A production-quality framework for measuring, profiling, and analyzing LLM inference performance using vLLM, async load generation, and GPU monitoring.
- Problem Statement
- Why LLM Inference Benchmarking Matters
- Architecture
- Setup
- Running the vLLM Server
- Running Benchmarks
- Metrics Definitions
- Experiment Methodology
- Hardware & Software Environment
- Results
- Graphs
- Analysis
- Failure & OOM Analysis
- Limitations
- What I Learned
- Future Improvements
Deploying large language models in production requires more than getting a model to generate text. The difference between an LLM system that works and one that works at scale comes down to understanding and controlling:
- Latency: How long does a user wait for the first token? For the full response?
- Throughput: How many requests per second can the system handle?
- Memory: How does VRAM scale with model size, batch size, and context length?
- Reliability: Does the system degrade gracefully under load, or crash with OOM errors?
This project builds a reproducible, instrumented benchmarking framework that answers these questions empirically — with actual measurements on real GPU hardware, not estimated numbers.
During training, throughput dominates: maximize GPU utilization over long jobs. During inference, the constraints differ fundamentally:
- Requests arrive unpredictably and must be served with bounded latency.
- Autoregressive generation is inherently sequential per token.
- Memory is split between fixed model weights and dynamic KV cache (grows per request).
- Users perceive latency; they don't care about GPU utilization directly.
Sequential benchmarks report average response time but miss:
- How continuous batching raises GPU utilization at concurrent load.
- How queuing latency explodes when the server is saturated.
- How context length changes KV-cache memory pressure and prefill time.
- How tail latency (p99) matters more than mean for SLA compliance.
vLLM introduced PagedAttention — which manages KV cache like virtual memory — and continuous batching, which dynamically interleaves requests rather than waiting for fixed batches. This makes benchmarking vLLM both practically relevant and technically rich.
┌──────────────────────────────────────────────────────────────────────┐
│ LLM Benchmarking Framework │
│ │
│ ┌─────────────┐ ┌──────────────────────────────────────────────┐ │
│ │ configs/ │───▶│ Experiment Runner │ │
│ │ *.yaml │ │ (experiments/*.py) │ │
│ └─────────────┘ └──────────┬───────────────────────────────────┘ │
│ ┌─────────────────┼──────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────────────┐ ┌─────────────────┐ ┌──────────────────────┐ │
│ │ Workload Gen │ │ Async Client │ │ GPU Monitor │ │
│ │ (workload.py) │ │ (client.py) │ │ (gpu_monitor.py) │ │
│ └───────┬───────┘ └────────┬────────┘ └──────────┬───────────┘ │
│ ▼ ▼ ▼ │
│ ┌───────────────┐ ┌─────────────────┐ ┌──────────────────────┐ │
│ │ Prompt List │ │ vLLM OpenAI API │ │ pynvml timestamped │ │
│ │ (token-aware) │ │ POST /v1/comp. │ │ GPU util + VRAM │ │
│ └───────────────┘ └────────┬────────┘ └──────────┬───────────┘ │
│ ▼ │ │
│ ┌──────────────────────────────────▼──────────┐ │
│ │ MetricsAggregator │ │
│ │ TTFT · TPOT · E2E · p50/p95/p99 │ │
│ │ throughput · GPU util · peak VRAM │ │
│ └──────────────────┬──────────────────────────┘ │
│ ▼ │
│ ┌───────────────────────┐ │
│ │ Results Storage │ │
│ │ results/raw/*.json │ │
│ │ results/processed/ │ │
│ └───────────┬────────────┘ │
│ ▼ │
│ ┌───────────────────────┐ │
│ │ Analysis & Plots │ │
│ │ analyze_results.py │ │
│ │ plot_results.py │ │
│ └───────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘
(separate process)
┌──────────────────────────────┐
│ vLLM Server │
│ OpenAI-compatible HTTP API │
│ POST /v1/completions │
│ GET /health │
└──────────────────────────────┘
llm-inference-benchmark/
├── README.md
├── requirements.txt
├── pyproject.toml
├── Dockerfile
├── .gitignore
├── configs/
│ ├── baseline.yaml Single-concurrency baseline
│ ├── concurrency.yaml Sweep [1, 2, 4, 8, 16, 32]
│ ├── context_length.yaml Input length sweep
│ ├── precision.yaml FP32 / FP16 / BF16
│ └── quantization.yaml INT8 / AWQ comparison
├── server/
│ ├── launch_vllm.sh Server launch script
│ └── server_config.py Typed config + GPU capability checks
├── benchmark/
│ ├── client.py Async HTTP client (SSE streaming)
│ ├── workload.py Deterministic prompt generation
│ ├── metrics.py TTFT, TPOT, E2E, percentiles
│ └── run_benchmark.py CLI entrypoint
├── profiling/
│ ├── gpu_monitor.py Background NVML polling
│ └── memory_profiler.py VRAM snapshots + OOM detection
├── experiments/
│ ├── concurrency.py Experiment 1
│ ├── context_length.py Experiment 2
│ ├── precision.py Experiment 3
│ ├── quantization.py Experiment 4
│ └── caching.py Experiment 5
├── analysis/
│ ├── analyze_results.py Aggregate JSON → CSV
│ └── plot_results.py Generate matplotlib figures
├── tests/
│ ├── test_metrics.py Formula unit tests (no GPU needed)
│ ├── test_workload.py Workload generation tests
│ └── test_client.py Client tests with mocked server
├── docs/
│ └── interview_notes.md Technical Q&A
└── results/
├── raw/ Per-run JSON (gitignored)
├── processed/ Aggregated CSVs (gitignored)
└── plots/ PNG figures (selectively committed)
| Requirement | Minimum | Notes |
|---|---|---|
| Linux | Ubuntu 20.04+ | vLLM is Linux-only |
| NVIDIA GPU | 4 GB VRAM | OPT-125M fits in ~4 GB fp16 |
| NVIDIA Driver | ≥ 525 | For CUDA 12.x |
| Python | 3.11 | |
| Docker | 24.0+ | Optional |
| NVIDIA Container Toolkit | latest | For docker run --gpus all |
Windows: vLLM requires Linux. Use WSL2 with GPU passthrough or a cloud instance.
python3.11 -m venv .venv
source .venv/bin/activatepip install torch==2.3.1 --index-url https://download.pytorch.org/whl/cu121
python -c "import torch; print(torch.cuda.get_device_name(0))"pip install vllm==0.5.4pip install -r requirements.txt
pip install -e ".[dev]"python -c "
import torch, pynvml
print('PyTorch:', torch.__version__)
print('GPU:', torch.cuda.get_device_name(0))
pynvml.nvmlInit()
h = pynvml.nvmlDeviceGetHandleByIndex(0)
m = pynvml.nvmlDeviceGetMemoryInfo(h)
print(f'VRAM: {m.total/1e9:.1f} GB total')
"Start the server before running any benchmark script.
bash server/launch_vllm.shMODEL=TinyLlama/TinyLlama-1.1B-Chat-v1.0 DTYPE=float16 bash server/launch_vllm.shcurl -s http://localhost:8000/health # Returns 200 when readyexport HF_HOME=/data/hf_cache # Control download location
export HUGGING_FACE_HUB_TOKEN=hf_... # For gated modelspython benchmark/run_benchmark.py --config configs/baseline.yamlpython benchmark/run_benchmark.py --config configs/baseline.yaml --concurrency 8python experiments/concurrency.py --config configs/concurrency.yaml
python experiments/context_length.py --config configs/context_length.yaml
python experiments/precision.py --config configs/precision.yaml
python experiments/quantization.py --config configs/quantization.yaml
python experiments/caching.pypython analysis/analyze_results.py --input results/raw/ --output results/processed/
python analysis/plot_results.py --input results/processed/ --output results/plots/pytest tests/ -vAll formulas are implemented in benchmark/metrics.py.
TTFT = t_first_token - t_request_start
Measures prefill latency — how long until the first output token arrives. Dominated by input sequence length (longer prompt → more prefill FLOPs) and server queue depth.
TPOT = (t_completion - t_first_token) / (n_output_tokens - 1)
Valid only when n_output_tokens >= 2. Measures the decode step cost per token.
Primarily affected by model size and concurrent batch size.
E2E = t_completion - t_request_start
Total wall-clock time including network, queuing, prefill, and decode.
requests_per_sec = n_successful / total_wall_time_sec
output_tokens_per_sec = sum(output_tokens) / total_wall_time_sec
p50 = median latency (50th percentile)
p95 = tail latency experienced by worst 5% of requests
p99 = tail latency experienced by worst 1% of requests
p95/p99 matter for SLA compliance — a system can have excellent mean latency but unacceptable tail latency when the server is under load.
Each experiment follows the scientific method:
Hypothesis → Controlled Experiment → Measurement → Analysis → Conclusion
Exactly one variable changes per experiment. All other parameters (model, dtype, input length, output length, concurrency, seed) are held constant and documented in each YAML.
The vLLM server is restarted between experiments that change server-side configuration (dtype, quantization). This prevents cache contamination across runs.
Fill in after running on GPU hardware. Run the commands below and paste the output.
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
python -c "import torch, vllm, sys; print(sys.version, torch.__version__, vllm.__version__)"
nvcc --version | grep release| Component | Value |
|---|---|
| GPU | [pending] |
| GPU VRAM | [pending] |
| NVIDIA Driver | [pending] |
| CUDA Version | [pending] |
| Python | [pending] |
| PyTorch | [pending] |
| vLLM | [pending] |
| Model (default) | facebook/opt-125m |
| OS | [pending] |
Results pending benchmark execution on GPU hardware.
All values in this section will be populated after running experiments. No values have been estimated or fabricated. The measurement framework is fully implemented and ready to run.
| Metric | Value |
|---|---|
| TTFT mean (ms) | pending |
| E2E mean (ms) | pending |
| E2E p95 (ms) | pending |
| E2E p99 (ms) | pending |
| Throughput (req/s) | pending |
| Output tokens/s | pending |
| GPU util mean % | pending |
| Peak VRAM (GB) | pending |
| Failed requests | pending |
| Concurrency | Throughput (req/s) | TTFT p50 (ms) | E2E p95 (ms) | GPU Util % | VRAM (GB) |
|---|---|---|---|---|---|
| 1 | pending | pending | pending | pending | pending |
| 2 | pending | pending | pending | pending | pending |
| 4 | pending | pending | pending | pending | pending |
| 8 | pending | pending | pending | pending | pending |
| 16 | pending | pending | pending | pending | pending |
| 32 | pending | pending | pending | pending | pending |
| Input Tokens | TTFT mean (ms) | E2E mean (ms) | Throughput (req/s) | Peak VRAM (GB) |
|---|---|---|---|---|
| 128 | pending | pending | pending | pending |
| 256 | pending | pending | pending | pending |
| 512 | pending | pending | pending | pending |
| 1024 | pending | pending | pending | pending |
| 1792 | pending | pending | pending | pending |
| dtype | VRAM (GB) | Throughput (req/s) | E2E mean (ms) | Failed |
|---|---|---|---|---|
| float32 | pending | pending | pending | pending |
| float16 | pending | pending | pending | pending |
| bfloat16 | pending | pending | pending | pending |
| Config | VRAM (GB) | Throughput (req/s) | E2E mean (ms) | TTFT mean (ms) |
|---|---|---|---|---|
| fp16 baseline | pending | pending | pending | pending |
| bitsandbytes INT8 | pending | pending | pending | pending |
| Config | TTFT p50 (ms) | TTFT p95 (ms) | Throughput (req/s) |
|---|---|---|---|
| Caching disabled | pending | pending | pending |
| Caching enabled | pending | pending | pending |
Graphs pending benchmark execution. Generated by
analysis/plot_results.pydirectly from measured data.
results/plots/concurrency_vs_throughput.pngresults/plots/concurrency_vs_p95_latency.pngresults/plots/concurrency_vs_ttft.pngresults/plots/context_length_vs_ttft.pngresults/plots/context_length_vs_vram.pngresults/plots/precision_vs_vram.pngresults/plots/quantization_comparison.png
Analysis pending benchmark execution. Each section will describe what the data showed — observations from actual measurements, not anticipated outcomes.
-
Model weights — fixed at load time.
weight_bytes = n_params × bytes_per_dtypeOPT-125M in fp16: ~250 MB. -
KV cache — grows dynamically per request per token.
kv_per_token = 2 × n_layers × n_heads × head_dim × bytes_per_dtype -
Activations — proportional to batch_size × seq_len × hidden_dim. Released after each layer.
-
CUDA runtime buffers — 200 MB – 1 GB depending on ops and driver version.
# 1. Check error message — "tried to allocate X.X GiB" tells you which allocation failed
# 2. Check VRAM per process
nvidia-smi --query-compute-apps=pid,used_memory --format=csv
# 3. Reduce GPU memory utilization
# --gpu-memory-utilization 0.75
# 4. Reduce max sequence length (shrinks KV cache budget)
# --max-model-len 1024
# 5. Reduce concurrency to lower simultaneous KV cache usage
# 6. Enable quantization to shrink model weight footprintResults pending. The memory pressure experiment gradually increases concurrency and context length while recording the first configuration that produces failures.
- Single GPU only — multi-GPU tensor parallelism is out of scope for v0.1.
- Small default model — OPT-125M is chosen for accessibility; larger models may show different bottlenecks.
- No quality measurement — only serving performance is benchmarked.
- Max-rate only — no Poisson-distributed arrival rate simulation.
- vLLM 0.5.4 pinned — newer versions may add features or change the API.
Will be completed after running all experiments. Observations will come from actual measurement data.
- Controlled request arrival rate (Poisson distribution).
- Multi-model comparison (OPT-125M vs TinyLlama vs Mistral-7B).
- Tensor parallelism across multiple GPUs.
- Speculative decoding analysis.
- Prometheus + Grafana real-time dashboard.
- Automated regression CI gate (fail if latency regresses > 10%).
- Quality benchmarks (MT-Bench, MMLU) alongside serving benchmarks.
# Record environment before every run
python -c "
import torch, vllm, sys, subprocess
print('Python:', sys.version)
print('PyTorch:', torch.__version__)
print('vLLM:', vllm.__version__)
print('GPU:', torch.cuda.get_device_name(0))
"
# All raw results include a metadata.environment block with these values.
# Use fixed random_seed: 42 (default in all configs) for reproducible prompts.All benchmark numbers come from actual GPU execution. "pending" means not yet run.