AI Infrastructure · LLM Inference Engineering

From inference performance to infrastructure decisions.

I am Feihong Guo. During my AI Infrastructure internship at Baidu AI Cloud, I benchmarked and tuned SGLang serving for DeepSeek-V4-Pro, V4.1-Flash and Kimi K3 across H200, B300 and GB300. I used workload measurements to compare deployments, size Prefill and Decode capacity, and trace performance and correctness failures to the runtime.

Deploy → measure → explain → improve → validate → size.
Throughput, latency, correctness, and stability belong in the same decision.

Authored and merged in SGLang · DP health routing #35721 · PP stream lifetime #47614 · FP8 fusion #43786 · EPLB validation #48404

SGLangDistributed LLM ServingGPU / NPU PerformanceCapacity & TCO
Feihong Guo
Feihong Guo · 郭飞鸿
Zhengzhou University · AI Infrastructure

Baidu AI Cloud · SGLang

Serving performance.
Runtime correctness.

I reproduce serving failures with controlled workloads, trace them through scheduling and device execution, and carry verified repairs upstream. My four merged PRs cover DP routing, PP tensor lifetime, FP8 kernel fusion and EPLB configuration validation.

Authored · merged

#35721 DP health routing
#47614 PP stream lifetime
#43786 FP8 fusion
#48404 EPLB validation

Formal technical reviews

#32467 planner-race repair · merged
#32183 verifier window · open

Collaborative repair · merged

#39487 DCP KV recovery
My diagnosis, draft fix, and validation; maintainer integration.

01 · PIPELINE / STREAM LIFETIME

Kimi-K3 produced long token-0 runs when I deepened the asynchronous PP Decode queue. I traced the failure to receive buffers being reused before the forward stream read them. After fixing tensor lifetime, affected requests fell from 51/192 and 98/192 to 0/192 at depths 2 and 3. PR #47614 merged on 2 November 2026.

Expand the investigation: reproduction, checksum evidence and the fix

1. Turn an apparent speedup into a correctness test. Deeper asynchronous PP queues appeared faster, but many Kimi-K3 requests emitted long runs of token 0. On PP2 × TP4, depth 1 produced no affected requests in the tested run. Raising the depth to 2 or 3 made the fault reproducible. Synchronizing after each launch also suppressed it, which directed the investigation to queued work and tensor lifetime.

2. Measure where the data changed. On the downstream stage, I checked the same proxy tensors just after receive on the communication stream and just before use on the forward stream. Two depth-2 runs had 76/7,498 and 378/7,498 checksum mismatches. Keeping the previous tensor reference alive until the next receive had allocated its buffers gave 0/7,499 mismatches with a similar observed queue depth. Checksum and reference-lifetime controls ↗

3. Extend storage lifetime at the shared launch boundary. CPU scheduling could drop the old reference while a forward-stream read was still queued. The next receive could reuse the same storage and overwrite that microbatch's activations. In _pp_launch_batch, I used record_stream(self.forward_stream) on each non-CPU proxy tensor before run_batch. This tells the allocator to retain storage until forward-stream use completes. Existing stream dependencies still determine execution order.

Kimi-K3 · 8×B300 · v0.5.22 · fake P/D · C64 · requests with at least 20 consecutive token-0 outputs
ConfigurationAsync depthBeforeAfter
PP2 × TP4251/1920/192
PP2 × TP4398/1920/192
PP2 × TP4 + DCP4259/1920/192
PP4 × TP236/1920/192
PP4 × TP2419/1920/192

The table reports single-node B300 correctness tests using fake transfer. The PR records the tested patch variants and separate communication-overlap follow-ups. The corrupted-output runs served as failure reproductions.

PR #47614authored · merged · 2 Nov 202651–98 → 0affected requests / 192 · PP2 depths 2 and 3 · tested B300 stack
02 · ROUTING STATE

Repeated DeepSeek-V4-Pro Prefill runs on 8×B300 showed intermittent throughput drops. I traced the trigger to generation-health requests changing shared DPC user-routing state before scheduler admission. After isolating health dispatch, all four patched health-ON runs reached 94.56–94.70k input tok/s. PR #35721 shipped in v0.5.22.

Expand the investigation: hypotheses, traces and 2×2 validation

1. Separate scheduling from GPU execution. Identical long-Prefill runs intermittently dropped throughput. I first tested DP Attention, chunk sizing and PrefillDelayer because LOW runs showed rank progress gaps and sustained mixed/delay activity. A Delayer admission-order backport still produced both GOOD and LOW, so I investigated whether a rank received less work or took longer to execute it. Backport counterexample ↗

2. Check whether observation perturbs the run. A continuous alternating 4 OFF / 4 ON DeviceTimer test gave healthy-state means of 94,563 and 94,569 input tok/s. The sample was too small to settle LOW entry probability. Moving from 5 s to 250 ms metrics sampling separated observed chunk completions, and a benchmark start marker made the fast window cover the onset. DeviceTimer A/B ↗

3. Trace the request across the control plane. Client POST, tokenizer, DataParallelController and scheduler admission traces exposed a hidden ordering: a generation-health request could advance shared user round-robin or touch load-budget state before a busy scheduler skipped it. The patch isolated health dispatch and preserved explicit-rank and P/D routing. The initial symptom report was filed on 18 August and the patch opened on 20 August. Health trigger evidence ↗

4. Isolate the cause with a 2×2. On 26 August I held the image, workload, DP Attention, PrefillDelayer and external GET /health polling fixed. Each round started a fresh server and varied only the patch and generation-health request creation. Four-arm experiment ↗

8×B300 · DeepSeek-V4-Pro · TP8/DP8 · 60k input / 1 output · C16 · 160 requests
ConfigurationResultInput tok/s
Original + generation health ON2/8 LOW68,403 / 82,883 LOW
Original + generation health OFF4/4 GOOD94,783–94,847
Patched + generation health ON4/4 GOOD94,558–94,699
Patched + generation health OFF4/4 GOOD94,850–94,934

What the follow-up revealed. In the severe LOW run, the active-window median of summed scheduler running requests was 48 vs 72–80 in the healthy controls. This is request inventory, measured separately from kernel batch dimensions. A milder LOW still had a median of 80. Rank dispatch traces and the four-arm result establish the causal link. Delayer timeouts persisted in healthy patched runs. Revised mechanism ↗

Full public investigation · #35241 ↗
30 / 30targeted CPU tests · rebased tree4 / 4B300 patched health-on repeats · tested workload
03 · INVESTIGATION / REVIEW

From July to August, I reported DSpark replay and startup failures, built targeted reproductions, and contributed experiments to the final planner fix. My technical review on PR #32467 validated the no-init variant with a controlled kernel comparison: 13,111/30,000 out-of-bounds plans → 0/30,000.

Expand the investigation: original reports, isolation and regression coverage

1. Follow the failure across replay and startup. On 13 July I filed #31023 for a TP8 compact target-verify replay failure, then opened diagnostic #31455 to bisect synchronization and observe graph metadata. In August I filed #33356 for an independent startup-capture reproducer on v0.5.16. The surfacing rank, shape and kernel varied, so I traced the inputs to those kernels back to their producer.

2. Separate TP agreement, capture ordering and invalid plans. After #31195, all 120/120 TP8 trials agreed on verify budget, lengths and graph key. A separate four-arm startup experiment compared planner patch #32467 and capture-ordering patch #33795. Full capture succeeded in 0/7 without either, 1/6 with capture ordering alone, and 3/3 with the planner patch alone or both. Each successful planner-patch run captured all 35 shapes. Startup matrix ↗

3. Observe the bad index at the producer and test the proposed fix. A single-GPU reproducer called the real GPU-input planner and checked max(valid ragged_id) < sum(extend_lens). Redundant initialization of shared warp scratch could overwrite another warp's reduction, misclassifying ragged input as uniform. One bad plan reached index 383 for only 360 valid rows. I tested DarkSharpness’s no-init suggestion in the same session, with isolated JIT caches and 30,000 calls per variant. Under the existing 32-warp launch, each warp writes its slot before the existing post-reduction barrier.

Single-GPU planner regression · same B30Z session · one uniform and two ragged inputs
VariantOut-of-bounds plans
A · original initialization13,111 / 30,000
B · initialization + added barrier0 / 30,000
C · remove redundant initialization0 / 30,000

4. Check sustained replay and ragged coverage separately. The disclosed v0.5.16 backport stack completed 18,560/18,560 decode-only requests across C64–C1024 with Graph replay observed. Natural traffic in that sweep produced uniform verify lengths. A separate forced-budget probe observed 39/43 genuinely ragged scheduling blocks, with no GPU fault or budget violation in that window. The higher-concurrency follow-up up to C2560 measured admission pressure at a fixed MRR of 1024. Runtime and coverage details ↗

I reported and reproduced the failures, contributed the controlled experiments and submitted the formal review. EanWang211123 authored the merged no-init fix in #32467. I tracked separate follow-ups for Breakable CUDA Graph ordering (#34286) and P/D draft-input handoff coverage (#34410).

PR #32467technical review · merged kernel fix13,111 → 0OOB plans · 30,000 calls per A/B/C variant
04 · KV STATE

Reported Issue #38645, localized the logical-to-physical KV index mismatch in DCP retraction backup/restore, and implemented and validated draft fix #38790. Unpatched DCP8 failed at its first retraction; my patched B300 run completed 2,048/2,048 requests with eight retractions. Maintainer kpham-sgl integrated the fix through a shared DCP layout helper, added compatibility and runtime checks, and merged #39487; I closed my superseded draft.

My repair and the maintainer’s integration

My implementation and six CPU tests covered rank ownership, local-row mapping, and restoration into newly allocated KV locations; four KDA/Mamba GPU regressions checked the separate recurrent-state path. The maintainer reused a shared layout helper, handled already-resolved physical indices and partial-page prefixes, and added B200 forced-retraction and logprob checks. My B300 evidence validates the documented local patch stack.

My validated draft · #38790 ↗ · Maintainer integration · merged #39487 ↗
2,048 / 2,048v0.5.17 patched requests · 8 real retractions6 / 6CPU correctness tests
05 · DISTRIBUTED COMMUNICATION

I reproduced degenerate Kimi-K3 outputs in graph-captured DCP a2a on two-node GB300. Disabling NCCL Graph Registration reduced degenerate outputs from 12/12 to 0/12 in the local probe. Upstream #43216, authored by zhihengy, introduced a broader default guard that covers this configuration.

Expand the investigation: local A/B, upstream guard and closure

Local diagnosis and alternatives. In the original TP8/DCP8 2×2 probe, only registration ON with temporary exchange buffers produced degenerate outputs. The startup guard in #45371 also completed 640/640 requests in a separate local C128 run. Persistent buffers (#45824) passed an early smoke test, but the no-guard configuration later produced 0/10 valid answers under load in a separate TP8/attention-DP2/DCP4 deployment. I withdrew the apparent speedups and closed that proposal unmerged.

Upstream resolution. #43216 was merged on 22 October 2026. When a deployment spans multiple TP, EP or DP ranks and any CUDA Graph phase is enabled, it defaults NCCL_GRAPH_REGISTER to 0, preserving explicit overrides. The original graph-enabled GB300 DCP a2a configuration meets those conditions, so I closed the narrower #45371 unmerged and marked Issue #45367 completed.

Closure is based on source-level coverage of the original configuration and the earlier local experiments. A matched GB300 end-to-end rerun on current main remains outstanding, and the precise NCCL replay-time mechanism remains unconfirmed.

12/12 → 0/12degenerate outputs · local registration ON vs OFF#43216upstream default guard · merged
Explore the public PR, review, and investigation index

Selected upstream records · contribution and evidence index

Reproduce, isolate, validate, upstream

The framework may change; the research method should not. I treat public issues, PRs, regressions, and RFCs as a reproducible evidence trail that distinguishes symptom, causal mechanism, repair, and validation.

01Reproduce
02Isolate
03Validate
04Upstream

Authored contributions · merged

PR #47614Authored · merged · 2 Nov 2026

Record received PP tensors on the forward stream

PR #35721Authored · merged · v0.5.22

Separate generation health state from user dispatch

PR #43786Authored · merged · 4 Nov 2026

Fuse inverse RoPE into CUDA FP8 wo_a quantization

PR #48404Authored · merged · 7 Nov 2026

Reject incompatible EPLB recorder modes at startup

Technical reviews & collaborative repair

PR #32467Technical review & causal validation · merged

My technical review of the final no-init fix ↗

Review · PR #32183Formal COMMENTED review · PR open

Separate verifier-window correctness from other DSpark failures

PR #39487Collaborative fix · maintainer-authored · merged

Land the DCP retraction fix following my report and draft

Authored engineering work · open

PR #45384Authored · open

Lazily initialise CUDA VMM multimodal transport

PR #45584Authored · open

Serialise multi-tokenizer general warmups

PR #34410Authored tests · open

Pin the P/D decode draft-input handoff contract

PR #34286Authored patch · open

Synchronize graph construction after final warmup

Investigation, RFCs & supporting records

PR #45371Authored investigation · closed unmerged

DCP graph-registration guard

PR #45824Authored investigation · closed unmerged

Persistent DCP all-to-all buffers

Issue #31023Authored report & evidence · closed

Isolate DSpark target-verify replay failures on TP8

Issue #33356Authored report & validation · closed

Separate plan-kernel corruption from graph capture ordering

Roadmap #34297Authored investigation roadmap · closed

Organize DSpark correctness contracts and validation gates

RFC #32432Authored design proposal · closed

Propose metadata, workspace, and stream-ownership contracts

Issue #38645Authored report & root-cause analysis · closed

Trace the DCP retraction logical-to-local KV index mismatch

SGLang RFC #44697Authored profiling proposal · open

Separate routing imbalance from placement and serving costs

DeepSeek EPLB #29Authored evaluation discussion · open

Discuss replica costs and the boundary with model routing

PR states checked 8 November 2026. Full contribution index ↗

SGLang deployment & performance

Serving benchmarks and deployment choices

Built reproducible phase-isolated and end-to-end benchmarks across input/output lengths and concurrency. Compared TP, PP, EP, CP/DCP, DP Attention, EAGLE3/DSpark, EPLB, CUDA Graph and Chunked Prefill. Used KV cache and Mamba state pools, MRR/MF and memory pressure to identify feasible configurations.

01

Long-context serving benchmarks

Benchmarked long-context Prefill and Decode on B300 and GB300, using throughput and latency to guide parallelism and deployment choices.

View selected benchmark work
Selected SGLang benchmark work
ModelBenchmark focus
DeepSeek-V4-ProPrefill and Decode tuning, comparing batching, cache capacity, and speculative decoding.
DeepSeek-V4.1-FlashPrefill and expert balancing under synthetic and real-token loads. RFC #44697 ↗
Kimi K3Prefill: compared tensor/expert parallelism with pipeline parallelism. Decode: evaluated DCP and pipeline trade-offs in throughput, per-token latency, and runtime correctness.

Related K3 runtime investigations · PR #47614 ↗ · Issue #45367 ↗

02

Estimate before the full-node test

Extended an InferFlux-based capacity model for long-context inference and used workload, compute/bandwidth limits, cache memory, parallel layout, and efficiency priors to narrow the test space. For Kimi K3, examined DCP and the capacity/rollback boundaries of KV and SSM state pools.

The output is a capacity estimate, not a guaranteed throughput number. Gaps between prediction and measurement help prioritize memory, communication, batching, or runtime investigation.

03

Parallelism is a workload decision

Compared TP8/DP8 and TP8/CP8 for DeepSeek-V4-Pro, and TP8/EP8/MegaMoE and PP8 for Kimi K3. A separate GB300 CP8 V4-Pro Prefill configuration measured median TTFT of 0.666 s at 60k and 1.516 s at 128k input, with one output token.

Select hardware and parallel layout against the target workload and SLO. Compare throughput, TTFT/TPOT and resource use at the qualifying operating point.

04

EPLB: measure balance, then test throughput

Separated logical expert popularity, physical placement, replica dispatch, padding and kernel time. Earlier V4-Pro B300 work motivated a follow-up on real-token V4.1-Flash 64K/C32 Prefill workloads.

In the reported V4.1-Flash comparison, frozen placement lowered rank-count CV from 0.149 to 0.047 and average MegaMoE duration from 7.512 to 7.180 ms. Input throughput rose 0.95%, compared with 1.01% for the synthetic balanced-routing diagnostic.

Routing diagnostics and recorder validation

The uniform-routing arm changes logical expert selection and serves as a performance diagnostic. The RFC records its measurement boundaries. Separately, #48404 rejects incompatible recorder modes at startup after an online rebalance crash was reproduced.

My SGLang profiling RFC · #44697 ↗ · My DeepSeek EPLB discussion · #29 ↗
05

Microbenchmarks and runtime profiling

Used a MegaMoE microbenchmark as a component reference for the serving trace. A 32K-chunk microbenchmark measured about 5.6 ms. In a separate V4.1-Flash real-input Prefill trace, mean main MegaMoE kernel times were 6.9–7.5 ms across ranks. Used nsys to follow execution and communication, and ncu to examine selected kernels.

Merged #43786 ↗ fuses inverse RoPE into CUDA UE8M0 FP8 activation quantization, preserving BF16 rounding and packed scales. Upstream B200 kernel tests passed 7/7, including numerical equivalence and CUDA Graph replay with updated inputs.

The microbenchmark and serving trace have different execution boundaries. The trace includes internal waits, so the gap guides attribution. The historical B300 fusion backport measured +1.61% Prefill throughput. Full-model performance of the final upstream implementation remains unmeasured.

06

Reproducible benchmark environments

Maintained SGLang container images and CI/CD workflows for multi-node experiments. Automated dependency setup, service lifecycle and benchmark collection, and recorded topology, cache budgets and validation settings for repeatable regressions.

Huawei ICT Academy · ZZU Supercomputing

The same method,
across NPU and CPU.

After the SGLang work, a complementary systems track: Ascend inference engines and kernels, plus Kunpeng memory and parallel performance. Hardware details matter when they change the serving path.

Ascend serving engines

Worked with xLLM and vLLM-Ascend on Ascend 910B: model support, MTP, ACL Graph, cache capacity, and benchmark configuration. Qwen3.5-9B throughput moved 2,301 → 30,451 tok/s at C960 in a combined configuration comparison.

A separate Gemma 4 E2B deployment comparison measured 1,159.53 → 1,274.00 tok/s with Full ACL Graph. The two benchmarks have different baselines.

Kernels into a valid runtime path

Used Ascend C, Triton, and TileLang to connect tiling and fusion to the Da Vinci data path: aligned transfers, on-chip buffers, compute, and synchronisation. Specialised paths need shape and layout checks, precision validation, and a fallback before serving integration.

Kernel timing and system throughput are measured separately. A faster local operator needs a reachable engine path and an end-to-end comparison.

Kunpeng performance foundations

On Kunpeng 920, tuned SVE/NEON, NUMA affinity, MPI/OpenMP and FFT plan reuse. The 256³ FFTW workload moved from 80.97 to 555.00 GFLOP/s.

CPU workloads are independent measurements, rather than evidence of an NPU or LLM serving speedup.

Ascend 910C · MiniCPM-o

Trained a DSpark draft with DeepSpec on B300/DP8 and exported it via GGUF to Ascend 910C. The project also covered prefix KV reuse, ACL Graph replay and TileLang AOT fusion in ggml-CANN. Matched-harness Core RTF went from 0.6754 to 0.4829 on the same 120 s duplex workload. Project ↗ · Packed vision projection #144 · open · Flow/Vocoder overlap #146 · draft

Performance → capacity → economics

Hardware selection, P/D sizing
and deployment economics

Compare serving performance, resource utilization and deployment cost under a defined SLO. Use measured phase capacity for an initial P/D allocation, then validate the combined service path.

01 / SLO

Find the feasible operating point

Sweep concurrency and configuration on the target request mix. Compare TTFT, TPOT, request throughput, memory pressure, failures, and output correctness. Select an operating point that meets the specified latency, quality, and stability criteria.

q_instance = measured requests/s
at the qualifying configuration

Prefill input tok/s, Decode output tok/s, and requests/s describe different quantities. Request lengths and phase balance must stay explicit.

02 / CAPACITY

Size Prefill and Decode separately

Convert Prefill and Decode rates to per-instance requests/s under the same workload and SLO. Size both stages for peak demand, with headroom and redundancy for the deployment.

N_P = ceil(λ_peak / q_P)
N_D = ceil(λ_peak / q_D)
Minimum GPU count = N_P × g_P + N_D × g_D

Validate the resulting P/D ratio with real KV transfers, queueing and end-to-end TTFT/TPOT. Stage-isolated rates are sizing inputs.

03 / TCO

Price delivered service

Use one accounting period for compute rental or hardware depreciation, power, networking, software, and operations. Divide by the correctly delivered token volume under the chosen service criteria. Treat input and output separately when the accounting requires it.

Cost per 1M valid tokens
= period cost / valid tokens delivered × 10⁶

Utilisation, idle reserve, uptime, retries, and traffic shape affect unit cost. Rental and depreciation require a consistent accounting boundary.

Evidence for sizing comes from benchmarks, resource limits and runtime validation. These inputs support hardware selection and comparative cost scenarios. Full TCO requires actual prices, power, utilization, redundancy, operations and delivered traffic.

Methodology reference · performance engineering

Databricks · LLM Inference Performance Engineering: Best Practices ↗

A useful framework for relating Prefill/Decode, batching, memory bandwidth, and parallel execution to measured server behaviour.

Methodology reference · capacity and cost

NVIDIA · How Much Does Your LLM Inference Cost? ↗

Connects latency-constrained benchmark points to deployment capacity and a cost model. These references guide the method; my results remain tied to my own workloads.

Inference engineering practice

Measure the service.
Explain the bottleneck.

A repeatable loop across SGLang, NVIDIA GPUs, and Ascend: make the workload and execution path explicit, prove the result, then use it to make a resource decision.

How much correct, reliable serving capacity can this configuration deliver within its latency and resource budget?

01

Deploy & define

Pin model and framework versions, topology, request distribution, cache policy, precision, and resource limits. Confirm the intended backend and communication path.

02

Measure & profile

Separate P/D tests from end-to-end service tests. Use nsys for critical paths, CPU/GPU gaps, communication overlap, synchronisation, and graph coverage; use ncu for selected kernels.

03

Repair & validate

Test one causal hypothesis at a time. Check output quality, failure behaviour, memory boundaries, repeatability, and throughput/latency together. Carry the regression upstream.

04

Size & account

Use qualifying serving capacity for instance sizing, GPU allocation, and cost scenarios. Keep measured results, estimates, and unvalidated assumptions visible.

AI Infra is the connecting layer: model behaviour and workload shape meet runtime scheduling, cache state, kernels, interconnect, and deployment economics. My focus is building and validating that connection.
↑