vLLM: Incomplete CVE-2025-62164 remediation can be bypassed by concurrent prompt parts

Story map

Why it is Yellow

• CVSS 6.3 medium
• On your watchlist: PyPI

Affected packages

References

• https://github.com/advisories/GHSA-pr7f-p5mw-fc87
• https://github.com/vllm-project/vllm/security/advisories/GHSA-pr7f-p5mw-fc87
• https://nvd.nist.gov/vuln/detail/CVE-2026-73557
• https://github.com/vllm-project/vllm/pull/48583
• https://github.com/vllm-project/vllm/commit/793cf79c89d4049124e756915468ac30318f2e50
• https://github.com/vllm-project/vllm/releases/tag/v0.26.0
• https://github.com/advisories/GHSA-pr7f-p5mw-fc87

Original description

## Executive Summary The follow-up protection for CVE-2025-62164 is incomplete at vLLM revision `26587f9519e22a5c4549ead7595ad9ca3229c4fd`. It wraps serialized prompt-embedding reconstruction and dense conversion in `torch.sparse.check_sparse_tensor_invariants()`, but PyTorch 2.11.0 implements that context with save/enable/restore operations over process-global state. Two prompt-embedding parts in one `/v1/chat/completions` request are gathered concurrently on the event loop's default executor. When one context exits before the other loads its tensor, it can restore the global flag to `False` while the second part remains inside its guard. In a deterministic run against hash-verified source from the affected revision, the actual target loader rejected an invalid sparse payload as a negative control. The frozen chat tracker then scheduled benign and malicious parts on distinct `asyncio_0` and `asyncio_1` threads. The benign context exited, the malicious loader observed the invariant flag disabled, and `torch.load(weights_only=True)` reconstructed indices `[[10], [10]]` for a declared shape of `[3, 3]`. The run intercepted the target's `to_dense()` call before it operated on the invalid tensor. This primary trigger requires `--enable-prompt-embeds`, which is default-off, but it does **not** require `renderer_num_workers > 1`, a multimodal model, or `--enable-mm-embeds`. API authentication is optional in the stock server: middleware is installed only when CLI or environment API keys are supplied. The lab proves bypass of the follow-up guard, invalid sparse reconstruction, and guarded-sink reachability. Crash and memory-corruption consequences are conditional on the behavior documented by the published CVE. ## Background CVE-2025-62164 / [GHSA-mrw7-hf4f-83pf](https://github.com/vllm-project/vllm/security/advisories/GHSA-mrw7-hf4f-83pf) concerns client-controlled serialized `prompt_embeds` reaching `torch.load(weights_only=True)` and an invalid sparse tensor reaching `to_dense()`. The advisory attributes memory corruption, denial of service, and potential code execution to that historical unsafe operation. The remediation chronology matters for duplicate handling: - PR [#27204](https://github.com/vllm-project/vllm/pull/27204), merge commit [`58fab50d82838d5014f4a14d991fdb9352c9c84b`](https://github.com/vllm-project/vllm/commit/58fab50d82838d5014f4a14d991fdb9352c9c84b) on 2025-10-22, introduced the default-off `enable_prompt_embeds` gate. It did not add the sparse-invariant context. - Commit [`84e23d103d3483f944780d0d42bcf0993fd27e3a`](https://github.com/vllm-project/vllm/commit/84e23d103d3483f944780d0d42bcf0993fd27e3a) on 2025-12-15, titled `additional protection for CVE-2025-62164 (#30649)`, added the process-global sparse-invariant context around load, type check, and dense conversion. - Refactor commit [`f0a1c8453ad1c664c8a04c83fe545195fcd556eb`](https://github.com/vllm-project/vllm/commit/f0a1c8453ad1c664c8a04c83fe545195fcd556eb) on 2026-01-31 moved the guarded loader into `vllm/renderers/embed_utils.py` while preserving the same context. - Chat content-part commit [`14043dfecd35dd2f12b4d51eb9fa166184a0ca0f`](https://github.com/vllm-project/vllm/commit/14043dfecd35dd2f12b4d51eb9fa166184a0ca0f) on 2026-05-01 introduced `prompt_embeds` chat parts and the concurrent one-request schedule described here. This report therefore does not present the malformed sparse payload or `to_dense()` sink as new. It reports a distinct concurrency root cause and trigger: unsynchronized save/enable/restore of the process-global follow-up guard, reachable through the later multi-part chat scheduler. The affected revision pins PyTorch 2.11.0 in `pyproject.toml:10`. ## Vulnerability Details The target's `safe_load_prompt_embeds` performs the guarded operation in `vllm/renderers/embed_utils.py:16-39`: ```python with torch.sparse.check_sparse_tensor_invariants(): tensor = torch.load( BytesIO(pybase64.b64decode(embed, validate=True)), weights_only=True, map_location=torch.device("cpu"), ) if not isinstance(tensor, torch.Tensor): raise VLLMValidationError(...) tensor = tensor.to_dense() ``` The context is not request-local. With the global flag initially disabled, we can describe the verified interleaving: 1. Benign part A enters, saves `False`, and enables the flag. 2. Malicious part B enters, saves `True`, and leaves the flag enabled. 3. A completes its load and exits, restoring its saved `False` value. 4. B remains lexically inside its context but observes the actual global flag as `False`. 5. B's `torch.load(..., weights_only=True)` reconstructs the malformed sparse tensor. 6. The target reaches `tensor.to_dense()` before later rank, hidden-size, and dtype checks. `weights_only=True` constrains deserialization types; it does not compensate for a sparse invariant check that another request has disabled. The complete stock actor-to-sink chain, traced in the affected source, is: `POST /v1/chat/completions` (`vllm/entrypoints/openai/chat_completion/api_router.py:41-61`) -> `OpenAIServingChat.create_chat_completion` -> `_create_chat_completion` -> `render_chat_request` (`vllm/entrypoints/openai/chat_completion/serving.py:206-280`) -> `OnlineRenderer.render_chat` (`vllm/renderers/online_renderer.py:95-190`) -> `preprocess_chat` (`vllm/renderers/online_renderer.py:335-380`) -> `BaseRenderer.render_chat_async` (`vllm/renderers/base.py:1070-1105`) -> `HfRenderer.render_messages_async` (`vllm/renderers/hf.py:1049-1085`) -> `parse_chat_messages_async` (`vllm/entrypoints/chat_utils.py:1911-1945`) -> content-part `parse_prompt_embeds` and `_load_prompt_embeds_async` (`vllm/entrypoints/chat_utils.py:1099-1120`) -> `AsyncMultiModalItemTracker.resolve_items` (`vllm/entrypoints/chat_utils.py:818-835`) -> `asyncio.gather` of both prompt parts -> `safe_load_prompt_embeds_async` -> `make_async` -> `loop.run_in_executor(executor=None, ...)` (`vllm/utils/async_utils.py