vLLM Config: Advanced Engine Deployment#

--vllm-config is a YAML-based configuration system for fine-grained control over vLLM engine deployment in vime. It enables multi-model serving, Prefill-Decode (PD) disaggregation, Encoder-Prefill-Decode (EPD) disaggregation for vision-language models, heterogeneous server groups, and can even serve as a standalone vLLM launcher for complex inference topologies.

Architecture Overview#

In the default setup (without --vllm-config), vime deploys a single model behind a single router with uniform server groups:

architecture overview

With --vllm-config, the vLLM deployment expands into a multi-model, multi-router topology:

vllm-config architecture

Key design principles:

Each model gets its own router. Models are isolated at the routing layer, allowing independent load balancing and fault tolerance.
Server groups within a model can be heterogeneous. Different groups can have different TP sizes, worker types (prefill/decode/encoder/regular), and vLLM engine argument overrides.
Weight sync is per-model. Only models with update_weights: true receive weight updates from training. Frozen models (reference, reward, etc.) are served as-is.

Config Format#

The config file is a YAML document with a top-level vllm key containing a list of model definitions:

vllm:
  - name: <model_name>              # Required. Unique identifier for this model.
    model_path: <path>              # Optional. HF checkpoint path. Defaults to --hf-checkpoint.
    update_weights: <bool>          # Optional. Whether to sync weights from training. Auto-inferred.
    num_gpus_per_engine: <int>      # Optional. Default TP size for all groups in this model.
    server_groups:                  # Required. List of server group configurations.
      - worker_type: <type>         # Required. One of: regular, prefill, decode, encoder, placeholder.
        num_gpus: <int>             # Required. Total GPUs allocated to this group.
        num_gpus_per_engine: <int>  # Optional. TP size override for this group.
        overrides: <dict>           # Optional. vLLM EngineArgs field overrides.

Field Reference#

Model-Level Fields#

Field	Type	Default	Description
`name`	`str`	Required	Unique name for this model (e.g., `"actor"`, `"ref"`, `"reward"`). Used as the key in `args.vllm_model_routers`.
`model_path`	`str`	`args.hf_checkpoint`	HuggingFace checkpoint path. All server groups within a model must use the same model path.
`update_weights`	`bool`	Auto	Whether this model receives weight updates from training. When not set, automatically inferred: `true` if `model_path` matches `--hf-checkpoint`, `false` otherwise.
`num_gpus_per_engine`	`int`	`args.rollout_num_gpus_per_engine`	Default TP size for server groups in this model. Individual groups can override.
`server_groups`	`list`	Required	List of `ServerGroupConfig` entries defining the engine topology. (`engine_groups` is accepted as a backward-compatible alias.)

Server Group Fields#

Field	Type	Default	Description
`worker_type`	`str`	Required	Engine type: `regular` (standard), `prefill` (PD prefill worker), `decode` (PD decode worker), `encoder` (EPD vision-encoder worker), or `placeholder` (reserve GPU slots without launching engines).
`num_gpus`	`int`	Required	Total number of GPUs for this group. Must be > 0.
`num_gpus_per_engine`	`int`	Model’s `num_gpus_per_engine`	TP size override. Number of GPUs per engine instance.
`overrides`	`dict`	`{}`	vLLM `EngineArgs` field overrides. Applied on top of `--vllm-*` CLI args with highest priority.

Worker Types#

Type	Description	Use Case
`regular`	Standard vLLM engine	Default mode, handles both prefill and decode
`prefill`	PD disaggregation prefill worker	Dedicated to prompt processing; paired with `decode` workers
`decode`	PD disaggregation decode worker	Dedicated to token generation; paired with `prefill` workers
`encoder`	EPD disaggregation vision-encoder worker	Runs the vision tower for VLMs; paired with `prefill` or `regular` workers, which become `language_only`
`placeholder`	Reserves GPU slots, no engine created	Reserve GPUs for training co-location or future use

Usage Patterns#

1. Basic Single-Model Deployment#

The simplest config replicates the default behavior:

# vllm_basic.yaml
vllm:
  - name: default
    server_groups:
      - worker_type: regular
        num_gpus: 8

python train.py \
  --vllm-config vllm_basic.yaml \
  --rollout-num-gpus 8 \
  --rollout-num-gpus-per-engine 2 \
  ...

This creates 4 engines (8 GPUs ÷ 2 GPUs/engine) behind a single router.

2. PD Disaggregation#

Separate prefill and decode phases onto dedicated server groups for better throughput in multi-turn and agentic workloads:

# vllm_pd.yaml
vllm:
  - name: actor
    server_groups:
      - worker_type: prefill
        num_gpus: 4
        num_gpus_per_engine: 2    # 2 prefill engines, TP=2
      - worker_type: decode
        num_gpus: 12
        num_gpus_per_engine: 4    # 3 decode engines, TP=4

python train.py \
  --vllm-config vllm_pd.yaml \
  --rollout-num-gpus 16 \
  ...

Why PD disaggregation? In multi-turn scenarios, prefill and decode have different compute characteristics. Prefill is compute-bound (processes the entire prompt), while decode is memory-bandwidth-bound (generates tokens one-by-one). Disaggregating them allows:

Using smaller TP for prefill (higher throughput per GPU)
Using larger TP for decode (lower latency)
Independent scaling of prefill vs. decode capacity

Note: PD disaggregation uses vllm-router with pd_disaggregation=True.

3. EPD Disaggregation (Vision Encoder Split)#

For vision-language models, worker_type: encoder moves the vision tower onto its own engines. Encoder engines compute image embeddings and publish them to an encoder cache; the language engines consume that cache instead of running the vision tower themselves.

EPD composes with PD on an orthogonal axis. Pair encoder with prefill + decode for a full three-stage split:

# vllm_epd.yaml
vllm:
  - name: actor
    server_groups:
      - worker_type: encoder
        num_gpus: 1
      - worker_type: prefill
        num_gpus: 1
      - worker_type: decode
        num_gpus: 1

Or pair encoder with regular to split off the vision tower without PD:

vllm:
  - name: actor
    server_groups:
      - worker_type: encoder
        num_gpus: 2
      - worker_type: regular
        num_gpus: 6

Roles assigned automatically. vime derives the encoder-cache (EC) wiring from the worker types, so you do not write ec_transfer_config by hand:

Worker type	`ec_role`	`kv_role` (when PD is also used)	Notes
`encoder`	`ec_producer`	—	Runs the vision tower. `enable_prefix_caching` is forced off (no language-model KV cache groups). Not registered with the router.
`prefill`	`ec_consumer`	`kv_producer`	Started with `language_only: true` and `encoder_urls` pointing at the encoder engines.
`regular`	`ec_consumer`	—	Same as prefill, when used without PD.
`decode`	—	`kv_consumer`	Receives KV from prefill; never touches the encoder cache.

Startup ordering. Encoder groups are launched first and awaited synchronously, because their URLs must be injected into the language engines’ server arguments before those engines start. The language engines then initialize in the usual deferred fashion.

Priming the encoder. A request must reach the encoder before the language engines generate, so the embeddings are in the cache when the consumer looks them up. The built-in rollout paths do this for you — both generate and generate_streaming call prime_encoder on multimodal samples. In a custom rollout function, call it yourself before rendering:

from vime.rollout.vllm_rollout import prime_encoder

await prime_encoder(args, messages, model_name="actor")

prime_encoder is a no-op when the sample has no images or when the deployment has no encoder group, so it is safe to call unconditionally. Encoder endpoints are published at runtime on args.vllm_model_encoder_endpoints, a dict { model_name: (model_path, [endpoint, ...]) }.

Swapping the connector. By default vime uses upstream vLLM’s ECExampleConnector over a per-deployment tmpfs directory (/dev/shm/vime-ec-<uuid>). An ec_transfer_config in a group’s overrides replaces the generated one for that group, and is then layered over the auto-derived ec_connector / ec_role defaults — so your keys win, but nothing is deep-merged. Because the replacement is per-group, set it on every participating group (encoder and its consumers) with a matching shared_storage_path, or the producer and consumer will point at different caches:

      - worker_type: encoder
        num_gpus: 1
        overrides:
          ec_transfer_config:
            ec_connector: MyProductionConnector
            ec_connector_extra_config:
              shared_storage_path: /mnt/shared/ec-cache

Limitations. ECExampleConnector is upstream’s reference implementation and is file-backed: the default /dev/shm path is node-local, so encoder and consumer engines must land on the same node unless you override shared_storage_path with a shared mount. EPD is also currently exercised in CI with update_weights: false (see tests/test_qwen2.5_vl_3B_ep_disaggregation.py); weight sync into a split encoder/language-only deployment is not yet covered.

4. Multi-Model Serving#

Deploy multiple models simultaneously, each behind its own router:

# vllm_multi_model.yaml
vllm:
  - name: actor
    update_weights: true              # receives weight updates from training
    server_groups:
      - worker_type: regular
        num_gpus: 8
        num_gpus_per_engine: 4

  - name: ref
    model_path: /path/to/ref_model    # different model checkpoint
    update_weights: false              # frozen, no weight updates
    server_groups:
      - worker_type: regular
        num_gpus: 4
        num_gpus_per_engine: 2

  - name: reward
    model_path: /path/to/reward_model
    update_weights: false
    server_groups:
      - worker_type: regular
        num_gpus: 4
        num_gpus_per_engine: 2

python train.py \
  --vllm-config vllm_multi_model.yaml \
  --rollout-num-gpus 16 \
  --hf-checkpoint /path/to/actor_model \
  --rollout-function-path my_rollout.generate_rollout \
  ...

Accessing models in custom rollout functions:

from vime.rollout.vllm_rollout import get_model_url
from vime.utils.http_utils import post

async def my_generate(args, sample, sampling_params):
    # Route to the actor model (default endpoint is /inference/v1/generate)
    actor_url = get_model_url(args, "actor")
    output = await post(actor_url, {
        "model": args.hf_checkpoint,
        "token_ids": sample.tokens,
        "sampling_params": {"max_tokens": 1024, "temperature": 1.0, "top_p": 1.0, "logprobs": 1},
    })
    # output["choices"][0] carries token_ids, logprobs.content[i].logprob, and finish_reason

    # Route to the reference model
    ref_url = get_model_url(args, "ref")
    ref_output = await post(ref_url, {
        "model": args.hf_checkpoint,
        "token_ids": sample.tokens,
        "sampling_params": {"max_tokens": 1024, "temperature": 1.0, "top_p": 1.0, "logprobs": 1},
    })

    # Route to the reward model (e.g., OpenAI-compatible API)
    reward_url = get_model_url(args, "reward", "/v1/chat/completions")
    reward_output = await post(reward_url, {...})

    ...

The get_model_url() helper reads from args.vllm_model_routers, a dict mapping model names to (ip, port) tuples that is automatically populated after engine startup.

5. Multi-Model with PD Disaggregation#

Combine multi-model and PD disaggregation for maximum flexibility:

# vllm_full.yaml
vllm:
  - name: actor
    update_weights: true
    server_groups:
      - worker_type: prefill
        num_gpus: 4
        num_gpus_per_engine: 2
      - worker_type: decode
        num_gpus: 8
        num_gpus_per_engine: 4

  - name: ref
    model_path: /path/to/ref_model
    update_weights: false
    server_groups:
      - worker_type: regular
        num_gpus: 4
        num_gpus_per_engine: 2

6. Placeholder Groups for GPU Reservation#

Use placeholder groups to reserve GPU slots without creating engines. This is useful for co-located training where some GPUs need to be reserved for training:

vllm:
  - name: actor
    server_groups:
      - worker_type: regular
        num_gpus: 6
        num_gpus_per_engine: 2
      - worker_type: placeholder
        num_gpus: 2                   # reserve 2 GPUs (no engines created)

7. Per-Group EngineArgs Overrides#

Use overrides to apply vLLM EngineArgs fields to specific server groups without affecting others:

vllm:
  - name: actor
    server_groups:
      - worker_type: regular
        num_gpus: 8
        num_gpus_per_engine: 4
        overrides:
          mem_fraction_static: 0.85
          context_length: 32768
          chunked_prefill_size: 4096
          enable_torch_compile: true

Overrides take highest priority, overriding both the base --vllm-* CLI args and model-level defaults. This is especially useful for:

Different memory configurations per group
Different context lengths for prefill vs. decode
Enabling experimental features on specific groups

8. Standalone vLLM Launcher#

While --vllm-config is designed for vime’s training pipeline, it also works as a powerful launcher for pure inference scenarios using external engine addresses or by configuring vime to focus solely on serving.

Using external engines with a pre-launched topology:

For complex production deployments, you may want to pre-launch vLLM engines independently and connect them to vime:

# Step 1: Launch vLLM engines externally
vllm serve /path/to/model --port 10090 ...
vllm serve /path/to/model --port 10091 ...

# Step 2: Connect vime to external engines
python train.py \
  --rollout-external-engine-addrs host1:10090 host2:10091 \
  ...

vime queries each external engine’s /server_info endpoint to infer rollout_num_gpus, per-engine GPU counts, vLLM parallel sizes, and prefill/decode worker types. If no --vllm-router-ip/--vllm-router-port is provided, vime launches its own router and registers the external engines to it.

Note: --vllm-config and --rollout-external-engine-addrs are mutually exclusive. Use --vllm-config when you want vime to manage the full engine lifecycle; use --rollout-external-engine-addrs when engines are pre-deployed.

For external-engine selection, update from disk, and delta disk transport, see External Rollout Engines Roadmap.

Router Configuration#

Each model in the config gets its own independent router (vllm-router by default).

Router Policies#

You can configure the routing policy:

--router-policy round_robin     # Simple round-robin
--router-policy consistent_hash # Session affinity for multi-turn
--router-policy cache_aware     # Cache-aware routing (default)

Session-Affinity Routing for Multi-Turn Agents#

For multi-turn dialogues and agentic workloads, session affinity ensures that all requests belonging to the same conversation are routed to the same backend worker. This significantly improves prefix cache hit rates because the worker already has the conversation history cached.

vime automatically assigns each sample a unique session_id (stored in sample.session_id). When the router policy is consistent_hash, this ID is passed as the x-session-id header, and vllm-router uses it to deterministically route all turns of the same session to the same worker.

--router-policy consistent_hash

How it works:

Each sample is assigned a unique session_id via UUID
On each request, vime passes x-session-id: <session_id> in the HTTP header
vllm-router’s consistent-hash policy maps this key to a specific worker
Subsequent turns reuse the same session_id, ensuring they hit the same worker

Resolution Rules#

When the config is loaded, vime applies the following resolution cascade:

GPU per engine fallback: Group num_gpus_per_engine → Model num_gpus_per_engine → args.rollout_num_gpus_per_engine
Model path fallback: Group overrides.model_path → Model model_path → args.hf_checkpoint
Weight update inference: If update_weights is not set:
- true if the effective model path matches --hf-checkpoint
- false otherwise (with a warning)
Total GPU validation: The sum of all num_gpus across all groups across all models must equal --rollout-num-gpus.

Mutual Exclusion#

--vllm-config is mutually exclusive with:

Flag	Conflict Reason
`--prefill-num-servers`	PD disaggregation is configured via `server_groups` in the YAML
`--rollout-external-engine-addrs`	External engines have their own topology; config manages the lifecycle internally

Complete Example: Multi-Model Agentic Training#

Below is a complete example showing a multi-model setup for agentic RL training with PD disaggregation on 32 GPUs:

Config file (vllm_agent.yaml):

vllm:
  - name: actor
    update_weights: true
    server_groups:
      - worker_type: prefill
        num_gpus: 4
        num_gpus_per_engine: 2
        overrides:
          chunked_prefill_size: 8192
      - worker_type: decode
        num_gpus: 12
        num_gpus_per_engine: 4
        overrides:
          mem_fraction_static: 0.88

  - name: ref
    model_path: /data/models/Qwen3-32B
    update_weights: false
    server_groups:
      - worker_type: regular
        num_gpus: 8
        num_gpus_per_engine: 4

  - name: reward
    model_path: /data/models/reward-model
    update_weights: false
    server_groups:
      - worker_type: regular
        num_gpus: 8
        num_gpus_per_engine: 4

Launch command:

python train.py \
  --vllm-config vllm_agent.yaml \
  --hf-checkpoint /data/models/Qwen3-8B \
  --rollout-num-gpus 32 \
  --rollout-function-path my_agent.rollout.generate_rollout \
  --custom-rm-path my_agent.reward.reward_func \
  --advantage-estimator grpo \
  --n-samples-per-prompt 8 \
  ...

Custom rollout function (my_agent/rollout.py):

from vime.rollout.vllm_rollout import get_model_url
from vime.utils.http_utils import post

async def generate_with_models(args, sample, sampling_params):
    """Generate using actor, score with reward model, compare with reference."""
    
    # Generate from actor (default endpoint is /inference/v1/generate)
    actor_url = get_model_url(args, "actor")
    actor_output = await post(actor_url, {
        "model": args.hf_checkpoint,
        "token_ids": sample.tokens,
        "sampling_params": {"max_tokens": 1024, "temperature": 1.0, "top_p": 1.0, "logprobs": 1},
    })
    response_ids = actor_output["choices"][0]["token_ids"]

    # Get reference logprobs over the prompt+response. max_tokens=1 + prompt_logprobs scores
    # the submitted token_ids; read them from the top-level "prompt_logprobs" field.
    ref_url = get_model_url(args, "ref")
    ref_output = await post(ref_url, {
        "model": args.hf_checkpoint,
        "token_ids": sample.tokens + response_ids,
        "sampling_params": {"max_tokens": 1, "temperature": 0.0, "prompt_logprobs": 1},
    })

    # Score with reward model (OpenAI-compatible)
    reward_url = get_model_url(args, "reward", "/v1/chat/completions")
    reward_output = await post(reward_url, {
        "model": "reward",
        "messages": [{"role": "user", "content": sample.prompt}],
    })
    
    # ... process outputs and return Sample

FAQ#

Q: Can I mix PD and regular groups in the same model?#

No. PD disaggregation requires that a model’s server groups are either all prefill/decode pairs or all regular. Mixing regular with prefill/decode in the same model is not supported.

encoder is the exception: it is an orthogonal axis and may be added to either layout, giving encoder + prefill + decode (full EPD) or encoder + regular (vision split without PD). See EPD Disaggregation.

Q: Do encoder GPUs count toward `--rollout-num-gpus`?#

Yes. Total-GPU validation sums num_gpus across every group of every model, including encoder groups. A three-group EPD layout with one GPU each needs --rollout-num-gpus 3.

Q: What happens if `num_gpus` is not divisible by `num_gpus_per_engine`?#

For multi-node engines (where num_gpus_per_engine > num_gpus_per_node), the division is based on the local GPU count per node. For example, with 8 GPUs/node and num_gpus_per_engine: 16, each engine spans 2 nodes.

Q: Can different server groups within a model use different model paths?#

No. All server groups within a model must share the same model_path. This is validated during resolve(). If you need different models, define them as separate model entries.

Q: How do I access the router address for a specific model at runtime?#

Use get_model_url(args, "model_name", "/endpoint") from vime.rollout.vllm_rollout. It reads from args.vllm_model_routers, which is a dict { model_name: (ip, port) } populated automatically.

Q: Can I use `--vllm-config` without training (inference only)?#

While --vllm-config is designed for vime’s training loop, you can effectively use it for inference-only scenarios by configuring a rollout-only run. For fully standalone vLLM serving, consider using vLLM’s native _run_vllm_server directly or --rollout-external-engine-addrs for connecting to pre-deployed engines.

Q: What is the relationship between `--vllm-config` and `--prefill-num-servers`?#

--prefill-num-servers is the legacy way to enable PD disaggregation (it creates a single model with prefill + decode groups). --vllm-config is the newer, more flexible approach. They are mutually exclusive. We recommend migrating to --vllm-config for all new deployments.

vLLM Config: Advanced Engine Deployment

Contents