vllm

About this Skill

Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), OpenAI-compatible API serving, throughput and latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Use when deploying or running a vLLM server (vllm serve, vllm/vllm-openai), sizing a model and its KV cache for GPUs, selecting quantization and parallelism, serving via /v1 endpoints, measuring serving throughput or latency, tuning batching, or diagnosing GPU, OOM, or startup failures in a vLLM deployment. Do not use for model training, fine-tuning, evaluation-set design, or engine-selection methodology (that is ml-engineering), or for operating the llama.cpp stack with GGUF models (that is llama-cpp); other inference engines (TGI, Ollama, Triton) are out of scope.

Anwendungsfälle

• Deploying vLLM with Docker or Kubernetes
• Sizing models and KV caches for GPUs
• Serving models through OpenAI-compatible endpoints
• Benchmarking throughput and latency
• Diagnosing OOM, GPU, startup, and upgrade issues

Funktionen

• Docker and Kubernetes deployment guidance
• Quantization, tensor parallelism, and KV cache configuration
• OpenAI-compatible API endpoint operations
• Throughput, latency, and batching benchmarks
• Read-only health and metrics probing with vllm-health

Anforderungen

• vLLM release v0.26.0 or a pinned older release
• NVIDIA CUDA, AMD ROCm, Intel XPU, or a supported CPU build
• Python 3.9+ for the bundled vllm-health script

Unterstützte Agents

Skill files

Related Skills