LLM Inference & Serving Lab

Overview

Training an LLM is only half the job — serving it efficiently in production is the other half. The LLM Inference Lab gives you a real deployment environment with an NVIDIA L4 GPU (the cost-efficient inference GPU from Google Cloud) and a GKE cluster to deploy models as scalable API endpoints. You'll learn to deploy models with vLLM (the fastest open-source LLM serving engine), apply quantization to reduce memory and cost, build FastAPI wrappers, and run load tests to understand throughput and latency tradeoffs. This is the exact workflow used by production AI teams to serve language models at scale.

What You'll Do in This Lab

• Deploy fine-tuned LLMs as REST APIs using vLLM and FastAPI
• Serve models on Vertex AI Endpoints with auto-scaling configuration
• Apply GPTQ, AWQ, and GGUF quantization and measure quality vs speed tradeoff
• Run load tests with Locust — measure tokens/second, latency P50/P95/P99
• Implement streaming responses for real-time chat applications
• Configure model caching, batching, and prefix caching for cost optimization

Lab Workflow