Staff Machine Learning Engineer, LLM Inference Optimization

About the role

Nebius Token Factory builds production-grade inference for open-source, customer and internally optimized models, owning the path from model artifact to fast, reliable and cost-efficient endpoints. This role sits at the intersection of model internals, inference engines, serving architecture, benchmarking and production deployment.

Key responsibilities

• Own a platform-level optimization pillar such as quantization, speculative decoding, KV-cache optimization, engine selection or long-context serving.
• Create reusable evaluation, benchmarking and rollout frameworks across multiple model families.
• Provide technical direction to MLEs, applied scientists, GPU kernel engineers and infrastructure engineers on shared performance goals.
• Optimize LLM and VLM endpoints for latency, throughput, memory efficiency, GPU utilization, quality and cost per token.
• Deploy, configure, benchmark and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo or similar.
• Build and productionize model-compression workflows including quantization, quantization-aware training, distillation, low-bit serving and accuracy recovery.
• Implement speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching and disaggregated prefill/decode serving.
• Build reproducible benchmarks for TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability and cost per token.
• Partner with GPU kernel and platform engineers to diagnose bottlenecks across model code, kernels, runtime, scheduler, gateway and cluster layers.
• Write design documents, performance reports, rollout plans and customer-facing technical explanations.

Required profile

• Hands-on experience deploying or optimizing LLM, VLM or high-throughput transformer inference systems.
• Ability to reason quantitatively about latency, throughput, quality, utilization and cost tradeoffs.
• Strong communication skills and collaboration with research, kernel, infrastructure, product and customer teams.

Required skills

• Strong Python and PyTorch engineering skills.
• Practical knowledge of at least one modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve or KServe.
• Understanding of transformer inference bottlenecks: KV cache, attention, memory bandwidth, batching, parallelism and long-context serving.

Preferred: experience with quantization-aware training, FP8/INT8/INT4, distillation, speculative decoding, CUDA or Triton, and open-source contributions to vLLM, SGLang, TensorRT-LLM, Triton, Ray or KServe.

What we offer

• Competitive compensation and career growth and learning opportunities.
• Flexibility and ownership in a collaborative, innovative culture.
• Opportunity to work on impactful AI projects in an international environment.