Senior Machine Learning Engineer, Inference

Accountabilities

You will lead end-to-end inference optimization, ensuring that advanced machine learning models perform efficiently and reliably in production environments. Your responsibilities will include:

• Own the optimization process from research models to production deployment.
• Improve latency and throughput for real-time voice and LLM inference at consumer scale.
• Work directly with inference engines such as vLLM, TensorRT-LLM (TRT-LLM), and SGLang to enhance performance.
• Implement model acceleration techniques, including quantization, distillation, speculative decoding, caching, continuous batching, and paged attention.
• Develop high-performance GPU workloads using C++, CUDA, Rust, or optimized Python.
• Build and scale distributed multi-GPU and multi-node inference systems using Kubernetes, Ray, and custom load-balancing solutions.
• Collaborate with research and product teams to bring new models into production.

Requirements

The ideal candidate is a senior-level machine learning inference or systems engineer with strong hands-on experience optimizing production inference engines and building scalable GPU infrastructure. You should bring:

• At least 5 years of experience in ML inference or systems engineering, at a senior level equivalent to Google L5 or Meta E5.
• Practical experience optimizing inference engines such as vLLM or TRT-LLM at the engine level, beyond basic model deployment.
• A proven background in developing backend and machine learning systems and serving models in production.
• Strong systems programming and GPU optimization skills using C++, CUDA, Rust, or optimized Python.
• Hands-on knowledge of model acceleration techniques, including quantization, distillation, speculative decoding, caching, continuous batching, and paged attention.
• Experience with distributed systems, Kubernetes, Ray, load balancing, multi-GPU and multi-node inference, and high-concurrency environments.
• Excellent English communication skills.

Residence in the United States, Canada, the United Kingdom, Germany, Switzerland, or Serbia.

Additional advantages include a track record of rapid career progression, contributions to open-source inference engines or technical publications, and a PhD in Computer Science, Physics, or Mathematics, or a degree from a leading computer science program. The ability to commute to Mountain View, California, is also a plus.