Design, build, and scale distributed production inference systems for large language models at Thinking Machines. Optimize latency, throughput, and reliability while collaborating with research and infrastructure teams to deploy cutting-edge model architectures. Focus on systems engineering, GPU utilization, and high-concurrency serving with strong observability and uptime requirements.
Key Highlights
Own end-to-end distributed inference systems for LLM serving (Inkling, Inkling-Small, Tinker)
Optimize latency, throughput, and cost efficiency at scale with GPU utilization and batching
Work closely with research and infrastructure teams to translate models into production-ready systems
Key Responsibilities
Design and build distributed infrastructure for large-scale model serving, including request routing, load balancing, and multi-node coordination
Optimize inference latency and throughput in production using techniques like KV cache management, continuous batching, speculative decoding, and quantization
Develop and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability
Benchmark, tune, and extend inference engines to support emerging model architectures in production
Partner with research and infrastructure teams to translate model designs into production-ready systems
Build tooling for tracing, debugging, and resolving issues across the serving stack
Participate in on-call rotation to support production inference systems
Technical Skills Required
Distributed Systems
Python
GPU Programming (CUDA)
Discover our full range of relocation jobs with comprehensive support packages to help you relocate and settle in your new location.
Benefits & Perks
Generous health, dental, and vision benefits
Unlimited paid time off (PTO)
Paid parental leave
Nice to Have
Experience with LLM inference engines (vLLM, SGLang, TensorRT-LLM)
Model parallelism, tensor/pipeline parallelism, or distributed inference techniques
Kubernetes or similar orchestration systems for GPU workloads
Open-source contributions to ML systems or inference infrastructure
Job Description
EngRadar / Thinking Machines LabSoftware Engineer, Production Inference (Distributed Inference) is the employer — EngRadar is a job radar, not a recruiter. We track this posting from their own careers page and send you straight there; we never handle applications or CVs.San Francisco Full-time Posted todayApply directly →The mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.
Want the full job description?
Read the complete details on LinkedIn, the original posting.
Continue on LinkedIn
This is a short excerpt. All rights to the full description belong to its original publisher.
