Senior Site Reliability Engineer (SRE) for Model Post-Training Infrastructure

About the Role

We're looking for a Site Reliability Engineer (SRE) to drive reliability for Model Post Training end-to-end. Post-training spans several distinct workloads — supervised fine-tuning, reinforcement learning, preference optimization, and distillation — each with its own resource profile, failure signature, and tolerance for interruption or delay. You'll set the reliability strategy across all of them, not just react to what breaks.

This is a senior individual-contributor role with real technical authority. You'll work alongside the engineers building Tinker's training and serving infrastructure and the research teams running experiments on top of it, shaping how the platform is architected before problems reach production rather than only after. You'll be trusted to define what reliable means for post-training workloads, push back on launches that put stability or correctness at risk, and lead incident response for issues that span multiple teams and layers of the stack.

We're looking for someone who has operated production ML systems before and understands, first-hand, how an infrastructure failure — a bad gradient sync, a stale checkpoint, a silent NaN — can look identical to a research problem until someone with the right instincts tells the difference.

What You'll Do

• Define and own end-to-end reliability for Model Post Training, from CI/CD flows to production observability and incident response, across SFT, RL, DPO, and distillation workloads.
• Develop Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency against development velocity, and tailoring them to each training method's tolerance for delay, restart, and data loss.

Looking to advance your Development & Programming career with relocation support? Explore Development & Programming Jobs with Relocation Packages that include comprehensive packages to help you move and settle in your new role.

• Design and implement monitoring and observability across the full training path — data loading, forward/backward passes, optimizer steps, checkpointing, and sampling — with enough signal to distinguish an infrastructure failure from a training or modeling issue.
• Drive incident response for Model Post Training platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence.
• Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation.
• Set technical direction on reliability for new post-training methods as they're adopted onto the platform, partnering with research early so resilience is designed in rather than retrofitted.
• Build and harden checkpointing and recovery for long-running training jobs, so trainer, optimizer, or sampler failures cost minutes of progress, not hours.
• Mentor other engineers on production ML reliability practices, and collaborate with security teams to address production vulnerabilities.

Skills and Qualifications

Minimum Qualifications

• 7+ years of experience in distributed systems, cloud infrastructure, or site reliability engineering, including direct on-call ownership of a live production service.
• Hands-on experience operating production machine learning systems — training, fine-tuning, or inference infrastructure — and reasoning about how infrastructure failures manifest as training or model-quality issues.
• Proficiency writing software to solve reliability problems, including building tooling and automation, typically in Python and/or a systems language such as Go or Rust.
• Track record of driving production incident response, postmortems, and systematic reliability improvement, including for complex, cross-team, or ambiguous issues.
• Strong communication skills and a track record of setting technical direction and coordinating across engineering and research teams.

Preferred Qualifications

• Deep experience operating production cloud services at scale (e.g., public cloud platforms, internal cloud services).
• Working knowledge of post-training methods — SFT, RL (e.g., PPO, GRPO), DPO, or knowledge distillation — and the distinct infrastructure demands each one places on a shared platform.
• Background in distributed training frameworks (e.g., PyTorch FSDP/DDP, Megatron, DeepSpeed) and experience distinguishing infrastructure failures from optimization or data issues in training runs.

Interested in relocating to United State? Check out our comprehensive Relocation Jobs in United State page with detailed relocation packages and benefits.

• Track record building checkpoint and recovery systems for long-running distributed jobs.
• Expertise in Kubernetes at scale: deploying, operating, debugging, and tuning clusters handling heterogeneous GPU workloads.
• Experience setting reliability standards or SRE practices for an ML platform from the ground up, rather than inheriting an established one.

Logistics

• Location: This role is based in San Francisco, California.
• Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 – $475,000 USD.
• Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
• Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.