Senior LLM Inference Systems Engineer

netpreme • United State

Visa Sponsorship Relocation

Apply Now

AI Summary

Own Netpreme’s end-to-end LLM inference-serving stack, optimizing performance for frontier models using vLLM, SGLang, TensorRT-LLM, and NVIDIA Dynamo. Architect distributed systems, benchmark optimizations, and collaborate with hardware/software teams to deploy cutting-edge AI models at scale. Requires 2+ years in LLM inference, GPU systems, and production ML infrastructure with Python, Linux, and Kubernetes expertise.

Key Highlights

End-to-end ownership of LLM inference-serving stack, from engine tuning to distributed system architecture

Collaborate with hardware and systems engineers to optimize performance for frontier models

Build benchmarks, measure system efficiency, and deploy models in production with strict SLAs

Key Responsibilities

Deploy and operate large language and multimodal models using inference engines like vLLM, SGLang, and TensorRT-LLM

Architect and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible, ensuring upgrades, monitoring, and reliability

Optimize parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets

Build reproducible benchmarks for latency (TTFT, TPOT, E2E), throughput, concurrency scaling, and system efficiency

Support diverse workloads including tool calling, agentic workflows, RAG, and long-context applications

Evaluate and integrate new models, frameworks, and optimizations into production systems

Looking to advance your Devops career with relocation support? Explore Devops Jobs with Relocation Packages that include comprehensive packages to help you move and settle in your new role.

Technical Skills Required

Python Kubernetes LLM Inference Optimization

Benefits & Perks

Competitive salary with performance-based bonus and early-stage equity grant

100% employer-paid Health, Dental, and Vision coverage for you and dependents

Visa sponsorship and relocation assistance available to one of our office hubs

Discover our full range of relocation jobs with comprehensive support packages to help you relocate and settle in your new location.

Nice to Have

Experience with Helm, Ansible, or similar deployment and configuration tools

Experience with distributed inference systems like NVIDIA Dynamo (prefill/decode disaggregation, KV-cache-aware routing)

Experience with advanced inference optimizations such as prefix caching, speculative decoding, and quantization

Experience analyzing LLM architectures and tuning deployments across different hardware platforms

Contributions to vLLM, SGLang, TensorRT-LLM, or related open-source projects

Experience owning production infrastructure at an early-stage startup

Job Description

The biggest bottleneck in AI inference today is memory. Chips can compute faster than they can be fed data, a problem known as the memory wall, and Netpreme is building the solution. Our approach spans the full stack, from a proprietary memory management library down to X-Mem™, our Memory Processing Unit (MPU) silicon. Netpreme was co-founded by researchers in optical networking and ML systems. We’re a small team in Cambridge, MA and Santa Clara, CA, and we’re looking for people who want to own hard problems end to end and help shape where the company goes next.

Want the full job description? Read the complete details on LinkedIn, the original posting.

Continue on LinkedIn

This is a short excerpt. All rights to the full description belong to its original publisher.