Research Engineer - Benchmark Development for Frontier AI Agents

hud • Singapore

Visa Sponsorship Relocation

This Job is No Longer Active This position is no longer accepting applications

AI Summary

Build high-quality benchmarks for evaluating frontier AI agents on domain-specific tasks. Own design, implementation, and quality of internal agent benchmarks while collaborating with subject-matter experts. Requires strong Python skills and experience with model evaluation environments.

Key Highlights

Create technically rigorous, practically useful, and credible benchmarks for frontier AI agents

Work with subject-matter experts to define tasks and create domain-specific evaluation workflows

Develop infrastructure to reliably run models and agents against benchmark tasks

Validate benchmark performance correlation with real-world evals and customer needs

Key Responsibilities

Own the design, implementation, and quality of HUD's internal agent benchmarks

Work with subject-matter experts to define tasks and create domain-specific benchmarks that evaluate agents on realistic workflows

Build infrastructure to reliably run models and agents against benchmark tasks

Develop metrics and analyses to understand benchmark difficulty, reliability, and failure modes

Validate whether benchmark performance correlates with real-world evals, customer needs, and lab expectations

Write clear documentation and benchmark reports that make results legible and credible to technical audiences

Looking to advance your Development & Programming career with relocation support? Explore Development & Programming Jobs with Relocation Packages that include comprehensive packages to help you move and settle in your new role.

Technical Skills Required

Python Docker Linux

Benefits & Perks

Competitive compensation

100% covered top-of-the-line medical, dental, and vision from Blue Shield of CA

Unlimited* access to tokens for ChatGPT, Claude Code, Cursor, etc.

Discover our full range of relocation jobs with comprehensive support packages to help you relocate and settle in your new location.

Nice to Have

Published papers or written technical blogs on relevant topics such as public benchmarks and their limitations, model failure modes, etc.

Experience working on environments and evals

Curiosity and ability to truly understand how workflows in various domains work

Detail-oriented and able to spot subtle inconsistencies or edge cases in tasks

Ability to reason from first principles about task design, scoring, and failure modes

Early-stage startup experience with ability to work independently in fast-paced environments

Strong communication skills for remote collaboration across time zones

Job Description

HUD is building infrastructure to create RL training data and evals for frontier AI agents, as well as a marketplace to sell these to frontier labs through the HUD marketplace. Our platform is used by frontier labs, Fortune 500 companies, and startups. We’ve raised $16M from top VCs and were YC W25.We’re looking for Research Engineers to build high-quality benchmarks for evaluating frontier agents on domain-specific tasks. You’ll build benchmarks that are technically rigorous, practically useful, and credible to frontier labs.

Want the full job description? Read the complete details on LinkedIn, the original posting.

Continue on LinkedIn

This is a short excerpt. All rights to the full description belong to its original publisher.