E

Founding Engineer, AI Agent Evaluation

early-stage, venture-backed ai startup (name not explicitly stated) • United States
Relocation
Apply Now
AI Summary

Early-stage AI startup seeks its first engineer to design evaluation systems for AI agents, scoring complex business tasks, building test scenarios, and ensuring statistically sound results. Requires experience in evaluation systems, benchmarks, or RL environments for LLMs/agents, with 2-6 years of relevant experience. Bonus for post-training, domain expertise, or research in evaluation.

Key Highlights
First engineer role at an early-stage AI startup
Focus on designing evaluation systems for AI agents
2-6 years of experience required, bonus for domain expertise or research

Looking to advance your Development & Programming career with relocation support? Explore Development & Programming Jobs with Relocation Packages that include comprehensive packages to help you move and settle in your new role.

Key Responsibilities
Design scoring systems for AI agent output on complex tasks
Develop realistic test scenarios and simulated workflows
Ensure statistical soundness and trustworthiness of evaluation results
Translate customer-specific approaches into core product features
Technical Skills Required
Evaluation Systems RL Environments LLM Benchmarking

Discover our full range of relocation jobs with comprehensive support packages to help you relocate and settle in your new location.

Benefits & Perks
Up to $300k base salary
Significant founding equity
Comprehensive health benefits
Relocation support to San Francisco

Interested in relocating to United States? Check out our comprehensive Relocation Jobs in United States page with detailed relocation packages and benefits.

Nice to Have
Experience with RLHF or SFT post-training techniques
Domain expertise in finance, legal, or healthcare
Background in statistics or published research on evaluation

Job Description

An early-stage, venture-backed AI startup in San Francisco is hiring its first engineer. The company helps enterprises understand whether their AI agents are actually doing the job: how well they perform, where they break, and how to improve them.

What you’ll work on:

  • Designing ways to score agent output on complex business tasks where there’s no single right answer
  • Building realistic test scenarios and simulated workflows that reflect how agents are used in practice
  • Deciding how much testing is enough, and making sure results are statistically sound and trusted by customers
  • Taking approaches that work in individual customer deployments and building them into the core product
Want the full job description? Read the complete details on LinkedIn, the original posting.
Continue on LinkedIn

This is a short excerpt. All rights to the full description belong to its original publisher.

Ready to apply?

Apply Now
See Jaabz jobs first on Google 1 tap · free · in AI Overviews Jaabz is on your Google Manage Preferred Sources
Job Overview
Posted Nov 12, 2026
Employment Full-time
Experience Mid-Senior level
Location United States
Category Programming
Company early-stage, venture-backed ai startup (name not explicitly stated)
Keep exploring

Similar Jobs

Explore other opportunities that match your interests

Hidden company

Log in to see location

Premium

Senior Software Engineer, Secure Signing Infrastructure

Log in free to see the details

Hidden company

Log in to see location

Premium

Software Engineer (ControlST & Windows Applications)

Log in free to see the details

Relocation Guides

Step-by-step relocation guides, from moving logistics to settling abroad.

Browse all articles

Subscribe to our newsletter

New visa sponsorship, relocation and remote tech jobs, delivered to your inbox.