Machine Learning Researcher - AI Model Evaluator

This opportunity is shared as part of a referral program

Mercor is partnering with a leading AI research lab on Project Vesuvius, an initiative designed to evaluate and enhance the ability of large language models (LLMs) to generate structured, high-quality research plans for open-ended machine learning problems.

We are seeking Machine Learning Researchers and PhDs to serve as annotators who will assess and provide structured feedback on AI-generated research plans. The goal is to improve how LLMs function as brainstorming partners for machine learning research.

This is a remote, short-term engagement with flexible hours and opportunities to contribute to frontier AI evaluation and research.

Key Responsibilities

• Evaluate and compare AI-generated research plans for clarity, feasibility, and technical soundness.
• Design and compile ML tasks based on real-world challenges and research competitions.
• Draft detailed, executable natural language plans for machine learning workflows.
• Implement and validate research plans in Python within a Docker environment.
• Assess outputs against structured rubrics, provide usefulness scores, and deliver concise, objective feedback.

Ideal Qualifications

• 5+ years of experience in applied machine learning or a PhD in machine learning or related fields.
• Strong understanding of ML research methodologies, experimental design, and evaluation practices.
• Excellent analytical and technical writing skills.
• Experience with reproducibility or benchmarking in ML research preferred.
• Detail-oriented and able to deliver high-quality, structured feedback independently.

Engagement Details

• Type: Independent contractor
• Mode: Fully remote and asynchronous — work from anywhere, on your own schedule.
• Commitment: Maximum 80 hours per week
• Project Name: Vesuvius

This role is designed for researchers and engineers who value autonomy, precision, and meaningful contribution to frontier AI development.

Compensation & Contract Terms

• Hourly Pay Rate: Up to $140/hour
• Payment: Weekly
• Contract Type: Independent contractor engagement
• Structure: Remote, milestone-based evaluation with flexible scheduling
• Application Process
• Submit your resume or CV highlighting relevant ML research or engineering experience.
• Complete a short AI-based interview and a brief questionnaire about your experience with reproducibility and model benchmarking.
• Selected candidates will receive detailed onboarding materials and access to the project environment.