AI Summary
Join a leading AI lab's GenAI team as a ground-truth expert to evaluate frontier AI models. Design real-world ML/NLP tasks, generate reference solutions, and analyze model outputs. Requires Python proficiency, deep ML/NLP experience, and ability to work ~20 hours/week remotely.
Key Highlights
Act as a ground-truth expert for frontier AI models.
Design and develop challenging ML/NLP tasks and reference solutions.
Evaluate model performance and identify capability gaps.
Key Responsibilities
Design challenging, real-world ML and NLP problems targeting specific capability gaps in a frontier AI model.
Integrate problems into an agentic development environment, preparing all necessary components using Python.
Evaluate the target model's performance on your tasks.
Identify tasks where the target model fails and classify the nature of the failure.
Work alongside fellow subject-matter experts to keep evaluations consistent and accurate.
Technical Skills Required
Benefits & Perks
W-2 employment
Fully remote within the United States
Part-time (~20 hours per week)
Nice to Have
Past experience in AI training, model evaluation, or data annotation.
Job Description
hackajob is collaborating with Mercor to connect them with exceptional professionals for this role.
- Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models.** ## 1\. Overview A leading AI lab is building and evaluating frontier models and needs experienced machine learning and natural language processing practitioners to act as ground-truth experts. You will author challenging, real-world tasks, generate reference solutions, and evaluate model outputs to surface reasoning and capability gaps in a target model. The work centers on designing robust ML and NLP problems, building them out with executable tests where applicable, and then analyzing how models and agents behave against them. All applicants are expected to have working proficiency in Python. This is a part-time W-2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI lab as part of their extended workforce. This role is fully remote within the United States, at approximately 20 hours per week. ## 2\. Key Responsibilities - **Task design and development:** Design challenging, real-world ML and NLP problems drawn from your area of expertise (e.g., model training and evaluation, language understanding and generation, retrieval, applied ML pipelines) that target specific capability gaps in a frontier AI model. - **Spec and golden-solution generation:** Integrate the problems into an agentic development environment, preparing all necessary components using Python. - **Evaluation and analysis:** Evaluate the target model's performance on your tasks. - **Headroom identification:** Identify tasks where the target model fails, and classify the nature of the failure. - **Collaborate with other experts:** Work alongside fellow subject-matter experts to keep evaluations consistent and accurate. ## 3\. Core Qualifications - Deep, hands-on experience in machine learning and/or natural language processing, from applied industry work, research, or a graduate/PhD background in the field. - Working proficiency in Python, applied in research, industry, or open-source work (not theoretical familiarity). - Strong command of modern ML/NLP methods, including model training and evaluation, transformers and large language models, and standard tooling. - Ability to engage reliably for approximately 20 hours per week. - Past experience in AI training, model evaluation, or data annotation is preferred. - Strong written communication and the ability to work independently and manage your own time. ## About Cincinnatus LLC Cincinnatus LLC is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for contingent and contract-based opportunities. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve part-time or full-time commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus LLC. ## Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Interested in remote work opportunities in Machine Learning & AI? Discover Machine Learning & AI Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.
Browse our curated collection of remote jobs across all categories and industries, featuring positions from top companies worldwide.
Similar Jobs
Explore other opportunities that match your interests
Visa Sponsorship
Relocation
Remote
Job Type
Contract
Experience Level
Associate
crossing hurdles
India
Senior Machine Learning Researcher - AI Lab Benchmark Expert
••••••
••••••
••••••
Job Type
••••••
Experience Level
••••••
hackajob
India
Visa Sponsorship
Relocation
Remote
Job Type
Full-time
Experience Level
Internship
skillscapital
India