Puffy
Emea
Head of AI Transformation and Business Operations
- AI
- Rally
- SQL
D
Principal AI Evaluation Engineer
Emea
Relocation
AI Summary
Seeking a Principal Evals Engineer for a major AI program in Abu Dhabi to define and build AI evaluation standards, frameworks, and infrastructure. This hands-on role involves ensuring AI systems are production-ready by designing shared evaluation architectures and automating quality measurement. Requires deep experience in modern AI evaluation, LLM quality assessment, and building robust, scalable evaluation systems.
Key Highlights
Define and own AI evaluation strategy, standards, frameworks, and infrastructure.
Design and build shared evaluation architecture for multiple AI products and engineering teams.
Ensure AI systems are production-ready through rigorous, evidence-based evaluation.
Key Responsibilities
Define the overall AI evaluation strategy and reference architecture across the engineering organization
Build shared evaluation frameworks, harnesses and reusable libraries
Develop and maintain golden datasets and dataset versioning strategies
Design automated evaluation and grading systems
Implement regression detection for non deterministic AI behaviour
Evaluate LLM output quality, hallucination, grounding, faithfulness and citation accuracy
Build evaluation frameworks for RAG and retrieval systems
Evaluate agentic systems, including tool usage, multi step reasoning and failure recovery
Design and calibrate LLM as a judge systems against human evaluation
Establish production monitoring, online evaluation and drift detection
Develop adversarial testing methodologies covering prompt injection, jailbreaks and data leakage
Integrate AI evaluation into CI/CD pipelines and release processes
Help engineering teams understand and apply rigorous evaluation standards independently
Contribute to building the broader evaluation engineering function as the organisation scales
Looking to advance your Development & Programming career with relocation support? Explore Development & Programming Jobs with Relocation Packages that include comprehensive packages to help you move and settle in your new role.
Technical Skills Required
Benefits & Perks
Relocation support available
Competitive tax-free UAE package
Discover our full range of relocation jobs with comprehensive support packages to help you relocate and settle in your new location.
Nice to Have
Ragas
DeepEval
Promptfoo
Braintrust
LangSmith
Langfuse
LangChain or LangGraph
Vector databases such as pgvector or Qdrant
AI red teaming or adversarial testing
Voice or conversational AI evaluation
Human evaluation programme design
Data pipeline or performance testing
Arabic language AI evaluation, including golden datasets, dialect coverage and right to left validation
Experience working within regulated, government or highly secure environments
Job Description
Principal Evals Engineer
Location: Abu Dhabi, UAE (relocation support available)
About the opportunity
We are hiring a Principal Evals Engineer to join a major new AI programme in Abu Dhabi. The organization is building a large scale AI platform designed to support next generation AI systems across multiple sectors.The work spans AI assistants, retrieval systems, agentic workflows, voice applications and document intelligence, with a strong focus on building reliable AI systems that can operate at scale in real world environments.
Want the full job description?
Read the complete details on LinkedIn, the original posting.
Continue on LinkedIn
This is a short excerpt. All rights to the full description belong to its original publisher.
Keep exploring
Similar Jobs
Explore other opportunities that match your interests
Hidden company
Log in to see location
Premium
Staff Software Engineer – Automation Library Development
Log in free to see the details
CyberCoders
United State
Senior/Staff/Principal Physical IC Design Engineer (ASIC)
- AI
- Cloud
- Machine Learning
- Perl
Relocation Guides
Step-by-step relocation guides, from moving logistics to settling abroad.
Can You Get Relocation Assistance for an Internal Job Transfer?
Changing locations for work is already a lot to manage. But when it happens within the same company, people often assume the finan...
Read Article
What Happens to Relocation Assistance If You Leave the Job?
If you leave your job after moving, your employer will likely ask for some or all of your relocation money back. Most jobs with re...
Read Article
Hidden Costs of Job Relocation: What Employers May Not Cover
You might think that relocation assistance is like an all-expenses-paid fresh start, but it’s usually not as simple and straightfo...
Read Article
Relocation Assistance vs. Visa Sponsorship: What’s the Difference?
For anyone trying to land a job abroad, visa sponsorship and relocation assistance are usually the first things they look for in a...
Read ArticleSubscribe to our newsletter
New visa sponsorship, relocation and remote tech jobs, delivered to your inbox.