Senior QA Automation Engineer – AI/LLM

We’re Hiring: Senior QA Automation Engineer – AI/LLM (Fully Remote)

📍 Abu Dhabi, UAE

📄 12 months extendable contract

We are looking for a Senior QA Automation Engineer to help define how advanced AI systems are tested, validated and approved for deployment within a highly regulated environment.

This is not a traditional QA role. You will build automated frameworks to assess LLM and RAG outputs for hallucination, factual accuracy, consistency, faithfulness and answer relevance.

Key responsibilities:

• Build Python-based test automation frameworks using Pytest

• Evaluate LLM outputs against ground-truth datasets

• Implement RAG evaluation metrics using RAGAS, DeepEval, TruLens or custom evaluators

• Develop prompt-regression testing to identify model and prompt drift

• Automate UI and API testing using Playwright/Selenium and Requests

• Validate integrations between AI agents and enterprise systems

• Test the latency and throughput of AI and predictive-model services

• Verify API security, RBAC and PII-redaction controls

• Validate data quality across SQL databases, data lakehouses and vector databases

• Integrate automated tests into GitLab CI/CD quality gates

• Produce traceable evidence for formal verification and release reviews

• Work closely with data scientists, developers and systems engineers

What we’re looking for:

• 5+ years of QA automation experience

• At least 2 years testing AI/ML systems, data-driven applications or AI agents

• Strong Python, Pytest, SQL and API automation experience

• Experience testing LLMs, RAG pipelines or probabilistic systems

• Knowledge of ground-truth datasets and AI evaluation metrics

• Experience with GitLab CI/CD and automated quality gates

• Familiarity with Jira, Jama, Polarion or similar traceability tools

Interested or know someone suitable? Please message me directly or send your CV.