Lead Quality Engineer – AI Delivery

You will work closely with Product Managers, Product Owners, Engineering Leads, Architects, and developers, combining strategic quality leadership with hands-on technical capability.

We will count on you to:

* Lead Quality Engineering across multiple AI delivery pods and establish consistent, risk-based quality standards.

* Partner with Product and Engineering to define test strategy, acceptance criteria, UAT approach, quality risks, and release-readiness criteria.

* Ensure appropriate coverage across functional, integration, end-to-end, non-functional, AI-specific, and business acceptance testing.

* Define testing and evaluation approaches for LLM, RAG, agentic, and AI-enabled solutions, including grounding, hallucination risk, regression, tool calling, workflow behaviour, and non-deterministic outputs.

* Establish appropriate AI evaluation datasets, test harnesses, baselines, thresholds, and regression approaches.

* Guide and, where needed, implement reusable automation across web, APIs, backend services, integrations, asynchronous workflows, and AI components.

* Ensure products are ready for UAT, with appropriate test evidence, environments, test data, defect status, entry/exit criteria, and known risks.

* Provide Pod Leads and programme leadership with clear quality metrics, risks, defect trends, AI evaluation results, UAT readiness, and release-readiness reporting.

* Conduct quality reviews across pods and identify gaps in testing, testability, observability, or quality evidence.

* Use production incidents, escaped defects, telemetry, and user feedback to continuously improve quality approaches.

* Coach engineers and delivery teams to strengthen shared ownership of quality across the lifecycle.

What you need to have

* Proven experience leading Quality Engineering for enterprise-scale AI / GenAI solutions in production.

* Hands-on experience testing LLM, RAG, agentic, workflow-based, or AI-enabled applications.

* Strong experience defining end-to-end test strategies, UAT approaches, quality gates, and release-readiness criteria.

* Experience working across multiple delivery teams or pods in complex enterprise environments.

* Strong software engineering and automation skills in TypeScript / JavaScript and/or Python.

* Experience with modern automation frameworks such as Playwright, Cypress, Selenium, Jest, Mocha, or PyTest.

* Strong experience testing web applications, APIs, backend services, integrations, and distributed systems.

* Experience with AI evaluation frameworks, regression suites, test harnesses, and representative evaluation datasets.

* Good understanding of CI/CD, cloud-native systems, observability, and Secure SDLC practices.

* Strong stakeholder management skills and the ability to translate quality evidence into clear risks, metrics, and recommendations for leadership.

What makes you stand out

* Experience with LangChain, LangSmith, Mastra, or similar AI engineering and evaluation tooling.

* Experience testing complex RAG, multi-agent, or tool-calling solutions.

* Experience with performance, resilience, accessibility, or security testing.

* Experience using production telemetry and incidents to improve test and evaluation strategies.

* Ability to balance strong quality controls with pragmatic, fast-moving AI delivery.

Technology context

Typical technologies include:

React, TypeScript, Node.js / NestJS, Python / FastAPI, REST APIs, async workflows, AWS / Azure, GitHub Actions / GitLab CI / Azure DevOps, LLM APIs, RAG, agentic workflows, LangChain, LangSmith, and Mastra.