Lead AI Testing Engineer

Job Description

We are looking for a Lead AI Testing Engineer to drive the quality strategy for AI-driven and data migration initiatives spanning knowledge graphs, ontologies and RAG-based LLM features. You will define and execute QA practices across data ingestion, graph layers, rule evaluation engines and AI/LLM components while building automation frameworks that scale evaluation across complex, data-heavy systems.ResponsibilitiesDesign and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rateBuild test harnesses to scale evaluation beyond human-in-the-loop processesCreate data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenariosVerify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validationDefine and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutionsEstablish quality gates, acceptance criteria and test coverage modelsMaintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layersValidate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validationIntegrate test suites into CI/CD pipelines and define release quality gatesChampion automation-first and shift-left quality practicesLeverage agentic AI and Gen AI tooling in testing and framework developmentRequirements8+ years of experience in software development, testing automation and DevOpsProficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integrationExpertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQLSkills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctnessProven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gatesSkills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systemsBackground in backend and API testing (REST) covering data validation, integration and E2E testingProficiency in frontend web application test automation using Playwright or equivalent Python-compatible frameworkHands-on experience using coding agents and agentic development dailyDemonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checksApplied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scaleExperience in Agile delivery using Git and Azure DevOps or JIRAEnglish proficiency at B2 level or higherNice to haveSkills in semantic search testing and evaluationFamiliarity with Vector Database integration and retrieval validationBackground in data science covering ML concepts, data pipelines or data engineering collaborationProficiency in API tooling such as Postman and BrunoExperience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or JenkinsWe offerWe gather like-minded people:Engineering community of industry professionalsFriendly team and enjoyable working environmentFlexible schedule and opportunity to work remotely within PolandChance to work abroad for up to 60 days annuallyBusiness-driven relocation opportunitiesWe provide growth opportunities:Outstanding career roadmapLeadership development, career advising, soft skills, and well-being programsCertification (GCP, Azure, AWS)Unlimited access to LinkedIn Learning, Get Abstract, Cloud GuruEnglish classesWe cover it all:Stable income (Employment Contract or B2B)Participation in the Employee Stock Purchase PlanBenefits package (health insurance, multisport, shopping vouchers)Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and moreReferral bonusesCorporate, social and well-being eventsPlease, note:The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview.We will reach out to selected candidates exclusively.EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliv