Senior QA Analyst

Job Description

Job Application for Senior QA Analyst at HugeInc Back to jobs Senior QA Analyst Colombia Apply Location: This position is remote within Colombia We are seeking a Senior QA Analyst & AI Evaluation Strategist to plan, design, and execute test strategies for our rapidly expanding suite of agentic and AI-powered product features. This is a unique evolution of the traditional QA role. Instead of focusing solely on predictable, binary (pass/fail) code paths, you will specialize in evaluating probabilistic software. You won’t be building or training AI models from scratch; instead, your focus will be on the integrity, orchestration, and evaluation of the system . You will be responsible for defining how we test multi-agent workflows, curating test data, and validating that our analytical dashboards, APIs, and data lakes process non-deterministic data accurately and safely. If you have an analytical, system-thinking QA mindset and are eager to strategize the future of AI product quality, we encourage you to apply. What you'll do Strategic QA Planning: Formulate comprehensive test plans, timelines, and execution strategies optimized for multi-agent architectures and AI-driven data pipelines. Evaluation Framework Design: Collaborate with data scientists and product managers to define and standardize statistical evaluation rubrics (e.g., tracking Share of Model, Net Sentiment, and Citation Authority). System Integrity & Governance: Establish and supervise risk-adaptive checkpoints to ensure data privacy, compliance, and accurate reasoning across user journeys. Cross-Functional Alignment: Act as the primary bridge between Engineering, Data Science, and Product to translate complex business objectives into clearly defined, structured QA requirements. Ecosystem Verification: Ensure end-to-end platform consistency, verifying everything from raw data ingestion and API gateways to final user-facing analytical surfaces. Orchestrate Testing Frameworks: Design and run manual and automated functional validation processes for core application layers, API endpoints, and multi-step agentic workflows. Curate "Golden" Test Sets: Oversee the compilation and maintenance of baseline validation datasets used to test edge cases, user intent, and potential model anomalies. Validate Analytics and Metrics: Audit the calculations and automated workflows behind complex composite data scores, ensuring reporting values match underlying metrics reliably. Execute Adversarial Scenarios: Coordinate functional "red-teaming" or boundary testing to evaluate how gracefully features handle prompt variations, ambiguous inputs, or conversational drift. Biases & Safety Auditing: Perform targeted validation checks to ensure compliance with strict internal data policies, verifying proper PII data security and content filter safety. Defect Profiling & Distribution: Document and classify standard functional bugs alongside statistical behavioral defects in JIRA, mapping out trends to help developers find root causes efficiently. Monitor Platform Metrics: Utilize modern cloud logging and observability dashboards to keep tabs on execution latency, data drift, and overall user feedback patterns. What we're looking for Education: Bachelor’s degree in Computer Science, Business Analytics, Information Technology, or a related field (or equivalent practical experience). Experience: 3+ years of proven experience in QA Engineering, Systems Analysis, or Software Testing, with a track record of leading test planning for complex digital platforms. Analytical & Spectrum Thinking: A strong data-driven mindset with the ability to shift away from purely binary testing toward evaluating data trends, thresholds, and statistical outcomes. Core QA Competencies: Mastery of traditional software testing methodologies, regression testing, API testing tools, Agile environments, and project tracking ecosystems (such as JIRA). Basic Scripting Familiarity: Ability to work with basic programming/scripting (ideally Python or JavaScript) to trigger automated test collections, query databases, or execute pipelines. Conceptual AI Familiarity: General understanding of how generative AI features function, including an operational awareness of typical behavior challenges (e.g., hallucinations, context limits, and instructions adherence). Hands-on experience with agentic QA workflows and AI-assisted testing tools (Claude Code, Codex, Antigravity, or similar) — this is the core of the role. Solid automation experience with Playwright, Cypress, or another JS framework — enough to design a suite from scratch and keep it healthy. Must have B2/C1 English Level Preferred Skills and Qualifications AI Tooling & Observability: Exposure to AI evaluation tools, prompt engineering playgrounds, or data lake reporting architectures (e.g., BigQuery, GA4 platforms). Advanced Automation: Experience utilizing automated test frameworks to validate non-static content or multi-t