AI Automation QA

Description

Key Accountabilities

• Definition and execution of testing and quality assurance strategies for AI‑enabled workflows
• Continuous evaluation and monitoring of system behavior in production environments
• Contribution to auditability, risk management, and continuous quality improvement

Principal Responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
• Apply LLM evaluation frameworks to...