Testing and Monitoring AI Systems

• Why AI systems require measurable evaluation
• Accuracy and its limitations
• Precision, recall, and F1
• Regression error metrics
• ROC and AUC concepts
• Technical and business performance indicators
• Baseline measurement
• Deployment thresholds

Hands-on Lab: Metrics and Thresholds Workshop — Calculate multiple AI performance metrics, compare the AI system with the existing process, and establish measurable go/no-go thresholds.

• Held-out testing and validation
• Accuracy and completeness of generative AI outputs
• Fact-checking
• Hallucination detection
• Bias detection
• Building reusable golden test sets
• Representative and edge-case tests
• Adversarial test inputs
• Red-teaming AI tools
• Prompt injection
• Jailbreak attempts
• Data-leak probes
• Connected-tool misuse
• Retesting after fixes

Hands-on Lab: Build and Run a Test Set — Create a structured test suite, score performance, identify failure patterns, conduct an adversarial red-team round, apply a fix, and retest the system.

• How bias enters AI systems
• Measuring differences between groups
• Approval and error rates
• Competing definitions of fairness
• Selecting fairness measures appropriate to the application
• Documenting findings and remediation actions

Hands-on Lab: Fairness Check — Calculate group-level outcome or error rates, evaluate the results using defined fairness measures, and document required technical or operational changes.

• Continuous AI monitoring
• Data drift
• Concept drift
• Technical performance metrics
• Business and adoption metrics
• Input-data quality monitoring
• Alert thresholds
• Avoiding alert fatigue
• Retraining triggers
• Validating models after updates

Hands-on Lab: Build a Monitoring Tracker and Detect Drift — Build a monitoring dashboard, configure threshold-based alerts, identify when drift began, and determine whether the system should be adjusted, retrained, or paused.

• Technical AI documentation
• Model cards and data documentation
• Test evidence
• Risk and change records
• Audit trails
• Maintaining documentation after model, data, or prompt changes
• Maintaining ongoing audit evidence

• Controlled experimentation
• A/B testing AI changes
• Comparing model or prompt versions
• User feedback as technical input
• Scaling successful configurations
• Identifying underperforming AI systems
• Retirement and deprecation
• Data handling during shutdown
• Transition and rollback planning

Create a complete technical validation and monitoring package for an AI system, including a reusable test set, test results, performance thresholds, red-team findings, monitoring tracker, improvement experiment, and defined criteria for retraining or retirement. The capstone is assessed against a skills-based rubric and must be successfully completed to earn the CERT4TECH Certificate in Testing and Maintaining AI Tools.