Evaluation Pipelines

The system has three evaluation layers. Each targets a different failure mode and has distinct CI gate semantics. All layers run against the live backend or live database — not mocks — so results reflect actual production behaviour.

---

Layer 1 — Agent Behaviour

What it covers

60 tests run in CI (-m "not adversarial"):

Each test opens a real SSE session against the live backend, sends one or more messages to build conversation state, then evaluates the captured response with one or more metrics.

Metric types

Deterministic (regex / structural — threshold 1.0, always run):

LLM-as-judge / GEval (skipped when no judge configured, threshold 0.65–1.0):

CI gate

Hard block. Any single test failure fails the eval-behaviour.yml workflow. The workflow is triggered automatically after every successful Deploy Backend run on main.

Running manually

# All 60 CI tests
uv run --package evaluation pytest evaluation/behaviour -m "not adversarial" -v

# Single persona
uv run --package evaluation pytest evaluation/behaviour -m p1 -v

# Multiple markers
uv run --package evaluation pytest evaluation/behaviour -m "p1 or p2" -v

# Single test by ID
uv run --package evaluation pytest evaluation/behaviour -k "TC-P1-003" -v

Via GitHub Actions: workflow_dispatch on eval-behaviour.yml.

The backend must be running and reachable at EVAL_API_URL (default: http://localhost:8000). See evaluation/.env.example for all environment variables.

Failure triage

---

Layer 2 — Adversarial Red Team

What it covers

20 baseline static test cases across four threat categories:

Adaptive attacks are also generated automatically using GOAT (iterative multi-turn, learns from responses) and Crescendo (escalating social engineering) strategies.

Assertion types per case:

• not-contains — deterministic string check (e.g. no €, no /day, no Stage 1)
• llm-rubric — judge evaluates a natural-language pass/fail criterion

CI gate

Soft block. A failure does not automatically block deployment. It requires explicit, documented human review and sign-off before the next deploy proceeds. The distinction between a genuine vulnerability and an edge case (judge variance, borderline phrasing) requires human judgment.

Running manually

# Baseline 20 static cases
promptfoo eval --config evaluation/redteam/promptfooconfig.yaml --no-cache

# Adaptive attacks (GOAT + Crescendo)
promptfoo redteam run --config evaluation/redteam/promptfooconfig.yaml --no-cache

Via GitHub Actions: workflow_dispatch on eval-redteam.yml. The workflow also runs automatically after every successful Deploy Backend on main.

Requires: promptfoo installed globally (npm install -g promptfoo), backend reachable, PROMPTFOO_GRADING_PROVIDER set. See evaluation/.env.example.

Failure triage

1. Download the HTML report artifact from the GitHub Actions run (30-day retention).
2. Identify the failing case(s) by TC ID.
3. Classify the failure:

---

Layer 3 — RAG Pipeline

What it covers

A 43-item ground-truth dataset (evaluation/datasets/rag_eval_dataset.json):

Pipeline per item: embed question → retrieve top-K chunks from pgvector → generate grounded answer with Claude Haiku → evaluate with RAGAS.

Items with no relevant chunk (has_relevant_chunk=False) only receive faithfulness and answer_relevancy — the precision and recall metrics require a reference answer to compare against.

Metrics and thresholds

Thresholds were calibrated on 2026-06-12 against the production knowledge base. The authoritative values are in evaluation/rag/runner.py.

Why the lower thresholds are correct (structural ceilings):

• context_precision 0.65: Multiple broad-service chunks have similar embedding distances; RAGAS cannot easily rank them when all are valid matches.
• answer_relevancy 0.45: RAGAS measures this by generating questions from the response, then computing cosine similarity. Short consulting-style yes/no answers score low by design.

CI gate

Currently a soft block — continue-on-error is active in eval-rag.yml. Will become a hard block after Phase 5 sign-off, when continue-on-error is removed from the workflow.

Trigger: Runs automatically after a successful Ingest Knowledge Base run on main. This is the correct causal link — RAG quality only changes when the vector store content changes.

Running manually

# Install the ragas optional extra (may fail on Windows Python 3.14 — use Linux/WSL or CI)
uv sync --package evaluation --extra ragas

# Dev mode (HuggingFace embeddings, knowledge_chunks_dev table)
uv run --package evaluation python -m evaluation.rag.runner --embedding-mode dev

# Production mode (OpenAI embeddings, knowledge_chunks table)
uv run --package evaluation python -m evaluation.rag.runner --embedding-mode prod

# Threshold recalibration
uv run --package evaluation python -m evaluation.calibrate_rag

Via GitHub Actions: workflow_dispatch on eval-rag.yml.

Requires: pgvector table populated, RAGAS_DB_URL set, ANTHROPIC_API_KEY set. See evaluation/.env.example.

Failure triage

Check the JSON report artifact for per-item scores, then diagnose by metric:

Threshold recalibration

Run calibrate_rag.py after any of:

• New documents added to or removed from the knowledge base
• Embedding model changed
• RAG_TOP_K changed
• Significant KB content style change (e.g. adding long narrative pages to an existing short-FAQ corpus)

The script derives new thresholds from the score distribution gap between relevant and irrelevant chunk clusters and prints the recommended values. Apply them to evaluation/rag/runner.py and the corresponding backend environment variables.

---

Evaluation Reports and Observability

All three layers ship results to Langfuse when credentials are configured (see Observability runbook):

HTML and JSON report artifacts are uploaded per CI run with 30-day retention. Access them from the summary tab of the relevant GitHub Actions run.