Evidence memo

RAG Knowledge Copilot

This memo explains what the system demonstrates, how to verify it quickly, and which production risks are explicitly addressed. The live demo is browser-only (Cloudflare-safe) and optimised to surface LLMOps reliability signals.

RAG LLMOps Retrieval Tracing Guardrails Evaluation Harness

Signal → Systems → Proof Ledger → RAG Copilot Memo

Open live system Back to anchor Systems index

CV anchor: /evidence/#rag-copilot

What this proves

The goal is not “a chatbot”. The goal is to demonstrate production-minded RAG: grounded answers, traceability, and refusal policies.

Grounded Drafting Citation Discipline Prompt Injection Defence Regression Safety Monitoring Mindset

Retrieval traceability: shows top-k sources, similarity signals, and readable snippets.
Grounding policy: answer is constrained to retrieved context; weak retrieval triggers refusal (strict mode).
Guardrails: designed to prevent silent hallucinations and prompt injection patterns.
Evaluation mindset: structured to support labelled queries, precision@k tracking, and citation correctness checks.

How to verify (60 seconds)

Step 1 — Run a “monitoring” question

Open the system and ask: "What should be monitored for deployed ML models? Provide citations."

Expected: retrieved sources appear with similarity scores.
Expected: answer includes citations like [1], [2], [3].

Step 2 — Test the guardrail

Ask an unrelated question (e.g. about cooking) while in Strict policy.

Expected: refusal due to low retrieval confidence.
Reason: prevents unsupported answers.

Operational design choices

The live demo is intentionally client-only (no network calls) to keep the portfolio reliable, fast, and safe for public hosting. The architecture patterns map directly to production RAG implementations with vector stores and LLM backends.

Cloudflare Pages No Backend Calls CORS-safe Trace Export

Production risks & mitigations

Hallucinations

Mitigation: strict grounding to retrieved snippets.
Mitigation: refusal on weak retrieval confidence.
Mitigation: citation discipline and trace display.

Prompt injection

Mitigation: treat retrieved content as data, not instructions.
Mitigation: allow-list behaviours; refuse policy overrides.
Mitigation: traceability for audit and incident review.

Next improvements (if deployed with an LLM backend)

Add a vector store (e.g. pgvector / managed service) + embedding pipeline.
Introduce query set + automated eval (precision@k, groundedness, refusal correctness).
Implement access control and PII redaction, plus prompt injection filters.
Monitor latency, cost per query, and retrieval drift over time.

Keywords (ATS trigger set)

RAG LLMOps Retrieval Tracing Vector Search Evaluation Harness Guardrails Prompt Injection Defence Groundedness Citation Discipline

Proof anchor for CV: /evidence/#rag-copilot