Essential session storage is required to sign in. Optional analytics remain off unless you allow them.
Build an agent that answers questions over a private document set using RAG. Chunk and EMBED the documents into a vector store (Chroma/pgvector/Qdrant/etc.), give the agent a retrieval tool, and have it retrieve relevant chunks into context and answer GROUNDED in them WITH CITATIONS (reducing hallucination). Practice context engineering — include only the relevant chunks within the budget. CRUCIALLY: build a small EVAL SET of question/expected-answer pairs and evaluate the agent (programmatic checks and/or LLM-as-judge), reporting task-success rate, and add basic tracing so you can see each retrieval and LLM call.
Chunk/embed/retrieve works; answers are grounded in retrieved chunks with citations; hallucination reduced; relevant-only context.
A real eval set with programmatic and/or LLM-as-judge scoring; task-success rate reported; used to improve the agent.
Tracing of retrievals + LLM calls so a run can be inspected/debugged.
Context kept within budget and relevant; not stuffing everything in.
Use this README structure to make the work reviewable.
DevPath fetches a read-only, size-capped snapshot. Submitted code is never executed.
# A grounded RAG agent with evals ## Goal Build an agent that answers questions over a private document set using RAG. Chunk and EMBED the documents into a vector store (Chroma/pgvector/Qdrant/etc.), give the agent a retrieval tool, and have it retrieve relevant chunks into context and answer GROUNDED in them WITH CITATIONS (reducing hallucination). Practice context engineering — include only the relevant chunks within the budget. CRUCIALLY: build a small EVAL SET of question/expected-answer pairs and evaluate the agent (programmatic checks and/or LLM-as-judge), reporting task-success rate, and add basic tracing so you can see each retrieval and LLM call. ## Acceptance criteria - [ ] RAG quality: Chunk/embed/retrieve works; answers are grounded in retrieved chunks with citations; hallucination reduced; relevant-only context. - [ ] Evaluation: A real eval set with programmatic and/or LLM-as-judge scoring; task-success rate reported; used to improve the agent. - [ ] Observability: Tracing of retrievals + LLM calls so a run can be inspected/debugged. - [ ] Context engineering: Context kept within budget and relevant; not stuffing everything in. ## Verification - [ ] Document installation and run commands. - [ ] Record automated checks and their exact commands. - [ ] Add representative output, screenshots, or a short demo where useful. - [ ] Confirm failure paths and known constraints. ## Decision log ### Decision title - Context: - Choice: - Alternatives considered: - Tradeoffs: ## Evidence - Link each rubric criterion to the file, test, or artifact that demonstrates it. ## Known limitations - List what is intentionally out of scope and what should be improved next.