RAG & Knowledge Bases over Your Documents

Advanced · Technical & Agentic track · ~35 min hands-on + readings and quiz

← Back to the Technical & Agentic track

Make AI answer from your documents — not its training data — and cite where each point came from.

What you’ll be able to do

  • Build a searchable knowledge base over your own documents
  • Ground answers in sources with citations
  • Check retrieval quality and handle gaps

Overview

Retrieval-augmented generation (RAG) connects a model to your document set: it retrieves relevant passages and answers from them, citing sources. NotebookLM is a friendly entry point; the same idea powers custom knowledge bases. It reduces hallucination but doesn’t eliminate it.

Quality depends on retrieval: if the right passage isn’t found, the answer is wrong or guessed. You check that citations actually support the claim.

Practice activities

Activity 1 · Advanced — Ground answers in your documents

Time ~18 min · Tools NotebookLM (public sources only)

Goal. Make AI answer from your sources, with citations you verify.

Setup. In NotebookLM, load these 5 public PDFs on pedestrian safety:

  1. Philadelphia Vision Zero Action Plan (2017) — the city’s strategy for eliminating traffic deaths
  2. NHTSA Pedestrian Safety Report (2023) — national pedestrian fatality trends
  3. WHO Global Status Report on Road Safety (2023) — global context for pedestrian injury
  4. FHWA Proven Safety Countermeasures: Pedestrian Safety — evidence-based infrastructure interventions
  5. Philadelphia Complete Streets Design Handbook — city design standards for pedestrian infrastructure

Upload all five to a new NotebookLM notebook. These are all public government or NGO documents — no data-sensitivity concerns.

Steps.

  1. Ask a question the sources can answer; for each cited passage, confirm it actually supports the answer.

  2. Probe a gap — ask something the sources don’t cover. Does it admit the gap or fabricate?

  3. Synthesize:

    Summarize what these sources agree and disagree on, with citations.

Expected result. Grounded answers with verifiable citations and a tested “gap” case.

Check your work. A citation marker isn’t proof — open the cited passage and read it.

Common pitfalls. RAG reduces but doesn’t eliminate hallucination; bad retrieval means a wrong answer. Public data only unless the tool is approved.

Stretch (optional). Add a source that contradicts the others and see how it handles the conflict.

Activity 2 · Advanced — Probe retrieval quality

Time ~12 min · Tools NotebookLM

Goal. Understand when retrieval misses relevant material.

Setup. The same notebook.

Steps.

  1. Ask the same question two different ways and compare which passages it retrieves.

  2. Note any obviously relevant source it missed.

  3. Diagnose:

    Why might you have missed [source] for this question, and how should I phrase it to retrieve it?

Expected result. Insight into how phrasing changes retrieval.

Check your work. If rephrasing changes the cited sources, the system is retrieval-sensitive — design questions accordingly.

Common pitfalls. Users assume RAG “read everything”; it retrieves only a few passages. Verify coverage.

Stretch (optional). Organize the sources (clear titles, sections) and see if answers improve.

Activity 3 · Advanced — See retrieval happen: build a minimal RAG pipeline with Claude Code

Time ~25 min · Tools Claude Code (consumer/PI-purchased — Low Risk Data only; these 5 PDFs are public)

Goal. Build the machinery NotebookLM hides: a from-scratch RAG pipeline where you can read the exact passages the system retrieved — and see when it retrieves nothing.

Setup. Open Claude Code in an empty folder. You’ll point it at the same 5 public PDFs from Activity 1:

  1. Philadelphia Vision Zero Action Plan (2017)
  2. NHTSA Pedestrian Safety Report (2023)
  3. WHO Global Status Report on Road Safety (2023)
  4. FHWA Proven Safety Countermeasures: Pedestrian Safety
  5. Philadelphia Complete Streets Design Handbook

Reminder: Claude Code is not Drexel-supported and is approved for Low Risk Data only — never feed it student records, PHI, or unpublished research data.

Steps.

  1. Ask Claude Code to build the pipeline and make retrieval visible:

    Download these 5 public PDFs [paste the 5 URLs], extract the text, and split it into paragraph-sized chunks tagged with source and page. Write a small Python script that takes a question, retrieves the 3 most relevant chunks (keyword or embedding similarity is fine), and prints each retrieved chunk IN FULL — source, page, similarity score, and the literal text — before printing an answer based only on those chunks.

  2. Run a question the sources can answer, and read the retrieved chunks yourself:

    Run the script with: “What speed-management countermeasures reduce pedestrian deaths?” Show me the 3 retrieved chunks in full and explain why each scored high.

    In Activity 1 you trusted a citation marker; here you can see the 3 passages chosen out of hundreds and judge whether retrieval picked the right ones.

  3. Repeat Activity 1’s gap probe — but this time watch retrieval fail in the open:

    Run the same script with: “What do these documents say about e-scooter injury rates in Philadelphia?” Show me the retrieved chunks and their similarity scores, and tell me whether anything relevant was actually found.

    With NotebookLM you had to infer a gap from the tone of the answer; here the low scores and off-topic chunks show you directly that nothing relevant exists in the sources.

  4. Break retrieval on purpose to see why chunking matters:

    Re-chunk the documents into much larger chunks (e.g., 2 pages each) and rerun question 2. Did the retrieved passages get noisier? Explain how chunk size changed what the answer was grounded in.

Expected result. A working script that answers from the 5 sources while printing the literal retrieved passages; one question where retrieval visibly succeeds, one gap question where you can see it visibly fail.

Check your work. Read the retrieved chunks before reading the answer — does each one actually contain the evidence the answer claims? If the answer says something no printed chunk supports, the model added it.

Common pitfalls. A high similarity score doesn’t mean the chunk is right — keyword overlap can retrieve the wrong passage confidently. Retrieval quality is the whole game: garbage chunks in, confident garbage out. Low Risk Data only in Claude Code unless separately approved.

Stretch (optional). Ask the same question here and in NotebookLM (Activity 1) and compare: where NotebookLM gave a citation marker, can you find the equivalent chunk your pipeline retrieved? Which experience made an error easier to catch?

Check your readiness

Answer these, then check — your score suggests whether to dive in or skim the readings first.

Useful resources