All in One View
Content from Before We Use AI: What Are We Practising?
Last updated on 2026-08-31 | Edit this page
Estimated time: 28 minutes
Overview
Questions
- What are we actually practising in this lesson?
- When does an AI coding tool help me learn, and when does it get in the way?
- How do I decide whether to trust a result the AI gave me?
Objectives
- Distinguish between code generation, code understanding, and code validation.
- Use a structured checkpoint to decide whether an AI-generated result is safe to run, revise, or reject.
This is not a lesson about letting AI do the work
It is tempting to treat an AI coding agent as a machine that turns a sentence into a finished script. You type “clean this data,” code appears, it runs, and you move on.
This lesson is about the opposite skill. The point is to learn how to inspect, question, constrain, and validate AI-generated code so that the result is trustworthy enough for research. Throughout the lesson you will be asked to explain what the AI did, not only run what it produced.
Three things are easy to confuse:
- Code generation is getting the AI to produce something that runs.
- Code understanding is being able to say, in plain language, what the code does and why.
- Code validation is having evidence that the code is correct for your data and your research question.
Generation is the easy part now. Understanding and validation are the parts that make code trustworthy, and they are still your job.
AI can make code appear before understanding appears
A chatbot or agent can produce a working-looking script faster than you can read it. That is the core risk for a learner. If the code shows up before you understand it, you have skipped the step where learning actually happens.
In research this has a second cost. Working code is not the same as trustworthy code. A script can run cleanly, pass a quick check, and still quietly change a threshold, drop rows, or misread a date format in a way that changes your conclusion.
Why generated code can raise your cognitive load
It sounds backwards: if the AI writes the code, shouldn’t that be less work? For a novice, often the opposite is true. To judge whether generated code is correct, you have to hold in your head:
- what you asked for,
- what the code actually does,
- what assumptions it made,
- and whether those assumptions fit your data.
Verifying code you did not write is frequently harder than writing it yourself. When the AI also reaches for a library you have not seen, an advanced language feature, or an abstraction the lesson has not covered, the work of understanding grows instead of shrinks. We will name these moments when they come up.
The goal of this episode is to reset expectations before learners open a terminal. Some will arrive expecting a productivity demo. Be explicit that the workshop measures whether they can explain and validate AI output, not how fast they can generate it.
Signs to watch for across the whole workshop:
- Learners who are impressed that the agent can read their files but cannot say what it changed.
- Learners who accept output they cannot explain because “it ran.”
- Learners who turn to the AI before turning to a helper, which hides where they are stuck.
Normalise bringing AI confusion back into the room. Treat confusing AI output as a shared teaching artefact, not a personal failure.
The run / revise / reject checkpoint
Every time the AI hands you something, you make a decision. Make it on purpose. Ask:
- Can I explain what this does? If not, you are not ready to approve it.
- What did it assume about my data? Name at least one assumption.
- What evidence do I have that it is correct? “It ran” is not evidence of correctness.
- Decision: run it, revise it, or reject it.
You will use this same checkpoint, in more detail, throughout the lesson.
Challenge: practise the checkpoint
Read this short snippet that an agent produced in response to “remove the bad rows from my data”:
PYTHON
import pandas as pd
df = pd.read_csv("data.csv")
df = df.dropna()
df.to_csv("clean.csv", index=False)Without running anything, answer in a sentence or two each:
- In plain language, what does this script do?
- What does it assume “bad rows” means?
- Name one way this could quietly damage a real research dataset.
- Would you run it, revise it, or reject it for a dataset where missing values are meaningful (for example, a survey where “no answer” is a valid response)?
- It reads
data.csv, drops every row that has a missing value in any column, and writes the result toclean.csv. - It assumes “bad rows” means “rows with any missing value.” That is the AI’s interpretation, not yours.
-
dropna()with no arguments removes a row if any column is empty. In a wide dataset, one empty optional field can delete an otherwise complete observation, silently shrinking your sample. - Revise or reject. For data where missingness is meaningful, dropping those rows changes what the data represents. You would constrain which columns to check, or impute, instead. The code runs perfectly and is still wrong for your question.
Quick check: what does dropna()
actually remove?
Given a table with columns site, date,
score, and notes, where only
notes has some blank cells, what does
df.dropna() (no arguments) do?
- Drops rows where every column is blank.
- Drops rows where the
notescolumn specifically is blank. - Drops rows where any column is blank, including
notes. - Drops the
notescolumn entirely, keeping all rows.
3. dropna() with no arguments checks
every column and drops a row if any of them is blank, even if
the missing value is in a column you would consider optional, like
notes. Options 1 and 2 describe common but wrong
assumptions about how it scopes the check; option 4 confuses dropping
rows with dropping a column, which is a different method
(drop(columns=...)) entirely.
Feedback checkpoint: what do you want from AI?
On a sticky note or in the shared Etherpad, write one line: what do you most want an AI tool to do for your research, and what are you most worried it will get wrong? Working through this alone? Write the line in your own notes file instead, you will revisit it at the end of the lesson either way.
We will come back to these at the end of the lesson.
- This lesson teaches you to inspect, question, constrain, and validate AI output, not to delegate your thinking.
- Generation, understanding, and validation are different skills. Generation is the easy one.
- For a novice, verifying generated code can cost more effort than writing it.
- Working code is not the same as trustworthy code. Use the run / revise / reject checkpoint every time.
Content from CLI-Based AI
Last updated on 2026-08-31 | Edit this page
Estimated time: 50 minutes
Overview
Questions
- What can each kind of AI tool see, and what can it change?
- Why use a CLI for AI instead of a browser?
- What is the Living Spec and why does it matter?
Objectives
- Describe what different AI tools can see and what they can change.
- Create a Living Spec (CLAUDE.md) that states your hard constraints, and explain why doing so keeps you the active reviewer of the agent’s output.
Why CLI matters for research
Most researchers start with chat-based AI in a browser. Browser and desktop AI products increasingly ship connectors, uploaded-file access, or even their own coding/agentic surfaces, so “browser tools are sandboxed, CLI tools aren’t” is no longer a reliable rule. What actually matters is the configured surface for the specific tool in front of you: what it’s connected to and what permissions it has, not which category it falls in. Check that directly rather than assuming from “browser” vs. “CLI.”
If you have not used a CLI agent before, that puts you with most of
your peers, not behind them: a 2026 survey of 868 scientists who program
found browser-based general-purpose chat tools accounted for the large
majority of primary tool choice, with CLI/agentic tools a small minority
(O’Brien et al., 2026).
This lesson teaches the less-common path deliberately, not because it is
trendy, but because of a structural advantage a stateless chat interface
cannot match: the persistent external brain described later in this
episode. A browser chat forgets your project’s rules the moment you
close the tab; a file like CLAUDE.md reloads them at the
start of every session, under version control, reviewable by you or
anyone else.
A CLI (Command Line Interface) agent runs in your terminal, the same
place you run Python scripts or navigate your filesystem with
ls and cd, and has access to three things a
browser tool does not.
Your files and data. The agent can read your actual datasets, inspect your directory structure, and write scripts directly to disk. You are not copying and pasting between a chat window and a code editor. The agent works in your project the way a collaborator sitting at your machine would.
Your installed tools. Your machine probably has domain-specific software on it: geospatial tools like GDAL, bioinformatics pipelines, R packages, custom scripts, institutional data connectors. A browser AI has no idea these exist. A CLI agent can call them directly, pass output between them, and build on what you already have installed.
An iterative loop. When a script fails, the agent sees the error output in the terminal and can try again. You are not copying stack traces back into a chat window. The feedback loop is tight and stays in one place.
What the tool can see, and what it can change
Before you trust any AI tool, the first question is always: what can it see, and what can it do? The more access a tool has, the more it can help, and the more it can quietly get wrong. These four kinds of tools sit along that spectrum.
| Tool type | What context it has | What it can do | What can go wrong | What a novice should verify |
|---|---|---|---|---|
| Chatbot (browser, e.g., ChatGPT, Gemini web) | Only what you paste in | Suggests code and text | No view of your real files; guesses at structure; you copy code by hand | That the code matches your actual columns and files, not the example it imagined |
| IDE assistant (e.g., Copilot in VS Code) | The file you have open, sometimes nearby files | Suggests and inserts code inline | Sees only part of the project; may complete code that fits the line but not the goal | That the suggestion does what you intended, not only what looks plausible |
| CLI agent (e.g., Claude Code, Codex CLI) | Your project directory: files, data, structure | Reads files, runs code, writes scripts to disk | Can edit or delete real files; can act on a misread of your data | What files it read, what it changed, and what it ran, before you approve |
| Fully agentic workflow (multi-step, runs tools on its own) | Whatever you grant, across many steps | Plans and executes a chain of actions with little input | Errors compound across steps; hard to see where it went wrong | That you can still explain each step and reproduce the result |
As you move down the table, the tool can do more for you and more to you. Nothing in this table removes your responsibility to understand the result.
Learners are often impressed that a CLI agent can read their files and run their code. Impressive access is not the same as a correct result. Watch for learners who can describe what the agent can do but not what it just did.
Before any learner approves a command that changes files, ask them to say out loud: what did the agent read, what is it about to change, and why. If they cannot answer, that is the moment to slow down, not speed up.
Data privacy and institutional context
Your institution decides which AI tools are approved for which kinds of data, and the free tools are usually not the ones you can point at sensitive research data. At UCLA, the centrally provided free tools (Gemini Basic, Microsoft Copilot, ChatGPT web) are web-only and approved for data classified P1-P3, with P4 requiring approval. None of them is a terminal agent. See UCLA’s available AI tools list.
For the terminal workflow in this lesson, think in two paths:
- Personal plan or API key for non-sensitive (P1-P3) work. Simple to set up; this is what most workshop exercises assume. Do not use it with sensitive or restricted data.
- UCLA Amazon Bedrock (Anthropic models) for sensitive (P3/P4) research data. Claude Code can run against Bedrock with the same commands; only the backend changes. Confirm your unit’s access and data-tier approval first.
Warning: Personal accounts often lack the privacy protections of an institutional agreement. Consult your campus data policy before using any AI tool with sensitive data. PHI and attorney-client privileged information are not approved for these tools.
Looking ahead: If your research requires fully local processing, these same skills transfer to open-weight models (like Qwen3, Gemma, or OpenAI’s gpt-oss) run via Ollama. Check both the license and the specific version before you rely on one for reproducibility.
From writer to active reviewer
People sometimes describe this shift as moving from “writer” to “orchestrator,” as if the AI now does the work and you just conduct. That framing is misleading, and for a learner it is risky.
A more honest version: AI may reduce the need to recall every detail of syntax, but it increases the need to understand intent, dependencies, assumptions, tests, and failure modes. You are not handing off the thinking. You are moving the work from typing towards reading, questioning, and judging. That is harder to do well, not easier.
You guide the agent using a Living Spec, and then you review what it produces against that spec. The diagram below shows the loop: you define the goal, the agent proposes a plan, you approve before any code is written, and you verify the result before it counts as done.
graph TD
accTitle: Living Spec approval and verification loop
accDescr {A researcher defines a goal in CLAUDE.md, the agent proposes a plan, and the researcher approves it or sends it back at an approval gate. Only after approval does the agent draft code, which then passes a verification step or returns for refinement.}
A[Researcher] -->|Define goal| B(CLAUDE.md\nLiving Spec)
B --> C[Request a plan]
C --> D{Approval Gate}
D -->|Approve| E(AI Agent executes)
D -->|Revise| C
E -->|Draft code| F{Verification}
F -->|Passed| G[Final Output]
F -->|Failed| H[Refinement]
H --> B
style D fill:#bbf,stroke:#333,stroke-width:2px
style F fill:#f9f,stroke:#333,stroke-width:2pxDescription of the diagram: a loop. The researcher defines a goal
in CLAUDE.md, the agent proposes a plan, and the researcher
approves it or sends it back at an approval gate. Only after approval
does the agent draft code, which then passes a verification step or
returns for refinement. The researcher stays in control at every
gate.
Ask learners: “Have you ever used ChatGPT to write code that looked correct but failed when you ran it?” This is a good time to introduce the concept of orchestration. The goal is not only to “fix” code, but to ensure the AI’s intent (the spec) is correct.
This introduces a new challenge: verification load. You must coordinate and validate the agent’s actions against your requirements.
Managing cognitive load
It is common to feel “out of the loop” when the AI generates many lines of code quickly. To manage this, focus on anchoring your understanding. Read the comments the AI generates and test small pieces of code frequently. If a block of logic is confusing, ask the AI to explain it before moving on.
File system access
Unlike browser tools, Claude Code has access to your working environment. It can read project context from the directory structure and modify files. Instead of copying and pasting code, the agent writes scripts to your disk and can iterate based on terminal errors.
Security responsibility
Giving an AI agent access to your filesystem is a security responsibility. A buggy or misconfigured agent could delete files or access sensitive data, such as passwords.
Always consider that your tools can have unintended consequences. Ensure files are backed up or under version control (like Git) so you can revert unwanted changes.
Long context
Like humans, we only have a certain amount of working memory, and large language models (LLMs) operate in a similar way. This is called the context window in LLM tools. Current models like Claude have long context windows (hundreds of thousands of tokens, up to a million in some configurations), but a large window doesn’t mean the agent should load your entire project at once. The more useful pattern is that the agent inspects and retrieves the specific files it needs within its configured access, rather than everything being crammed in up front.
This allows you to describe the desired state of your project, and the agent coordinates changes across multiple files. Call this intent specification, not “declarative programming”: you’re describing what you want, but the agent still writes an ordinary, imperative implementation, and the spec itself is not executable the way real declarative code is.
A large context window is not a free pass
The more you load into a session, the more the model has to track. Beyond a certain point, quality degrades, the model may lose track of earlier instructions, produce inconsistent output, or fixate on the wrong files. This is sometimes called context poisoning.
A large context window makes this easier to run into, not harder. Managing what goes into your context is part of the workflow, not an afterthought.
Let’s make sure this works
Open a terminal window and type claude --help. You
should see a usage summary listing the options and slash commands
available. Claude Code defaults to an interactive session; the
-p (or --print) flag runs a single prompt
non-interactively (headless mode), which we use for quick one-off
checks.
Navigate to your project folder, coastal-water-quality,
and run a quick headless check:
BASH
cd coastal-water-quality
claude -p "What operating system am I on? List the files here and in data/."Compare the output to what you see when you run ls and
ls data/. Did the AI describe your project accurately? The
AI should return something like:
You are on macOS (Darwin). This looks like a data-cleaning project.
Top level: README.md, CLAUDE.example.md, validate_data.py, data/, expected_outputs/
data/: site_A.csv, site_B.csv, site_C.csvA first look at the data
Before you let the agent touch anything, look yourself. In your terminal:
Then ask the agent to describe the same file, and compare:
BASH
claude -p "Describe the columns in data/site_A.csv and note anything inconsistent or risky for analysis."Did its description match what you saw with head? Where
it added an interpretation (for example, guessing what a column means),
note that as the agent’s assumption, not fact. This habit,
checking what the tool claims against the data itself, is the whole
point of the lesson.
Now let’s initialize the project so the agent has persistent context.
Working directory matters
Always start Claude Code from inside your project folder. The agent uses the current directory to find your files and spec. Starting from the wrong folder, such as your home directory, is one of the most common sources of confusion in a workshop.
Initialize your project
Claude Code includes an /init command that creates a
CLAUDE.md file describing your project in your working
directory:
You are now inside a Claude Code session. Type / to see
the available slash commands and page through the full list. Notice
/init, this is the command that will initialize our
project. Let’s run it:
Notice that it inspects your files and folders. After it finishes,
let’s see what new files have been created. You can run a shell command
from inside the session by starting the line with !:
This shows the files that are present. You should see a file named
CLAUDE.md. Let’s look inside it:
Here is the kind of thing Claude Code generates for this project:
MARKDOWN
# CLAUDE.md
## Project Overview
This is a data-cleaning project. The `data/` directory holds three water quality
files (`site_A.csv`, `site_B.csv`, `site_C.csv`) with inconsistent column names and
date formats. The goal is to merge them into `data/master_dataset.csv` and analyse
water quality trends. See `README.md` for the target schema.
## Key Files
- **`data/site_*.csv`**: raw per-site measurements (do not edit).
- **`validate_data.py`**: checks for the merged dataset.
- **`CLAUDE.md`**: project context and rules, loaded automatically each session.
## Usage
Re-run `/init` after the project changes, and edit this file by hand to record your
goals, constraints, and rules.A few things to notice. Claude Code scanned the directory and described the real project, including the messy site files. Because this folder has data, the generated spec is already useful. You will still edit it by hand to add the goals, constraints, and rules the agent must follow.
The Living Spec
To get the most out of a CLI agent, provide it with persistent context about your project. This acts as a “Living Spec”, a set of rules and goals the agent must follow across every session.
Every major CLI tool has its own native spec file that it loads automatically when you start a session:
| Tool | Native spec file |
|---|---|
| Claude Code | CLAUDE.md |
| OpenAI Codex | AGENTS.md |
| Cursor | .cursorrules |
AGENTS.md is also emerging as a
portable convention across tools: OpenAI released it in
August 2025, and it was contributed to the Linux Foundation’s Agentic AI
Foundation in December 2025 alongside Anthropic’s MCP and Block’s goose.
But whether a tool auto-loads it varies, so check before
assuming:
| Tool | Auto-loads AGENTS.md? |
|---|---|
| OpenAI Codex CLI | Yes, it’s the native file above |
| Claude Code | No. It reads CLAUDE.md only. Add
@AGENTS.md as an import inside your CLAUDE.md,
or symlink CLAUDE.md to AGENTS.md, to bring it
in |
| Gemini CLI | No by default; needs explicit configuration |
For a tool that doesn’t auto-load it, you can still reference it
explicitly in a prompt: "Read AGENTS.md and then...".
That’s what makes it portable: it travels with your project even when
the tool you’re using doesn’t pick it up on its own.
What to include in your spec file
Use this file to define:
- Current Goal: What you are working on right now.
-
Rules of the Road: Technical constraints (e.g.,
“Always use
pandasfor dataframes”). - Verification Gates: How you will confirm the code is correct.
Your project’s external brain
A model forgets everything between sessions, and even within a session its context window is limited. The fix researchers have settled on is to keep the project’s memory in plain markdown files that the agent reads and updates. Andrej Karpathy popularized calling this an “external brain.” Three files do most of the work:
-
CLAUDE.md: durable rules, goals, and constraints, auto-loaded every session (the file you just created). -
PLAN.md: the step-by-step plan for the task at hand. It is temporary, and you will meet it in the next episode. - A running notes file (for example
NOTES.md): a dated log of what was tried, what worked, and why you made key choices.
For research, that notes file is not bureaucracy. It is provenance: it is how you, a reviewer, or future-you reconstruct what the agent knew and why a result came out the way it did. Because the external brain lives in your repo and under version control, it stays reviewable and reproducible, unlike the model’s hidden and disposable memory.
Challenge: Initialize and customize your spec file
Inside your Claude Code session, run /init to create a
CLAUDE.md file. Then open it in a text editor and add one
“Hard Constraint” (something the AI must do) and one “Success
Metric” (how you know it’s done).
MARKDOWN
# Project: Arctic Sea Ice Analysis
## Goal
To analyse trends in sea ice extent from 1980-2020.
## Rules of the Road
- **Hard Constraint**: Only use the `xarray` library for spatial data processing.
- **Success Metric**: All final plots must include a valid DOI reference for the data source.
## Conventions
- Use snake_case for variable names.
- Save all plots to the `figures/` directory.Feedback checkpoint: describe the agent’s context
Before we move on, turn to the person next to you and answer out
loud: when you ran /init, what did the agent look at, and
what file did it create? If you are not sure, say so. In the shared
Etherpad, paste one thing the agent did that surprised you. Working
alone? Answer out loud to yourself, or write it as a comment at the top
of your CLAUDE.md, saying it out loud (or writing it down)
is what surfaces the gaps, not who is listening.
- Different AI tools see and change different things; always ask what a tool can see and do before trusting it.
- A CLI agent can read, run, and edit your real files, which makes verifying what it changed part of the workflow.
- Run
/initinside Claude Code to create aCLAUDE.mdLiving Spec that reduces context drift. - A portable
AGENTS.mdlets the same spec travel across different AI tools, but auto-loading it isn’t universal; check per tool (Claude Code needs an explicit@AGENTS.mdimport). - The shift is from writing syntax to actively reviewing intent, assumptions, and evidence; it does not remove your responsibility.
Content from Best Practices for Prompting
Last updated on 2026-08-31 | Edit this page
Estimated time: 75 minutes
Overview
Questions
- How do I write effective prompts?
- How do I review a plan before the agent acts on it, not just after?
- What are common AI failures?
- How can I make the AI fix its own mistakes, and when should I not trust that it has?
Objectives
- Refine a vague prompt into one with context, specificity, and output instructions.
- Use Claude Code’s plan mode to review an agent’s approach before it writes any files.
- Use introspection to refine AI-generated code.
Working inside Claude Code
All prompts in this episode are typed inside an active Claude Code session. Start one in your project folder before the exercises:
Then type prompts directly at the prompt. Shell commands (like
python script.py) are run in a separate terminal window, or
from inside the session by prefixing the line with !.
Five principles of effective prompting
Effective prompting is clear technical communication. To get the best results, start by being specific. Include constraints, filenames, and a description of your expected output. Vague requests lead to generic answers, while precise instructions result in usable code.
Provide context. Explain why you need the code and what data you have (e.g., “I am processing a CSV file with these columns…”). This helps the AI understand the goal. Specify outputs clearly, tell the AI where to save files or how to format tables.
Treat prompting as an iterative process. Start with a simple request and add complexity in follow-up prompts. Include validation steps by asking the AI to verify or test its own work.
The CO-STAR framework
Optional on a first pass. CLEAR (below) is enough to start; reach
for CO-STAR when a prompt gets complex. While CLEAR helps with
conversation flow, CO-STAR structures complex research prompts that
eventually become part of your CLAUDE.md:
- Context: Provide background (e.g., “I am a biologist analysing RNA-seq data”).
- Objective: Define the specific task (“Write a script to normalise these counts”).
- Style: Specify the coding style (“Use the Tidyverse style guide in R”).
- Tone: Set the personality (“Be concise and prioritise readable code”).
- Audience: Who is this for? (“For a graduate student who knows R but not bioinformatics”).
- Response: Define the format (“A single R script with comments and a plot output”).
The Bootstrap Workflow
Instead of writing a full CLAUDE.md by hand, use the
Bootstrap Workflow. This lets the agent assist in
defining the project spec from the start.
- Scan: Ask the agent to scan your directory and data files.
-
Draft: Ask the agent to write an initial
CLAUDE.mdbased on what it sees and your high-level goal. - Gate: You review, edit, and approve the spec before any code is written.
Example bootstrap prompt
“Scan the CSV files in data/raw/. Based on my goal of
‘Analysing water quality trends’, draft an CLAUDE.md file
that defines the column schema, required libraries, and a plan for
cleaning the data.”
Concrete example: From bad to good
| Aspect | Bad prompt | Good prompt |
|---|---|---|
| Vague vs specific | “Clean this data.” | “In data.csv, remove rows with missing
values in the ‘age’ column and save as
clean_data.csv.” |
| No context vs context | “Write a plot script.” | “I am building a report for a climate study. Write a
Python script using seaborn to create a line plot of ‘temp’ over ‘year’
from results.csv.” |
| Silent vs validated | “Run a t-test.” | “Perform a paired t-test between ‘pre’ and ‘post’ columns. Print the t-statistic, p-value, and an interpretation of the result at alpha=0.05.” |
Prompts that preserve learning
Good prompting is technical communication, but in a learning setting it is also learning design. A prompt can be specific and well-formed and still rob you of the understanding you came to build. The prompts below are written to keep you in the loop: the AI helps, but you still do the thinking that makes the result yours.
Prompts that preserve learning:
- “Do not give me the final code yet. Ask me three questions about the data first.”
- “Give me a plan using only concepts we have covered so far.”
- “Write the simplest possible version. Avoid list comprehensions, classes, and external libraries.”
- “Explain what assumptions you are making about my data.”
- “Give me one small change to make myself, and tell me where to make it.”
- “Ask me to predict the output before you show me the answer.”
- “Give me a hint, not the solution.”
Prompts to avoid, and what to ask instead
The prompts on the left feel efficient but hand over the parts that make code trustworthy. The versions on the right keep you able to explain and validate the result.
| Avoid | Ask instead |
|---|---|
| “Do this exercise for me.” | “Walk me through how to approach this; do not write the final answer.” |
| “Fix everything.” | “List the problems you see, ranked. I will choose which to fix first.” |
| “Make this production ready.” | “Name the three biggest risks in this script for research use.” |
| “Clean this data.” | “Tell me what inconsistencies you find in these files. Do not change anything yet.” |
| “Write the whole pipeline.” | “Outline the pipeline in steps. We will build and check one step at a time.” |
Generated prompts often pull in syntax, libraries, or abstractions the lesson has not introduced. Signs that AI output is adding extraneous load:
- The agent uses an advanced feature (comprehensions, classes, decorators) before it has been taught.
- It imports a library that is not installed locally.
- It writes several files when one short script would do.
- It buries the core logic under heavy comments.
- The answer is correct but the learner cannot explain it.
Interventions: ask the learner to request a simpler version, to remove one abstraction, to trace the code line by line, or to compare it with a minimal reference solution. Slowing down here is the lesson, not a detour.
Write CLEAR vertically on a whiteboard. As you explain each letter, add the keyword (Concise, Logical, Explicit, Adaptive, Reflective). This helps students remember the framework.
The CLEAR framework
The CLEAR framework, developed by Leo Lo, provides a structured approach to prompt engineering:
graph LR
accTitle: The CLEAR prompting loop
accDescr {Effective prompts move from Concise to Logical to Explicit to Adaptive to Reflective, with a feedback loop from Reflective back to Adaptive when the output needs another pass.}
C[Concise] --> L[Logical]
L --> E[Explicit]
E --> A[Adaptive]
A --> R[Reflective]
R -->|Feedback Loop| A
style R fill:#bbf,stroke:#333,stroke-width:2pxThe diagram traces the CLEAR loop, from Concise to Logical to Explicit to Adaptive to Reflective, with a feedback arrow from Reflective back to Adaptive. Effective prompts are concise and logical, prioritising important information and following a sequence of steps. They are also explicit, specifying the scope, persona, and tone of the output. When the AI produces poor results, be adaptive by rephrasing or splitting tasks. Finally, be reflective, evaluate the output and verify facts using other sources rather than trusting the response.
Introspection
The CLEAR framework guides your input, but you can also force the AI to critique its own output. This is often called self-correction.
Emphasize this section. Most learners treat AI output as final. The idea that they can ask the AI to fix its own work is often a new concept. It is like asking a student, “Are you sure you checked your work?”, they often find their own mistakes when asked.
Asking the AI to review its own code often surfaces problems, but it cannot decide which problems matter for your research; that judgement stays with you. Never accept the first draft. Follow up with an introspection prompt:
- “Review the code you just wrote. Are there any edge cases or security vulnerabilities?”
- “Did you hardcode any file paths?”
- “Critique your own implementation. Is there a more efficient way?”
Reasoning effort
Optional: useful once you are comfortable with the basics.
Frontier models no longer split cleanly into separate “standard” and
“reasoning” model families the way they did in 2025. The current
generation instead lets you turn up a reasoning-effort setting on the
same model. In Claude Code this is the effort setting
(low through xhigh, with max on
some models); the default is high, which suits most complex
reasoning and coding work.
When to raise it:
- Reach for more effort when a task has multiple interacting constraints, or chains many steps of tool use together, not just because it “feels hard.”
- Leave it at the default for routine formatting, quick scripts, and brainstorming.
- Higher isn’t free: it costs more time and tokens, and on some tasks
it can lead to overthinking rather than a better answer. Check your
session’s
/statusfor the exact model and effort level in use, since both change over time.
At higher effort levels, a model already does more internal reasoning before answering, so you often need less explicit introspection prompting than you would at a lower setting.
Plan before you act
As tasks grow more complex, asking the agent to write code immediately leads to more rewrite time. Claude Code has an actual enforced planning mode for this, not just a prompting convention: use it rather than relying on the agent to honor a polite request.
Use plan mode, not just a polite request
Start a session in plan mode from the command line:
You can also switch into plan mode mid-session; check
/help or your installed version’s documentation for the
current way to do that, since the exact command has changed across
releases. While in plan mode, the agent can read files and reason, but
is blocked from writing files or running mutating commands, no matter
what it decides to do. That is a real difference from a prompt like “do
not write any files yet”: a prompt is a request the agent usually
follows but is not required to, while plan mode is enforced by the tool
itself.
Review the plan, push back on steps you disagree with, and ask for alternatives. When you are satisfied, exit plan mode and let the agent proceed.
Checkpoint prompts
Break large tasks into explicit phases so you review the output at each stage before moving forward. This is useful whether or not you are in plan mode, since it scopes what the agent does even after you have approved the overall plan:
Step 1 only: read the three CSV files and tell me what inconsistencies you find. Do not write any code yet.This is especially valuable in research because it catches misunderstandings about your data before they propagate into broken code.
The plan file
For complex projects, ask the agent to write a PLAN.md
first:
Write a PLAN.md outlining the steps to clean and merge these files. I will review and edit it before you write any code.This makes the plan a reviewable, editable artefact, a more formal version of the Bootstrap Workflow. Once approved, refer back to it in follow-up prompts: “Proceed with step 2 from PLAN.md.”
Plan files vs. the Living Spec
A PLAN.md and your CLAUDE.md serve
different purposes. The spec defines persistent rules and constraints
that apply across all sessions. The plan describes the steps for a
specific task. Keep them separate: plans are temporary, specs are
durable.
Challenge: Plan before you clean
Practise using plan mode before moving on to the data cleaning episode. Start (or switch into) plan mode, then inside your Claude Code session, type:
Read the three site files in data/. They have inconsistent column names and date formats. Outline a step-by-step plan for cleaning and merging them into a single dataset.You do not need to add “do not write any files yet” here — plan mode already guarantees that. Review the plan. Does it include an audit step? Does it address missing values? Revise the plan in the conversation until you are satisfied, then exit plan mode and save it by asking: “Write this plan to PLAN.md.”
- An audit step, inspect files before changing them
- A schema harmonisation step, standardise column names
- A date standardisation step
- A missing value strategy
- An output verification step
If the agent skipped any of these, ask it to revise before you proceed. The goal is to catch gaps in the plan, not in the code.
AI failures
AI agents are designed to be helpful, which can lead them to take shortcuts.
Common failure modes
-
Determinism collapse: Small variations in prompts
or model updates can lead to different outputs for the same task, which
affects reproducibility.
- Fix: There isn’t a setting that makes an agentic run reproducible — tool calls, retries, and file-system state all vary run to run regardless of sampling settings. Instead, log the exact model, prompt, and relevant outputs (a provenance header, see the next episode) so a different run can be compared, not guaranteed identical.
-
Over-correction loops: If an agent runs its own
tests, it might fix the test to match its buggy code.
- Fix: Write your own requirements and key tests.
- Synthetic data substitution: The AI may generate fake data if it cannot find the real file.
-
Silent failure: The AI uses
try/exceptblocks that hide errors.
How to catch failures
Have you seen an AI make a confident mistake? In your research, what signs indicate the AI is hallucinating?
Common strategies:
- Always ask: “Show me the first 10 rows of the data you loaded.”
- Demand proof: “How did you calculate that p-value? Show the intermediate steps.”
- Check file sizes: Is the cleaned file 0 bytes?
Challenge: The prompt refinement loop
Practise the CLEAR framework on a file you already have: one raw site
file, data/site_A.csv. (You have not cleaned the data yet,
so use the raw column names.)
-
Start with a vague prompt, type this inside your Claude Code session:
Plot my data.Observe: Does it work? Which file did it use? Is the plot useful? Where did it save it?
Refine the prompt: Write a new prompt that applies context (what the data is), specificity (which file and columns, scatterplot with a trendline), and output instructions (where to save it).
Using data/site_A.csv, create a Python script that plots WaterQualityScore over Collection_Date as a scatterplot with a linear trendline. Label the axes. Save the plot to fig/site_A_trend.png (create the directory if it does not exist).Challenge: The introspection loop
Test the AI as a verifier principle. Ask the AI to find flaws in its code before you run it.
-
Generate a script, type this prompt inside your Claude Code session:
Write a Python script that reads 'data.csv' and calculates the rolling 7-day average of a 'score' column. Handle missing values. -
Force introspection: Once the code is generated, do not run it. Follow up in the same session:
Review the rolling average script you just wrote. Are there any edge cases (like having fewer than 7 days of data) where this would fail? If so, provide an updated version. Compare: Did the AI find a mistake in its first draft? Did it add a guard clause like
min_periods=1?
A second, critique-focused pass often catches issues the first draft missed. Treat it as a prompt to look harder yourself, not as a guarantee the code is now correct.
Feedback checkpoint: paste one line you don’t understand
In the shared Etherpad, paste one line of AI-generated code from this episode that you cannot fully explain yet. We will pick a few and work through them together. There is no penalty for not understanding a line; the penalty is shipping it without knowing what it does. Working alone? Ask the agent to explain that line back to you, then check its explanation line by line against what you already know before accepting it.
- Be specific and provide context.
- Plan before you act: use plan mode so the agent can’t write files until you approve its approach, not just a prompt asking it to wait.
- Prefer prompts that preserve learning: ask for plans, hints, and the simplest version, not the finished answer.
- Always validate AI outputs, and never ship a line you cannot explain.
- Introspection can surface issues the first draft missed, but it is not a guarantee of correctness; treat what it finds as something to verify, not proof.
Content from Data Cleaning with AI
Last updated on 2026-08-31 | Edit this page
Estimated time: 98 minutes
Overview
Questions
- Can I trust AI to standardise inconsistent files?
- How do I check that a cleaning script did what I needed, not only what the AI assumed?
Objectives
- Predict what a cleaning script must handle before you prompt for it.
- Build a data processing pipeline for inconsistent files using a spec.
- Explain and validate AI-generated code before you trust its output.
- Document the cleaning process and record its provenance.
Prerequisites
Ensure you are signed in to Claude Code and have a session running in your project folder. Generating scripts can take 10-30 seconds.
This episode uses live coding. Learners should follow along by running commands on their own machines.
Cleaning messy data
Cleaning and merging inconsistent files is a common bottleneck in research. We will use Claude Code to clean and merge the coastal water quality files into one analysis-ready dataset.
The project data
Your project folder (coastal-water-quality) contains
three files in data/, one per monitoring site, collected
weekly from January to May 2023: site_A.csv,
site_B.csv, and site_C.csv (20 samples each,
60 in total). They are messy in exactly the ways real multi-site data
is: the site, date, and score columns are named differently in each
file, the date formats differ, and a few values are missing. The
folder’s README.md describes the target schema and what
“done” looks like. Our goal is one clean
data/master_dataset.csv, then (in the capstone) a trend
plot of score over time by site.
Optional: generate your own messy data
To practise on a fresh problem later, you can have the AI synthesise a similar dataset:
Create a python script named 'make_messy_data.py' that generates 3 CSV files with inconsistent column names, varied date formats, and some missing values, mimicking a multi-site study.For the lesson, use the provided files so everyone is working from the same data.
Cleaning with AI, one checkpoint at a time
It is tempting to type “clean and merge these files” and run whatever comes back. Resist that. We will work the task as a sequence of checkpoints so that you can explain and validate the result, not only produce one. This is the pattern you will reuse for the rest of the lesson: predict, prompt for a plan, inspect, modify, validate, reflect.
A. Predict before you prompt
Open the three files in data/ and look at them yourself
first. Without using the AI, write down two inconsistencies you
expect any cleaning script will have to handle. Real examples
in this data:
- The site column is
SiteIDin site A,idin site B, andStationIDin site C. - The score column is
WaterQualityScore,score, andQuality_Index. - The dates differ:
2023/01/05(A),Jan 5 2023(B), and05-01-2023(C). Watch site C:05-01-2023is day-month-year, so it means 5 January, not 1 May.
Keep your list. It is your yardstick for judging what the AI proposes. If you want a faster overview, you can have the AI write an inspection script, but read its output against your own list:
Write a Python script called 'inspect_data.py' that reads every CSV in data/. For each file, print the filename, the column names, and the number of missing values per column. Do not change any files.Quick check: site C’s date format
Site C stores dates as 05-01-2023, day-month-year. Under
that format, what date does 05-01-2023 represent, and what
would a parser that assumes month-day-year (the common US default)
silently produce instead?
- 5 January; a month-day-year parser would also read it as 5 January.
- 5 January; a month-day-year parser would silently read it as 1 May.
- 1 May; a month-day-year parser would silently read it as 5 January.
- There is not enough information to tell.
2. 05-01-2023 in day-month-year is day
05, month 01, 5 January 2023. A parser that assumes month-day-year
instead reads the same string as month 05, day 01, 1 May 2023, silently,
with no error. This is exactly the trap this episode’s cleaning script
has to avoid: the string never changes, only which format you tell the
parser to assume.
B. Ask the AI for a plan only
Do not ask for code yet. Inside your Claude Code session:
Read 'CLAUDE.md' and the three CSVs in data/. Before writing any code, give me a numbered plan for cleaning and merging them into data/master_dataset.csv. Do not write any files yet.Compare the plan with the inconsistencies you wrote down in step A. Did it catch the site C date format? Did it say how it will handle missing values? Did it propose anything you did not expect?
C. Approve or revise the plan
If the plan is missing a constraint you care about, add it to
CLAUDE.md rather than only mentioning it in chat. The spec
is what travels across sessions. For example:
D. Generate the code
Now ask for the script, pointing it at the spec:
Read 'CLAUDE.md' and the three CSVs in data/. Write a script called 'clean_and_merge.py' that follows the plan and the spec rules. Save the result to data/master_dataset.csv. Add comments linking code steps to spec rules.Your script will not match the lesson’s, and that is fine
AI output varies, so your clean_and_merge.py will look
different from your neighbour’s and from any example. That is the point:
the goal is not to match a reference, it is to read what you got and
decide whether it is correct. Judge it against your spec and the checks
below, not against someone else’s code.
If a learner’s AI fails to generate working code on the coastal data,
provide the pre-written versions from instructors/files/: -
backup_inspect_data.py -
backup_clean_and_merge.py
E. Explain before you run
Hands off the keyboard. Open clean_and_merge.py and read
it. In pairs, each person explains one section of the script out
loud to the other: what it does and which spec rule it serves.
If a section uses something you have not seen, that is exactly the line
to ask about. Working alone? Explain each section out loud to yourself,
or write the explanation as a comment above it, then ask the agent to
check your explanation against what the code actually does.
You are responsible for the final output. You cannot validate what you cannot explain.
F. Validate against checks, not vibes
Run python clean_and_merge.py. A correct run prints
something like Wrote data/master_dataset.csv with 60 rows.
Then confirm the result with concrete checks. “It ran” is not one of
them.
- Row-count check: 60 rows (3 sites x 20 samples), nothing dropped.
- Missing-value check: missing scores filled per your strategy; nothing filled that should have stayed blank.
-
Date check: don’t just confirm the dates parse
without error and fall in 2023 — a wrong parse can still produce
valid-looking 2023 dates. Compare a few site C sample dates against the
raw file by hand (
05-01-2023must become2023-01-05, not2023-05-01), or better, re-parse site C’s raw dates with the explicit%d-%m-%Yformat and compare the result to your merged file row by row.
You can ask the AI to write these checks, but read them before you
trust them. The project folder already ships a
validate_data.py with some checks written and others left
as TODO; you will finish it in the next episode.
G. Reflect
- What did the AI handle well?
- What did you have to already know in order to judge its answer?
- Did anything in the script change the data in a way the plan did not mention?
Feedback checkpoint: surprises and uncertainty
In the shared Etherpad, post two short lines: one thing the AI did that surprised you, and one thing it made sound certain that you are still unsure about. Bring these back into the room. Working alone? Write both lines in your notes file, and follow up on the uncertain one before moving on.
This is the most technically demanding episode, and generated
cleaning code is where extraneous load shows up most. Watch for scripts
that reach for apply with lambdas, regex date parsing, or
multiple helper files when a short linear script would do. If a learner
cannot explain a block in step E, have them ask the agent for a simpler
version before continuing. The “hands off keyboard” read in step E is a
feedback checkpoint: it is where you find out who is lost.
This challenge requires modifying existing code. If learners are
stuck, suggest they ask the AI to read clean_and_merge.py
before asking for modifications.
Challenge: Update the script
Imagine your analysis should cover February to May only, so you need to exclude the January samples.
- Predict first. Before prompting, write down how many rows you expect this to remove and the row count you expect afterward. (Samples are weekly; January has four sampling dates per site.)
-
Then update. Use Claude Code to write a
new script,
clean_feb_onward.py, that reusesclean_and_merge.py’s cleaning logic but filters to February 2023 onward. Do not modifyclean_and_merge.pyitself: the capstone and later episodes reuse it and expect the full 60-row dataset, so it needs to stay untouched. - Verify against your prediction. Run the new script and compare the actual row count to what you predicted. If they differ, explain why in one sentence. Pay special attention to site C: a naive parse can silently swap day and month for some rows and not others, so some misparsed January rows may land outside the window and vanish while others land inside it and wrongly survive.
January has four weekly sampling dates per site (5, 12, 19, 26), so a
correct filter removes 12 rows, leaving 48. If you get a different
number, the most likely cause is the site C date format: a naive parse
can silently swap day and month for some rows and not others.
For example, 05-01-2023 can silently become May 1 and
12-01-2023 can become December 1, while
19-01-2023 happens to parse correctly regardless, since 19
can’t be a month. So a partial misparse doesn’t move cleanly to one edge
of the range, it scatters. This is a small, safe version of a silent
error that would quietly bias a real trend analysis.
Read 'clean_and_merge.py'. Write a new script, 'clean_feb_onward.py', that reuses its cleaning logic but keeps only samples collected in February 2023 or later. Do not modify clean_and_merge.py.Did the AI edit the relevant part or rewrite the whole file? Did it parse the dates correctly before filtering? Did you check the changes before running?
Automating documentation
For the final step, have the AI generate a short pipeline summary.
This folder already ships a README.md describing the schema
and target output, so ask for a differently-named file rather than
letting the agent overwrite it:
Create a PIPELINE.md file that explains the data processing pipeline we just built. List the original files, the cleaning steps performed, and the final output format. Do not modify or overwrite README.md.Challenge: Provenance tracking
To ensure research is reproducible, track which model generated your code and when.
- Use Claude Code to add a provenance header to
clean_and_merge.py. - The header should be a Python docstring containing:
- The model used (check your session’s
/statusor status line for the exact name) - The date
- A summary of the prompt.
- The model used (check your session’s
Read 'clean_and_merge.py'. Add a docstring at the very top of the file as a provenance header. Include the exact model name from this session (check /status if unsure), today's date, and a summary of the prompt: 'Standardise site IDs, format dates, and impute missing scores with site medians.'- Predict the inconsistencies yourself before prompting, so you can judge the AI’s plan.
- Ask for a plan first, put constraints in the spec, then generate code.
- Explain the script before you run it; you cannot validate what you cannot explain.
- Validate with concrete checks (row count, missing values, date format), not because it ran.
Content from Validation Strategies: The Approval Gate
Last updated on 2026-08-31 | Edit this page
Estimated time: 100 minutes
Overview
Questions
- What evidence makes AI-generated code safe to approve?
- What can rewrite time tell me about my workflow, and what can it not?
- How can I use one AI to catch the errors of another without treating it as an authority?
Objectives
- Given an AI-generated script, state the evidence supporting it, name the missing evidence, and decide to approve, revise, or reject.
- Describe the four-layer validation stack (immutable requirements, automated checks, metamorphic tests, domain plausibility).
- Turn validation into checks you can run, by finishing
validate_data.py. - Use a multi-model critique, in a fresh session, to widen review without treating it as an authority.
The approval gate: verification over generation
In an agentic research workflow, you read, question, and approve code as much as you write it. The standard has shifted from vibe coding to a disciplined, validated workflow.
The approval gate is the point where you decide AI-generated code is robust enough for research production. It separates a working prototype from validated science.
The review-first standard
The bottleneck in research is no longer writing code; it is verifying it. A high-performance workflow follows this cycle: Plan → Agent Implementation → Automated AI-Powered Testing → Human Review.
Rewrite time: a signal, not a score
Rewrite time is the manual effort, in minutes, you spend making AI-generated output ready to trust. It is useful, but be careful what you claim from it.
Rewrite time does not prove AI makes researchers faster in general. Measuring programmer productivity is genuinely hard, and a single timing on a single task tells you almost nothing about productivity overall. This caution isn’t hypothetical: a 2026 survey of 868 scientists who program found that feeling more productive with a genAI tool was associated with less programming experience and less use of practices like testing and code review, not with better outcomes. The single strongest predictor of feeling productive was simply how many lines of generated code someone accepted at once (O’Brien et al., 2026) — the authors’ own conclusion is that scientists may be “gauging productivity by code generation rather than validation.” That gap between feeling productive and being correct is exactly what this validation stack exists to close. Treat rewrite time as a formative signal about this workflow, on this task:
- It is local and contextual.
- High rewrite time usually means the task was underspecified, the model overreached, or you did not yet have the mental model to evaluate the output.
- It helps you decide what to improve next: the prompt, the spec, or the validation step.
Don’t turn rewrite time into a productivity claim
A low rewrite time on one script is not evidence that AI saved you time overall. It does not count the time spent prompting, reviewing, debugging, or validating, and it says nothing about whether the result is scientifically correct. Use it to tune your workflow, not to justify it.
Think aloud while you work
Work in pairs. One person uses the AI for five minutes on a small, constrained task (for example, adding a single validation check to the cleaning script). Say out loud what you are doing as you do it: what you expect, what you trust, where you hesitate.
The observer takes notes: Where did the worker hesitate? Where did they trust the output without checking? Where did they backtrack or get confused?
Afterward, discuss as a group: what one change to the prompt, spec, or validation step would have helped most? That is what rewrite time is really pointing at.
Working alone? Keep a two-column log instead: what you expected before each step, and what you actually had to change afterward. The gap between the two columns is what the observer would have noticed.
The four-layer validation stack
To minimise rewrite time and ensure research rigor, use a structured validation stack.
Layer 1: Immutable Requirements (No-Go Zones)
Before the AI writes code, define immutable requirements in your
CLAUDE.md. These are rules the AI is not allowed to
break.
Example (from our project): “Do not change the column names
in data/site_*.csv” and “Do not drop any rows: the merge
must have 60.”
Layer 2: Automated checks you can run
Validation should be executable, not just a feeling. Your project
ships validate_data.py with two checks written and three
left as TODO. Ask the agent to help you implement the rest, then read
and run them, so that “valid” means “these checks passed,” not “it
looked fine.”
Layer 3: Metamorphic and invariant checks
Test the relationships in your data that should never change. -
Invariants: the merged file must have 60 rows (3 sites
x 20 samples), and no sample_id should be lost. -
Metamorphic checks: if you shuffle the order of the
input rows, the merged mean score should not change.
Layer 4: Domain plausibility
This is where your research expertise is irreplaceable. A check can pass while the science is wrong. In our data, a water quality score above 100 is impossible, and site C’s January samples must not appear in May. The clearest case: if the date-format trap goes uncaught, the cleaning runs, the row count is right, and the trend you plot is still wrong.
Validation is yours to own
The stack only works if you stay in charge of it. Four things to keep in front of you:
- You cannot validate what you cannot explain. If you cannot say what the code does, you are not validating it, you are hoping.
- Passing tests is not the same as being scientifically correct. Tests check what you thought to check. They do not check the assumption you missed.
- A second AI model is a reviewer, not an authority. It has its own blind spots. Use it to widen your view, not to settle the question.
- Domain plausibility is where your expertise matters most. No model can replace knowing what a sensible result looks like in your field.
Challenge: build the validator
Open validate_data.py. Two checks are written for you;
three are left as TODO (every sample’s date matches the raw source,
every sample_id is present exactly once, scores within
0-100).
Implement the three TODO checks. Write them yourself, or ask the agent and then read every line before you trust them. For the date check specifically: don’t settle for “parses without error and falls in 2023” — a misparsed site C date can still produce a valid-looking 2023 date. Compare against the raw file instead (see the hint already in
validate_data.py).-
Run
python validate_data.pyagainst yourdata/master_dataset.csv. Make the checks pass by fixing the data or the cleaning script, never by editing a value to satisfy a check. A passing run looks like this (captured from the reference implementation,instructors/files/backup_validate_data.py):[PASS] row count is 60 [PASS] all canonical columns present [PASS] every sample's date matches the raw source [PASS] every original sample_id is present exactly once [PASS] score values fall within 0-100 All 5 checks passed. -
Now break it on purpose: ask the agent to re-clean site C’s dates using
pd.to_datetime(..., format="mixed")instead of the explicit%d-%m-%Yformat. Run the validator again. A good date check should now fail, for example:[FAIL] every sample's date matches the raw source - 8 mismatch(es), e.g. ['SC001', 'SC002', 'SC006']Notice this doesn’t crash, and it doesn’t obviously bunch the whole site C line at one edge of a plot either — only some rows are wrong, because
format="mixed"guesses each date individually and gets some of them right by accident. If instead everything still passes, your date check is too weak (see the solution).
If your validator still passes on the misparsed data, the date check
is too weak. A good check reconstructs the expected date for each sample
from the raw file (parsed with the explicit format) and compares it
exactly — not merely that pd.to_datetime did not raise an
error, or that the year is 2023. See
instructors/files/backup_validate_data.py for a complete
reference implementation. A validator that cannot fail on a known-bad
input is not protecting you.
Quick check: what does a passing validator actually tell you?
Your validate_data.py reports all five checks passed:
row count, date accuracy, ID uniqueness, score range, and no lost rows.
Which of the following can you now claim?
- The data is correct, no further review is needed.
- The five specific properties you wrote checks for hold on this run; anything you didn’t write a check for is still unverified.
- The AI’s cleaning logic is correct in general, since it produced data that passes validation.
- The result would validate the same way on a re-run with a different model.
2. A passing validator tells you exactly what it checked, nothing more. Option 1 overclaims: domain plausibility (Layer 4) and anything outside your five checks are still unverified. Option 3 mistakes passing output for correct logic, the same misparse bug could resurface on a different input the checks don’t happen to cover. Option 4 assumes determinism the lesson has already ruled out, a different run or model could produce different intermediate code that still happens to pass, or fail differently. A validator earns trust by what it would catch, not by passing once.
Multi-model critique
A second model can widen your review, but switching models inside the same Claude Code session is not independent verification: it shares your conversation history, tools, and context with the first model. Treat what it returns as a hypothesis to check, not a second opinion from a clean slate.
Challenge: get a multi-model critique
Use Model A (Claude Code) to generate a data cleaning script.
-
Provide the code to Model B, a different model, in a fresh session if you can (a new terminal, a different tool such as Codex CLI, or at minimum
/clearfirst) — not just a/modelswitch inside the same conversation, which still carries the first model’s framing. Give it the specification, the code, and any tests, not the first model’s narrative about its own work. Then give it this prompt:“Read this script. Act as a skeptical senior data scientist. Identify three potential edge cases where this script will fail, such as empty strings, NaN values, or encoding issues. Suggest specific assert statements to catch these.”
Reflect: Did the second model find something the first missed? Which of its findings still need a deterministic check or your own domain judgment before you’d act on them?
Models have different blind spots, so a second read can surface edge
cases and review questions the first one glossed over. But it is a
hypothesis generator, not an authority: it shares training data and
general tendencies with the first model, and a same-session
/model switch shares even more (conversation history,
tools, framing). Treat its findings as more questions to investigate
with a deterministic check or your own expertise, not as proof.
Warn learners about approval fatigue, the tendency to accept AI suggestions without reading them. The four-layer stack is designed to make the AI prove it is correct before you review the code.
Challenge: explain the approval gate
Take a script you (or a partner) generated earlier and walk through the approval gate out loud or in writing. Answer all five:
- What did the AI produce?
- What evidence do you have that it works?
- What evidence is still missing?
- What domain assumption could still be wrong?
- Would you approve, revise, or reject this output, and why?
A strong answer names the output precisely, points to specific checks as evidence (row counts, invariant checks, a test that passed), and is honest about gaps (“I have not checked the date parsing on the 2019 files”). The domain-assumption answer is the hardest and the most important: it is where you show you are judging the science, not only the code. “Approve” is only justified when the evidence covers the claim; otherwise the honest answer is “revise.”
- The approval gate separates experimental prototypes from validated research.
- Rewrite time is a local, formative signal about your workflow, not a productivity score.
- Immutable requirements prevent the AI from drifting away from research specs.
- A multi-model critique, run in a fresh session, is a reviewer, not an authority.
- You cannot validate what you cannot explain.
Content from Limitations and Cautions
Last updated on 2026-08-31 | Edit this page
Estimated time: 54 minutes
Overview
Questions
- When should I not use AI?
- What are common failure modes?
Objectives
- Identify a hallucinated package name using the official package registry.
- Choose between a proprietary and an open-weight model given a task’s reproducibility needs and data sensitivity.
The jagged frontier
AI capability is inconsistent. A model may solve a complex differential equation but fail a simple logic puzzle. Researchers must identify where AI is reliable and where it is a liability for their specific field.
When not to trust AI code
Using AI-generated code can introduce risks to research integrity. Security-critical tasks, like authentication, encryption, or handling sensitive data, require expert oversight.
AI may also fail when research involves new statistical methods or domain-specific details. Models synthesise information from training data, which might not include the latest breakthroughs or specific sensor patterns. In performance-critical code, AI often prioritises common algorithms over the most efficient ones, which can cause bottlenecks in large-scale processing.
When not to use AI in a workshop exercise
The goal of this lesson is not “never use AI.” It is to use AI where it supports learning and rigor, and to step back where it does not. Avoid AI when:
- The exercise is designed to build basic syntax fluency. Generating the answer skips the practice that builds the skill.
- You cannot yet explain the output. If you cannot judge it, you cannot use it responsibly.
- The data are sensitive and the tool’s privacy terms are unclear.
- The task involves security, authentication, encryption, access control, or regulated data.
- The model keeps introducing concepts beyond the scope of the lesson.
- Reaching for AI prevents the instructor from seeing where the group is actually stuck.
When AI can support learning
AI is genuinely useful when it helps you understand, not when it replaces understanding. Good uses include:
- Explaining an error message you are stuck on.
- Generating a simpler example of a concept you just met.
- Asking you concept-check questions before you answer.
- Comparing two possible solutions so you can choose.
- Suggesting tests for code you already understand.
- Helping you document code after you understand what it does.
Common failure modes
Understanding AI failure modes helps you identify errors before they affect results.
Spec Drift
Spec Drift occurs when the code and the CLAUDE.md
(Living Spec) become unaligned. The agent may fix a bug in the code but
forget to update the spec, leading to future hallucinations. -
Prevention: Regularly ask the agent to “Sync the spec with the
current code.”
Bootstrap Failures
In the “Bootstrap Workflow,” the AI may miss nuances in raw data
during the initial scan. If you approve a flawed spec, the error will
propagate through the entire project. - Prevention: Thoroughly
audit the agent’s first draft of CLAUDE.md.
Silent semantic drift
Semantic drift occurs when an agent makes a change that alters data assumptions or logic without breaking the code. - Example: The code runs and tests pass, but a filtering threshold was changed or a column was renamed incorrectly, affecting the research conclusion. - Prevention: Use metamorphic testing and invariant checks to ensure core logic remains unchanged.
Other failure modes
- Hallucinated functions: The model uses libraries or APIs that do not exist.
- Outdated approaches: The AI uses deprecated syntax from its training data.
- Confident incorrectness: The AI presents wrong formulas or logic as certain.
- Tool poisoning via MCP: When an agent calls external tools through MCP, a misconfigured or malicious MCP server can inject instructions into the agent’s context (prompt injection) through its tool descriptions or returned data, even from a server that is otherwise sandboxed. This can cause the agent to take unintended actions or leak data. Mitigation: only install MCP servers from trusted, audited sources; treat everything a server returns as untrusted input, not just the server itself; and sandbox with minimal filesystem/network access where you can. Registry listing (MCP now has an official one) is not a safety certification.
- Over-engineering: The model generates complex code for simple problems.
Environmental cost
Data centers consume large amounts of electricity and water. Frequent, iterative prompting can be resource-intensive.
- Energy use: Every AI query requires complex calculations, and running them costs real electricity and water. Published estimates of exactly how a single query compares to a web search vary widely by methodology and model size, and the comparison is genuinely disputed, but the underlying point holds regardless of the exact multiplier: iterative, exploratory prompting has a real resource cost that a single web search does not.
- Code efficiency: AI models often prioritise working code over efficient code. Inefficient software uses more energy and resources over time.
Sustainable practices
To code responsibly:
- Think before prompting: Use the CLEAR framework to get the right answer in fewer attempts.
- Request optimisation: Prompt the AI to optimise for memory or speed once the logic is correct.
- Use documentation: If you need simple syntax, check the documentation instead of querying an LLM.
Current models tend to flag uncertainty more often than older ones, but they still hallucinate. Do not promise learners it won’t happen. * If it refuses: Acknowledge that the model correctly identified its own limitations. * Backup: Have a screenshot of a known hallucination ready to show if the AI performs perfectly during the session.
Challenge: Test for hallucinations, then verify independently
Inside your Claude Code session, type:
How do I use the 'pypanda-researcher' library to automatically write my conclusion?Note whether the model admits it does not know, hedges with
uncertainty, or confidently invents instructions. Then check for
yourself, independent of what the model told you: search PyPI (or your
language’s package index) for pypanda-researcher. Record
what you find and the date you checked. Would you accept, revise, or
reject the model’s answer based on that evidence, not on how confident
it sounded?
Current models are somewhat better at flagging uncertainty than earlier generations — you may get a clean “this doesn’t exist” response, and that’s the correct behaviour when it happens. But don’t count on it: a 2026 study testing five frontier models found they still hallucinate nonexistent package names 4.6-6.1% of the time, and more surprisingly, 127 of those hallucinated package names were invented identically by all five models (Churilov, 2026). That second finding matters for your validation habits specifically: asking a different model to double-check a package name is weaker protection than it sounds, because current models increasingly confabulate the same wrong answers, not different ones. The lesson here is not that hallucination always happens, but that you cannot assume it won’t, and cross-checking with another model is not a substitute for checking the official package registry or documentation directly.
Open science and proprietary AI
Claude Code is not open source, which creates a tension in open research.
- Proprietary models (Gemini, GPT, Claude): These are closed-weight models. You cannot verify their training data, and they may update silently. Institutional agreements provide data privacy but do not solve reproducibility issues.
- Open-weight models (e.g. Qwen3, Gemma, OpenAI’s gpt-oss): These can be run locally using tools like Ollama. They offer better reproducibility because you can pin a specific, frozen model revision — though license terms still vary by model, so check them alongside the version. “Open-weight” is the precise term here: the weights are downloadable, but that alone doesn’t make the system open source under the fuller definition (training data, code, and license may still be closed or restricted).
Recommendation: There is no blanket right answer between proprietary and open-weight for prototyping and cleaning. Base the choice on your data’s classification and authorization, the task’s reproducibility and auditability needs, and cost, not on a default. Whichever you use, archive the generated code rather than relying on the model to regenerate the same result later.
Quick check: which model class fits the scenario?
For each scenario, would a proprietary model (Claude, GPT, Gemini) or an open-weight model (Qwen3, Gemma, gpt-oss, run locally) fit better, and why?
- Cleaning a throwaway exploratory script you will discard by the end of the day.
- A pipeline that must reproduce the same output in five years for a methods reviewer.
- A first pass on de-identified patient records at an institution with an approved, contracted AI tool.
- Either. Cost and convenience win here; nothing about the task needs reproducibility or data controls.
- Open-weight. You can pin the exact model revision and re-run it unchanged; a proprietary model can update or be deprecated out from under you, and an institutional agreement does not fix that.
- Whichever the institution’s approved tool actually is, not “proprietary” by default. An institutional agreement can cover data privacy and compliance, but it does not by itself solve reproducibility, that is a separate question from #2, and the two can both apply to the same project.
Key lesson
AI can generate code, but it does not take on your expertise or your responsibility. Your work shifts towards understanding, questioning, and verifying the code the AI produces. The accountability for the result stays with you.
Feedback checkpoint: certainty vs evidence
In the shared Etherpad, post one thing an AI tool told you this session that it made sound certain, but that you have not actually verified. These are the items most worth a second look. Working alone? Write it in your notes file, then actually go verify it before you move on, that follow-through is the point of the exercise.
- Avoid AI for security-critical tasks, sensitive data, and basic syntax practice.
- Know when AI supports learning and when it gets in the way.
- You are responsible for the final output.
- Open-weight models offer better reproducibility (a pinned revision); the right choice for a task still depends on data sensitivity, auditability, and cost, not a default.
Content from From AI Output to a Review-Ready Bundle
Last updated on 2026-08-31 | Edit this page
Estimated time: 110 minutes
Overview
Questions
- Can I take one task from prompt to trustworthy, documented result?
- What does “review-ready” actually include, beyond a script that runs?
Objectives
- Run the full workflow on one project, from messy files to a validated, documented result.
- Produce a review-ready bundle: spec, plan, code, validation, a result, and provenance.
- Make and defend an approve / revise / reject decision on your own output.
So far you have practised each move on its own: a spec, a plan, a cleaning script, a validator. The capstone puts them together on the project you have been carrying all along, the coastal water quality data, and ends with an actual finding: what pattern is visible in the water quality score at each site over the monitoring period? (A descriptive line plot can show a visible pattern; it does not by itself establish a statistically significant trend or a difference between sites — that would need a stated method and uncertainty, which is out of scope here.)
Work through the steps below. The goal is not just a plot. It is a bundle you could hand to a collaborator, or your future self, and have them trust.
The workflow, end to end
Run these as checkpoints, the same pattern as the cleaning episode. Do not rush to the plot.
- Confirm the task. In one sentence, write what “done” means here. (Example: a merged 60-row dataset and a per-site trend plot, both validated and documented.)
-
Check the spec. Open your
CLAUDE.md. Does it state the schema, the no-go zones (no dropped rows, parse site C dates explicitly), and how missing values are handled? Fix it before generating anything. - Ask for a plan only. Have the agent outline the steps from raw files to merged dataset to trend plot. Do not let it write code yet. Review the plan against your spec.
-
Generate the cleaning + analysis code. If you
already have
clean_and_merge.py, reuse it. Then ask for a short analysis script that loadsdata/master_dataset.csvand plotsscoreoverdatewith one line per site, saved tofig/score_trend.png. - Explain one block. Pick one function or section and explain, out loud or in a comment, what it does and why. If you cannot, that block is not validated yet.
-
Validate. Run
python validate_data.py(with your finished checks). Confirm 60 rows, each date matches its source record exactly (not just “falls in 2023,” a misparsed date can still land in range), scores in range, no lost IDs. - Run the analysis. Produce the plot. Look at it.
- Judge the result (domain plausibility). Does the trend make sense? Do all three site lines span January to May with a continuous weekly pattern, or does site C look scattered or out of order, the sign the date trap bit you? A plot can render cleanly and still be wrong.
-
Document provenance. Add a header or a short
PROVENANCE.md: model used, date, the prompt summary, and which checks passed. - Decide. Approve, revise, or reject, and write one line saying why.
What can go wrong here (and that is the point)
The most likely failure is silent: if the cleaning script parses site
C’s dates with format="mixed" instead of the explicit
%d-%m-%Y, some rows silently swap day and month
(05-01-2023 becomes May 1) while others accidentally parse
correctly. The script runs, the row count is 60, and the trend plot
still misleads — but the wrong points scatter rather than bunching
neatly at one edge, so eyeballing the plot alone won’t reliably catch
it. This is exactly the kind of error that “it ran” would have hidden,
and exactly what your validator (checking exact per-sample dates, not
just “falls in 2023”) and your eyes on the plot are for.
Capstone challenge: the review-ready bundle
Produce and submit (or share with a partner) a bundle for the coastal project:
-
CLAUDE.md(your spec, with constraints) -
PLAN.md(the approved plan, or a one-paragraph summary) -
clean_and_merge.py(generatesdata/master_dataset.csv) -
validate_data.py(all five checks implemented and passing) -
fig/score_trend.png(the per-site trend) - a provenance note (model, date, prompt summary, checks passed)
- a short approval decision: approve, revise, or reject, and why
Then write three sentences of reflection:
- What did I understand? Name one part of the pipeline you could rebuild without AI.
- What did I verify? Name the evidence, not the feeling.
- What remains uncertain? Name one thing you would check before using this for real.
The script running is the least interesting part. A strong bundle shows judgement: the spec names the date trap before the code is written; the validator would fail on a misparsed file; the provenance note lets someone reproduce the run; and the approval decision is honest about what is and is not yet checked. “Revise” is a perfectly good answer when the evidence does not yet cover the claim.
Budget most of the time for doing, not explaining. Expect the site C
date bug to surface for several learners; treat it as the highlight, not
a snag, it is the lesson’s whole thesis in one concrete failure. If a
group is far behind, have them use
instructors/files/backup_clean_and_merge.py so they still
reach the validate-and-judge steps, which are the point. Collect a few
approval decisions and read them aloud; the honest “revise” answers are
the best teaching.
Feedback checkpoint: revisit your opening note
Look back at the sticky note (or notes-file line, if you worked through Episode 1 alone) from Episode 1, what you wanted from AI and what you feared it would get wrong. Did the workshop change either answer? Post one line in the Etherpad, or add it to your notes file.
- Review-ready means spec, plan, code, validation, a result, provenance, and a decision, not just a script that runs — a descriptive plot is not itself a statistical finding.
- The same checkpoint pattern scales from one script to a whole small project.
- A plot can render cleanly and still be wrong; domain plausibility is your job.
- “Revise” is a valid, honest outcome when the evidence does not yet cover the claim.
Content from Resources and Next Steps
Last updated on 2026-08-31 | Edit this page
Estimated time: 52 minutes
Overview
Questions
- Which tool should I reach for, and when?
- How do I take this back to my own data?
Objectives
- Choose an appropriate AI coding tool for a given task and data sensitivity.
- Draft a workflow plan for one of your own datasets specifying tool, backend, no-go zones, opening prompt, validation check, and provenance record.
- Recognize signs of AI hype using a stated set of criteria (scope, transparency, citations, privacy).
This episode is a short reference and a plan for what you do next. The detail is deliberately light: the skills you practised (spec, plan, validate, judge, document) transfer across every tool below.
A tool-choice table
| Tool | What it is | Reach for it when |
|---|---|---|
| Claude Code (used in this lesson) | Anthropic’s CLI agent | terminal-native work on your real files |
| Codex CLI / Cursor | OpenAI CLI / AI code editor | you prefer a different vendor or an editor-integrated agent |
| Aider + Ollama | open CLI agent + local model runner | sensitive data that must stay on your machine, or reproducibility |
| NotebookLM, Elicit, Consensus | document/literature tools | grounding answers in your own PDFs or the literature, not coding |
Local models (run via Ollama) keep data on your hardware and let you pin a frozen version for reproducibility, at the cost of needing a capable GPU. Many researchers use a hybrid approach: a cloud model for general scripting, a local model for sensitive data.
Quick check: which pairing is a problem?
Which of these tool-and-data pairings should give you pause?
- Claude Code, for cleaning a public, already-published dataset.
- Aider + Ollama, for a pipeline that must reproduce identically in five years.
- Claude Code (default cloud backend, no institutional agreement), for a first pass on sensitive, unpublished human-subjects data.
- NotebookLM, for asking questions grounded in a folder of your own PDFs.
3. Sending sensitive, unpublished human-subjects data to a general cloud endpoint with no institutional agreement or approved backend behind it is exactly the pairing to avoid, that is what “which approved backend” in the workflow card is asking you to check before you start, not after. The other three are reasonable defaults: public data carries no sensitivity constraint (1), a local pinned model fits a reproducibility requirement well (2), and a document-grounding tool used on your own files is what it is built for (4).
MCP and the shadow-IT risk
The Model Context Protocol (MCP) lets agents connect to external tools and data sources (databases, file systems, services). Anthropic donated MCP governance to the Linux Foundation’s Agentic AI Foundation in December 2025, and an official server registry now exists — but registry listing is not a safety certification. MCP servers are still often installed without institutional oversight, and prompt injection and silent data exfiltration remain real risks: a server’s tool descriptions and returned data should be treated as untrusted input, not just the servers themselves. Before connecting one, check it is actively maintained and from a trusted source, sandbox it with minimal filesystem/network access where you can, and confirm your institution’s policy covers it.
Citing and crediting AI
Transparent attribution is part of open science. Major standards (COPE, Nature, Elsevier) agree AI tools cannot be authors, because they cannot take accountability. Cite them as methodological tools instead.
- In a repo
README.md: note the model, its role, and who verified the output (for example, “Claude [current model name] drafted the cleaning script; verified by [you] viavalidate_data.py”). - In a manuscript: name the model in methods or acknowledgements, and keep prompts and outputs available.
- References: COPE on AI authorship, Elsevier AI policy, CRediT taxonomy.
Challenge: your Monday workflow card
Pick one real scenario from your own work, then fill out a short plan
you could actually use next week. A ready-made template,
monday-worksheet.md, ships with the lesson (in
learners/files/); open it and fill it in.
Choose a scenario: a small public CSV, sensitive human-subjects data, large geospatial files, or an existing messy repository.
For it, write down:
- Tool and backend: which tool, and (if your data is sensitive) which approved backend?
-
What the AI can see / must not touch: the no-go
zones for your
CLAUDE.md. - First prompt: your “plan only, no code yet” opener.
- Validation: the one check that would most likely catch a silent error in this task.
- Provenance: where you will record model, date, and prompt.
A good card is specific to your data. The validation line is the tell: if you cannot name a check that would fail when the result is wrong, you do not yet understand the task well enough to delegate it. That is fine, it just means you start by understanding, not generating.
Spotting hype
New tools appear daily, and many are more marketing than substance. Before adopting one:
- Scope: beware tools that claim to do everything; specialised tools usually work better.
- Transparency: can you see the intermediate steps, or just the answer?
- Citations: does it give real, checkable DOIs and URLs?
- Privacy: if it is free, is your data used for training?
Quick check: spot the hype
For each tool description, name the one criterion above it fails most clearly.
- “Answers any research question instantly, across any field, no setup required.”
- “Summarises your uploaded PDFs and gives you a final answer, no need to see the underlying search or reasoning.”
- “Free forever. Just sign up and start uploading your data today.”
- Scope. “Any field, any question” is the specialised-vs-does-everything red flag, a tool that claims no domain limits usually has no domain depth either.
- Transparency. Hiding the intermediate steps means you cannot check its work, only trust its answer.
- Privacy. A free tool that just wants your data uploaded, with no mention of what happens to it, is the exact case the privacy criterion is asking you to check before you use it, not after.
A few sources that stay practical and skeptical: Simon Willison (AI engineering and security), Ethan Mollick (AI and cognitive work), Hamel Husain (systematic evaluation), and The Batch (balanced industry coverage).
For the tool used here, see the Claude Code documentation. For definitions of terms used throughout this lesson (Living Spec, external brain, approval gates, and more), see the learner reference page.
- The workflow transfers across tools; match the tool and backend to the task and data sensitivity.
- Attribute AI use transparently; it cannot be an author.
- Leave with a concrete plan for your own data, including the one check that would catch a silent error.
- Before adopting a new tool, check its scope claims, transparency, citations, and data-privacy terms; new tools appear daily, and many are more marketing than substance.