Replication as Curriculum: What a 2026 Undergraduate Re-run of Three Open-Data Papers Revealed

On a Saturday in spring 2026, a mid-sized US public university turned a teaching lab into a replication fair. The premise was simple: give undergraduate teams one day, three published papers with open data, and a shared worksheet. Ask them to reproduce the main analysis, not to judge the authors. The papers came from psychology, ecology, and materials science, chosen because each had accessible data and a analysis that a well-prepared team could attempt in a few hours.

The event was not designed to produce a verdict on reproducibility. It was designed to teach students how published methods become working analyses. That distinction shaped everything from the paper selection to the language used in the final debrief.

How the day was structured

Each team received the paper, the available data, any public code, and a decision log. The morning was spent mapping the analysis. Teams marked every point where a reasonable analyst could choose differently: exclusion rules, variable coding, transformations, model specification, software version, and the handling of missing values. Faculty facilitators did not provide answers. Their role was to keep the documentation moving and to prevent teams from treating ambiguity as failure.

The afternoon was reserved for running the analysis and comparing results with the published claim. In the final hour, teams classified their outcome as matched, sensitive, or blocked. A match did not mean identical numbers. It meant the same qualitative conclusion with compatible uncertainty. Sensitive meant the result changed in magnitude or precision under a defensible alternative choice. Blocked meant the team could not proceed without information that was not in the paper or the public materials.

What matched, and what did not

The psychology paper reproduced most cleanly. Descriptive statistics and the direction of the main effect matched, and the effect size fell close to the published estimate. The team still had to resolve a reverse-coded item that was described in the text but implemented differently in the public code. That discrepancy did not overturn the result, but it required a judgment call that the published article did not record.

The ecology paper matched at the level of model structure and broad conclusion, but the confidence intervals widened after the team reconciled taxonomic names across two data files. The published analysis had used an older naming convention. The team's decision log recorded the update, the alternative, and the effect on the estimate. The result was not wrong; it was more uncertain than a casual reading of the abstract would suggest.

The materials science paper was the most instructive. The qualitative trend matched, but the reported magnitude shifted under different baseline corrections. The public dataset did not include the instrument calibration metadata that the original authors had presumably used. The team could reproduce the shape of the finding but not the precise number. That outcome was classified as sensitive rather than blocked because the team could still test how much the conclusion depended on the correction.

Where replication broke down

Most breaks were documentation gaps, not errors. One dataset existed in two versions with no clear note about which version supported the published figure. One paper described an exclusion rule in prose that the public code implemented at a different stage of the pipeline. One team spent an hour tracking a software dependency that had changed behavior since the original analysis. None of these discoveries required a gotcha frame. They required a place to write them down.

The decision log became the most valuable artifact of the day. It had five columns: the original language, the team's assumption, a defensible alternative, the consequence for the result, and the evidence used. Teams color-coded entries as green, amber, or red. By the end of the afternoon, the room contained a visual map of where each published result depended on undocumented judgment. That map was more useful to students than a simple pass or fail.

What students learned

Students reported that methods sections became less transparent after they tried to use them. That is not cynicism. It is a realistic reading habit. They also learned to separate 'we could not reproduce this' from 'the original is wrong.' Those are different claims, and the fair required different evidence for each.

Several teams contacted authors during the event. Some received quick clarifications; others did not hear back before the debrief. The facilitators treated both outcomes as normal. Replication is a social process as well as a technical one, and students need practice asking precise questions without implying misconduct.

A reusable organizing checklist

• Select papers with open data and, where possible, public code.
• Pilot each paper before the event so that facilitators know where teams are likely to stall.
• Prepare a standardized computing environment or document software versions in advance.
• Define 'match' before the event, and include uncertainty rather than relying only on significance.
• Give every team a decision log with fields for original text, assumption, alternative, consequence, and evidence.
• Assign facilitators who are not authors of the selected papers.
• Build in time for documentation, not only analysis.
• Brief participants on neutral language: 'ambiguous' and 'underdocumented' are acceptable; 'sloppy' is not.
• Invite authors to respond after the event, with no obligation to participate.
• Publish an anonymized summary so future teams can learn from the same ambiguities.

Limits and the next iteration

A one-day fair is not a formal replication study. Teams were not blinded, the analyses were not preregistered, and the sample of papers was small. The event cannot settle questions about reproducibility in any field. It can, however, teach students how published claims become computable claims, and it can show departments where their own training in methods and documentation is thin.

The next fair will add a pre-event pilot report, a shared software container, and a short author-response window. The organizing team also plans to compare decision logs across departments to see which kinds of ambiguity recur. The point is not to make replication easier. The point is to make the work of replication visible enough to teach.