greenhorn: how we got here

The problem

A reviewer who is new to a product opens a pull request description that is full of project words. They cannot follow it. We already had tools for the sentence: controlled-English rules and Hemingway rules. They fix how a sentence is written. They do not fix which concepts a stranger needs, in what order, or with which example. We wanted to close that gap.

Our first design

Our first idea was a Context block at the top of the pull request description. It held a list of up to 5 terms with one-sentence definitions. Next came a one-line before and after. Then came the brief. The author, or the model that writes for the author, picked the terms. The caps (5 terms, one line) were guesses.

We attacked it before building it

Before we built anything, we attacked our own design. We found seven problems.

Definitions first and the example second contradicted the sources we cited. Those sources say concrete before abstract.
A terms block at the top makes the reader flip between the block and the text.
Authors cannot see their own jargon, so the author cannot be the one who finds the terms.
A near analogy can mislead, so an analogy needs its limit stated.
The caps were guesses.
A neighbouring tool already existed. coldread by patrykwiener fixes text for a reader who was not in the session. greenhorn targets a different gap: the domain concepts that a newcomer lacks. We have not compared the two tools on the same documents.
Most of our citations came from reviews and summaries, not from the papers.

The design changed in these ways:

A term gets its definition inline, at first use.
One concrete example comes before the mechanism.
A separate reader with no context finds the stumbles.
We take the answers from the code and never invent them.
An analogy appears only with its limit.
The output is skippable.
The limits stay as config defaults, not fixed numbers.

Then we read the papers

Problem 7 sent us to the papers themselves. We read 19 in full and 2 by abstract only. The table at the end lists each one.

Some of our ideas held up, with limits.

Rawson et al. (2015) ran three experiments that enrolled 131, 176 and 197 undergraduates. The main analyses kept only students who already knew three or fewer of the ten concepts. Students who studied examples scored higher on a classification test than students who only re-studied definitions (the authors report d = 0.74 to 1.67). For examples that students had not seen in study, d was 0.40, 1.22 and 0.90. The test used short concepts, not documents, and came after a short delay.
Sweller and Cooper (1985) ran four experiments with 20 to 40 Year 8 and Year 9 students learning algebra. Students who studied worked examples spent less time in the practice phase in Experiments 2 to 4. On later problems with the same structure, they were faster and made fewer errors (Experiments 3 to 5). They had no advantage on problems with a different structure. The authors say that the step to extended classroom learning "must be tested directly". The test compared studying solved problems with solving them. It did not compare examples with definitions.
Kalyuga et al. (2003) reviewed studies in which help for beginners lost its benefit, or hurt, more experienced learners. This is why our output is skippable. We know the studies only through the review, and none tests a skippable section.
Gray and Holyoak (2021) reviewed analogy research for classroom STEM teaching. They recommend that instructors "point out the limits of the analogy". That is a review's recommendation, not an experiment on that step. They also say that analogies help most when learners already know the source. So this advice may not carry over to readers with little prior knowledge.
Bacchelli and Bird (2013) studied code review at Microsoft. They found that understanding the change is the key challenge for reviewers. The study tests no fix.
Camerer, Loewenstein and Weber (1989) found that participants who knew a company's earnings showed bias toward that knowledge when they predicted the estimates of people who did not. They did not fully ignore it. Markets cut the bias by about half, and feedback alone did not reduce it. Pinker advises, in an essay and not in a study, that writers show a draft to people like the intended audience. These support why the author cannot be the one who finds the terms. They do not show that our fix works. The 1989 task was forecasting, not writing.
For inline definitions, the nearest test is Yeung, Jin and Sweller (1998, abstract only). They reported that definitions inside a passage helped 5th graders' comprehension but reduced vocabulary learning, and that adults showed the reverse. We found no test of inline definitions against a glossary for technical documents. Inline definitions stay a design choice.
Chandler and Sweller (1992) ran two experiments on material that points to other material. In the second, 20 educational psychology students read an experimental report. In one version the parts that refer to each other sat together. Those students spent less time (the authors call it significant, t(18) = 1.85, which is weaker than the usual two-sided cutoff) and scored higher on test questions. That version had no headings. The authors warn that experts or harder reports may give different results. The material was a report, not definitions or jargon.

Other ideas did not hold up. For each one, we believed something, the paper said something else, and we changed our claim.

Order. We believed that an example before the definition beats a definition first. In the one direct test (Rawson et al.), performance was similar when examples came before or after the definition. So we dropped our claim about order. The tool still opens with an example, and we claim no benefit from that order.
ICER 2023. We believed that Hassan et al. showed that novices learn better from concrete to abstract. The paper is a qualitative study of 42 interviews with undergraduates in an introductory Python course. It measured no learning gain, and its authors call their suggestions conjecture. So we no longer cite it for that.
Program comprehension. We leaned on studies of how people read code. Pennington (1987) studied 80 and 40 professional programmers, so experts. von Mayrhauser and Vans (1995) report mixed and switching strategies, and say that no one model accounts for all behavior. So we dropped the claim that novices read bottom-up and experts read top-down.
Advance organizers. We believed that organizers work best when short and more abstract. We saw that in nothing we read. Luiten et al. (1979, a conference paper) analyzed 135 studies and reported an average effect size of .21 on learning. So we dropped the claim.
The 20% to 40% figure. We wrote that reviewers ask the author 20% to 40% of the time. Bacchelli and Bird write, from interviews, that reviewers often (from 20% to 40% of the times) "even go to talk in person" to the author. That is an interview report from one company. So we now say so.
Plain rewriting. We counted Bullock et al. (2019) as support for plain rewriting. They ran an online experiment with 650 participants. They measured how fluent a text felt, not comprehension. Jargon lowered felt fluency. Hover-over definitions had no significant effect on it (p = .54). They were hover text, not inline text. So we cite it only for that.
Analogies. We wrote that an analogy can teach a wrong model. Gray and Holyoak say that inappropriate prior knowledge can produce misconceptions unless the instructor maps the correspondences and states the limits. So we changed our wording.

What we could not read

We could not open some papers. The publisher page showed a login or returned a 403 error, and we found no open copy. These are Ausubel (1960), Brooks (1983), Masson and Waldron (1994), McNamara et al. (1996) and the journal version of Luiten, Ames and Ackerson (1980). We know McNamara et al. only through the Kalyuga et al. review. We read the 1979 conference version of Luiten et al. instead. We did not attempt Newton (1990). We read Chandler and Sweller (1992), but not their 1991 paper. No claim in this account leans on any of them.

We used Claude to find and read the sources. We checked the key numbers and quotes a second time against the text of the papers. On 2 November 2026, separate Claude agents read each claim again against the paper text. They found errors that our check missed. We rewrote those lines.

Then we tested the tool

We tested the tool, not only the idea. An independent audit tried to break the checker. It found real bugs, and each one got a regression test.

Two --disable flags did not combine.
Prose such as "Before the change it broke. After a while it worked." counted as a before and after example.
An unclosed code fence counted as an example and hid the rest of the text.
The parser slowed down quadratically on long input.

One weak point remains, and we state it openly. The reader agent has the setting tools: [] so that it has no tools. The Claude Code docs say that an omitted tools field inherits every tool, and they say nothing about an empty list. In Claude Code 2.1.316 we saw that tools: [] leaves the agent only its hand-back tool. A later version may change that. So we also list every tool that we know of in disallowedTools, and a test keeps both lines.

Where it stands

We have a hypothesis, not a result. A newcomer follows technical text better when a reader with no context marks where they get lost, when every gap is filled from the real source and not guessed, and when the text opens with one concrete example before the mechanism.

No study tests that combination for technical documents or for software newcomers. The numbers in the defaults (5 terms, threshold 3, 25 words) are untested guesses. The reader in the pipeline is a model, not a human newcomer.

These results would show we are wrong:

Fresh readers explain the greenhorn rewrite no better than they explain the original.
The example-first order confuses more readers than it helps.
The model reader lists stumbles that human newcomers do not hit, and misses stumbles that they do hit.

The human eval tests all three. It compares three versions of each document: the original, the original after polish only, and the full greenhorn rewrite. Fresh readers see one version each. They explain the text in two sentences. Two scorers who do not know the version score each answer from 0 to 2 against the author's intent. Nobody ran it yet, and the folder holds no results.

Read the full protocol. If you run it, send us the results. We will publish them even if the result is a tie.

Sources

Each row is a paper that the account above uses. We mark a paper as read in full only when we read the text of the paper. We mark it as abstract only when we read the abstract and no more.

Source	What we read	Population and task	What it found	Limit
Rawson, Thomas and Jacoby (2015)	Read in full	Three experiments that enrolled 131, 176 and 197 undergraduates at two universities. They learned ten judgment-and-decision-making concepts and then classified examples. The main analyses kept only students who knew three or fewer of the concepts already. By our reading of the degrees of freedom, this is roughly 55 to 90 students per analysis.	Students who studied examples did better than students who studied definitions only. The authors report d = 0.74 to 1.67. For examples not seen in study, d = 0.40, 1.22 and 0.90. The two order groups did not differ significantly (d = 0.07 to 0.36, about 25 students per group).	The control group re-studied definitions. Short declarative concepts, one topic, lab sessions, a short delay. The upper bound of 1.67 is in the abstract, and the largest d in the results text is 1.43. The order test cannot show that the two orders are equal. We saw no clear benefit for students who already knew most of the concepts. During study, the example groups were told which concept each example showed.
Kalyuga, Ayres, Chandler and Sweller (2003)	Read in full	A review of studies of trades trainees, science and database instruction, plus two text-reading studies.	Reports that techniques that help beginners can lose their effect or hurt more experienced learners.	A secondary description of the studies, in a review that mostly cites the authors' own group. Sample sizes not seen. It does not test a skippable section. We read a copy hosted by a third party.
Gray and Holyoak (2021)	Read in full	A narrative review of analogy research for classroom STEM teaching.	Recommends that instructors map the correspondences and "point out the limits of the analogy".	A review, not a new experiment. We saw no effect size for the advice. Nothing about technical documents. The authors say analogies help most when learners already know the source.
Bacchelli and Bird (2013)	Read in full	Code review at Microsoft: 17 observed developers, 570 classified comments, survey answers from 165 managers and 873 programmers.	Found that understanding the change is the key challenge for reviewers. The authors write, from interviews, that reviewers often (from 20% to 40% of the times) "even go to talk in person" to the author.	One company (Microsoft). The surveys covered January to June 2012. The 20% to 40% figure is an interview report. The study tests no fix and does not test PR descriptions. It notes that reviewers who read the change description first took longer to comment.
Camerer, Loewenstein and Weber (1989)	Read in full	Economic experiments. 51 Wharton students made the uninformed estimates. Groups of nine Wharton students traded in markets. 32 University of Chicago MBA students made individual judgments. Informed participants knew the actual 1980 earnings and guessed the mean estimate of the uninformed group.	Informed participants showed bias toward what they knew. They did not fully ignore it. Markets cut the bias by about half. Feedback alone did not reduce it.	A forecasting task, not writing or reading. The individual conditions had small groups (19 and 13). It does not show that a writer's text is harder to read.
Pinker, "The Source of Bad Writing"	Read in full	An essay. No participants.	Argues that writers should show a draft to people like the intended audience.	Opinion and anecdote. No data of his own. He mentions research on overconfidence without citing it.
Yeung, Jin and Sweller (1998)	Abstract only	Five experiments with 5th graders, adults and 8th-grade ESL learners who read passages with explanatory notes.	Reported: definitions inside the passage helped 5th graders' comprehension compared with a separate list but reduced vocabulary learning. Adults showed the reverse.	Abstract only. Sample sizes not seen. School-age and ESL readers, not technical documents. The result flipped with expertise and with the outcome.
Chandler and Sweller (1992)	Read in full	Experiment 1: The researchers allocated 26 first-year apprentices, and 20 (10 per group) took the test. They learned machine programming from text and diagrams. Experiment 2: 20 educational psychology students, about 10 per group, read one simplified experimental report.	Experiment 1: the group that got text and diagrams together scored higher overall. Only 4 of 10 items were individually significant, and the authors call the effects at best moderate. Experiment 2: students who read the integrated report spent less time and scored higher on test questions.	Small groups. Instructions and a report, not definitions or jargon. The integrated report had no headings, and the authors call its time difference significant at t(18) = 1.85, which is weaker than the usual two-sided cutoff. The authors say experts or harder reports may give different results. The copy is a scan with some text errors.
Sweller and Cooper (1985)	Read in full	Experiments 2 to 5: 20, 22, 40 and 24 Year 8 and Year 9 students learning algebra in Sydney high schools. Experiment 1: 62 students (22 in Year 9, 22 in Year 11 and 18 university mathematics teacher-education students), on recall and classification measures of expertise.	Worked-example groups spent less time in the acquisition phase in Experiments 2 to 4. In Experiment 5 the time was equal, and they worked through about three times as many problems. On later problems with the same structure, they were faster and made fewer errors (Experiments 3 to 5). They had no advantage on problems with a different structure. Experiment 2 showed no difference in test time.	Algebra only, with novices, in short lab sessions, in small groups tested with nonparametric tests. It compares studying solved problems with solving them, not examples with definitions. The authors say the wider implications must be tested directly.
Hassan, Cunningham and Zilles (2023, ICER)	Read in full	42 think-aloud interviews with undergraduates in an introductory Python course for non-technical majors.	Observed that beacons and variable-role names were rarely helpful. Tracing helped when students misread a common pattern or syntax, and tracing a single input could distract them. The authors call their suggestions conjecture.	Students, not reviewers. Small code snippets, not documents. No outcome scores. One course, one semester.
Pennington (1987)	Read in full	80 professional programmers in Study 1 and 40 in Study 2, who read and modified programs.	Procedural units dominated the experts' first mental representation, though task goals may shift what dominates later.	Experts only. Small programs. It cannot say how a reader without domain knowledge reads a PR description.
von Mayrhauser and Vans (1995)	Read in full	A survey of program-comprehension models.	Reports mixed and switching strategies. States that no one model accounts for all behavior as programmers understand unfamiliar code.	A secondary survey about code, not prose. Results come from small programs.
Luiten, Ames and Ackerson (1979), conference paper	Read in full	A meta-analysis of 135 advance-organizer studies.	Reported an average effect size of .21 on learning (110 effect sizes), which the authors read as a facilitative effect. They did not test significance. Learning effects were .26 for college and .28 for special-education groups, and .17 for secondary and primary groups. Spoken organizers did better than written ones (.37 and .17).	Mixed studies, written organizers mostly. The studies defined ability levels in different ways. Scanned text, noisy. The journal version may differ.
Bullock, Colon Amill, Shulman and Dixon (2019)	Read in full	An online experiment with 650 participants who read short science texts.	Jargon lowered self-reported processing fluency. Hover-over definitions had no significant effect on fluency (p = .54), but they raised support for the technologies (p < .05).	The outcome is felt fluency, not a comprehension test. Lay readers of science news, short texts, a non-representative online sample. The mediation analysis was cross-sectional, and the definitions were hover text, not inline text.
Fyfe, McNeil, Son and Goldstone (2014)	Read in full	A review of concreteness fading in math and science teaching. It cites experiments with undergraduates (modular arithmetic; a science simulation) and with children (math equivalence).	It recommends starting concrete and fading to abstract. In McNeil and Fyfe (2012), fading gave the best transfer at all three test times. In Goldstone and Son (2005), concrete-to-abstract beat the other three conditions.	The title says systematic review, but we found no search method in the text. The authors write that some direct experimental evidence exists and that more is needed. The review lists the two child studies as unpublished. The authors call the evidence for the full three-step transition "admittedly small". All of it is math and science teaching, none of it prose.
Fyfe and Nathan (2018)	Read in full	A theory paper. No new data.	Describes concreteness fading as underspecified despite increasing empirical and theoretical support.	A theory paper, useful as the authors' own caution.
Fyfe and McNeil, poster	Read in full	63 children aged 7 to 9 who could not solve math-equivalence problems, assigned to one of four conditions (the poster does not say how).	Concrete-to-abstract fading beat the other conditions on transfer problems.	A poster, not the journal article. One session and an immediate test only. One math concept, young children, a main-effect p-value close to .05.

How we got here