Method and standards

The source is the submission record of the PrepClubs platform: every practice attempt sat, with the answer given to each item, the order they were given in, the seconds spent on each, and the total time taken. There is no survey instrument anywhere in this work and no self-reported behaviour. What is measured is what the test player recorded.

Attempts are grouped by form and each form is classified by how many seconds it allows per question. Speeded means under thirty seconds an item, which on this platform is the CCAT, Wonderlic, PI Cognitive and Cubiks Logiks formats. Generous clock means thirty-three seconds or more. The classification is made from the form’s time limit and item count, not from any judgement about difficulty.

1. Re-grade against the bank on disk. Every attempt is scored again against the form as it now exists, and discarded if that disagrees with the stored score. Question banks were edited after launch, so an old attempt cannot honestly be scored against a paper it was not sat on. This is by far the largest discard and it is the one we would most like to be smaller.
2. Discard attempts with no answers recorded. A submission with an empty answer map carries no information about anything.
3. Keep attempts that used at least half the clock. Evidence that the paper was actually sat rather than opened and closed. Half is the standard cut for every report.

Candidates who failed to finish are not discarded. Not finishing is the outcome most of this work measures. Filtering on it would be selecting on the dependent variable, which is the error most likely to produce a flattering and worthless result.

Ninety percent of the clock, as a standard cut. It looks like a stricter and therefore better test of seriousness. It is not, because it selects for time pressure. Applied to the comparison in Report 2026-01 it removes from the generous-clock group exactly the people who found the paper easy and handed in early, which is the group the comparison depends on. It survives only as a robustness check, reported alongside the headline and never in place of it.

Excluding incomplete attempts. Proposed for the sensible-sounding reason that an unfinished paper is not a serious one. It would have deleted the finding. An attempt that ran the full clock and left the last ten questions blank is the most informative record in the corpus.

More than half the attempts on record are excluded. The ledger below appears in identical form on every report, computed at the same moment as that report’s figures.

The working rule is that a confound gets a control, not a caveat. A caveat is what we write when a control is impossible, and in that case the claim is weakened to match rather than published at full strength with a disclaimer attached.

• Item difficulty, in Report 2026-02. Slow answers land on hard questions, so a comparison across questions is worthless. The comparison is drawn inside each individual question, splitting its own sittings at its own median time. Difficulty is then held exactly constant because it is the same item.
• Selection, in Report 2026-03. On a speeded form the later items are answered only by candidates fast enough to reach them. The gradient is recomputed on attempts that answered every single item, where no candidate is missing from any point on the curve. Both columns are published.
• Ability, which is not controlled anywhere. None of this work holds the candidate constant. Where a residual gap is consistent with abler candidates simply being faster, the report says so and does not claim the stronger reading.

Withdrawn, 21 September 2026. The claim that time pressure degrades the answers a candidate does give. Accuracy appeared to collapse across the paper on speeded forms and hold on generous-clock forms. Splitting speeded attempts by how much of the clock they actually used killed it: candidates who finished with time to spare declined 27.6 accuracy points against 31.3 for the time-pressured. Nearly the whole effect was present in people under no pressure, so it was never the clock. It was a property of the paper, which became Report 2026-03.

Measured and not published. Repeat attempts show a gain of roughly twelve points. Of the eighty repeat pairs, sixty-one re-sat the same form, so most of that is item memory rather than improvement. It is the one number in this corpus that would flatter the company that collected it, and it stays unpublished until a comparison across different forms of the same test is possible.

Not published at all. The per-question records. Joined to an attempt, timing data at this sample size is re-identifiable. Aggregated figures are published as CSV on every report; the underlying rows are not.

Every figure is written by the same script that computes the numbers behind it, in the same pass, so a chart cannot drift from its table. Each report links a CSV of its figures and a vector copy of each chart.

If a number here looks wrong, or you want a cut that has not been published, tell us which and we will run it. Ask us to check a figure.

These apply to every report in the series and are not repeated as news in each one.

• These are practice attempts. They are sat on PrepClubs forms written to each test’s published format, not on the publishers’ own instruments, and they are not issued, endorsed, scaled or normed by any publisher.
• The stakes are not the same. An attempt at home and an assessment attached to a job offer are different events. Nothing here measures the difference.
• The sample is self-selected. People practising for a test knew it was coming and chose to prepare. That is not a random sample of applicants and cannot be read as one.
• Sample sizes vary a great deal by form. Every table carries its attempt count so a reader can weigh a finding on 41 attempts differently from one on 215.
• No negative marking anywhere in the corpus. Every form measured scores a wrong answer as zero rather than as a deduction. Findings about guessing do not transfer to a test that penalises errors.