Claude Certified Architect Foundations (CCAR-F) Practice Test

With --output-format json and --json-schema, Claude Code in -p mode returns structured output that conforms to a declared schema, such as a list of objects that each have a target path and test code. Downstream scripts can then parse it reliably. The deciding detail is that the tests themselves are fine and the failure is split_tests.py being unable to parse free-form text with inconsistent prose and fences. A schema removes that ambiguity at the source.

Why the others are wrong: An independent review instance is the right tool when the generated content has errors that self-review misses, but the test content is acceptable here and a reviewer does not make the output parseable. Strengthening CLAUDE.md is the right move when tests ignore team conventions, but the tests already follow the conventions and prose instructions do not guarantee a machine-parseable output format. Supplying the existing test file is the fix when generated tests duplicate covered scenarios, but reviewers already report few duplicates and the file is already provided.

Source: CCAR-F exam guide, Task Statement 3.6

The logs show the output is structurally valid, so this is a semantic gap in the enum and not a formatting failure. The model has no correct label for multiple dependent claims, so it picks the nearest allowed value. Adding 'multiple_dependent' covers the known category. An 'other' value with a detail field gives categories nobody has catalogued yet an accurate place to land, instead of being forced into an existing label.

Why the others are wrong: Strict mode guarantees conformance to the schema, but responses already validate with zero errors, so it cannot fix a label the schema does not offer; it would win if outputs were violating the schema. Forcing tool_choice ensures a tool call, but every response already arrives as a tool_use block; it would win if the model sometimes replied in plain text instead of calling the tool. A nullable field suits information that is absent from the source, but these claims are present and classifiable, so returning null would discard real data; it would win if the claim type were genuinely missing from some documents.

Source: CCAR-F exam guide, Task Statement 4.3

The auditor's concern is about errors that appear later, on new layouts, among fields that are auto-approved. Those fields are never seen by reviewers, so no existing check measures them. Reviewing a stratified random sample of auto-approved, high-confidence extractions on live traffic measures ongoing error rates in exactly that population. It also surfaces novel failure modes that a fixed validation set cannot contain.

Why the others are wrong: A per-type, per-field breakdown is the right check before reducing review, and it has already passed, but recomputing it on last month's static set cannot reveal errors on layouts that did not exist when that set was labeled. Raising the threshold helps when borderline fields are slipping through with errors, but it only moves more fields into review and still gives no measurement of the errors hiding among the fields that stay auto-approved. Calibrating against a labeled set is the right step when confidence scores are untrusted, but repeating it on the same fixed data only confirms past calibration and never samples new production layouts.

Source: CCAR-F exam guide, Task Statement 5.5