Independent audit · October 2026

Our Commitments

The questions you never saw

A practice exam can only ask you about what its source material covers. When that material has a hole, the exam has the same hole — and you are the last person who would ever notice.

Think about how that failure actually reaches you. You do not get a wrong answer. You do not get a badly worded question. You get nothing — a topic on the syllabus simply never comes up. You sit the exam, you score well, you feel ready, and the one area you were never asked about is still waiting for you on the day.

An absence makes no noise. A wrong question gets reported: somebody complains and it gets fixed. A missing topic produces no complaint from anyone, because nobody can report a question that was never asked. It does not show up in a score, a pass rate, or a count of questions per domain — that count comes out right whether the questions are spread across the whole syllabus or bunched onto the parts the source material happens to cover well.

Why this is harder than it sounds

A syllabus is written in the certifying body’s language: broad statements of what a competent practitioner should be able to do. The documentation an exam has to be built from is written in the vendor’s language — product names, settings, limits. The two vocabularies do not have to meet, and often do not.

So you cannot check coverage by searching for it. Search the sources for a syllabus topic and you will get hits whether or not anything behind them could settle a question, because unrelated products share the same words — the same term can mean four different things to one vendor. Counting pages tells you no more. The only way to find out whether a topic can be examined is to try to examine it.

What we do about it

Before we build exams for a certification, we probe our knowledgebase against the syllabus one objective at a time — not domain by domain, which is exactly where a gap hides behind a healthy-looking total. Each individual item is tested for whether a real question could be grounded in sources we actually hold, and everything that cannot is written down as a gap.

Then a person goes and finds the missing documentation. Not a crawler: somebody who knows the subject, searching for what the probe says is absent and judging what is worth adding. Then we probe again, and keep going until the syllabus is covered. It is slow and it is hands-on, and we have not found an honest shortcut.

What we will not promise

We cannot cover what a vendor has never published. Some certifying bodies document their subject thoroughly and in public; others keep it behind a book they sell. Where the sources genuinely do not exist, we will not invent material to fill the hole — we won’t ask you a question unless we know we can give you the right answer (and a link to it, so you can add to your study guide).

We have found gaps in our own knowledgebase doing this. That is not something to hide; it is the whole reason for looking. The knowledgebase to worry about is the one nobody has ever probed.

Can you pass our exams without knowing anything?

Some practice exams can be “gamed”: beaten by pattern alone — always pick the longest answer, avoid the one that explains itself. We ran those tricks against every exam we create. Here is what they score.

Always pick the longest answer

27.8% against 26.1% for pure guessing

The shaded span is the 95% range you would expect from chance alone on a sample this size. We land OUTSIDE it, 3.14 standard deviations from the middle.

Outside the band

Why lower would be just as bad

Our target is chance, not zero. An exam where the longest answer wins 40% of the time is gameable — but so is one where it wins 10%, because the trick simply inverts to avoid the longest answer. The only result worth claiming is one indistinguishable from chance, which means the length of an option carries no information in either direction.

That is why this page reports a band rather than a target, and why we would report a score of 12% as a failure just as readily as 38%.

The other tells

Measured across the same 2316 single-answer questions. One standard deviation on a sample this size is 0.9 points, so anything within about 2 points of expectation is noise.

Each row is a trick a candidate could apply without reading the question.
TrickScoresExpected
Always pick the longest The classic tell. Wordier answers are often the careful, correct ones. 27.8%25.0% 3.1σ out
Always pick the same letter Keys cluster when a writer drafts without checking the spread. 0.6–25.8%25.0% 27.2σ out
Pick the odd one out 254 questions out of 2316 have three options opening with the same word and a fourth that does not. If the key were the different-looking one, this would win. A further 142 have every option opening alike, where there is no odd one out to pick; those are not counted either way. 14.6%25.0% 3.8σ out
Pick the third-longest Not a trick anyone would think of — but we measure it, and it is reported here whatever it says. 25.7%25.0% Inside band

The numbers we did not like

We report our audit figures every month. Even the ones we don’t like.

Always pick the longest: 27.8%

3.1 standard deviations from chance — not noise. A candidate applying it still scores 27.8% and fails, and it is a heuristic nobody would arrive at unprompted; part of it is likely an artifact of how we order options of equal length.

Always pick the same letter: 0.6–25.8%

27.2 standard deviations from chance — not noise. A candidate applying it still scores 0.6–25.8% and fails, and it is a heuristic nobody would arrive at unprompted; part of it is likely an artifact of how we order options of equal length.

Pick the odd one out: 14.6%

3.8 standard deviations from chance — not noise. The key is never the odd-looking option, and that is as usable as always being it — a candidate who crosses out the different-looking answer improves from one-in-four to one-in-three on these questions without reading them. It happens because distractors get written as variations on the key’s own phrasing, so the key ends up sharing an opening word with them. The fix belongs where the exam is written, and that is where we are making it; this page will say so when the figure moves.

One exam tells you nothing

A single exam carries about 45 single-answer questions. At that size one standard deviation is 6.5 points, so any individual exam can score anywhere between roughly 12% and 38% on the longest-answer trick and mean nothing at all.

Of the 52 exams in this run, 3 sat outside their own bands. It is why we publish the aggregate rather than a league table of our own exams.

Our guide, Certification Exams: Tactics and Strategy, recommends checking for these tells in sample exams. We figured we should live up to that. Full disclosure: we sell the exams we’re measuring here.

What this audit does not cover

Drills are not audited, and that is on purpose.

An exam measures you. A drill teaches you. Everything on this page is about our exams, because an exam’s job is to discriminate — a score you could have got without knowing the material is worthless, which is what all this measurement is for.

A drill has the opposite job. It is ten questions on one thing you got wrong, and letting the answer be findable is how teaching works. We do not audit drills for gameability because a gameable drill is not a defect; it is a drill. The trade is real and we would rather say it plainly: a drill score is not evidence of knowledge, and we do not report one. Use the exams to find out where you stand.

Are these the questions you will see?

No. The real questions are proprietary, and anyone who tells you otherwise is selling you a leak. But if you can answer ours, you can answer theirs.

We write from the same published documentation the exam is built on, against the same syllabus, in the vendor’s own vocabulary — the Vocabulary tab measures how closely. What we do not have, and would not use, is the question bank.

So we aim past it. A practice exam that matches the real one in difficulty tells you you are borderline; ours tell you that you are ready. If our exams feel unfair in places, that is the intent — the margin is deliberate, and it is the reason a score here is worth something.

That is also why every figure on this page matters. We cannot show you the real questions, so the only honest thing we can offer instead is evidence about the questions we do write: whether they can be gamed, whose words they use, and what we found when we looked.

Whose words do we use

Organizations write their certification exams using their vocabulary, and you will need to know it to succeed on their exams.

We use the vendor’s terminology when we write our exams so that product names and API identifiers match the vendor’s. Across 48 exams and 13 credentials we counted every technical term in every stem and option — 1673 uses — and checked each one against the vendor’s live pages.

68.1% of technical terms we use appear in the vendor’s own documentation

By credential: PCE 100%, SAA-03 100%, SC-500 100%, SCS-C03 94%, AZ-400 94%, CSE 78%, SAA-C03 76%, DOP-C02 74%, CCAR-F 70%, AI-200 66%, ACE 60%, PDE 55%, AZ-104 54%.

The other 31.9% is not invented product vocabulary. Judge that on the list rather than on our word for it — here are all 241 terms:

lookup_order, search-api, status-api, summarizer, replan, player_id, route-planner, fault-intake, summariser, alarms, archive-api, card_txn, dispatch-docs, pg-password, prescriptions, arrivals-feed, dispatch-api, fleet-state, fw-adhoc, interval-agg, plant-net, refunds-sub, trailer_temps, billing-work, data.tags, dispatch-worker-v41, dispatch-worker-v42, escalate_to_human, extract_invoice, ledger-platform, passages, price-import, vc-analytics, vision-net, article_id, billing-sub, calcLateFee, computeLegCost, corrandocs, costcentre, customs_hold_minutes, data.severity, depot-cache, digital-net, fetch_url, films/nightjar-1080.mp4, hg-logs, mes-db, meter-state, meter_readings, mg-ops, passage_embeddings, payment_terms, pellmoorrefs, revenue_daily, revenue_hourly_mv, scorecards, spawn_subagent, sub-analytics, subscriber_devices, time_entries, transcriber, adhoc, agg-sub, booking_items, cdr_daily, chunks_archive, chunks_v2, ci-deploy, ClaimsAdjusters, claude_args, Contoso.Json, corran-apps, daily_article_views_mv, dba_ops, delivery_date, dispatch-api--v12, effective_date, end_date, erx, extract_contract, fuel_surcharges, IntakeQueue, lateFee, lookup_vendor, lp-eu, lp-rx-landing, lp-shared-temp, map-service, mcp__support__process_refund, mg-finance, migration-guide, net60, netwatch, notifier, PBL-, pellmoorvhd, pg-password-v2, pricing-worker, pw-dataflow-temp, review-portal, sample_point, session_logs, settlement, shipper_id, showId, StockChanged, trailers/nightjar.mp4, trial-etl, trial-reader, truck-events, vnet-hub, weight_band, admissions, adt-events, answer-audit, api-5xx-high, AppTraces, archive-sub, article_meta, aws.securityhub, bigquery.googleapis.com/tables.delete, BurstBalance, cell-counters, Claims/, claims_features, Clinic, cm-masters, ConnectionStrings.Catalog, ConnectionStrings__Catalog, ConnectionStrings_Catalog, console.log, crash_reports, createSecret, CreateStackSet, customer_usage, customerId, daily_retention_mv, daily_sales_mv, db-docs, DecodeEvents, delay_training, departed, device_model, direct_prompt, dispatch-worker, dispensed_date, embed-key, event_date, event_id, event_name, extract_bill_of_lading, failed_documents, fees/core.ts, fees/index.ts, fetch_all_pages, fetch_page_image, fleet-builds, fw-dash, game-events, game_id, get_billing_note, get_routing_note, get_tracking_note, governing_law, grp-line-supervisors, guidelines/queueTrigger, guild-chat, hf-customs, hgauditstore, iam.serviceAccountKeys.list, iam.serviceAccounts.disable, kenton, kg-delivery-notes, larkspuracr, law-digital, LedgerLockService, legacy-billing-conventions, limits.cpu, limits.memory, load_document, match_start_failures, MediaAllowList, mem_used_percent, meterReader, mf-admin, mf-finance, mf-keys, Microsoft.Compute/virtualMachines/delete, ModuleNotFoundError, net30, net90, new-documents, npm-v1, order-worker, outage-tickets, outdoor-furniture, packages/billing, packages/billing/api/CLAUDE.md, packages/billing/ui, ParseEvents, partner_availability, Patient, pellmoorpacs, player_scores, purchase_date, pw-lab-results, qa-records, rate-tables, region_name, reportedAt, requests.memory, rg-content, ride_history, robot-, roles/cloudsql.client_withcond_, roles/storage.objectViewer_withcond_, route_id, Router.cs, router.cs, rsv-brack-plant, sailing-notice, ServiceBusConnection, ServiceBusReceiverOptions.ReceiveMode, SettlementBatchWriter, shop.alderandfinch.com, spec.loadBalancerIP, start_date, stbrackenart, summarise, Tarnwood/GameServer, thicket-base, timetable-db.ops.meridian.internal, total_confidence, tournaments, trailers/, truckId, ValidationError, verify_fact, vm-erp-db01, WAREHOUSE_TOKEN

They fall into three kinds, and none of them is a product we made up. Most are the furniture of our own scenarios — the fields, tools and systems belonging to the fictional companies our questions are set in, which have to be invented because the company is. Some belong to somebody else’s technology: a question about caching can say allkeys-lru, and a question about containers linux/arm64, without either being in this vendor’s documentation. And some are our own measurement being too strict — usage.input_tokens is real and is documented, but it appears in the vendor’s pages inside a block of JSON rather than as that literal string, so our test scores it a miss. We would rather the instrument understated us than flattered us, so we have left it that way.

CSE, PCE, SAA-03, SC-500 sits lowest, on too few measurable terms to mean anything - the same small-sample caution that applies everywhere else on this page. This counts things that are named in code: product names, API fields, configuration keys. It says nothing about ordinary English used as a term of art — a vendor who uses sweep or headline to mean something particular will still catch you out, and this measurement would never know.

Method

Sample
52 exams live in October 2026 — 2316 single-answer questions across ACE-EXAM, AI-200, AZ-104, AZ-305, AZ-400, CCAR-F, CSE-EXAM, DOP-C02, PCE-EXAM, PDE-EXAM, PML-EXAM, SAA-03, SAA-C03, SC-500, SCS-C03. Multiple-response items are excluded; the tricks do not apply to them.
Ties
Where two options tie for longest, the trick scores a fraction rather than a win. Counting a tie as a hit would flatter us.
Baseline
Chance is computed per question from its own option count, not assumed to be 25%.
Left out
Nothing on sale. All the exams in our live catalogue are counted above. Exams we have retired are not: they cannot be bought, so they are not evidence about what we ship.
Reproducing
We update our exams regularly (at least every month) and we re-audit our exams as part of that. We’ll republish this page after every audit.