Independent audit · October 2026
A practice exam can only ask you about what its source material covers. When that material has a hole, the exam has the same hole — and you are the last person who would ever notice.
Think about how that failure actually reaches you. You do not get a wrong answer. You do not get a badly worded question. You get nothing — a topic on the syllabus simply never comes up. You sit the exam, you score well, you feel ready, and the one area you were never asked about is still waiting for you on the day.
An absence makes no noise. A wrong question gets reported: somebody complains and it gets fixed. A missing topic produces no complaint from anyone, because nobody can report a question that was never asked. It does not show up in a score, a pass rate, or a count of questions per domain — that count comes out right whether the questions are spread across the whole syllabus or bunched onto the parts the source material happens to cover well.
A syllabus is written in the certifying body’s language: broad statements of what a competent practitioner should be able to do. The documentation an exam has to be built from is written in the vendor’s language — product names, settings, limits. The two vocabularies do not have to meet, and often do not.
So you cannot check coverage by searching for it. Search the sources for a syllabus topic and you will get hits whether or not anything behind them could settle a question, because unrelated products share the same words — the same term can mean four different things to one vendor. Counting pages tells you no more. The only way to find out whether a topic can be examined is to try to examine it.
Before we build exams for a certification, we probe our knowledgebase against the syllabus one objective at a time — not domain by domain, which is exactly where a gap hides behind a healthy-looking total. Each individual item is tested for whether a real question could be grounded in sources we actually hold, and everything that cannot is written down as a gap.
Then a person goes and finds the missing documentation. Not a crawler: somebody who knows the subject, searching for what the probe says is absent and judging what is worth adding. Then we probe again, and keep going until the syllabus is covered. It is slow and it is hands-on, and we have not found an honest shortcut.
We cannot cover what a vendor has never published. Some certifying bodies document their subject thoroughly and in public; others keep it behind a book they sell. Where the sources genuinely do not exist, we will not invent material to fill the hole — we won’t ask you a question unless we know we can give you the right answer (and a link to it, so you can add to your study guide).
We have found gaps in our own knowledgebase doing this. That is not something to hide; it is the whole reason for looking. The knowledgebase to worry about is the one nobody has ever probed.
Some practice exams can be “gamed”: beaten by pattern alone — always pick the longest answer, avoid the one that explains itself. We ran those tricks against every exam we create. Here is what they score.
Always pick the longest answer
The shaded span is the 95% range you would expect from chance alone on a sample this size. We land OUTSIDE it, 3.14 standard deviations from the middle.
Outside the bandOur target is chance, not zero. An exam where the longest answer wins 40% of the time is gameable — but so is one where it wins 10%, because the trick simply inverts to avoid the longest answer. The only result worth claiming is one indistinguishable from chance, which means the length of an option carries no information in either direction.
That is why this page reports a band rather than a target, and why we would report a score of 12% as a failure just as readily as 38%.
Measured across the same 2316 single-answer questions. One standard deviation on a sample this size is 0.9 points, so anything within about 2 points of expectation is noise.
| Trick | Scores | Expected | |
|---|---|---|---|
| Always pick the longest The classic tell. Wordier answers are often the careful, correct ones. | 27.8% | 25.0% | 3.1σ out |
| Always pick the same letter Keys cluster when a writer drafts without checking the spread. | 0.6–25.8% | 25.0% | 27.2σ out |
| Pick the odd one out 254 questions out of 2316 have three options opening with the same word and a fourth that does not. If the key were the different-looking one, this would win. A further 142 have every option opening alike, where there is no odd one out to pick; those are not counted either way. | 14.6% | 25.0% | 3.8σ out |
| Pick the third-longest Not a trick anyone would think of — but we measure it, and it is reported here whatever it says. | 25.7% | 25.0% | Inside band |
We report our audit figures every month. Even the ones we don’t like.
3.1 standard deviations from chance — not noise. A candidate applying it still scores 27.8% and fails, and it is a heuristic nobody would arrive at unprompted; part of it is likely an artifact of how we order options of equal length.
27.2 standard deviations from chance — not noise. A candidate applying it still scores 0.6–25.8% and fails, and it is a heuristic nobody would arrive at unprompted; part of it is likely an artifact of how we order options of equal length.
3.8 standard deviations from chance — not noise. The key is never the odd-looking option, and that is as usable as always being it — a candidate who crosses out the different-looking answer improves from one-in-four to one-in-three on these questions without reading them. It happens because distractors get written as variations on the key’s own phrasing, so the key ends up sharing an opening word with them. The fix belongs where the exam is written, and that is where we are making it; this page will say so when the figure moves.
A single exam carries about 45 single-answer questions. At that size one standard deviation is 6.5 points, so any individual exam can score anywhere between roughly 12% and 38% on the longest-answer trick and mean nothing at all.
Of the 52 exams in this run, 3 sat outside their own bands. It is why we publish the aggregate rather than a league table of our own exams.
Drills are not audited, and that is on purpose.
An exam measures you. A drill teaches you. Everything on this page is about our exams, because an exam’s job is to discriminate — a score you could have got without knowing the material is worthless, which is what all this measurement is for.
A drill has the opposite job. It is ten questions on one thing you got wrong, and letting the answer be findable is how teaching works. We do not audit drills for gameability because a gameable drill is not a defect; it is a drill. The trade is real and we would rather say it plainly: a drill score is not evidence of knowledge, and we do not report one. Use the exams to find out where you stand.
No. The real questions are proprietary, and anyone who tells you otherwise is selling you a leak. But if you can answer ours, you can answer theirs.
We write from the same published documentation the exam is built on, against the same syllabus, in the vendor’s own vocabulary — the Vocabulary tab measures how closely. What we do not have, and would not use, is the question bank.
So we aim past it. A practice exam that matches the real one in difficulty tells you you are borderline; ours tell you that you are ready. If our exams feel unfair in places, that is the intent — the margin is deliberate, and it is the reason a score here is worth something.
That is also why every figure on this page matters. We cannot show you the real questions, so the only honest thing we can offer instead is evidence about the questions we do write: whether they can be gamed, whose words they use, and what we found when we looked.
Organizations write their certification exams using their vocabulary, and you will need to know it to succeed on their exams.
We use the vendor’s terminology when we write our exams so that product names and API identifiers match the vendor’s. Across 48 exams and 13 credentials we counted every technical term in every stem and option — 1673 uses — and checked each one against the vendor’s live pages.
By credential: PCE 100%, SAA-03 100%, SC-500 100%, SCS-C03 94%, AZ-400 94%, CSE 78%, SAA-C03 76%, DOP-C02 74%, CCAR-F 70%, AI-200 66%, ACE 60%, PDE 55%, AZ-104 54%.
The other 31.9% is not invented product vocabulary. Judge that on the list rather than on our word for it — here are all 241 terms:
lookup_order, search-api, status-api, summarizer, replan, player_id, route-planner, fault-intake, summariser, alarms, archive-api, card_txn, dispatch-docs, pg-password, prescriptions, arrivals-feed, dispatch-api, fleet-state, fw-adhoc, interval-agg, plant-net, refunds-sub, trailer_temps, billing-work, data.tags, dispatch-worker-v41, dispatch-worker-v42, escalate_to_human, extract_invoice, ledger-platform, passages, price-import, vc-analytics, vision-net, article_id, billing-sub, calcLateFee, computeLegCost, corrandocs, costcentre, customs_hold_minutes, data.severity, depot-cache, digital-net, fetch_url, films/nightjar-1080.mp4, hg-logs, mes-db, meter-state, meter_readings, mg-ops, passage_embeddings, payment_terms, pellmoorrefs, revenue_daily, revenue_hourly_mv, scorecards, spawn_subagent, sub-analytics, subscriber_devices, time_entries, transcriber, adhoc, agg-sub, booking_items, cdr_daily, chunks_archive, chunks_v2, ci-deploy, ClaimsAdjusters, claude_args, Contoso.Json, corran-apps, daily_article_views_mv, dba_ops, delivery_date, dispatch-api--v12, effective_date, end_date, erx, extract_contract, fuel_surcharges, IntakeQueue, lateFee, lookup_vendor, lp-eu, lp-rx-landing, lp-shared-temp, map-service, mcp__support__process_refund, mg-finance, migration-guide, net60, netwatch, notifier, PBL-, pellmoorvhd, pg-password-v2, pricing-worker, pw-dataflow-temp, review-portal, sample_point, session_logs, settlement, shipper_id, showId, StockChanged, trailers/nightjar.mp4, trial-etl, trial-reader, truck-events, vnet-hub, weight_band, admissions, adt-events, answer-audit, api-5xx-high, AppTraces, archive-sub, article_meta, aws.securityhub, bigquery.googleapis.com/tables.delete, BurstBalance, cell-counters, Claims/, claims_features, Clinic, cm-masters, ConnectionStrings.Catalog, ConnectionStrings__Catalog, ConnectionStrings_Catalog, console.log, crash_reports, createSecret, CreateStackSet, customer_usage, customerId, daily_retention_mv, daily_sales_mv, db-docs, DecodeEvents, delay_training, departed, device_model, direct_prompt, dispatch-worker, dispensed_date, embed-key, event_date, event_id, event_name, extract_bill_of_lading, failed_documents, fees/core.ts, fees/index.ts, fetch_all_pages, fetch_page_image, fleet-builds, fw-dash, game-events, game_id, get_billing_note, get_routing_note, get_tracking_note, governing_law, grp-line-supervisors, guidelines/queueTrigger, guild-chat, hf-customs, hgauditstore, iam.serviceAccountKeys.list, iam.serviceAccounts.disable, kenton, kg-delivery-notes, larkspuracr, law-digital, LedgerLockService, legacy-billing-conventions, limits.cpu, limits.memory, load_document, match_start_failures, MediaAllowList, mem_used_percent, meterReader, mf-admin, mf-finance, mf-keys, Microsoft.Compute/virtualMachines/delete, ModuleNotFoundError, net30, net90, new-documents, npm-v1, order-worker, outage-tickets, outdoor-furniture, packages/billing, packages/billing/api/CLAUDE.md, packages/billing/ui, ParseEvents, partner_availability, Patient, pellmoorpacs, player_scores, purchase_date, pw-lab-results, qa-records, rate-tables, region_name, reportedAt, requests.memory, rg-content, ride_history, robot-, roles/cloudsql.client_withcond_, roles/storage.objectViewer_withcond_, route_id, Router.cs, router.cs, rsv-brack-plant, sailing-notice, ServiceBusConnection, ServiceBusReceiverOptions.ReceiveMode, SettlementBatchWriter, shop.alderandfinch.com, spec.loadBalancerIP, start_date, stbrackenart, summarise, Tarnwood/GameServer, thicket-base, timetable-db.ops.meridian.internal, total_confidence, tournaments, trailers/, truckId, ValidationError, verify_fact, vm-erp-db01, WAREHOUSE_TOKEN
They fall into three kinds, and none of them is a product we made up. Most are the furniture of our own scenarios — the fields, tools and systems belonging to the fictional companies our questions are set in, which have to be invented because the company is. Some belong to somebody else’s technology: a question about caching can say allkeys-lru, and a question about containers linux/arm64, without either being in this vendor’s documentation. And some are our own measurement being too strict — usage.input_tokens is real and is documented, but it appears in the vendor’s pages inside a block of JSON rather than as that literal string, so our test scores it a miss. We would rather the instrument understated us than flattered us, so we have left it that way.
CSE, PCE, SAA-03, SC-500 sits lowest, on too few measurable terms to mean anything - the same small-sample caution that applies everywhere else on this page. This counts things that are named in code: product names, API fields, configuration keys. It says nothing about ordinary English used as a term of art — a vendor who uses sweep or headline to mean something particular will still catch you out, and this measurement would never know.