A CMP comparison can show how often sampled implementations match a defined consent policy. To make that comparison useful, report the check, visitor region, eligible site count, and incomplete tests for each provider.
This article explains how to read and build that comparison. It contains no measured CookieTrail vendor rankings; the demonstration uses invented providers and counts.
See how the sample changes a comparison
Switch between all sites and the two regional samples. The invented providers change order because their site groups differ.
declared rejection policy
- Eligible independent sites
- 8 matches / 10 evaluated
- Inconclusive sites
- 2 shown separately; excluded from the rate
declared rejection policy
- Eligible independent sites
- 20 matches / 30 evaluated
- Inconclusive sites
- 4 shown separately; excluded from the rate
The pooled sample puts A ahead. Switch to Germany to see why region composition matters.
All counts, providers, and outcomes are invented. This teaches denominators and sample composition; it is not a CookieTrail benchmark, a representative sample, or a vendor recommendation.
Define the outcome before naming a winner
A consent management platform (CMP) displays the banner and communicates choices. The deployed outcome also depends on tag settings, integrations, custom code, and the policy the site intends to apply.
- Before-choice behavior: does the site follow its declared rule for the tested tool and purpose?
- Rejection: was refusal successfully operated, stored, and enforced for the named check?
- Purpose mapping: does selective permission control the relevant optional use?
- Withdrawal: does changing a saved choice affect subsequent collection?
Publish those outcomes separately. The latter two require additional test paths beyond CookieTrail's current first-load, accept-all, and reject-all audit. A single all-checks score can hide whether a provider's sample lacks coverage for a difficult path.
Compare independent sites with comparable evidence
Identify the deployed CMP and product family using reliable evidence. Keep custom implementations, ambiguous detections, versions, and major customizations visible. Multiple banners or installations on the same site need review before attribution.
Count independent sites for prevalence. A hundred repeated runs of one template do not create a hundred independent implementations. Select pages consistently and record which integrations were actually tested.
Compare regions and site types separately where the sample allows it. A provider used mainly on simple marketing pages cannot fairly inherit a claim of easier installation from a comparison against providers serving complex publishing stacks.
Keep denominators and uncertainty visible
Calculate each rate from sites eligible for that specific outcome, with numerator and denominator beside it. Show unreadable choices, unsuccessful interactions, incomplete captures, and unknown detections separately. They must not become successful results.
A rate from a handful of sites can change substantially with one new observation. Publish uncertainty and the sampling method; with a convenience sample, a statistical interval still does not establish representativeness. Avoid precise rank ordering when the evidence cannot separate providers.
The demonstration shows why pooling regions can conceal a different regional comparison. Its numbers teach the method and must not be quoted as benchmark results.
Separate the observed result from the explanation
A higher observed match rate may justify investigating clearer documentation, safer defaults, or simpler integration. It doesn't establish any of those causes. Review the configuration and ask implementing teams about their experience.
Use the tracking-strategy comparison to distinguish deliberate limited measurement from an integration failure.
For any named failure, preserve the evidence and give the implementation owner an opportunity to explain it. Confirm what the detector and stored-choice reader can establish for that provider. An untested readback isn't a failed implementation.
Repeat a published study consistently
Keep the original release as a dated snapshot. For a later edition, separate the same-site panel from newly sampled sites, note detector and method changes, and show which checks had enough comparable evidence. A rising success rate can otherwise reflect a changing sample.
CookieTrail can retain regional observations and matching retests for individual implementations. Turning those records into a public benchmark requires a defined study sample, consistent coverage review, aggregation, and appropriate permission for identifiable examples.