Blog
What the published cases show
Every post here is computed from the graded case and evidence records and lists the record ids it read. Nothing is padded: a post whose data minimum is not met is listed on the not-published page instead. Frameworks are counted, never compared.
- 31
- posts
- 280
- records cited
- 9
- framework notes
- 2026-11-12
- data as of
Atom feed for new posts and refreshed counts
Start here
Three notes on reading vendor case studies, each built on the case files of this site.
Note for buyers
How to read a test automation case study: the fields that are usually missing
222 of 263cases omit start year
Field by field, how often 263 verified automation cases leave out team size, duration, baseline, client name and CI.
data as of 2026-11-12 · 263 records
Note for buyers
Why a percentage without a baseline is a claim, not a result
246percentage claims
246 percentage claims in verified automation cases, and how many state the starting value.
data as of 2026-11-12 · 126 records
Note for buyers
How this ranking grades a case: A, B, C and X with real examples
18grade A
Grade rules with examples from 263 verified cases: A 18, B 108, C 137; 0 removed as X.
data as of 2026-11-11 · 4 records
Framework notes 9
One per framework page that passes the threshold, with the grades, clients, metrics and gaps of the cases naming it.
Cross-case counts 10
Counts across every verified case: baselines, clients, team sizes, durations, migrations, coverage, languages, CI tools and industries.
Cross-case count
How often published automation case studies state a baseline
75 of 263cases state a baseline
Of 263 verified automation cases, 75 state a before value for at least one metric. Counts by vendor and metric kind.
data as of 2026-11-12 · 263 records
Cross-case count
Execution time claims across published cases: with and without baselines
120execution time claims
120 execution time claims from verified automation cases, split by whether the vendor states the before value.
data as of 2026-11-12 · 86 records
Cross-case count
Named versus anonymous clients in automation case studies, by vendor
91 of 263cases name the client
91 of 263 verified automation cases name the client. Counts per vendor.
data as of 2026-11-12 · 263 records
Cross-case count
Team sizes vendors disclose in automation engagements
149cases state a team size
149 of 263 verified automation cases state a team size. The distribution and the cases.
data as of 2026-11-12 · 149 records
Cross-case count
Engagement lengths disclosed in automation cases
109cases state a duration
109 of 263 verified automation cases state the engagement length. Distribution and cases.
data as of 2026-11-12 · 109 records
Cross-case count
What migrations vendors publish: from and to, counted
11migration cases
11 verified automation cases describe moving a test suite between frameworks. Flows counted.
data as of 2026-11-12 · 11 records
Cross-case count
Coverage claims in automation cases: how many define the denominator
88coverage claims
88 coverage claims from verified automation cases, sorted by whether the quote says what the figure is a share of.
data as of 2026-11-12 · 76 records
Cross-case count
Languages paired with each web framework in published cases
139cases name a language
139 verified cases name a programming language. Pairings with each web UI framework, counted.
data as of 2026-11-12 · 139 records
Cross-case count
CI tools named in automation cases
127cases name a CI tool
127 verified automation cases name a CI tool. Counts per tool and per vendor.
data as of 2026-11-12 · 127 records
Cross-case count
Industries that appear in automation case studies
251distinct industry labels
Industries stated across 263 verified automation cases, counted.
data as of 2026-11-12 · 263 records
Notes for buyers 12
Questions to ask, fields to check and limits of the data, each citing case files from this site.
Note for buyers
What an automation coverage figure can mean: the denominators seen in published cases
88coverage claims
88 coverage claims sorted by what they measure: test cases, regression effort, features or code.
data as of 2026-11-12 · 76 records
Note for buyers
Questions to ask a vendor about a migration case before you rely on it
11migration cases
What 11 published migration cases disclose and leave out, turned into questions for the vendor.
data as of 2026-11-12 · 11 records
Note for buyers
RFP questions per framework layer, derived from what cases disclose
168web UI cases
Request-for-proposal questions for web UI, mobile, API and performance automation, each tied to a disclosure gap counted in published cases.
data as of 2026-11-12 · 393 records
Note for buyers
Flakiness claims: how cases describe them and what to ask for
4flakiness claims
4 metrics in verified automation cases mention flaky or unstable tests.
data as of 2026-11-12 · 4 records
Note for buyers
Team size and ramp-up: what disclosed cases show
149cases state a team size
149 verified automation cases state a team size; what they say about ramp-up.
data as of 2026-11-12 · 149 records
Note for buyers
Device and browser clouds named in cases, and what that implies for your bill
79cases name a cloud
79 verified automation cases name a device or browser cloud.
data as of 2026-11-12 · 79 records
Note for buyers
What this ranking cannot tell you, and where to get it
3things the data does not measure
Service quality, current staff and pricing are outside the data; where to look instead.
data as of 2026-11-11 · 3 records
Note for buyers
Reading a Clutch review for framework evidence
17reviews name a framework
Collectors kept only verified third-party reviews whose text names a listed framework: 17 across 6 of 17 vendors.
data as of 2026-11-12 · 17 records
Note for buyers
How to request a correction and what gets logged
10working days to answer
The correction process for vendors and readers, and what the changelog records.
data as of 2026-11-11 · 3 records
Related pages
Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.