Cross-case count
How often published automation case studies state a baseline
Research desk, testautomationvendors.com · published 2026-11-12 · data as of 2026-11-12 · 263 records
- 75 of 263
- cases state a baseline
- 35
- with baseline and method
- 53
- publish no metric
75 of 263 verified cases state a baseline for at least one metric, and 35 also state how or over what period it was measured. 53 cases publish no metric at all.
What stands out
- 01a1qa publishes 61 of the 263 verified cases; the three vendors with the most publish 153, so a reader is looking at a concentrated set of sources.
- 02209 of 639 published metrics state a before value, and 58 also state how or over what period they were measured.
- 03Cases with a named client state a baseline in 40 of 91; anonymous cases do so in 35 of 172. A name and a baseline are separate disclosures, and the grade needs both for A.
Counts come from verified records. A baseline is the before value a vendor states for a metric; quoted claims carry their grade and baseline flag.
- baseline:
- The measured value before the work started, against which a later value is compared. source
The count
A baseline is the before value. A case counts here when any of its metrics states one in measurable terms.
209 of 639 individual metrics carry a baseline.
Table view
| Item | Count |
|---|---|
| Cases with a baselined metric | 75 |
| Cases with metrics but no baseline | 135 |
| Cases with no metric | 53 |
| Total | 263 |
By vendor
Each row counts verified cases and the subset with at least one baselined metric.
| Vendor | Cases | With baseline |
|---|---|---|
| a1qa | 61 | 12 |
| DeviQA | 52 | 43 |
| Codoid | 40 | 4 |
| TestFort | 30 | 2 |
| Qualitest | 12 | 1 |
| TestingXperts | 10 | 2 |
| ThinkSys | 9 | 4 |
| QAwerk | 8 | 1 |
| Testrig Technologies | 8 | 1 |
| Zuci Systems | 8 | 1 |
| Abstracta | 7 | 1 |
| ImpactQA | 4 | 0 |
| TestDevLab | 4 | 0 |
| QualityLogic | 3 | 1 |
| BugRaptors | 2 | 2 |
| QA Madness | 2 | 0 |
| QA Wolf | 2 | 0 |
| betterqa | 1 | 0 |
The evidence ladder
Each row counts the verified cases that make that disclosure; the rows are separate counts, ordered from the most basic disclosure to grade A. Of 263 verified cases, 18 reach grade A, and the rarest disclosure below it is "reach grade a" (18 cases).
The right-hand column gives the same steps for all 263 verified cases, so a reader can set this subset against the whole dataset.
Table view
| Item | Count |
|---|---|
| Name a listed framework | 263 |
| Name the client | 91 |
| Publish at least one metric | 210 |
| State a baseline for a metric | 75 |
| State baseline and method for one metric | 35 |
| Reach grade A | 18 |
| Total | 692 |
| Step | Verified cases | All verified cases |
|---|---|---|
| Name a listed framework | 263 of 263 | 263 of 263 |
| Name the client | 91 of 263 | 91 of 263 |
| Publish at least one metric | 210 of 263 | 210 of 263 |
| State a baseline for a metric | 75 of 263 | 75 of 263 |
| State baseline and method for one metric | 35 of 263 | 35 of 263 |
| Reach grade A | 18 of 263 | 18 of 263 |
Named client against baseline
The table crosses the two disclosures the grade rules rely on.
| Baseline stated | No baseline | |
|---|---|---|
| Named client | 40 | 51 |
| Anonymous client | 35 | 137 |
By metric kind
Execution time and suite size carry baselines more often than coverage or cost when the counts below say so; the table is the source.
| Metric kind | Metrics | With baseline | With baseline and method |
|---|---|---|---|
| other | 177 | 60 | 11 |
| execution-time | 120 | 55 | 20 |
| suite-size | 119 | 44 | 7 |
| coverage | 88 | 39 | 13 |
| defects | 79 | 5 | 4 |
| cost | 28 | 0 | 0 |
| release-frequency | 28 | 6 | 3 |
Questions
What counts as a baseline here?
The source must state the before value or before state in measurable terms, for example a regression run of 8 hours before the work.
Why does a baseline matter for the grade?
Grade A needs a named client and a metric with both a baseline and a method. Anonymous cases reach B only with a baselined metric.
Are cases without metrics excluded?
No. They count toward case volume at their grade weight.
How was each baseline checked?
The verifier re-opened each source page and matched the quoted claim and the before value against the page text.
Records used
This post reads 263 records; the newest verifiedOn date among them sets the data-as-of date.
Show record ids
a1qa-1, a1qa-2, a1qa-3, a1qa-4, a1qa-5, a1qa-6, a1qa-7, a1qa-8, a1qa-9, a1qa-10, a1qa-11, a1qa-12, a1qa-13, a1qa-14, a1qa-15, a1qa-16, a1qa-17, a1qa-18, a1qa-19, a1qa-20, a1qa-21, a1qa-22, a1qa-23, a1qa-24, a1qa-25, a1qa-26, a1qa-27, a1qa-28, a1qa-29, a1qa-30, a1qa-31, a1qa-32, a1qa-33, a1qa-34, a1qa-35, a1qa-36, a1qa-37, a1qa-38, a1qa-39, a1qa-40, a1qa-41, a1qa-42, a1qa-44, a1qa-60, a1qa-61, a1qa-62, a1qa-63, a1qa-64, a1qa-65, a1qa-66, a1qa-67, a1qa-68, a1qa-69, a1qa-70, a1qa-71, a1qa-72, a1qa-73, a1qa-74, a1qa-75, a1qa-76, a1qa-77, abstracta-1, abstracta-2, abstracta-3, abstracta-4, abstracta-5, abstracta-6, abstracta-7, betterqa-1, bugraptors-4, bugraptors-6, codoid-1, codoid-2, codoid-3, codoid-4, codoid-5, codoid-6, codoid-7, codoid-8, codoid-9, codoid-10, codoid-11, codoid-12, codoid-13, codoid-14, codoid-15, codoid-16, codoid-17, codoid-18, codoid-19, codoid-20, codoid-21, codoid-22, codoid-23, codoid-24, codoid-25, codoid-26, codoid-27, codoid-28, codoid-29, codoid-30, codoid-31, codoid-32, codoid-33, codoid-34, codoid-35, codoid-36, codoid-37, codoid-38, codoid-39, codoid-40, deviqa-1, deviqa-2, deviqa-3, deviqa-4, deviqa-5, deviqa-6, deviqa-7, deviqa-8, deviqa-9, deviqa-10, deviqa-11, deviqa-12, deviqa-13, deviqa-14, deviqa-15, deviqa-16, deviqa-17, deviqa-18, deviqa-19, deviqa-20, deviqa-21, deviqa-22, deviqa-23, deviqa-24, deviqa-25, deviqa-26, deviqa-27, deviqa-28, deviqa-29, deviqa-30, deviqa-31, deviqa-32, deviqa-33, deviqa-34, deviqa-35, deviqa-36, deviqa-37, deviqa-38, deviqa-39, deviqa-40, deviqa-41, deviqa-42, deviqa-43, deviqa-44, deviqa-45, deviqa-46, deviqa-47, deviqa-48, deviqa-49, deviqa-50, deviqa-51, deviqa-52, impactqa-1, impactqa-2, impactqa-3, impactqa-4, qa-madness-1, qa-madness-2, qa-wolf-1, qa-wolf-2, qawerk-1, qawerk-2, qawerk-3, qawerk-4, qawerk-5, qawerk-6, qawerk-7, qawerk-8, qualitest-1, qualitest-2, qualitest-3, qualitest-4, qualitest-5, qualitest-6, qualitest-7, qualitest-8, qualitest-9, qualitest-10, qualitest-11, qualitest-12, qualitylogic-1, qualitylogic-2, qualitylogic-3, testdevlab-1, testdevlab-2, testdevlab-3, testdevlab-4, testfort-1, testfort-2, testfort-3, testfort-4, testfort-5, testfort-6, testfort-7, testfort-8, testfort-9, testfort-10, testfort-11, testfort-12, testfort-13, testfort-14, testfort-15, testfort-16, testfort-17, testfort-18, testfort-19, testfort-20, testfort-21, testfort-22, testfort-23, testfort-24, testfort-25, testfort-26, testfort-27, testfort-28, testfort-29, testfort-30, testingxperts-1, testingxperts-2, testingxperts-3, testingxperts-4, testingxperts-5, testingxperts-6, testingxperts-7, testingxperts-8, testingxperts-9, testingxperts-10, testrig-1, testrig-3, testrig-6, testrig-7, testrig-8, testrig-9, testrig-10, testrig-11, thinksys-1, thinksys-2, thinksys-3, thinksys-4, thinksys-5, thinksys-6, thinksys-7, thinksys-8, thinksys-9, zuci-systems-1, zuci-systems-2, zuci-systems-3, zuci-systems-4, zuci-systems-5, zuci-systems-6, zuci-systems-7, zuci-systems-8
More from the blog
Cross-case count
Execution time claims across published cases: with and without baselines
120execution time claims
120 execution time claims from verified automation cases, split by whether the vendor states the before value.
data as of 2026-11-12 · 86 records
Cross-case count
Named versus anonymous clients in automation case studies, by vendor
91 of 263cases name the client
91 of 263 verified automation cases name the client. Counts per vendor.
data as of 2026-11-12 · 263 records
Cross-case count
Team sizes vendors disclose in automation engagements
149cases state a team size
149 of 263 verified automation cases state a team size. The distribution and the cases.
data as of 2026-11-12 · 149 records
Related pages
Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.