Cross-case count
Execution time claims across published cases: with and without baselines
Research desk, testautomationvendors.com · published 2026-11-12 · data as of 2026-11-12 · 86 records
- 120
- execution time claims
- 55
- state the before value
- 20
- also state the method
The dataset holds 120 execution time claims. 55 state the before value and 65 state only an after value or a relative change. 20 also state how or over what period the time was measured.
What stands out
- 0137 claims give a before and an after time in units that can be compared; the median claimed speed-up among them is 4.8x, and the range runs from 1.6x to 168.0x.
- 02These ratios restate vendor claims; none of them says whether the same tests ran before and after.
- 03DeviQA publishes 22 of the 37 cases with a baselined execution time claim; the three vendors with the most publish 30, so a reader is looking at a concentrated set of sources.
Counts come from verified records. A baseline is the before value a vendor states for a metric; quoted claims carry their grade and baseline flag.
- baseline:
- The measured value before the work started, against which a later value is compared. source
Claims with a before value
55 claims give the time before the work; each card shows the before and after values as quoted.
- “cut release decision time from about two working weeks to under one”before about two working weeks after under one working week a1qa a1qa-2 grade B, partial baseline yes method yes
- “regression testing time was reduced from 5 days to 1.5 days”before 5 days after 1.5 days a1qa a1qa-4 grade B, partial baseline yes method yes
- “reducing testing time from several hours to just 3 minutes”before several hours after 3 minutes a1qa a1qa-25 grade B, partial baseline yes method yes
- “Testing is completed faster, within hours rather than days”before days after hours a1qa a1qa-29 grade B, partial baseline yes method yes
- “a one-week scope was covered in only 1 hour”before one week after 1 hour a1qa a1qa-30 grade B, partial baseline yes method yes
- “the entire regression testing flow was reduced from 45 hours to 2 hours”before 45 hours after 2 hours a1qa a1qa-63 grade B, partial baseline yes method yes
- “Cut combined mobile + web regression execution to ~35 minutes (from multi-day cycles)”before multi-day cycles after ~35 minutes Abstracta abstracta-2 grade B, partial baseline yes method no
- “6 Days”before 10+ Days after 6 Days BugRaptors bugraptors-6 grade B, partial baseline yes method no
- “Regression cycle reduced from 4 days to 3 hours.”before 4 days after 3 hours Codoid codoid-1 grade B, partial baseline yes method no
- “Reduced 2 hours of testing to 25 minutes”before 2 hours after 25 minutes Codoid codoid-10 grade B, partial baseline yes method yes
- “Regression window cut from 5 days → 4 hours”before 5 days after 4 hours Codoid codoid-19 grade B, partial baseline yes method no
- “CI/CD in Azure DevOps cut execution time from 9 hours to 40 minutes.”before 9 hours after 40 minutes Codoid codoid-19 grade B, partial baseline yes method no
Claims without a before value
65 claims state only the result or a relative change; the first 10 are shown.
- “2 working days saved per regression decision for the selected scope”before about 3 working days per release after not stated a1qa a1qa-3 grade B, partial baseline no method yes
- “curtailed the time required for release verification testing by 90%”a1qa a1qa-6 grade C, claim only baseline no method yes
- “7.5 boost in smoke testing due to test automation”before not stated after 7.5 a1qa a1qa-16 grade C, claim only baseline no method yes
- “10.4 speed-up in regression testing thanks to test automation”before not stated after 10.4 a1qa a1qa-16 grade C, claim only baseline no method yes
- “~5× drop in regression cycle duration”before not stated after ~5x a1qa a1qa-19 grade C, claim only baseline no method no
- “~2× decrease in average time per test case”before not stated after ~2x a1qa a1qa-19 grade C, claim only baseline no method no
- “the engineers reached a 3X faster testing time”before not stated after 3X faster a1qa a1qa-20 grade C, claim only baseline no method yes
- “Development time of an automated test was reduced by 28%”before 8 hours average per automated test after 28% less a1qa a1qa-24 grade C, claim only baseline no method yes
- “The result was a fourfold reduction in testing time across these areas.”before not stated after 4x reduction a1qa a1qa-29 grade B, partial baseline no method yes
- “40X faster API testing with test automation”before not stated after 40X a1qa a1qa-30 grade B, partial baseline no method yes
Claimed speed-ups, computed
Before and after values were converted to minutes where both name a unit; ranges use their midpoint. 37 claims qualify. The factor is before divided by after.
| Case | Vendor | Grade | Before | After | Factor |
|---|---|---|---|---|---|
| deviqa-51 | DeviQA | A | Manual regression testing took about 2 weeks | 2 hours | 168.0x |
| deviqa-5 | DeviQA | B | ~3 hours to create test data for specific test scenarios | 2 minutes | 90.0x |
| thinksys-1 | ThinkSys | A | 5-7 days (manual) | ~2 hours (automated) | 72.0x |
| deviqa-16 | DeviQA | A | 4 days were needed to run regression testing | ~2 hours | 48.0x |
| codoid-1 | Codoid | B | 4 days | 3 hours | 32.0x |
| codoid-19 | Codoid | B | 5 days | 4 hours | 30.0x |
| qualitest-4 | Qualitest | B | 8 hours | 19 minutes | 25.3x |
| a1qa-63 | a1qa | B | 45 hours | 2 hours | 22.5x |
| deviqa-13 | DeviQA | B | 2.5 weeks | 1 day | 17.5x |
| deviqa-1 | DeviQA | B | 2 days to run regression testing manually | 3 hours | 16.0x |
| codoid-19 | Codoid | B | 9 hours | 40 minutes | 13.5x |
| deviqa-30 | DeviQA | B | Regression testing took about 20 days | Up to 2 days | 10.0x |
| deviqa-44 | DeviQA | A | 20 hours | 2 hours | 10.0x |
| deviqa-13 | DeviQA | B | 7 days | 1 day | 7.0x |
| deviqa-5 | DeviQA | B | >3 hours to execute manual smoke testing | ~30 minutes | 6.0x |
| deviqa-18 | DeviQA | B | >3 hours to run a regression test suite | 30 minutes | 6.0x |
| deviqa-51 | DeviQA | A | 12 hours | 2 hours | 6.0x |
| deviqa-12 | DeviQA | B | >20 hours to run smoke testing | 4 hours | 5.0x |
| codoid-10 | Codoid | B | 2 hours | 25 minutes | 4.8x |
| deviqa-11 | DeviQA | A | 40+ minutes to run automated E2E testing on each pull request | 10 minutes | 4.0x |
| deviqa-12 | DeviQA | B | >70 hours to run regression testing | 18 hours | 3.9x |
| deviqa-7 | DeviQA | B | Regression testing required 2 weeks | 4 days | 3.5x |
| deviqa-11 | DeviQA | A | 7 days to run regression testing | 2 days | 3.5x |
| a1qa-4 | a1qa | B | 5 days | 1.5 days | 3.3x |
| deviqa-1 | DeviQA | B | 1 hour to run automated smoke testing | 20 minutes | 3.0x |
| deviqa-4 | DeviQA | B | Regression testing required more than 12 hours | 4 hours | 3.0x |
| deviqa-10 | DeviQA | B | 60 minutes | approximately 20 minutes | 3.0x |
| deviqa-32 | DeviQA | B | >3 hours were needed to execute manual smoke testing | <1 hour | 3.0x |
| thinksys-1 | ThinkSys | A | 6 hours sequential | ~2 hours with sharding | 3.0x |
| thinksys-3 | ThinkSys | B | ~5 minutes | under 2 minutes | 2.5x |
| deviqa-6 | DeviQA | B | 3-4 days for release testing | 1-2 days | 2.3x |
| testrig-7 | Testrig Technologies | B | 12 hours | 5-6 hours | 2.2x |
| deviqa-50 | DeviQA | B | 2 weeks were needed to run regression testing | <1 week | 2.0x |
| qawerk-5 | QAwerk | A | 3-4 days | 2 days | 1.8x |
| thinksys-2 | ThinkSys | A | 3-4 weeks, fully manual | 2 weeks | 1.8x |
| bugraptors-6 | BugRaptors | B | 10+ Days | 6 Days | 1.7x |
| testrig-7 | Testrig Technologies | B | 5-6 hours | 3-4 hours | 1.6x |
Reading these claims
A relative claim without a before value cannot be checked against a baseline, so it stays a vendor claim.
A before and after pair without a stated period or suite size still leaves open whether the same tests ran in both runs.
Questions
How many execution time claims state the before value?
55 of 120.
Does a faster run mean the same tests ran?
Only when the case states the suite size or scope for both runs; most cases here do not.
Are these figures audited?
No. They are quotes matched against the vendor's own page. Grades describe what the page discloses.
Where are the full claims listed?
Each case file carries a claim ledger with every metric, its before and after values and both flags.
Records used
This post reads 86 records; the newest verifiedOn date among them sets the data-as-of date.
Show record ids
a1qa-2, a1qa-3, a1qa-4, a1qa-6, a1qa-16, a1qa-19, a1qa-20, a1qa-24, a1qa-25, a1qa-29, a1qa-30, a1qa-33, a1qa-44, a1qa-61, a1qa-62, a1qa-63, a1qa-73, a1qa-77, abstracta-2, bugraptors-6, codoid-1, codoid-6, codoid-9, codoid-10, codoid-11, codoid-12, codoid-19, codoid-21, codoid-22, codoid-26, codoid-31, codoid-37, deviqa-1, deviqa-3, deviqa-4, deviqa-5, deviqa-6, deviqa-7, deviqa-9, deviqa-10, deviqa-11, deviqa-12, deviqa-13, deviqa-16, deviqa-18, deviqa-19, deviqa-21, deviqa-22, deviqa-23, deviqa-24, deviqa-27, deviqa-28, deviqa-29, deviqa-30, deviqa-32, deviqa-34, deviqa-35, deviqa-36, deviqa-39, deviqa-40, deviqa-41, deviqa-42, deviqa-44, deviqa-46, deviqa-49, deviqa-50, deviqa-51, qawerk-5, qawerk-8, qualitest-1, qualitest-3, qualitest-4, qualitest-6, qualitest-12, qualitylogic-1, qualitylogic-3, testfort-7, testfort-28, testingxperts-10, testrig-7, thinksys-1, thinksys-2, thinksys-3, zuci-systems-1, zuci-systems-3, zuci-systems-5
More from the blog
Cross-case count
How often published automation case studies state a baseline
75 of 263cases state a baseline
Of 263 verified automation cases, 75 state a before value for at least one metric. Counts by vendor and metric kind.
data as of 2026-11-12 · 263 records
Cross-case count
Named versus anonymous clients in automation case studies, by vendor
91 of 263cases name the client
91 of 263 verified automation cases name the client. Counts per vendor.
data as of 2026-11-12 · 263 records
Cross-case count
Team sizes vendors disclose in automation engagements
149cases state a team size
149 of 263 verified automation cases state a team size. The distribution and the cases.
data as of 2026-11-12 · 149 records
Related pages
Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.