Framework note
Cucumber, SpecFlow, Behave: what the published cases show
Research desk, testautomationvendors.com · published 2026-11-12 · data as of 2026-11-12 · 53 records
- 53
- verified cases
- 9
- vendors
- 19 of 53
- name the client
- 44 of 134
- metrics with a baseline
53 verified cases in this dataset name Cucumber, SpecFlow, Behave, published by 9 vendors. Grades: A 4, B 22, C 27. 19 of 53 name the client, and 44 of 134 published metrics state a baseline.
What stands out
- 01a1qa publishes 15 of the 53 Cucumber, SpecFlow, Behave cases; the three vendors with the most publish 40, so a reader is looking at a concentrated set of sources.
- 0244 of 134 metrics in these cases state a before value; across all 639 metrics in the dataset the count is 209.
- 034 of 53 reach grade A, which needs a named client and one metric with both a before value and a stated method.
- 04The field left out most often is start year: 46 of 53 cases do not state it.
Counts come from verified records. A baseline is the before value a vendor states for a metric; quoted claims carry their grade and baseline flag.
- baseline:
- The measured value before the work started, against which a later value is compared. source
01Cases, vendors and grades
The count covers every verified case whose text names Cucumber, SpecFlow, Behave, including vendors outside the ranked 12.
Vendors with cases: a1qa (15), Codoid (15), DeviQA (10), QAwerk (1), Qualitest (1), TestDevLab (3), TestFort (1), ThinkSys (1) and Zuci Systems (6).
Table view
| Item | Count |
|---|---|
| Grade A | 4 |
| Grade B | 22 |
| Grade C | 27 |
| Total | 53 |
02Clients and industries
Named clients appear in 19 cases; 34 keep the client anonymous.
Industries stated: Accounts payable automation (1), AI-driven digital growth optimization SaaS (1), Automotive services B2B marketplace (1), Clinical trial data science (1), Cloud customer support software (1), Cloud ERP software for construction (1), Construction quality management software (1) and Consumer electronics retail (1).
Client types stated: enterprise (3), public (1) and startup (1).
03Metric kinds and baselines
The Cucumber, SpecFlow, Behave cases publish 134 metrics. 44 state a baseline and 16 state both a baseline and a method.
- “regression testing time was reduced from 5 days to 1.5 days”before 5 days after 1.5 days a1qa a1qa-4 grade B, partial baseline yes method yes
- “the number of total handled requests increased from 400 to 46,000”before 400 after 46,000 a1qa a1qa-9 grade B, partial baseline yes method yes
- “minimizing scripts code smells from 190 to 1”before 190 after 1 a1qa a1qa-23 grade B, partial baseline yes method yes
- “a full exploration of all 6 modules took 2 months instead of a year”before a year after 2 months a1qa a1qa-67 grade B, partial baseline yes method no
- “Reduced 2 hours of testing to 25 minutes”before 2 hours after 25 minutes Codoid codoid-10 grade B, partial baseline yes method yes
- “Regression window cut from 5 days → 4 hours”before 5 days after 4 hours Codoid codoid-19 grade B, partial baseline yes method no
| Metric kind | Metrics | With baseline |
|---|---|---|
| other | 35 | 17 |
| execution-time | 29 | 15 |
| suite-size | 28 | 6 |
| coverage | 15 | 6 |
| defects | 13 | 0 |
| cost | 8 | 0 |
| release-frequency | 6 | 0 |
04Languages, CI tools and clouds named alongside
Languages: Java (20), C# (11), Ruby (7), Python (6), JavaScript (5), C/C++ (1), Groovy (1) and TypeScript (1).
CI tools: Jenkins (24), Azure DevOps (9), GitHub Actions (4), Bamboo (2), CircleCI (2), GoCD (1), TeamCity (1) and TFS (1).
Device or browser clouds: BrowserStack (10), AWS (5), Sauce Labs (3), Azure (2), Device Farm (1) and Kobiton (1).
05Migration flows
No verified case tags a migration into or out of Cucumber, SpecFlow, Behave.
06The evidence ladder
Each row counts the Cucumber, SpecFlow, Behave cases that make that disclosure; the rows are separate counts, ordered from the most basic disclosure to grade A. Of 53 Cucumber, SpecFlow, Behave cases, 4 reach grade A, and the rarest disclosure below it is "reach grade a" (4 cases).
The right-hand column gives the same steps for all 263 verified cases, so a reader can set this subset against the whole dataset.
Table view
| Item | Count |
|---|---|
| Name a listed framework | 53 |
| Name the client | 19 |
| Publish at least one metric | 45 |
| State a baseline for a metric | 14 |
| State baseline and method for one metric | 9 |
| Reach grade A | 4 |
| Total | 144 |
| Step | Cucumber, SpecFlow, Behave cases | All verified cases |
|---|---|---|
| Name a listed framework | 53 of 53 | 263 of 263 |
| Name the client | 19 of 53 | 91 of 263 |
| Publish at least one metric | 45 of 53 | 210 of 263 |
| State a baseline for a metric | 14 of 53 | 75 of 263 |
| State baseline and method for one metric | 9 of 53 | 35 of 263 |
| Reach grade A | 4 of 53 | 18 of 263 |
07Frameworks named alongside Cucumber, SpecFlow, Behave
47 of 53 cases name at least one other listed framework. The pairing shows which stacks vendors describe together, not which combination works.
Table view
| Item | Count |
|---|---|
| Selenium WebDriver | 32 |
| Postman and Newman | 15 |
| Appium | 11 |
| Apache JMeter | 9 |
| REST Assured | 6 |
| Playwright | 3 |
| WebdriverIO | 3 |
| Cypress | 1 |
| Total | 80 |
08Where the cases come from
a1qa (15), Codoid (15) and DeviQA (10) publish the most Cucumber, SpecFlow, Behave cases. A vendor with many cases here shows breadth of published work; the grade columns show how much of it can be checked.
| Vendor | Cases | Grade A or B | With a baselined metric |
|---|---|---|---|
| a1qa | 15 | 4 | 4 |
| Codoid | 15 | 5 | 2 |
| DeviQA | 10 | 10 | 7 |
| Zuci Systems | 6 | 1 | 0 |
| TestDevLab | 3 | 3 | 0 |
| QAwerk | 1 | 1 | 1 |
| Qualitest | 1 | 1 | 0 |
| TestFort | 1 | 0 | 0 |
| ThinkSys | 1 | 1 | 0 |
09What vendors leave out
25 of 53 cases do not state a team size and 29 do not state a duration.
8 of 134 metrics are cost figures and 1 mention flakiness, so maintenance cost and flaky test rates are rarely published.
10Questions for a Cucumber, SpecFlow, Behave proposal
Ask for suite size and run time before and after the work: 43 of 53 cases give no before value for either.
Ask for the flaky test rate over a stated number of runs: 1 of 134 metrics mention flakiness at all.
Ask who owns maintenance after hand-over and what it costs per month: 8 of 134 metrics are cost figures.
Ask for a reference client you can call: 34 of 53 cases keep the client anonymous.
Questions
How many vendors have published Cucumber, SpecFlow, Behave cases?
9 vendors in this dataset, with 53 verified cases between them, as of the dataAsOf date on this page.
How many Cucumber, SpecFlow, Behave cases name the client?
19 of 53. A named client is required for grade A.
Do Cucumber, SpecFlow, Behave case studies publish baselines?
44 of 134 metrics state a before value.
Does this page rank Cucumber, SpecFlow, Behave against other frameworks?
No. The site counts which vendors have evidence on a framework and never compares frameworks.
Records used
This post reads 53 records; the newest verifiedOn date among them sets the data-as-of date.
Show record ids
a1qa-4, a1qa-6, a1qa-8, a1qa-9, a1qa-13, a1qa-15, a1qa-17, a1qa-21, a1qa-23, a1qa-37, a1qa-38, a1qa-41, a1qa-44, a1qa-60, a1qa-67, codoid-2, codoid-5, codoid-6, codoid-10, codoid-11, codoid-12, codoid-15, codoid-19, codoid-20, codoid-21, codoid-22, codoid-24, codoid-26, codoid-27, codoid-40, deviqa-9, deviqa-10, deviqa-12, deviqa-16, deviqa-19, deviqa-25, deviqa-28, deviqa-37, deviqa-44, deviqa-49, qawerk-5, qualitest-11, testdevlab-2, testdevlab-3, testdevlab-4, testfort-10, thinksys-8, zuci-systems-2, zuci-systems-4, zuci-systems-5, zuci-systems-6, zuci-systems-7, zuci-systems-8
More from the blog
Framework note
Playwright: what the published cases show
48verified cases
48 verified cases from 14 vendors name Playwright: grades, client types, metric kinds, stacks and the fields vendors leave out.
data as of 2026-11-12 · 48 records
Framework note
Cypress: what the published cases show
25verified cases
25 verified cases from 10 vendors name Cypress: grades, client types, metric kinds, stacks and the fields vendors leave out.
data as of 2026-11-12 · 25 records
Framework note
Selenium WebDriver: what the published cases show
88verified cases
88 verified cases from 13 vendors name Selenium WebDriver: grades, client types, metric kinds, stacks and the fields vendors leave out.
data as of 2026-11-12 · 88 records
Related pages
Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.