Skip to content
Test Automation Vendors / 263 cases

Note for buyers

RFP questions per framework layer, derived from what cases disclose

Research desk, testautomationvendors.com · published 2026-11-12 · data as of 2026-11-12 · 393 records

168
web UI cases
48
mobile cases
107
API cases
70
performance cases

The questions below come from fields that verified cases in each layer leave out most often. Web UI has 168 cases, mobile 48, API 107 and performance 70.

What stands out

  1. 01web UI: 168 cases; 56 state a baseline and 15 reach grade A.
  2. 02mobile: 48 cases; 14 state a baseline and 1 reach grade A.
  3. 03API: 107 cases; 40 state a baseline and 8 reach grade A.
  4. 04performance: 70 cases; 20 state a baseline and 3 reach grade A.
  5. 05BDD: 53 cases; 14 state a baseline and 4 reach grade A.

Counts come from verified records. A baseline is the before value a vendor states for a metric; quoted claims carry their grade and baseline flag.

baseline:
The measured value before the work started, against which a later value is compared. source

Web UI layer

168 cases. Team size missing in 84; duration missing in 108; no baselined metric in 112.

Ask for suite size, run time, flaky test rate and CI gate rules: 92 of 168 cases name a CI tool.

Mobile layer

48 cases. Team size missing in 18; duration missing in 34; no baselined metric in 34.

Ask which devices and OS versions ran, and whether a device cloud was used: 27 of 48 cases name one.

API layer

107 cases. Team size missing in 22; duration missing in 51; no baselined metric in 67.

Ask how many endpoints are covered and how contract changes are caught: 37 of 107 cases publish a coverage figure.

The evidence ladder

Each row counts the verified cases that make that disclosure; the rows are separate counts, ordered from the most basic disclosure to grade A. Of 263 verified cases, 18 reach grade A, and the rarest disclosure below it is "reach grade a" (18 cases).

The right-hand column gives the same steps for all 263 verified cases, so a reader can set this subset against the whole dataset.

The evidence ladder
Table view
ItemCount
Name a listed framework263
Name the client91
Publish at least one metric210
State a baseline for a metric75
State baseline and method for one metric35
Reach grade A18
Total692
StepVerified casesAll verified cases
Name a listed framework263 of 263263 of 263
Name the client91 of 26391 of 263
Publish at least one metric210 of 263210 of 263
State a baseline for a metric75 of 26375 of 263
State baseline and method for one metric35 of 26335 of 263
Reach grade A18 of 26318 of 263

Performance layer

70 cases. Team size missing in 28; duration missing in 41; no baselined metric in 50.

Ask for the load profile, the p95 response time before and after, and the test environment: 20 of 70 cases give a before value.

Questions

Are these questions specific to one framework?

No. They are grouped by layer and apply to any framework in it.

Why tie questions to gaps?

A gap that most published cases share is the one a written proposal is least likely to answer unprompted.

Where are the per-framework counts?

On each framework page and its framework note.

Does the site rank layers or frameworks?

No.

Records used

This post reads 393 records; the newest verifiedOn date among them sets the data-as-of date.

Show record ids

a1qa-1, a1qa-2, a1qa-3, a1qa-4, a1qa-5, a1qa-6, a1qa-8, a1qa-10, a1qa-12, a1qa-14, a1qa-15, a1qa-16, a1qa-19, a1qa-20, a1qa-21, a1qa-22, a1qa-23, a1qa-26, a1qa-27, a1qa-29, a1qa-32, a1qa-38, a1qa-40, a1qa-41, a1qa-60, a1qa-62, a1qa-63, a1qa-71, a1qa-75, a1qa-76, abstracta-2, abstracta-6, abstracta-7, codoid-1, codoid-2, codoid-3, codoid-5, codoid-7, codoid-8, codoid-9, codoid-10, codoid-11, codoid-12, codoid-13, codoid-14, codoid-15, codoid-16, codoid-17, codoid-18, codoid-19, codoid-20, codoid-21, codoid-22, codoid-23, codoid-24, codoid-27, codoid-30, codoid-33, codoid-34, codoid-35, codoid-39, deviqa-1, deviqa-2, deviqa-3, deviqa-4, deviqa-5, deviqa-6, deviqa-7, deviqa-8, deviqa-9, deviqa-10, deviqa-11, deviqa-12, deviqa-13, deviqa-14, deviqa-16, deviqa-18, deviqa-19, deviqa-20, deviqa-21, deviqa-23, deviqa-24, deviqa-27, deviqa-28, deviqa-29, deviqa-33, deviqa-34, deviqa-35, deviqa-38, deviqa-42, deviqa-43, deviqa-46, deviqa-49, deviqa-51, deviqa-52, impactqa-1, impactqa-4, qa-madness-1, qa-madness-2, qa-wolf-1, qa-wolf-2, qawerk-1, qawerk-2, qawerk-4, qawerk-5, qawerk-6, qawerk-7, qawerk-8, qualitest-2, qualitest-3, qualitest-4, qualitest-5, qualitest-7, qualitest-8, qualitest-10, qualitylogic-1, qualitylogic-3, testdevlab-1, testdevlab-2, testdevlab-3, testdevlab-4, testfort-1, testfort-3, testfort-4, testfort-5, testfort-6, testfort-7, testfort-8, testfort-9, testfort-11, testfort-12, testfort-15, testfort-21, testfort-22, testfort-23, testfort-24, testfort-27, testfort-30, testingxperts-1, testingxperts-2, testingxperts-3, testingxperts-4, testingxperts-8, testingxperts-9, testingxperts-10, testrig-1, testrig-3, testrig-6, testrig-7, testrig-8, testrig-10, testrig-11, thinksys-1, thinksys-2, thinksys-3, thinksys-4, thinksys-5, thinksys-6, thinksys-7, thinksys-8, zuci-systems-1, zuci-systems-2, zuci-systems-3, zuci-systems-4, zuci-systems-5, zuci-systems-6, zuci-systems-7, zuci-systems-8, a1qa-9, a1qa-10, a1qa-14, a1qa-24, a1qa-40, a1qa-66, a1qa-73, abstracta-1, abstracta-2, codoid-1, codoid-6, codoid-9, codoid-11, codoid-13, codoid-15, codoid-18, codoid-23, codoid-25, codoid-26, codoid-30, codoid-39, deviqa-8, deviqa-12, deviqa-15, deviqa-18, deviqa-19, deviqa-21, deviqa-25, deviqa-30, deviqa-37, deviqa-38, deviqa-41, deviqa-46, qawerk-2, qualitest-1, qualitest-2, qualitest-3, qualitest-12, testdevlab-1, testdevlab-3, testfort-9, testfort-10, testfort-11, testfort-28, testrig-6, testrig-8, testrig-10, zuci-systems-7, a1qa-1, a1qa-2, a1qa-4, a1qa-6, a1qa-9, a1qa-11, a1qa-12, a1qa-14, a1qa-15, a1qa-16, a1qa-24, a1qa-25, a1qa-30, a1qa-31, a1qa-32, a1qa-33, a1qa-34, a1qa-39, a1qa-41, a1qa-42, a1qa-60, a1qa-62, a1qa-64, a1qa-66, a1qa-67, a1qa-70, a1qa-72, a1qa-74, a1qa-77, codoid-1, codoid-7, codoid-8, codoid-28, codoid-29, codoid-30, codoid-31, codoid-36, codoid-37, deviqa-1, deviqa-3, deviqa-5, deviqa-6, deviqa-8, deviqa-9, deviqa-10, deviqa-11, deviqa-12, deviqa-13, deviqa-16, deviqa-17, deviqa-18, deviqa-22, deviqa-23, deviqa-24, deviqa-25, deviqa-26, deviqa-28, deviqa-29, deviqa-31, deviqa-32, deviqa-34, deviqa-35, deviqa-36, deviqa-38, deviqa-39, deviqa-40, deviqa-41, deviqa-44, deviqa-45, deviqa-46, deviqa-47, deviqa-48, deviqa-49, deviqa-50, deviqa-52, qa-madness-2, qawerk-2, qawerk-3, qawerk-8, qualitest-6, testdevlab-4, testfort-6, testfort-8, testfort-9, testfort-10, testfort-13, testfort-14, testfort-15, testfort-16, testfort-17, testfort-18, testfort-19, testfort-20, testfort-22, testfort-23, testfort-25, testfort-27, testfort-28, testfort-29, testingxperts-3, testingxperts-5, testingxperts-6, testingxperts-7, testrig-1, testrig-6, testrig-9, zuci-systems-7, a1qa-4, a1qa-5, a1qa-6, a1qa-7, a1qa-9, a1qa-18, a1qa-20, a1qa-21, a1qa-22, a1qa-28, a1qa-35, a1qa-36, a1qa-60, a1qa-61, a1qa-62, a1qa-65, a1qa-66, a1qa-68, a1qa-69, abstracta-3, abstracta-4, abstracta-5, betterqa-1, bugraptors-4, bugraptors-6, codoid-1, codoid-4, codoid-7, codoid-13, codoid-14, codoid-28, codoid-31, codoid-32, codoid-35, codoid-37, codoid-38, deviqa-5, deviqa-6, deviqa-8, deviqa-9, deviqa-11, deviqa-12, deviqa-14, deviqa-19, deviqa-21, deviqa-23, deviqa-24, deviqa-25, deviqa-38, deviqa-51, impactqa-2, impactqa-3, qawerk-3, qualitest-6, qualitest-9, qualitylogic-2, testfort-2, testfort-3, testfort-6, testfort-7, testfort-11, testfort-15, testfort-16, testfort-23, testfort-26, testfort-27, testingxperts-6, testingxperts-7, testrig-8, thinksys-9

More from the blog

Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.