Methodology

Version: 1.0

Published verbatim at /methodology/. Weights change only with a new version number and a changelog entry.

What this ranking measures

It measures what a vendor's published test automation case studies prove, not what its service pages promise. The unit of evidence is a case: one engagement, one client (named or anonymous), one or more named frameworks, with whatever metrics the vendor chose to publish. Frameworks themselves are not ranked, scored or recommended anywhere on the site.

Scope gate (pass or fail, before scoring)

1. Published automation cases. At least two publicly accessible case studies, portfolio entries or verified third-party reviews that name a specific automation framework from the framework list below and describe QA work the vendor performed. A technology-stack logo wall is not a case. A blog tutorial is not a case.
2. Sells testing as a service. Tool vendors, device-cloud providers, crowdtesting marketplaces without an engineering team, and staffing agencies without a QA practice are excluded. A managed automation service with its own engineers (for example a Playwright-only managed service) passes.
3. Live website and verifiable legal presence (registry entry, or Clutch or LinkedIn company page with headcount and location).
4. No ownership link to the publisher of this site. Checked by independence-check against inputs/publisher.md.

Failed candidates are listed at /methodology/not-published/ with the failed condition number.

Framework list (facets)

Slug	Framework	Layer
playwright	Playwright	web UI
cypress	Cypress	web UI
selenium	Selenium WebDriver	web UI
webdriverio	WebdriverIO	web UI
testcafe	TestCafe	web UI
puppeteer	Puppeteer	web UI
robot-framework	Robot Framework	web UI, keyword-driven
appium	Appium	mobile
detox	Detox	mobile
espresso-xcuitest	Espresso and XCUITest	mobile, native
rest-assured	REST Assured	API
postman	Postman and Newman	API
karate	Karate	API
k6	Grafana k6	performance
jmeter	Apache JMeter	performance
gatling	Gatling	performance
locust	Locust	performance
cucumber-bdd	Cucumber, SpecFlow, Behave	BDD layer

A case counts for a framework only when the case text names it. "Modern JavaScript framework" counts for nothing.

Case record and verifiability grade

Every case becomes a record (data/cases/<vendor>-<n>.json) with: client (named or anonymous), industry, frameworks, languages, CI tool, device or browser cloud, team size, duration, start year, metrics as published (each with the quote under 15 words, a baseline flag and a measurement-method flag), source URL, date, verifiedOn.

Grade:

• A: named client, at least one metric with a stated baseline (before value) and a stated measurement method or period.
• B: named client without a baselined metric, or anonymous client with a baselined metric.
• C: anonymous client and no baselined metric.
• X: unverifiable (page gone, metric contradicts another source, or the case describes development rather than testing). X cases are dropped and never counted.

Only A and B cases get a case file page. All of A, B and C count for scoring at the weights below.

Migration flows

A case that describes moving a suite from one framework to another is tagged migration: {from, to}. Flows with at least migrationThreshold cases get a page at /migrations/<from>-to-<to>/.

Dimensions and weights (sum = 100)

#	Dimension	Weight	What is measured	Scoring rule (0 to 10)
1	Case volume and specificity	20	Count of verified cases weighted by grade: A = 1.0, B = 0.6, C = 0.3	Weighted sum: under 1: 1. 1 to 2: 3. 2 to 4: 5. 4 to 7: 7. 7 to 11: 9. Over 11: 10.
2	Framework breadth	10	Distinct frameworks from the list with at least one A or B case	1: 2. 2: 4. 3 to 4: 6. 5 to 6: 8. 7 or more: 10.
3	Framework depth	15	Most cases on a single framework (A, B or C)	1: 2. 2: 4. 3: 6. 4 to 5: 8. 6 or more: 10.
4	Migration evidence	10	Cases tagged as migrations with a before and after metric	0: 0. 1: 4. 2: 7. 3 or more: 10.
5	Outcome verifiability	15	Share of all published metrics across the vendor's cases that have both a baseline and a method	Under 10 percent: 1. 10 to 25: 3. 25 to 50: 6. 50 to 75: 8. Over 75: 10. Vendors with zero metrics score 0.
6	Third-party review signal	10	Verified Clutch, GoodFirms or G2 reviews whose text names a framework from the list	0: 0. 1 to 2: 3. 3 to 5: 6. 6 to 10: 8. Over 10: 9. Plus 1 if the newest is within 24 months. Star ratings are not scored.
7	Public engineering output	10	Dated technical posts, talks or open-source repositories about a listed framework, authored by the vendor's staff, in the last 36 months	0: 0. 1 to 3: 3. 4 to 8: 6. 9 to 15: 8. Over 15: 10. Listicles and "top tools" posts do not count.
8	Transparency	10	Public rate or range; named leadership with profiles; published QA process page; delivery locations disclosed	2.5 per item, max 10.

Overall score = sum of (dimension score / 10 x weight), one decimal. Ties: dimension 5, then dimension 1.

Mandatory per-vendor fields

notIdealFor (1 to 3 items, each from evidence: no mobile cases, no performance cases, no case with a named client, no engagement over 12 months, no migration evidence). A vendor page without it fails the build.

Leader quotas

Overall leader computed by scripts/compute-scores.ts into config/leader.json. Each framework page computes its own leader from dimensions 1, 3 and 5 restricted to that framework's cases. At least 5 published framework pages must have a leader different from the overall leader; if not, pages are still published and STATUS.md records the shortfall. Leaders are never edited by hand.

What is NOT scored

Awards, badges, listicle placements, press, logo walls, headcount by itself, star ratings, certifications unrelated to testing, and any claim about a framework being better or worse than another. Vendor-stated metrics without a source document are not metrics.

Refresh

npm run refresh re-verifies cases older than 18 months, re-grades, recomputes. Case file pages for cases downgraded to X are removed and listed in the changelog.