Skip to content
Test Automation Vendors / 263 cases

Note for buyers

Flakiness claims: how cases describe them and what to ask for

Research desk, testautomationvendors.com · published 2026-11-12 · data as of 2026-11-12 · 4 records

4
flakiness claims
1
with a before value

4 of 639 metrics mention flaky or unstable tests. 1 state a before value.

Counts come from verified records. A baseline is the before value a vendor states for a metric; quoted claims carry their grade and baseline flag.

baseline:
The measured value before the work started, against which a later value is compared. source

The claims

4 metrics use the words flaky, unstable or false failure.

  • “Flaky tests cut from 18% to 4.6%”before 18% flaky test rate after 4.6% a1qa a1qa-1 grade B, partial baseline yes method yes
  • “reduced test flakiness to under 2%.”before not stated after under 2% flakiness Codoid codoid-19 grade B, partial baseline no method no
  • “Intelligent waits instead of hard-coded sleeps cut flaky failures by 85%.”before not stated after 85% fewer flaky failures ThinkSys thinksys-2 grade A, verified baseline baseline no method no
  • “Flaky failures fell by more than 90%.”before not stated after more than 90% fewer flaky failures ThinkSys thinksys-3 grade B, partial baseline no method no

What to ask for

The flaky test rate as failures without a code change per 100 runs, before and after, over a stated number of runs.

Questions

How many cases mention flakiness?

4.

What is a flaky test?

A test that passes and fails on the same code without any change to that code.

Is flakiness scored?

Only as a metric under outcome verifiability, like any other metric.

What figure is most checkable?

A rate per runs, before and after, over a stated period.

Records used

This post reads 4 records; the newest verifiedOn date among them sets the data-as-of date.

Show record ids

a1qa-1, codoid-19, thinksys-2, thinksys-3

More from the blog

Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.