We're publishing SCARCE, our first public technical report - a placebo-controlled, cross-domain benchmark measuring exactly where and how much synthetic data helps on the hardest, most data-starved class in each dataset.
Synthetic data is easy to claim and hard to prove. We wanted to answer one question under conditions we could not fool ourselves with: does adding Synthgen synthetic data to a real dataset actually make a model better on the cases that matter most - the rare, hard, data-starved class where real examples are scarce and models fail?
SCARCE - Synthetic-data Cross-domain Assessment of Rare-Class Efficiency - is our answer, and today we are publishing it in full.
On the single hardest, most starved class in each dataset, we compared a model trained on real data alone against the same model trained on the same real data plus Synthgen synthetic data. Nothing else changed. We report per-class accuracy only, never an averaged or whole-dataset number.
The benchmark spans 8 industries and three imaging types, from everyday color photos to grayscale surface scans to X-ray.
Adding Synthgen synthetic data lifted accuracy on the hardest class by +9 to +35 percentage points, on every dataset we tested. A few of the results from the report:
Every one of these is paired against a real-only baseline and statistically significant.
The less real data you have, the more synthetic data is worth. On MVTec Pill, two labelled examples per class with Synthgen reached the same accuracy as five real labelled examples alone - the same result on 60% fewer real labels. The gap is widest exactly when your real data is scarcest.
We think honesty is the point of a benchmark, so the report is explicit about its scope:
We make no claims about easy classes, overall dataset accuracy, or any setting we did not measure. The gains say what they say, and nothing more.
Read the full SCARCE technical report (PDF)
The headline numbers and the full per-class breakdown also live on our research page.
