FIELD NOTES / DATA AND EGGS

Data Quality Is Candle Light

A short history of candling, and how it maps to modern data testing.

March 3, 20267 min readBy Dolores Peckham
In shortOld craft, new data / Test early / Test what matters / Alert on trends.

1. Old craft, new data

Farmers shine a light through the egg to find cracks. We shine tests through the data to find nulls and duplicates.

2. Test early

Catch problems at ingestion, not in the dashboard. The earlier you candle, the cheaper the fix.

3. Test what matters

Do not write 500 checks. Write 20 that would make you panic if they failed.

4. Alert on trends

A table that slowly worsens is more dangerous than one that fails loudly. Track failure rates over time.

Written by Dolores Peckham, Head of Data and AI. Disagree? Tell us. We read every message, usually before the second coffee.