FIELD NOTES / CLOUD COOPS

The Great Egg Migration: Moving 4 Petabytes to FeedBucket

A war story about moving the largest barn in the company to object storage, and what we learned about patience.

June 3, 20259 min readBy Maurice Wingate
In shortInventory first / Move in waves / Verify everything / The surprising bit.

1. Inventory first

We found 11 petabytes of data we thought were 4. The other 7 were backups of backups, and one very confusing folder named "misc".

2. Move in waves

We started with cold data, moved to warm, and finished with hot. Each wave taught us something for the next.

3. Verify everything

Checksums on every object. If a hash did not match, we copied it again. We found 412 mismatches, all of them harmless.

4. The surprising bit

The move paid for itself in four months, and we discovered a fridge-sized server in the corner that nobody knew about.

Written by Maurice Wingate, Chief Security Rooster. Disagree? Tell us. We read every message, usually before the second coffee.