Data Cleaning

What is data cleaning?

Real-world data is never tidy. Spreadsheets come with duplicate rows, empty cells, dates in five different formats and numbers in mismatched units. Data cleaning is the pre-training “washing the rice” step — picking out the stones, grit and bad grains and keeping only what's usable.

What mess does it clean up?

Duplicates
The same sample appearing multiple times makes a model over-memorize certain patterns, which biases it.
Missing and outlier values
Empty fields or absurd numbers — like an age of 200 — need to be filled in or dropped.
Format and encoding issues
Dates written every which way, mixed full-width and half-width characters, mojibake — all of it has to be standardized.
Noise and mislabeled data
Wrong labels or irrelevant junk leaking in will steer the model in the wrong direction.

Why is it underrated?

Plenty of people worship “a bigger model” and ignore “cleaner data”. In practice, a strong model can't rescue bad data. Seasoned teams will tell you they often spend more time cleaning than tuning the model — and the payoff is more reliable.

How cleaning usually goes

Automated deduplication and format normalization first, then rules plus human spot-checks for the nasty cases, and finally some stats to confirm the distribution still makes sense. More teams now let large models help with the scrubbing, but people still set the tricky boundaries.

Bottom line: data cleaning keeps junk out of training. The cleaner the data, the more dependable the model.