A training job that finishes without errors tells you almost nothing about whether the data was sound. Models will happily fit garbage and return a loss curve that looks fine. Validation checks are the cheap gate that sits in front of the expensive step, and the goal is simple: fail loudly before training, not quietly after deployment.

Where Validation Belongs

Run checks at two points. Validate at ingestion, so bad upstream data is caught where it enters your system and the owner can be told. Then validate again on the assembled training set, because joins, filters, and sampling can introduce problems that the raw tables never had. A check that only runs on raw data will miss a join that silently duplicated half your rows.

Schema and Structural Checks

Start with the boring ones, because they catch the most incidents.

• Expected columns and types. If a column vanished or changed from integer to string, stop.
• Row count within a plausible range. Compare with the previous run. A sudden halving or doubling deserves a human look. Choose thresholds from your own history, not from a generic rule.
• Key uniqueness. If a table should have one row per entity per day, assert that.
• Freshness. Confirm the newest timestamp is recent enough for the run to be meaningful.

Value-Level Checks

Next, look inside the columns.

• Null rates per column. A feature that was rarely missing and now is missing often usually means a broken upstream job.
• Allowed ranges and categories. Ages below zero, unknown category codes, and impossible dates are all common.
• Constant or near-constant columns. A feature that has the same value for every row carries no signal and often signals a pipeline fault.
• Duplicate rows. Exact duplicates inflate some examples and can leak between train and validation splits.

Distribution Checks

These are the checks teams skip and later regret. Compare summary statistics, such as mean, spread, and category proportions, against a reference snapshot from a known-good run. You are not testing statistical significance in a formal sense; you are looking for sharp jumps. Tolerances should come from observing how much your data normally moves week to week, then set slightly outside that.

Label and Split Checks

Labels deserve their own attention because errors there are the most damaging.

• Class balance. If the positive rate suddenly shifts, find out whether the world changed or the labeling logic did.
• Label delay. If labels arrive days after events, make sure recent rows with not-yet-known outcomes are not being treated as negatives. This is one of the most frequent silent bugs.
• Leakage. Check that no feature is derived from information that would not exist at prediction time, such as a field populated after the outcome.
• Split integrity. Confirm no entity appears in both train and validation when it should not, and that time-based splits actually respect time.

A Check Summary Table

Check	What it catches	Typical response
Schema match	Dropped or retyped columns	Block the run
Row count vs. last run	Failed or duplicated loads	Block, notify owner
Null rate per column	Broken upstream jobs	Block above threshold
Key uniqueness	Join fan-out	Block
Distribution vs. reference	Drift, logic changes	Warn or block by severity
Label rate and delay	Mislabeled recent data	Block or trim window
Train/validation overlap	Leakage	Block

A Hypothetical Example

Suppose a team retrains a churn model weekly. One week, a source team renames a status field and the ingestion job starts filling it with nulls. The training job still runs; the model simply loses an important feature and accuracy dips. With a null-rate check on that column, the pipeline would have failed on Monday morning with a message naming the field, and the fix would have been a conversation instead of a post-mortem.

Severity matters here. A null rate rising slightly might warrant a warning in the run log, while a column going from rarely null to mostly null should block.

Choosing Tools Without Overbuilding

You can write these checks as plain assertions in your pipeline code, and for a first version that is often the right answer. Open-source libraries exist for declarative expectations, and your orchestrator may support check tasks natively; look at the current documentation for what each provides. Whatever you choose, keep expectations in version control next to the pipeline, so changes to thresholds are reviewed like any other change.

Handling Failures Gracefully

A blocked run is only useful if someone sees it. Route failures to the dataset owner, include the failing check and the observed versus expected values, and make it obvious how to override with a recorded reason. Overrides without records turn into habits. Also store every check result, passing or failing, alongside the model version. When a model misbehaves months later, that history lets you ask what the data looked like on training day.

Avoid the opposite failure as well: dozens of noisy checks that fire every week teach people to ignore them. Start with a short list, tune thresholds after a few runs, and add a check only when an incident shows a gap.

Frequently Asked Questions

How many checks are enough to start? Five to ten covering schema, row count, nulls, key uniqueness, and label rate. Expand based on incidents rather than imagination.

Should validation failures always stop training? Not always. Block on structural and leakage problems, and warn on mild distribution shifts, so the team can review without halting everything.

Where do I get reference statistics? From a previous accepted training run. Save summary statistics each time a model is approved and compare new data against that stored snapshot.

Do these checks apply to unstructured data like text or images? Yes in adapted form: file counts, corrupt or empty files, length and size distributions, language or format mix, and duplicate detection.

Can validation replace monitoring in production? No. Training-time checks and production monitoring cover different moments, and you need both.

Next Step

Open your current training pipeline and write down what happens if a key column is entirely null tomorrow. If the honest answer is that training proceeds, add a null-rate check on your three most important features today.