How do you version datasets and models for reproducibility?

The outcome you should expect

The honest promise of dataset and model versioning is not that every experiment becomes magically repeatable — it is that the *cost of asking "what produced this?"* drops from days to minutes. That single change reshapes how a data team spends its week.

Before versioning, the typical failure looks like this: a model in production starts drifting, someone asks which training snapshot it came from, and the answer is a Slack thread, a notebook with an absolute path to /mnt/data/final_v3_REAL.parquet, and a shrug. Reconstructing that snapshot takes an engineer two to five days of archaeology, and even then confidence is partial. After versioning, the same question is a hash lookup: the model registry entry carries a data hash, a Git SHA, and a locked environment file. Someone runs one command, gets the exact inputs, and reruns training.

Concretely, expect these outcomes within one to two quarters of adoption:

Replay time collapses. A previously "unreproducible" run goes from multi-day reconstruction to a checkout plus a pull — typically minutes plus however long training itself takes. The training time doesn't shrink; the *setup* time does, and setup was almost always the expensive part.

Onboarding gets faster. A new data scientist joining a versioned repo clones, pulls the pinned dataset, and reproduces last quarter's baseline on day one instead of week three. This is the most consistently reported benefit and the easiest to observe internally — track it as "days to first reproduced baseline."

Audit questions stop being projects. In regulated settings — lending, insurance underwriting, clinical decision support — a model risk review will ask for the training data, the transformation code, and evidence they were paired. With lineage in place that's a report you export. Without it, it's a staffed workstream.

Silent regressions surface earlier. When each pipeline stage is content-hashed, a changed input invalidates downstream stages automatically. You stop shipping models trained on a partially-refreshed feature table because the stale stage no longer silently passes through.

Storage bills go up, engineering hours go down. Versioning is a deliberate trade: you pay in object storage and a little cognitive overhead to stop paying in reconstruction labor. For most teams the swap is strongly favorable, but it *is* a swap and you should say so out loud when you propose it.

What you should *not* expect: versioning does not make a badly-specified experiment meaningful, does not fix nondeterministic GPU kernels on its own, and does not eliminate the discipline of writing down what you were testing. It removes an entire class of excuse, not the need for rigor.

What drives that outcome

Reproducibility is not one control. It is four independent axes that all have to be pinned simultaneously, and a team that pins three of four still gets non-reproducible runs — usually blaming the wrong axis when it happens.

Axis one: data identity. The core mechanic in every credible tool is content addressing. DVC writes a small .dvc metafile into Git containing an MD5-style hash of the file or directory; the bytes themselves live in a remote — S3, GCS, Azure Blob, SSH, or a shared NFS mount. Git stays small (kilobytes of pointers instead of gigabytes of Parquet), while dvc pull reconstitutes the exact bytes. lakeFS takes the same idea to the bucket level: it gives you Git-like branch, commit, and merge semantics over an existing object store without copying objects, using a metadata layer that maps branch state to physical keys. Delta Lake solves it at the table level — a transaction log over Parquet files that lets you query VERSION AS OF 42 or TIMESTAMP AS OF '2026-05-01' and get exactly the rows that existed then.

Pick the layer that matches how your data actually moves. File-shaped assets (images, audio, model weights, CSVs) fit DVC. Bucket-shaped, many-writer data lakes fit lakeFS. Table-shaped analytics data already in Spark fits Delta Lake. Mixing two is normal; mixing four is a maintenance tax.

Axis two: code identity. This is the axis everyone assumes is solved because Git exists — and it's the one that quietly breaks. The training script is versioned, but the SQL that built the feature table lives in a BI tool, the preprocessing lives in a notebook nobody committed, and the feature definitions live in a spreadsheet. Reproducibility requires that *every transformation* between raw bytes and model input be in version control. If a step happens in a UI, that step is unversioned, full stop.

Axis three: environment identity. A model trained under one library version and reloaded under another can fail to deserialize or, far worse, deserialize and silently score differently. Pin with a lockfile — requirements.txt with exact pins, a Poetry or uv lock, a conda environment export, or a container digest (not a floating :latest tag). MLflow addresses this by writing a conda.yaml or python_env.yaml next to the logged model along with a signature describing expected input and output schemas; Pachyderm addresses it by executing every pipeline stage inside a declared container image, which is the strongest version of this guarantee because the OS-level libraries are pinned too.

Axis four: execution identity. Random seeds, data shuffling order, GPU nondeterminism, and parallel reduction order all inject variance that no amount of data versioning removes. Set and log seeds for Python, NumPy, and your framework; enable deterministic kernel flags where your framework offers them; accept that some GPU operations remain nondeterministic and record a tolerance instead of pretending you have bitwise equality.

The registry entry at the bottom is the load-bearing piece. A model file sitting in a bucket with no binding to its four inputs is an orphan — you can deploy it, but you cannot explain it. That binding is what turns four separate version systems into one reproducibility guarantee, and it is exactly what MLflow's Model Registry, Neptune's model registry, and W&B Artifacts each provide in slightly different shapes.

Benchmarks and realistic ranges

Anchor expectations with ranges you can sanity-check against your own environment rather than vendor claims.

Setup effort. DVC on an existing Git repo is a couple of hours for someone comfortable with Git: dvc init, configure a remote, dvc add your data directory, commit the pointer. MLflow tracking against a local or shared backend is a similar afternoon. The heavier options are genuinely heavier — Pachyderm assumes Kubernetes, containerized pipeline stages, and a declarative pipeline spec, so plan on a week or two before the first real pipeline runs cleanly, and only take that on if you have DevOps capacity. Delta Lake is near-free if you already run Spark and a real project if you don't.

Repository size. The whole point of pointer-based versioning is that Git stays small. A .dvc file is a few hundred bytes regardless of whether it points at 50 MB or 500 GB. If your Git repo is growing by gigabytes, someone has committed data directly — that's the signal to audit. Practical guardrail: add a pre-commit hook that rejects any file over ~10 MB unless explicitly allowlisted.

Storage growth. This is where teams get surprised. Content-addressed storage deduplicates identical files, so re-versioning an unchanged 100 GB corpus costs almost nothing. But a *modified* large file usually stores as a whole new object, not a delta — change one column in a monolithic 40 GB Parquet file and you may store another 40 GB. The fix is structural: partition large tables so a change touches a few partitions instead of the whole file, and version the *processed* artifact rather than every intermediate. Teams that version everything at every stage routinely see storage multiply several times over within a year.

Retention. Set an explicit policy rather than discovering costs later. A common shape: keep every version referenced by a registered model or a published result indefinitely; keep unreferenced experiment branches 30 to 90 days; garbage-collect the rest. Delta Lake's VACUUM removes files no longer needed by the retention window; lakeFS has garbage collection for unreferenced objects; DVC has dvc gc with flags controlling which workspaces and branches to preserve. Every one of these is destructive — run them with a dry-run flag first, and never point them at a remote shared with an unversioned system.

Egress and transfer. The cost people forget is not storage but movement. Pulling a 200 GB dataset from cloud storage to on-prem compute repeatedly gets expensive fast. Co-locate compute with the bucket, cache pulled data on the training host, and use shallow or partial pulls (dvc pull path/to/subset) so a developer who needs 3 GB doesn't drag down 200.

Model registry footprint. Model artifacts are small compared to data — typically megabytes to low gigabytes — so registries rarely dominate cost. Keep every promoted version; prune failed experiment artifacts aggressively.

Where the revenue argument lives. If you need to justify this work to someone holding a budget, don't lead with reproducibility as a virtue. Lead with the revenue-facing consequence: a scoring model that can't be reproduced can't be safely retrained, and a model that can't be safely retrained silently decays until it starts misrouting leads, mispricing quotes, or misranking accounts. Every one of those is a revenue leak with no alarm attached to it. Reproducibility is what makes a fast, confident retrain possible — which is the actual mechanism by which model quality, and the revenue that depends on it, holds up over time.

Risks, edge cases, and failure modes

The unversioned upstream. Your pipeline is immaculate and it reads from a production table that a nightly job overwrites in place. You have versioned everything downstream of a moving target. Fix by snapshotting at ingest — write an immutable dated copy to object storage, version *that*, and never read production directly from training code. This is the single most common reproducibility hole and it lives outside whatever tool you adopted.

Deleted or mutated remote objects. A .dvc pointer is only as good as the bytes it points to. If someone applies an S3 lifecycle rule to the DVC remote, or a teammate "cleans up" a bucket, your pointers dangle and Git looks perfectly healthy. Enable bucket versioning and deletion protection on any bucket serving as a versioning remote, and restrict delete permissions to a role humans don't hold day to day.

Personal data and the right to erasure. Immutable versioned history collides directly with deletion obligations under privacy regimes. If a subject requests erasure and their record is baked into forty historical dataset versions, you have a real problem. Mitigations: keep raw personal data out of versioned artifacts entirely and version *derived, pseudonymized* features instead; or keep a short retention window on raw snapshots with a documented purge path; or hold identifiers in a separate governed store that versioned data joins to by key. Decide this before you have four hundred versions, not after.

Nondeterminism you cannot version away. Even with data, code, and environment pinned, GPU floating-point reductions, multi-threaded data loading order, and library-level parallelism can produce runs that differ in the fifth decimal place. Sometimes that compounds — a slightly different early gradient leads to a materially different final model. Handle it by defining a reproducibility tolerance in advance ("held-out AUC within ±0.002") and treating exact-match reproduction as a goal only where you've enabled deterministic execution and accepted the throughput cost.

Registry as a lie. A model registry entry that records only the artifact and a metric is worse than no registry, because it *looks* like governance. If the entry doesn't carry the data hash, code SHA, and environment reference, you've built a filing cabinet, not a lineage system. Enforce it in CI: reject registration when required lineage fields are absent.

Storage cost blowup. Covered above, but the failure mode is social — nobody owns the bill, versions accumulate for eighteen months, finance escalates, and the reflex is to purge history wholesale, which destroys exactly the audit trail you built. Put a named owner and a written retention policy in place at adoption time.

Tool sprawl. DVC for files, lakeFS for the lake, Delta for tables, MLflow for models, W&B for experiments, plus a homegrown script — each individually reasonable, collectively a system nobody can explain. Two tools is usually the right number: one for data, one for models and experiments. Adding a third requires justifying which existing tool it replaces.

Notebooks. Notebooks execute out of order, embed absolute paths, and carry stale state in their outputs. Treat them as exploration surfaces and require that anything reproducible be moved into a versioned script or pipeline stage before it counts. Committed .ipynb outputs also bloat diffs — strip them on commit.

Large-file merge conflicts. Two branches that both modify a versioned binary produce a conflict Git cannot resolve. There's no merge algorithm for a 40 GB Parquet file. Establish a convention — a single owning branch for dataset mutations, or generation-based naming — instead of expecting the tool to arbitrate.

A practical rollout plan

Roll this out narrow and prove it, rather than announcing a platform. The sequence below front-loads the cheap, high-signal steps.

Week one — pick the pilot and instrument nothing else. Choose one model that already matters: something in production, with a known owner, ideally one that has already embarrassed someone with an unanswerable lineage question. Resist the urge to start with the largest dataset. You want a project where the whole loop finishes in an afternoon so you can iterate on the process rather than waiting on transfers.

Week one — stop the bleeding upstream. Before any tool, add an immutable ingest snapshot. Write raw inputs to a dated, write-once prefix in object storage. Everything downstream reads only from snapshots. This alone fixes the majority of real-world irreproducibility and requires no new vendor.

Week two — version the data. dvc init in the existing repo, point the remote at your bucket, dvc add the processed dataset, commit the pointer alongside the code that produced it. Verify by cloning fresh into a scratch directory and pulling — if the clone-and-pull doesn't reconstitute the dataset for a colleague who wasn't involved, you don't have versioning yet, you have a local cache.

Week two — pin the environment. Freeze exact dependency versions into a lockfile, or build and push a container image referenced by digest. Record the reference in the run config, not in a README.

Week three — define the pipeline as stages. Express preprocessing and training as declarative stages with explicit inputs and outputs (dvc.yaml, or the equivalent in whichever tool). The payoff is automatic invalidation: change a preprocessing parameter and only affected stages rerun. This is also the step that finally forces every hidden transformation into version control, because a stage that doesn't declare its inputs breaks visibly.

Week three to four — register models with lineage. Log runs to MLflow (or W&B, or Neptune — the choice matters less than the discipline) and register the trained model with the data hash, Git SHA, and environment reference attached as required fields. Add stage transitions your org actually uses: staging, production, archived.

Week four — automate the proof. Wire CI so that opening a pull request touching the pipeline triggers a run and posts metrics and lineage back on the PR. CML exists precisely for this pattern with GitHub Actions or GitLab CI, and it converts reproducibility from a claim into a check that fails loudly.

Week five — write the retention policy and the runbook. One page: what's kept forever, what expires, who runs garbage collection, how to restore a model from a registry entry. Then rehearse a restore from a cold start, on a machine that has never seen the project.

Adjacent surfaces worth pulling in once the pilot holds. A feature store makes the training/serving skew problem tractable by serving the same feature definitions to both paths — versioning feature definitions is the natural next step after versioning datasets. Analytics engineering has already solved a sibling problem: dbt versions transformation logic in Git and tests it in CI, and a team that runs dbt has half the muscle memory this requires. And on the revenue side of the house, the same lineage question shows up in reporting — when a board deck number can't be traced to a query and a snapshot, that's the identical failure wearing different clothes. The techniques transfer directly.

Related questions

Do I need both DVC and MLflow, or does one cover it?

They cover different axes. DVC versions data and pipeline stages; MLflow tracks runs and registers models. Many teams run both, letting MLflow record the DVC data hash on each run. Use one only if you genuinely have just one problem.

How do I version a dataset that's too large to snapshot?

Version at the table or catalog layer instead of the file layer. Delta Lake, Iceberg, and Hudi record a transaction log so you can query the table as of a version or timestamp without duplicating bytes. Pair the version number with your run metadata.

What's the minimum viable setup for a solo practitioner?

Git for code, DVC pointing at a personal S3 or GCS bucket for data, a lockfile for the environment, and a text file recording seeds. That's an afternoon of setup and it covers all four reproducibility axes.

How does versioning interact with data privacy deletion requests?

Poorly, unless you plan for it. Keep raw personal data out of versioned artifacts, version pseudonymized derivatives instead, and hold identifiers in a separately governed store. Otherwise erasure requests force you to rewrite immutable history.

Can I reproduce a model without reproducing the training run?

You can reload a registered model artifact and get identical inference given the same environment. But you cannot verify *how* it was made, retrain it safely, or defend it in review. Artifact preservation and reproducibility are different guarantees.

FAQ

Does DVC require Git?

Yes. DVC stores its metafiles inside a Git repository and relies on Git for branching, history, and commit identity. If your workflow genuinely cannot use Git — for example a data lake with many concurrent writers and no repo — lakeFS provides branch-and-merge semantics directly over object storage instead.

What actually gets stored in Git versus the remote?

Git holds small text metafiles containing content hashes plus your code, pipeline definitions, parameters, and metrics files. The remote holds the actual bytes, addressed by hash. This is why a repo tracking terabytes can still clone in seconds — you pull pointers first and fetch only the data you need.

How do I handle nondeterministic training runs?

Log seeds for every random source, enable your framework's deterministic execution flags where available, and record hardware and driver details in the run metadata. Accept that some GPU operations remain nondeterministic; define an acceptable metric tolerance up front rather than chasing bitwise equality you may not be able to get.

Should I version raw data, processed data, or both?

Version the immutable raw snapshot and the processed artifact that feeds training. Skip most intermediates — they're regenerable from the raw snapshot plus versioned code, and versioning every intermediate is the fastest route to a storage bill nobody wants to defend. If an intermediate takes many hours to compute, version that one too.

How much extra storage should we budget?

It depends entirely on churn, not on dataset size. Unchanged data costs nothing extra because identical content deduplicates. Frequently-modified large monolithic files are the expensive case, since a small edit typically stores a whole new object. Partition large files, version processed artifacts rather than everything, and set a retention window with garbage collection from day one.

What's the first thing to fix if reproducibility is already broken?

Immutable ingest snapshots. Most broken pipelines read from a source that gets overwritten, which makes every downstream version meaningless. Write raw inputs to dated write-once storage, point all training at those snapshots, and you'll have fixed the largest hole before adopting any tool at all.

Sources

• DVC documentation
• MLflow Model Registry
• Delta Lake time travel
• lakeFS documentation
• Weights & Biases Artifacts
• Hugging Face Hub repositories
• CML (Continuous Machine Learning)
• Pachyderm documentation
• Apache Iceberg documentation
• PyTorch reproducibility notes

Related on PULSE

• [How do you monitor machine learning models in production?](/knowledge/ai0384)
• [How do you build a feature store for machine learning?](/knowledge/ai0385)
• [How do you set up CI/CD for machine learning pipelines?](/knowledge/ai0386)
• [How do you detect and handle data drift?](/knowledge/ai0387)
• [How do you document data lineage across a warehouse?](/knowledge/ai0388)