MLOpsThe MLOps lifecycle · MLflow 3.17 · DVC 3.67 · FastAPI · Docker · GitHub Actions · Evidently · Python 3.11+
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
50 small wins to finish your pathNext lesson →

Why git cannot hold data

Git is a version control system that keeps the whole content of every version of every file it tracks, which is what makes it the wrong home for a dataset.

Last updated: 19 Oct, 2026 · MLOps

The tracking part closed with A remote tracking server, which gave each training run a permanent record of its parameters and its metrics. That record names a data file; it does not hold one. This part adds the missing half: a way to version the data a run read, and the pipeline that turned it into a model. The video reaches for DVC at the same point and describes it as version control applied in two directions, one over the data and one over the pipeline.

Storing a changed file the git way

Git stores content, not edits. When a tracked file changes, git writes a new object holding the whole new content and leaves the old object alone. Later it packs objects together, looking for files of similar name and size and keeping the difference between them, with the newest version stored whole.

  • A text file packs well. Two versions of a Python file share nearly every line, so the difference between them is a few bytes.
  • A regenerated data file does not. A CSV written out again, a joblib model, a parquet file: the bytes move around the whole file, so there is little shared structure to keep and each version costs close to its full size.
  • History travels. A clone copies every object ever committed, so a 200 MB file committed ten times is carried by everyone who clones the repository, for as long as it exists.
  • The host sets a ceiling. GitHub warns above 50 MiB per file, blocks anything above 100 MiB, and asks that a repository stay under 1 GB.

What a model project needs and git does not give

  • Which data made this model. A commit pins the code and the config. Nothing in it pins the 56 KB of rows the model was fitted on.
  • A way to hand the data over. A teammate clones the code and has no dataset, so the file travels by chat message or shared drive and the two copies drift apart.
  • Storage that fits the file. Data belongs where storage is cheap and large. Code belongs in git. One repository cannot be both.

Keeping the pointer in git and the content outside

DVC splits the two. The content of the file moves into a cache folder DVC owns, and a small text file takes its place in the working folder. That text file, the pointer, names the content by its md5 hash, its size and its path, and it is what git commits. The hash is the link: one hash names one exact version of the bytes. A copy of the cache goes to storage you choose, which DVC calls a remote, so whoever clones the repository has the pointer and can fetch the content it names.

  • The working folder holds the real file, the one your code opens.
  • The cache, under .dvc/cache, holds one copy of each distinct content, filed under its hash.
  • The remote holds the same content again, somewhere a teammate or a build machine can reach.
  • Git holds the pointer, the pipeline files and the code, and stays small.

Git vs DVC for a dataset

GitDVC
What a commit carriesThe whole file contentAn 89-byte pointer naming the md5
Where the bytes liveInside .git, in every cloneIn .dvc/cache and in a remote you choose
Cost of a new versionClose to the file size, for everyoneOne copy in the cache, one in the remote
Size limitA host blocks a large fileWhatever storage you pay for
SuitsSource, config, small text filesDatasets, models, large binaries

Where versioning data pays off

  • Reproducing last quarter's model. Check out the old commit, pull the data its pointer names, and the inputs are the ones that run used.
  • Onboarding somebody. They clone the repository and fetch the data with one command instead of asking where the file is.
  • A retraining job on a build machine. The job has no shared drive, so it fetches the data from the remote the same way a person does.
Watch out. DVC does not replace git, it rides on it. Every DVC command expects a git repository around it, and a pointer is worth nothing to a teammate until the content behind it has been sent to a remote. A repository whose data was never pushed hands the next person a hash and nothing to match it to.
Try it yourself
  • Run du -sh .git in a project that has a dataset committed, then compare the number with the size of the data folder itself.
  • List the files your project writes and mark each one as text that changes line by line, or as a file that is rewritten whole. The second group is the group DVC is for.
  • Work out what a 50 MB export committed once a week for a year would add to every clone of your repository.
PreviousA remote tracking server

Slow is fine. Stopping is the only problem.