Data: data lake, versions, feature store

Why a separate store

A production database like Postgres stores current state: orders, customers, inventory. Training a model directly on it is impractical. Heavy queries over millions of rows slow down the service, and old values disappear after an UPDATE.

So data for analytics and ML is copied to a separate place. There are two kinds of such places.

A data warehouse is an analytical database where data arrives already cleaned, into tables with a schema defined in advance. Examples: ClickHouse, Greenplum, BigQuery, Snowflake. You write SQL queries against it for reports.

A data lake is object storage where data is dropped in its raw form: table dumps, logs, images, JSON. It's usually S3 or MinIO, an S3-compatible store you run on your own servers. The schema is applied at read time, not at write time.

Parquet

Tables in a data lake are usually stored in Parquet format. It's a columnar format: values from the same column sit next to each other.

This is convenient for training. The churn model needs 10 columns out of 200, and Parquet lets you read just those 10. A CSV would have to be read in full.

Paths in a data lake can look like this:

s3://lake/raw/orders/date=2026-10-19/part-0001.parquet
s3://lake/raw/orders/date=2026-10-20/part-0001.parquet

The date= folder is called a partition. A query for the last week reads only seven such folders instead of the whole history.

Iceberg and the lakehouse

A data lake made of plain Parquet files has a problem with updates. A table consists of hundreds of files. While an export is rewriting some of them, a reader can see half old files and half new ones. You also can't look at the table as it was a week ago.

Table formats like Apache Iceberg and Delta Lake solve this. Alongside the Parquet files, they keep metadata: which files make up each version of the table. A table change becomes a transaction: a reader sees either the old version or the new one. This gives you time travel, reading the table as it looked at a given point in time.

A data lake with a table format is called a lakehouse. Queries against it run through engines like Spark and Trino. For ML, time travel is useful because it lets you retrain a model on exactly the data it was first trained on.

Dataset versions

A run's metadata needs to include the data version. For a table in Iceberg, that's its snapshot. For datasets that are plain files, there are dedicated tools.

DVC (Data Version Control) works alongside Git. The actual data file goes to S3, while a small pointer file gets committed to Git:

dvc init
dvc add data/train.csv              # creates the pointer data/train.csv.dvc
git add data/train.csv.dvc data/.gitignore
git commit -m "dataset v1"
dvc remote add -d storage s3://ml-data/dvc
dvc push                            # pushes the actual file to S3

To get a particular version of the dataset, you switch to its commit and run dvc pull.

lakeFS adds Git-like branches and commits to an entire S3 bucket, without copying data when a branch is created. On a branch you can prepare a new dataset, check it, and merge it into main.

In November 2025, lakeFS took over DVC's development. According to their announcement, DVC remains a separate tool for small datasets, while lakeFS is built for large ones.

Feature store

Training-serving skew from the first lesson shows up when features for training and production are computed by different code. A feature store keeps features in one place for both cases.

• The offline store serves historical feature values for training.
• The online store serves fresh values quickly for a production request.

One open source feature store is called Feast. A service gets a customer's features from it like this:

from feast import FeatureStore

store = FeatureStore(repo_path=".")
features = store.get_online_features(
    features=["user_stats:orders_30d", "user_stats:avg_check"],
    entity_rows=[{"user_id": 42}],   # which customer
).to_dict()

The Google paper lists the feature store as an optional component of level 1. Without it, you reach the same goal with a shared function that computes features and is called both during training and in the service.

What to remember

Data for ML is copied from the production database into a data warehouse or a data lake. A data lake usually sits on S3 or MinIO, tables in it are stored in Parquet and split into partitions.

Iceberg and Delta Lake add transactions and time travel to a data lake. Such a data lake is called a lakehouse.

Dataset versions are tracked with DVC or lakeFS. A feature store serves the same features to training and production and protects against training-serving skew.