File Pipelines in Python
Curate robot demonstrations, inspect saved results and revise a training selection. The examples below also cover MCAP logs, WebDataset samples, images and PyTorch.
Robot demonstrations
Install datachain[video] in an empty project directory. Save the following as
build_episodes.py and run python build_episodes.py:
The public DROID sample contains 100 LeRobot v3 episodes at 15 fps. The reader attaches
all three cameras as VideoFragment values: the original file identity and the episode's
start/end times in the shared MP4. Only the wrist camera is decoded for this score.
The score is mean absolute RGB pixel change between the first frame and one second later,
or the last frame for shorter episodes. It measures visual change, not grasp success.
A reference run selected 63 episodes. The manifest keeps the scores, metadata and camera references; creating it does not repack videos. Accessing pixels reads video bytes. Source-store versioning and retention determine how long the original contents remain available.
Inspect and revise a saved selection
Saved datasets have versions, schemas and recorded dependencies. Run named scripts to capture the code that produced each version. A fresh process can inspect the saved result:
Query a pinned version or change the selection rule using the saved scores:
This creates a new selection version without decoding the cameras again. The previous version
remains readable. To hand the selection to another project, share training_manifest.parquet
and access to the source storage. Load a selected camera frame in that project:
Dataset versions preserve rows and source identities, not credentials or a copy of the videos. Reusing computed scores is separate from reading the selected contents for training.
Select successful robot demonstrations
DROID's success labels live in its per-frame data, separately from the episode metadata. For a label-based selection, read the needed data columns and aggregate by episode:
This shard contains 1,074 episodes, 901 marked successful. It is a separate dataset from
droid_100; join labels to camera episodes from the same dataset on episode_index.
The copied meta/ totals describe a larger dataset, so count the rows actually present.
Public images
Run this in an empty project directory. Read image dimensions from a public 300-image prefix, save a training selection and export a manifest. No credentials or repository checkout are needed.
Save this as build_images.py:
Run it with python build_images.py. A reference run saved 300 rows in pet_images and
291 in training_images:
The reader returns typed columns for SQL filtering. It reads file headers over the network; buffering
may read a small image in full. Saved results stay in .datachain/ after Python exits.
Feed PyTorch
First run the public image example above. For its training handoff, install datachain[torch].
This also works in a new Python process:
This step downloads image contents. A new session needs the public source's anon=True access setting;
saved dataset versions record rows and file identities, not storage credentials.
Dataset versions · PyTorch integration
For video (datachain[video]), VideoFile.get_frames() yields VideoFrame objects with timestamps.
Catalog MCAP recordings
dc.read_mcap() indexes recording summaries without scanning all messages.
This standalone local demo makes two tiny recordings, then saves the one containing joint states.
Run it in a separate project directory after installing datachain:
Use a cloud URI instead of recordings/*.mcap for your own logs. The reader requires complete
MCAP summaries. Topic presence does not establish sensor validity or synchronization.
For message processing, dc.read_mcap(path, topics="/joint_states", duration=10) produces
McapSlice windows. Apply your Python reader with .map() to compute features from their messages.
duration is in seconds; message timestamps and slice boundaries are in nanoseconds.
WebDataset samples
Index a public uncompressed TAR shard once, then save caption-based selections from its metadata.
WDSLaion gives the JSON metadata typed columns; sample images remain file references:
A new process can select cats from dc.read_dataset("image_samples") without scanning the TAR again.
The initial index reads the archive. Reading selected images fetches their bytes; it is separate from
metadata filtering. For many training epochs, repacking a small subset can reduce repeated reads.
See measured workflow costs.
Inventory across clouds
Configure access for each source, then use the same API across stores. This template lists current objects without downloading their contents or saving a dataset:
Keep the source with the path: identical paths can exist in different stores. Each iteration lists again;
use read_storage and save when you want to reuse the inventory.
Update and resume
- Captured scripts do not bundle imported modules, environments or model weights. Source-store versioning and retention determine whether the original file contents remain available.
- For compatible file-delta pipelines, start with
read_storage(..., update=True, delta=True)and rerun to update changed files and remove deleted ones. See supported operations and migration. - After a model or preprocessing change, rebuild with
DATACHAIN_IGNORE_CHECKPOINTS=1andupdate=True, delta=False, then save a new version under the same name. Use another name when keeping parallel model variants; earlier versions remain readable. - After interruption, rerun the same script to reuse written checkpoints. Work not yet checkpointed may repeat.