Datachain

Data Types

Data types supported by DataChain must be of type DataType. DataType includes most Python types supported in Pydantic fields, as well as any class that inherits from Pydantic BaseModel.

Pydantic models can be used to group and nest multiple fields together into a single type object. Any Pydantic model must be registered so that the chain knows the expected schema of the model. Alternatively, models may inherit from DataModel, which is a lightweight wrapper around Pydantic's BaseModel that automatically handles registering the model.

DataModel

Bases: BaseModel

Pydantic model wrapper that registers model with DataChain.

__pydantic_init_subclass__ classmethod 

__pydantic_init_subclass__()

It automatically registers every declared DataModel child class.

Source code in datachain/lib/data_model.py 
58
59
60
61
62
@classmethod
def __pydantic_init_subclass__(cls):
    """It automatically registers every declared DataModel child class."""
    promote_default_none(cls)
    ModelStore.register(cls)

hidden_fields classmethod 

hidden_fields() -> list[str]

Returns a list of fields that should be hidden from the user.

Source code in datachain/lib/data_model.py 
73
74
75
76
@classmethod
def hidden_fields(cls) -> list[str]:
    """Returns a list of fields that should be hidden from the user."""
    return cls._hidden_fields

register staticmethod 

register(models: DataType | Sequence[DataType])

For registering classes manually. It accepts a single class or a sequence of classes.

Source code in datachain/lib/data_model.py 
64
65
66
67
68
69
70
71
@staticmethod
def register(models: DataType | Sequence[DataType]):
    """For registering classes manually. It accepts a single class or a sequence of
    classes."""
    if not isinstance(models, Sequence):
        models = [models]
    for val in models:
        ModelStore.register(val)

DataType module-attribute 

DataType = type[BaseModel] | StandardType

is_chain_type 

is_chain_type(t: type) -> bool

Return true if type is supported by DataChain.

Source code in datachain/lib/data_model.py 
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
def is_chain_type(t: type) -> bool:
    """Return true if type is supported by `DataChain`."""
    if ModelStore.is_pydantic(t):
        return True
    if any(t is ft or t is get_args(ft)[0] for ft in get_args(StandardType)):
        return True

    inner, is_optional = unwrap_optional(t)
    if is_optional:
        return is_chain_type(inner)

    # Deliberately not using `annotation_parts` here. This is validation, not
    # traversal: it must see the raw args, since normalising away `Ellipsis` would
    # let `list[int, ...]` through to be serialized as `list[int]`. Only `list` and
    # `dict` at the exact arity `type_to_str` can write back out are accepted --
    # abstract origins serialize as a bare "Sequence"/"Mapping", and `python_to_sql`
    # mis-types tuples. Matching on the origin identity also avoids `issubclass`
    # against generics that reject it (TypedDicts, some protocols).
    orig = get_origin(t)
    args = get_args(t)
    if orig is list:
        return len(args) == 1 and is_chain_type(args[0])
    if orig is dict:
        return len(args) == 2 and all(is_chain_type(arg) for arg in args)

    return False