Batched Inference
batch_map() calls a function on up to batch rows at once and stores one result per input row, keeping the existing columns. Use it when a model or service handles many inputs in one call, such as a GPU forward pass or an embeddings request. For one row per call, use map().
setup() loads the model once per process and passes it to encode() for each batch. A runnable version is in examples/get_started/udfs/stateful.py.
Inputs and outputs
batchis required. Each column parameter receives a list of up tobatchvalues, also withbatch=1; the last batch can be shorter.- Resources from
setup()are passed unchanged, not as lists. - The function returns
list[T]orIterator[T]with one result per input row, in input order. Any other number of results raisesUdfError. - The output column has type
T, and an explicitoutput=describes one row.
Use dc.BatchMapper for a class with setup() and teardown(); dc.Mapper is for map(). Multiple named functions work as in map(), for example batch_map(embedding=encode, label=classify, batch=32). dc.llm operations call the model once per row, so use them with map() and settings(parallel=N).
Row independence
Delta updates and retries can form different batches. Each row's result must be independent of the other rows in its batch.
A rerun reuses rows already stored in checkpoints, even with a different batch size. For example, if a run fails because GPU memory runs out, lower batch and rerun the script.
Prefetch
In batch_map(), settings(prefetch=N) counts batches: up to N batches download at a time, the files of each batch one after another. Up to 2N+1 batches can be on local disk per process: the batch being processed and 2N ahead. Budget about (2N+1) × batch × average file size of disk per process, and start with prefetch=1 or 2.
settings(batch_size=N) sets database insert buffering; it does not change the number of rows per call.