Error handling & troubleshooting¶

Failures in datafast come in two kinds, and the second kind is the expensive one.

Loud failures raise. A pipeline that is built wrong, a missing API key, a prompt placeholder with no matching column — these stop before or during the run and tell you what happened.

Quiet failures do not raise. A record whose LLM call failed is logged and dropped, and the run finishes normally with fewer records than you asked for. Nothing in the return value says so.

This page covers both, starting with the errors you can read and ending with the ones you have to go looking for.

The two exceptions datafast defines¶

Everything else you will see is a plain ValueError, RuntimeError, KeyError or ImportError.

Both are on the top-level package:

from datafast import PipelineChangedError, PipelineValidationError

Before the run: compile()¶

compile() checks the shape of a pipeline without running it. run() calls it for you, so these errors arrive before a single LLM call is paid for. Call it yourself to check a pipeline you are still writing:

from datafast import Map, Sink, Source

pipeline = Source.list([{"text": "hello"}]) >> Map(lambda r: r) >> Sink.list()
pipeline.compile()

It raises on the first problem it finds:

Column checking is deliberately conservative. Map, FlatMap, Group, Pair, Join and Concat reshape records in ways compile() cannot read, so after one of them the schema is unknown and no column is reported missing. A clean compile() means the structure is right, not that every column reference is.

During the run: what stops and what does not¶

This is the most important distinction on the page.

A prompt placeholder with no matching column is a loud failure, which is worth knowing because it looks like the quiet kind:

An LLMStep whose prompt is "Write about {topic}" while the records only carry a text column stops the run with KeyError: 'topic'. This happens while the calls are being built, so nothing has been spent.

A failed call costs only that call¶

The runner sends LLM calls in batches, grouped by served model. A batch is all-or-nothing at the provider, so if any call in it raises, the whole batch comes back as one error.

When that happens the runner retries the group one call at a time. Only the calls that really fail are lost, whatever batch_size you set. With eight records and a single failure on the second call, you keep seven — at batch_size 1, 4 or 8 alike.

The retry costs a second attempt for the calls that had already succeeded inside the failed batch. That is the price of not losing them, and it is only paid on the failure path.

on_parse_error and the records that disappear¶

Every LLM step takes on_parse_error, and it has two values:

One thing about it is not obvious: it is not only about parsing. The try block around each record covers the whole call — building the messages, the provider request, and the parse. A rate limit, a timeout or a bad key is treated exactly like a malformed reply. With the default the record is dropped and the run continues; with "raise" the run stops on the first one, whichever kind it is.

Both values behave the same way under Pipeline.run() as under step.process(records).

With the default, the way to find out whether a run went well is to count:

from datafast import LLMStep, ListSink, Source, openai

records = [{"text": "a"}, {"text": "b"}, {"text": "c"}]

pipeline = (
    Source.list(records)
    >> LLMStep(
        prompt="Summarize: {text}",
        input_columns=["text"],
        output_column="summary",
        model=openai(),
    )
    >> ListSink()
)

results = pipeline.run()
if len(results) < len(records):
    print(f"{len(records) - len(results)} records were dropped")

If the count is short, the log holds one LLM call failed line per lost record, with the model, the call id and the error.

Warnings¶

datafast raises five Python warnings. None of them stops a run, and each means something different.

The first one names the parameter as it goes on the wire, not as you set it. Setting thinking=True on a served model with no reasoning support warns about reasoning_effort, because that is the request field it would have filled.

Choosing how strict to be¶

unsupported_params on any served model decides what the first warning does:

from datafast import anthropic

model = anthropic(unsupported_params="fail")

Use "fail" while you are working out what a served model supports. It turns a silent difference between two providers into an error you cannot miss.

Checkpoints and resume¶

A checkpoint is the on-disk state of a run: a manifest.json plus one JSONL file per step. Resume reads it and skips the steps that finished.

PipelineChangedError¶

The manifest holds a fingerprint of the pipeline. If it does not match, datafast will not reuse the checkpoint:

PipelineChangedError: Pipeline structure has changed since checkpoint.
Use resume=False to start fresh.

With resume=False there is no error — the old checkpoint is cleared and the run starts over, with Pipeline changed, clearing old checkpoint in the log.

The fingerprint only sees step names and step classes. Change a prompt, swap the served model, or rewrite the function inside a Map, and the fingerprint is identical. Resume will continue as if nothing changed and mix old records with new ones. Point a changed pipeline at a fresh checkpoint_dir.

resume_from and stop_after errors¶

resume_from re-runs one step and everything after it. All three of its failure modes are ValueError, raised before anything executes:

stop_after takes the same kind of value and fails the same way:

The names are step class names — ListSource, LLMStep, JSONLSink — not variable names, unless you set one with as_step.

What actually survives a crash¶

Completed steps always survive. A partly finished LLM step is the hard case, and the answer depends on checkpoint_every, which defaults to 100.

An LLM step saves its first progress file after checkpoint_every calls, and resume needs that file. A run that dies at call 60 on the default has no progress file, so resume runs all 60 calls again and you pay twice. Lower it for slow or expensive steps:

from datafast import Map, Sink, Source

pipeline = Source.list([{"text": "hello"}]) >> Map(lambda r: r) >> Sink.list()
pipeline.run(checkpoint_dir="./checkpoints", checkpoint_every=10)

After the first progress file, the crash point no longer matters. Each record's call id is written right after the record itself, so resume knows exactly which calls finished and makes only the missing ones. In both cases a resumed run produces each record once.

If a crash lands between those two writes, the record is discarded and its call runs again — one repeated call rather than a duplicated record.

Provider errors¶

Calls to a served model follow one rule: ValueError means you asked for something impossible; everything else is wrapped in RuntimeError naming the provider.

The RuntimeError message carries the full traceback of the original error, so the provider's own message — the 401, the 429, the model-not-found — is at the bottom of it. Read from the end.

Only five kinds of failure are retried: rate limits, connection errors, timeouts, internal server errors and service-unavailable errors. A bad key or a malformed request fails on the first attempt, whatever retry_limit says.

Inside a pipeline you will not see these at all, because the runner catches them. They appear as LLM call failed in the log and as missing records in the result.

Things that fail silently¶

The list worth re-reading when a run finishes and the output looks wrong.

• Dropped records. The default on_parse_error="skip" swallows every exception, not just parse failures. Compare the output count with the input count, or set on_parse_error="raise" to stop on the first one.
• A mistyped prompt file path raises. prompt=Path("prompts/typo.txt") raises FileNotFoundError at the first record when the file does not exist, before any call.
• The checkpoint fingerprint ignores prompts and served models. Only step names and classes are hashed.
• A parse mode never fails loudly in XML. A missing tag gives an empty string.

Reading the log¶

datafast logs through loguru. The lines that matter:

Where to go next¶

• Pipelines & execution — checkpoints, resume and the run controls in full.
• LLM step — every option on a step, including on_parse_error.
• Structured output & parse modes — why a reply failed to parse.
• Calling a served model directly — the error contract outside a pipeline.
• Served models — declaring capabilities yourself.