Specialized LLM Steps¶

Classify, Score, Compare, Rewrite and Extract are LLM steps for five common jobs. Each one writes the prompt for you and parses the answer into named columns, so you declare the job instead of the wording.

LLMStep is the general one: you give it the prompt and say how to parse the answer. Reach for a specialized step when your job is one of these five, and for LLMStep when it is not.

At a glance¶

Each of them also adds a _model column naming the served model that answered.

Shared parameters¶

These are keyword arguments on all five steps, apart from the two exceptions the table notes.

Passing a list of served models, or a Sample of them, runs every record against every served model. Ten records and three served models is thirty records out, and thirty LLM calls.

The placeholders prompt accepts differ per step:

An LLM or a plain function¶

Classify, Score, Compare and Extract also run on a plain Python function. Pass fn instead of llm; passing both, or neither, raises ValueError when you build the step.

from datafast import Classify

step = Classify(
    labels=["short", "long"],
    input_columns=["text"],
    output_column="length_class",
    fn=lambda r: "short" if len(r["text"]) < 100 else "long",
)

Use fn when the answer is a rule, not a judgement: a length cutoff, a regular expression, a lookup table. It costs nothing, it is exact, and it is repeatable. Mixing both in one pipeline is normal — cheap rules first, the LLM only on what is left.

Rewrite has no fn. It always calls an LLM, and llm is required.

Classify¶

Assigns one label, or several, from a set you fix up front.

from datafast import Classify, openai

step = Classify(
    labels=["politics", "sports", "tech"],
    input_columns=["article"],
    output_column="topics",
    multi_label=True,
    labels_description={"tech": "Software, hardware, the internet"},
    llm=openai("gpt-4o-mini"),
)

labels_description may cover some labels only; the rest appear with their name alone.

The two label modes handle a bad answer differently. With multi_label=False an unknown label is written to the record anyway, after a warning. With multi_label=True unknown labels are dropped, which can leave an empty list. Check the column against labels before you trust it.

Score¶

Gives a number inside a range, against criteria you describe.

from datafast import Score, openai

step = Score(
    input_columns=["question", "answer"],
    output_column="quality",
    score_range=(1, 5),
    criteria="helpfulness and accuracy",
    rubric={1: "Wrong or unhelpful", 3: "Adequate", 5: "Excellent"},
    llm=openai("gpt-4o-mini"),
)

criteria is one sentence of guidance. rubric anchors particular numbers; you do not have to describe every number, only the ones worth pinning. Keys are shown in ascending order.

The score is always forced into score_range. A model answering 99 on a (1, 5) scale is stored as 5, not rejected — so an out-of-range answer looks like a top score. The same clamping applies to what fn returns.

Compare¶

Judges two columns of the same record against each other. The usual use is preference data: two candidate answers, which is better.

output_mode decides which columns you get. Any other value raises ValueError.

from datafast import Compare, openai

step = Compare(
    column_a="response_chosen",
    column_b="response_rejected",
    criteria="helpfulness and accuracy",
    output_mode="detailed",
    llm=openai("gpt-4o-mini"),
)

score_range is only used by "scores" and "detailed"; under "winner" it does nothing. As with Classify, a winner that is not "a", "b" or "tie" is written to the record anyway, after a warning.

Rewrite¶

Rewrites text in one of eight ways. It writes plain text, not JSON, so the answer goes into the column as it comes back.

The three required companions are checked when you build the step: mode="custom" without custom_instruction raises ValueError immediately, not mid-run. An unknown mode raises too.

from datafast import Rewrite, openai

step = Rewrite(
    input_column="text",
    llm=openai("gpt-4o-mini"),
    mode="custom",
    custom_instruction="Rewrite as a single question",
    preserve=["technical terms"],
    num_variations=3,
)

num_variations=3 makes three LLM calls per record and yields three records, each with a _variation column holding 0, 1 or 2. That column only appears when num_variations is above 1.

Extract¶

Pulls named values out of free text.

Give exactly one of fields, extractor or fn. Zero or two raises ValueError.

fields maps a name to a description. The name becomes the column, the description tells the model what to put there. Use it when you know what you want.

from datafast import Extract, openai

step = Extract(
    input_column="product_description",
    fields={
        "product_name": "The name of the product",
        "price": "Price in dollars, number only",
    },
    flatten=True,
    llm=openai("gpt-4o-mini"),
)

flatten decides the shape of the output. With flatten=False, the whole result is one dict in output_column. With flatten=True, each value becomes its own column and output_column is ignored. Flatten when the next step reads the values as columns, or when you write to CSV; keep the dict when you only want to carry the result along.

A value the model leaves out is written as an empty string "", even for values that are normally lists. An "entities" result with no organizations gives "", not [].

Things worth knowing¶

• fn mode ignores forward_columns and exclude_columns in Classify, Score and Compare, and adds no _model column. It copies the whole record and adds its result. Extract is the exception: its fn mode honours both.
• fn results are not checked against labels. Classify in fn mode writes whatever the function returns. Score in fn mode is still clamped to score_range.
• on_parse_error="skip" skips every error, not just parse errors. A timeout, a refusal or a bad key drops that record with a warning, and you end a run with fewer records than you started. Use "raise" while you are developing.
• A bad label or winner is still written. Only multi_label=True filters. Validate the column before training on it.
• Score clamps instead of failing, so an out-of-range answer becomes the top or bottom of score_range.
• Several served models multiply your records, one output record per record per served model. So does num_variations on Rewrite.
• Extract fills missing values with "", including where you expected a list.

Where to go next¶

• LLM step — LLMStep, the general one, and parse modes.
• Served Models — building the llm argument.
• Source and Seed — where the records come from.
• Full Glossary — the precise meaning of record, column, step and served model.
• API reference — generated signatures for everything on this page.