Save and load evaluation inputs

A dataset stores inputs for one trace or span definition within an app. Inputs are copied into independent storage and remain available after the source trace expires. Datasets are shared across the app's Live and Test environments.

Save an input in the console

1. Record inputData on a run or invocation; see input capture.
2. Open that trace or span in the console and choose Add to dataset.
3. Select a compatible dataset or New dataset. The captured source determines its definition and source kind.
4. Open Datasets, select the collection, and copy its dataset ID.

Sources without inputData cannot be added. Dataset inputs must match the collection's definition name and kind. Versions of the same agent definition can coexist. Creating an empty dataset is also available from the Datasets page.

Download and validate before execution

Use a read key for the dataset's app. Either environment's read key works.

import Foundation
import LedgeKit

struct SummaryInput: Codable, Sendable {
    let text: String
}
enum InputError: Error { case emptyText }

func loadSummaryInputs(datasetID: UUID, readKey: String) async throws
    -> (LedgeDataset, [LedgeDatasetInput<SummaryInput>]) {
    let dataset = try await LedgeDatasetClient(apiKey: readKey).load(datasetID)
    let inputs = try dataset.decodeInputs(
        SummaryInput.self,
        sourceKind: .span,
        definitionName: "summarizer"
    ) { input in
        guard !input.text.isEmpty else { throw InputError.emptyText }
    }
    return (dataset, inputs)
}

Use .trace and the workflow definition name for workflow inputs. The name is supplied by your application; it does not register a function in the SDK.

The SDK downloads one complete snapshot and validates every input before returning the array. There is no partially successful decode. The snapshot does not change while your test runs, even if someone edits the dataset in the console.

What each decoded item contains

Decoding or application-validation failures identify the offending item UUID. Empty datasets and incompatible definitions fail explicitly. Download limits are 1,000 items and 4 MiB; oversized downloads are rejected, not truncated.

Your application chooses the function, builds its current prompt, and executes the model. The SDK does not compile saved templates or choose a function based on the definition name.

Continue with evaluating the dataset and the dataset reference.