Evaluate a dataset

Execute current application code against saved inputs and preserve report provenance.

This guide uses Apple Evaluations in an iOS/macOS 27 test target. LedgeKit loads inputs and reports results; your evaluation owns execution and scoring.

Construct native samples

Use the SummaryInput type from the dataset guide. Preserve each item's ID, original JSON, and source in its native sample:

import Evaluations
import Foundation
import LedgeKit
import LedgeKitTesting

@available(iOS 27.0, macOS 27.0, *)
struct SummarySample: SampleProtocol {
    let id: String
    let source: LedgeDatasetSource
    let savedInput: LedgeJSONValue
    let arguments: SummaryInput
    var input: String { id }
    var expected: LedgeEvaluationResponse? { nil }

    init(_ item: LedgeDatasetInput<SummaryInput>) {
        id = item.id.uuidString.lowercased()
        source = item.source
        savedInput = item.input
        arguments = item.value
    }
}

The saved source describes where the input came from. A fresh model request made during evaluation has its own identity.

The following evaluation calls the current summarize(_:client:) function from the Foundation Models guide. The completion metric reports execution status. Review semantic quality from the captured responses.

@available(iOS 27.0, macOS 27.0, *)
struct SummaryEvaluation: Evaluation {
    let samples: [SummarySample]
    var dataset: ArrayLoader<SummarySample> { .init(samples: samples) }

    func subject(from sample: SummarySample) async throws -> ModelSubject<LedgeEvaluationResponse> {
        let capture = LedgeEvaluationCapture()
        let directory = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
        defer { try? FileManager.default.removeItem(at: directory) }
        let client = try LedgeClient(serviceName: "my-app", transport: capture, localDirectory: directory)
        // Execute the real implementation. Its application return value is not model output.
        _ = try await summarize(sample.arguments.text, client: client)
        await client.flush()
        guard let trace = await capture.envelopes.first?.trace else {
            throw LedgeEvaluationError.invalidReport("No session captured")
        }
        return ModelSubject(value: trace.response)
    }

    var evaluators: Evaluators {
        Evaluator<SummarySample> { _, subject in
            subject.value.trace?.status == .success
                ? Metric("completed").passing()
                : Metric("completed").failing()
        }
    }

    func aggregateMetrics(using aggregator: inout MetricsAggregator) {
        aggregator.computeMean(of: Metric("completed"))
    }
}

The SDK response contains the captured trace. For an isolated model call, use span.response. The viewer displays the actual text or structured response; requests, timing, errors and transcripts remain details. Pipelines retain every call, including retries and fallbacks. The uploader rejects arbitrary application values and outputs that differ from their captured transcript.

Run, save, and validate

let dataset = try await LedgeDatasetClient(readKey: readKey).load(datasetID)
let inputs = try dataset.decodeInputs(
    SummaryInput.self, sourceKind: .span, definitionName: "summarizer"
) { input in
    guard !input.text.isEmpty else { throw InputError.emptyText }
}
let evaluation = SummaryEvaluation(
    samples: inputs.map(SummarySample.init)
)
let result = try await evaluation.run(info: dataset.evaluationInfo())
let reportURL = try result.saveJSON(
    to: reportDirectory, includeReportMetadata: true
)
try dataset.validateResults(in: reportURL)
let reporter = LedgeEvaluationReporter(writeKey: writeKey)
try await reporter.submit(reportURL)

reportDirectory is a writable URL where Apple Evaluations exports the report; the reporter manages its own durable upload storage.

evaluationInfo() records the full expected dataset membership before execution. validateResults(in:) requires every loaded item ID exactly once in the report. It does not score outputs or assert that native execution succeeded. Verify that result.errors.hasFailures is false and assert your quality metrics in the test as well.

Save the report before validation so incomplete results remain inspectable. A delivery retry should submit the saved file instead of calling run again.

Review the result

Open Evaluations in the write key's app and environment. Inspect sample responses and metrics, then use the recorded dataset/source information to follow the result back to its inputs where available.

Construct native samples

Define execution and scoring

Run, save, and validate

Review the result

On this page