Evaluate a dataset

Execute current application code against saved inputs and preserve report provenance.

This guide uses Apple Evaluations in an iOS/macOS 27 test target. LedgeKit loads inputs and reports results; your evaluation owns execution and scoring.

Construct native samples

Use the SummaryInput type from the dataset guide. Preserve each item's ID, original JSON, and source in its native sample:

import Evaluations
import Foundation
import LedgeKit
import LedgeKitTesting

@available(iOS 27.0, macOS 27.0, *)
struct SummarySample: SampleProtocol {
    let id: String
    let source: LedgeDatasetSource
    let savedInput: LedgeJSONValue
    let arguments: SummaryInput
    var input: String { id }
    var expected: String? { nil }

    init(_ item: LedgeDatasetInput<SummaryInput>) {
        id = item.id.uuidString.lowercased()
        source = item.source
        savedInput = item.input
        arguments = item.value
    }
}

The saved source describes where the input came from. A fresh model request made during evaluation has its own identity.

The following evaluation calls the current summarize(_:client:) function from the Foundation Models guide. A nonempty-output metric is a minimal example; add quality checks appropriate to your application.

@available(iOS 27.0, macOS 27.0, *)
struct SummaryEvaluation: Evaluation {
    let samples: [SummarySample]
    let client: LedgeClient
    var dataset: ArrayLoader<SummarySample> { .init(samples: samples) }

    func subject(from sample: SummarySample) async throws -> ModelSubject<String> {
        let value = try await summarize(sample.arguments.text, client: client)
        return ModelSubject(value: value)
    }

    var evaluators: Evaluators {
        Evaluator<SummarySample> { _, subject in
            subject.value.isEmpty
                ? Metric("nonempty").failing()
                : Metric("nonempty").passing()
        }
    }

    func aggregateMetrics(using aggregator: inout MetricsAggregator) {
        aggregator.computeMean(of: Metric("nonempty"))
    }
}

This uses a string-valued native subject. If your evaluation uses a richer native model subject, retain its native capture when exporting; the uploader does not construct or replace it.

Run, save, and validate

let (dataset, inputs) = try await loadSummaryInputs(
    datasetID: datasetID, readKey: readKey
)
let evaluation = SummaryEvaluation(
    samples: inputs.map(SummarySample.init), client: client
)
let result = try await evaluation.run(info: dataset.evaluationInfo())
let reportURL = try result.saveJSON(
    to: reportDirectory, includeReportMetadata: true
)
let report = try LedgeEvaluationFile(data: Data(contentsOf: reportURL))
try dataset.validateResults(in: report)
try await reporter.submit(report)

client is your configured trace client, reporter is the evaluation reporter, and reportDirectory is a writable URL in your test environment.

evaluationInfo() records the full expected dataset membership before execution. validateResults(in:) requires every loaded item ID exactly once in the report. It does not score outputs or assert that native execution succeeded. Verify that result.errors.hasFailures is false and assert your quality metrics in the test as well.

Save the report before validation so incomplete results remain inspectable. A delivery retry should submit the saved file instead of calling run again.

Review the result

Open Evaluations in the write key's app and environment. Inspect sample responses and metrics, then use the recorded dataset/source information to follow the result back to its inputs where available.

Construct native samples

Define execution and scoring

Run, save, and validate

Review the result

On this page