Upgrade trace-level evaluators

Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of the Langfuse v4 upgrade. Existing evaluators continue to run while you upgrade.

Observation-level evaluators run in real time and scale with larger traces and higher evaluation volume. Any future multi-span support will also use the observation-centric data model, so upgrading is the foundation for future evaluation capabilities.

Setting up a new evaluator? Start with an observation-level evaluator in the LLM-as-a-Judge setup guide. This page is only for upgrading existing trace-level evaluators.

Upgrade your evaluator

Before you start

First, upgrade your SDK or ingestion path. Observation-level evaluators run in real time with:

Python SDK: v4.7.0+ (upgrade guide)
JS/TS SDK: v5.4.0+ (upgrade guide)
Direct OpenTelemetry ingestion: x-langfuse-ingestion-version: 4 (setup guide)

After upgrading to the v4 data model, input, output, and context are no longer available as separate trace fields. The equivalent values must be available on an observation.

Upgrade in Langfuse

Open Evaluation > LLM-as-a-Judge. The upgrade screen walks you through all evaluators that need to be upgraded and recommends one of two paths for each evaluator:

Langfuse Assistant: Selects an existing observation and translates the evaluator configuration in the app.
Coding assistant: Uses the provided evaluator context and prompt to make any required SDK or instrumentation changes in your coding editor.

The agentic flow makes the upgrade conversational. You can inspect the proposed target observation, filters, and variable mappings, ask why a change is needed, and refine the configuration before creating the evaluator. You do not need to translate or recreate the configuration manually.

Review, create, and validate

How the configuration may change

Review the target observation, filters, and variable mappings proposed by the assistant. Depending on the existing evaluator, the configuration may target the same observation as before, replace trace fields with values from one observation, or require an instrumentation change so all required values are available together. The assistant guides you through the appropriate change; the technical reference below explains the cases in detail.

How resulting scores may differ

Before creating the evaluator, confirm how the resulting scores may change:

A trace-level evaluator produces one score per matching trace.
An observation-level evaluator produces one score per matching observation.
If three observations in one trace match the filters, the upgraded evaluator produces three scores. To keep one score per trace, narrow the filters to one observation using its name, type, or root-observation status.

Create and validate

Create the observation-level evaluator, then confirm that it receives the expected input, output, and context and produces the intended number of scores. You can keep the legacy evaluator active temporarily to compare results, or deactivate it immediately.

If you need to revert, deactivate the new evaluator and reactivate the existing legacy evaluator. Historical evaluation results remain accessible.

Technical reference

The recommended assistant guides you through every change. You do not need to classify or update configurations manually. Expand this section only if you want to understand how an existing configuration may be translated.

View technical configuration cases and a before/after example

Langfuse groups existing configurations by their variable mappings and recommends an assistant based on the required change:

Current variable mapping	Typical configuration change
`Observation` for every variable, all from one observation	Target the same span or generation and translate the filters and variable mappings.
`Trace` for every variable	Select the observation holding the equivalent input, output, and metadata. For end-to-end evaluations, this is often a root observation.
A mix of `Trace` and one `Observation`	Target that observation and make the trace fields available there via `propagate_attributes()` or an instrumentation update.
More than one `Observation`, with or without `Trace`	Write the required values to a root or dedicated evaluation observation, then target it directly. Any future multi-span support will also use the observation-centric model.

Before and after example

This example upgrades a trace-level evaluator that reads trace input and output to target the span containing the same values.

Before: trace-level evaluator

Target: Trace
Filters:
  environment not in ["sdk-experiment"]
  name = "user-workflow"
Variable mapping:
  query = trace.input.key
  generation = trace.output

After: observation-level evaluator

Target: Observation
Filters:
  environment not in ["sdk-experiment"]
  traceName = "user-workflow"
  type = "SPAN"
Variable mapping:
  query = observation.input.key
  generation = observation.output

Upgrade checklist

Upgrade the SDK or OpenTelemetry ingestion path.
Open the evaluator upgrade screen in Evaluation > LLM-as-a-Judge.
Follow the recommended Langfuse Assistant or coding assistant path.
Review the target observation, filters, and variable mappings.
Confirm the expected number and placement of scores.
Decide whether to keep the legacy evaluator active for comparison.
Validate the evaluator results and deactivate the legacy evaluator when ready.

Getting help

Troubleshooting: Why is my observation-level evaluator not executing?
Documentation: LLM-as-a-Judge step-by-step set up guide
GitHub: Report issues at github.com/langfuse/langfuse
Support: Contact support@langfuse.com for enterprise customers

Last updated: July 19, 2026

Was this page helpful?

On this page