Annotate Traces

Annotations in Agenta let you enrich the traces created by your LLM applications. You can add scores, comments, expected answers and other metrics to help evaluate your application's performance.

Open in Google Colaboratory

What You Can Do With Annotations

Collect user feedback on LLM responses
Run custom evaluation workflows
Measure application performance in real-time

Understanding How Annotations Work

Annotations link to specific points (spans) in your application traces. You can link them to:

Parent spans (like an agent invocation)
Child spans (like a single LLM call within a larger process)

Each annotation can include:

Numbers (scores, ratings)
Categories (labels, classifications)
Text (comments, reasoning)

Annotation Structure

Annotations are stored as traces and created through POST /api/simple/traces/. Here's what one looks like:

{
    "trace": {
        "data": {
            "outputs": {
                "score": 80,
                "normalized_score": 0.8,
                "reasoning": "The answer is mainly correct"
            }
        },
        "references": {
            "evaluator": {
                "slug": "my_evaluator"
            }
        },
        "links": {
            "invocation": {
                "trace_id": "47cb4a13b42a47bab2198509543ff14d",
                "span_id": "1a9130f6ad398e55"
            }
        },
        "meta": {
            "someattr": "somevalue"
        }
    }
}

An annotation has four main parts:

Data: The actual evaluation content (scores, comments)
References: Which evaluator the annotation belongs to, identified by slug
Links: The trace_id and span_id of the instrumented span you're annotating
Meta (optional): Any extra information you want to include

What You Can Include in Annotations

Annotations are flexible - you can include whatever makes sense for your evaluation:

// Just a simple score
{"score": 3}

// A score with explanation
{"score": 3, "comment": "The response is not grounded"}

// Multiple metrics with reference information
{"score": 3, "normalized_score": 0.5, "comment": "The response is not grounded", "expected_answer": "The capital of France is Paris"}

You can use numbers, text, booleans, or categorical values.

Connecting Annotations to Traces

Each annotation links to a specific span in your trace using a trace_id and span_id. These IDs follow the OpenTelemetry format.

warning

Agenta doesn't check if the trace and span IDs exist. Make sure they're valid before sending.

Working with Evaluators

Every annotation references an Evaluator by slug. The evaluator:

Defines what your annotation should contain
Provides context for interpreting the annotation

Annotations accept any slug in references.evaluator.slug. The call succeeds even if no evaluator entity with that slug exists yet. To see the evaluator (and its aggregated stats) in the Agenta UI, create the evaluator explicitly through POST /api/simple/evaluators/ or from the UI. Evaluators are not auto-created from the annotation payload.

You can add multiple annotations from the same evaluator to a single span, which is useful for collecting feedback from different sources.

Using Meta

The meta field lets you add extra information to your annotation. Use it to store things like user IDs, session information, or other context.

Creating Annotations

Python
JS/TS

import requests

base_url = "https://cloud.agenta.ai"
headers = {
    "Content-Type": "application/json",
    "Authorization": "ApiKey YOUR_API_KEY"
}

# Annotation payload
trace_payload = {
    "trace": {
        "data": {
            "outputs": {
                "score": 40,
                "reasoning": "Response contains some inaccuracies",
                "labels": ["Partially Correct", "Needs Improvement"]
            }
        },
        "references": {
            "evaluator": {
                "slug": "content_quality"
            }
        },
        "links": {
            "invocation": {
                "trace_id": "47cb4a13b42a47bab2198509543ff14d",
                "span_id": "1a9130f6ad398e55"
            }
        }
    }
}

# Make the API request
response = requests.post(
    f"{base_url}/api/simple/traces/",
    headers=headers,
    json=trace_payload
)

# Process the response
if response.status_code in (200, 202):
    print("Annotation created successfully")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

async function createAnnotation() {
  const baseUrl = 'https://cloud.agenta.ai';

  const headers = {
    'Content-Type': 'application/json',
    'Authorization': 'ApiKey YOUR_API_KEY'
  };

  const tracePayload = {
    trace: {
      data: {
        outputs: {
          score: 40,
          reasoning: "Response contains some inaccuracies",
          labels: ["Partially Correct", "Needs Improvement"]
        }
      },
      references: {
        evaluator: {
          slug: "content_quality"
        }
      },
      links: {
        invocation: {
          trace_id: "47cb4a13b42a47bab2198509543ff14d",
          span_id: "1a9130f6ad398e55"
        }
      }
    }
  };

  try {
    const response = await fetch(`${baseUrl}/api/simple/traces/`, {
      method: 'POST',
      headers: headers,
      body: JSON.stringify(tracePayload)
    });

    if (response.ok) {
      const data = await response.json();
      console.log("Annotation created successfully");
      console.log(data);
    } else {
      console.error(`Error: ${response.status}`);
      console.error(await response.text());
    }
  } catch (error) {
    console.error("Request failed:", error);
  }
}

createAnnotation();

info

Annotations accept any evaluator slug. To make the evaluator appear in the UI with aggregated stats, create it explicitly via POST /api/simple/evaluators/ or from the Agenta UI.

Understanding the Response

When you create an annotation, you receive back the stored trace with its own trace_id and span_id:

{
  "count": 1,
  "trace": {
    "created_at": "2025-05-13T16:54:52.245664Z",
    "created_by_id": "019315dc-a332-7ba5-a426-d079c43ab776",
    "span_id": "a1713ea8f59291f6",
    "trace_id": "bb39070937f74f169b60dfc420729ad3",
    "origin": "custom",
    "kind": "adhoc",
    "channel": "api",
    "data": {
      "outputs": {
        "score": 0,
        "reasoning": "The answer is totally false.",
        "expected_answer": "The capital of France is Paris",
        "normalized_score": 0
      }
    },
    "meta": {
      "user_id": "miamia"
    },
    "references": {
      "evaluator": {
        "id": "0196ca64-9cbf-7ee3-b507-2b559dd13090",
        "slug": "main-scorer"
      }
    },
    "links": {
      "invocation": {
        "span_id": "09383e1124b264d0",
        "trace_id": "5d52a529b48135fd58790f8aa3e33fb8"
      }
    }
  }
}

info

The annotation's own trace_id and span_id are different from the invocation it's linked to. Use the annotation's trace_id to fetch, update, or delete it later.

Viewing Annotations in the UI

You can see all annotations for a trace in the Agenta UI. Open any trace and check the Annotations tab to see detailed information. The right sidebar shows average metrics for each evaluator.

Querying Annotations

Fetch a single annotation by its own trace_id with GET /api/simple/traces/{trace_id}. Find all annotations linked to an instrumented span by calling POST /api/simple/traces/query with a links filter.

Python
JS/TS

1. Fetch a specific annotation by its trace_id:

import requests

base_url = "https://cloud.agenta.ai"
headers = {
    "Content-Type": "application/json",
    "Authorization": "ApiKey YOUR_API_KEY"
}

annotation_trace_id = "bb39070937f74f169b60dfc420729ad3"

response = requests.get(
    f"{base_url}/api/simple/traces/{annotation_trace_id}",
    headers=headers
)

if response.status_code == 200:
    print("Query successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

2. Query all annotations for an invocation:

# Filter by the invocation trace/span the annotations are linked to
query_data = {
    "trace": {
        "links": {
            "invocation": {
                "trace_id": "5d52a529b48135fd58790f8aa3e33fb8",
                "span_id": "09383e1124b264d0"
            }
        }
    },
    "windowing": {"limit": 50}
}

response = requests.post(
    f"{base_url}/api/simple/traces/query",
    headers=headers,
    json=query_data
)

3. Query all annotations produced by an evaluator:

query_data = {
    "trace": {
        "references": {
            "evaluator": {"slug": "content_quality"}
        }
    },
    "windowing": {"limit": 50}
}

response = requests.post(
    f"{base_url}/api/simple/traces/query",
    headers=headers,
    json=query_data
)

1. Fetch a specific annotation by its trace_id:

async function fetchAnnotation() {
  const baseUrl = 'https://cloud.agenta.ai';

  const headers = {
    'Content-Type': 'application/json',
    'Authorization': 'ApiKey YOUR_API_KEY'
  };

  const annotationTraceId = "bb39070937f74f169b60dfc420729ad3";

  try {
    const response = await fetch(`${baseUrl}/api/simple/traces/${annotationTraceId}`, {
      method: 'GET',
      headers: headers
    });

    if (response.ok) {
      const data = await response.json();
      console.log("Query successful");
      console.log(data);
    } else {
      console.error(`Error: ${response.status}`);
      console.error(await response.text());
    }
  } catch (error) {
    console.error("Request failed:", error);
  }
}

fetchAnnotation();

2. Query all annotations for an invocation:

async function queryAnnotationsForInvocation() {
  const baseUrl = 'https://cloud.agenta.ai';

  const headers = {
    'Content-Type': 'application/json',
    'Authorization': 'ApiKey YOUR_API_KEY'
  };

  const queryData = {
    trace: {
      links: {
        invocation: {
          trace_id: "5d52a529b48135fd58790f8aa3e33fb8",
          span_id: "09383e1124b264d0"
        }
      }
    },
    windowing: { limit: 50 }
  };

  try {
    const response = await fetch(`${baseUrl}/api/simple/traces/query`, {
      method: 'POST',
      headers: headers,
      body: JSON.stringify(queryData)
    });

    if (response.ok) {
      const data = await response.json();
      console.log("Query successful");
      console.log(data);
    } else {
      console.error(`Error: ${response.status}`);
      console.error(await response.text());
    }
  } catch (error) {
    console.error("Request failed:", error);
  }
}

queryAnnotationsForInvocation();

Removing Annotations

Call DELETE /api/simple/traces/{trace_id} with the annotation's own trace_id (returned when the annotation was created), not the invocation trace.

Python
JS/TS

import requests

base_url = "https://cloud.agenta.ai"
headers = {
    "Content-Type": "application/json",
    "Authorization": "ApiKey YOUR_API_KEY"
}

# The annotation's own trace_id (not the invocation it's linked to)
annotation_trace_id = "bb39070937f74f169b60dfc420729ad3"

response = requests.delete(
    f"{base_url}/api/simple/traces/{annotation_trace_id}",
    headers=headers
)

if response.status_code in (200, 204):
    print("Annotation deleted successfully")
else:
    print(f"Error: {response.status_code}")
    print(response.text)

async function deleteAnnotation() {
  const baseUrl = 'https://cloud.agenta.ai';

  const headers = {
    'Content-Type': 'application/json',
    'Authorization': 'ApiKey YOUR_API_KEY'
  };

  // The annotation's own trace_id (not the invocation it's linked to)
  const annotationTraceId = "bb39070937f74f169b60dfc420729ad3";

  try {
    const response = await fetch(`${baseUrl}/api/simple/traces/${annotationTraceId}`, {
      method: 'DELETE',
      headers: headers
    });

    if (response.ok) {
      console.log("Annotation deleted successfully");
    } else {
      console.error(`Error: ${response.status}`);
      console.error(await response.text());
    }
  } catch (error) {
    console.error("Request failed:", error);
  }
}

deleteAnnotation();

What You Can Do With Annotations​

Understanding How Annotations Work​

Annotation Structure​

What You Can Include in Annotations​

Connecting Annotations to Traces​

Working with Evaluators​

Using Meta​

Creating Annotations​

Understanding the Response​

Viewing Annotations in the UI​

Querying Annotations​

Removing Annotations​