Observability concepts

This page covers key concepts that are important to understand when logging traces to LangSmith.

A trace records the sequence of steps your application takes—from receiving an input, through intermediate processing, to producing a final output. Each step within a trace is represented by a run. Multiple traces are grouped together within a project.

The following diagram displays these concepts in the context of a simple RAG app, which retrieves documents from an index and generates an answer.

Runs

A run is a span representing a single unit of work or operation within your LLM application. This could be anything from a single call to an LLM or chain, to a prompt formatting call, to a runnable lambda invocation. If you are familiar with OpenTelemetry, you can think of a run as a span.

Traces

A trace is a collection of runs for a single operation. For example, if you have a user request that triggers a chain, and that chain makes a call to an LLM, then to an output parser, and so on, all of these runs would be part of the same trace. If you are familiar with OpenTelemetry, you can think of a LangSmith trace as a collection of spans. Runs are bound to a trace by a unique trace ID.

Projects

A project is a collection of traces. You can think of a project as a container for all the traces that are related to a single application or service. You can have multiple projects, and each project can have multiple traces.

Feedback

Feedback allows you to score an individual run based on certain criteria. Each feedback entry consists of a feedback tag and feedback score, and is bound to a run by a unique run ID. Feedback can be continuous or discrete (categorical), and you can reuse feedback tags across different runs within an organization.

You can collect feedback on runs in a number of ways:

1. Sent up along with a trace from the LLM application.
2. Generated by a user in the app inline or in an annotation queue.
3. Generated by an automatic evaluator during offline evaluation.
4. Generated by an online evaluator.

To learn more about how feedback is stored in the application, refer to the Feedback data format guide.

Tags

Tags are collections of strings that can be attached to runs. You can use tags to do the following in the LangSmith UI:

• Categorize runs for easier search.
• Filter runs.
• Group runs together for analysis.

Learn how to attach tags to your traces.

Metadata

Metadata is a collection of key-value pairs that you can attach to runs. You can use metadata to store additional information about a run, such as the version of the application that generated the run, the environment in which the run was generated, or any other information that you want to associate with a run. Similarly to tags, you can use metadata to filter runs in the LangSmith UI or group runs together for analysis.

Learn how to add metadata to your traces.

Data storage and retention

For traces ingested on or after Wednesday, May 22, 2024, LangSmith (SaaS) retains trace data for a maximum of 400 days past the date and time the trace was inserted into the LangSmith trace database.

After 400 days, the traces are permanently deleted from LangSmith, with a limited amount of metadata retained for the purpose of showing accurate statistics, such as historic usage and cost.

If you wish to keep tracing data longer than the data retention period, you can add it to a dataset. A dataset allows you to store the trace inputs and outputs (e.g., as a key-value dataset), and will persist indefinitely, even after the trace gets deleted.

Deleting traces from LangSmith

If you need to remove a trace from LangSmith before its expiration date, you can do so by deleting the project that contains it.

You can delete a project with one of the following ways:

• In the LangSmith UI, select the Delete option on the project’s overflow menu.
• With the delete_tracer_sessions API endpoint
• With the delete_project() (Python) or deleteProject() (JS/TS) in the LangSmith SDK.

LangSmith does not support self-service deletion of individual traces.

If you have a need to delete a single trace (or set of traces) from LangSmith project before its expiration date, the account owner should reach out to LangSmith Support with the organization ID and trace IDs.