LLM Observability with Grafana and OpenLLMetry
First, go to the Grafana Cloud account page under https://grafana.com/orgs/<your org name>
,
and click on Send Traces under Tempo. In Grafana Data Source settings,
note the URL
value. Click Generate now to generate an API key and copy it.
Note also the Stack ID
value (you can find it in the URL https://grafana.com/orgs/<Your Org Name>/stacks/<Stack ID>
).
With Grafana Agent
Make sure you have an agent installed and running in your cluster.
The host to target your traces is constructed is the hostname of the URL
noted above, without the https://
and the trailing /tempo
.
Add this to the configuration of your agent:
traces:
configs:
- name: default
remote_write:
- endpoint: <Grafana Hostname>:443
basic_auth:
username: <Stack ID from Grafana Cloud>
password: <Grafana Cloud API Token>
receivers:
otlp:
protocols:
grpc:
Note the endpoint. The URL you need to use is without https
and the trailing
/
. So https://tempo-us-central1.grafana.net/tempo
should be used as
tempo-us-central1.grafana.net:443
.
Set this as an environment variable in your app:
TRACELOOP_BASE_URL=<grafana-agent-hostname>:4317
Without Grafana Agent
Grafana cloud currently only supports sending traces to some of its regions. Before you begin, check out this list and make sure your region is supported.
In a terminal, type:
echo -n "<your stack id>:<your api key>" | base64
Note the result which is a base64 encoding of your user id and api key.
The URL you’ll use as the destination for the traces depends on your region/zone. For example, for AWS US Central this will be prod-us-central-0
.
See here for the names of the zones you should use below.
Then, you can set the following environment variables when running your app with Traceloop SDK installed:
TRACELOOP_BASE_URL=https://otlp-gateway-<zone>.grafana.net/otlp
TRACELOOP_HEADERS="Authorization=Basic%20<base64 encoded stack id and api key>"