Observability for Bedrock with Grafana Tempo

Step 1

Install the Traceloop SDK and initialize it. It will automatically log all calls to Bedrock using boto3 - both prompts and completions.

import boto3
import json

from traceloop.sdk import Traceloop

Traceloop.init(app_name="joke_generation_service")
brt = boto3.client(service_name='bedrock-runtime')

body = json.dumps({
    "prompt": "Tell me a joke about opentelemetry",
    "max_tokens": 200,
    "temperature": 0.5,
    "p": 0.5,
})

response = brt.invoke_model(
    body=body,
    modelId='cohere.command-text-v14',
    accept='application/json',
    contentType='application/json'
)

response_body = json.loads(response.get('body').read())

response = response_body.get('generations')[0].get('text')
print(response)

import * as traceloop from "@traceloop/node-server-sdk";
import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";

traceloop.initialize({
  appName: "sample_bedrock_ai21",
  apiKey: process.env.TRACELOOP_API_KEY,
  disableBatch: true,
});

// Create a BedrockRuntimeClient with your configuration
const client = new BedrockRuntimeClient();

const prompt = "How do you say Good Morning in French?";

const input = {
  modelId: "ai21.j2-ultra-v1",
  contentType: "application/json",
  accept: "application/json",
  body: JSON.stringify({
    prompt,
    maxTokens: 200,
    temperature: 0.7,
    topP: 1,
    stopSequences: [],
    presencePenalty: { scale: 0 },
    frequencyPenalty: { scale: 0 },
  }),
};

async function generateTextContent() {
  return await traceloop.withWorkflow("sample_completion", {}, async () => {
    // Create an InvokeModelCommand with the input parameters
    const command = new InvokeModelCommand(input);

    // Send the command to invoke the model and await the response
    client.send(command).then((response) => {
      // Save the raw response
      const rawRes = response.body;

      // Convert it to a JSON String
      const jsonString = new TextDecoder().decode(rawRes);

      // Parse the JSON string
      const parsedResponse = JSON.parse(jsonString);

      console.log(">>> non-stream", parsedResponse);
    });
  });
}

generateTextContent();

Step 2

Make sure you have a Grafana Agent installed and running in your system. Go to the Grafana Cloud account page under https://grafana.com/orgs/<your org name>, and click on Send Traces under Tempo. In Grafana Data Source settings, note the URL value. Click Generate now to generate an API key and copy it. Note also the Stack ID value (you can find it in the URL https://grafana.com/orgs/<Your Org Name>/stacks/<Stack ID>). Add this to the configuration of your Grafana Agent.

TRACELOOP_BASE_URL=https://api.honeycomb.io
TRACELOOP_HEADERS="x-honeycomb-team=<YOUR_API_KEY>"

Docs

Observability for Bedrock with Grafana Tempo

Step 1

Step 2

Discover use cases

Trace prompts and completions in Bedrock

More things you can connect to {platform}

Chroma

Cohere

OpenAI

Watsonx

Anthropic

LangChain

LlamaIndex