Skip to main content

Tracing

Introduction​

Tracing Infrastructure​

Grafana Tempo is an open source tracing backend (Prometheus for traces). Tempo is deeply integrated with Grafana, Prometheus, and Loki. Tempo can ingest common open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.

Client instrumentation is required before traces can be generated, most of the popular client instrumentation frameworks have SDKs in the most commonly used programming languages.

OpenTelemetry has the most active development in the community and may be a better long-term choice.

alt text

How to send Traces to Grafana Cloud​

The infrastructure team currently relies on a Grfana Cloud managed Tempo instance. Retention is set to 30days.

Since most applications do not currently generate Traces, Ansible support is lacking.

Configure Environment Variables​

Ensure you set up the following environment variables on the Grafan alloy container:

  • GRAFANA_CLOUD_OTLP_ENDPOINT: The endpoint URL for Grafana Cloud.
  • GRAFANA_CLOUD_INSTANCE_ID: Your Grafana Cloud instance ID.
  • GRAFANA_CLOUD_API_KEY: The API key for authenticating to Grafana Cloud.

Run Grafana alloy collector​

Docker run example:​

docker run --env GRAFANA_CLOUD_OTLP_ENDPOINT=<> --env GRAFANA_CLOUD_INSTANCE_ID=<> --env GRAFANA_CLOUD_API_KEY=<> faithtosin/grafana-alloy:<tag>

You can also pass the OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES environmental variables to your application for additional tracing labels.

Docker compose example:​

version: "3"
services:
  alloy:
    image: faithtosin/grafana-alloy:v1.4.2
    container_name: grafana_alloy
    ports:
      - "12345:12345"
      - "4318:4318"
      - "4317:4317"
    environment:
      GRAFANA_CLOUD_API_KEY: 'glc_eyJvIjxxxxxxxxxxxxxxxxxxx'
      GRAFANA_CLOUD_INSTANCE_ID: '782677'
      GRAFANA_CLOUD_OTLP_ENDPOINT: 'tempo-prod-03334-prod-us-east-34.grafana.net:443'
  my-tracing-app:
    image: jaegertracing/example-hotrod:latest
    container_name: my-tracing-app
    ports:
      - "8080:8080"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
      - OTEL_SERVICE_NAME=test-service-name
      - OTEL_RESOURCE_ATTRIBUTES=environment=stage,service.namespace=test-namespace,service.version=1.4,service.instance.id=
    restart: unless-stopped

View Traces:​

With Grafana alloy forwarding traces to Grafana Cloud Tempo, traces will be visible at Grafana cloud. alt text