​

Troubleshooting Guide

This guide helps you diagnose and resolve common issues you might encounter when self-hosting Traceloop.

​

Connection Issues

​

ClickHouse Connection Failures

The Traceloop Helm chart expects specific configuration for external ClickHouse connections:

Required Configuration:

• Fill out values-external-clickhouse.yaml with your ClickHouse host, port, and connection details
• Ensure traceloop-clickhouse-secret exists with required credentials:
  • CLICKHOUSE_USERNAME
  • CLICKHOUSE_PASSWORD
  • Note: This secret is automatically created if using Traceloop Terraform, otherwise create it manually

# Verify the ClickHouse secret exists
kubectl get secret traceloop-clickhouse-secret -n traceloop

# Check secret keys (without exposing values)
kubectl describe secret traceloop-clickhouse-secret -n traceloop

Common issues and their solutions when connecting to ClickHouse:

• Verify the host and port are correct
• Ensure the database exists and the user has appropriate permissions
• Check if SSL is required (secure: true)
• Verify network connectivity and firewall rules
• Ensure traceloop-clickhouse-secret contains correct credentials
• Verify values-external-clickhouse.yaml has correct host and configuration

​

Kafka Connection Issues

The Traceloop Helm chart expects specific configuration for external Kafka connections:

Required Configuration:

• Fill out values-external-kafka.yaml with your Kafka broker addresses, connection details, and credentials
• Ensure the following Kafka topics exist: traces, spans, and metrics
  • Note: These topics are automatically created if using Traceloop Terraform, otherwise create them manually

# Verify your Kafka configuration is properly applied
kubectl get configmap -n traceloop | grep kafka

If you’re having trouble connecting to Kafka:

• Confirm broker addresses are accessible from the Kubernetes cluster
• Ensure required topics exist: traces, spans, and metrics
• For Confluent Cloud:
  • Ensure SASL authentication is properly configured
  • Verify SSL is enabled
• Check topic creation permissions
• Verify network policies allow Kafka connectivity
• Verify values-external-kafka.yaml has correct broker addresses, credentials, and configuration

​

PostgreSQL Connection Problems

The Traceloop Helm chart expects specific configuration for external PostgreSQL connections:

Required Configuration:

• Fill out values-external-postgres.yaml with your PostgreSQL address and connection details
• Ensure traceloop-postgres-secret exists with required credentials:
  • POSTGRES_DATABASE_USERNAME
  • POSTGRES_DATABASE_PASSWORD
  • Note: This secret is automatically created if using Traceloop Terraform, otherwise create it manually

# Verify the PostgreSQL secret exists
kubectl get secret traceloop-postgres-secret -n traceloop

# Check secret keys (without exposing values)
kubectl describe secret traceloop-postgres-secret -n traceloop

Common PostgreSQL connectivity issues:

• Verify database exists and is accessible
• Check user permissions (needs ability to create schemas and tables)
• For SSL connections, ensure proper SSL mode is set
• Verify PostgreSQL version compatibility (9.6 or higher)
• Ensure traceloop-postgres-secret contains correct credentials
• Verify values-external-postgres.yaml has correct database address and configuration

​

Kubernetes Deployment Issues

​

Pod Startup Failures

If pods aren’t starting properly:

# Check pod status
kubectl get pods -n traceloop

# Check pod events
kubectl describe pod -n traceloop <pod-name>

# Check pod logs
kubectl logs -n traceloop <pod-name>

Common solutions:

• Verify resource requests and limits are appropriate
• Ensure all secrets are properly created
• Check container image pull permissions

​

Ingress Issues

Our Ingress Architecture: Traceloop uses a custom ingress setup rather than traditional Kubernetes ingress controllers. A load balancer handles SSL termination and forwards HTTP traffic directly to port 30800 on the Kubernetes cluster. This port maps to a Kong gateway pod, which serves as the unified ingress point for all HTTP endpoints.

If you can’t access Traceloop through your load balancer:

Load Balancer Configuration:

• Verify the load balancer is properly provisioned
• Check DNS records point to the correct load balancer endpoint
• Verify SSL certificate is properly attached to load balancer for HTTPS termination
• Ensure security groups allow traffic from load balancer to Kubernetes cluster on port 30800
• Check load balancer target group health - targets should be healthy on port 30800

Kong Gateway Pod Health:

• Verify Kong gateway pod is running and healthy
• Check that Kong is listening on the correct port (default mapped to 30800)
• Ensure Kong domain configuration is correct in values-customer.yaml - verify kong-gateway.kong.domain is properly filled

# Check Kong gateway pod status
kubectl get pods -n traceloop | grep kong

# Check Kong pod logs
kubectl logs -n traceloop -l app=kong

# Verify port mapping and service
kubectl get svc -n traceloop

​

Image Pull Failures

Traceloop images are stored on Docker Hub and require authentication. A regcred secret is provided by Traceloop and must be manually created in the traceloop namespace.

Setup Requirements:

• Create the regcred secret in the traceloop namespace using credentials provided by Traceloop
• Ensure the secret is properly referenced in your Helm values

# Verify the regcred secret exists
kubectl get secret regcred -n traceloop

# Check secret details (without exposing credentials)
kubectl describe secret regcred -n traceloop

Common Issues:

• regcred secret not created in the traceloop namespace
• Incorrect Docker Hub credentials in the secret
• Secret not properly referenced in pod specifications
• Network connectivity issues to Docker Hub

​

Common Error Messages

​

”Error: connection refused”

Check component connectivity:

# Test ClickHouse connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-clickhouse-host 9000

# Test Kafka connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-kafka-broker 9092

# Test PostgreSQL connectivity
kubectl exec -it -n traceloop deploy/traceloop-api -- nc -vz your-postgres-host 5432

​

“Error: authentication failed”

Traceloop uses PropelAuth as the authentication provider. Ensure proper configuration:

Required Configuration:

• Fill out values-customer.yaml with PropelAuth settings:
  Copy

  Ask AI

  customerConfig:
    propelauth:
      authURL: "traceloop-provided"
  
  customerSecret:
    propelauth:
      verifierKey: "traceloop-provided"
      apiKey: "traceloop-provided"
  

Authentication issues:

• Ensure service accounts have necessary permissions
• Verify SSL/TLS configuration if required
• Ensure PropelAuth configuration is correct in values-customer.yaml
• Verify PropelAuth secrets contain the correct keys provided by Traceloop

​

”Error: insufficient permissions”

Permission-related problems:

• Check RBAC configuration
• Verify service account permissions
• Ensure necessary Kubernetes roles are created
• Check database user permissions

​

Quick Validation Checklist

1. Component Connectivity

   • All infrastructure components are reachable
   • Proper ports are open
   • Network policies allow required traffic

2. Authentication & Permissions

   • Database credentials are correct
   • Kubernetes secrets exist and are mounted
   • Service accounts have required permissions

3. Infrastructure Health

   • Kubernetes nodes are healthy
   • Sufficient resources are available
   • Storage is properly configured

4. Logging & Monitoring

   • Check pod logs: kubectl logs -n traceloop -l app=traceloop
   • Monitor resource usage
   • Review infrastructure metrics