Grafana, Kibana, Datadog & Splunk

Navigate the major observability platforms test engineers encounter in production environments.

Observability Platform Landscape

Grafana

Grafana connects to data sources (Prometheus, InfluxDB, Loki, Elasticsearch) and visualises metrics and logs.

Key panels for test engineers:

Error Rate Dashboard:
├── Time series: error_rate by service over 24h
├── Stat: current p95 latency
├── Table: top 10 slowest endpoints
└── Logs panel: ERROR-level log stream

Grafana annotations — mark test runs:

# Add annotation when tests start
curl -X POST $GRAFANA_URL/api/annotations \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"Playwright regression started","tags":["test","ci"]}'

This overlays test-run timing on your metrics graphs — visible correlation between test execution and error rate spikes.

Kibana (ELK Stack)

Elasticsearch + Logstash + Kibana. Search and visualise logs.

KQL query examples for test investigation:

# Find all errors for a specific user
service.name: "checkout-api" AND level: "ERROR" AND user.id: "user-123"

# Find slow requests
service.name: "api" AND http.response.duration_ms > 1000

# Find all logs for a trace
trace.id: "abc-def-123"

# Count errors by service in last hour

Kibana Discover vs Dashboard:

• Discover: ad-hoc log investigation (forensics after test failure)
• Dashboard: persistent views of aggregated metrics (SLO monitoring)

Datadog APM

Datadog provides APM (Application Performance Monitoring) with automatic distributed tracing.

// Instrument Node.js for Datadog APM
// dd-trace auto-instruments popular libraries
require('dd-trace').init({
  service: 'checkout-api',
  env: process.env.DD_ENV ?? 'staging',
  version: process.env.DD_VERSION ?? '1.0.0',
  logInjection: true,    // inject traceId into logs automatically
});

Datadog for test engineers:

• Synthetic Tests — scheduled Playwright-like browser tests from global nodes
• CI Visibility — import test results, track flaky tests, failure trends
• Log correlation — filter logs by traceId from a failing trace

# Send Playwright JUnit results to Datadog CI Visibility
DD_API_KEY=$DATADOG_API_KEY \
DD_SITE=datadoghq.com \
npx @datadog/datadog-ci junit upload \
  --service playwright-guide \
  test-results/results.xml

Splunk

Splunk uses SPL (Splunk Processing Language) for powerful log analysis.

-- Find all errors for checkout service in last hour
index=app_logs service=checkout-api level=ERROR earliest=-1h

-- Calculate error rate
index=app_logs service=checkout-api
| stats count(eval(level="ERROR")) as errors, count as total by _time span=5m
| eval error_rate = errors/total*100

-- Find slow API calls
index=app_logs duration_ms>1000
| stats avg(duration_ms) p95(duration_ms) count by endpoint
| sort -p95(duration_ms)

Connecting Test Failures to Observability

test('checkout failure — link to Datadog trace', async ({ page }) => {
  // Capture trace ID from response header
  const traceId = await page.evaluate(async () => {
    const res = await fetch('/api/checkout', { method: 'POST', body: '{}' });
    return res.headers.get('x-datadog-trace-id');
  });

  // On failure: log the observability URL for quick investigation
  if (traceId) {
    console.log(`Datadog trace: https://app.datadoghq.com/apm/trace/${traceId}`);
  }
});

Best practice: Attach the trace URL to the test failure output so on-call engineers jump straight to the trace.