Add custom application metrics to your existing demo app using the OpenTelemetry SDK. You'll create counters, gauges, and histograms for business logic, verify them in Prometheus, and build Grafana dashboards. Starting state: ~/observability-lab/ running from Module 1 (Prometheus, Grafana, OTel Collector, demo app code at ~/observability-lab/app/). After this lesson: demo app enhanced with custom metrics (app_http_requests_total, app_orders_processed_total, app_active_connections, app_http_request_duration_seconds) and new endpoints (/order, /slow, /error).
Auto-instrumentation from Lesson 103 gives you HTTP request metrics for free. Custom metrics let you measure business-specific things that auto-instrumentation cannot see: orders placed, cache hits, queue depth, or payment processing time.
cat <<'EOF'
=== AUTO vs CUSTOM METRICS ===
Auto-instrumentation gives you (from Lesson 103):
http_server_request_duration_seconds - Request latency histogram
http_server_active_requests - Current in-flight requests
Custom metrics let you measure (what you will add now):
app_http_requests_total - Request count with status codes
app_orders_processed_total - Business event counter
app_active_connections - Connection pool gauge
app_http_request_duration_seconds - Custom latency histogram
Rule: Start with auto-instrumentation. Add custom metrics only for
things the auto-instrumentation cannot see.
EOFAuto-instrumentation captures the 'shape' of HTTP traffic. Custom metrics capture the 'meaning' — how many orders were placed, how deep the queue is, how long a payment took. The OTel Meter API lets you define counters, gauges, and histograms that flow through the same OTel Collector pipeline your auto-instrumented metrics already use.
You see the distinction between auto-instrumented and custom metrics, plus the four custom metrics you will add in this lesson.