Incidents

observability Package

Overview

Package observability provides comprehensive OpenTelemetry instrumentation for the incident management platform.

This package implements distributed tracing, metrics collection, and observability integration to provide production-grade monitoring, debugging, and performance analysis capabilities. It integrates with Prometheus for metrics export and supports various tracing backends including Jaeger, Zipkin, and cloud-native solutions.

Core capabilities:

  • Distributed tracing with automatic HTTP and database instrumentation
  • Custom business metrics for incident lifecycle and system health
  • Structured logging integration with trace correlation
  • Resource detection and service identification
  • Configurable exporters for different deployment environments

Architecture and integration:

  • OpenTelemetry SDK v1.37.0 with auto-instrumentation capabilities
  • Prometheus metrics export with standardized labels and dimensions
  • Trace context propagation across service boundaries
  • Performance-optimized sampling strategies for high-throughput scenarios
  • Resource detection for containerized and cloud environments

Usage example:

// Initialize observability during application startup
shutdown, err := observability.Initialize(ctx, observability.Config{
	ServiceName:      "incident-management",
	ServiceVersion:   "1.0.0",
	Environment:      "production",
	MetricsEndpoint:  ":9090/metrics",
	TracingEndpoint:  "http://jaeger:14268/api/traces",
})
if err != nil {
	log.Fatal("Failed to initialize observability:", err)
}
defer shutdown(ctx)

// Use instrumented HTTP client automatically
client := observability.InstrumentedHTTPClient()
resp, err := client.Get("https://api.external.com/data")

// Add custom business metrics
observability.RecordIncidentCreated(ctx, "web-service", "critical")
observability.RecordConnectorSync(ctx, "servicenow", time.Since(start), err == nil)

Import Path: github.com/systmms/incidents/internal/observability

Types

Config

Config provides comprehensive configuration for OpenTelemetry initialization.

This configuration structure supports various deployment scenarios from development to production environments with flexible exporter configuration and sampling strategies.

{<nil> 25946 type 0 [0x1400059cac0] 0}

Methods

DefaultConfig

DefaultConfig returns a production-ready configuration with sensible defaults.

This configuration enables comprehensive observability suitable for production deployments with Prometheus metrics export and distributed tracing. Environment variables can override default values for deployment-specific customization.

{<nil> <nil> DefaultConfig 0x140001a5be0 <nil>}

metricsRegistry

metricsRegistry holds all custom business metrics for the incident management platform.

This registry provides standardized metrics collection for key business operations including incident lifecycle events, connector health, system performance, and user activity patterns. All metrics follow OpenTelemetry conventions with consistent labeling and dimensional attributes.

{<nil> 519 type 0 [0x1400023bf00] 0}

Methods

newMetricsRegistry

newMetricsRegistry creates and initializes all custom business metrics.

This function sets up comprehensive metrics collection with appropriate units, descriptions, and dimensional attributes for effective monitoring and alerting.

{<nil> <nil> newMetricsRegistry 0x140002a7840 <nil>}

responseWriter

responseWriter wraps http.ResponseWriter to capture response metrics.

This wrapper enables collection of response status codes and content length for comprehensive HTTP request monitoring and performance analysis.

{<nil> 16824 type 0 [0x1400078a600] 0}

telemetryState

telemetryState holds global telemetry instances and shutdown functions.

{<nil> 27954 type 0 [0x1400059d440] 0}

Variables 

var globalTelemetry

Global telemetry state

Functions

ConnectorMiddleware

ConnectorMiddleware provides instrumentation for external connector operations.

This function creates spans for connector synchronization operations with provider attribution and performance tracking for monitoring integration health and reliability.

Example usage:

func (c *ServiceNowConnector) SyncIncident(ctx context.Context, incident *Incident) error {
	ctx, span := observability.ConnectorMiddleware(ctx, "servicenow", "sync_incident")
	defer span.End()

	startTime := time.Now()
	err := c.performSync(ctx, incident)
	duration := time.Since(startTime)

	// Record connector performance
	observability.RecordConnectorSync(ctx, "servicenow", duration, err == nil)

	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
	}

	return err
}

{<nil> <nil> ConnectorMiddleware 0x140000b0720 <nil>}

DatabaseMiddleware

DatabaseMiddleware provides instrumentation for database operations.

This function creates spans for database queries and commands with comprehensive attribute collection for performance monitoring and query optimization analysis.

Example usage:

func (s *Store) GetIncident(ctx context.Context, id string) (*Incident, error) {
	ctx, span := observability.DatabaseMiddleware(ctx, "SELECT", "incidents", "get_incident_by_id")
	defer span.End()

	startTime := time.Now()
	incident, err := s.queryIncident(ctx, id)

	// Record query performance
	observability.RecordDatabaseQuery(ctx, "SELECT", "incidents", time.Since(startTime))

	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
	}

	return incident, err
}

{<nil> <nil> DatabaseMiddleware 0x140000b01a0 <nil>}

GetMeter

GetMeter returns the configured meter for manual metrics collection.

This function provides access to the OpenTelemetry meter for creating custom instruments and recording business-specific metrics.

{<nil> <nil> GetMeter 0x140002367c0 <nil>}

GetTracer

GetTracer returns the configured tracer for manual instrumentation.

This function provides access to the OpenTelemetry tracer for creating custom spans and adding detailed tracing to business logic operations.

Example usage:

tracer := observability.GetTracer()
ctx, span := tracer.Start(ctx, "process-incident")
defer span.End()

span.SetAttributes(
	attribute.String("incident.id", incidentID),
	attribute.String("incident.severity", severity),
)

{<nil> <nil> GetTracer 0x14000231400 <nil>}

HTTPMiddleware

HTTPMiddleware provides comprehensive OpenTelemetry instrumentation for HTTP requests.

This middleware automatically creates spans for incoming HTTP requests, records performance metrics, and propagates trace context for distributed tracing across service boundaries. It includes detailed attribute collection for debugging, monitoring, and performance analysis.

Features:

  • Automatic span creation with HTTP semantic conventions
  • Request and response attribute collection (method, URL, status code, size)
  • Performance metrics integration with business metric recording
  • Error status tracking and span status management
  • Trace context propagation for downstream service calls
  • Configurable attribute collection and sampling

Usage:

router := mux.NewRouter()
router.Use(observability.HTTPMiddleware("incident-api"))

// All routes automatically instrumented
router.HandleFunc("/api/v1/incidents", handleIncidents)

{<nil> <nil> HTTPMiddleware 0x140004db0c0 <nil>}

HealthCheckMiddleware

HealthCheckMiddleware provides basic health check endpoint with observability.

This middleware creates a simple health check endpoint that responds with service status and basic telemetry information for monitoring and load balancer health checks.

{<nil> <nil> HealthCheckMiddleware 0x140001a44c0 <nil>}

Initialize

Initialize sets up comprehensive OpenTelemetry instrumentation for the service.

This function configures distributed tracing, metrics collection, and resource detection based on the provided configuration. It returns a shutdown function that must be called during application termination to ensure proper cleanup and data export.

The initialization process:

  • Creates OpenTelemetry resource with service identification
  • Configures trace provider with sampling and exporters
  • Sets up metrics provider with Prometheus export
  • Registers global providers for auto-instrumentation
  • Initializes custom business metrics collectors

Returns a shutdown function that handles graceful telemetry cleanup.

{<nil> <nil> Initialize 0x140002238a0 <nil>}

InstrumentedHTTPClient

InstrumentedHTTPClient returns an HTTP client with automatic OpenTelemetry instrumentation.

This client automatically creates spans for outgoing HTTP requests and propagates trace context to downstream services. It’s suitable for connector integrations and external API calls.

{<nil> <nil> InstrumentedHTTPClient 0x14000236b20 <nil>}

PolicyMiddleware

PolicyMiddleware provides instrumentation for policy evaluation operations.

This function creates spans for policy engine operations with policy type attribution and decision tracking for monitoring authorization system performance and usage patterns.

Example usage:

func (e *Engine) Evaluate(ctx context.Context, req *Request) (*Decision, error) {
	ctx, span := observability.PolicyMiddleware(ctx, "incident_access", "evaluate")
	defer span.End()

	startTime := time.Now()
	decision, err := e.performEvaluation(ctx, req)
	duration := time.Since(startTime)

	decisionResult := "deny"
	if decision != nil && decision.Allow {
		decisionResult = "allow"
	}

	// Record policy evaluation performance
	observability.RecordPolicyEvaluation(ctx, "incident_access", duration, decisionResult)

	return decision, err
}

{<nil> <nil> PolicyMiddleware 0x140000b0ea0 <nil>}

RecordAPIRequest

RecordAPIRequest records metrics for HTTP API requests.

This function provides comprehensive API performance monitoring with endpoint, method, and status code attribution for identifying bottlenecks and errors.

{<nil> <nil> RecordAPIRequest 0x140002adf40 <nil>}

RecordConnectorSync

RecordConnectorSync records metrics for external connector synchronization operations.

This function tracks connector health, performance, and reliability with provider-specific attribution for monitoring ServiceNow, Jira, PagerDuty, and other integrations.

{<nil> <nil> RecordConnectorSync 0x140002ad5a0 <nil>}

RecordDatabaseQuery

RecordDatabaseQuery records metrics for database query operations.

This function tracks database performance with operation type and table attribution for identifying slow queries and database bottlenecks.

{<nil> <nil> RecordDatabaseQuery 0x1400070a5a0 <nil>}

RecordIncidentCreated

RecordIncidentCreated records metrics when a new incident is created.

This function captures incident creation events with service, severity, and source attribution for comprehensive incident analytics and trending analysis.

{<nil> <nil> RecordIncidentCreated 0x140002a7ea0 <nil>}

RecordIncidentResolved

RecordIncidentResolved records metrics when an incident is resolved.

This function captures resolution metrics including total duration from creation to resolution for MTTR (Mean Time To Recovery) analysis and SLA monitoring.

{<nil> <nil> RecordIncidentResolved 0x140002ac920 <nil>}

RecordPolicyEvaluation

RecordPolicyEvaluation records metrics for policy evaluation operations.

This function tracks policy engine performance and usage patterns with policy type attribution for monitoring authorization system health and decision latency.

{<nil> <nil> RecordPolicyEvaluation 0x1400070b720 <nil>}

RecordUserAction

RecordUserAction records metrics for specific user actions.

This function provides detailed user activity tracking with action type and feature attribution for understanding user workflows and feature adoption patterns.

{<nil> <nil> RecordUserAction 0x14000598600 <nil>}

RecordUserSession

RecordUserSession records metrics for user session events.

This function tracks user engagement and system usage patterns with session lifecycle attribution for monitoring platform adoption and user activity.

{<nil> <nil> RecordUserSession 0x1400070bce0 <nil>}

RecordWebSocketConnection

RecordWebSocketConnection records changes in WebSocket connection count.

This function tracks real-time communication system health with connection lifecycle events for monitoring system capacity and user engagement.

{<nil> <nil> RecordWebSocketConnection 0x1400070a860 <nil>}

RecordWebhookProcessing

RecordWebhookProcessing records metrics for webhook processing operations.

This function monitors webhook reliability and performance with provider attribution for tracking external system integration health and processing delays.

{<nil> <nil> RecordWebhookProcessing 0x1400070b080 <nil>}

WebhookMiddleware

WebhookMiddleware provides instrumentation for webhook processing operations.

This function creates spans for webhook processing with provider attribution and performance tracking for monitoring external system integration reliability.

Example usage:

func (h *WebhookHandler) ProcessPagerDutyWebhook(ctx context.Context, payload []byte) error {
	ctx, span := observability.WebhookMiddleware(ctx, "pagerduty", "process_webhook")
	defer span.End()

	startTime := time.Now()
	err := h.processPayload(ctx, payload)
	duration := time.Since(startTime)

	// Record webhook processing performance
	observability.RecordWebhookProcessing(ctx, "pagerduty", duration, err == nil)

	return err
}

{<nil> <nil> WebhookMiddleware 0x140000b1820 <nil>}

createMeterProvider

createMeterProvider sets up metrics collection with Prometheus export.

{<nil> <nil> createMeterProvider 0x14000230800 <nil>}

createResource

createResource builds an OpenTelemetry resource with comprehensive service identification.

{<nil> <nil> createResource 0x1400022a320 <nil>}

createTracerProvider

createTracerProvider sets up distributed tracing with configurable exporters and sampling.

{<nil> <nil> createTracerProvider 0x1400022b620 <nil>}

getEnvOrDefault

getEnvOrDefault returns an environment variable value or a default if not set.

{<nil> <nil> getEnvOrDefault 0x14000236e40 <nil>}

Generated automatically from Go source code. Last updated: 2025-08-25T07:51:05-04:00

Edit this page on GitHub
Prev
notifications Package
Next
policy Package