Incidents

ChatOps Integration

Overview

The incident management platform provides comprehensive ChatOps integration that works with multiple chat platforms. This enables teams to manage incidents directly from their preferred collaboration tools through slash commands, bot mentions, and interactive elements.

Supported Platforms:

  • Slack
  • Microsoft Teams
  • Discord
  • Mattermost
  • Extensible for additional platforms

Features

✅ Multi-platform Support: Works with Slack, Teams, Discord, Mattermost, and more
✅ Slash Commands: Comprehensive incident management commands
✅ Interactive Notifications: Rich incident notifications with action buttons
✅ Bot Mentions: Natural language queries and commands
✅ Real-time Updates: Automatic incident status notifications
✅ Flexible Routing: Channel routing based on incident severity
✅ Webhook Support: Bi-directional integration with chat platforms

Quick Start

Step 1: Add a Chat Provider

Interactive Configuration 

# Add Slack provider interactively
./im chatops add slack company-slack

# You'll be prompted for:
# - Bot token (xoxb-...)
# - Default channel ID
# - Slash command configuration
# - Notification settings

JSON Configuration 

# Add with pre-configured settings
./im chatops add slack company-slack --config '{
  "bot_token": "xoxb-your-bot-token",
  "default_channel": "C1234567890",
  "enable_slash_commands": true,
  "enable_notifications": true,
  "notification_channels": {
    "SEV-1": "C_CRITICAL_INCIDENTS",
    "SEV-2": "C_MAJOR_INCIDENTS",
    "SEV-3": "C_MINOR_INCIDENTS",
    "SEV-4": "C_INFO_INCIDENTS"
  }
}'

Step 2: Test the Connection 

./im chatops test company-slack

# Output:
✓ Connected to Slack workspace
✓ Bot permissions verified
✓ Default channel accessible
✓ Slash commands configured

Step 3: Send Test Notification 

./im chatops notify company-slack INC-12345

# Sends a test incident notification to verify integration

Platform-Specific Setup

Slack Integration

Prerequisites

  • Slack workspace admin access
  • Ability to create and install apps

Setup Steps

  1. Create Slack App:

    • Go to https://api.slack.com/apps
    • Click “Create New App” → “From scratch”
    • Name: Incident Management Bot
    • Workspace: Select your workspace

  2. Configure Bot Scopes:

    • Go to OAuth & Permissions
    • Add Bot Token Scopes: 
      chat:write        # Send messages
commands          # Respond to slash commands
app_mentions:read # Read bot mentions
channels:read     # List channels
users:read        # Look up users

  3. Install App:

    • Click “Install to Workspace”
    • Copy Bot User OAuth Token (starts with xoxb-)

  4. Configure Slash Commands:

    • Go to Slash Commands → “Create New Command”
    • Command: /im
    • Request URL: https://your-domain.com/chatops/slack-company
    • Description: Manage incidents
    • Usage Hint: declare --title "Issue description"

  5. Enable Events:

    • Go to Event Subscriptions
    • Enable Events: On
    • Request URL: https://your-domain.com/chatops/slack-company
    • Subscribe to Bot Events: app_mention

Slack Features

✅ Slash commands (/im declare, /im ack, etc.)
✅ Interactive buttons and menus
✅ Rich notifications with attachments
✅ Bot mentions with natural language
✅ Channel management
✅ User lookups

Microsoft Teams Integration

Prerequisites

  • Microsoft Teams admin access
  • Azure Bot Service access

Setup Steps

  1. Create Bot Registration:

    • Go to https://portal.azure.com
    • Create Azure Bot resource
    • Note App ID and generate App Password

  2. Configure Messaging:

    • Set Messaging endpoint: https://your-domain.com/chatops/teams-company
    • Enable Microsoft Teams channel

  3. Create Teams App:

    • Use App Studio or Developer Portal
    • Configure bot with App ID from step 1
    • Add slash command: im

  4. Deploy to Teams:

    • Upload app package to Teams
    • Install for your organization

Teams Features

✅ Slash commands via Bot Framework
✅ Rich notifications with Adaptive Cards
✅ Bot mentions
🔶 Interactive elements (limited support)
✅ Channel management
✅ User lookups

Discord Integration

Prerequisites

  • Discord server admin access
  • Discord Developer Portal access

Setup Steps

  1. Create Discord App:

    • Go to https://discord.com/developers/applications
    • Click “New Application”
    • Name: Incident Management Bot

  2. Create Bot:

    • Go to Bot section
    • Click “Add Bot”
    • Copy bot token

  3. Add to Server:

    • Go to OAuth2 → URL Generator
    • Scopes: bot, applications.commands
    • Permissions: Send Messages, Use Slash Commands
    • Use generated URL to invite bot

  4. Configure Slash Commands:

    • Commands are registered automatically via Discord API
    • No manual configuration needed

Discord Features

✅ Slash commands
✅ Rich embeds for notifications
✅ Bot mentions
✅ Interactive buttons
✅ Channel management
✅ User lookups

Slash Commands Reference

All platforms support these core incident management commands:

Incident Lifecycle 

/im declare [title]                    # Create new incident
/im ack [incident-id]                  # Acknowledge incident
/im resolve [incident-id] [note]       # Resolve incident
/im close [incident-id]                # Close incident

Information & Status 

/im status [incident-id]               # Show incident details
/im list                               # List open incidents
/im help                               # Show help message

Advanced Commands 

/im escalate [incident-id] [severity]  # Change severity
/im assign [incident-id] [user]        # Assign to user
/im update [incident-id] [note]        # Add timeline update
/im timeline [incident-id]             # Show incident timeline

Example Usage

Declaring Incidents 

# Simple declaration
/im declare Database connection timeout

# With severity and service
/im declare --sev SEV-2 --service checkout "Payment processing down"

# With full details
/im declare --sev SEV-1 --service payments --description "Complete payment system outage affecting all customers"

Managing Incidents 

# Acknowledge with note
/im ack INC-2024-001 "Investigating MySQL cluster"

# Resolve with detailed note
/im resolve INC-2024-001 "Fixed by restarting MySQL primary. Root cause: connection pool exhaustion"

# Quick status check
/im status INC-2024-001

Interactive Notifications

Incident Alerts

When incidents are created or updated, rich notifications are sent to configured channels:

Slack Example 

🚨 **New Incident Declared**
**INC-2024-001**: Database connection timeout

**Severity**: SEV-1  |  **Status**: Open  |  **Service**: Checkout
**Started**: 2 minutes ago

*Multiple users reporting payment failures*

[Acknowledge] [Join War Room] [View Details] [Escalate]

Teams Example (Adaptive Card) 

{
  "type": "AdaptiveCard",
  "body": [
    {
      "type": "TextBlock",
      "text": "🚨 New Incident Declared",
      "weight": "Bolder",
      "color": "Attention"
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "ID", "value": "INC-2024-001" },
        { "title": "Severity", "value": "SEV-1" },
        { "title": "Service", "value": "Checkout" }
      ]
    }
  ],
  "actions": [
    { "type": "Action.Submit", "title": "Acknowledge" },
    { "type": "Action.Submit", "title": "Join War Room" }
  ]
}

Interactive Elements

Quick Actions

  • Acknowledge: Take ownership of incident
  • Escalate: Change severity level
  • Assign: Assign to team member
  • Status Update: Add timeline note
  • Join War Room: Create/join dedicated incident channel

Selection Menus

  • Severity Selection: SEV-1 through SEV-4
  • Service Selection: From configured service list
  • User Assignment: From team member list
  • Status Transition: Available status changes

Notification Routing

Configure different channels for different incident severities:

{
  "notification_channels": {
    "SEV-1": "C_CRITICAL_INCIDENTS",     # High-priority alerts
    "SEV-2": "C_MAJOR_INCIDENTS",        # Team notification channel
    "SEV-3": "C_MINOR_INCIDENTS",        # General incident channel
    "SEV-4": "C_INFO_INCIDENTS",         # Low-priority notifications
    "default": "C_GENERAL_INCIDENTS"     # Fallback channel
  },
  "escalation_channels": {
    "sla_breach": "C_SLA_ALERTS",        # SLA breach notifications
    "management": "C_EXECUTIVE_ALERTS"   # Executive escalations
  }
}

Channel Routing Rules

  • SEV-1 incidents → Critical channel + management alerts
  • SLA breaches → Escalation to management channel
  • Resolution updates → Original incident channel
  • Metrics summaries → Weekly reports to team channels

Bot Mentions & Natural Language

Enable conversational interaction by mentioning the bot:

Query Examples 

@incident-bot what incidents are currently open?
@incident-bot show me the status of INC-2024-001
@incident-bot how many SEV-1 incidents this week?
@incident-bot who is on call for payments service?

Command Examples 

@incident-bot create an incident for API errors
@incident-bot acknowledge INC-2024-001
@incident-bot resolve the database incident
@incident-bot escalate to SEV-1

Natural Language Processing

The bot processes natural language and converts to appropriate commands:

  • Intent recognition: Understands “create”, “acknowledge”, “resolve”
  • Entity extraction: Finds incident IDs, severities, services
  • Context awareness: Remembers previous conversation context
  • Clarification: Asks for missing information

CLI Management

List Providers 

./im chatops list

# Output:
Provider      Platform    Status    Channels
company-slack slack       active    5
corp-teams    teams       active    3
dev-discord   discord     inactive  1

Provider Management 

# Add provider
./im chatops add slack production-slack

# Update configuration
./im chatops update production-slack --config '{
  "notification_channels": {
    "SEV-1": "C_NEW_CRITICAL_CHANNEL"
  }
}'

# Remove provider
./im chatops remove production-slack

# Test connectivity
./im chatops test production-slack

Send Notifications 

# Send test notification
./im chatops notify production-slack INC-12345

# Send custom message
./im chatops message production-slack "System maintenance starting in 10 minutes"

# Broadcast to all providers
./im chatops broadcast "Planned maintenance window: 2AM-4AM EST"

API Integration

REST API Endpoints

Provider Management 

# List providers
GET /api/v1/chatops/providers

# Add provider
POST /api/v1/chatops/providers
{
  "name": "my-slack",
  "type": "slack",
  "config": {
    "bot_token": "xoxb-...",
    "default_channel": "C1234567890"
  }
}

# Test provider
POST /api/v1/chatops/providers/my-slack/test

# Send notification
POST /api/v1/chatops/providers/my-slack/notify
{
  "type": "incident_created",
  "incident_id": "INC-123",
  "data": {
    "title": "New Incident",
    "severity": "SEV-2",
    "message": "Database issues detected"
  }
}

Webhook Endpoints 

# Generic provider webhook
POST /chatops/{provider-name}

# Platform-specific endpoints
POST /chatops/slack/{instance-name}      # Slack events & interactions
POST /chatops/teams/{instance-name}      # Teams Bot Framework
POST /chatops/discord/{instance-name}    # Discord interactions

Security & Compliance

Authentication

  • Bot tokens securely stored and encrypted
  • Webhook signature validation for all platforms
  • User identity verification through platform APIs
  • Service account mapping between chat and incident systems

Authorization

  • RBAC integration with existing policy system
  • Command restrictions based on user roles
  • Channel-based permissions for sensitive operations
  • Audit trail for all ChatOps actions

Data Privacy

  • Configurable message retention policies
  • Sensitive data redaction in logs and notifications
  • Compliance with chat platform privacy policies
  • GDPR compliance for user data handling

Integration with Incident Lifecycle

Automatic Notifications

  • Incident Created → Notification to appropriate channels
  • Status Changes → Updates broadcast to incident channels
  • SLA Breaches → Escalation alerts to management channels
  • Resolution → Summary notifications with metrics

Timeline Integration

All ChatOps actions are recorded in the incident timeline:

{
  "event_type": "chatops.acknowledge",
  "timestamp": "2025-08-23T15:30:00Z",
  "actor": "chatops:user123",
  "source": "slack:company-slack",
  "data": {
    "channel": "C1234567890",
    "command": "/im ack INC-123",
    "response": "Incident acknowledged"
  }
}

Advanced Configuration

Custom Commands

Extend the command system with organization-specific commands:

// Register custom command handler
manager.RegisterCommandHandler("deploy", func(cmd *SlashCommand) (*InteractionResponse, error) {
    // Custom deployment command logic
    return &InteractionResponse{
        Text: "Deployment initiated for " + cmd.Args[0],
    }, nil
})

Message Templates

Customize notification formatting per platform:

templates:
  incident_created:
    slack: |
      🚨 *{{.Title}}*
      {{.Description}}
      Severity: {{.Severity}} | Service: {{.Service}}
    teams: |
      **{{.Title}}**

      {{.Description}}

      **Severity:** {{.Severity}}
      **Service:** {{.Service}}
    discord: |
      🚨 **{{.Title}}**
      ```
      {{.Description}}
      ```
      **Severity:** {{.Severity}} | **Service:** {{.Service}}

Workflow Integration

Trigger external workflows from chat commands:

# GitHub Actions workflow triggered by ChatOps
on:
  repository_dispatch:
    types: [chatops-deploy]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy Application
        run: |
          echo "Deploying ${{ github.event.client_payload.service }}"

Monitoring & Analytics

Metrics Tracked

  • Command usage by platform and user
  • Response times for interactive elements
  • Notification delivery success rates
  • User engagement with incident notifications
  • Error rates by platform and command

Analytics Dashboard

  • Most used commands by team and time period
  • Average response time from notification to acknowledgment
  • Channel activity during incident response
  • Bot mention patterns and natural language usage

Logging & Debugging 

# Enable debug logging
export CHATOPS_DEBUG=true
./im serve

# View recent ChatOps events
./im timeline --source chatops --limit 20

# Check provider health
./im chatops health --provider company-slack

Best Practices

Channel Organization

  • Dedicated incident channels per severity level
  • Service-specific channels for team notifications
  • Executive channels for management escalations
  • Archive channels for closed incidents

Command Usage

  • Use descriptive titles in declare commands
  • Include context in acknowledgment notes
  • Leverage interactive elements over manual typing
  • Follow incident lifecycle (declare → ack → resolve → close)

Bot Configuration

  • Descriptive bot names and avatars (e.g., “Incident Bot” with alert icon)
  • Minimal required permissions for security
  • Proper webhook security with signature validation
  • Regular token rotation for enhanced security

Team Adoption

  • Training sessions on available commands
  • Integration with existing workflows
  • Regular usage analytics review
  • Feedback collection for improvements

Troubleshooting

Common Issues

Commands Not Working

Symptoms: Slash commands return errors or no response

Solutions:

# Verify slash command configuration in platform
/im help

# Check bot permissions
./im chatops test company-slack

# Validate webhook endpoints
curl -X POST https://your-domain.com/chatops/company-slack \
  -H "Content-Type: application/json" \
  -d '{"type": "ping"}'

Notifications Not Sent

Symptoms: Incident notifications not appearing in channels

Solutions:

# Test provider connection
./im chatops test company-slack

# Verify channel permissions
./im chatops channels --provider company-slack

# Check notification routing
./im chatops config --provider company-slack

Interactive Elements Not Responding

Symptoms: Buttons and menus don’t work

Solutions:

# Confirm interactivity enabled in platform
# Validate webhook signature verification
# Check request/response format compatibility

# Test with simple interaction
./im chatops interact-test company-slack

Debug Commands 

# Check all provider status
./im chatops status

# View recent webhook events
./im chatops webhooks --recent 10

# Test specific functionality
./im chatops test-notifications --provider company-slack
./im chatops test-commands --provider company-slack

Extensibility

Adding New Providers

  1. Implement Provider interface
  2. Register with provider registry
  3. Add CLI configuration support
  4. Create platform-specific webhook handlers
  5. Update documentation

Custom Interactive Elements

Create platform-specific interactive components:

type CustomButton struct {
    ID          string
    Text        string
    Style       ButtonStyle
    Handler     func(*Interaction) (*InteractionResponse, error)
}

External System Integration

Connect ChatOps to external systems:

  • CI/CD pipelines (trigger deployments)
  • Monitoring systems (query metrics)
  • Documentation platforms (create runbooks)
  • Calendar systems (schedule maintenance)

This comprehensive ChatOps integration transforms how teams interact with incident management, bringing the full power of the platform directly into daily collaboration workflows.

Edit this page on GitHub
Prev
Testing & Validation
Next
Kubernetes Deployment