ChatOps Troubleshooting

Slack Integration#

Bot Not Responding to Commands#

Symptoms:

• Slash commands return no response
• Bot appears offline or unresponsive

Common Causes and Solutions:

1. Check Bot Token

   # Test bot connection
   ./im connector slack status
   
   # Verify token is valid
   curl -X POST 'https://slack.com/api/auth.test' \
     -H 'Authorization: Bearer xoxb-your-token'

2. Socket Mode Issues

   • If using Socket Mode, ensure the app is connected
   • Check: Slack App Settings → Socket Mode → Connections

3. Request URL Not Reachable

   • For HTTP mode, verify the endpoint is publicly accessible
   • Check: Slack App Settings → Interactivity & Shortcuts

Slash Commands Return “dispatch_failed”#

Symptoms:

• Commands show “dispatch_failed” error
• Logs show timeout errors

Solutions:

1. Response Time

   • Slack requires acknowledgment within 3 seconds
   • Use async response for long-running operations:

   // Acknowledge immediately
   w.WriteHeader(http.StatusOK)
   // Process and respond via response_url

2. Check Request URL

   • Must be HTTPS with valid certificate
   • Must respond to Slack’s verification challenge

Missing OAuth Scopes#

Symptoms:

• missing_scope errors
• Some features work, others don’t

Solutions:

1. Required Scopes

   channels:history
   channels:read
   chat:write
   commands
   im:history
   im:write
   users:read

2. Reinstall App

   • After adding scopes, users must reinstall
   • Go to: Slack App Settings → Install App → Reinstall

Notifications Not Delivered#

Symptoms:

• Incident updates don’t post to channels
• No errors in logs

Debugging:

1. Check Channel Configuration

   ./im chatops slack list-channels
   # Verify target channels are configured

2. Bot Membership

   • Bot must be invited to private channels
   • Use /invite @incident-bot in the channel

3. Channel ID vs Name

   • Use channel IDs (C0123ABC) not names (#general)
   • Names can change; IDs are stable

Microsoft Teams Integration#

Bot Not Responding#

Symptoms:

• Messages to bot get no response
• Bot shows as offline

Solutions:

1. Check App Registration

   ./im connector teams status

2. Verify Bot Framework Registration

   • Check: Azure Portal → Bot Services → [your bot]
   • Ensure messaging endpoint is correct

3. Certificate Issues

   • Teams requires valid TLS certificates
   • Self-signed certificates won’t work

Adaptive Cards Not Rendering#

Symptoms:

• Messages appear as raw JSON
• Cards show “Unable to render”

Solutions:

1. Card Version

   • Teams supports Adaptive Cards up to version 1.5
   • Check card schema version in templates

2. Unsupported Elements

   • Some Adaptive Card elements aren’t supported
   • Test cards at: https://adaptivecards.io/designer/

Permission Errors#

Symptoms:

• 403 Forbidden when posting messages
• Unauthorized errors in logs

Solutions:

1. Check App Permissions

   • Navigate to: Azure Portal → App Registration → API Permissions
   • Required: ChannelMessage.Send, Chat.ReadWrite

2. Admin Consent

   • Some permissions require admin consent
   • Contact your Teams administrator

Discord Integration#

Bot Offline#

Symptoms:

• Bot shows as offline in Discord
• Commands don’t work

Solutions:

1. Check Bot Token

   ./im connector discord status

2. Gateway Connection

   • Verify bot has GUILDS and GUILD_MESSAGES intents
   • Check: Discord Developer Portal → Bot → Privileged Gateway Intents

3. Bot Permissions

   • Ensure bot role has necessary permissions in server
   • Check: Server Settings → Roles → [bot role]

Slash Commands Not Registered#

Symptoms:

• Commands don’t appear in Discord
• “Unknown command” errors

Solutions:

1. Register Commands

   ./im connector discord sync-commands

2. Guild vs Global Commands

   • Guild commands appear immediately
   • Global commands take up to 1 hour to propagate

3. Bot Scope

   • Bot must be invited with applications.commands scope
   • Reinvite if scope was missing initially

Common Issues (All Platforms)#

Real-Time Updates Not Working#

Symptoms:

• Incident updates don’t appear in chat
• Manual refresh required

Debugging:

1. Check Event Bus Connection

   ./im events status
   # Should show connected and subscribed

2. Verify Subscription

   ./im chatops subscriptions list

3. Check WebSocket/SSE

   • Some networks block WebSocket connections
   • Try falling back to SSE if available

Rate Limiting#

Symptoms:

• Messages delayed or dropped
• 429 Too Many Requests errors

Solutions:

1. Implement Backoff

   • All chat platforms have rate limits
   • Platform implements exponential backoff automatically

2. Batch Notifications

   ./im chatops configure --batch-window 5s

3. Priority Queuing

   • Critical notifications skip the queue
   • Non-critical updates are batched

Message Formatting Issues#

Symptoms:

• Rich formatting not displaying
• Markdown rendering incorrectly

Solutions:

1. Platform-Specific Formatting

   Platform	Format
   Slack	mrkdwn (Slack-flavored markdown)
   Teams	Adaptive Cards or HTML
   Discord	Discord markdown

2. Check Template Configuration

   ./im chatops templates list
   ./im chatops templates validate

Metrics and Observability#

ChatOps Metrics#

# Query metrics
curl -s http://localhost:9090/metrics | grep chatops_

Enable Debug Logging#

export LOG_LEVEL=debug
export CHATOPS_DEBUG=true
./im serve

Getting Help#

If issues persist:

1. Enable debug logging and collect relevant log sections
2. Check the GitHub Issues for similar problems
3. Open a new issue with:
   • Platform (Slack/Teams/Discord) and version
   • Error messages and log excerpts
   • Bot configuration (redact tokens!)
   • Steps to reproduce