Skip to main content

​
Operations Runbook

Procedures for operating Lite Claw in production.

​
Token Rotation

​
Google OAuth Client Secret

When rotating the OAuth client secret:
1

Update in Google Cloud Console

Generate a new client secret in APIs & Services → Credentials.
2

Update in Railway

Set new GOOGLE_OAUTH_CLIENT_SECRET value.
3

Redeploy worker

Trigger a new deployment.
4

Verify

Test /integrations connect calendar with a test account.

​
Token Encryption Key

This is a breaking change. Tokens encrypted with the old key cannot be decrypted. Users must reconnect their integrations.
1

Generate new key

node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"
2

Update TOKEN_ENCRYPTION_KEY in Railway

3

Redeploy

4

Notify users to reconnect

/integrations disconnect calendar
/integrations disconnect gmail
/integrations connect calendar
/integrations connect gmail

​
Incident Response

​
Suspected Token Compromise

If you suspect OAuth tokens have been compromised:
1

Disconnect integrations immediately

/integrations disconnect calendar
/integrations disconnect gmail
2

Revoke in Google

Go to Google Account Security and revoke the app’s access.
3

Rotate secrets

  1. Rotate GOOGLE_OAUTH_CLIENT_SECRET
  2. Rotate TOKEN_ENCRYPTION_KEY
4

Redeploy and reconnect

After redeploying, reconnect integrations through the OAuth flow.

​
Claim Code Abuse

If you see unauthorized claim attempts:
1

Rotate OWNER_CLAIM_CODE

Generate a new claim code and update in Railway.
2

Audit database

Check ownership_state and whitelist tables for unauthorized entries.
3

Remove unauthorized users

Delete any unauthorized IDs from the whitelist.
4

Check audit logs

Look for claim_failed_invalid_code spikes in audit_log.

​
Heartbeat Troubleshooting

​
Quick Checks

  1. Verify cron services are running and calling pnpm heartbeat:run
  2. Check user timezone in user_profiles.timezone
  3. Check schedule in heartbeat_jobs.schedule_cron

​
Log Interpretation

Log EntryMeaningAction
sentHeartbeat deliveredNone (success)
skippedNotDueNot time yetNone (expected)
skippedDuplicateAlready sent this slotNone (Redis dedup working)
failedExecution errorInvestigate immediately

​
Common Issues

  1. Check cron service is deployed and running
  2. Verify HEARTBEAT_JOB_TYPE is set correctly
  3. Check user has heartbeats enabled (/heartbeats)
  4. Verify timezone is set in user profile
  1. Check Redis connection (UPSTASH_REDIS_REST_URL)
  2. Verify slot key TTL is working
  3. Check for multiple cron service instances
  1. Check user_profiles.timezone value
  2. Verify server time vs user expectation
  3. Check cron expression in heartbeat_jobs.schedule_cron

​
Monitoring

​
Key Metrics

MetricAlert ThresholdNotes
claim_failed_invalid_code> 5/hourPossible brute force
heartbeat.failed> 0Investigate immediately
OAuth token refresh failures> 0Token may be revoked
Worker restarts> 3/hourCrash loop
Daily:
  • Scan error logs for exceptions
  • Check heartbeat delivery counts
Weekly:
  • Review OAuth token refresh success rate
  • Check claim attempt patterns
  • Verify cron job execution history

​
Useful Commands

# Run all checks (lint, typecheck, tests, build)
pnpm check

# Run specific test files
pnpm test -- test/google-oauth-service.test.ts
pnpm test -- test/provider-clients.test.ts
pnpm test -- test/cron-lite.test.ts

# Manual heartbeat trigger (for testing)
pnpm heartbeat:run

# Database migrations
pnpm migrate