> ## Documentation Index
> Fetch the complete documentation index at: https://docs.liteclaw.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbook

> Day-2 operations, token rotation, and incident response.

# Operations Runbook

Procedures for operating Lite Claw in production.

## Token Rotation

### Google OAuth Client Secret

When rotating the OAuth client secret:

<Steps>
  <Step title="Update in Google Cloud Console">
    Generate a new client secret in **APIs & Services → Credentials**.
  </Step>

  <Step title="Update in Railway">
    Set new `GOOGLE_OAUTH_CLIENT_SECRET` value.
  </Step>

  <Step title="Redeploy worker">
    Trigger a new deployment.
  </Step>

  <Step title="Verify">
    Test `/integrations connect calendar` with a test account.
  </Step>
</Steps>

### Token Encryption Key

<Warning>
  This is a breaking change. Tokens encrypted with the old key cannot be decrypted. Users must reconnect their integrations.
</Warning>

<Steps>
  <Step title="Generate new key">
    ```bash theme={null}
    node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"
    ```
  </Step>

  <Step title="Update TOKEN_ENCRYPTION_KEY in Railway" />

  <Step title="Redeploy" />

  <Step title="Notify users to reconnect">
    ```
    /integrations disconnect calendar
    /integrations disconnect gmail
    /integrations connect calendar
    /integrations connect gmail
    ```
  </Step>
</Steps>

***

## Incident Response

### Suspected Token Compromise

If you suspect OAuth tokens have been compromised:

<Steps>
  <Step title="Disconnect integrations immediately">
    ```
    /integrations disconnect calendar
    /integrations disconnect gmail
    ```
  </Step>

  <Step title="Revoke in Google">
    Go to [Google Account Security](https://myaccount.google.com/permissions) and revoke the app's access.
  </Step>

  <Step title="Rotate secrets">
    1. Rotate `GOOGLE_OAUTH_CLIENT_SECRET`
    2. Rotate `TOKEN_ENCRYPTION_KEY`
  </Step>

  <Step title="Redeploy and reconnect">
    After redeploying, reconnect integrations through the OAuth flow.
  </Step>
</Steps>

### Claim Code Abuse

If you see unauthorized claim attempts:

<Steps>
  <Step title="Rotate OWNER_CLAIM_CODE">
    Generate a new claim code and update in Railway.
  </Step>

  <Step title="Audit database">
    Check `ownership_state` and `whitelist` tables for unauthorized entries.
  </Step>

  <Step title="Remove unauthorized users">
    Delete any unauthorized IDs from the whitelist.
  </Step>

  <Step title="Check audit logs">
    Look for `claim_failed_invalid_code` spikes in `audit_log`.
  </Step>
</Steps>

***

## Heartbeat Troubleshooting

### Quick Checks

1. Verify cron services are running and calling `pnpm heartbeat:run`
2. Check user timezone in `user_profiles.timezone`
3. Check schedule in `heartbeat_jobs.schedule_cron`

### Log Interpretation

| Log Entry          | Meaning                | Action                     |
| ------------------ | ---------------------- | -------------------------- |
| `sent`             | Heartbeat delivered    | None (success)             |
| `skippedNotDue`    | Not time yet           | None (expected)            |
| `skippedDuplicate` | Already sent this slot | None (Redis dedup working) |
| `failed`           | Execution error        | Investigate immediately    |

### Common Issues

<AccordionGroup>
  <Accordion title="Heartbeats not sending">
    1. Check cron service is deployed and running
    2. Verify `HEARTBEAT_JOB_TYPE` is set correctly
    3. Check user has heartbeats enabled (`/heartbeats`)
    4. Verify timezone is set in user profile
  </Accordion>

  <Accordion title="Duplicate heartbeats">
    1. Check Redis connection (`UPSTASH_REDIS_REST_URL`)
    2. Verify slot key TTL is working
    3. Check for multiple cron service instances
  </Accordion>

  <Accordion title="Wrong time delivery">
    1. Check `user_profiles.timezone` value
    2. Verify server time vs user expectation
    3. Check cron expression in `heartbeat_jobs.schedule_cron`
  </Accordion>
</AccordionGroup>

***

## Monitoring

### Key Metrics

| Metric                       | Alert Threshold | Notes                   |
| ---------------------------- | --------------- | ----------------------- |
| `claim_failed_invalid_code`  | > 5/hour        | Possible brute force    |
| `heartbeat.failed`           | > 0             | Investigate immediately |
| OAuth token refresh failures | > 0             | Token may be revoked    |
| Worker restarts              | > 3/hour        | Crash loop              |

### Recommended Checks

**Daily:**

* Scan error logs for exceptions
* Check heartbeat delivery counts

**Weekly:**

* Review OAuth token refresh success rate
* Check claim attempt patterns
* Verify cron job execution history

***

## Useful Commands

```bash theme={null}
# Run all checks (lint, typecheck, tests, build)
pnpm check

# Run specific test files
pnpm test -- test/google-oauth-service.test.ts
pnpm test -- test/provider-clients.test.ts
pnpm test -- test/cron-lite.test.ts

# Manual heartbeat trigger (for testing)
pnpm heartbeat:run

# Database migrations
pnpm migrate
```