Skip to content

Operations Runbook

Start The API

Pre-conditions

Use the V2 backend workspace at /home/user/centurion-staker/centurion-seat-manager. Configure CHAIN_ID, DEPOSIT_CONTRACT_ADDRESS, VAULT_FACTORY_ADDRESS, PRINCIPAL_DESTINATION_ADDRESS, VAULT_INIT_CODE_HASH, ENROLLMENT_JWT_PUBLIC_KEY_PEM, and P10_RESERVE_COVERAGE_LIMIT_WEI; the minimum environment list is documented in seat-manager/README.md:93. For Postgres, set ONBOARDING_STORE_DRIVER=postgres and ONBOARDING_POSTGRES_URL, which are parsed in seat-manager/src/config/loader.ts:101.

Steps

  1. Install dependencies.
  2. Build TypeScript and Prisma client.
  3. Run migrations before the API uses Postgres.
  4. Start the API process.
cd /home/user/centurion-staker/centurion-seat-manager
npm install
npm run build
ONBOARDING_POSTGRES_URL=postgresql://user:pass@host:5432/seat_manager npm run prisma:migrate:deploy
npm run start

The script npm run start maps to node dist/src/index.js api in seat-manager/package.json:36, and the api command calls runApiServer in seat-manager/src/index.ts:34.

Expected output

[seat-manager-v2] bootstrap command=api chainId=<chain_id> mode=<mode>
seat-manager-v2 API listening on http://<host>:<port> (seed seats=<n>, store=postgres)

The listening log is emitted by runApiServer in seat-manager/src/api/runtime.ts:243.

Troubleshooting

  • Fatal startup error: SEAT_MANAGER_ADMIN_ENABLE=true requires SEAT_MANAGER_ADMIN_USERS_JSON means admin mode was enabled without users. Set SEAT_MANAGER_ADMIN_USERS_JSON or disable admin; the guard is in seat-manager/src/config/loader.ts:381.
  • SEAT_MANAGER_ADMIN_SESSION_SECRET with length >= 32 means the admin secret is missing or too short. Set a production secret; the guard is in seat-manager/src/config/loader.ts:387.
  • Database connection failure means ONBOARDING_POSTGRES_URL is wrong or migrations did not run. Re-run npm run prisma:migrate:deploy, which is defined in seat-manager/package.json:19.

Rollback

Stop the API process first. Restore the previous environment file or deployment secret set, then restart npm run start. If a migration applied and the schema is wrong, follow the migration rollback procedure below; do not run the API against a half-rolled schema.

Start The Watcher

Pre-conditions

The API build must exist, the store must be durable, and execution credentials must match the selected operation mode. watch starts both operation execution and lifecycle watching in seat-manager/src/api/runtime.ts:313. Set SEAT_MANAGER_WATCH_POLL_MS, ETH_RPC_URL, and optionally SEAT_MANAGER_CL_BEACON_API_URL; the watch variables are documented in seat-manager/README.md:114.

Steps

  1. Start a dedicated worker process.
  2. Keep the API process separate unless a single combined process is required.
  3. Use api-watch only when one process must host both API and workers.
cd /home/user/centurion-staker/centurion-seat-manager
node dist/src/index.js watch

For a combined process:

node dist/src/index.js api-watch

watch and api-watch are valid runtime commands in seat-manager/src/index.ts:11.

Expected output

[seat-manager-v2] bootstrap command=watch chainId=<chain_id> mode=<mode>
seat-manager-v2 watch loop started (store=postgres, poll_ms=1000)

The watch-loop log is emitted at seat-manager/src/api/runtime.ts:328. When the process stops cleanly, it prints processed, succeeded, failed, and requeued counts in seat-manager/src/api/runtime.ts:346.

Troubleshooting

  • No lifecycle progress and /admin/v1/health shows missing el_last_scanned_block: verify ETH_RPC_URL and deposit contract address. The EL watcher returns early when the RPC URL is missing in seat-manager/src/watchers/lifecycleWatchLoop.ts:333.
  • Repeated lifecycle watcher loop error means RPC, log decoding, or block reads are failing; the catch path logs and delays without advancing the cursor in seat-manager/src/watchers/lifecycleWatchLoop.ts:504.
  • Repeated operation failures mean the executor adapter is failing; non-retryable errors mark the operation failed in seat-manager/src/watchers/operationWorker.ts:190.

Rollback

Stop the watcher. If it wrote bad lifecycle evidence, do not edit rows manually; move affected seats to EXCEPTION through a documented admin/governance path or restore a database snapshot. If the issue is only a stuck operation, use the stuck-operation procedure.

Seed Seats Today

Pre-conditions

Foundation Ops must have a unique seatId and operatorId. No non-revoked seat can already exist for that operator; the store checks this and the database enforces it in seat-manager/src/onboarding/prismaStore.ts:296 and seat-manager/prisma/migrations/20260514165745_one_active_seat_per_operator/migration.sql:37.

Steps

  1. Add the seed JSON to the API or API+watch environment.
  2. Restart the process so startup seeding runs.
  3. Confirm the seat appears as CREATED.
export SEAT_MANAGER_CREATED_SEATS_JSON='[{"seatId":"seat_001","operatorId":"operator_001"}]'
npm run start

The parser requires a JSON array with non-empty seatId and operatorId in seat-manager/src/api/runtime.ts:17. Startup calls seedCreatedSeat for each row in seat-manager/src/onboarding/prismaStore.ts:263.

Expected output

seat-manager-v2 API listening on http://<host>:<port> (seed seats=1, store=postgres)

Re-seeding the same seat id for the same operator is idempotent in seat-manager/src/onboarding/prismaStore.ts:282.

Troubleshooting

  • SEAT_MANAGER_CREATED_SEATS_JSON must be a JSON array means the environment variable is not an array; the parser raises this at seat-manager/src/api/runtime.ts:22.
  • Each seed seat must include non-empty seatId and operatorId means one object is missing a required field; the parser raises this at seat-manager/src/api/runtime.ts:30.
  • OPERATOR_ALREADY_HAS_ACTIVE_SEAT means the operator already has a non-revoked seat. Revoke the old seat with governance approval before retrying.

Rollback

If the seat was seeded in error and remains CREATED, revoke it through the admin console so the audit trail records the decision. Do not delete the row; Seat history is retained and revocation is the lifecycle path.

Rotate The Executor Key

Pre-conditions

The backend is running in owner-send execution mode with SEAT_MANAGER_OWNER_PRIVATE_KEY configured. Sponsored claims use the configured owner key as claim executor; beneficiary private keys are not required by the backend in seat-manager/README.md:121. Existing live vaults need claim executor grants for the new signer.

Steps

  1. Stop API and watcher processes.
  2. Replace SEAT_MANAGER_OWNER_PRIVATE_KEY in the runtime secret store.
  3. Start the API.
  4. Run the grant backfill.
  5. Start the watcher.
  6. Confirm claim signer readiness through status.
cd /home/user/centurion-staker/centurion-seat-manager
export SEAT_MANAGER_OWNER_PRIVATE_KEY=0x<new_private_key>
npm run start
node dist/src/index.js backfill-claim-executor
node dist/src/index.js watch

The backfill command requires owner-send mode and scans live non-revoked deposited/CL-visible/active seats in seat-manager/src/api/runtime.ts:351.

Expected output

[seat-manager-v2] bootstrap command=backfill-claim-executor chainId=<chain_id> mode=OWNER_SEND
seat-manager-v2 claim executor grant backfill complete scanned=<n> updated=<n>

Grant TTL defaults to 30 days in the operation executor constants, and SEAT_MANAGER_CLAIM_EXECUTOR_GRANT_EXPIRES_AT_UNIX can set a fixed expiry in seat-manager/src/contracts/operationExecutor.ts:630.

Troubleshooting

  • backfill-claim-executor requires SEAT_MANAGER_OPERATION_MODE=OWNER_SEND means the runtime did not resolve owner-send mode; the guard is in seat-manager/src/api/runtime.ts:351.
  • Claim requests fail with Claim sponsorship signer is not configured when the owner key is absent; preflight reports that in seat-manager/src/contracts/operationExecutor.ts:225.
  • Claim requests fail with missing executor scope when the controller grant is absent or expired; the backend checks claimExecutorAuthorized before enqueueing in seat-manager/src/api/server.ts:2151.

Rollback

Stop workers, restore the previous SEAT_MANAGER_OWNER_PRIVATE_KEY, restart the API, and run the grant backfill for the previous signer. Then use status on a live seat to prove claim_signer_ready is true before allowing claim traffic.

Handle A Stuck Operation

Pre-conditions

You need admin console access with OPERATOR or ADMIN role for retry/cancel, an active session, CSRF token, and fresh step-up. Read-only review is available to VIEWER, OPERATOR, and ADMIN, while retry and cancel require higher roles in seat-manager/src/api/server.ts:3151, seat-manager/src/api/server.ts:3260, and seat-manager/src/api/server.ts:3333.

Steps

  1. Open Admin Console -> Operations.
  2. Filter by status and kind.
  3. Open the operation detail or use the API.
  4. Retry only failed or proposed.
  5. Cancel only queued, processing, or proposed.
  6. Add a reason note and complete step-up.
curl -sS -b admin.cookie https://seat-manager.example.org/admin/v1/operations?seat_id=<seat_id>
curl -sS -b admin.cookie https://seat-manager.example.org/admin/v1/operations/<operation_id>
curl -sS -X POST -b admin.cookie \
  -H "X-CSRF-Token: <csrf>" \
  -H "Idempotency-Key: <uuid>" \
  -H "Content-Type: application/json" \
  -d '{"reason_code":"MANUAL_RETRY","note":"retry after RPC recovery"}' \
  https://seat-manager.example.org/admin/v1/operations/<operation_id>/retry

Expected output

{"ok":true,"action":"operation.retry","operation_id":"op_...","status":"queued"}

Cancel returns:

{"ok":true,"action":"operation.cancel","operation_id":"op_...","status":"failed"}

The retry response is built at seat-manager/src/api/server.ts:3301, and the cancel response is built at seat-manager/src/api/server.ts:3374.

Troubleshooting

  • Retry rejected with Operation retry not allowed from status <status> means the operation is not failed or proposed; the gate is seat-manager/src/api/server.ts:3290.
  • Cancel rejected with Operation cancel not allowed from status <status> means it is not queued, processing, or proposed; the gate is seat-manager/src/api/server.ts:3363.
  • Step-up rejected means the session has no fresh step-up window; run the step-up dialog again. Freshness is enforced in seat-manager/src/api/adminAuth.ts:342.
  • Stale processing rows re-enter the claim path when the lease expires; claimNextQueuedOperation scans expired processing leases in seat-manager/src/onboarding/prismaStore.ts:1318.

Rollback

Retry rollback is another cancel if the operation is still queued, processing, or proposed. Cancel rollback is a retry only when the failed row represents work that is safe to requeue; admin retry moves it back to queued and records admin.operation.retry in seat-manager/src/api/server.ts:3316. Every action emits an audit event, so add an audit note that explains the reversal.

Roll Back A Migration

Pre-conditions

There is no automatic rollback command in this repo. You need a database snapshot, the exact migration file, and confirmation that API and watcher processes are stopped. Prisma migration deploy is defined in seat-manager/package.json:19, and the current schema depends on raw SQL indexes such as uniq_operations_live_seat_kind and uniq_seats_active_operator_id in seat-manager/prisma/migrations/20260507080000_security_remediation_hardening/migration.sql:19 and seat-manager/prisma/migrations/20260514165745_one_active_seat_per_operator/migration.sql:37.

Steps

  1. Stop API, watcher, and admin writes.
  2. Take a fresh database snapshot.
  3. Identify the migration to reverse under seat-manager/prisma/migrations/.
  4. Write a manual reversal SQL file reviewed by another engineer.
  5. Apply it in a maintenance window.
  6. Run npm run prisma:migrate:deploy to confirm Prisma metadata state.
  7. Start API in status mode before opening traffic.
cd /home/user/centurion-staker/centurion-seat-manager
node dist/src/index.js status
ONBOARDING_POSTGRES_URL=postgresql://user:pass@host:5432/seat_manager npm run prisma:migrate:deploy
node dist/src/index.js status

The status command prints runtime status without starting the API in seat-manager/src/index.ts:49.

Expected output

[seat-manager-v2] bootstrap command=status chainId=<chain_id> mode=<mode>

For migrations:

No pending migrations to apply.

Troubleshooting

  • Prisma reports drift: stop and compare the actual schema to seat-manager/prisma/schema.prisma:10; restore the snapshot when the reversal target is unclear.
  • API fails after rollback with unique index errors: check raw SQL partial indexes, especially operation and active-operator indexes in seat-manager/prisma/migrations/20260507080000_security_remediation_hardening/migration.sql:19 and seat-manager/prisma/migrations/20260514165745_one_active_seat_per_operator/migration.sql:37.
  • Watcher fails after rollback: verify WatcherState still has required keys and compatible columns in seat-manager/prisma/schema.prisma:204.

Rollback

Rollback of a failed rollback is snapshot restore. Restore the pre-maintenance snapshot, run migrations forward, run node dist/src/index.js status, then restart API and watcher only after the schema matches the application.