Skip to content
quest'n'click

Clever CloudExplanationFor adminsApplies from 4.2.3

RPO and RTO

Cette page en françaisThis page on the GitHub wiki

Two numbers say what a hosted Studio can lose and how long it can be down: the RPO (recovery point objective, the changes a recovery may lose) and the RTO (recovery time objective, how long the recovery takes). This page defines both for the Studio Server, says how each is measured on your installation, and ties them to the measures of the reference deployment. They are observations, never promises.

Why

A team writes the game together, all day. If the database is lost, the question is not whether a backup exists but which changes come back, and when people can write again. The Studio Server answers with two mechanisms, each tied to one number:

  • The journal shipped to the bucket bounds the RPO. Each confirmed operation is copied to Cellar as an immutable segment, in order, as soon as it is written (a failure is retried within 30 s).
  • The daily backup plus the replay bounds the RTO. A restore writes the latest backup into a new database, then replays every segment shipped since (restore drill).

The RPO, defined

What a restore can lose is the operations confirmed but not yet shipped to the bucket when the database was lost. At any moment that is the age of the oldest operation not shipped yet:

  • GET /readyz gives it as rpoMs (0 when everything is shipped);
  • with metrics on, studio_journal_rpo_seconds says the same (Monitoring).

The target is one minute (D42): watched, with an alert past 60 s for 5 min, never promised.

The RTO, defined

The time from the decision to restore to the moment the Studio serves writes again (/readyz 200 on the restored database). It adds up:

  1. creating a new PostgreSQL add-on (not measured);
  2. downloading the latest backup (not measured);
  3. the restore with its replay;
  4. pointing the studio app at the new add-on (clever service link-addon, unlinking the lost one) and redeploying (a studio deploy took 29 s to 1 min 29 s in the reference deployment, migrations included);
  5. the new instance taking the service lease.

Only step 3 is measured:

What is not a loss

  • A redeploy refuses writes for a while (503 with Retry-After) and loses none: a write retried with its operation id is applied once (Redeploys).
  • A restore logs everyone out (sessions are not in a backup), and drafts still in a member's browser were never confirmed: they are not part of the RPO, and the Studio keeps a refused draft in the editor.

Alternatives set aside

  • PostgreSQL's own backups only. They would restore the tables, not tie them to the file contents in Cellar at the same instant. The Studio's backup lists the blobs it names and refuses to restore when one is missing.
  • A continuous replica. Not offered: one instance, one database, and the journal shipped to the bucket instead.

Trade-offs

  • The journal and the backups share the bucket with the contents: one place to lose. Copy it elsewhere.
  • The RPO depends on Cellar being reachable: while it is not, rpoMs grows (Worker, PostgreSQL or Cellar down).
  • Measures come from one small game and one member writing, on the test organisation: not a production load.

Decisions and ADRs