Skip to content
quest'n'click

Clever CloudTroubleshootingFor adminsApplies from 4.2.3

Clever Cloud troubleshooting

Cette page en françaisThis page on the GitHub wiki

The problems a hosted Studio on Clever Cloud has shown, or is built to show: a 503, connections exhausted, a missing cookie, a slow preview, jobs refused for a clock skew, a daily backup that did not run. Each starts from what you see.

Symptom

You seeSection
A save refused with 503 and Retry-After: 2, "this instance does not serve writes yet; try again", during or right after a deploy503 during a deploy
"too many connections for role …" in the logs, a 500 on a write during a deployConnections exhausted
The game's frames empty in one browser (Safari, GNOME Web), or 401 "open the game from the Studio" in the frameMissing cookie
The room view showing an older revision for tens of seconds after a changeSlow preview
Health and previews failing now and then; the processing worker refused the job (403: the job has expired)Clock skew
No "event":"backup" line one morning, no new object under backups/Silent cron
503 "the edit cannot be judged now", or "the session is suspended"A service down: Worker, PostgreSQL or Cellar down

Possible causes

  • A redeploy: two instances overlap for minutes and hand the service lease over.
  • The PostgreSQL DEV plan (5 connections) with two instances running.
  • publicUrl and playUrl on two sites (*.cleverapps.io), or a browser that keeps no cookie in a cross-host frame.
  • The preview's build on the worker, one at a time, with each revision's files stored.
  • The studio app's clock ahead of the worker's (before 4.2.3's fix, a few milliseconds were enough).
  • The cron's conditions: STUDIO_BACKUP_CRON, instance number 0, a login shell, absolute paths.

Diagnosis

Keep the logs of both apps while you look (clever logs follows until stopped: run it in the background and stop it by its process id), and read /readyz:

# on your machine, in the engine repository
clever logs --alias my-game-studio > studio.log 2>&1 &
curl -s https://studio.example.fr/readyz

Each section below says which line or answer tells its case apart.

Fix

503 during a deploy

  1. A 503 with Retry-After: 2 while /readyz says "writes":"waiting" and the logs show a new instance before its lease-taken line is the redeploy's handoff: it lasts 10 to 16 s after the handoff, about a minute in all, and about 2.5 min for the first deploy onto 4.2.3 or later from an older version. The Studio retries the write with the same operation id and it is applied once. Nothing to fix; tell members before deploying (Redeploys). If it lasts longer than 2.5 min, look at PostgreSQL (Worker, PostgreSQL or Cellar down).

Connections exhausted

  1. "too many connections" (leases-read-failed, reconcile-failed, ship-failed) during a deploy comes from two instances, about 4 connections each, on a plan of 5. Take a PostgreSQL plan with more connections (Apps and add-ons). Outside a deploy, look for connections you opened yourself (psql) on the same add-on.
  1. The game's frame needs its own session cookie, on playUrl. Check that publicUrl and playUrl are two subdomains of one domain of yours: two *.cleverapps.io names are two sites, where only a partitioned cookie (CHIPS) survives. WebKit on Linux (GNOME Web) stores no cookie a frame of another host sets, even on one site: there the game's frames stay empty whatever you do. A real Safari on macOS and iOS has not been checked yet (the studio-hosted pass). A game frame left idle past its 15-minute session shows nothing new: "Reload view" opens it again (Domains and TLS).

Slow preview

  1. After each change the worker builds the head; a burst of changes builds its last revision only, and the room view shows the last ready preview meanwhile, with the revision it is. The "event":"preview" log line (since 4.4.0) says where the time goes: queuedMs, snapMs, jobMs, storeMs, sinceWriteMs. Upgrade to 4.4.0 or later if you run an older version:

The worker's job itself stays the main part: it builds the engine cold at each revision.

Jobs refused for a clock skew

  1. the processing worker refused the job (403: the job has expired) in health-failed or preview-failed lines, for some jobs and not others, came on the reference deployment from the studio app's clock running ahead of the worker's: the Studio sets a job's expiry, the worker checked it to the millisecond. Since 4.2.3 (#192) the worker allows 30 s of skew (CLOCK_SKEW_MS), logs worker-job-refused with its reason, and the Studio repeats the reason (200 characters at most). If you still see it: check both apps run the same version, and read the worker's worker-job-refused line; a skew past 30 s is the platform's to fix.

It still refuses a job that has expired, to the millisecond, or that would live longer than a job plus the skew.

Silent cron

  1. The daily backup runs at 03:17 UTC from clevercloud/cron.json, and its script does nothing, silently, when:

    • the studio app does not set STUDIO_BACKUP_CRON=1, or was not redeployed or restarted since it was set;
    • INSTANCE_NUMBER is not 0 (another instance, or Clever did not say which one it is);
    • the app was installed from the release archives, which do not carry the script: add your own (Backups), starting with #!/bin/bash -l (without it, a cron runs with no environment) and using absolute paths ($ROOT).

    Run it by hand in clever ssh (cd "$APP_HOME" && ops/studio/backup-cron.sh): it prints its "event":"backup" line, or the error that stops it.

How to check it is fixed

/readyz 200 with rpoMs 0; a save goes through; the game shows in the room view and the Play tab; the next "event":"preview" line is short; the worker logs no worker-job-refused; the next morning has its "event":"backup" line.

Neighbouring cases