Browse the documentation

Operations

Turn on workspaces

Enable multi-tenancy in stages, give each workspace its domain, and turn it off again.

This is the runbook for turning one Thallo site into many: the enablement flow stage by stage, host routing, the commands that create workspaces, domains and members, and the way back. Read it to the end before you run the first command: the middle of the flow rewrites the schema and blocks every write to the site until it finishes.

Workspaces explains what a workspace owns and what the install keeps in common. This page assumes you have read it.

What you need before you start

  • PostgreSQL. The retrofit refuses any other driver before it touches anything. See known limitations.
  • A cache driver that can purge by pattern. Thallo probes the live driver: a driver that cannot delete by pattern fails the first stage with Tenancy requires a cache driver that supports pattern purge. CACHE_DRIVER=redis passes; the Memcached driver does not implement pattern deletion at all.
  • No data collections. Enabling is refused while any collection definition exists, at the first stage, before anything is installed.
  • The account that will own the first workspace, by email or uuid. The admin shows the uuid as User ID at the top of the person's page under Users & Access › Users.

Settings › Workspaces and thallo:tenancy:status list both refusals under blockers before you start, and the admin keeps Enable workspaces off while any stands.

  • A window with no writes, and a fresh backup. The retrofit raises a write barrier and keeps it up until the flow reaches on.

Every step below has an equivalent in the admin, on Settings › Workspaces: the Multi-workspace mode card runs the enablement flow, the Domain routing card runs host activation, and both show the same steps this page names.

Read the current state

Three read-only commands.

bash
php glueful thallo:tenancy:status
php glueful thallo:tenancy:resolution:status
php glueful thallo:tenancy:diagnose

thallo:tenancy:status prints the enablement machine as a table: step, enabled, schema_state, progress, reloading, mode, pending slug, pending name, failure, cli fallback and blockers. --json prints the same status as JSON for a script. On an install that has never been through the flow, step is off, enabled is no and schema_state is none. The other steps are migrating_extension, awaiting_confirm, retrofitting, enabling_enforcement, reloading, finalizing, on, disabling, disabled_widened and failed.

thallo:tenancy:diagnose prints one line per check — schema, state, enforcement, provenance, collections, domain_reverification, static_write_audit, role_policy and public_signup — each with its status and its detail, and exits non-zero if any of them fails.

Turn multi-workspace mode on

thallo:tenancy:enable advances the machine one stage per run and stops. Run it again to take the next stage.

  1. Start the flow. The step becomes migrating_extension; nothing is enforced yet.

    bash
    php glueful thallo:tenancy:enable
  2. Run it again. It applies the tenancy package's migrations and stops at awaiting_confirm, telling you to re-run with --slug, --name and --owner. Passing those options earlier does nothing: they are read only on the run that confirms.

  3. Confirm the first workspace. This is the stage that changes the database.

    bash
    php glueful thallo:tenancy:enable --slug=my-site --name="My Site" [email protected]

    --owner takes the owner's email or uuid; an address no account has is refused before anything changes. thallo:create-admin prints the uuid of the account it creates.

    In order, it raises the write barrier; creates the first workspace with that slug and name and gives the owner the owner role in it; proves that no business key would collide once every existing row belongs to that workspace, and stops before any schema change if one would; moves install-wide system keys out of the settings table; adds a workspace column to every table that holds content, the content model or settings and widens their unique constraints; and records the schema as widened. It ends at reloading, with the barrier still up.

    Recording widened is one transaction with moving the store's orders and payments into the new workspace, so nothing ever sees the workspace before its rows are in it. A payment already under way when this stage starts finishes first; one that starts while the rows move is refused and can be retried a moment later. If payment work is still running after 10 seconds, the stage fails without changing anything: a queue:work worker that has handled a payment keeps holding it, so stop the workers, then run it with --retry. From here on the database refuses a payment row with no workspace.

  4. Restart the app: PHP-FPM, and every queue:work worker.

  5. Finish. It verifies enforcement, then lowers the barrier and sets the step to on in one transaction.

    bash
    php glueful thallo:tenancy:enable

You have finished when thallo:tenancy:status reports step on, enabled yes and schema_state widened, and thallo:tenancy:diagnose exits 0. The admin's sidebar now has a Workspaces group.

If a stage fails, the step becomes failed and failure carries the reason, and the command prints it. Fix the cause, then run it with --retry, which picks up at the stage that failed — or use Retry in Settings › Workspaces. A failure before the confirm stage can be abandoned instead with --cancel, which returns the step to off.

Payments on a site that turned workspaces on earlier

In earlier releases, turning workspaces on moved orders into the first workspace but left their payments behind: that workspace could not see them, refunds and payment webhooks could not find them, and paying an order again could open a second payment for it. On such a site, repair them:

  1. See what there is. This changes nothing.

    bash
    php glueful thallo:tenancy:payments:repair

    It prints, for each payments table, how many rows have no workspace, how many belong to the default workspace and how many to others. Then it lists anything that would stop the move: a row whose key (a plan name, an invoice number, a payment attempt) the default workspace already holds, or a row that belongs with another workspace's order or plan. Last come the orders with more than one payment attempt.

  2. Resolve what it lists. Two open payment attempts for one order is the usual case, and it stops the move until one is gone. Check with your payment provider which attempt the customer paid, then supersede the other by its uuid from the report:

    bash
    php glueful thallo:tenancy:payments:repair --retire-intent=<uuid>
    php glueful thallo:tenancy:payments:repair --retire-intent=<uuid> --apply

    The first run shows the attempt — its workspace, order, status, provider and reference — and changes nothing; --apply supersedes it. It reaches only attempts with no workspace or in the default workspace, and never supersedes one that has meanwhile been paid. Superseding only marks the attempt abandoned in Thallo: it does not cancel or refund anything at the payment provider. If the customer paid both attempts, refund one with the provider as well. A payment that still arrives for a superseded attempt is recorded, so you can refund it.

  3. Move the rows into the default workspace.

    bash
    php glueful thallo:tenancy:payments:repair --apply

    Only rows with no workspace move, keeping their ids and links; rows of other workspaces are not touched. If anything from step 1 is still there it changes nothing and says why. Payment work that started before it finishes first; if some is still running after 10 seconds — often a queue:work worker that has handled a payment — it changes nothing and asks you to try again, so stop the workers and rerun it. A payment webhook that arrives while it moves rows is refused and retried by the payment queue afterwards, so no update is lost. Run it again and it reports Nothing to repair. --json prints the same report as JSON.

    Run step 3 once even when step 1 finds nothing to move: it is also what makes the database refuse payment rows with no workspace from then on. Until it has run, every request that touches payments takes the repair's lock for as long as it runs.

Turn on domain routing

Until host routing is active the install serves exactly one workspace, and creating a second one is refused. Activating it is a separate flow.

  1. Set the hosts. In .env:

    text
    TENANCY_BASE_DOMAIN=sites.example.com
    TENANCY_DEFAULT_HOSTS=sites.example.com,www.sites.example.com
    TENANCY_PUBLIC_SCHEME=https

    The same two values can be set in the Public origin card on Settings › Workspaces, as Base domain and Default hosts; saved there they override .env. A process that started before the change refuses to activate until the app is restarted.

  2. Run the activation, once per stage:

    bash
    php glueful thallo:tenancy:resolution:activate

    The stages are mapping_hosts (each default host is attached to the first workspace, already verified — these hosts need no DNS proof), verifying_wiring (each one is probed and must resolve to that workspace), rebuilding_routes (the compiled route table is cleared) and then awaiting_fresh_boot, where the command tells you to re-run in a fresh process.

  3. Restart the app, then run the command once more. It probes the hosts again and the step becomes full.

Check it with thallo:tenancy:resolution:status: step is full and mode is full_resolution. A failed activation is retried or reset from the Domain routing card with Retry or Reset; the command does nothing from failed.

Create workspaces, domains and members

Each of these commands prints its result as JSON.

bash
php glueful thallo:tenancy:tenant create --slug=acme --name="Acme" --owner=<user-uuid>
php glueful thallo:tenancy:tenant list --status=active
php glueful thallo:tenancy:tenant suspend --uuid=<workspace-uuid>
php glueful thallo:tenancy:tenant reactivate --uuid=<workspace-uuid>

create refuses unless domain routing is full. It registers the workspace, then seeds its starter content types, block types and regions and activates it. --status accepts provisioning, active, suspended, deleted and purging. With a base domain set, a slug that would produce a reserved host — www, api or admin under that base domain — is refused; every other workspace answers at <slug>.<base domain>.

A custom host is a proof of ownership. Add it, publish the TXT record, then verify:

bash
php glueful thallo:tenancy:domain add --tenant=<workspace-uuid> --host=www.example.com
php glueful thallo:tenancy:domain verify --domain=<domain-uuid>

add answers with the domain's uuid and a token. Publish that token as the value of a TXT record named _thallo-verify.www.example.com. verify answers verified when it finds the record and pending when it does not, so you can run it again. The other actions are list (--tenant) and enable, disable and remove (each --domain).

Membership is per workspace:

bash
php glueful thallo:tenancy:member add --tenant=<workspace-uuid> --user=<user-uuid> --role=admin
php glueful thallo:tenancy:member set-role --tenant=<workspace-uuid> --user=<user-uuid> --role=viewer
php glueful thallo:tenancy:member remove --tenant=<workspace-uuid> --user=<user-uuid>
php glueful thallo:tenancy:member list --tenant=<workspace-uuid>

--role takes owner, admin, member, viewer or the slug of a custom role that workspace has defined.

After an upgrade adds or changes a starter definition, bring every workspace up to date:

bash
php glueful thallo:tenant:sync --all

It adds what is missing and leaves anything you have customised alone. --kind=block_type, --kind=content_type or --kind=region narrows it; a workspace uuid in place of --all does one workspace. thallo:tenant:blocks:sync is the block-type-only form of the same thing. thallo:tenant:seed <workspace-uuid> repairs a workspace whose starter surface is incomplete.

Keep it healthy

  • php glueful thallo:tenancy:diagnose after every change to the flow. Its exit code is the check.

  • php glueful thallo:tenancy:purge:recover redispatches workspace purges that were requested, failed or lost their lease. Run it after a queue outage, and whenever the admin says a purge is waiting for a worker.

  • php glueful thallo:tenancy:hosts:sweep queues the expired host-cooldown sweep. Nothing schedules it for you; add a daily cron line:

    text
    0 5 * * * php /path/to/site/glueful thallo:tenancy:hosts:sweep
  • Both of those jobs run on the tenancy-purge and tenancy-maintenance queues, so a worker has to be consuming them. See the scheduler and the queue.

Turn it off again

Disabling stops workspace scoping. It leaves the widened schema and every row in place, so nothing is lost and turning it back on is quick. Three things must be true first: exactly one workspace exists, domain routing is not active, and no starter definition is out of step.

  1. Turn domain routing off. This also requires exactly one workspace.

    bash
    php glueful thallo:tenancy:resolution:deactivate
  2. Disable. If it refuses because starter definitions are unsynchronised, run php glueful thallo:tenant:sync --all and try again.

    bash
    php glueful thallo:tenancy:disable

    It removes the enforcement provider, sets enabled to false and the step to disabled_widened, and tells you to re-run in a fresh process. The barrier stays up.

  3. Restart the app, then run php glueful thallo:tenancy:disable once more. It verifies that the install serves correctly unscoped and lowers the barrier.

You are done when thallo:tenancy:status reports step disabled_widened, enabled no and reloading no. To turn workspaces back on, run thallo:tenancy:enable, restart, and run it again: there is no second retrofit.

Commands that are not part of this flow

  • php glueful extensions:enable and extensions:disable on the tenancy enforcement provider. config/extensions.php lists it under protected, and every enable and disable surface refuses it: workspace enforcement is owned by the enablement flow. Never add or remove that provider line by hand.
  • tenant:create, tenant:activate and tenant:suspend. These are the framework's own commands. tenant:create inserts a row in the tenants table and stops — no owner membership, no starter content model, and none of the guards above. Use thallo:tenancy:tenant.
  • thallo:tenancy:single-store:repair. This is the single-site repair, not part of enabling workspaces. It establishes the default workspace identity an install needs before workspaces are on, and it is the fix when a single-site install reports that no single-store workspace is established. It takes --owner, and optionally --slug (default default) and --name (default Default).