Two POS providers, one core. The failure modes, written down.

The build

What was built, and what it is not.

Platestead is a restaurant commerce platform: storefronts, an admin and a public API on one shared codebase, with Square and Clover behind one provider-neutral POS core. Each restaurant brand runs as its own deployment with its own database, secrets and encryption keys; one brand's API key is refused by the other brand's backend. Two demonstration restaurants run on it: The Local Table, on a Square sandbox seller, and Ember & Fork, which runs two units in a single deployment, Mill Bend and Riverside, each with its own menu, prices, stock, orders and Clover sandbox merchant.

Ruby on Rails and PostgreSQL, on Spree Commerce, for the backend and admin, with a modern JavaScript storefront alongside. Designed, directed and operated end to end by Prayantr's principal between August and September 2026, with AI coding agents doing much of the implementation under his direction and review. Both restaurants are demonstrations operated in-house: Square and Clover run against sandbox accounts, and card payments in test mode.

Prayantr's own build · Not client work
Square and Clover sandbox accounts · No real merchant or customers yet

What follows is the mechanism: the boundaries, the decisions and the failure modes, including the parts that are deliberately not finished. The platform's public site is platestead.com, and a shorter case study is on amitsolanki.com.

Architecture

One integration core. Thin adapters at the edge.

Square was integrated directly, in an earlier single-restaurant build. When Clover arrived, the shared machinery was extracted into a provider-neutral core, spree_pos, and Square was moved onto it as an adapter, so the Clover integration was written against the same seams rather than beside the first.

• The core owns the state. Connections with encrypted credentials, provider locations mapped to local stock locations, external references, stored webhook events de-duplicated by event id, catalog sync, counted sync runs, order push and a nightly reconciliation.
• Adapters own the protocol. Each adapter translates one provider's auth, payloads and error model into the core's interface. The core's runtime code has no provider-specific branches.
• One catalog source per store, enforced twice: a model validation and a PostgreSQL partial unique index, so a race cannot slip a second source in.
• Writes ship inert. Outbound writes to a POS are switched on as an explicit, audited step. Stock write-back to Clover is built and deployed, and switched off until two named issues are fixed.

The abstraction stops where the providers genuinely differ. Duplicate protection and authentication are not hidden behind a common interface, because pretending the providers behave alike is how an integration duplicates writes or trusts a forged payload. The next two sections are those two differences.

Hard problem 01

Two auth lifecycles, two webhook trust models.

The two providers disagree on almost everything that matters to an integration's reliability:

• Token lifetime. Square issues access tokens that last about 30 days with a stable refresh token. Clover issues tokens that last about 30 minutes and rotates the refresh token on every use, so each refresh has to be persisted before the old token is discarded. Both refresh lazily, ahead of expiry.
• A refresh that had never run. The Clover adapter's refresh method existed, nothing called it, and it targeted the wrong endpoint. No test could show that: it surfaced only when a real sandbox token reached its refresh window. It was fixed and then verified on the real clock, by watching a refresh happen minutes before expiry.
• Signed webhooks. Square signs each request against the URL it was delivered to. The adapter verifies that signature and de-duplicates on event id. Events for one sandbox order arrived more than 16 hours late; all were accepted, verified and applied, and nothing was dropped.
• Unsigned webhooks. Clover authenticates with a single static code shared by every merchant on the app, so the body proves nothing about which merchant sent it or what changed. Its webhooks are treated as untrusted notifications: the adapter re-fetches the state from Clover with that merchant's own credential and acts on the answer. In a sandbox test, a payload claiming a stock count of 999 produced Clover's real count of 24. A delivery that covers several merchants is split per connection first.

Honest gap: the Square connect flow checks its state parameter, but further hardening of that flow is recorded as open work.

Hard problem 02

Idempotent order delivery, where one side has no idempotency keys.

A completed website order is pushed to the restaurant's POS by a background job. Square accepts deterministic idempotency keys derived from the order, so a re-push returns the same Square order. Clover has no native idempotency, so the adapter builds it:

1. Look up before creating

   Search Clover by external reference. None found: create with the total, add lines, attach payment. One found: adopt it and add only what is missing, which covers a retry after a half-finished push. Two or more: mark it ambiguous and leave the decision to a person.

2. A per-order advisory lock

   No automatic path pushes an order twice, but manual and abnormal paths could: a retried job whose original worker was still alive, or a console re-push during an incident. Against a provider without idempotency keys that means a duplicate kitchen ticket, or an adopted half-order with every line and the payment doubled. On Ember & Fork, the push now runs under a session-level PostgreSQL advisory lock per order, taken with a try-once call that never waits. Session-level is the point: the lock belongs to a live database connection, so a killed worker's lock is released by the server and its retry can proceed, with no transaction held open across the provider calls and no lease to expire. The regression spec fails against the previous release (two Clover orders instead of one), so it proves the fix rather than decorating it.

3. A stuck-work sweep

   A worker killed after committing pending (a deploy's short shutdown grace, or memory pressure) left the push stranded with no alert, because the queue fails the claimed execution directly and the job's own retry never runs. A sweep every 5 minutes finds pushes pending for over 10 minutes, alerts once, and re-enqueues the same job; the lock and adopt-on-retry keep the retry to one Clover order. Alerts are de-duplicated so a stuck order does not open a new issue every sweep.

Every push ends in one of three recorded outcomes: succeeded with remote ids saved, failed and safe to retry, or ambiguous. This is idempotent per order by design, not a promise of exactly-once delivery, and the adopt-on-retry path has been validated on one Clover sandbox order.

Hard problem 03

Catalog sync that refuses to delete on partial data.

The menu flows one way, from the POS into the storefront: webhook-driven updates, with a nightly full reconciliation as the backstop for anything a webhook missed. The dangerous step is deletion, because "missing from the response" and "deleted at the provider" look identical when a listing is cut short by a timeout or a paging bug.

• A complete listing or nothing. An item is archived as deleted only after the sync has read the provider's full listing. A partial read prunes nothing.
• A blast-radius limit. A sync that would remove more than 25 percent of the catalog is withheld for a human instead of applied.
• Archive, not delete. Removed items are archived, and dishes archived by the restaurant stay archived when the POS still lists them.
• Proven, not assumed. The prune path and the guard were exercised against the Square sandbox: a real deletion archived exactly one item, and no removal was withheld.

Known and accepted: a burst of catalog webhooks during heavy menu editing in the POS can trigger overlapping syncs. Sync runs are serialized and counted, and the nightly reconciliation remains the backstop.

Hard problem 04

A live hostname migration, with the webhooks edited in place.

The Local Table moved to its platestead.com hostname while live. The awkward dependency is the signed webhook: Square signs each request against the URL it was delivered to, so the subscription's URL, the application's idea of its own host and the signature check all have to move together.

1. Rebuild the branch first

   The migration branch was checked for ancestry against what production was running. An earlier version of it would have silently dropped live fixes, so it was rebuilt on the production heads before anything else happened.

2. Backup, then deploy in order

   A verified database backup, then the admin deploy, then the storefront deploy, then the database-held URLs.

3. Edit the webhook subscription, do not recreate it

   Editing the existing Square webhook subscription in place kept its signing key, so there was no window in which events signed with a new key met a verifier holding the old one. The OAuth redirect was updated alongside.

4. Prove it end to end

   A test event was delivered and signature-verified on the new host; canonical URLs and the sitemap moved to it; error monitoring stayed clean afterwards.

5. Keep the way back open

   The old hostnames kept serving, and the rollback was written before the change, not during it.

It is not described as a zero-downtime migration, because downtime was not measured.

Production practice

Release guardrails that refuse, not remind.

Two restaurant brands on one server raise the cost of a careless deploy, so the release path is written down, and the riskiest checks are mechanical. Each deploy runs through its tenant's own wrapper, which refuses on the first two checks below; the release runbook adds the gate:

• Ancestry-checked deploys. The wrapper refuses a build whose history does not contain the commit running now, so a stale branch cannot quietly roll production back. An override exists and is printed loudly.
• Tenant file checks. Configuration must name only this tenant; secret files must be locked to the owner, and are never printed.
• A pre-deploy gate, from the runbook. Before each deploy of the September 2026 release: a verified database backup, zero claimed or pending outbound pushes, and clean test suites and static security scan.
• Rollback targets. The previous image stays on the server, so rollback starts it without a rebuild.

The guardrails came partly from a failure. An early cutover, moving The Local Table's Square integration onto the new core, stopped at a post-deploy check when the first catalog import against real Square SDK objects raised. The documented rollback ran: 18 minutes 26 seconds of maintenance, no data changed, no restore needed. The root cause was test fakes shaped like the bug rather than like the SDK, so the fix included specs built from real, sanitized SDK responses.

Between releases: nightly database backups for both tenants, each dump verified, with a restore rehearsal whose record counts matched production. Backups are kept on one machine, by decision, for a build with no customer data. Error monitoring and uptime checks cover the public endpoints; queue-backlog, stalled-worker and memory alerts are recorded as open debt rather than implied.

Decisions and tradeoffs

The calls that shaped it, with their costs.

• A deployment per restaurant brand, not a shared multi-tenant database. Each brand gets its own database, keys and cookies, which makes a cross-tenant data leak a configuration error rather than a missing WHERE clause. The cost: both tenants share one server with no network segmentation between them yet, and they report into shared error-monitoring projects, tagged per deployment.
• A restaurant unit is a store, not a stock location. Spree scopes the catalog source, products, prices and API keys to a store. A unit with its own menu at its own prices therefore had to be a store; modelling it as a stock location would have fought the framework on every query. The storefront resolves the unit from the URL (/us/en is Mill Bend, /riverside/us/en is Riverside), sends that unit's key, partitions its caches per unit, and ignores a spoofed unit header. Cross-unit isolation was verified for catalog, carts, orders, stock locations and POS connections; a Riverside dish returns a real 404 under Mill Bend's path. Not unit-aware yet, and said so: admin roles are store-wide (the code fails closed), and loyalty, menu chat and the courier admin are shared across units.
• Backport, don't force an upgrade. The Local Table stays on an older line of the core with fixes backported to it, instead of being upgraded mid-release to match Ember & Fork.
• Enforce invariants at every entry point. A release sweep found that required dish choices, such as a spice level, were enforced on the add-to-cart endpoint but not on a second cart path the UI never used. Both paths now refuse an incomplete item, verified in production with a request matrix. The same sweep refused anonymous file uploads, capped request bodies at the proxy, and closed a critical advisory in the storefront framework.
• Negative controls on fixes. A fix counts when its test fails without it. Production checks are read-only probes, open issues are triaged against a written release bar, and the release closed with an end-to-end acceptance order through Riverside's Clover sandbox merchant.

Proven, and not

Where the evidence stops.

• Proven against Square and Clover sandbox accounts: catalog sync and reconciliation for both, idempotent order push on both, token refresh on both, the deletion guard, the stuck-work sweep, the hostname migration and the rollback path.
• Not proven: a real merchant, or real customer traffic. None has been connected.
• Built, switched off: stock write-back to Clover.
• Not built: order status flowing back from Clover, and cancellation or refund sync to either POS.