How Gurwi and Clatri scale under real load

Worker architecture and distributed leasing

Each product splits its backend into two services. The API answers HTTP requests, and the worker runs background jobs such as processing an uploaded document. In both Gurwi and Clatri the worker is a separate Railway service with 1 replica, so a heavy job never competes with chat requests for the API's memory or event loop.

A single worker replica avoids a classic bug. If two workers poll the same jobs table at the same moment, both can read the same pending row and process it twice, which means double billing for model calls and duplicate chunks in the index. Running one worker makes that race impossible by design.

Leases protect the system even when that assumption breaks, for example during a deploy when the old and new worker overlap for a few seconds. A lease is a time-limited claim written to the database [Supabase]. The worker marks a job as taken by itself until a given timestamp, and other workers skip any job whose lease has not expired. Postgres makes the claim atomic with a row lock, typically through `FOR UPDATE SKIP LOCKED`, so two workers can never grab the same row in the same instant.

If the worker dies mid-job, it stops renewing the lease. Once the timestamp passes, the job becomes claimable again and the next worker picks it up. The database therefore acts as the referee, and the worker count becomes a performance decision rather than a correctness risk.

Gurwi document processing: throughput gains from concurrency

Gurwi's worker used to process documents one at a time. The setting `RESOURCEWORKERCONCURRENCY` was 1, so a second upload waited in the queue until the first finished. The value is now 3, letting the single worker replica handle three documents in parallel.

Supabase records from October 7 show 6 documents taking about 111 seconds each [Supabase]. With serial processing, the sixth document finishes after 6 × 111 = 666 seconds, a little over 11 minutes, even though its own processing took less than 2.

With concurrency 3, the six documents run in two waves. The first three finish at about 111 seconds and the next three at about 222 seconds, so the batch completes in under 4 minutes. Throughput rises from roughly 0.54 to 1.6 documents per minute.

The average wait matters most to the user. With serial processing the completion times are 111, 222, 333, 444, 555 and 666 seconds, which average 388.5. With three slots, three documents finish at 111 and three at 222, averaging 166.5. That math assumes each document still takes 111 seconds when three run at once. If they share a slow external API or saturate the worker's CPU, per-document time grows, so the October 7 durations should be compared against post-change measurements before treating the gain as final.

Pinpointing bottlenecks: Clatri database load versus Gurwi ingestion queues

Clatri and Gurwi strain in different places, and the telemetry makes that visible. Clatri has more live conversation. It recorded 4561 messages in 30 days and a peak of 6 users active in the same minute. Gurwi peaks at 2 chats per minute, and its pressure came from documents waiting in the worker queue.

For Clatri, the API is idle and the database is not. [Railway] shows 0.02 vCPU of 32 on average, while [Supabase] shows 74% RAM used and 53 of 90 connections open on the small compute tier. Postgres uses RAM for its cache and for each connection's working memory, so rising RAM and connection counts tend to move together.

Connection pool saturation looks different from CPU saturation. When the 90 connections fill, new queries wait for a free slot or fail with a "too many connections" error, even if the database CPU is calm. Adding API replicas makes this worse, since each replica opens its own pool. The fixes live in Supabase, through a transaction-mode pooler, smaller per-replica pools or a larger compute tier.

For Gurwi, the symptom was latency for the user who uploads a document while others are already in line. Chat requests were fine, and the database had headroom. The fix was the worker concurrency change from the previous page, which touched the queue, not the API.

Operational troubleshooting runbook: Railway CLI and Supabase telemetry

Each layer has its own instruments, and reading the wrong one wastes time. The Railway CLI shows what the containers are doing, and the Supabase dashboard shows what Postgres is doing. A quick pass through both usually places the problem in under five minutes.

Railway layer

These commands run from a terminal linked to the project. `railway link` selects the project and environment, `railway status` shows which service is active, `railway logs` streams that service's output, and `railway variables` lists settings such as `RESOURCEWORKERCONCURRENCY`.

railway link
railway status
railway logs
railway variables

In the Railway dashboard, each service's Metrics tab shows CPU, memory and network per replica, and the Deployments tab shows whether a healthcheck failed. Run these against the API when requests fail, and against the worker when jobs stall.

Supabase layer

In the Supabase dashboard, the Reports section shows memory, CPU and connection counts for the database, which is where the 74% RAM and 53 of 90 connections came from. Query Performance lists the slowest and most frequent statements. For a live view, a query on `pgstatactivity` shows every open connection and its state.

psql "$SUPABASE_DB_URL" -c "select state, count(*) from pg_stat_activity group by state;"

Many connections sitting in `idle` usually mean application pools holding slots they do not use. Many in `active` with long durations point to slow queries, which Query Performance can then name.

End-to-end architecture and incident diagnosis

Picture a Tuesday afternoon. Clatri users report that some messages fail with a server error, and the problem started shortly after a marketing email went out. Cloudflare shows normal traffic patterns, and the TLS certificate is valid.

The first check goes to [Railway]. Both API replicas are healthy, CPU sits well under one vCPU, and `railway logs` shows errors mentioning that the database rejected new connections. The containers are fine, and they are reporting a problem one hop further down.

The second check goes to [Supabase]. Reports show connections pinned at 90 of 90 and RAM above its usual 74%. `pgstatactivity` reveals dozens of idle connections held by API pools. The diagnosis is connection pool saturation in the database, and the response is to route the API through the transaction pooler, shrink per-replica pool sizes or move up from small compute. Adding Railway replicas would open more connections and deepen the outage.

A Gurwi incident on the same day would look different. Users say their uploaded PDFs stay in processing for many minutes while chat works normally. The API metrics are calm, Supabase has headroom, and the jobs table shows a growing number of pending rows. The constraint is the worker queue, and the levers are `RESOURCEWORKERCONCURRENCY` and the per-document processing time.

Across both products, the architecture keeps the same shape. Cloudflare fronts the domain, Railway routes among 2 API replicas and runs 1 worker per product, and Supabase holds the data, the leases and the connection limit. Most incidents today will surface in the database for Clatri and in the queue for Gurwi, long before the API needs more compute.

When Supabase read replicas make sense for Clatri and Gurwi

A read replica in Supabase is a second Postgres instance with its own compute, its own copy of the data and its own connection limits, both direct and through the pooler. It serves only reads. Every insert, update and lease claim still goes to the primary, so Gurwi's worker would keep writing chunks and renewing leases on the same database it uses today.

Supabase sets several conditions for adding one. The project needs a Pro, Team or Enterprise plan, has to run on AWS with Postgres 15 or later, and needs at least a Small compute add-on. The replica inherits the primary's compute size. Projects below XL can have up to 2 replicas, and projects on XL or above can have up to 5. Supabase bills each replica's compute, disk, IOPS and throughput separately, so every replica adds a second line to the invoice.

Why a replica leaves Clatri's problem in place

Clatri's limit is the number of connections the primary accepts, the 90 slots that filled up during the Tuesday incident. A replica does nothing for those slots unless the app actually sends read traffic to it, and even then the replica brings its own connection ceiling that API pools can fill in the same way. Adding one while the pools stay oversized gives Clatri two instances to manage and pay for, each able to run out of connections. The fix for connections stays the same as before, with the transaction-mode pooler, smaller pools per replica or a larger compute tier.

The order of the levers

The first step is to optimize the slow queries and confirm the cost with `EXPLAIN ANALYZE`, which runs a query and reports where Postgres spends its time. If reads make up less than 80% of the workload, a larger compute tier helps more than a replica. If the workload is heavily read-oriented and the primary is already close to the top of vertical scaling (16XL), replicas become the next step.

Replicas can make sense earlier for two reasons. One is isolating heavy work, such as reporting or analytics, so a long report never slows the queries that serve live chats. The other is serving reads from a region closer to users, which cuts latency.

How each product would grow

If Clatri's traffic grows, the natural design sends heavy reads, such as message history and reports, to a replica and keeps every write on the primary, with the pooler in front of both. Gurwi would follow a different path. Its worker queue stays the first place to watch, and replicas would only arrive once the volume of reads justifies the extra instance.