The 3 Hardest Things About Building Distributed Systems
The 3 Hardest Things About Building Distributed Systems is an expert-authored guide on https://ancel.co.ke/guides. Engineering resilient e-commerce platforms in emerging markets: soft inventory reservations, asynchronous payment reconciliation, voucher double-spend prevention and background healing jobs. Lessons from Fits by Aliv. It answers how teams should think about: E-commerce architecture in emerging markets operates under hostile network and payment constraints. Ancel Ajanga wrote this guide tying patterns…
Part of topic: Backend & Modular Systems, Full-Stack Systems Design, Zero-Trust & Auditable Infrastructure
What problem does this guide address?
E-commerce architecture in emerging markets operates under hostile network and payment constraints. Unlike card-based transactions with synchronous authorizations, mobile money (M-Pesa) payments require customer interaction on external mobile handsets. If an e-commerce platform decrements stock immediately when a checkout starts, abandoned carts lock up inventory. If it waits until payment confirms, high-demand items will oversell. Furthermore, unreliable mobile connections and untrusted webhooks frequently cause order state desynchronization.
How is the system architected?
Fits by Aliv solves these constraints using a four-tier reliability model. First, checkouts acquire time-limited soft reservations on inventory backed by Redis and PostgreSQL. Second, BullMQ background queues enforce strict reservation time-to-live (TTL) limits, automatically restoring abandoned stock. Third, payment transitions are strictly decoupled from webhook callbacks: rather than trusting incoming callbacks unconditionally, the system validates transaction status using outbound Daraja STK Query polling. Fourth, voucher redemptions utilize atomic conditional SQL updates to prevent double-spending across concurrent checkouts.
What outcomes can you measure?
A production-grade, payment-safe e-commerce engine. Eliminates inventory overselling and voucher double-spending through database-level conditional mutations and automated background reservation reclamation. Replaces blind webhook trust with verified query reconciliation.
Deep dive
The Realities of Emerging Market E-Commerce
Building an online store in an environment with high-speed internet and credit card rails is largely a solved problem. Platforms like Shopify abstract the complexities of checkout, inventory, and payment processing.
However, operating in East Africa introduces architectural challenges that break standard e-commerce patterns:
1. Asynchronous Handset Interactions: Customers pay via M-Pesa STK Push. A payment prompt appears on their phone, requiring them to enter a PIN. This handshake takes anywhere from 5 to 60 seconds, during which the checkout remains pending. 2. Volatile Network Connectivity: Mobile networks in transit frequently drop TCP connections right as the user completes their payment, preventing the mobile app from receiving confirmation directly from the client session. 3. Voucher and Coupon Exploitation: Flash sales create high-concurrency spikes where users attempt to redeem limited-use promotional vouchers simultaneously across multiple browser tabs or devices.
Solving these problems requires moving away from naive CRUD state machines toward asynchronous, failure-tolerant workflows.
The Inventory Dilemma: Soft Reservations with TTL
When a customer proceeds to checkout, when should inventory be deducted?
* Deduct Immediately: If the user abandons the checkout or takes 10 minutes to find their phone, that stock is unavailable to other paying buyers. * Deduct After Payment: If two customers attempt to purchase the last pair of sneakers at the same moment, both STK prompts will be sent, both customers will enter their PINs, and the merchant will have accepted money for an item they cannot fulfill.
### The Solution: Two-Phase Soft Reservations Fits by Aliv implements a two-phase reservation pattern:
``` [User Clicks Checkout] | v [Acquire Soft Reservation] ---> (Stock Decremented, TTL = 10 min) | +-------------------------------------+ | | [Payment Confirmed] [Timeout / Abandoned] | | v v [Convert to Permanent Sale] [BullMQ Reclaims Stock] ```
When checkout initiates, an atomic transaction reserves the items and schedules a delayed job in BullMQ:
```typescript await prisma.$transaction(async (tx) => { // 1. Decrement available stock conditionally const updatedItem = await tx.inventoryItem.updateMany({ where: { id: itemId, availableStock: { gte: requestedQuantity } }, data: { availableStock: { decrement: requestedQuantity }, reservedStock: { increment: requestedQuantity } } });
if (updatedItem.count === 0) { throw new BadRequestException("Insufficient inventory available."); }
// 2. Create reservation with expiration timestamp const reservation = await tx.reservation.create({ data: { orderId, expiresAt: new Date(Date.now() + 10 * 60 * 1000) } });
// 3. Enqueue reclamation job in BullMQ await inventoryQueue.add( "expire-reservation", { reservationId: reservation.id }, { delay: 10 * 60 * 1000 } ); }); ```
If payment is completed within the window, the reservation converts into a finalized order. If the BullMQ job fires and finds the order still unpaid, it atomically returns the reserved count back to `availableStock`.
Preventing Voucher Double-Spend Under High Concurrency
Promotional discounts are prime targets for race conditions. If a single-use voucher has a database boolean flag `is_used = false`, reading the record and checking the boolean before updating it creates a classic time-of-check to time-of-use (TOCTOU) vulnerability.
Fits by Aliv guarantees atomic single redemption by executing conditional updates directly in SQL:
```sql UPDATE vouchers SET is_used = true, used_by_order_id = $1, redeemed_at = NOW() WHERE id = $2 AND is_used = false; ```
If two concurrent checkouts attempt to apply the exact same voucher code, the database's internal row lock ensures only one transaction updates 1 row. The losing transaction updates 0 rows, prompting the application code to instantly reject the voucher without requiring distributed locks.
Never Trust Callbacks Alone: Outbound Verification
Payment webhooks are public endpoints exposed to the internet. While payloads can include HMAC signatures or shared secrets, relying exclusively on incoming webhooks leaves systems vulnerable to delayed delivery or upstream provider issues.
Fits by Aliv treats incoming M-Pesa callbacks as untrusted hints.
When a webhook arrives: 1. The endpoint validates the header signature and immediately returns an HTTP 200 response to Safaricom's servers. 2. Rather than updating the order to `PAID` directly from the webhook body, the backend enqueues a verification task. 3. The background worker calls Safaricom's authoritative STK Query API directly:
```typescript async function verifyPaymentWithUpstream(checkoutRequestId: string): Promise<PaymentStatus> { const token = await generateDarajaToken(); const response = await axios.post( DARAJA_QUERY_URL, { BusinessShortCode: process.env.MPESA_SHORTCODE, Password: generateDarajaPassword(), Timestamp: generateTimestamp(), CheckoutRequestID: checkoutRequestId }, { headers: { Authorization: `Bearer ${token}` } } );
return response.data.ResultCode === "0" ? PaymentStatus.PAID : PaymentStatus.FAILED; } ```
By independently querying the upstream authority, the system guarantees that orders are only fulfilled when funds are genuinely confirmed on Safaricom's core ledger, completely neutralizing spoofed or corrupted webhook payloads.
Frequently asked questions
- What engineering problem does this guide tackle?
- E-commerce architecture in emerging markets operates under hostile network and payment constraints. Unlike card-based transactions with synchronous authorizations, mobile money (M-Pesa) payments require customer interaction on external mobile handsets. If an e-commerce platform decrements stock immediately when a checkout starts, abandoned carts lock up inventory. If it waits until payment confirms, high-demand items will oversell. Furthermore, unreliable mobile connections and untrusted webhooks frequently cause order state desynchronization.
- How is the system architected?
- Fits by Aliv solves these constraints using a four-tier reliability model. First, checkouts acquire time-limited soft reservations on inventory backed by Redis and PostgreSQL. Second, BullMQ background queues enforce strict reservation time-to-live (TTL) limits, automatically restoring abandoned stock. Third, payment transitions are strictly decoupled from webhook callbacks: rather than trusting incoming callbacks unconditionally, the system v…
- What measurable outcomes can you expect?
- A production-grade, payment-safe e-commerce engine. Eliminates inventory overselling and voucher double-spending through database-level conditional mutations and automated background reservation reclamation. Replaces blind webhook trust with verified query reconciliation.
Related case studies & projects
Developer Journalfor narrative deep dives, or get in touch for a conversation.