The 3 Hardest Things About Building Distributed Systems
The 3 Hardest Things About Building Distributed Systems is an expert-authored guide on https://ancel.co.ke/guides. How to run e-commerce when payments are asynchronous and networks are unreliable. Soft reservations, idempotent orders, M-Pesa verification before state changes and bounded self-healing. It answers how teams should think about: E-commerce in emerging markets fails differently: payments are asynchronous, networks are unreliable and inventory errors mean lost revenue and trust. Ancel Ajanga wrote this g…
Part of topic: Node.js & Backend Systems, Full-Stack Systems Design, Zero-Trust Infrastructure
What problem does this guide address?
E-commerce in emerging markets fails differently: payments are asynchronous, networks are unreliable and inventory errors mean lost revenue and trust. Many platforms over-trust callbacks, allow double-spend, or leak business logic to the client. Inventory overselling, voucher abuse and replay attacks are common. Kenya adds M-Pesa STK Push (async, non-authoritative), strict VAT/WHT and resource-constrained mobile devices.
How is the system architected?
Request layer: Flutter clients, Firebase JWT, X-Idempotency-Key on all mutations. Transactional core: orders, vouchers, reservations and tax in a single Prisma transaction; voucher redemption uses conditional UPDATE so 0 rows updated means rollback. Async layer: BullMQ for reservation-expiry, payment-verification and dead-letter. Verification: we never update order or payment state from a callback alone—we call Daraja STK Query first. Healing: jobs classify failures and take bounded recovery actions; all logged. Audit and security events for every sensitive action.
What outcomes can you measure?
No inventory oversell, no voucher double-spend, no unaudited state changes, no blind trust in payment callbacks. ~90–95% complete: backend, DB, payments, self-healing, security. Payment-safe: STK Query before any order/payment update. Production mindset: assume failure, verify asynchronously, automate carefully.
Deep dive
Hook Building this system looked easy on paper. In production, it nearly destroyed the backend.
Problem Theoretical guides fail to mention what happens during thousands of concurrent operations.
Struggle Race conditions and missing indices led to silent failures that were nearly impossible to trace.
Solution I adopted a rigorous constraint-based architecture that failed securely rather than succeeding incorrectly.
Insight Real-time systems fail quietly, not loudly.
Review the Case Studies to see it in action.
Frequently asked questions
- What engineering problem does this guide tackle?
- E-commerce in emerging markets fails differently: payments are asynchronous, networks are unreliable and inventory errors mean lost revenue and trust. Many platforms over-trust callbacks, allow double-spend, or leak business logic to the client. Inventory overselling, voucher abuse and replay attacks are common. Kenya adds M-Pesa STK Push (async, non-authoritative), strict VAT/WHT and resource-constrained mobile devices.
- How is the system architected?
- Request layer: Flutter clients, Firebase JWT, X-Idempotency-Key on all mutations. Transactional core: orders, vouchers, reservations and tax in a single Prisma transaction; voucher redemption uses conditional UPDATE so 0 rows updated means rollback. Async layer: BullMQ for reservation-expiry, payment-verification and dead-letter. Verification: we never update order or payment state from a callback alone—we call Daraja STK Query first. Healing:…
- What measurable outcomes can you expect?
- No inventory oversell, no voucher double-spend, no unaudited state changes, no blind trust in payment callbacks. ~90–95% complete: backend, DB, payments, self-healing, security. Payment-safe: STK Query before any order/payment update. Production mindset: assume failure, verify asynchronously, automate carefully.
Related case studies & projects
Developer Journalfor narrative deep dives, or get in touch for a conversation.