Scaling Isn’t the Hard Part — Staying Correct Is
Scaling Isn’t the Hard Part — Staying Correct Is is an expert-authored guide on https://ancel.co.ke/guides. Architecting financial platforms for correctness under failure: double-entry ledger invariants, idempotent M-Pesa payment processing, distributed lock semantics and fault isolation across asynchronous networks. Lessons from NestFi. It answers how teams should think about: Financial engineering is fundamentally different from typical CRUD application development. Ancel Ajanga wrote this guide tying patterns t…
Part of topic: Backend & Modular Systems, Zero-Trust & Auditable Infrastructure, Full-Stack Systems Design
What problem does this guide address?
Financial engineering is fundamentally different from typical CRUD application development. In social or e-commerce apps, a temporary glitch or dropped message is an inconvenience; in financial systems, a single race condition or unhandled payment webhook can create unauthorized money, corrupt account balances, or cause untraceable financial loss. External payment providers operate asynchronously over unreliable mobile networks, frequently replaying webhooks or timing out ambiguously.
How is the system architected?
NestFi implements a tiered storage architecture designed around the fundamental accounting equation: Assets = Liabilities + Equity. PostgreSQL serves as the single source of financial truth, enforcing immutable double-entry ledger constraints where every transaction consists of balanced debits and credits within an atomic database transaction. MongoDB captures high-volume, append-only operational audit trails and webhook payloads. Redis provides short-lived distributed locks (`redlock`) with strict TTLs to serialize concurrent withdrawal attempts. The NestJS backend processes M-Pesa Daraja callbacks idempotently using database-enforced transaction reference constraints, isolating financial state mutations from third-party API availability.
What outcomes can you measure?
A resilient financial coordination platform engineered for zero ledger drift. Eliminates double-crediting through database-level idempotency on M-Pesa transaction IDs. Replaces vulnerable mutable balance fields with verifiable append-only ledger entries, accompanied by real-time WebSocket status updates and complete audit trails.
Deep dive
The Fundamental Rule of Financial Software
In consumer applications, eventual consistency is often an acceptable trade-off. If a user's follower count or profile picture update takes a few seconds to propagate, no real damage is done.
In financial software, eventual consistency can be fatal. If a user initiates two simultaneous withdrawals from a wallet containing $100, both requests must not succeed concurrently. If an M-Pesa payment callback is delayed by 45 seconds and replayed three times by the telecom carrier, the user's balance must be credited exactly once.
Most financial software bugs do not stem from complex math. They stem from a failure to account for asynchronous network realities: dropped callbacks, duplicate retries, and race conditions against mutable database rows. NestFi was architected around a single guiding principle: assume every network call will fail, retry, or arrive out of order.
The Double-Entry Ledger Invariant
The most dangerous design pattern in fintech is storing account balances as a mutable numeric column in a database table:
```sql -- ANTI-PATTERN: Never do this in production financial systems UPDATE accounts SET balance = balance - 100 WHERE id = 42; ```
If an unhandled exception occurs mid-operation, or if an administrator edits the database directly, the relationship between historical transactions and current balance is permanently severed.
In NestFi, account balances are mathematical derivatives of an immutable double-entry ledger. Money is never created or destroyed; it is merely moved between accounts. Every transaction must contain at least two entries—a debit and a credit—and their sum must equal exactly zero:
$$\sum \text{Debits} + \sum \text{Credits} = 0$$
```typescript export async function recordTransfer( client: PoolClient, sourceAccountId: string, destinationAccountId: string, amountInCents: bigint, idempotencyKey: string ): Promise<void> { // 1. Verify idempotency const existing = await client.query( 'SELECT id FROM financial_transactions WHERE idempotency_key = $1', [idempotencyKey] ); if (existing.rows.length > 0) return;
// 2. Insert master transaction record const txResult = await client.query( `INSERT INTO financial_transactions (idempotency_key, created_at) VALUES ($1, NOW()) RETURNING id`, [idempotencyKey] ); const txId = txResult.rows[0].id;
// 3. Insert balanced debit and credit entries atomically await client.query( `INSERT INTO ledger_entries (transaction_id, account_id, amount, direction) VALUES ($1, $2, $3, 'DEBIT'), ($1, $4, $3, 'CREDIT')`, [txId, sourceAccountId, amountInCents, destinationAccountId] ); } ```
Because this executes inside a single PostgreSQL transaction with strict serialization, a database crash or server failure can never leave money in limbo.
Taming Asynchronous M-Pesa Payment Callbacks
In Kenya, M-Pesa is the dominant payment infrastructure. Interfacing with Safaricom's Daraja API presents unique distributed systems challenges:
1. STK Push is Asynchronous: When an app initiates a payment prompt, Safaricom sends a USSD prompt to the user's handset. The HTTP response to the initial request merely acknowledges that the prompt was dispatched, not that funds were transferred. 2. Callbacks Can Arrive Twice: If Safaricom's webhook listener does not receive an immediate 200 OK response within a tight timeout, its gateway retries the callback multiple times. 3. Callbacks Can Arrive Out of Sequence: In edge cases, a user might retry a payment after an apparent timeout, causing multiple overlapping webhooks to hit the server.
### Idempotency Enforcement To protect against duplicate processing, NestFi implements database-enforced idempotency on the unique Safaricom transaction code (`MpesaReceiptNumber`, e.g., `NLJ7RT61KV`):
```sql CREATE TABLE payment_receipts ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), receipt_number VARCHAR(64) NOT NULL UNIQUE, checkout_request_id VARCHAR(64) NOT NULL, amount_cents BIGINT NOT NULL, phone_number VARCHAR(16) NOT NULL, created_at TIMESTAMPTZ DEFAULT NOW() ); ```
When a callback arrives: 1. The raw payload is immediately logged to MongoDB for immutable auditing. 2. An insert into `payment_receipts` is attempted within a PostgreSQL transaction. 3. If the `receipt_number` already exists, PostgreSQL throws a unique constraint violation (`code 23505`). The application cleanly acknowledges the callback with HTTP 200 without executing the credit logic a second time.
Concurrency Protection with Fail-Closed Distributed Locks
When users request withdrawals, checking the ledger balance and initiating the outward payment requires multiple steps. If a user triggers two rapid withdrawal requests before the first has finished executing, a race condition could permit withdrawing funds twice.
NestFi guards these multi-step operations using Redis distributed locks with fail-closed semantics:
```typescript const lockKey = `lock:withdrawal:${userId}`; const acquired = await redis.set(lockKey, transactionId, 'PX', 5000, 'NX');
if (!acquired) { throw new ConflictException("A transaction is already processing for this account."); }
try { // Execute balance verification and payment dispatch await processWithdrawal(userId, amount); } finally { // Release lock only if we still own it (Lua script verification) await releaseLockSafely(redis, lockKey, transactionId); } ```
If Redis is unreachable, the system fails closed—rejecting the withdrawal rather than proceeding unguarded. In financial engineering, availability must always yield to correctness.
Frequently asked questions
- What engineering problem does this guide tackle?
- Financial engineering is fundamentally different from typical CRUD application development. In social or e-commerce apps, a temporary glitch or dropped message is an inconvenience; in financial systems, a single race condition or unhandled payment webhook can create unauthorized money, corrupt account balances, or cause untraceable financial loss. External payment providers operate asynchronously over unreliable mobile networks, frequently replaying webhooks or timing out ambiguously.
- How is the system architected?
- NestFi implements a tiered storage architecture designed around the fundamental accounting equation: Assets = Liabilities + Equity. PostgreSQL serves as the single source of financial truth, enforcing immutable double-entry ledger constraints where every transaction consists of balanced debits and credits within an atomic database transaction. MongoDB captures high-volume, append-only operational audit trails and webhook payloads. Redis provid…
- What measurable outcomes can you expect?
- A resilient financial coordination platform engineered for zero ledger drift. Eliminates double-crediting through database-level idempotency on M-Pesa transaction IDs. Replaces vulnerable mutable balance fields with verifiable append-only ledger entries, accompanied by real-time WebSocket status updates and complete audit trails.
Related case studies & projects
Developer Journalfor narrative deep dives, or get in touch for a conversation.