I Built My Own Messaging App — And It Was Way Harder Than I Expected
I Built My Own Messaging App — And It Was Way Harder Than I Expected is an expert-authored guide on https://ancel.co.ke/guides. Architecting private, distributed real-time messaging: client-side cryptographic key derivation, horizontally scalable WebSocket sync with Redis Pub/Sub, three-tier storage partitioning and network resilience. Lessons from Inkly. It answers how teams should think about: Real-time communication platforms face twin engineering challenges that pull in opposite directions: security and scalab…
Part of topic: real-time-systems, Full-Stack Systems Design, Zero-Trust & Auditable Infrastructure, Backend & Modular Systems
What problem does this guide address?
Real-time communication platforms face twin engineering challenges that pull in opposite directions: security and scalability. Many modern messaging apps sacrifice user privacy by storing plaintext conversations on centralized servers, creating lucrative targets for data breaches. Conversely, implementing end-to-end encryption often degrades performance and introduces complex synchronization challenges across distributed server clusters when users roam between networks or go offline.
How is the system architected?
Inkly resolves these tensions through a zero-knowledge, three-tier distributed architecture. Client devices derive cryptographic keys locally using X25519 elliptic-curve Diffie-Hellman and HKDF key derivation; the server infrastructure acts purely as an encrypted transit and storage medium, with zero visibility into plaintext content. The backend tiers responsibilities across three data layers: PostgreSQL manages authoritative identity and relationship state, MongoDB stores encrypted message archives, and Redis handles distributed locking, presence tracking, and cross-node WebSocket event distribution via Redis Pub/Sub.
What outcomes can you measure?
A functional, privacy-first real-time messaging architecture. Enforces zero-knowledge confidentiality where message contents cannot be read by intermediate servers. Demonstrates horizontal WebSocket scalability across isolated Node.js instances via Redis adapters, backed by circuit breakers to prevent cascade failures.
Deep dive
The Illusion of Easy Real-Time Messaging
Every software engineer has seen a 15-minute tutorial showing how to build a chat application with Node.js and WebSockets. You set up a server, listen for `connection`, and broadcast incoming messages with `io.emit('message', data)`.
That works for five users on localhost. In production, building a messaging platform that is genuinely private, horizontally scalable, and resilient to packet loss is one of the most demanding problems in software engineering:
* The Single-Node Bottleneck: A single WebSocket server is bound by CPU and memory. When you scale horizontally by spinning up ten backend instances behind a load balancer, User A connected to Server 1 cannot talk to User B connected to Server 4 without a distributed pub/sub backbone. * The Plaintext Vulnerability: If your server decrypts and re-encrypts messages, a single SQL injection, misconfigured backup, or malicious insider can expose every conversation. True privacy demands that the server never possesses the keys. * The Offline Delivery Guarantee: In mobile environments, devices lose connectivity constantly. Messages dispatched while a recipient is offline must be safely stored, ordered, and delivered without duplication when the device reconnects.
Inkly was built from the ground up to solve these architectural constraints.
Zero-Knowledge Cryptographic Foundation
Inkly implements a zero-knowledge protocol termed "Sacred Bonds". Before two users can exchange messages, their clients establish a cryptographically verifiable channel without revealing private keys to the server:
1. Key Generation: Each client generates an asymmetric X25519 key pair on device upon account registration. The public key is published to the directory; the private key remains exclusively in secure device storage (e.g., iOS Keychain, Android Keystore). 2. Channel Derivation: When User A initiates a conversation with User B, both clients compute a shared master secret using Elliptic-Curve Diffie-Hellman (ECDH). 3. Session Key Ratchet: Using HMAC-based Extract-and-Expand Key Derivation (HKDF), the client derives ephemeral symmetric AES-256-GCM keys for each message:
```typescript // Client-side execution only async function encryptMessage( plaintext: string, sharedSecret: Uint8Array, nonce: Uint8Array ): Promise<EncryptedPayload> { const sessionKey = await crypto.subtle.deriveKey( { name: "HKDF", hash: "SHA-256", salt: new Uint8Array(), info: new TextEncoder().encode("InklyMessagePayload") }, sharedSecretKey, { name: "AES-GCM", length: 256 }, false, ["encrypt"] );
const ciphertext = await crypto.subtle.encrypt( { name: "AES-GCM", iv: nonce }, sessionKey, new TextEncoder().encode(plaintext) );
return { ciphertext: bufferToHex(ciphertext), nonce: bufferToHex(nonce) }; } ```
The Node.js server receives only the recipient ID, sender ID, timestamp, and hex-encoded ciphertext. Even under subpoena or catastrophic database compromise, the message contents cannot be read.
Horizontal WebSocket Scaling via Redis Pub/Sub
When scaling WebSockets across multiple server nodes, clients connecting through a reverse proxy (e.g., NGINX) are distributed across different application instances.
If Alice is connected to Node A and Bob is connected to Node B, Node A has no direct socket reference to Bob.
``` [Alice] ---> (Node A) / (Node A) ---> [Alice] ===> [Redis Pub/Sub Backbone] === [Bob] ---> (Node B) / (Node B) ---> [Bob] ```
Inkly integrates the `@socket.io/redis-adapter`. When Alice sends a message to Bob's room: 1. Node A publishes the event to Redis on the channel `socket.io#/#`. 2. Redis broadcasts the payload across all subscribed Node instances. 3. Node B identifies that Bob is connected to its local process and forwards the event over Bob's active WebSocket connection. 4. Node A recognizes that Alice is the sender and emits an acknowledgment back to her client.
By decoupling connection termination from message routing, server nodes remain completely stateless and can be scaled out or in dynamically without dropping active conversations.
Three-Tier Storage Partitioning
To balance high-throughput ingestion with strict relational constraints, Inkly partitions persistence across three dedicated storage tiers:
1. PostgreSQL (Authoritative Identity & Access): * Handles user accounts, cryptographic public keys, contact rosters, and channel permissions. * Relational constraints prevent phantom users or unauthorized channel subscriptions. 2. MongoDB (Encrypted Message Archives): * Optimized for append-heavy write patterns. * Stores ciphertext payloads, message delivery receipts, and timestamps as document collections partitioned by conversation ID. * TTL indexes automatically clean up transient ephemeral media messages after their retention window expires. 3. Redis (Ephemeral Presence & Rate Limiting): * Tracks user online status and heartbeat pings with 30-second TTLs. * Enforces sliding-window token-bucket rate limits per client socket to prevent spam floods.
Failure Isolation with Circuit Breakers
In distributed real-time systems, the slowest component dictates the stability of the entire cluster. If the MongoDB cluster encounters high disk I/O and write latencies increase from 5ms to 500ms, active WebSocket connections can quickly back up, exhausting Node.js event loop memory.
Inkly wraps secondary database operations in circuit breakers (using Opossum):
```typescript const messageSaveBreaker = new CircuitBreaker(saveToMongo, { timeout: 3000, errorThresholdPercentage: 50, resetTimeout: 10000 });
messageSaveBreaker.fallback(() => { // Graceful degradation: Buffer message to Redis temporary queue return bufferMessageToRedisFallback(messagePayload); }); ```
If MongoDB becomes unresponsive, the circuit breaker opens immediately. In-flight messages are temporarily buffered into Redis memory while real-time WebSocket delivery continues unimpeded, preventing database degradation from crashing the real-time communications layer.
Frequently asked questions
- What engineering problem does this guide tackle?
- Real-time communication platforms face twin engineering challenges that pull in opposite directions: security and scalability. Many modern messaging apps sacrifice user privacy by storing plaintext conversations on centralized servers, creating lucrative targets for data breaches. Conversely, implementing end-to-end encryption often degrades performance and introduces complex synchronization challenges across distributed server clusters when users roam between networks or go offline.
- How is the system architected?
- Inkly resolves these tensions through a zero-knowledge, three-tier distributed architecture. Client devices derive cryptographic keys locally using X25519 elliptic-curve Diffie-Hellman and HKDF key derivation; the server infrastructure acts purely as an encrypted transit and storage medium, with zero visibility into plaintext content. The backend tiers responsibilities across three data layers: PostgreSQL manages authoritative identity and rel…
- What measurable outcomes can you expect?
- A functional, privacy-first real-time messaging architecture. Enforces zero-knowledge confidentiality where message contents cannot be read by intermediate servers. Demonstrates horizontal WebSocket scalability across isolated Node.js instances via Redis adapters, backed by circuit breakers to prevent cascade failures.
Related case studies & projects
Developer Journalfor narrative deep dives, or get in touch for a conversation.