Skip to main content
Inkly: Architecting a High-Availability Real-Time Messaging Platform with Zero-Knowledge Privacy

Inkly: Architecting a High-Availability Real-Time Messaging Platform with Zero-Knowledge Privacy

April 10, 2025 3 min read Node.js, Flutter, WebSockets, Security, PostgreSQL, MongoDB, Redis, Cryptography

Inkly: Architecting a High-Availability Real-Time Messaging Platform with Zero-Knowledge Privacy is a Developer Journal article by Ancel Ajanga on https://ancel.co.ke. Deep-dive into Inkly messaging architecture: Zero-Knowledge Sacred Bond key agreements (X25519/HKDF), horizontally scalable Redis WebSocket adapters, and 3-tier storage. It focuses on: Building modern messaging requires balancing two opposing forces: absolute cryptographic privacy (the server cannot read user messages) and high operational performan…

Deep-dive into Inkly messaging architecture: Zero-Knowledge Sacred Bond key agreements (X25519/HKDF), horizontally scalable Redis WebSocket adapters, and 3-tier storage.

Who this is for

Distributed systems engineers, mobile messaging architects, and security engineers designing end-to-end encrypted, low-latency communication platforms.

Problem

Building modern messaging requires balancing two opposing forces: absolute cryptographic privacy (the server cannot read user messages) and high operational performance with offline synchronization and fast message delivery.

Business outcome

A scalable messaging system with zero-knowledge cryptographic ratchets, Redis Pub/Sub WebSocket clustering, and a resilient three-tier data layer (PostgreSQL for vault metadata, MongoDB for message streams, Redis for ephemeral caching).

Metrics

  • Real-time message delivery over WebSockets
  • Horizontal WebSocket scaling across nodes via a Redis-backed adapter
  • Zero server access to message plaintext through X25519/HKDF key agreement

Building a real-time messaging application is often treated as a solved tutorial exercise: connect a client to a WebSocket, broadcast an event, and append the message to a database.

In production, however, messaging is one of the most demanding distributed systems problems in software engineering. You must deliver messages quickly to many concurrent mobile clients, maintain reliable offline backfill queues, scale WebSockets across multiple physical server nodes, and—above all—guarantee that **even if your servers are compromised, user message contents remain completely unreadable**.

I architected Inkly to solve this dual challenge of **Zero-Knowledge Privacy** and **High-Availability Scale**.

The Problem: The Privacy-Performance Paradox

Modern communication platforms routinely fail on either security or scale: 1. **Server Snooping and Data Breaches**: Traditional platforms store message plaintexts in relational tables. A single compromised database credential or rogue employee exposes millions of private conversations. 2. **WebSocket Single-Node Bottlenecks**: A single server process can only hold so many open TCP connections before running out of file descriptors. When you scale to multiple nodes, a user connected to Node A cannot message a user connected to Node B without a low-latency pub/sub coordination backbone. 3. **Mobile Network Instability**: Mobile phones constantly lose connectivity when moving between cell towers, switching to Wi-Fi, or entering tunnels. A production messaging system must handle reconnection storms and order-preserving message reconciliation without dropping packets.

Architectural Approach: Zero-Knowledge Bonds and Distributed Nodes

Inkly is built with a **Node.js backend, Flutter mobile client**, and a distributed three-tier persistence layer.

### 1. Zero-Knowledge "Sacred Bonds" (X25519 & HKDF) Inkly implements client-side cryptographic key agreements: - When two users establish a connection, their client devices perform an **X25519 Elliptic Curve Diffie-Hellman (ECDH)** key exchange. - Symmetric session keys are derived using **HMAC-based Extract-and-Expand Key Derivation Function (HKDF)** with SHA-256. - Messages are encrypted using **AES-256-GCM** on the sender's device before touching the network. - The server acts as a blind message relay. It inspects metadata (recipient ID, message timestamp, sequence number), but has zero mathematical ability to decrypt the ciphertext payload.

### 2. Horizontally Scalable WebSockets with Redis Pub/Sub To scale beyond a single Node.js process: - WebSockets are decoupled across multiple server instances using the **Redis Socket.IO adapter**. - When User 1 sends a message to User 2, Node A publishes the encrypted payload to a Redis channel (`channel:user:${recipientId}`). - Node B, which holds the active WebSocket connection for User 2, receives the message from Redis and pushes it immediately down the TCP socket. - Delivery latency remains under 100 milliseconds even across distributed nodes.

ypescript
// Blind message relay with Redis Pub/Sub broadcast
async function relayEncryptedMessage(
  senderId: string,
  recipientId: string,
  encryptedPayload: EncryptedMessagePacket
) {
  // 1. Verify sender identity and rate limits
  await rateLimiter.consume(senderId);

  // 2. Persist encrypted ciphertext blob in MongoDB
  const saved = await messageRepo.create({
    senderId,
    recipientId,
    ciphertext: encryptedPayload.ciphertext,
    iv: encryptedPayload.iv,
    tag: encryptedPayload.authTag,
    timestamp: new Date()
  });

  // 3. Publish to Redis for real-time delivery across nodes
  await redisPublisher.publish(
    `channel:user:${recipientId}`,
    JSON.stringify(saved)
  );
}

### 3. Three-Tier Storage Strategy Rather than forcing all data into one database: - **PostgreSQL**: Stores authoritative relational identities, cryptographic public key bundles, user profiles, and organization memberships. - **MongoDB**: Stores high-throughput append-only encrypted message streams and conversation metadata. - **Redis**: Handles real-time distributed pub/sub routing, rate limiting, and presence tracking (online/offline/typing).

Resilience Under Network Failure

To handle flaky mobile connections: - **Client-Side SQLite Outbox**: If a user sends a message while offline, the message is queued locally in SQLite with a provisional ID. - **Incremental Sequence Sync**: Upon reconnecting, the client issues a `sync` command with its last received message sequence number. The backend queries MongoDB for missed messages since that sequence, backfilling the client in exact chronological order.

Key Engineering Takeaways

1. **Shift Cryptography to the Client**: True zero-knowledge privacy requires that encryption keys never leave the user's physical device. The server must remain an untrusted relay. 2. **Decouple WebSockets with Pub/Sub Early**: Never build real-time communication assuming a single server instance. Always route socket events through Redis or a message broker from day one. 3. **Tier Your Storage by Access Pattern**: Use relational databases for identity and key vaults, document stores for high-volume encrypted messages, and in-memory caches for ephemeral routing.

Explore the Inkly Project Page or review my other [software engineering case studies](/case-studies).

Trade-offs

Stored encrypted message blobs in MongoDB for rapid ingestion while keeping identity accounts and cryptographic public keys in PostgreSQL for strict relational integrity; shifted cryptographic key derivation entirely to client runtimes.

Challenges faced

Handling reliable offline message queuing and backfill synchronization when mobile clients reconnect after days of intermittent network coverage.