Author: robert

Posted on by

Design of a Real-Time Collaborative Document Editing System (like Google Docs)

Below is a comprehensive design for a cloud-based real-time collaborative document editing system that meets the specified functional and non-functional requirements. The system allows multiple users to edit documents simultaneously, supports offline editing with synchronization, and scales to millions of users while ensuring security, low latency, and efficient storage.

Solution Overview

The system is designed to provide a seamless collaborative editing experience similar to Google Docs, addressing challenges such as real-time synchronization, conflict resolution, offline support, scalability, and security. Here’s how each component is architected to meet the requirements.

Key Components and Design Decisions

1. Data Model & Storage

  • Document Storage:
    Documents are stored using a Conflict-Free Replicated Data Type (CRDT)-based system (e.g., Yjs or Automerge). CRDTs represent the document as a sequence of operations (e.g., insert, delete, update) rather than plain text, enabling efficient merging of concurrent edits. This approach ensures that multiple users can edit the same document without conflicts.
  • Versioning Storage:
    An event log records every operation applied to the document, allowing reconstruction of any previous version by replaying the log. To optimize performance and avoid replaying the entire log for older versions, periodic snapshots of the document state are saved (e.g., every 100 operations or at fixed time intervals). This balances storage efficiency with fast version retrieval.
  • Database Choice:
    A NoSQL database (e.g., DynamoDB or MongoDB) is used to persist the CRDT data and event logs, as it supports high write throughput and horizontal scaling better than traditional SQL databases.

Why?
CRDTs natively handle concurrent edits, and the event log with snapshots minimizes storage overhead while enabling efficient versioning.

2. Concurrency & Conflict Resolution

  • Technique:
    Use operation-based CRDTs (e.g., Logoot or Treedoc) to manage concurrent edits. Each edit is an operation with a unique identifier and timestamp, ensuring that operations can be applied in any order and still converge to the same document state.
  • Handling Same-Word Edits:
    If two users edit the same word simultaneously (e.g., User A inserts “x” at position 5, and User B deletes character 5), the CRDT assigns unique identifiers to each operation based on user ID and timestamp. The merge function ensures both changes are preserved (e.g., “x” is inserted, and the deletion shifts accordingly), avoiding data loss.

Why?
CRDTs simplify conflict resolution compared to Operational Transformations (OT) and eliminate the need for locking, providing a seamless user experience under high concurrency.

3. Real-Time Synchronization

  • Communication Protocol:
    Use WebSockets for real-time, bidirectional communication between clients and the server. When a user makes an edit, the operation is sent to the server via WebSocket, which broadcasts it to all connected clients editing the same document. Clients apply the operation to their local CRDT state.
  • Event Propagation:
    The server acts as a central coordinator, receiving operations from clients and pushing them to others in near real-time (targeting sub-100ms latency). WebSocket connections are maintained for each active user per document.

Why?
WebSockets offer low-latency, full-duplex communication, making them ideal for real-time updates compared to Server-Sent Events (unidirectional) or gRPC (more complex for this use case).

4. Offline Editing & Sync

  • Offline Editing:
    When a user goes offline, their edits are stored locally in the browser using IndexedDB as a queue of operations. The client continues to apply these operations to its local CRDT state, allowing uninterrupted editing.
  • Synchronization:
    Upon reconnection, the queued operations are sent to the server. The server merges them with the current document state using the CRDT merge function, resolving conflicts automatically. If the merge result is ambiguous (e.g., significant divergence), users can optionally review changes via a manual conflict resolution interface.

Why?
Local storage ensures offline functionality, and CRDTs handle conflict resolution naturally, minimizing data loss during sync.

5. Scalability & Performance

  • Sharding:
    Documents are sharded across multiple servers based on document ID, distributing the load and enabling horizontal scaling. Each shard manages a subset of documents and their associated event logs.
  • Event-Driven Architecture:
    A message broker (e.g., Kafka or RabbitMQ) handles operation propagation. When an edit occurs, the operation is published to the broker, and relevant servers and clients consume it. This decouples the system, improving scalability and fault tolerance.
  • Caching:
    Frequently accessed documents and their current states are cached in memory (e.g., using Redis) to reduce database load and ensure fast reads.
  • Performance Optimization:
    Sub-100ms latency is achieved through WebSockets, in-memory caching, and efficient CRDT operations. Periodic snapshots reduce computation for version reconstruction.

Why?
Sharding and an event-driven approach scale the system to millions of users, while caching and snapshots ensure high performance even with large documents and frequent edits.

6. Security & Access Control

  • Authentication:
    Users are authenticated using OAuth2 or JWT, ensuring only authorized individuals can access the system.
  • Authorization:
    Implement Role-Based Access Control (RBAC) with roles such as:
  • Owner: Can edit, share, and delete the document.
  • Editor: Can edit the document.
  • Viewer: Can only view the document.
    Role assignments are stored in a centralized database and checked for every operation (read, write, delete).
  • Encryption:
  • In Transit: Use TLS to secure all WebSocket and HTTP communications.
  • At Rest: Encrypt sensitive document data in the database using AES-256.

Why?
RBAC ensures fine-grained permissions, and encryption protects data confidentiality, meeting security requirements.

7. Versioning & Undo Feature

  • Document History:
    The event log stores all operations (delta changes) applied to the document, enabling reconstruction of any version by replaying operations from the start or the nearest snapshot. Snapshots are taken periodically to optimize retrieval.
  • Undo Feature:
    Clients maintain a local stack of recent operations. An undo reverses the last operation (e.g., deleting an inserted character) and sends the reversal to the server, which propagates it to other clients.

Why?
Delta changes are storage-efficient, and snapshots improve performance. The local undo stack provides a responsive user experience.

Final Architecture

  • Clients: Web browsers or mobile apps connect via WebSockets for real-time collaboration and use IndexedDB for offline edits.
  • API Gateway: Authenticates users, enforces RBAC, and routes requests to services.
  • Document Service: Manages document CRUD operations, coordinates real-time updates, and applies CRDT logic.
  • CRDT Engine: Merges operations and maintains document consistency.
  • Event Log Database: Persists operations and snapshots (e.g., DynamoDB).
  • Message Broker: Distributes operations across servers and clients (e.g., Kafka).
  • Caching Layer: Stores document states in memory (e.g., Redis).
  • Storage Layer: Holds encrypted document data and metadata.

Meeting Requirements

Functional Requirements

  1. CRUD Operations: Supported via the document service.
  2. Concurrent Editing: Enabled by CRDTs and WebSockets.
  3. Conflict Resolution: Handled automatically by operation-based CRDTs.
  4. Version History: Provided by the event log and snapshots.
  5. Offline Editing: Supported with local storage and sync via CRDTs.
  6. Speed: Optimized with caching and efficient CRDT operations.

Non-Functional Requirements

  1. Low Latency: Sub-100ms updates via WebSockets and caching.
  2. Scalability: Achieved with sharding and event-driven architecture.
  3. Fault Tolerance: Message broker and geo-redundant storage ensure minimal data loss.
  4. Security: RBAC, TLS, and encryption protect access and data.
  5. High Availability: Sharding and redundancy target 99.99% uptime.
  6. Efficient Storage: Delta changes and periodic snapshots minimize duplication.

Conclusion

This design delivers a robust, scalable, and secure real-time collaborative document editing system. By leveraging CRDTs for conflict-free editing, WebSockets for low-latency synchronization, and an event-driven architecture for scalability, it meets all specified requirements while providing a user experience comparable to Google Docs.

Posted on by

Design a Real-Time Collaborative Document Editing System (like Google Docs)

Problem Statement:

You need to design a cloud-based real-time collaborative document editor that allows multiple users to edit the same document simultaneously. The system should support real-time updates, conflict resolution, and offline editing.

Requirements:

Functional Requirements:

  1. Users can create, edit, and delete documents.
  2. Multiple users can edit the same document concurrently and see real-time changes.
  3. The system should handle conflicting edits and merge them efficiently.
  4. Users should be able to view document history and revert to previous versions.
  5. Users should be able to edit documents offline, and changes should sync once they’re back online.
  6. The system should be fast, even for large documents with thousands of edits per second.

Non-Functional Requirements:

  1. Low latency (sub-100ms updates for real-time collaboration).
  2. Scalability: The system should support millions of users simultaneously.
  3. Fault tolerance: Ensure minimal data loss even if servers crash.
  4. Security: Handle role-based access control (RBAC) for documents (read-only, edit, admin).
  5. High availability: 99.99% uptime with geo-redundancy.
  6. Efficient storage: Maintain versions without excessive data duplication.

What I Expect –

  1. A quick architecture diagram on – https://excalidraw.com/ outlining major blocks
  2. Any database design / schema.

Discussion Points:

  1. Data Model & Storage
    • How will you store documents? (SQL vs NoSQL, CRDT-based storage, event logs)
    • How do you efficiently store document history without excessive duplication?
  2. Concurrency & Conflict Resolution
    • Which technique would you use to merge concurrent edits? (CRDTs, Operational Transformations, or custom locking mechanisms)
    • How do you handle two users making changes to the same word at the same time?
  3. Real-Time Synchronization
    • How would you design the event propagation system for real-time updates?
    • Would you use WebSockets, Server-Sent Events (SSE), or gRPC?
  4. Offline Editing & Sync
    • How do you handle offline users editing a document and later syncing changes?
    • How would you resolve conflicts when an offline user makes a conflicting edit?
  5. Scalability & Performance
    • How do you scale the system to handle millions of documents and users?
    • Would you use sharding, leader-based consensus, or event-driven architecture?
    • How would you ensure that real-time updates are fast and efficient?
  6. Security & Access Control
    • How do you prevent unauthorized users from modifying a document?
    • How do you implement role-based access control (RBAC)?
  7. Versioning & Undo Feature
    • How do you store document history efficiently?
    • Would you store delta changes or full snapshots?

Key Points

  • It seems likely that a real-time collaborative document editing system can be designed using ShareJS for real-time updates and MongoDB for storage, supporting features like concurrent editing and offline syncing.
  • Research suggests that Operational Transformation (OT) is effective for handling conflicts, while Conflict-Free Replicated Data Types (CRDTs) like Automerge could simplify offline editing but may need additional conflict resolution for text.
  • The evidence leans toward using sharding and a pub-sub system like Redis for scalability, ensuring low latency and high availability for millions of users.

System Overview

This design aims to create a cloud-based real-time collaborative document editor, similar to Google Docs, that supports multiple users editing simultaneously, handles conflicts, and allows offline editing with sync capabilities. The system will be fast, scalable, and secure, meeting all functional and non-functional requirements.

Architecture

The system architecture includes:

  • Clients: Web browsers using WebSockets for real-time communication.
  • Load Balancer: Distributes connections to appropriate shards based on document ID for scalability.
  • Shards: Each shard handles a subset of documents, with multiple server instances running ShareJS for real-time collaboration.
  • Database: A MongoDB cluster stores document data, with replication for fault tolerance.
  • Pub-Sub System: Redis facilitates real-time updates within shards, ensuring low latency.

For offline editing, clients queue local operations and sync them when reconnected, leveraging ShareJS’s OT for conflict resolution.

Database Design

The database schema includes:

  • documents collection: Stores current document state and version.
  • permissions collection: Manages role-based access control (RBAC) for users.
  • operations collection: Logs all operations for versioning and undo functionality.

This design ensures efficient storage and quick access to document history without excessive duplication.

Detailed System Design and Analysis

This section provides a comprehensive analysis of designing a real-time collaborative document editing system, addressing all requirements and discussion points. The design leverages established technologies and methodologies to ensure scalability, performance, and user experience.

System Requirements and Design Goals

The system must support:

  • Functional Requirements: Creation, editing, and deletion of documents; real-time concurrent editing; conflict resolution; document history and versioning; offline editing with sync; high performance for large documents.
  • Non-Functional Requirements: Low latency (sub-100ms updates), scalability for millions of users, fault tolerance, security with RBAC, high availability (99.99% uptime), and efficient storage.

The design aims to balance these requirements using a combination of Operational Transformation (OT) for real-time collaboration and considerations for offline editing, ensuring a robust and scalable solution.

Data Model and Storage

Storage Strategy

The system uses ShareJS, which implements OT, for real-time collaboration. Documents are stored in a MongoDB cluster for scalability and fault tolerance. The storage strategy involves:

  • Document State: Stored as JSON objects in the documents collection, with each document having a type (e.g., “text”) and current data.
  • Operation History: Maintained in the operations collection for versioning and undo, logging each operation with details like document ID, operation data, timestamp, and user ID.
  • Snapshots: Considered for efficiency, where periodic snapshots of the document state are stored to reduce the need to replay long operation logs for historical versions.
Database Schema

The database design is as follows:

CollectionFieldsDescription
documents_id (string), type (string), data (object), v (integer)Stores current document state and version number
permissions_id (string), users (array of objects with user_id and role)Manages RBAC for each document
operations_id (string), document_id (string), operation_data (object), timestamp (date), user_id (string)Logs all operations for versioning and undo

This schema ensures efficient storage and retrieval, with indexing on document_id for quick access to operations and permissions.

SQL vs. NoSQL

MongoDB was chosen over SQL due to its flexibility with JSON-like documents and scalability for handling large volumes of concurrent writes and reads, essential for real-time collaboration.

CRDT-Based Storage

Initially, CRDTs like Automerge were considered for their offline-first capabilities and conflict-free merging. However, for real-time text editing, OT was preferred due to better handling of concurrent edits without manual conflict resolution, which Automerge might require for text overlaps. Automerge remains a viable option for offline editing, but the design leans toward OT for consistency.

Concurrency and Conflict Resolution

Technique Selection
  • Operational Transformation (OT): Chosen for real-time collaboration, as implemented by ShareJS. OT transforms concurrent operations to maintain document consistency, ensuring that when two users edit the same word simultaneously, the server adjusts operations to merge changes seamlessly.
  • Conflict Resolution: OT handles conflicts by transforming operations based on their order and position, preserving all changes without loss. For example, if User A inserts text at position 5 and User B deletes at position 5 concurrently, OT adjusts the operations to apply both changes correctly.
Comparison with CRDTs

CRDTs were evaluated, particularly Automerge, for their decentralized merging capabilities. However, for text editing, CRDTs might preserve conflicting edits (e.g., both insertions at the same position), requiring application-level resolution, which could disrupt real-time flow. OT’s centralized approach ensures a single consistent view, making it more suitable.

Real-Time Synchronization

Event Propagation System
  • WebSockets: Used for bidirectional communication, enabling clients to send edits and receive updates in real-time. Each document has a unique channel in the pub-sub system (Redis), ensuring updates are broadcast to all connected users.
  • Pub-Sub Implementation: Redis facilitates efficient message passing within each shard, with server instances subscribing to document channels to propagate changes, achieving sub-100ms latency.
Technology Choice

WebSockets were preferred over Server-Sent Events (SSE) or gRPC due to their bidirectional nature, essential for real-time collaboration. gRPC could be considered for high-performance backend communication, but WebSockets align better with browser-based clients.

Offline Editing and Sync

Handling Offline Users
  • When offline, clients queue local operations using ShareJS’s client-side capabilities, storing them locally. Upon reconnection, these operations are sent to the server.
  • The server applies these operations, transforming them based on the current document state to handle any intervening changes, ensuring consistency.
Conflict Resolution for Offline Edits
  • The server uses OT to merge offline operations with the current state. If conflicts arise (e.g., offline user edited the same part as online users), OT transforms the operations to resolve them, maintaining document integrity.
  • This approach ensures that offline edits are not lost and are seamlessly integrated, with the server broadcasting the updated state to all connected clients.

Scalability and Performance

Scaling Strategy
  • Sharding: Documents are distributed across multiple shards based on document ID, with each shard handling a subset. This distributes load and ensures scalability for millions of users and documents.
  • Leader-Based Consensus: Each shard has a primary server instance for document updates, with secondary instances for failover, ensuring consistency and availability.
  • Event-Driven Architecture: The pub-sub system (Redis) enables event-driven updates, reducing server load by broadcasting changes efficiently.
Ensuring Fast Updates
  • Low latency is achieved by routing users to the nearest data center (geo-redundancy) and using WebSockets for real-time communication. Redis’s in-memory data structure ensures quick message passing, meeting the sub-100ms requirement.
  • For large documents, ShareJS’s OT implementation is optimized for frequent updates, with periodic snapshots reducing the need for full operation replays.

Security and Access Control

Preventing Unauthorized Access
  • All communications are encrypted using HTTPS for web traffic and secure WebSockets, ensuring data privacy.
  • Authentication is handled through an identity provider, with user sessions validated before allowing operations.
Implementing RBAC
  • The permissions collection stores user roles (read-only, edit, admin) for each document. Before applying operations, the server checks the user’s role, denying unauthorized actions. This ensures fine-grained access control, meeting security requirements.

Versioning and Undo Feature

Efficient Storage of History
  • Document history is maintained in the operations collection, logging each edit with timestamp and user ID. This allows replaying operations to reconstruct any version, supporting undo functionality.
  • To optimize storage, periodic snapshots are stored in the documents collection, reducing the need to process long operation logs for historical access.
Delta Changes vs. Full Snapshots
  • The system uses delta changes (operations) for real-time updates, stored in the operation log. Full snapshots are taken at intervals (e.g., every 1000 operations) to balance storage efficiency and quick access, ensuring users can revert to previous versions without excessive computation.

Unexpected Detail: Hybrid Approach Consideration

While OT is central to real-time collaboration, the initial exploration of CRDTs like Automerge highlights a potential hybrid approach for offline editing, where CRDTs could simplify syncing but require additional conflict resolution for text. This dual consideration adds flexibility but increases complexity, which was ultimately resolved by favoring OT for consistency.

Conclusion

This design leverages ShareJS for OT-based real-time collaboration, MongoDB for scalable storage, and Redis for efficient pub-sub, ensuring low latency, high availability, and support for offline editing. The sharding mechanism and RBAC implementation meet scalability and security needs, with operation logs and snapshots providing robust versioning and undo features.

Key Citations

Posted on by

Plugins to install into vscode.

Here’s a collection of essential VS Code extensions for a lead front-end developer working with React and TypeScript:

Core TypeScript & React Extensions

  • ESLint – For code quality and style enforcement
  • Prettier – For consistent code formatting
  • TypeScript – Official extension with IntelliSense
  • ES7+ React/Redux/React-Native snippets – Provides shortcuts for common React patterns

Developer Experience

  • IntelliCode – AI-assisted development with context-aware completions
  • GitLens – Supercharged Git capabilities within editor
  • Import Cost – Shows the size of imported packages
  • Error Lens – Improves error visibility in the editor
  • Auto Rename Tag – Automatically renames paired HTML/JSX tags

Debugging & Testing

  • Jest – For running and debugging tests
  • Debugger for Chrome/Edge – Debugging React apps in browser
  • Redux DevTools – If using Redux for state management

Performance & Code Quality

  • Lighthouse – For performance auditing
  • SonarLint – Detect and fix quality issues
  • Better Comments – Improve comments with alerts, TODOs, etc.

Component Development

  • vscode-styled-components – Syntax highlighting for styled-components
  • Tailwind CSS IntelliSense – If using Tailwind
  • Material Icon Theme – Better file/folder icons

Architecture & Documentation

  • Draw.io Integration – For creating architecture diagrams
  • Todo Tree – Track TODO comments in your codebase
  • Path Intellisense – Autocompletes filenames in imports

Team Collaboration

  • Live Share – Real-time collaborative editing
  • CodeStream – Code discussions and reviews inside VS Code

These extensions cover the technical aspects of React and TypeScript development while also addressing the leadership aspects of documentation, collaboration, and maintainability that are important for a lead developer role.

Posted on by

Pandas JavaScript Equivalent

Pandas is a powerful data manipulation and analysis library in Python, providing data structures and operations for manipulating numerical tables and time series. It is widely used for data manipulation due to its ease of use and extensive functionality.

For Node.js, there are several libraries that offer similar functionalities to Pandas. One such library is PandasJS, which is built on TensorFlow.js and supports tensors out of the box, allowing for groupby, merging, joining, and plotting operations. Another option is Data-Forge, a library inspired by LINQ and Pandas, designed to handle data wrangling tasks efficiently. Additionally, D3.js, although primarily a visualization library, also offers data manipulation capabilities that can be useful for data analysis tasks.13

These libraries provide robust solutions for handling and analyzing data within JavaScript applications, offering features comparable to Pandas in Python.3

For the most up-to-date and actively maintained options, you might want to consider Polars JS, which is noted for being even better than Pandas in Python.

Posted on by

Time Series + Predictive Analytics

I have had some interesting back-end questions posted to me recently.

Implementing a time series store / and sum_submerge method.
In this particular vein, I felt like solutions similar to ReductStore and PyStore were worth a look.

But I felt that I was at a loss for the overall theory in terms of time-series data vs a more traditional relational data used to model and create SaaS like I have been building most of my life.
I can definitely see how having a fleet of GPU’s one would want to collect telemetry data, and then use said data to quantify their performance and lifespan.

Using predictive analytics to predict the failure of a device, and preemptively removing it a top tier where the best clients are paying top dollar for said fleet of devices.
Would seem like a good idea to try and create a dataset of devices, and optimal telemetry vs thresholds for failure.
Also tracking deltas on things which could signify performance degradation.

Another crushing boulder has been dropped on me, with all the stuff that I don’t know being tacked on. Feels like atlas has become a splatter.

Posted on by

JavaScript New Features from ES5 to ESNext

ECMAScript 5 (2009)

  • Strict Mode ('use strict'): Enforces better coding practices.
  • Array Methods: forEach, map, filter, reduce, some, every.
  • Object Methods: Object.keys(), Object.create().
  • Getter/Setter Properties: Define computed properties.
  • JSON Support: JSON.parse(), JSON.stringify().
  • bind() Method: Binds this to a function.
  • Property Descriptors: Control property attributes like writable, configurable.

ECMAScript 6 (ES6) – 2015

  • let and const: Block-scoped variables.
  • Arrow Functions (=>): Shorter syntax for functions.
  • Template Literals: String interpolation using backticks.
  • Default Parameters: Function parameters with default values.
  • Destructuring: Extract values from objects/arrays.
  • Spread (...) and Rest Parameters: Expanding and collecting values.
  • Classes (class): Syntactic sugar over constructor functions.
  • Modules (import / export): Native module support.
  • Promises: Handle asynchronous operations.
  • Map and Set: New data structures.
  • Generators (function*): Pause and resume execution.

ECMAScript 7 (ES7) – 2016

  • Exponentiation Operator (**): 2 ** 3 === 8.
  • Array.prototype.includes(): Check if an array contains a value.

ECMAScript 8 (ES8) – 2017

  • Async/Await: Simplifies working with Promises.
  • Object Entries and Values: Object.entries(), Object.values().
  • String Padding: padStart(), padEnd().
  • Trailing Commas in Function Parameters: Avoid syntax errors in version control.
  • Shared Memory & Atomics: Multi-threaded JS via SharedArrayBuffer.

ECMAScript 9 (ES9) – 2018

  • Rest/Spread in Objects: { ...obj }.
  • Promise.prototype.finally(): Runs after a Promise resolves/rejects.
  • Asynchronous Iteration (for await...of): Async iterators.

ECMAScript 10 (ES10) – 2019

  • Array.prototype.flat() & flatMap(): Flatten nested arrays.
  • Object.fromEntries(): Convert key-value pairs into objects.
  • Optional Catch Binding: catch { } without explicitly defining an error variable.
  • String Trim Methods: trimStart(), trimEnd().
  • Symbol Description: Symbol('desc').description.

ECMAScript 11 (ES11) – 2020

  • BigInt (123n): Large integer support.
  • Dynamic import(): Asynchronous module loading.
  • Nullish Coalescing (??): x = a ?? 'default'.
  • Optional Chaining (?.): Safe property access.
  • Promise.allSettled(): Resolves after all Promises settle.
  • String matchAll(): Returns all matches in a string.
  • Global This (globalThis): Unified global object access.

ECMAScript 12 (ES12) – 2021

  • Numeric Separators (1_000_000): Improves readability.
  • replaceAll(): Replace all instances in a string.
  • WeakRefs & FinalizationRegistry: Manage memory manually.
  • Logical Assignment (&&=, ||=, ??=): Shorter conditional assignments.

ECMAScript 13 (ES13) – 2022

  • at() Method: Access array elements via negative indices.
  • Object.hasOwn(): Better alternative to hasOwnProperty.
  • Class Private Fields & Methods: #privateField.
  • Top-Level await: await outside async functions.

ECMAScript 14 (ES14) – 2023

  • Array findLast() & findLastIndex(): Find last matching element.
  • Set Methods: union(), intersection(), difference(), symmetricDifference().
  • Hashbang (#!) in Scripts: Support for Unix-style shebangs.
  • Symbols as WeakMap Keys: Improved memory handling.

Upcoming Features (ESNext)

  • Explicit Resource Management (using): Auto-dispose resources.
  • Temporal API: Improved date/time handling.
  • Pipeline Operator (|>): Streamline function chaining.

Posted on by

ES2023 (ECMAScript 2023) Features

ES2023 focused on minor improvements and consistency updates.

1. Array.prototype.toSorted(), toSpliced(), and toReversed()

  • Immutable versions of sort(), splice(), and reverse(), preventing in-place modifications.

Example:

const nums = [3, 1, 4];

console.log(nums.toSorted()); // ✅ [1, 3, 4] (original array remains unchanged)
console.log(nums.toReversed()); // ✅ [4, 1, 3]
console.log(nums.toSpliced(1, 1, 99)); // ✅ [3, 99, 4] (removes index 1, adds 99)

console.log(nums); // ✅ [3, 1, 4] (unchanged)

2. Array.prototype.findLast() and findLastIndex()

  • Similar to find() and findIndex(), but search from the end.

Example:

const arr = [1, 2, 3, 4, 5];

console.log(arr.findLast(n => n % 2 === 0)); // ✅ 4
console.log(arr.findLastIndex(n => n % 2 === 0)); // ✅ 3

3. RegExp.prototype.hasIndices

  • Checks if a regex was created with the /d flag.

Example:

const regex = /test/d;
console.log(regex.hasIndices); // ✅ true

4. Symbol.prototype.description Now Writable

  • The description property of Symbol objects can be modified.

Example:

const sym = Symbol("original");
console.log(sym.description); // ✅ "original"

5. WeakMap.prototype.emplace() and WeakSet.prototype.emplace() (Proposal)

  • A shortcut for setting values only if a key doesn’t already exist. (Not finalized but expected in future updates.)

Example:

const weakMap = new WeakMap();
weakMap.emplace({}, () => "newValue"); // ✅ Sets value only if key doesn’t exist

Summary of Features

FeatureES2022ES2023
Private fields/methods in classes✅
Static fields/methods in classes✅
Object.hasOwn()✅
RegExp /d flag (match indices)✅
Error.cause✅
Array.prototype.at()✅
Top-level await in modules✅
Array.prototype.toSorted(), toReversed(), toSpliced()✅
Array.prototype.findLast() and findLastIndex()✅
RegExp.prototype.hasIndices✅
Symbol.prototype.description writable✅
Posted on by

ES2022 (ECMAScript 2022) Features

ES2022 introduced several improvements, including new class features, array and object enhancements, and top-level await.

1. Class Fields and Private Methods

  • Public and private fields (# prefix denotes private).
  • Private methods and accessors (# for methods and getters/setters).

Example:

class Person {
    name; // Public field
    #age; // Private field

    constructor(name, age) {
        this.name = name;
        this.#age = age;
    }

    #getAge() { // Private method
        return this.#age;
    }

    getInfo() {
        return `${this.name} is ${this.#getAge()} years old`;
    }
}

const alice = new Person("Alice", 25);
console.log(alice.getInfo()); // ✅ "Alice is 25 years old"
// console.log(alice.#age); // ❌ SyntaxError: Private field '#age' must be declared in an enclosing class

2. Static Class Fields and Methods

  • Classes can now define static fields and private static fields.

Example:

class Counter {
    static count = 0; // Public static field
    static #secret = 42; // Private static field

    static increment() {
        this.count++;
    }

    static getSecret() {
        return this.#secret;
    }
}

Counter.increment();
console.log(Counter.count); // ✅ 1
console.log(Counter.getSecret()); // ✅ 42

3. Object.hasOwn() (Finalized)

  • A safer alternative to Object.prototype.hasOwnProperty().

Example:

const obj = { a: 1 };
console.log(Object.hasOwn(obj, "a")); // ✅ true
console.log(Object.hasOwn(obj, "b")); // ✅ false

4. RegExp Match Indices (/d Flag)

  • Provides start and end positions of matches.

Example:

const regex = /hello/d;
const match = regex.exec("hello world");
console.log(match.indices[0]); // ✅ [0, 5] (start and end positions)

5. Error.cause Property

  • Allows errors to store their original cause.

Example:

try {
    throw new Error("Something went wrong", { cause: "Database connection failed" });
} catch (error) {
    console.log(error.message); // ✅ "Something went wrong"
    console.log(error.cause);   // ✅ "Database connection failed"
}

6. Array.prototype.at()

  • Allows negative indexing for arrays and strings.

Example:

const arr = [10, 20, 30];
console.log(arr.at(-1)); // ✅ 30 (last element)

7. Top-Level await in Modules

  • await can be used outside async functions in ES modules.

Example:

const data = await fetch("https://jsonplaceholder.typicode.com/todos/1").then(res => res.json());
console.log(data);

(Works in ES modules, not in CommonJS.)

Posted on by

Summary of ES2021 feature

ES2021 (ECMAScript 2021) introduced several new features to JavaScript. Here are the key additions:

1. Numeric Separators (_)

  • Helps improve the readability of large numbers.
  • Example: const billion = 1_000_000_000; // Same as 1000000000 const bytes = 0xFF_FF_FF_FF; // Hexadecimal format

2. String replaceAll()

  • Adds a built-in way to replace all occurrences of a substring.
  • Example: const text = "hello world, world!"; console.log(text.replaceAll("world", "JS")); // Output: "hello JS, JS!"

3. Promise any()

  • Similar to Promise.race(), but resolves with the first fulfilled promise (ignores rejected ones).
  • If all promises reject, it throws an AggregateError.
  • Example:
  • const p1 = Promise.reject("Error 1"); const p2 = new Promise(resolve => setTimeout(resolve, 100, "Success!")); const p3 = Promise.reject("Error 2"); Promise.any([p1, p2, p3]).then(console.log).catch(console.error); // Output: "Success!"

4. WeakRefs and FinalizationRegistry

  • Allows for weak references to objects, preventing memory leaks in certain cases.
  • Used for caching and cleaning up resources.
  • Example:
  • let obj = { name: "Alice" }; const weakRef = new WeakRef(obj); obj = null;
  • // The object can now be garbage collected
  • const registry = new FinalizationRegistry((heldValue) => { console.log(`${heldValue} was garbage collected`); }); registry.register(weakRef.deref(), "Alice");

5. Logical Assignment Operators (&&=, ||=, ??=)

  • Shorter syntax for conditional assignments.
  • &&= (AND assignment): let x = true; x &&= false; // x becomes false
  • ||= (OR assignment): let y = null; y ||= "default"; // y becomes "default"
  • ??= (Nullish coalescing assignment): let z = undefined; z ??= "fallback"; // z becomes "fallback"

6. Object.hasOwn()

  • A safer alternative to Object.prototype.hasOwnProperty, avoiding prototype chain issues.
  • Example: const obj = { a: 1 }; console.log(Object.hasOwn(obj, "a")); // true console.log(Object.hasOwn(obj, "b")); // false

Summary of ES2021 Features:

FeatureDescription
Numeric Separators (_)Improves number readability
String.prototype.replaceAll()Replaces all occurrences of a substring
Promise.any()Resolves with the first fulfilled promise
WeakRefs & FinalizationRegistryEnables weak references for memory management
Logical Assignment Operators (&&=, `
Object.hasOwn()A safer alternative to hasOwnProperty

Posted on by

Best Practices for Writing Unit Tests in Node.js

When writing unit tests in Node.js, following best practices ensures your tests are effective, maintainable, and reliable. Additionally, choosing the right testing framework can streamline the process. Below, I’ll outline key best practices for writing unit tests and share the testing frameworks I’ve used.

  1. Isolate Tests
    Ensure each test is independent and doesn’t depend on the state or outcome of other tests. This allows tests to run in any order and makes debugging easier. Use setup and teardown methods (like beforeEach and afterEach in Jest) to reset the environment before and after each test.
  2. Test Small Units
    Focus on testing individual functions or modules in isolation rather than entire workflows. Mock dependencies—such as database calls or external APIs—to keep the test focused on the specific logic being tested.
  3. Use Descriptive Test Names
    Write clear, descriptive test names that explain what’s being tested without needing to dive into the code. For example, prefer shouldReturnSumOfTwoNumbers over a vague testFunction.
  4. Cover Edge Cases
    Test not just the typical “happy path” but also edge cases, invalid inputs, and error conditions. This helps uncover bugs in less common scenarios.
  5. Avoid Testing Implementation Details
    Test the behavior and output of a function, not its internal workings. This keeps tests flexible and reduces maintenance when refactoring code.
  6. Keep Tests Fast
    Unit tests should execute quickly to support frequent runs and smooth development workflows. Avoid slow operations like network calls by mocking dependencies.
  7. Use Assertions Wisely
    Choose the right assertions for the job (e.g., toBe for primitives, toEqual for objects in Jest) and avoid over-asserting. Ideally, each test should verify one specific behavior.
  8. Maintain Test Coverage
    Aim for high coverage of critical paths and complex logic, but don’t chase 100% coverage for its own sake. Tools like Istanbul can help measure coverage effectively.
  9. Automate Test Execution
    Integrate tests into your CI/CD pipeline to run automatically on every code change. This catches regressions early and keeps the codebase stable.
  10. Write Tests First (TDD)
    Consider Test-Driven Development (TDD), where you write tests before the code. This approach can improve code design and testability, though writing tests early is valuable even without strict TDD.

Testing Frameworks I’ve Used

I’ve worked with several testing frameworks in the Node.js ecosystem, each with its strengths. Here’s an overview:

  1. Jest
    • What It Is: A popular, all-in-one testing framework known for simplicity and ease of use, especially with Node.js and React projects.
    • Key Features: Zero-config setup, built-in mocking, assertions, and coverage reporting, plus snapshot testing.
    • Why I Like It: Jest’s comprehensive features and parallel test execution make it fast and developer-friendly.
  2. Mocha
    • What It Is: A flexible testing framework often paired with assertion libraries like Chai.
    • Key Features: Supports synchronous and asynchronous testing, extensible with plugins, and offers custom reporting.
    • Why I Like It: Its flexibility gives me fine-grained control, making it ideal for complex testing needs.
  3. Jasmine
    • What It Is: A behavior-driven development (BDD) framework with a clean syntax.
    • Key Features: Built-in assertions and mocking, plus spies for tracking function calls—no external dependencies needed.
    • Why I Like It: The intuitive syntax suits teams who prefer a BDD approach.
  4. AVA
    • What It Is: A test runner focused on speed and simplicity, with strong support for modern JavaScript.
    • Key Features: Concurrent test execution, async/await support, and a minimalistic API.
    • Why I Like It: Its performance shines when testing asynchronous code.
  5. Tape
    • What It Is: A lightweight, minimalistic framework that outputs TAP (Test Anything Protocol) results.
    • Key Features: Simple, no-config setup, and easy integration with other tools.
    • Why I Like It: Perfect for small projects needing a straightforward testing solution.
<em>// Define the function to be tested</em>
function add(a, b) {
    return a + b;
}

<em>// Test suite for the add function</em>
describe('add function', () => {
    test('adds two positive numbers', () => {
        expect(add(2, 3)).toBe(5);
    });

    test('adds a positive and a negative number', () => {
        expect(add(2, -3)).toBe(-1);
    });

    test('adds two negative numbers', () => {
        expect(add(-2, -3)).toBe(-5);
    });

    test('adds a number and zero', () => {
        expect(add(2, 0)).toBe(2);
    });

    test('adds floating-point numbers', () => {
        expect(add(0.1, 0.2)).toBeCloseTo(0.3);
    });
});

Explanation

  • Purpose: The add function takes two parameters, a and b, and returns their sum. The test suite ensures this behavior works correctly across different types of numeric inputs.
  • Test Cases:
    • Two positive numbers: 2 + 3 should equal 5.
    • Positive and negative number: 2 + (-3) should equal -1.
    • Two negative numbers: (-2) + (-3) should equal -5.
    • Number and zero: 2 + 0 should equal 2.
    • Floating-point numbers: 0.1 + 0.2 should be approximately 0.3. We use toBeCloseTo instead of toBe due to JavaScript’s floating-point precision limitations.
  • Structure:
    • describe block: Groups all tests related to the add function for better organization.
    • test functions: Each test case is defined with a clear description and uses Jest’s expect function to assert the output matches the expected result.
  • Assumptions: The function assumes numeric inputs. Non-numeric inputs (e.g., strings) are not tested here, as the function’s purpose is basic numeric addition.

This test suite provides a simple yet comprehensive check of the add function’s functionality in Jest.

How to Mock External Services in Unit Tests with Jest

When writing unit tests in Jest, mocking external services—like APIs, databases, or third-party libraries—is essential to ensure your tests are fast, reliable, and isolated from real dependencies. Jest provides powerful tools to create mock implementations of these services. Below is a step-by-step guide to mocking external services in Jest, complete with examples.

Why Mock External Services?

Mocking replaces real external services with fake versions, allowing you to:

  • Avoid slow or unreliable network calls.
  • Prevent side effects (e.g., modifying a real database).
  • Simulate specific responses or errors without depending on live systems.

Steps to Mock External Services in Jest

1. Identify the External Service

Determine which external dependency you need to mock. For example:

  • An HTTP request to an API.
  • A database query.
  • A third-party library like Axios.

2. Use Jest’s Mocking Tools

Jest offers several methods to mock external services:

Mock Entire Modules with jest.mock()

Use jest.mock() to replace an entire module with a mock version. This is ideal for mocking libraries or custom modules that interact with external services.

Mock Specific Functions with jest.fn()

Create mock functions using jest.fn() and customize their behavior (e.g., return values or promise resolutions).

Spy on Methods with jest.spyOn()

Mock specific methods of an object while preserving the rest of the module’s functionality.

3. Handle Asynchronous Behavior

Since external services often involve asynchronous operations (e.g., API calls returning promises), Jest provides utilities like:

  • mockResolvedValue() for successful promise resolutions.
  • mockRejectedValue() for promise rejections.
  • mockImplementation() for custom async logic.

4. Reset or Restore Mocks

To maintain test isolation, reset mocks between tests using jest.resetAllMocks() or restore original implementations with jest.restoreAllMocks().

Example: Mocking an API Call

Let’s walk through an example of mocking an external API call in Jest.

Code to Test

Imagine you have a module that fetches user data from an API:

javascript

<em>// api.js</em>
const axios = require('axios');

async function getUserData(userId) {
  const response = await axios.get(`https://api.example.com/users/${userId}`);
  return response.data;
}

module.exports = { getUserData };

javascript

<em>// userService.js</em>
const { getUserData } = require('./api');

async function fetchUser(userId) {
  const userData = await getUserData(userId);
  return `User: ${userData.name}`;
}

module.exports = { fetchUser };

Test File

Here’s how to mock the getUserData function in Jest:

javascript

<em>// userService.test.js</em>
const { fetchUser } = require('./userService');
const api = require('./api');

jest.mock('./api'); <em>// Mock the entire api.js module</em>

describe('fetchUser', () => {
  afterEach(() => {
    jest.resetAllMocks(); <em>// Reset mocks after each test</em>
  });

  test('fetches user data successfully', async () => {
    <em>// Mock getUserData to return a resolved promise</em>
    api.getUserData.mockResolvedValue({ name: 'John Doe', age: 30 });

    const result = await fetchUser(1);
    expect(result).toBe('User: John Doe');
    expect(api.getUserData).toHaveBeenCalledWith(1);
  });

  test('handles error when fetching user data', async () => {
    <em>// Mock getUserData to return a rejected promise</em>
    api.getUserData.mockRejectedValue(new Error('Network Error'));

    await expect(fetchUser(1)).rejects.toThrow('Network Error');
  });
});

Explanation

  • jest.mock(‘./api’): Mocks the entire api.js module, replacing getUserData with a mock function.
  • mockResolvedValue(): Simulates a successful API response with fake data.
  • mockRejectedValue(): Simulates an API failure with an error.
  • jest.resetAllMocks(): Ensures mocks don’t persist between tests, maintaining isolation.
  • Async Testing: async/await handles the asynchronous nature of fetchUser.

Mocking Other External Services

Mocking a Third-Party Library (e.g., Axios)

If your code uses Axios directly, you can mock it like this:

javascript

const axios = require('axios');
jest.mock('axios');

test('fetches user data with Axios', async () => {
  axios.get.mockResolvedValue({ data: { name: 'John Doe' } });
  const response = await axios.get('https://api.example.com/users/1');
  expect(response.data).toEqual({ name: 'John Doe' });
});

Mocking a Database (e.g., Mongoose)

For a MongoDB interaction using Mongoose:

javascript

const mongoose = require('mongoose');
jest.mock('mongoose', () => {
  const mockModel = {
    find: jest.fn().mockResolvedValue([{ name: 'John Doe' }]),
  };
  return { model: jest.fn().mockReturnValue(mockModel) };
});

test('fetches data from database', async () => {
  const User = mongoose.model('User');
  const users = await User.find();
  expect(users).toEqual([{ name: 'John Doe' }]);
});

Advanced Mocking Techniques

Custom Mock Implementation

Simulate complex behavior, like a delayed API response:

javascript

api.getUserData.mockImplementation(() =>
  new Promise((resolve) => setTimeout(() => resolve({ name: 'John Doe' }), 1000))
);

Spying on Methods

Mock only a specific method:

javascript

jest.spyOn(api, 'getUserData').mockResolvedValue({ name: 'John Doe' });

Best Practices

  • Isolate Tests: Always reset or restore mocks to prevent test interference.
  • Match Real Behavior: Ensure mocks mimic the real service’s interface (e.g., return promises if the service is async).
  • Keep It Simple: Use the minimal mocking needed to test your logic.

By using jest.mock(), jest.fn(), and jest.spyOn(), along with utilities for handling async code, you can effectively mock external services in Jest unit tests. This approach keeps your tests fast, predictable, and independent of external systems.

Final Thoughts

By following best practices like isolating tests, using descriptive names, and covering edge cases, you can write unit tests that improve the reliability of your Node.js applications. As for frameworks, I’ve used Jest for its ease and features, Mocha for its flexibility, AVA for async performance, Jasmine for BDD, and Tape for simplicity. The right choice depends on your project’s needs and team preferences, but any of these can support a robust testing strategy.

To test the add function using Jest, we need to verify that it correctly adds two numbers. Below is a simple Jest test suite that covers basic scenarios, including positive numbers, negative numbers, zero, and floating-point numbers.