Sliding-Window Worker Pools for Concurrent Node.js Agent Pipelines

By the Engineering Team at WebDesigner.LA

Over the last two days, we embarked on a massive technical SEO modernization project across our 116-domain web portfolio. With thousands of pages needing comprehensive audits, metadata generation, and semantic restructuring, manual execution was out of the question.

We deployed an agent-driven AI framework to autonomously crawl, evaluate, and enrich the SEO infrastructure of every single page. As we learned firsthand, giving 25 autonomous AI agents concurrent access to a shared Node.js repository and database creates an entirely new class of distributed systems challenges. Here is how we engineered our orchestration pipeline for maximum throughput and zero data corruption.

The Bottleneck: Static Batches vs. Sliding-Window Concurrency

Our initial architecture was straightforward: we partitioned 116 domains into static batches of 5. The orchestrator used a standard Promise.all() barrier to wait for all 5 domains in a batch to finish before starting the next batch.

The flaw? Multi-tenant web portfolios are heavily skewed in size. While four domains in a batch might contain only 2 pages each and finish in ten seconds, the fifth domain (such as a 411-page e-commerce property) could take nearly an hour. Because of the static batch barrier, 80% of our worker lanes sat idle waiting for the largest domain in the cohort to complete, capping throughput at roughly 1.5 pages per minute.

The Solution: We replaced static Promise.all() chunking with a Sliding-Window Worker Pool built on Set and Promise.race(). The moment any worker finishes a small domain, the orchestrator immediately pulls the next queued domain into the freed execution lane:

export async function runSlidingWindowPool<T>(
  items: T[],
  concurrency: number,
  workerFn: (item: T) => Promise<void>
): Promise<void> {
  const activeWorkers = new Set<Promise<void>>();

  for (const item of items) {
    const task = Promise.resolve()
      .then(() => workerFn(item))
      .finally(() => activeWorkers.delete(task));

    activeWorkers.add(task);

    if (activeWorkers.size >= concurrency) {
      await Promise.race(activeWorkers);
    }
  }

  await Promise.all(activeWorkers);
}

By keeping all 25 parallel execution lanes continuously saturated, throughput jumped from 1.5 pages per minute to over 4.5 pages per minute.

Orchestration Pattern Lane Utilization Sustained Throughput Head-of-Line Blocking
Static Batching (Promise.all) ~20–35% (stalled on outliers) 1.5 pages / min Severe when 400+ page domain enters batch
Sliding-Window Pool (Set + Promise.race) 100% until queue drain 4.5+ pages / min Eliminated; fast domains cycle continuously

Resolving Concurrent I/O and State Contention

When 25 concurrent worker streams read, compute, and write against a shared filesystem and database, three failure modes emerge immediately:

1. Global State Overwrites

To support crash recovery, we tracked per-page progress in a central progress.json ledger. Initially, each worker loaded the state object into memory at startup, mutated its domain slice, and flushed the object back to disk. With 25 workers operating on independent timelines, stale in-memory snapshots overwrote newly completed domain statuses. We eliminated long-lived in-memory state and enforced atomic read-modify-write cycles serialized through a mutex lock right before flushing to disk.

2. Git Index Lock Collisions

Triggering git commit after every individual page optimization at 25x concurrency immediately caused fatal .git/index.lock collisions because Git index updates are single-writer operations. We decoupled version control from page-level workers and serialized repository commits through a central lock file (data/git.lock) so only one completed batch commits at a time.

3. Subagent Boundary Enforcement and Sharded Storage

Our most critical lesson involved autonomous subagent tool access. Early subagent prompts permitted workers to invoke database export scripts (mongoexport) directly against monolithic JSON files. When multiple agents exported simultaneously, interleaved writes corrupted JSON files. We solved this permanently by:

  • Strict Subagent Capabilities: Stripping subagents of raw export and git shell commands so all persistence flows exclusively through the validated CMS API client (scripts/cms_api_client.js).
  • Domain-Sharded Storage: Partitioning post storage by domain into deterministic 100-post shards (data/posts/<domain>/posts_0.json) alongside SQLite revision ledgers.

Architectural Takeaways

Once we isolated subagent permissions, replaced static batch barriers with a sliding-window worker pool, and enforced serialized disk/git locking, the multi-agent pipeline ran with zero lock collisions and 3x higher throughput. In production agentic systems, model reasoning is only half the equation—deterministic concurrency control and I/O isolation are where reliability is won.