We deployed an advanced, agent-driven AI framework to autonomously crawl, evaluate, and rewrite the SEO infrastructure of every single page. But as we learned the hard way, giving 25 autonomous AI agents free rein over a centralized Node.js database creates an entirely new class of engineering nightmares.

Here is how we scaled our orchestration pipeline to maximum throughput and the architectural lessons we learned along the way.

The Bottleneck: Static Batches vs. The Sliding Window

Our initial architecture was simple: we chunked our 116 domains into batches of 5. The orchestrator script used a standard Promise.all() to wait for the batch to finish before booting up the next 5 domains.

The problem? Web portfolios are rarely uniform. While four domains in a batch might only have 2 pages each and finish in ten seconds, the fifth domain (like our massive 411-page prima-jewelry.com property) would take an hour. Because of the static batching, 80% of our execution lanes sat completely idle, waiting for the heavy hitter to finish. Our throughput flatlined at roughly 1.5 pages per minute.

The Solution: We tore out Promise.all() and replaced it with a highly-efficient Sliding-Window Worker Pool using Set and Promise.race(). The millisecond a small domain finished processing, the orchestrator immediately grabbed the next domain from the queue and dropped it into the open execution lane.

By keeping the concurrency strictly saturated at exactly 25 parallel AI streams at all times, we instantly multiplied our throughput to over 4.5 pages per minute.

The Chaos of Concurrent I/O

When you unleash 25 parallel processes that are aggressively reading, computing, and writing to the same centralized file system, you immediately run into severe race conditions.

1. Global State Overwrites

To ensure the script could gracefully resume if it crashed, we tracked the progress of every page in a global progress.json state tracker. Initially, each domain loaded the global state into memory, updated its pages, and wrote it back.

But with 5 domains operating on different timelines, one domain's write would frequently overwrite the progress of another, blindly reverting completed domains back to a "pending" status. We solved this by abandoning in-memory global state altogether, forcing the orchestrator to execute blocking, synchronous reads (fs.readFileSync) directly from disk right before every write, guaranteeing atomic updates.

2. The Git Lock Collision

We initially instructed the pipeline to run git commit after every successful page optimization. At 25x concurrency, this immediately triggered fatal index.lock collisions because Git cannot execute concurrent commits on the same repository. We had to lift the version control step entirely out of the page-level logic, wrapping the repository in a central lock file (data/git.lock) so that only one domain could commit its batch to the remote branch at any given time.

3. Subagent Autonomy and Database Corruption

This was our most critical failure. The AI agents we deployed were autonomous—meaning they followed their own internal skill instructions on how to solve problems. One of those instructions involved running a shell script (sync_database_to_git.sh) that executed a mongoexport command to dump the MongoDB layout data into our local posts.json file.

Because 25 agents were independently deciding to export the database at the exact same millisecond, the write collisions completely corrupted the JSON syntax of posts.json. This fatal read error crashed the entire pipeline.

The Fix: We had to implement strict subagent boundaries. We modified the orchestrator's central prompt to explicitly strip the agents of their authority:

CRITICAL RULE: DO NOT run database export scripts or git commands! The orchestrator handles all synchronization.

The Result

Once we locked down the file system and constrained the agents, the pipeline became structurally bulletproof. We successfully optimized thousands of pages across 116 domains in a matter of hours, all while running a highly complex matrix of autonomous AI agents.

If you are building multi-agent AI systems, remember: the AI is the easy part. Managing the parallel I/O and orchestration is where the real engineering begins.