In high-scale multi-tenant environments, managing dozens of disparate background tasks—spanning automated QA audits, commercial lead discovery, outbound email routing, and content synchronization—rapidly becomes unsustainable when executed as uncoordinated cron jobs or ad-hoc background scripts. Without a centralized orchestration framework, long-running processes risk orphaning themselves, saturating server CPU cores, and lacking real-time observability.

To solve this architectural challenge, we engineered Job Runner v2.0: a resilient, high-throughput TypeScript platform built on a modular plugin architecture. By designing pluggable subsystem interfaces, Job Runner successfully consolidated three previously independent legacy automation engines—QA Watchdog, Lead Hunter, and Article Sync—into a single, observable, and process-isolated platform.

1. The Problem with Fragmented Background Scripts

Prior to consolidating under a unified runner, each operational tool operated in its own silo:

  • QA Watchdog: Dispatched standalone headless Playwright spiders to audit 404 links, discover dynamic contact forms, and inspect JSON-LD schemas.
  • Lead Hunter: Executed multi-step discovery loops across local business registries, dispatched outreach via the Google Gmail API, and synchronized records to Frappe CRM.
  • Article Sync & SEO Indexing: Ran scheduled database export/import routines to synchronize flat-file JSON state with MongoDB and push URLs to IndexNow search engine endpoints.

When executed via detached SSH sessions or standard system crons, unhandled promise rejections or infinite loop parser bugs could pin CPU cores at 100% indefinitely. Furthermore, operational teams lacked a single pane of glass to monitor queue depths, inspect real-time log streams, or terminate runaway processes safely.

2. The Modular Plugin Architecture

Job Runner v2.0 solves this by abstracting every operational domain into a strictly-typed module under src/modules/. Each subsystem registers its jobs through a unified declarative definition contract: 

export interface JobDef {
  key: string;
  name: string;
  description: string;
  queue: 'qa' | 'lead-hunter' | 'seo' | 'maintenance' | 'social';
  bin: 'node' | 'bash';
  script: string;
  cwd: string;
  baseArgs: string[];
  schedule?: string;
  retries: number;
  logFile: string;
}

A. Consuming QA-Watchdog

All automated testing suites were migrated into src/modules/qa/ as native TypeScript spiders:

  • 404 Fast Spider (qa:spider): Concurrently traverses all tenant domains over HTTPS, validating response codes and following 301 redirects to their destination DOMs.
  • Contact Form Liveness (qa:contact-forms): Scans dynamic DOM trees to locate, validate, and verify public inquiry forms.
  • Static Asset Auditor (qa:asset-audit): Deep-crawls CSS, JS bundles, and media assets to guarantee visual integrity.
  • SEO Schema Validator (qa:seo-audit): Inspects 3,400+ published articles for structured JSON-LD compliance.

B. Consuming Lead-Hunter

The commercial outreach and discovery engine was refactored into src/modules/marketing/, establishing an automated 5-stage cascade:

  1. Discovery: Autonomous SMB discovery across regional market slots (West Coast, East Coast, Southeast).
  2. Flaw Auditing: Identifies technical site regressions (missing SSL, slow LCP, broken viewport meta).
  3. CRM Synchronization: Deduplicates and creates structured leads inside Frappe CRM v15 via REST API.
  4. OAuth Email Routing: Dispatches personalized audits exclusively through the Google Gmail API microservice on port 8085.
  5. Bounce Tracking: Ingests delivery notifications and tags bounced records automatically.

C. Consuming Article Sync & Versioned Content Engine

Content synchronization was redesigned into src/modules/content/ with full versioning:

  • Database Dual-Writing: Synchronizes MongoDB collections and repository flat-file JSON records atomically.
  • Instant Search Indexing (IndexNow): Notifies Bing, Yandex, and IndexNow API gateways immediately when new articles or revisions are published.
  • Immutable Revisions: Every update captures a complete state snapshot (v1 → v2), enabling instant non-destructive rollbacks via the CMS API client.

3. Hard Process Isolation & Telemetry Integration

To prevent CPU saturation and orphaned tasks, Job Runner enforces two critical stability invariants:

  • Process Group Sandboxing (-pid): When a job is stopped or times out, Job Runner issues SIGTERM / SIGKILL to the negative process group ID, guaranteeing all spawned child processes (including headless Chromium instances) are cleanly terminated.
  • Dual-Push Telemetry: All job logs and lifecycle metrics are streamed to Prometheus and Promtail, populating real-time Grafana dashboards and push-button action panels across both primary (.32) and shard (.18) cluster nodes.

4. Conclusion

By uniting QA validation, marketing automation, and content synchronization into a modular plugin architecture, we transformed a collection of fragile background scripts into a hardened, enterprise-grade automation engine. The result is total operational visibility, zero-downtime reliability, and a clean foundation for future autonomous microservices.