The Exim Architecture: Routers, Transports & System Filters

To diagnose why an Exim filter halts mail delivery across an entire server, one must understand how Exim processes inbound and outbound messages:

• ACL Processing: Access Control Lists evaluate incoming SMTP connections, validating HELO hostnames, recipient syntax, and SPF/blacklists during the initial TCP dialogue.
• System Filter Execution: Before a message is passed to routers, Exim executes an optional global System Filter (configured via system_filter = /etc/system_filter.exim in exim.conf). The system filter inspects message headers and bodies globally, enforcing regulatory disclaimers, antispam scoring, or blocking dangerous email attachments.
• Routers and Transports: If the system filter passes, routers determine the delivery path (local Maildir mailbox vs. remote DNS MX exchange) and transports deliver the message.

The critical operational catch: If the system filter contains a single syntax error, Exim panics and freezes every single message entering or leaving the system.

The Root Cause: The Disconnected Filter File Syntax Error

Why did grep find $h_subject: in exim.conf at line 845 instead of line 46? Because the error message referred not to exim.conf, but to the secondary script referenced by the system_filter directive:

# Inside /etc/exim.conf
system_filter = /etc/system_filter.exim
system_filter_user = mail

Opening /etc/system_filter.exim revealed the fatal defect at line 46:

# /etc/system_filter.exim - Broken Syntax
# Exim filter << Mandatory header magic string!

if error_message then finish endif

# Broken line 46: A stray period or malformed expansion caused the failure!
if $h_subject:. contains "viagra" then
    fail text "Spam content rejected"
    seen finish
endif

The developer had accidentally typed a colon followed by a dot ($h_subject:.) instead of a clean header expansion test (if $h_subject: contains "..."). Because Exim's filter language is strictly whitespace and token-sensitive, the lexer could not resolve the token, classifying it as an "unknown filtering command" and generating gigabytes of panic logs on every queued retry.

Exim System Filter Execution Lifecycle

Diagnostic Runbook: Testing Exim Filter Files Safely

Never test Exim filter modifications in production by sending real emails. Exim provides dedicated command-line flags to test configuration files and system filters safely in user space:

1. Validate System Filter Syntax without Restarting

# Syntax check the system filter using a dummy email payload
exim -bF /etc/system_filter.exim < /dev/null

# If syntax is valid, Exim outputs:
# "Filter took 0.000 seconds of CPU time"

2. Simulate Filter Logic with a Test Email (.eml)

# Pass an authentic raw email message to verify rule actions
exim -bf /etc/system_filter.exim < test_spam_sample.eml

# Exim prints the exact execution trace and final delivery verdict!

3. Flusing and Unfreezing the Mail Queue

Once the syntax error is corrected, thousands of queued messages must be unthawed and flushed:

# View current mail queue count and message IDs
exim -bpr

# Unfreeze all frozen messages in the queue
exiqgrep -z -i | xargs exim -Mt

# Force a global queue run across all pending messages
exim -qf -v

DirectAdmin CustomBuild Architecture: Templating vs. Ad-Hoc Edits

A critical operational trap when administering DirectAdmin servers is modifying service configurations directly. Hosting control panels employ template generation engines that overwrite configuration files during automated maintenance routines:

• The CustomBuild Overwrite Trap: Direct edits made to /etc/exim.conf will be completely wiped out whenever DirectAdmin executes an automated update or when an administrator runs ./build exim_conf.
• Implementing Custom Templates Cleanly: To persist customizations permanently, DirectAdmin provides a dedicated template override directory. Administrators must copy the base template to /usr/local/directadmin/custombuild/custom/exim/exim.conf and apply configuration directives there.
• Externalizing System Filters: Rather than embedding complex filtering logic directly inside the main Exim configuration template, keep the system filter in a standalone file (e.g. /etc/system_filter.exim) tracked under Git version control. This ensures that MTA upgrades modify only routing logic while leaving anti-abuse filters intact.
• Handling DNSBL Timeouts: In addition to syntax bugs, Exim queues often choke on unresponsive DNS Blackhole Lists (such as outdated Spamhaus or Sorbs mirrors). Always configure aggressive lookup timeouts (dns_again_means_non_exist = true) in the ACL configuration to prevent thread starvation.

Systems Engineering Lessons

The DirectAdmin Exim suffering of 2012 highlights timeless truths of systems administration:

1. Decouple Daemon Logs from Root Filesystems: Storing log files on unpartitioned /var partitions creates severe risks. A runaway panic log generating 1GB/hour can exhaust inode space and crash database servers.
2. Validate Before Restarting: Always execute syntax verification commands (exim -bV, nginx -t, apachectl configtest) before reloading system services.
3. Understand the Control Panel Layer: Hosting panels like DirectAdmin and cPanel are wrappers around standard Linux daemons. True troubleshooting requires stepping outside the GUI and analyzing the underlying daemon configuration files directly.