1. What is smbtorture and Why is it Essential?

Unlike synthetic network benchmarks like iperf that only measure raw TCP pipe saturation, smbtorture simulates real-world enterprise file server semantics:

• Protocol State Machine Verification: It rigorously exercises SMB1, SMB2, and SMB3 dialect state machines, testing complex command compound sequences, directory change notifications, and file leasing.
• Byte-Range Lock Concurrency: In multi-user office environments (e.g., Microsoft Access databases or collaborative CAD software), multiple workstations request byte-range read/write locks simultaneously. smbtorture initiates thousands of competing asynchronous lock requests to ensure Samba never permits dirty reads or conflicting write leases.
• Opportunistic Locking (Oplocks) & File Leases: Clients cache file data locally to boost perceived speed. When a second client attempts to write to that same file, the server must break the first client's oplock cleanly. smbtorture deliberately triggers oplock break races to detect deadlocks.

2. The TDB Locking Subsystem & Storage Architecture

To understand why Samba servers lock up or corrupt files under multi-user strain, one must understand how Samba arbitrates file state:

• Trivial Database (TDB) Storage: Samba does not rely directly on the Linux kernel for byte-range lock tracking. Instead, it maintains high-speed in-memory/disk databases formatted as TDB files—chiefly locking.tdb (open file handles and share modes) and brlock.tdb (active byte-range locks).
• The Networked Storage Anti-Pattern: Placing Samba's private TDB lock files on remote NFS mounts or shared cluster filesystems is a fatal architecture mistake. TDB relies on fast POSIX fcntl() locking with sub-microsecond latency. Remote locking latency over NFS triggers cascading timeouts, lock corruption, and complete daemon freezing. TDB directories must always reside on high-speed local NVMe or dedicated tmpfs mounts.
• Inspecting Live Locks: Administrators can inspect active locks and oplocks in real time using the smbstatus utility:

  # Inspect all active file locks and oplock states
  sudo smbstatus --locks
  
  # Inspect connected client sessions and negotiated SMB dialects
  sudo smbstatus --shares

3. Key smbtorture Test Profiles & Execution

In modern Ubuntu distributions, smbtorture is available via the smbclient development packages or can be compiled directly from the Samba source tree:

# Install smbtorture on Ubuntu
sudo apt update
sudo apt install -y smbclient samba-testsuite

# Basic syntax to torture an SMB share:
# smbtorture //server/share -U "username%password" TESTNAME

# 1. Test raw byte-range locking semantics:
smbtorture //127.0.0.1/testshare -U "smbuser%SecretPass123" RAW-LOCK

# 2. Test opportunistic locking (oplocks) and lease transitions:
smbtorture //127.0.0.1/testshare -U "smbuser%SecretPass123" RAW-OPLOCK

# 3. Simulate high-concurrency client load (10 parallel simulated workstations):
smbtorture //127.0.0.1/testshare -U "smbuser%SecretPass123" -t 60 --num-procs=10 BENCH-NBENCH

Dissecting the Test Profiles

1. RAW-LOCK: Exercises concurrent byte-range lock acquisition, testing overlapping ranges, upgrade/downgrade requests from read to write, and non-blocking vs. blocking lock queues. A single failure indicates potential data corruption in multi-user databases.
2. RAW-OPLOCK: Simulates client-side caching transitions. When Client A holds an exclusive write oplock on a 50 GB video file and Client B requests read access, the server must signal an oplock break, wait for Client A to flush its dirty cache buffers, and transition Client A to a shared read oplock before replying to Client B. Any timing glitch causes file truncation.
3. BENCH-NBENCH: Replays standard Ziff-Davis NetBench network profiles, simulating dozens of simulated office workers opening, seeking, editing, and saving files across multi-gigabit Ethernet links.

4. High-Performance Samba Tuning Parameters & Filesystem Options

When smbtorture reveals bottlenecks or locking timeouts, updating /etc/samba/smb.conf with the following production-hardened directives optimizes throughput and concurrency:

[global]
   workgroup = WORKGROUP
   server role = standalone server
   server min protocol = SMB2_10
   server max protocol = SMB3_11

   # Socket & TCP Stack Optimization
   socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072

   # Asynchronous I/O via Kernel Threads
   aio read size = 16384
   aio write size = 16384
   vfs objects = aio_pthread

   # Concurrency & Oplocks
   oplocks = yes
   level2 oplocks = yes
   kernel oplocks = yes
   strict locking = auto

[data-share]
   path = /srv/storage/fastshare
   read only = no
   browsable = yes
   create mask = 0664
   directory mask = 0775

Filesystem Selection: ZFS, XFS, and System Extended Attributes

The underlying Linux storage filesystem dictates Samba performance under high concurrency:

• ZFS with System Attributes: On ZFS pools, always configure xattr=sa (system attribute based extended attributes). Standard Linux extended attributes on ZFS create hidden directory structures for every file, creating severe IOPS penalties when Windows clients query NTFS file permissions.
• XFS for High Sequential Throughput: XFS excels at large continuous file streaming and allocation groups, making it ideal for media editing NAS clusters. Ensure the filesystem is mounted with noatime to prevent synchronous inode writes on every file read.

5. Conclusion: Continuous Storage Verification

Never assume an SMB server is stable simply because it serves files to a single developer laptop. Incorporating smbtorture into automated staging pipelines guarantees that Linux storage backends maintain rock-solid data integrity under severe multi-client concurrency before deploying into mission-critical production environments.