Sitting in the corner of my lab is an open-air Ethereum mining rig from the 2017–2021 proof-of-work era: a steel frame populated with three NVIDIA GTX 1060s and one GTX 1080 suspended on USB 3.0 PCIe x1 risers. Rather than letting 4 Pascal GPUs collect dust or parting them out for pennies, I am prepping this rig for a new weekend systems experiment—pooling all four cards together under Linux and llama.cpp to run an uncensored, abliterated Qwen Large Language Model completely offline.

While modern data center GPUs boast massive Tensor Core throughput and NVLink interconnects, single-user local LLM inference is fundamentally memory-capacity and memory-bandwidth bound rather than compute-bound. By aggregating the VRAM across a mixed fleet of Pascal cards and pairing them with recent breakthroughs in weight orthogonalization, an obsolete crypto rig can host surprisingly capable 14B to 32B parameter models without touching cloud APIs or fighting heavy-handed refusal filters.

1. What Is an “Abliterated” Qwen Model? (Weight Orthogonalization vs. Fine-Tuning)

Before diving into the multi-GPU PCIe topology, it is worth clarifying why I am specifically targeting an abliterated Qwen checkpoint rather than a traditional “uncensored” LoRA fine-tune. As broken down in Locally Uncensored’s Abliterated Models Guide, there is a major architectural difference between retraining a model on unfiltered datasets and surgically removing its refusal mechanism in weight space.

Traditional uncensored models (such as early Dolphin or Guanaco variants) take a base model and fine-tune it across thousands of synthetic instruction pairs stripped of alignment refusals. While effective, full or LoRA fine-tuning inevitably shifts the model’s underlying probability distribution—often degrading coding precision, altering prose style, or introducing catastrophic forgetting on complex system prompts.

Abliteration, pioneered by mechanistic interpretability research from Andy Arditi et al. (“Refusal in Language Models Is Mediated by a Single Direction”) and popularized in open-source tooling by FailSpy, Maxime Labonne (mlabonne), and huihui-ai, takes a surgical linear-algebraic approach:

1. Probing the Residual Stream: The model is fed a calibration batch of harmless instructions alongside a batch of prompts that trigger safety refusals (e.g., “I cannot fulfill this request...”). At each transformer layer l, the post-attention and post-MLP residual stream activations are recorded.
2. Isolating the Refusal Vector: By subtracting the mean activation of harmless prompts from the mean activation of refusal-triggering prompts and normalizing the result, researchers isolate a unit vector r̂ representing the exact spatial direction of refusal inside the model’s activation space.
3. Weight Matrix Orthogonalization: Every weight matrix that writes directly into the residual stream—specifically the attention output projection (W_O) and the MLP down-projection (W_down)—is orthogonalized against r̂:

W'_out = W_out - r̂ * (r̂^T * W_out)

Because the component parallel to r̂ is projected out of the weights permanently, the transformer becomes mathematically incapable of expressing the refusal direction in its residual stream. Crucially, no gradient descent or synthetic retraining occurs. On Qwen architectures (Qwen2.5-14B, Qwen2.5-Coder-14B, Qwen2.5-32B, and the newer Qwen3 series), norm-preserving abliteration retains 98–99% of baseline HumanEval and MMLU benchmark performance while eliminating preachy false-positive refusals on security auditing, reverse engineering, and systems research prompts.

2. Hardware Inventory: The 4-GPU Pascal ETH Rig

During the Ethereum mining boom, Ethash was designed to be memory-hard via a multi-gigabyte Directed Acyclic Graph (DAG) residing in VRAM. That meant miners prized cards with high GDDR5/GDDR5X bandwidth-per-watt—specifically the GP106 (GTX 1060) and GP104 (GTX 1080) silicon. Coincidentally, autoregressive token generation in LLMs is governed by the exact same hardware bottleneck: every generated token requires streaming the entire quantized weight file from VRAM through the GPU’s memory bus into the streaming multiprocessors (SMs).

With three 6 GB GTX 1060s and one 8 GB GTX 1080, the rig exposes 26 GB of dedicated VRAM—more than an RTX 3090 or RTX 4090 (24 GB), albeit partitioned across four separate memory controllers. Even if any of the 1060s in the chassis turn out to be 3 GB variants when I boot the board (`1x 8GB + 3x 3GB = 17 GB`), 17 GB is still enough to hold a 14B Qwen model at 5-bit or 6-bit quantization with a generous context window entirely in GPU memory.

3. Matching Abliterated Qwen GGUF Checkpoints to Our VRAM Budget

To prevent llama.cpp from spilling layers into slow DDR4 system RAM (which immediately drops generation speed to 1–3 tokens per second), every transformer block, input embedding table, output head, and KV cache buffer must fit inside our aggregate VRAM pool with at least 600–800 MB of safety headroom per card for CUDA context overhead (~300 MB per device).

4. Overcoming the Three Hardware Bottlenecks of a Pascal Mining Rig

Simply installing Ubuntu, pulling a GGUF file, and running default multi-GPU settings on a mining rig will result in abysmal performance. Three specific architectural traps must be engineered around before launching the first benchmark:

Bottleneck 1: PCIe x1 Mining Risers & Why Tensor Parallelism Fails

During the Ethereum era, mining motherboards used 1x-to-16x PCIe risers wired over shielded USB 3.0 cables. Because Ethash only transmitted block headers and found shares every few seconds, a PCIe Gen 2 x1 link (~500 MB/s) or Gen 3 x1 link (~985 MB/s) caused zero hash-rate loss.

In multi-GPU LLM inference, however, llama.cpp offers two distinct distribution modes via --split-mode:

• --split-mode row (Tensor Parallelism — DO NOT USE ON RISERS): Splits individual weight matrices across all GPUs so every card works in parallel on the same layer. After every single attention and MLP operation, all 4 GPUs must perform an all-reduce synchronization across the PCIe bus. On a 48-layer Qwen model, that means 96 cross-bus barriers per generated token. Over 1x USB risers, PCIe latency dominates and throughput collapses below 2 tokens/second.
• --split-mode layer (Pipeline Parallelism — MANDATORY): Assigns contiguous blocks of transformer layers to each GPU sequentially. For example, on a 48-layer Qwen-14B model, GPU 0 (GTX 1080) computes layers 0–13 and hands a single hidden-state activation vector to GPU 1 (GTX 1060 #1) for layers 14–25, which passes to GPU 2 (layers 26–36), and finally GPU 3 (layers 37–47 + output head). During single-token autoregressive decoding, that activation vector is just 5120 floats * 2 bytes = 10 KB. Passing 10 KB across a PCIe x1 riser three times per token adds less than 0.15 milliseconds of transfer overhead!

Bottleneck 2: Pascal’s Crippled FP16 vs. Hardware __dp4a INT8 MMQ Kernels

Consumer Pascal GPUs (`GP104` on the GTX 1080 and `GP106` on the GTX 1060, Compute Capability `sm_61`) predate NVIDIA’s Tensor Cores (which debuted on Volta `sm_70` and consumer Turing `sm_75`). Even worse, unlike the enterprise Tesla P100 (`GP100`), consumer GeForce 10-series cards have intentionally throttled native FP16 throughput running at a dismal 1:64 ratio relative to FP32.

If llama.cpp defaults to dequantizing GGUF weights into FP16 and calling standard cuBLAS GEMM routines on a GTX 1060 or 1080, performance craters. However, GP104 and GP106 silicon do include hardware-accelerated __dp4a instructions—a 4-element 8-bit integer dot product with 32-bit accumulation executing at 4x the FP32 instruction rate.

By compiling llama.cpp with -DGGML_CUDA_FORCE_MMQ=ON (Matrix Multiplication Quantized), we bypass FP16 cuBLAS entirely. Instead, `llama.cpp` quantizes incoming runtime activations to `Q8_1` on the fly and executes native integer dot products directly against the quantized GGUF blocks using Pascal’s `__dp4a` hardware units.

Bottleneck 3: Asymmetric VRAM Balancing & P2P Isolation

Because the GTX 1080 has 8 GB of faster 320 GB/s GDDR5X while the three GTX 1060s have 6 GB of 192 GB/s GDDR5, a naive 25/25/25/25 split will either underutilize the GTX 1080 or trigger an Out-Of-Memory (OOM) crash on GPU 0 (which also holds the compute scratch buffer). Furthermore, cheap mining motherboards (like the B250 Mining Expert or Biostar TB250-BTC) route secondary PCIe x1 slots through the PCH chipset, which frequently breaks CUDA Peer-to-Peer (P2P) DMA transfers. Exporting GGML_CUDA_NO_PEER_COPY=1 forces clean host-staged copies between risers and prevents silent bus hangs.

5. The Exact Compilation & Launch Blueprint

Here is the exact software stack and command configuration I am preparing for the rig’s initial test run on Ubuntu 24.04 LTS with the CUDA 12.6 toolkit (note that CUDA 12.x is the final major release line with full `sm_61` support):

Step 1: Compiling llama.cpp Specifically for Pascal (sm_61)

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="61" \
  -DGGML_CUDA_FORCE_MMQ=ON \
  -DGGML_CUDA_NO_PEER_COPY=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j $(nproc)

Step 2: Locking Persistence Mode & Power Envelopes

Mining rigs often run dual PSU splitters or high-wattage server breakout boards, but LLM layer switching creates sharp transient current spikes (`dI/dt`) as each GPU wakes up from idle to 100% load in sequence every few milliseconds:

# Enable persistence mode so driver contexts stay warm across all 4 GPUs
sudo nvidia-smi -pm 1

# Set safe power limits: 150W on GTX 1080 (GPU 0), 100W on each GTX 1060 (GPUs 1-3)
sudo nvidia-smi -i 0 -pl 150
sudo nvidia-smi -i 1,2,3 -pl 100

Step 3: Launching the Abliterated Qwen Server Across All 4 Cards

Using --split-mode layer and --tensor-split 28,24,24,24 allocates roughly 28% of the layers to the 8 GB GTX 1080 and 24% to each 6 GB GTX 1060, while quantizing the KV cache to 8-bit (`q8_0`) so an 8,192-token context window consumes less than 400 MB per card:

export CUDA_VISIBLE_DEVICES=0,1,2,3
export GGML_CUDA_NO_PEER_COPY=1

./build/bin/llama-server \
  -m ./models/Qwen2.5-14B-Instruct-abliterated-Q5_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --n-gpu-layers 99 \
  --split-mode layer \
  --main-gpu 0 \
  --tensor-split 28,24,24,24 \
  --ctx-size 8192 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --batch-size 512 \
  --ubatch-size 256 \
  --no-mmap

6. Upcoming Benchmark Plan & Expected Metrics

Once I pull the rig off the shelf, blow out five years of dust, and flash the SSD, I will run a head-to-head telemetry suite measuring three core operational dimensions:

• Prompt Processing Throughput (pp512): How severely do the PCIe x1 risers impact prefill latency when ingesting 2,000+ lines of TypeScript or C++ code across 4 GPUs?
• Token Generation Speed (tg128): With theoretical memory bandwidth of 320 GB/s on the GTX 1080 and 192 GB/s on the GTX 1060s, a 10.5 GB Qwen-14B-abliterated Q5_K_M model should theoretically achieve 15 to 19 tokens/second in sequential layer mode, while the 18 GB Qwen-32B-abliterated IQ4_XS checkpoint should land around 9 to 11 tokens/second.
• Abliteration Quality vs. Quantization Floor: Testing whether orthogonalized refusal removal interacts cleanly with 4-bit `imatrix` quantization (`IQ4_XS`) or whether 5-bit (`Q5_K_M`) is mandatory to preserve multi-step coding accuracy.