Alignment-free QC benchmark: snipe

What the benchmark shows. On the Canis familiaris CanFam3.1 reference, across simulated sequencing workloads, snipe's alignment-free QC estimates reproduce alignment-derived ground truth closely, with best-achievable composite accuracy above 0.96 for the whole-genome workloads and for exome capture, and about 0.90 for RNA, the hardest case. These are the best scores over the k-mer sweep; a single compromise configuration stays within 0.02 of each workload's own best, RNA excepted (below).

Genomic QC is recovered without alignment

For whole-genome (150 bp), indel-enriched and metagenomically-contaminated reads, composite scores sit near 0.98 (WGS 0.98, indels 0.99, metagenomic 0.98). Sequence-change rate is the strongest signal: snipe's estimate matches the event-based alignment truth at r ≥ 0.999 for all three. Where a metric carries genuine dynamic range (sequence-change, and depth in coverage-varying designs), snipe tracks it closely; where the truth is saturated by construction (mapping and breadth for fully on-reference reads) the estimates agree in value even though a correlation there is uninformative. Because those two metrics are near 1 by construction for on-reference reads, they contribute high closeness almost automatically; the sequence-change-weighted score is the more stringent test of snipe's skill.

Exome capture is close behind, as a targeted assay

Whole-exome capture (WXS) is scored on its own because it is a targeted assay: reads cover only the captured regions, so depth and breadth are measured on-target rather than genome-wide. Its best composite accuracy is about 0.97, close to the whole-genome workloads, but it peaks at small k1 (best at k1 = 21) rather than large, because only a small fraction of the genome is on-target and the informative sketch is correspondingly small. Sequence change is recovered well (r ≈ 0.97 at k1 = 21, falling to 0.90 by k1 = 51), while mapping and breadth saturate near 1.0, as expected for reads that are almost entirely on-target.

Mapping rate reflects on-reference content in mixed samples

The metagenomic workload is dog whole-genome reads spiked with a fixed block of off-reference reads; mapping such a sample to the dog reference recovers only its on-reference fraction. Measured against the appropriate read-level ground truth (mapped reads / total), snipe's alignment-free mapping estimate agrees closely (0.46 versus the fixed true fraction of 0.458): the sketch-based rate reflects the same on-reference fraction the aligner does. The base-level Qualimap rate is computed only over reads that already mapped, so it saturates near 1.0 and is blind to the unmapped fraction; using it as ground truth is what previously made this metric look like a failure. This is agreement on a mixed sample once the unmapped noise is isolated, not an active contamination-detection claim.

RNA is the systematic exception

Scored independently against the transcriptome, RNA is the one workload where accuracy is materially lower (0.88–0.90) and where k-mer choice matters. Its sequence-change accuracy rises from r ≈ 0.53 at k1 = 51 to r ≈ 0.87 at k1 = 21, and the composite peaks near k1 = 21–25 before declining monotonically. This is consistent with edge-mer saturation on a short, isoform-redundant reference: a single change removes roughly k2 consecutive k-mers, so long extensions leave too few surviving k-mers and the inferred per-base rate runs to its ceiling. A residual ~1% floor in the alignment truth itself (cross-isoform mismatches under a splice-unaware aligner) further limits agreement at low divergence. Small core k-mers are therefore the appropriate operating point for transcriptomic input.

Why the best core k-mer size depends on the assay

The core k-mer size (k1) that gives the most accurate scores is not the same for every assay. For whole-genome, indel and metagenomic reads, accuracy climbs as k1 grows, so the best size is the largest one tested (composite accuracy, averaged over the edge and scale settings at each k1 rather than the best-achievable maximum, rises from about 0.95 at k1 = 21 to 0.97 at k1 = 51). For exome and RNA reads it does the reverse, falling as k1 grows (exome 0.95 down to 0.92, RNA 0.84 down to 0.64), so the best size is the smallest one tested. The dominant driver is visible directly in the sketches, and it centres on a single quantity: how many informative k-mers, meaning the k-mers a sample shares with the reference, survive in the sketch.

A larger core k-mer makes each estimate cleaner, because long k-mers rarely match the reference by chance, so the error, depth and coverage signals are sharper. But long k-mers are also rarer, so the sketch keeps fewer of them. Whether this cleaner-but-sparser trade pays off depends on how many informative k-mers the reference can supply. The dog genome (2.4 Gb) always supplies plenty: its reference sketch even grows, from about 201,000 to 225,000 distinct k-mers, as k1 rises, and a whole-genome sample still matches roughly 100,000 of them at k1 = 21 and 67,000 at k1 = 51. With that much signal to spare, the cleaner long-k-mer estimate wins, so the genomic assays do best at large k1.

The exome target and the transcriptome are tiny by comparison, so their sketches are starved of informative k-mers, and a larger k1 only makes them sparser. An exome sample matches only about 11,000 k-mers (only around 5% of the genome is on-target), and an RNA sample only about 4,000 at k1 = 21, dropping to 2,700 at k1 = 51 as the transcriptome sketch itself shrinks by roughly 30%. At the limit, an RNA sample at k1 = 51 has to estimate its entire error and mutation rate from only a handful of distinguishing k-mer observations (single digits for the low-divergence libraries), so the estimate turns noisy and its agreement with truth falls from 0.87 to 0.53. With so little signal, the smaller k-mer wins, so exome and RNA peak at k1 = 21.

The evidence points to sketch size, rather than any property of the sequences themselves, as the driver. Shrinking the sketch a different way, by coarsening the FracMinHash scale at a fixed k1 = 51, reproduces the same collapse: RNA's sequence-change agreement falls from 0.84 to 0.60 to 0.15 as the scale discards ten and then a hundred times more k-mers, with k1 never changing. Two plausible alternative explanations do not hold. The simulated error and mutation levels are identical across all five assays, so divergence cannot be responsible; and coverage runs the opposite way, since the exome reaches the highest on-target depths tested (up to 80×) yet still prefers the smallest k1, because extra reads add depth per k-mer but cannot add new informative k-mers.

Accuracy is robust to k, with one caveat

For genomic workloads the composite is nearly invariant to core k-mer size, edge extension and FracMinHash scale: a single compromise configuration (k1 = 31, k2-extension = 6, scale = 1,000) lands within 0.02 of every genomic workload's own optimum, and of the exome's. Only RNA benefits materially from tuning (+0.06 by moving to k1 = 21–25). Searching the full sweep instead of the fixed k1 ∈ {21, 31, 51} grid recovers only marginal accuracy (for example the genomic bundle rises from 0.980 to 0.984 by allowing k1 = 71), indicating the method is not sensitive to precise parameter choice for genomic and exome workloads within a sensible range (RNA excepted, as above).

One caveat on the yardstick itself: the alignment is treated as ground truth, but it is not error-free. Under the most diverged conditions (5% error or mutation) bwa mis-maps and soft-clips a fraction of reads, so the truth carries its own uncertainty, most visibly for RNA, where cross-isoform mapping leaves a ~1% floor, but present for the genomic assays too. A perfect score is agreement with the aligner, not with absolute reality. More broadly, these results are simulation-derived and specific to CanFam3.1; absolute values will shift with reference completeness, read chemistry and divergence. The metrics and configurations emphasized in the manuscript are selected for the reasons stated there; this page previews the entire configuration space so the sensitivity of each conclusion to the k-mer parameters can be inspected below.