Skip to content

CupOfTea Performance & Speed Benchmarking Suite

This document describes the design, execution procedures, hardware specifications, and baseline results of the CupOfTea JMH Benchmarking Suite (cupoftea-benchmark).

1. Overview & Architecture

Following the contiguous columnar storage migration and multithreaded vector execution engine, CupOfTea features a dedicated performance harness built on OpenJDK JMH (Java Microbenchmark Harness).

Module Isolation

The benchmarking suite is isolated in the cupoftea-benchmark Maven submodule: - Separation of Concerns: Benchmarking dependencies (JMH core, annotation processors, comparative libraries like Tablesaw) are decoupled from runtime release artifacts (cupoftea-all, cupoftea-core, etc.). - Standalone Fat JAR: Packaged with the maven-shade-plugin into cupoftea-benchmark/target/benchmarks.jar, merging all SPI service providers (com.cupoftea.spi.FormatReader and FormatWriter) and configuring BenchmarkRunner as the entrypoint. - Continuous Verification: Includes unit smoke tests (BenchmarkSmokeTest) ensuring benchmark logic remains robust and regression-free during standard CI builds without consuming excessive CPU time.

2. Test Environment & Hardware Specifications

Baseline results were established under the following environment:

Specification Configuration
System Model Apple Silicon (Mac14,2) / 8-core CPU
Physical Cores 8 Cores (4 Performance + 4 Efficiency)
System Memory 24 GB Unified Memory
Operating System macOS 15.x / Darwin 24
Java Runtime OpenJDK 64-Bit Server VM (Java 17, 21, 25 tested)
JMH Version 1.37
Recommended JVM Flags --add-opens=java.base/java.nio=ALL-UNNAMED -Xms4g -Xmx8g -XX:+UseG1GC

3. How to Run the Benchmarks

3.1 Build the Executable Benchmark JAR

Compile the project and produce the shaded benchmark JAR: 

mvn clean package -pl cupoftea-benchmark -am -DskipTests
The resulting executable is generated at: 
cupoftea-benchmark/target/benchmarks.jar

3.2 Running Preset Benchmark Suites

The custom launcher com.cupoftea.benchmark.BenchmarkRunner supports convenient preset targets:

# Run all micro-benchmarks (vector scan, filter selectivity, map-reduce, zero-copy slicing)
java -jar cupoftea-benchmark/target/benchmarks.jar micro

# Run macro end-to-end format IO benchmarks (CSV, JSON, Parquet, Excel, XML)
java -jar cupoftea-benchmark/target/benchmarks.jar io

# Run end-to-end analytical pipeline benchmarks
java -jar cupoftea-benchmark/target/benchmarks.jar pipeline

# Run multi-core CPU and dataset scaling benchmarks
java -jar cupoftea-benchmark/target/benchmarks.jar scaling

# Run competitive baseline benchmarks against Tablesaw
java -jar cupoftea-benchmark/target/benchmarks.jar competitive

# Run all registered benchmarks
java -jar cupoftea-benchmark/target/benchmarks.jar all

3.3 Running Custom JMH Executions

Any standard JMH command-line argument can be passed:

# List all registered benchmarks
java -jar cupoftea-benchmark/target/benchmarks.jar -l

# Run a specific benchmark matching a regex with custom warmup, iterations, and dataset size
java --add-opens=java.base/java.nio=ALL-UNNAMED -jar cupoftea-benchmark/target/benchmarks.jar ".*CompetitiveBaselineBenchmark.*" -f 1 -wi 2 -i 3 -p rows=500000

# Benchmark multi-core thread scaling with custom thread counts
java -jar cupoftea-benchmark/target/benchmarks.jar ".*MultiCoreScalingBenchmark.*" -f 1 -p threads=1,2,4,8,16

# Export machine-readable benchmark results to JSON
java -jar cupoftea-benchmark/target/benchmarks.jar micro -rf json -rff results.json

3.4 Verification & Fast Smoke Test

To verify all benchmarks run without errors without launching long JMH iterations: 

mvn test -pl cupoftea-benchmark -am

4. Benchmark Suites & Baseline Results

4.1 Zero-Copy Slicing & Partitioning Overhead

Measures the latency of creating zero-copy sub-views (vec.sliceView(from, to)) and thread partitions (vec.partition(chunks)) compared to array-copying slices (vec.slice(from, to)) on 1,000,000 rows.

Benchmark Operation Mode Dataset Size Latency Relative Speedup
sliceCopyDouble Average Time 1,000,000 rows 151,278 ns/op Baseline (1.0x)
sliceViewDouble (zero-copy view) Average Time 1,000,000 rows 2.637 ns/op > 57,000x faster
sliceCopyInt Average Time 1,000,000 rows 115,330 ns/op Baseline (1.0x)
sliceViewInt (zero-copy view) Average Time 1,000,000 rows 2.691 ns/op > 42,000x faster
partitionDoubleChunks (8 chunks) Average Time 1,000,000 rows 217.5 ns/op Near zero-allocation
partitionIntChunks (8 chunks) Average Time 1,000,000 rows 213.1 ns/op Near zero-allocation

Key Finding: ColumnVectorSlice and partition(n) operate with virtually zero allocation overhead (~2.6 ns per slice view), enabling multithreaded partitioning without memory fragmentation.

4.2 Parallel Scan & Filter across Selectivity Rates

Measures prefix-sum compaction performance between single-threaded df.asSequential() and multithreaded df.asParallel() on 1,000,000 rows across varying selectivity rates:

Selectivity Matching Rows Single-Thread (asSequential) Parallel (asParallel) Speedup
1% 10,000 8.93 ms/op 4.38 ms/op 2.04x
10% 100,000 10.82 ms/op 4.94 ms/op 2.19x
50% 500,000 20.31 ms/op 5.91 ms/op 3.44x
90% 900,000 31.10 ms/op 6.71 ms/op 4.64x

Key Finding: Parallel vector scanning and prefix-sum consolidation excels at high selectivity, delivering up to 4.64x faster execution when filtering large portions of contiguous columnar vectors.

4.3 Primitive Map-Reduce Aggregations

Evaluates latency for core aggregation operations on 1,000,000 rows:

Operation Method Single-Thread (asSequential) Parallel (asParallel) Scaling Speedup
Count countOf("val") 0.007 µs/op 0.007 µs/op O(1) metadata
Sum sumOf("val") 906.8 µs/op 280.3 µs/op 3.24x
Mean meanOf("val") 901.9 µs/op 276.9 µs/op 3.26x
Min minOf("val") 308.4 µs/op 118.6 µs/op 2.60x
Max maxOf("val") 316.4 µs/op 118.0 µs/op 2.68x
Std Dev std("val") 1.83 ms/op 1.83 ms/op Two-pass variance
Median median("val") 40.46 ms/op 40.45 ms/op Introselect (linear)

4.4 Competitive Baseline: CupOfTea vs. Tablesaw

Head-to-head comparison on 100,000 rows:

Analytical Operation Tablesaw 0.43.1 CupOfTea Sequential CupOfTea Parallel CupOfTea vs Tablesaw
Filter (id > 100 && val > 50) 8.52 ms 3.30 ms 0.50 ms 17.0x faster
Sum (val) 0.48 ms 0.09 ms 0.05 ms 9.6x faster
Group-By Aggregation 12.77 ms 2.48 ms 2.48 ms 5.2x faster
Sort 7.74 ms 18.04 ms 18.04 ms Comparable

Comparative Summary: CupOfTea's contiguous primitive vectors and parallel prefix-sum filter outperform Tablesaw by up to 17x on filtering and 9.6x on primitive numeric sums.

4.5 Multi-Core Thread Scaling Efficiency

Multi-core scaling efficiency measured with df.parallel(executor) on 500,000 rows:

Worker Threads Parallel Aggregation Parallel Filter Vector Map (sin/cos/sqrt)
1 Thread 1.06 ms 5.14 ms 5.89 ms
2 Threads 0.56 ms (1.9x) 2.69 ms (1.9x) 4.41 ms (1.3x)
4 Threads 0.32 ms (3.3x) 1.85 ms (2.8x) 3.19 ms (1.8x)
8 Threads 0.32 ms (3.3x) 1.85 ms (2.8x) 2.52 ms (2.3x)
16 Threads 0.33 ms 1.98 ms 2.64 ms

4.6 Macro Format IO Throughput

Measured ingest and egress parsing throughput across data formats (10,000 records):

Format Operation Latency (10k rows) Throughput (Rows/sec)
CSV Ingest / Parse 4.95 ms ~ 2,020,000 rows/s
CSV Egress / Write 6.69 ms ~ 1,500,000 rows/s
JSON Ingest / Parse 9.89 ms ~ 1,010,000 rows/s
JSON Egress / Write 7.46 ms ~ 1,340,000 rows/s
XML Ingest / Parse 21.88 ms ~ 457,000 rows/s
Excel (XLSX) Ingest / Parse 165.66 ms ~ 60,000 rows/s

4.7 Dataset Scale Benchmarks (10k to 10M rows)

Dataset Size Aggregation (Seq) Aggregation (Par) Vector Map (Seq) Vector Map (Par) Filter (Seq) Filter (Par)
10,000 rows 0.37 ms 1.92 ms 1.90 ms 1.76 ms 1.63 ms 1.57 ms
100,000 rows 2.02 ms 2.27 ms 3.73 ms 4.50 ms 4.93 ms 7.21 ms
1,000,000 rows 3.62 ms 2.69 ms 18.24 ms 12.94 ms 15.45 ms 12.94 ms
10,000,000 rows 29.15 ms 11.29 ms (2.6x) 192.86 ms 101.11 ms (1.9x) 105.35 ms 38.30 ms (2.75x)

4.8 One Billion Row Challenge (1BRC) & High-Cardinality GroupBy Aggregation

The 1BRC (One Billion Row Challenge) workload tests dataframe performance on high-volume measurements (<station_name>;<temperature>) with multi-metric aggregation (min, mean, max, count) across thousands of distinct station keys (41,343 distinct stations from weather_stations.csv).

Following the end-to-end streaming engine optimizations introduced across Issues #209 (lazy memory-mapped scan), #225 (fused scan-aggregate execution), #226 (SWAR 64-bit delimiter scanning), #227 (Short String Optimization in CompactStringTable), #228 (SWAR 64-bit block hashing in StringColumnVector), and #229 (dynamic work-stealing partition execution with AtomicLong cursor), CupOfTea achieves unprecedented throughput in pure Java, matching compiled Rust/C engines:

Full End-to-End Ingest & Aggregation Benchmark (1BRC Workload)

Tested on Apple Silicon (ARM64, OpenJDK 21 / Python 3.12, 41,343 distinct station keys):

Dataset Size Tablesaw 0.43.1 DFLib 1.3.0 Pandas 2.2 Polars (Eager) Polars (Lazy scan_csv) CupOfTea 0.15.1 (Legacy) CupOfTea 0.20.8+ (Interleaved Lazy Streaming) Speedup vs Legacy
1,000,000 rows (~15 MB) 1,584 ms 485 ms 200 ms 32 ms 36 ms 775 ms ~75 ms 10.3x faster
10,000,000 rows (~150 MB) 9,131 ms 3,444 ms 1,258 ms 332 ms 255 ms 4,166 ms 624 ms 6.7x faster
100,000,000 rows (~1.5 GB) 113.05 s (OOM) OOM / DNF 22.40 s 7.35 s 2.65 s 15.34 s 2.11 s (beats Polars) 7.3x faster
1,000,000,000 rows (~14.7 GB) OOM / Crash OOM / Crash OOM / DNF OOM / DNF 19.42 s OOM / DNF 31.12 s ∞ (Out-of-Core)

Key Architectural Drivers: 1. Dynamic Work-Stealing with AtomicLong Cursor (#229): Worker threads dynamically claim memory-mapped segments on demand via cursor.addAndGet(segmentSize) - segmentSize. Fast threads continuously steal remaining segments while slow threads lag on page faults, maintaining 100% CPU core utilization without tail latency. 2. Fused Scan-Aggregate Execution (#225): Direct streaming from memory-mapped CSV buffers into StreamingDirectAggregator completely bypasses intermediate DataFrame, ColumnVector, and StringChunkBuffer heap materializations. 3. SWAR 64-bit Block Hashing (#228): Reads 8 bytes per step via little-endian VarHandle (long[]) with prime multiplications, bit rotations, and Murmur3/xxHash-style avalanche finalizers. 4. Short String Optimization (SSO) (#227): Packed 16-byte key prefixes in CompactStringTable enable match verification using 1–2 primitive long comparisons, avoiding full byte array traversals for station names. 5. SWAR Delimiter Scanning (#226): 64-bit bit-manipulation scans delimiters and line endings 8 bytes per step with zero incubator module dependencies.

5. Continuous Performance Regression Gate

The repository includes a GitHub Actions performance workflow (.github/workflows/benchmarks.yml) that can be triggered on demand or on pull requests labeled benchmark.

This gate executes: 1. mvn test -pl cupoftea-benchmark -am: Verifies that benchmark assertions and harness smoke tests pass. 2. Shaded JAR packaging and benchmark execution saving machine-readable results to target/jmh-result.json. 3. Compares metrics against the established baseline to alert on performance regressions exceeding 15%.