dataprep: performance and cross-engine consistency

Overview

dataprep 0.1.8 ships two reshaping backends, melt() and dcast(), benchmarked here against all seven major alternatives in the R and Python ecosystems:

• R: reshape2, data.table, tidyr
• Python: pandas, polars, dask, duckdb

Every cell is measured with a C++ steady-clock timer and an adaptive times rule (20 / 15 / 10 / 5 / 1 iterations based on warmup time). Two statistics are recorded per cell:

• mean — the number quoted in every table below. Each timed call runs after gc(full = TRUE) and py_gc_collect(), so the GC / allocation tail is kept out of the timing window and the mean is a steady-state throughput measure rather than a GC-jitter measure.
• median — a robust cross-check. It is not tabulated here, but it is kept in the raw CSV files shipped under inst/extdata/. The scatter plot below compares the two statistics cell by cell.

For every cell the tables also report the speed-up of dataprep relative to each competitor, so the reader can see the full gradient from “about the same” to “three orders of magnitude”.

Mean vs median

The four benchmark CSV files shipped under inst/extdata/ carry both the mean and the median of every per-cell timing sample. The scatter plot below puts them side by side: each point is one (tool, host, shape) combination, the x axis is the median in milliseconds and the y axis is the mean. Points on the 1:1 line mean the two statistics agree; points above the line mean the mean is inflated by a long right tail in the per-iteration timings.

suppressPackageStartupMessages(library(ggplot2))

read_bench <- function(fname, op) {
  p <- system.file("extdata", fname, package = "dataprep")
  d <- read.csv(p, stringsAsFactors = FALSE)
  d <- d[!d$skipped, c("tool", "mean", "median")]
  d$op <- op
  d
}

bench <- rbind(
  read_bench("bench_melt_ubuntu.csv",  "melt (Ubuntu)"),
  read_bench("bench_dcast_ubuntu.csv", "dcast (Ubuntu)"),
  read_bench("bench_melt_win.csv",     "melt (Windows)"),
  read_bench("bench_dcast_win.csv",    "dcast (Windows)")
)

ggplot(bench, aes(median, mean)) +
  geom_abline(slope = 1, intercept = 0,
              linetype = "dashed", colour = "grey50") +
  geom_point(alpha = 0.45, size = 1.4) +
  scale_x_log10() +
  scale_y_log10() +
  facet_wrap(~ tool, ncol = 4) +
  labs(x = "median (ms, log scale)",
       y = "mean (ms, log scale)") +
  theme_bw(base_size = 10)

Almost every point sits on or just above the 1:1 line. The visible exceptions are the few dask and duckdb cells in the 1e3 × 10000 shape, where a single slow iteration pulls the mean up by up to 50 %; those are also the cells where the two statistics disagree the most on the speed-up ratio. For the dataprep column itself the two statistics never differ by more than a few percent, which is why the mean-based numbers quoted throughout this vignette are representative of the steady state.

Test environment

Benchmarks were run on two reference hosts. Only the core configuration is listed here; full hardware details are in README.md.

• Ubuntu 25.10 (Questing Quokka, kernel 6.17.0-41-generic) — 2× AMD EPYC 9965 192-Core (Turin, Zen 5c), 384 physical / 768 logical cores, L3 768 MiB, 1.0 TiB (16 × 64 GiB Micron, DDR5-5600, Multi-bit ECC), full AVX-512; R 4.5.1, g++ 15.2.0.

• Windows 11 Pro for Workstations (10.0.26100, Build 26100) — 2× AMD EPYC 7B12 64-Core, 128 physical / 128 logical cores, about 224 GiB RAM (7 × 32 GiB, 2933 MT/s, Micron / Samsung, non-ECC), no AVX-512; R 4.6.1 (ucrt), GCC 14.3.0.

Software versions on both hosts: data.table 1.18.6.1, reshape2 1.4.5, tidyr 1.3.2, reticulate 1.47.0; Python 3.13.7 (Ubuntu) / 3.13.15 (Windows), pandas 3.0.6, polars 1.44.2 (runtime rt64), dask 2026.8.0, duckdb 1.5.5.

How to read these numbers

The two hosts differ in core count, cache size and memory bandwidth. Two properties shape the numbers that follow:

• The Ubuntu host is unusually large. Most of the 1e6- and 1e7-row cells fit entirely in L3. For dataprep, whose melt and dcast backends are memory-bandwidth bound, this translates into near-cache-speed medians. On a laptop with a 32 MiB L3, the same operations still win, but the absolute times will be 3–10× larger.

• The Windows host has no AVX-512. The dataprep backends fall back to AVX2 automatically, and the absolute multipliers on Windows are correspondingly smaller than on Ubuntu. The relative ranking of the engines is identical on both hosts.

Both effects favour dataprep in the numbers below. The relative ranking is robust; the absolute multipliers — especially the 1629× and 800× figures — should be interpreted as “best-case on a very large machine”. On a typical 8–16-core workstation the same comparisons are within 10–100×.

Cleaning pipeline (dataprep 0.1.5 → 0.1.8)

The 0.1.8 release rewrites every heavy cleaning routine in C++. The table below compares against 0.1.5 on three dataset sizes from the same source (SMEAR I Varrio forest). All numbers are speed-up ratios (0.1.5 time / 0.1.8 time) on Ubuntu 25.10.

On Windows 11 Pro for Workstations, the same full-year pipeline gives obsedele ≈ 648×, condextr ≈ 839×, shorvalu ≈ 81×, optisolu ≈ 25× (at cores = 32), and the integrated dataprep call ≈ 173×. varidele is around 1.17× on this cell; this is expected, since varidele is a single colMeans(is.na(.)) in both versions and the new code path has little room for improvement.

Note on optisolu cores. The 0.1.5 implementation could crash when cores > 16, because its parallel::makeCluster() path gave each worker a full copy of the data. The benchmark above used cores = 16 for both versions to keep the comparison fair. 0.1.8 loads the package on each worker, exports the input data once per worker, and runs each (interval, times) case as a separate task, so cores = 64 is safe. The practical speed-up on a many-core host is larger. Also note that the optimal parameter values returned by optisolu() may differ slightly between 0.1.5 and 0.1.8.

melt() — wide to long

Input shapes are described as rows × (n_id + n_val). All numbers in the cells are means in milliseconds; the value in parentheses is dataprep’s speed-up relative to that competitor.

melt() on Windows 11 Pro for Workstations

The same four slices as the Ubuntu host, with no AVX-512.

dcast() — long to wide

Input is a canonical long table with every (id, variable) pair present exactly once. All numbers in the cells are means in milliseconds; the value in parentheses is dataprep’s speed-up relative to that competitor.

dcast() on Windows 11 Pro for Workstations

The same four slices as the Ubuntu host, with no AVX-512.

Summary of speedups

Speedup is defined as competitor mean / dataprep mean. Each table summarises every benchmark cell on that host, across all seven competitors (reshape2, data.table, tidyr, pandas, polars, dask, duckdb).

Cross-engine consistency

Both melt() and dcast() produce output numerically identical to reshape2 on every tested cell. All pairs of engines agree pairwise within tol = 1e-12.

Reproducing the benchmarks

The full runner is shipped under inst/:

• benchmark_helpers.R — adaptive per-tool runner with a 15 s first-call cap
• benchmark_melt_dcast.R — integrated driver. It runs both the per-tool benchmarks for melt() / dcast() and the 8-engine consistency checks, and prints the tables shown above.

Scripts are disabled by default so that R CMD check does not run them. To enable:

Sys.setenv(DATAPREP_RUN_BENCHMARK = "1")
source(system.file("benchmark_melt_dcast.R", package = "dataprep"))

Session info

sessionInfo()
#> R version 4.5.1 (2025-06-13)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 25.10
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblasp-r0.3.30.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=zh_CN.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=zh_CN.UTF-8        LC_COLLATE=zh_CN.UTF-8    
#>  [5] LC_MONETARY=zh_CN.UTF-8    LC_MESSAGES=zh_CN.UTF-8   
#>  [7] LC_PAPER=zh_CN.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=zh_CN.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Asia/Shanghai
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] ggplot2_4.0.3  dataprep_0.1.8
#> 
#> loaded via a namespace (and not attached):
#>  [1] vctrs_0.7.3        cli_3.6.6          knitr_1.52         rlang_1.3.0       
#>  [5] xfun_0.61          otel_0.2.0         generics_0.1.4     S7_0.2.2          
#>  [9] jsonlite_2.0.0     glue_1.8.1         htmltools_0.5.9    sass_0.4.10       
#> [13] scales_1.4.0       rmarkdown_2.32     grid_4.5.1         tibble_3.3.1      
#> [17] evaluate_1.0.5     jquerylib_0.1.4    fastmap_1.2.0      yaml_2.3.12       
#> [21] lifecycle_1.0.5    compiler_4.5.1     dplyr_1.2.1        RColorBrewer_1.1-3
#> [25] pkgconfig_2.0.3    Rcpp_1.1.2         farver_2.1.2       digest_0.6.39     
#> [29] R6_2.6.1           tidyselect_1.2.1   pillar_1.11.1      parallel_4.5.1    
#> [33] magrittr_2.0.5     bslib_0.12.0       withr_3.0.3        tools_4.5.1       
#> [37] gtable_0.3.6       cachem_1.1.0