dataprep 0.1.8

Upgrading from 0.1.5? Read this first

Three behaviour changes affect row and column counts. All three are bug fixes, but each one changes the output on real data. If your downstream analysis depends on exact row counts, read the quantified comparison below before upgrading.

1. obsedele() now scans each column independently. The 0.1.5 implementation collapsed all selected columns into one long vector before computing missing runs; this changed NA run boundaries and could both over-delete boundary rows and retain rows that should have been deleted. The 0.1.8 implementation scans each column independently: a row is deleted when any selected column has a missing run longer than half minutes on both sides.

2. half is now always in minutes, and the boundary is inclusive. In 0.1.5, half counted grid rows in units of by: with by = "5 min", half = 30 the effective window was 150 minutes. In 0.1.8, half is always in minutes, independent of by. Rows whose nearest anchor is exactly half minutes away are retained (within half minutes is a <= condition).

3. optisolu() no longer crashes with cores > 16. The 0.1.5 parallel::makeCluster() path exhausted memory when the worker processes each received a full copy of the input. The 0.1.8 implementation loads the package on each worker, exports the input data only once per worker, and runs each (interval, times) case in a separate task, so cores = 64 and cores = NULL (automatic) are both safe.

Quantified effect on a full-year dataset

On SMEAR I Varrio 2025 (49,422 rows × 61 numeric channels, 10-minute sampling), running the same pipeline with the same parameters:

Net change: 0.006% of the input. The six rows that differ between versions all sit at run boundaries where the anchor distance is within one sampling interval of half minutes.

See vignette("dataprep-migration") for the full upgrade guide and minimal reproductions of both changes.

Test environments

Two reference hosts were used. Their relative ranking of the engines is identical; the absolute multipliers scale with the hardware.

Reference host A — Ubuntu 25.10

Reference host B — Windows 11 Pro for Workstations

Performance summary

Cleaning pipeline (dataprep 0.1.5 → 0.1.8)

Speedup relative to 0.1.5 on the same input, same parameters. Values below 1.0× mean the new implementation is marginally slower on that cell.

On Windows 11 Pro for Workstations, the same full-year pipeline gives obsedele ≈ 648×, condextr ≈ 839×, shorvalu ≈ 81×, optisolu ≈ 25× (at cores = 32), and the integrated dataprep call ≈ 173×. varidele is around 1.17× on this cell; this is expected, since varidele is a single colMeans(is.na(.)) in both versions and the new code path has little room for improvement.

Note on optisolu cores. The 0.1.5 implementation could crash when cores > 16. The benchmark above used cores = 16 for both versions to keep the comparison fair. 0.1.8 loads the package on each worker, exports the input data once per worker, and runs each (interval, times) case as a separate task, so cores = 64 is safe. The practical speed-up on a many-core host is larger than the table above.

melt() — speed-up vs every one of the 7 major alternatives

Speed-ups relative to each competitor span 0.6×–1628.9× across both hosts. The sub-1.0× cells are concentrated at 1e7 rows with 10 id columns (Ubuntu) and 1e5 rows with 10 id columns (Windows), where polars is faster than dataprep; every other cell has dataprep ahead of or on par with the fastest competitor.

Means in milliseconds (Ubuntu 25.10):

Means in milliseconds (Windows 11 Pro for Workstations):

The median speed-up across all melt cells and all competitors is 11.3× on Ubuntu and 5.6× on Windows. The mean is 67.8× and 46.6× respectively. The median is pulled down by the 1e5-row small tables; at larger scales the speed-up is much higher.

dcast() — speed-up vs every one of the 7 major alternatives

Speed-ups relative to each competitor span 1.9×–799.8× across both hosts. Every cell has dataprep ahead of every other engine.

Means in milliseconds (Ubuntu 25.10):

Means in milliseconds (Windows 11 Pro for Workstations):

The median speed-up across all dcast cells and all competitors is 46.5× on Ubuntu and 41.4× on Windows. The mean is 90.2× and 76.6× respectively. On the 1e8-row cells (8 GB of input), dataprep completes in 40–214 ms while several competitors exceed 12 s on Ubuntu and 12 s on Windows.

Cross-engine consistency

melt() and dcast() produce output numerically identical to reshape2, data.table, tidyr, pandas, polars, dask, and duckdb on every tested shape, within tol = 1e-12.

Full tables — including mean, median, and the full per-competitor gradient — are in vignette("dataprep-performance"). The reproducible runner is shipped under inst/.

Behaviour changes

obsedele() semantics

The 0.1.8 C++ backend (obsedele_cpp) implements the retention criterion with an anchor-based scan: for each missing value and each selected column, the time distance to the nearest non-missing anchor on the left and on the right is computed directly. A row is deleted when any selected column has both distances exceeding half minutes.

This is mathematically identical to the running-mean criterion used in 0.1.0 and the rleid-based criterion used in 0.1.5, but:

• Scans each column independently. The 0.1.5 implementation merged columns before computing runs, which changed run boundaries and could both over-delete and under-delete boundary rows.
• Runs in O(n) time with O(1) extra allocation per column. No grid materialisation, no run-length state.
• Parallelises over columns with OpenMP without any shared mutable state.

optisolu() multi-core safety

The 0.1.5 parallel::makeCluster() path gave each worker a full copy of the input. With cores > 16 and large data this exhausted memory and aborted the R session. The 0.1.8 implementation shares read-only data across workers and accepts up to 64 cores safely.

melt() new major and as.factor arguments

The major argument is now honoured strictly. NULL (default) is equivalent to "col": column-major, identical to reshape2::melt. major = "row" produces tidyr-compatible row ordering. The earlier implementation could switch automatically based on input shape, and the tiny fast path silently ignored major; both are fixed. There is no longer any automatic switching.

A new as.factor argument controls the type of the variable column. NULL (default) uses TRUE for major = "col" and FALSE for major = "row". Explicit TRUE / FALSE overrides that default. The return value is always a plain data.frame; no tibble attributes are attached.

prep_fit() / prep_transform() degenerate columns

A constant training column has sd = 0, IQR = 0, or max - min = 0. prep_fit() now stores 1 as the scale value for such columns, so the transform becomes x - center. prep_transform() additionally guards against scale_val == 0 in case a plan is edited by hand.

Bug fixes

• dcast(): na.rm = TRUE combined with an explicit non-NA fill no longer loses the fill value. On the block-path (canonical melt output), the tile transpose wrote the input values unconditionally after the fill pass, so cells whose input was NA/NaN ended up as NA instead of the requested fill. The transpose kernels now receive na_rm and the resolved fill_val and substitute them while the tile is built. The general path was not affected; both paths now produce identical output. Repro: dcast(data.frame(id=c(1,1,2,2), variable=c("x","y","x","y"), value=c(1,NA,3,4)), id="id", variable="variable", value="value", na.rm=TRUE, fill=-1) now returns (1,"y") = -1.

• dcast(): duplicate (id, variable) pairs now resolve consistently with the documented “last occurrence wins” rule. The block path previously kept the first occurrence of a duplicated block; the general path kept the last. The block path now keeps the last occurrence, matching dcast()’s documentation and reshape2::dcast(). This only affects non-canonical inputs (canonical melt() output has no duplicates); canonical round-trips are unchanged.

New functions

Cleaning

• balance_panel() — balance an unbalanced panel by filling or completing.
• bin_data() — discretize continuous variables.
• clean_strings() — trim / case / regex cleaning of character columns.
• deduplicate() — exact and fuzzy duplicate removal.
• encode_categorical() — label / frequency / one-hot encoding.
• filter_high_cor() — drop highly correlated variables.
• filter_low_var() — drop near-constant variables.
• phys_filter() — physical range filtering.
• validate_data() — rule-based data validation.
• winsorize() — cap extreme values.
• zerona() — replace zeros with NA.

Imputation and transformation

• impute_missing() — linear / LOCF / NOCB / mean / median.
• log_returns() — log returns.
• transform_data() — log / sqrt / Box-Cox / Yeo-Johnson and z-score / min-max / robust scaling.

Diagnostics and reporting

• na_diagnose() — missing-value run statistics.
• data_report() — compact data quality report.
• dry_run() — simulate preprocessing without changing data.

Time series

• create_lags() — grouped lag / lead columns.
• day_night_flag() — day / night indicator.
• season_flag() — season / month / quarter indicator.
• decompose_ts() — additive / multiplicative decomposition.
• detrend_ts() — linear detrending.
• remove_diurnal_cycle() — subtract mean diurnal cycle.
• resample_time() — resample to coarser period.
• roll_apply() — rolling statistics with alignment.
• drift_detect() — rolling drift detection.

Sampling and workflow

• sample_data() — simple and stratified sampling.
• prep_fit() / prep_transform() — fit / transform style preprocessing plan that prevents data leakage.

Reshaping

• dcast() — long-to-wide reshaping, paired with melt().

API changes

• Argument cols now consistently accepts names, integer indices, or logical masks across the package.
• All exported functions gain a verbose = FALSE argument that controls progress and timing messages.
• Functions that operate on a time column accept date_col = NULL; when NULL, the first column matching date, Date, or DATE is used.
• melt() gains id.vars, measure.vars, variable.name, value.name, na.rm, cores, major, as.factor, verbose, parallel_threshold. id.vars is an alias of id for reshape2 / data.table compatibility.
• dcast() ships a formula interface (id1 + id2 ~ variable), value.var as an alias of value, and fun.aggregate for reducing duplicate (id, variable) pairs, plus fill, na.rm, cores, and verbose.
• dataprep(), shorvalu(), descdata(), melt(), dcast(), prep_fit(), and prep_transform() gain a cores argument for OpenMP control (the cleaning functions obsedele(), condextr(), percoutl(), and optisolu() already accepted cores since 0.1.5; their backends now route it to OpenMP as well). The global option options(dataprep.cores = ...) is respected by melt() and dcast().
• descdata() now accepts stats as either numeric indices or character names.
• percplot() now prints both the sample size (n) and the number of missing values (na) in each facet when a grouping column is supplied.
• descplot() and percplot() gain a num_xaxis argument that overrides the automatic choice between log and linear x-axis scales when column names are numeric.

Documentation

Seven vignettes ship with the package:

• vignette("dataprep-philosophy") — design philosophy and preprocessing methodology.
• vignette("dataprep-cleaning") — step-by-step walkthrough of the four cleaning steps.
• vignette("dataprep-performance") — full benchmark tables and 8-engine consistency checks.
• vignette("dataprep-migration") — 0.1.5 → 0.1.8 upgrade guide.
• vignette("dataprep-workflow") — leakage-free preprocessing with prep_fit() / prep_transform().
• vignette("dataprep-melt-dcast") — fast reshaping usage and implementation notes.
• vignette("dataprep-plots") — descriptive statistics and diagnostic plots.

Performance notes

Three benchmark cells sit close to, or marginally behind, the fastest competitor — dcast() at 100 id columns, melt() at 1e7 rows × 10 id columns, and varidele() on full-year data — and all three share the same root cause: the affected internal buffers (the 96-bit fingerprint table, the per-id-column id blocks, and the is.na scratch matrix) are allocated with Rf_allocVector, which routes through R’s default malloc-based allocator and therefore cannot be directed to hugepages, a per-process free pool, or a NUMA-aware arena, so first-touch page faults dominate those cells. Two possible fixes were prototyped on the reshaping backends and both were measured to close the gap: routing those buffers through Rf_allocVector3 with a hypothetical custom R_allocator_t adds a further few-fold speed-up on top of the existing hundred-fold to thousand-fold margins, and returning ALTREP virtual objects from melt() / dcast() — a lazy variable column and a deferred id block, so that large parts of the output are never materialised — adds another order of magnitude (tens of times) on the wide-table shapes. Neither is used in this release. Rf_allocVector3 is not recommended by CRAN, and an ALTREP return value, while fast to produce, is slow for downstream complex statistics because every element access re-enters the virtual-object layer, which shifts the cost from dataprep to the caller’s analysis code.

The shipped 0.1.8 backends therefore keep the standard allocation path, and the reported speed-ups stand as measured: melt() spans 0.6×–1628.9× and dcast() spans 1.9×–799.8× across the two reference hosts, with the sub-1.0× melt() cells confined to the 1e7-row (Ubuntu) and 1e5-row (Windows) shapes with 10 id columns, where polars is faster than dataprep. On the cleaning pipeline the 0.1.5 → 0.1.8 speed-ups of 1.1×–1146× stand as reported; Rf_allocVector3 and ALTREP were not evaluated there.

Environment variables (optional)

• DATAPREP_RUN_BENCHMARK = 1 enables the shipped benchmark scripts. They are disabled by default so that R CMD check does not execute them.

See also

• Design philosophy — why the pipeline has the shape it does.
• Cleaning pipeline — full walkthrough of varidele / obsedele / condextr / shorvalu.
• Performance — benchmark tables and cross-engine consistency.
• Migration — upgrade guide.
• Leakage-free workflow — prep_fit / prep_transform.
• Reshaping — melt / dcast.
• Descriptive statistics and plots — descplot / percplot.

dataprep 0.1.5

• Initial public release on CRAN.
• Core cleaning pipeline: varidele, obsedele, condextr, percoutl, optisolu, shorvalu, dataprep.
• Descriptive statistics and percentile helpers: descdata, descplot, percdata, percplot.
• Example datasets: data, data1.