dataprep: fast reshaping with melt() and dcast()

Basic usage

df <- data.frame(
  id       = 1:3,
  category = factor(c("a", "b", "c")),
  v1       = c(1.1, 2.2, 3.3),
  v2       = c(4.4, 5.5, 6.6)
)
melt(df, id.vars = c("id", "category"))
#>   id category variable value
#> 1  1        a       v1   1.1
#> 2  2        b       v1   2.2
#> 3  3        c       v1   3.3
#> 4  1        a       v2   4.4
#> 5  2        b       v2   5.5
#> 6  3        c       v2   6.6

Measure-side specification

melt(df, measure.vars = c("v1", "v2"))
#>   id category variable value
#> 1  1        a       v1   1.1
#> 2  2        b       v1   2.2
#> 3  3        c       v1   3.3
#> 4  1        a       v2   4.4
#> 5  2        b       v2   5.5
#> 6  3        c       v2   6.6

Automatic ID inference

Non-numeric and factor columns are treated as IDs by default.

melt(df)
#>   category variable value
#> 1        a       id   1.0
#> 2        b       id   2.0
#> 3        c       id   3.0
#> 4        a       v1   1.1
#> 5        b       v1   2.2
#> 6        c       v1   3.3
#> 7        a       v2   4.4
#> 8        b       v2   5.5
#> 9        c       v2   6.6

Custom column names and na.rm

df_na <- data.frame(
  id = 1:3,
  x  = c(1, NA, 3),
  y  = c(4, 5, NA)
)
melt(df_na, id.vars = "id",
     variable.name = "var", value.name = "val",
     na.rm = TRUE)
#>   id var val
#> 1  1   x   1
#> 2  3   x   3
#> 3  1   y   4
#> 4  2   y   5

Memory layout: major = "row" vs major = "col"

Row-major ("row") is usually faster when there are few measure columns; column-major ("col") is often faster when there are many measure columns because bulk copies dominate. major = NULL (the default) is equivalent to "col"; there is no automatic switching based on the input shape.

wide50 <- data.frame(id = 1:100,
                     matrix(rnorm(100 * 50), ncol = 50))
res_row <- melt(wide50, id.vars = "id", major = "row")
res_col <- melt(wide50, id.vars = "id", major = "col")
identical(as.data.frame(res_row), as.data.frame(res_col))
#> [1] FALSE

Thread control

cores controls OpenMP. options(dataprep.cores = ...) sets a default for the whole session.

options(dataprep.cores = 4L)
melt(df, id.vars = "id")
#>   id variable value
#> 1  1 category    NA
#> 2  2 category    NA
#> 3  3 category    NA
#> 4  1       v1   1.1
#> 5  2       v1   2.2
#> 6  3       v1   3.3
#> 7  1       v2   4.4
#> 8  2       v2   5.5
#> 9  3       v2   6.6
options(dataprep.cores = NULL)

1. Two layout paths, selected by major

melt() produces the same long-format table as reshape2::melt (column-major, major = "col") or tidyr::pivot_longer (row-major, major = "row"). The two layouts have very different memory access patterns:

• Column-major writes each output column as one contiguous block: for measure column k, the output block [k * n .. (k+1) * n) is a direct memcpy of the input column. This is the fastest possible path when the number of value columns is moderate.

• Row-major writes every row as n_meas consecutive doubles. For small n_meas this is compact, but for large n_meas it requires a per-row transpose.

major = NULL (the default) is equivalent to "col"; there is no automatic switching based on the input shape.

2. SIMD streaming stores

For large outputs, melt_cpp uses AVX-512 or AVX2 streaming stores (_mm512_stream_pd, _mm256_stream_si256) to write directly to memory, bypassing the CPU cache. This avoids the cache pollution that would otherwise evict useful input data, and it is the reason the 1e8-row melt finishes in under 0.5 s on Ubuntu (vs 2.6 s for the next-fastest engine, polars) and in under 1.3 s on Windows. A _mm_sfence() is issued at the end of each streaming region to guarantee visibility.

3. Hugepage hint for large outputs

Output vectors larger than 512 KB are allocated through Rf_allocVector() and then hinted with madvise(MADV_HUGEPAGE), so the kernel can back them with 2 MB pages. This reduces first-touch page faults on the 1e8-row case. There is no custom R_allocator_t, no MAP_POPULATE, and no free pool: those were described in earlier drafts but are not part of the shipped 0.1.8 backend.

4. Fast paths for small inputs

Two fast paths are used. Tiny inputs (n <= 2048, n_meas <= 64, n_id <= 8) route to melt_tiny_cpp; small inputs (total <= 131072, n_meas <= 256) route to melt_small_cpp. Both skip hugepage hinting, OpenMP setup, and thread-cap detection. This is what makes melt() the fastest engine even on 1e3-row tables, where the initialization overhead of the other backends dominates their runtimes.

5. Cache-aware block sizing

For the row-major path, the per-thread block size is computed from the L3 cache size (read once from /sys/devices/system/cpu/cpu0/cache/index3/size on Linux). Each block is sized to fit in half of L3, which keeps both the input reads and the output writes inside the cache for the duration of the block.

6. Per-call caches for SEXP and factor levels

The "data.frame" class tag, the "factor" class tag, the default "variable" / "value" column names, and the factor levels vector are constructed once per process and reused afterwards via R_PreserveObject. This removes a small but measurable per-call cost that shows up on the small-input benchmarks.

Basic usage

long <- melt(df, id.vars = c("id", "category"))
dcast(long, id = c("id", "category"),
      variable = "variable", value = "value")
#>   id category  v1  v2
#> 1  1        a 1.1 4.4
#> 2  2        b 2.2 5.5
#> 3  3        c 3.3 6.6

Formula interface

dcast(long, formula = id + category ~ variable,
      value.var = "value")
#>   id category  v1  v2
#> 1  1        a 1.1 4.4
#> 2  2        b 2.2 5.5
#> 3  3        c 3.3 6.6

Fill missing cells

dcast(long, id = c("id", "category"),
      variable = "variable", value = "value",
      fill = 0)
#>   id category  v1  v2
#> 1  1        a 1.1 4.4
#> 2  2        b 2.2 5.5
#> 3  3        c 3.3 6.6

Aggregating duplicate pairs

If a (id, variable) pair appears more than once, pass fun.aggregate. The default is “last occurrence wins”, matching data.table::dcast’s fun.aggregate = NULL behaviour.

long_dup <- data.frame(
  id       = c(1, 1, 2),
  variable = c("x", "x", "x"),
  value    = c(1, 2, 3)
)
dcast(long_dup, id = "id",
      variable = "variable", value = "value",
      fun.aggregate = mean)
#>   id   x
#> 1  1 1.5
#> 2  2 3.0

na.rm

long_na <- data.frame(
  id       = c(1, 1, 2, 2),
  variable = c("x", "y", "x", "y"),
  value    = c(1, NA, 3, 4)
)
dcast(long_na, id = "id",
      variable = "variable", value = "value",
      na.rm = TRUE)
#>   id x  y
#> 1  1 1 NA
#> 2  2 3  4

1. Block-path detection (Phase 0a–0c)

When the input is a canonical melt() output — variable is periodic and every id column is constant within one period — dcast_cpp skips the hash tables entirely and performs a tile transpose: each tile of TILE x period doubles is read contiguously into an L1 buffer, transposed in place, and written contiguously to the output columns. This keeps both reads and writes sequential and enables OpenMP parallelisation. The cost per tile is O(TILE * period), independent of the number of levels, which is why wide-level tables scale well.

2. 64-bit packed keys and 96-bit fingerprint fallback

For block-aligned input with few id columns, dcast_cpp packs the row key into a single uint64_t. When the combined bit budget of the id columns exceeds 64, it falls back to a 96-bit fingerprint (uint64_t h1 + uint32_t h2) computed by a 4-way parallel FNV-1a and stored in a 16-byte slot table, which is more cache-friendly than the previous 128-bit scheme.

3. Radix sort for shuffle-friendly input

When the input is not sorted, a 4-pass LSD radix sort over the packed keys replaces the hash table entirely. For inputs whose rows are already grouped — which is the common case after a melt() + arrange() pipeline — the sort is skipped.

4. Permutation shortcut

When the block path is taken and the id column is a permutation of 1..n_blocks (the most common case for canonical melt() output), a direct-index shortcut bypasses both the hash table and the sort.

The full pipeline is described in the dcast() help page.

melt consistency

Engines: dataprep, reshape2, data.table, tidyr, pandas, polars, dask, duckdb.

dcast consistency

The consistency scripts are shipped under inst/:

Sys.setenv(DATAPREP_RUN_BENCHMARK = "1")
source(system.file("benchmark_melt_dcast.R", package = "dataprep"))
melt_all_engines(10000L, n_id = 1L, n_val = 9L)
dcast_all_engines(1000L, n_id = 2L, n_val = 5L)

Ubuntu 25.10

Windows 11 Pro for Workstations

For melt(), on the largest cells (1e8 rows, 8 GB of input), dataprep is the only engine that completes within 2.5 s: under 0.3 s on Ubuntu and under 1.0 s on Windows.