---
title: "Long event output for large simulations"
author: "TA Trikalinos"
date: "2026-08-28"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Long event output for large simulations}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
set.seed(20260828)
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library(nhppp)
K <- 10^5
```

## What this vignette shows

The vectorized samplers return, by default, a **dense matrix** of event times:
one row per point process, one column per event, with rows that have fewer
events than the widest row padded with `NA` on the right. This is convenient
and fast, but the matrix is allocated at the width of the *widest* row. When
event counts vary a lot across point processes -- a few heavy processes among
many light ones -- almost all of that memory holds `NA`.

The `output = "long"` option returns the same draw in **long format** instead:
one entry per *event*, with no padding at all. This vignette shows the format,
verifies that the two outputs describe the same realization, and measures the
memory and speed differences.

## The two formats

A population of `r format(K, big.mark = ",")` processes over $(0, 4]$, where
most processes have a low constant rate (about 1 expected event) and 100
"heavy" processes have rate 10 (about 40 expected events):

```{r fixture}
rate <- c(rep(0.25, K - 100), rep(10, 100))
lambda_matrix <- matrix(rep(rate, 5), ncol = 5)
time_breaks <- seq(0, 4, length.out = 6)
```

The dense default:

```{r dense}
Z_dense <- vdraw_sc_step(
  lambda_matrix = lambda_matrix,
  time_breaks = time_breaks
)
dim(Z_dense)
mean(is.na(Z_dense)) # fraction of the matrix that is padding
```

The 100 heavy processes set the width of the matrix for everyone, so -- as
printed above -- nearly all cells are `NA` padding.

The long format, from the same sampler:

```{r long}
Z_long <- vdraw_sc_step(
  lambda_matrix = lambda_matrix,
  time_breaks = time_breaks,
  output = "long"
)
str(Z_long)
```

One entry per event: `id` is the (1-based) index of the point process,
ascending, with event times ascending within each `id`; `n_draws` records how
many processes were sampled. There is no `NA` anywhere.

**How "no events" is represented.** A process with zero events contributes no
entries -- its id is simply absent. `n_draws` is what distinguishes "process
present but with no events" from "process never sampled":

```{r no-events}
counts <- tabulate(Z_long$id, nbins = Z_long$n_draws)
sum(counts == 0) # processes with no events: present in n_draws, absent in id
```

For `data.table` users the long format is one line from a keyed event table,
ready to join against a population table by `id`:

```{r data-table, eval = requireNamespace("data.table", quietly = TRUE)}
events <- data.table::data.table(id = Z_long$id, time = Z_long$time)
events[c(1, .N)]
```

## The two outputs describe the same realization

The long format is not a different sampler -- the algorithms and their use of
the random number stream are identical, and only the recording of events
differs. A same-seed pair of draws contains exactly the same event times:

```{r same-seed}
set.seed(1)
Zd <- vdraw_sc_step(
  lambda_matrix = lambda_matrix[1:1000, ], time_breaks = time_breaks
)
set.seed(1)
Zl <- vdraw_sc_step(
  lambda_matrix = lambda_matrix[1:1000, ], time_breaks = time_breaks,
  output = "long"
)
# flatten the dense matrix by row (the long format's grouping) and compare
dense_events <- as.vector(t(Zd))
identical(Zl$time, dense_events[!is.na(dense_events)])
identical(tabulate(Zl$id, Zl$n_draws), as.integer(rowSums(!is.na(Zd))))
```

## Memory

The dense matrix costs $8 \, n \max_i N_i$ bytes; the long format costs about
$12 \sum_i N_i$ bytes (8 for each time, 4 for each id). The ratio is therefore
driven by $\max_i N_i / \bar N$ -- the *heterogeneity* of the event counts, not
the number of processes. Three populations of `r format(K, big.mark = ",")`
processes with the same expected total number of events but increasing
heterogeneity:

```{r memory}
mem_case <- function(rate_vector, label) {
  lm <- matrix(rep(rate_vector, 5), ncol = 5)
  Zd <- vdraw_sc_step(lambda_matrix = lm, time_breaks = time_breaks)
  Zl <- vdraw_sc_step(lambda_matrix = lm, time_breaks = time_breaks, output = "long")
  data.frame(
    population = label,
    max_events = ncol(Zd),
    dense_MB = round(as.numeric(object.size(Zd)) / 2^20, 1),
    long_MB = round(as.numeric(object.size(Zl)) / 2^20, 1)
  )
}
# equal expected totals (~1.25 e5 events), increasingly concentrated
homogeneous <- rep(0.3125, K)
mild <- c(rep(0.25, K - 5000), rep(1.5, 5000))
extreme <- c(rep(0.25, K - 100), rep(10, 100))
tab <- rbind(
  mem_case(homogeneous, "homogeneous"),
  mem_case(mild, "mildly heterogeneous"),
  mem_case(extreme, "extremely heterogeneous")
)
knitr::kable(tab)
```

With homogeneous counts the two formats cost about the same (the dense matrix
is narrow, and the long format pays 4 extra bytes per event for the id). As
heterogeneity grows, the dense matrix widens with the heaviest row while the
long format stays proportional to the actual number of events. In the extreme
case the dense matrix is dozens of times larger. The effect scales: at
$10^7$ processes the extreme fixture costs gigabytes dense and tens of
megabytes long.

## Speed

The long format is not a time-memory trade-off -- skipping the allocation,
`NA`-filling, and trimming of the wide matrix also saves time:

```{r speed}
time_case <- function(rate_vector, label, n_rep = 3) {
  lm <- matrix(rep(rate_vector, 5), ncol = 5)
  td <- system.time(for (i in seq_len(n_rep)) {
    vdraw_sc_step(lambda_matrix = lm, time_breaks = time_breaks)
  })["elapsed"]
  tl <- system.time(for (i in seq_len(n_rep)) {
    vdraw_sc_step(lambda_matrix = lm, time_breaks = time_breaks, output = "long")
  })["elapsed"]
  data.frame(
    population = label,
    dense_ms = round(1000 * td / n_rep),
    long_ms = round(1000 * tl / n_rep)
  )
}
knitr::kable(rbind(
  time_case(homogeneous, "homogeneous"),
  time_case(mild, "mildly heterogeneous"),
  time_case(extreme, "extremely heterogeneous")
))
```

## The conditioned samplers work the same way

`output = "long"` composes with the reporting and generation-conditioning
options. For instance, conditioning on at least two events and reporting in
long format:

```{r conditioned}
Z <- vdraw_sc_step(
  lambda_matrix = lambda_matrix[1:1000, ],
  time_breaks = time_breaks,
  generate_at_least_K = 2,
  output = "long"
)
min(tabulate(Z$id, Z$n_draws)) # every process has at least 2 events
```

Under a pure upper bound, zero-event processes have positive probability and
appear -- correctly -- as absent ids:

```{r conditioned-upper}
Z <- vdraw_sc_step(
  lambda_matrix = lambda_matrix[1:1000, ],
  time_breaks = time_breaks,
  generate_at_most_K = 1,
  output = "long"
)
table(tabulate(Z$id, Z$n_draws))
```

`vdraw_cumulative_intensity()` also accepts `output = "long"`; on its
conditioned (order-statistics) path the long result is built without any dense
intermediate, and `Lambda_inv` is evaluated once on the event vector.

## Which format to use

- **Dense (default)**: first-event workloads (`report_first_K = 1`),
  homogeneous event counts, or any code that expects the matrix layout.
  Nothing changes for existing code.
- **Long**: many processes with ragged event counts -- event-history
  simulations, heavy-tailed intensities, or whenever the events feed a
  `data.table`/long-format pipeline anyway. Memory is proportional to the
  events actually drawn, and the draw is somewhat faster.
