# Separable vs non-separable spatio-temporal disaggregation ## 50-seed study (+ independent 10-seed replication) Data generated from the **non-separable** model (`stModel "121"`), then fitted with both the non-separable ("121") and separable ("102") models on the identical dataset. Metric: RMSE of the fine-grid linear predictor against the true field (24x24 grid x 24 time points = 13,824 points per fit). Design: 3 configs x 5 nsf x 3 ntf x 50 seeds x 2 models = **4,500 fits**, **2,250 paired comparisons**. Complete, no gaps. 153 h on a 16-core EPYC 7F52. --- ## HEADLINE Non-separable achieves lower RMSE in **56.8%** of 2,250 paired comparisons (95% CI 54.7-58.8%, sign test p = 1.6e-10). Median difference +0.000195, i.e. **~0.1% of an RMSE of ~0.20**. Real, but small. ## The 10-seed run OVERSTATED the effect An earlier independent 10-seed run (different field realisations, partly a different machine) gave **64.0%** [95% CI 59.4-68.4]. That estimate lies **outside** the 50-seed confidence interval — the two barely fail to overlap. | config | 10-seed | 50-seed | |-------------|------------------|--------------------| | 1 (rt=2) | 58.7% (p=0.041) | **50.5% (p=0.80)** | | 2 (rt=6) | 58.0% (p=0.060) | 54.8% (p=0.0095) | | 3 (rt=24) | 75.3% (p=4e-10) | 64.9% (p=2e-16) | | **overall** | 64.0% | **56.8%** | **Config 1's effect disappeared entirely** — 50.5%, a coin flip. The 10-seed p=0.041 was not reproducible. Treat any 10-seed per-config claim as unreliable. ## Spatial aggregation is the dominant factor Win rate for the non-separable model, by spatial coarsening (n=450 each): | nsf | 2 | 3 | 4 | 6 | 8 | |-------------|-------|-------|-------|-------|-------| | NS wins | 86.9% | 68.0% | 59.3% | 37.1% | 32.4% | Monotone, and it **crosses 50% between nsf 4 and 6**. At nsf 6-8 the *separable* model wins about two-thirds of the time. Temporal coarsening matters far less (ntf 2/3/4 -> 60.4% / 57.3% / 52.5%, n=750 each). ## The non-separable model is occasionally CATASTROPHIC at nsf=8 The mean difference is **negative** (-0.00333) while the median is positive (+0.000195). That is not a rounding quirk — it is a heavy left tail: | config | nsf | ntf | seed | rmse_NS | rmse_S | diff | |--------|-----|-----|------|---------|--------|---------| | 1 | 8 | 3 | 3 | 2.952 | 0.238 | -2.714 | | 1 | 8 | 2 | 8 | 2.190 | 0.201 | -1.989 | | 1 | 8 | 3 | 20 | 1.739 | 0.242 | -1.497 | | 3 | 8 | 3 | 26 | 1.423 | 0.238 | -1.185 | | 3 | 8 | 2 | 33 | 1.172 | 0.234 | -0.937 | In these cases the non-separable fit is **~10x worse** than the separable one. 12 comparisons (0.53%) have |diff| > 0.05; **10 of the 12 favour separable**. All the extreme ones are at nsf=8. Consequences: - The **paired t-test is non-significant** (p=0.073) because the tail dominates the mean. The sign test (p=1.6e-10) and Wilcoxon are the statistics to quote. - 10% trimmed mean = +0.000221, i.e. close to the median. The central tendency favours non-separable; the tail favours separable. Corroborating evidence: the only two **INLA convergence failures** in the whole run (2 of 350 nsf=8 fits; 0 of 1,404 elsewhere) were also at nsf=8. Both needed a retry. The same cells that fail to converge are the ones that blow up. ## Recommended framing > Assuming separability costs little in predictive accuracy and only when the > data are spatially informative. At fine spatial resolution (nsf 2-3) and long > temporal range (config 3) the non-separable model wins consistently but by > ~0.1-1% RMSE. Once the data are coarsely aggregated (nsf >= 6) the separable > model is better more often than not, and the non-separable model becomes > numerically unstable, occasionally producing predictions an order of magnitude > worse. The simpler model is the safer default unless the spatial resolution > supports the extra structure. ## Method notes 1. **Warm start removed.** `control.mode = list(theta = theta.ini, restart=TRUE)` was measured at **1.80x SLOWER** than a cold fit on the identical dataset (893.1s vs 495.7s). Each seed is an independent field realisation, so the previous seed's mode is a poor starting point. `control.mode` affects only where the optimiser starts, not where it converges, so results are unchanged. Removing it also made seeds independent and fully parallelisable. 2. **Bug fixed in `runa_s.R`.** A bare `load()` of the NS results dumped the whole saved global environment into `.GlobalEnv`, overwriting the separable chain's `theta.ini` with the NON-separable chain's — so the separable model was always initialised at the non-separable fit's mode. Any earlier results produced with that code path are affected. Moot after change 1, but fixed. 3. **No seeds were set** in the original scripts (neither `inla.qsample` nor the `rnorm` observation noise), so runs were not reproducible. Field generation is now seeded: 9001/9002/9003 for the 50-seed run, 1235/1236/1237 for the 10-seed. 4. The two runs use **disjoint field realisations** and separate output trees, so they are genuine independent replications, not a re-analysis. ## Files - `results/per-scenario-50seeds.csv` — 45 rows: config/nsf/ntf with mean+median RMSE per model, mean_diff, median_diff, pct_NS_wins - `results/rmse-50seeds.csv` — all 2,250 comparisons, per seed, with fit times - `results/per-scenario-summary.csv`, `results/rmse-all-10seeds.csv` — 10-seed run - `code/` — `drive_cell.R` (one config/nsf/ntf/seed unit), `run50.sh` (driver), `drive_generate_ns.R` (fields), `analyse_all.R` Raw `.RData` fit objects are **129 GB** and not included; each is reproducible via `drive_cell.R `.