Thorbjörn SievertLet's try this again outside of the heat wave and weekend:
I'm looking for help with #rstats, specifically larger #bam / #gam in #mgcv with NCV.
It's running, but it has been about 60h by now, minimal CPU usage, and steadily climbing RAM usage.
Any pointers how to speed things up are welcome!
The #bam in question:
```
gam_roi_treatment_TW <- bam(
cumulative_time_in_roi ~
s(time_point_s, bs = "tp", k = 10) +
s(time_point_s, treatment, bs = "sz", k = 10) +
s(time_point_s, sex, bs = "sz", k = 10) +
s(time_point_s, strain, bs = "sz", k = 10) +
s(time_point_s, treatment, sex, bs = "sz", k = 10) +
s(time_point_s, treatment, strain, bs = "sz", k = 10) +
s(time_point_s, sex, strain, bs = "sz", k = 10) +
s(time_point_s, treatment, sex, strain, bs = "sz", k = 10) +
s(age_baseline, bs = "tp", k = 10) +
s(time_point_s, ID, bs = "fs", k = 10),
data = roi_no_talad,
family = tw(),
select = FALSE,
method = "NCV",
nei = roi_notalad_nei,
control = gam.control(ncv.threads = N_THREADS)
)
```
Thorbjörn Sievert (@[email protected])
Question for the #stats folks working with #mgcv and #gam / #bam: I'm dealing with a dataset with a high temporal autocorrelation, so after a lengthy discussion with the author of `gratia` (my current PI), we concluded that I should move away from AR(1) models and rather work with #NCV. No problems with the setup, but the run time is forever. My first model has been running for 20h by now, with no indication of how long it will take. I have already switched over to #OpenBLAS, but it seems weird that my CPU load is constantly at only 15-25%. The only indication that something is happening is that RAM usage is slowly but steadily increasing. I'm dealing with ~40k data points across ~300 time series, and a NCV window of 7 data points, so nothing crazy. The model includes two- and three-way interactions, and random smooths, but all of those things are biologically relevant. I'd happily take any suggestions on how to speed things up, how to get proper CPU usage, or at least get an estimate of how far along the calculations are. I might eventually need to switch over to #twlss, from the current `tw()`, and that will make things worse as that is not supported by `bam()`. I'm currently on Windows, but I might be able to run things on an #HPC if needed, but not sure if I can easily fiddle with #BLAS there.
David Lawrence Miller