Futureverse
batchtools_slurm fails to retrieve job status #810
Unanswered
baldikacti
asked this question in
Q&A
Replies: 3 comments 5 replies
|
What happens if you use: plan(future.batchtools::batchtools_slurm, scheduler.latency = 120)? Please also try with a longer-running task, e.g. library(future.batchtools)
plan(batchtools_slurm)
f <- future({ Sys.sleep(120); Sys.info() })
value(f)The above is just for troubleshooting purposes, but it might help us narrow in on what's going on. PS. Please avoid screenshots as they're not accessible, searchable, non-cut'n'pasteable |
0 replies
|
Sorry about the images. TL.DR: It appears that setting
Tested conditions:
Results: Script: library(future.batchtools)
plan(future.batchtools::batchtools_slurm, scheduler.latency = 120)
f <- future(Sys.info())
value(f)
Console output: Results: Script: library(future.batchtools)
plan(batchtools_slurm)
f <- future({ Sys.sleep(120); Sys.info() })
value(f)
Console output: Results: Script: library(future.batchtools)
plan(future.batchtools::batchtools_slurm, scheduler.latency = 60)
f <- future({ Sys.sleep(120); Sys.info() })
value(f) |
4 replies
|
Thanks this is very useful. I suspect this is related to futureverse/future.batchtools#74, which already has some ideas on what is going on. I'll report back here ... |
|
I'm still a bit puzzled what's going on here. Since I cannot reproduce it on my end (neither SGE nor Slurm), it's hard for me to troubleshoot other than carefully trying to emulate the future.batchtools and batchtools function calls in my head. Oh well... Can you please install the develop version of future.batchtools (>= 0.12.2.9966) using: remotes::install_github("futureverse/future.batchtools, ref = "develop")and then try with: library(future)
plan(future.batchtools::batchtools_slurm)
f <- future(Sys.info())
withr::with_options(list(future.debug = TRUE), { r <- result(f) })What debug output do you get? You should see something like: except that you'll end up with an error (which I don't above). |
|
It looks like it is working now. Below are the results from your example. But I've also tried the original code in the Mastodon thread, and it works even without adding
Script: |
|
Thank you. It's good that you've got a successful run. That establishes a baseline on your system. That said, I haven't made any updates that should have solved your original problem. My best guess is that whether it will succeed or fail depends on how busy your Slurm scheduler or your global file system is. On my end, FWIW, I've just verified that the develop version of future.batchtools works on three different Slurm clusters; I list the details of these clusters under https://future.batchtools.futureverse.org/reference/batchtools_slurm.html#details. |
|
future.batchtools 0.20.0 is on CRAN as of 2025-08-25. It comes with lots of improvements (https://future.batchtools.futureverse.org/news/index.html). |
1 reply
|
future.batchtools 0.21.0 is on CRAN as of 2025-09-21. Even more things were improved (https://future.batchtools.futureverse.org/news/index.html). @baldikacti , please retry if you have a chance. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This is a continuation of the Mastodon thread. (https://fosstodon.org/@henrikbengtsson@mastodon.social/115035868081050874)
System Details:
Ubuntu 24.04.2 LTS
slurm-wlm 24.11.6
Script run:
Command run:
Rscript main.RConsole output:
.futuredirectory attached below.future.zip
Submitted job resource usage:
Submitted
sbatchfile:All reactions