Comments (10)
Could you give me some information on the system you are trying to submit a job on? What are the usual commands you enter when submitting a job?
from qsub.
It is a university cluster running CentOS 5.11 and I am using R 3.5.1 which has to be sourced prior to calling R
. I use the command qsub myscript.sh
to submit a job and qrsh
for an interactive login. The cluster uses GridEngine for jobs.
These are my commands locally:
config <- create_qsub_config(remote = remote, local_tmp_path=local_tmp, remote_tmp_path=remote_tmp, execute_before=r_source)
qsub_lapply(1:3, function(x) x + 1, qsub_config=config)
This is my error:
Error in rsync_remote(remote_src = qs$remote, path_src = qs$remote_logdir, : rsync failed:
Along with those error messages in log.1.e, log.2.e, log.3.e files mentioned in the original post.
from qsub.
Just to make sure something can be executed on the remote system, could you do:
qsub::run_remote("echo hello", remote = remote)
?
If that completes successfully, could you try to rsync something to the remote system:
file <- tempfile()
write("hello", file)
qsub::rsync_remote(FALSE, file, remote, "~/test")
Because of the first error you gave, it's highly unlikely that these commands will succeed, but then we can troubleshoot further!
from qsub.
It looks like the first command succeeded, but the second failed with the same error mentioned above.
from qsub.
Could you then run
file <- tempfile()
write("hello", file)
qsub::rsync_remote(FALSE, file, "prism", "~/test", verbose = TRUE)
and then run the first line you get (without the #
) in the command line? Then we'll know the exact error
from qsub.
This did not complete successfully. Returned the error:
Error in parse_remote(remote_dest) : object 'username' not found
I also tried running rsync_remote
line by line and it seems fetch_hostname_from_config
returns NULL
. I tried to debug but not familiar with keep
in lInes 19, 24, 29.
from qsub.
Sorry, I meant
file <- tempfile()
write("hello", file)
qsub::rsync_remote(FALSE, file, remote, "~/test", verbose = TRUE)
(prism is our remote 😉 )
from qsub.
So the first line that was returned from that command was the rsync
command and when I ran that in command line, it seems to have executed correctly. There is now a test
file with 'hello' written in my remote directory.
from qsub.
You said that this is the code you are trying to run:
config <- create_qsub_config(remote = remote, local_tmp_path=local_tmp, remote_tmp_path=remote_tmp, execute_before=r_source)
qsub_lapply(1:3, function(x) x + 1, qsub_config=config)
What is the content of r_source
?
Also, could you run the following code?
library(qsub)
config <- create_qsub_config(
remote = remote,
local_tmp_path = local_tmp,
remote_tmp_path = remote_tmp,
execute_before = r_source,
remove_tmp_folder = FALSE
)
qsub_lapply(1:3, function(x) x + 1, qsub_config = config)
By setting remove_tmp_folder
to FALSE
, the folder that qsub creates at remote_tmp
on the remote will not be removed, even if an error was produced. Inside, you should find a script.sh
and script.R
. script.sh
should look something like this:
#!/bin/bash
#$ -t 1-3
#$ -N myjob
#$ -l h_vmem=10G
cd /path/to/your/folder
module unload R
Rscript --default-packages=methods,stats,utils,graphics,grDevices script.R $SGE_TASK_ID
One of the errors (Unable to locate a modulefile for 'R') will be caused by the module unload R
. The other errors are likely related to whatever is in r_source
. What does your script.sh look like?
from qsub.
Closing this due to inactivity. Let me know if you have any more updates on this.
from qsub.
Related Issues (6)
- Handle error messages correctly HOT 1
- Remove the temporary directories even if an error was thrown HOT 1
- Add a progress bar while waiting for the output to finish HOT 3
- Offer an alternative method, instead of a qsublapply function, split it up into qsubsubmit and qsubretrieve. HOT 1
- Error and output log paths cause scripts to fail HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from qsub.