Error in rsync_remote yields no output about qsub HOT 10 CLOSED

Could you give me some information on the system you are trying to submit a job on? What are the usual commands you enter when submitting a job?

from qsub.

It is a university cluster running CentOS 5.11 and I am using R 3.5.1 which has to be sourced prior to calling R. I use the command qsub myscript.sh to submit a job and qrsh for an interactive login. The cluster uses GridEngine for jobs.

These are my commands locally:
config <- create_qsub_config(remote = remote, local_tmp_path=local_tmp, remote_tmp_path=remote_tmp, execute_before=r_source)

qsub_lapply(1:3, function(x) x + 1, qsub_config=config)

This is my error:
Error in rsync_remote(remote_src = qs$remote, path_src = qs$remote_logdir, : rsync failed:

Along with those error messages in log.1.e, log.2.e, log.3.e files mentioned in the original post.

from qsub.

Just to make sure something can be executed on the remote system, could you do:
qsub::run_remote("echo hello", remote = remote) ?

If that completes successfully, could you try to rsync something to the remote system:

file <- tempfile()
write("hello", file)
qsub::rsync_remote(FALSE, file, remote, "~/test")

Because of the first error you gave, it's highly unlikely that these commands will succeed, but then we can troubleshoot further!

from qsub.

It looks like the first command succeeded, but the second failed with the same error mentioned above.

from qsub.

Could you then run

file <- tempfile()
write("hello", file)
qsub::rsync_remote(FALSE, file, "prism", "~/test", verbose = TRUE)

and then run the first line you get (without the #) in the command line? Then we'll know the exact error

from qsub.

This did not complete successfully. Returned the error:

Error in parse_remote(remote_dest) : object 'username' not found

I also tried running rsync_remote line by line and it seems fetch_hostname_from_config returns NULL. I tried to debug but not familiar with keep in lInes 19, 24, 29.

from qsub.

Sorry, I meant

file <- tempfile()
write("hello", file)
qsub::rsync_remote(FALSE, file, remote, "~/test", verbose = TRUE)

(prism is our remote 😉 )

from qsub.

So the first line that was returned from that command was the rsync command and when I ran that in command line, it seems to have executed correctly. There is now a test file with 'hello' written in my remote directory.

from qsub.

You said that this is the code you are trying to run:

config <- create_qsub_config(remote = remote, local_tmp_path=local_tmp, remote_tmp_path=remote_tmp, execute_before=r_source)
qsub_lapply(1:3, function(x) x + 1, qsub_config=config)

What is the content of r_source?

Also, could you run the following code?

library(qsub)
config <- create_qsub_config(
  remote = remote,
  local_tmp_path = local_tmp,
  remote_tmp_path = remote_tmp, 
  execute_before = r_source,
  remove_tmp_folder = FALSE
)
qsub_lapply(1:3, function(x) x + 1, qsub_config = config)

By setting remove_tmp_folder to FALSE, the folder that qsub creates at remote_tmp on the remote will not be removed, even if an error was produced. Inside, you should find a script.sh and script.R. script.sh should look something like this:

#!/bin/bash
#$ -t 1-3
#$ -N myjob
#$ -l h_vmem=10G
cd /path/to/your/folder
module unload R
Rscript --default-packages=methods,stats,utils,graphics,grDevices script.R $SGE_TASK_ID

One of the errors (Unable to locate a modulefile for 'R') will be caused by the module unload R. The other errors are likely related to whatever is in r_source. What does your script.sh look like?

from qsub.

Closing this due to inactivity. Let me know if you have any more updates on this.

from qsub.