hepcloud / fnalhpc_startd Goto Github PK

COBALT_JOBID is not propagated inside the container, hence the job_name_prefix takes no value. When doing the matching for job/file processing it will always match, damaging the synchronous nature of the setup.

TODO: No inbound, hangs, FNAL site firewall blocks this, maybe we can do it from a WMAgent, need to test this with cmsgwms-submit1?. Would be wonderful if script would just scp everything before HPC/COBALT job is submitted.

╰─ [$] scp -vvv [email protected]:/home/cmsdataops/macosta/CMSSW-mcgen-testjob.sh /home/macosta/fnalhpc_startd/lumberjack/hpc/cobalt-cms-13499/sandbox/CMSSW-mcgen-testjob.sh
Executing: program /usr/bin/ssh host fermicloud510.fnal.gov, user root, command scp -v -f /home/cmsdataops/macosta/CMSSW-mcgen-testjob.sh
OpenSSH_7.9p1, OpenSSL 1.1.0i-fips  14 Aug 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 2: Applying options for *
debug2: resolving "fermicloud510.fnal.gov" port 22
debug2: ssh_connect_direct
debug1: Connecting to fermicloud510.fnal.gov [131.225.152.148] port 22.

In the meantime, login node has SSH inbound connectivity and can be reached by the Schedd, so we need to scp the files over. This can't be automated in any way since SSH login to ALCF requires MFA. But the script will generate the command... or maybe write it to a mini script. Will have to implement that as it sounds useful.

The standalone starter uses mira-home as base, we should take advantage of the node SSD by moving the execute directory over there.