Status -2 returned by oc client about jenkins-client-plugin HOT 21 CLOSED

Do you need a proxy to reach your openshift api server from the location where you're running the jenkins client plugin? Is this a dupe of #24 ?

from jenkins-client-plugin.

I don't think it is a proxy problem. Otherwise I would get some error message. I just get an exit code -2 and no error message. If I run exactly the same command from command line then it works.

from jenkins-client-plugin.

I have been able to reproduce if not the exact error something very similar. Bottom line, either the tag operation is broke or how you invoke it is not obvious.

from jenkins-client-plugin.

actually, oc tag fails for me from the command line (running latest 3.6 origin), independent of jenkins, so let me sort out what is up (general openshift issue or user issue) and will report back.

from jenkins-client-plugin.

It fails for me with any command, not only oc tag. And from command line it works.

from jenkins-client-plugin.

@GolanTrevize10 - given the failure for any command, can you add an openshift.loglevel(8) prior to any of the commands being executed and provide the output. Thanks.

from jenkins-client-plugin.

Hi, I reinstalled Jenkins and now it is working. I think it was some problem after upgrading the plugin.

from jenkins-client-plugin.

Glad you are up and running @GolanTrevize10

from jenkins-client-plugin.

After installing the latest version of the plugin in a new Jenkins instance I am getting this error again. This is the verbose output

Verbose sub-step output:
Command> oc new-project test --skip-config-write --insecure-skip-tls-verify --server=https://xxxxxxx --loglevel=8 --token=XXXXX
Status> -2
StdOut>
StdErr>
Reference> {}

It is not proxy problem. Any idea what the problem can be?

from jenkins-client-plugin.

@GolanTrevize10 - to clarify, your prior comment implies to me that you see your problem running the oc binary from the jenkins host?

If so, this plugin does has not have bearing on that scenario. We can still try to help sort the issue out, but the answer to the question dictates the next steps we take.

from jenkins-client-plugin.

from jenkins-client-plugin.

OK, then at this point I simply need to see if I can reproduce.

I am going to need

• as simple of a pipeline script as you can provide that recreates the problem ... and by precise, I mean an actual script
• if you are using a non-default cluster config, those specifics so I can try and replicate something similar
• given your other issues, I'm assuming you are running outside of openshift, but please confirm
• the precise version of the oc binary you are using

thanks

from jenkins-client-plugin.

Hi, this is the script I am using

node{
// Get OC directory. Indicate name of tool configuration in Jenkins configuration.
def ocDir = tool "OC Tool"
// Define env variables
withEnv(["PATH+OC=${ocDir}/openshift-origin-client-tools-v3.6.0-c4dd4cf-linux-64bit","SKIP_TLS=true","https_proxy=http://b2b-http.dhl.com:8080"]) {
// Name of cluster configuration and user token. It is recommended to use a service account instead of your personal user.
openshift.withCluster( 'example', 'E5c8SuYQ5LYhazSJuWP4M6_PmvTs31lFLqYfulUjLBY' ) {
openshift.verbose()
openshift.newProject('test')

    }
}

}

I am running oc client 3.6.0 on a Jenkins instance outside Openshift. It doesn't even try to connect with Openshift cluster.

from jenkins-client-plugin.

I was checking the source code of durable task plugin. It looks like it returns status -2 when the script does not start after 15 seconds. I tried to increase it to 60 but still it is failing

if (_pid > 0 && !ProcessLiveness.isAlive(workspace.getChannel(), _pid, launcher)) {
// it looks like the process has disappeared. one last check to make sure it's not a result of a race condition,
// then if we still don't have the exit code, use fake exit code to distinguish from 0 (success) and 1+ (observed failure)
// TODO would be better to have exitStatus accept a TaskListener so we could print an informative message
status = super.exitStatus(workspace, launcher);
if (status == null) {
status = -1;
}
return status;
} else if (_pid == 0 && /* compatibility */ startTime > 0 && System.currentTimeMillis() - startTime > 1000 * LAUNCH_FAILURE_TIMEOUT) {
return -2; // apparently never started
}
return null;

from jenkins-client-plugin.

I changed the plugin to print the ouput from BourneShellScript and I see an error message now

nohup: failed to run command ‘sh’: Permission denied

And this is the command that it is trying to run

echo $$$$ > '/var/lib/jenkins/jobs/Openshift/jobs/test/workspace@tmp/durable-52f95ee0/pid'; jsc=durable-e2569e192fcc8c4971e94c22736a2ecc; JENKINS_SERVER_COOKIE=$$jsc '/var/lib/jenkins/jobs/Openshift/jobs/test/workspace@tmp/durable-52f95ee0/script.sh' > '/var/lib/jenkins/jobs/Openshift/jobs/test/workspace@tmp/durable-52f95ee0/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/jobs/Openshift/jobs/test/workspace@tmp/durable-52f95ee0/jenkins-result.txt'

Any idea how I can fix it?

from jenkins-client-plugin.

Hi @GolanTrevize10

So working off of the example you provided, I was unable to reproduce the issue you are seeing, even after making a modification of two in an attempt to produced an error.

As to what you uncovered in your durable task plugin debug, we have no idea what would cause that. The only remotely definitive that comes to mind is that given reinstalling jenkins solved things for you last time, along with the investigation you provided, uncovering what the durable task plugin is doing, it leads us to think that it is external to this plugin and something related to the jenkins set up.

As you noted, the nohup: failed to run command ‘sh’: Permission denied is the crux of the issue for you.

If you have not already, it might be worth working with the durable task plugin owners to understand what could case that. Once we've passed the "script" to BourneShellScript, it is in the hands of durable task plugin.

Between myself and this plugin's creator, @jupierce, this is a list of theories (wild guesses really) on possible causes, as well as additional things to try to obtain data points to compare:

• possible differences in the DTC plugin version ... as a point of reference, our jenkins image currently uses v1.13

• from your example script, and the node{, I'm presuming you are running off of jenkins master ... is that correct? or is a jenkins/slave involved (which could bring up a variety of concerns based on how it is launched) ? Is it possible the user id running this in Jenkins does not have permissions to leverage the oc you have installed for example ?

• ultimately, the script passed into BourneShellScript should have something that looks like oc new-project test --skip-config-write --insecure-skip-tls-verify --server=https://xx.xx.xx.xx:443 --loglevel=8 --token=XXXXX ... if you print out the contents of /var/lib/jenkins/jobs/Openshift/jobs/test/workspace@tmp/durable-52f95ee0/script.sh do you see that ?

• if you just run sh 'oc new-project test --skip-config-write --insecure-skip-tls-verify --server=https://xx.xx.xx.xx:443 --loglevel=8 --token=XXXXX ' from your pipeline, does that work? If it did, understanding how that shows up differently in BourneShellScript and the debug you've added would help

from jenkins-client-plugin.

Thanks for your answer Gabe. I'm using the DTC plugin 1.14. I will try with 1.13. I am running the job in master, not in a slave.
The folder doesn't exist. It exists only /var/lib/jenkins/jobs/Openshift/jobs/test/workspace@tmp/

from jenkins-client-plugin.

I checked if I run the command using the sh step it also fails with same error.

from jenkins-client-plugin.

Thanks for the sh 'oc ...' run @GolanTrevize10 .... that definitely rules out this plugin then.

We can still leave this issue open for now, and see if we can help you sort out what is going on.

Comparing results when changing the versions of durable task plugin and the oc binary would be the next logical steps, along with debug in the plugin.

Yeah, for /var/lib/jenkins/jobs/Openshift/jobs/test/workspace@tmp/durable-52f95ee0/script.sh I am pretty sure that is a temporary file. You'd have to change DTC to dump the contents of the file prior to the nohup: failed to run command ‘sh’: Permission denied

from jenkins-client-plugin.

Thanks Gabe, I found the problem. It is related to https://issues.jenkins-ci.org/browse/JENKINS-41339. I didn't notice it because the OC plugin is swallowing messages from the DTC plugin. I changed my Global variables in Jenkins and now it is working. You can close the issue.

from jenkins-client-plugin.

Ah - there you go! Thanks for the info and research @GolanTrevize10 - I've added a watch to that jira - I could certainly see other users hitting this issue.

@jupierce - a thought/question on the client plugin "swallowing messages from DTC": opinions on whether our use of the QuietTaskListener have any impact?

I'm wondering if we should make the user of that listener vs. the normal one controllable (perhaps via the verbose() method).

from jenkins-client-plugin.