eficode / codesonar-plugin Goto Github PK

Stacktrace follows.

17:45:02 FATAL: Java heap space

17:45:02 java.lang.OutOfMemoryErrorhttp://stacktrace.jenkins-ci.org/search?query=java.lang.OutOfMemoryError: Java heap space

17:45:02 at java.util.Arrays.copyOf(Unknown Source)

17:45:02 at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)

17:45:02 at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)

17:45:02 at java.lang.AbstractStringBuilder.append(Unknown Source)

17:45:02 at java.lang.StringBuffer.append(Unknown Source)

Likely culprit:
http://stackoverflow.com/questions/13612441/string-replace-using-huge-heap-space

Currently versions are handled as float comparing floats is dangerous. We should Introduce a Version object that is always returned, but implement it as an Optional object, which would allow for us to check for Unknown versions.

Following the semver patch, we should implement a newerThan(Version v) for safe comparison.

This happens with a 3.8Gb log file

java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.String.<init>(String.java:207)
	at java.io.BufferedReader.readLine(BufferedReader.java:356)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at org.apache.commons.io.IOUtils.readLines(IOUtils.java:1033)
	at org.jenkinsci.plugins.codesonar.CodeSonarPublisher.perform(CodeSonarPublisher.java:109)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
	at hudson.model.Build$BuildExecution.post2(Build.java:186)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
	at hudson.model.Run.execute(Run.java:1749)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)

Reverting to polling. Always works.

https://wiki.jenkins.io/display/JENKINS/CodeSonar+Plugin#CodeSonarPlugin-JenkinsJobDSL

Running the following job dsl results in an error:

job("build-linux") {
  parameters {
    stringParam("HUB", "")
    stringParam("PROJ", "linux","Project name")
  }

  scm {
    git("https://github.com/torvalds/linux.git","master")
  }

  steps {
    shell('''export PATH=$PATH:/home/ubuntu/ressources/codesonar-4.4p0/codesonar/bin
codesonar analyze $PROJ -foreground $HUB make -j2''')
  }

  publishers {
    codesonar {
      protocol('http')
      hubAddress('${HUB}')
      projectName('${PROJ}')
     }
  }
}

Jenkins ver. 2.121.1 and CodeSonar Plugin v. 2.0.7

Error:

Processing provided DSL script
ERROR: (script, line 17) the following options are required and must be specified: credentialId
Finished: FAILURE

Could query more details informations. Add sql.html to the url. Read the docs.

There might be a way to ask for what sql query builded this page, so it is easy to figure out these queries.

Long time ago we was supposed to run analysis of Linux kernel, and see how huge data set the plugin can handle (due to earlier memory leak issues and bad coding around handling large data set).

It was possible and #33 wasn't successful, but this is now possible with the latest addition of https://github.com/Praqma/codesonar-plugin/tree/master/test and commit e5d898d from solving #39.

So now we only need to run it and report results.

• Run wgen.py job (need to be created first)
• Run linux kernel analysis
• If something fails, try to do some java monitoring
• Update repository root README with link and short description of the test setup we have available now

This piece of code is not of any use, because the hub-address is collected from the configuration in job where the user specify the hub-address.

This menthod will also in current version of the plugin and code sonar hub resolve to null, falling back to using the job configuration specified by the user as this piece of code shows: https://github.com/Praqma/codesonar-plugin/blob/master/src/main/java/org/jenkinsci/plugins/codesonar/services/AnalysisService40.java#L65

https://github.com/Praqma/codesonar-plugin/blob/master/src/main/java/org/jenkinsci/plugins/codesonar/services/AnalysisService40.java#L46

We don't have historic reasons why we needed to resolve it from the build console, when we could just have it as a configuration.

The plugin documentation specifies to use the HUB as a parameter, so they same value is used in the codesonar analyze command as in the post-build action as ${HUB} so we do not have any user-stories around specifying difrent HUB urls.

So it should be safe to clean up this code.

"the memory requirements of the plugin on large analyses are impractical and the user had given up on the plugin and instead set up Jenkins to run codesonar manually. The user suspected a memory leak, however I suspect it may simply be an issue where it attempts to keep too much data in memory at once."

This issue will transfer our internal testing setup to the plugin, so it can be used as reference.

Everybody can use it for testing, if they have AWS and can get a CodeSonar license.

http://code.praqma.net/ci/view/Open%20Source/view/codesonar-plugin/job/1_unit-tests_codesonar/

We can easily use visualvm, but it would be nice to have a trend graph of key data over the execution period of a build, so we don't have to keep an eye on visualvm.

Automated trends and data collection using the jstat and jmx configuration should be done.

We're fixing an out of memory error in #27 caused by reading a very large Jenkins console output searching for an analysis ID to query latest results from Code Sonar. This was not even needed as we could look up ID of the project from version 4.4 of Code Sonar query API for latest analysis ID.

For version older of 4.4 it might still not be needed, as codesonar build command seems to be the only one omitting the needed information (codesonar analyze do not).

So in all circumstances of CodeSonar hub 4.0, 4.2 and 4.4 it seems like we never need to parse the console log of Jenkins, but can instead use the HUB URI and projectname that are configured in the Jenkins CodeSonar plugin post-build step.

So I suggest as a first fix to get rid of any Jenkins console log parsing, and use the query method we already have in place based on the plugin configuration.

Second I suggest that we also stop assuming the Jenkins job should use the latest CodeSonar results, but use a specific ID.
It seems we can query that based on a file in the workspace, Dave Vitek suggest foo.prj_files/aid.txt where foo equals our projectname configuration in the plugin as the name of the CodeSonar project.

It would be good from an end-user perspective that the plugin had an explicit configuration saying something similar to:

Which project analysis ID to query on CodeSonar hub?
[  ] Latest  or [ ] project ID from file $file

where the user would actually see $file as the replaced project name file real location.

The help should further explain what latest mean and how we query the hub.

We're moving to Jenkins as Code at https://jenkins.praqma.cloud/
We could move the current DSL directly, but we would like to modernise. Therefore the current DSL should
be converted to declarative pipeline script.

We need to simplify the current pom. That means we do not need to have advanced configurations at all.

• cobertura (doesn't work with java8)
• javancss
• etc...

Is there a way to set the parent path of a project with the plugin?

If we set the project name just to the name it works (e.g. dev_feature2).
If we set the project name including the parent path (e.g.
/ABC/EUR/Sandbox_multibranch/dev_feature2) we get an error response:
"ERROR: Project by the name /ABC/EUR/Sandbox_multibranch/dev_feature2 was
not found on the hub"

Customer:

The background is that we want to be sure to retrieve the correct analysis
result if we have two projects with the same name under different parent
paths.

Codesonar Plugin version: 2.0.5
Codesonar version: 4.4p0

This is the notes from the handover session specifying how to build, test and run the codesonar plugin

We'll need to revert the one commit on master that is not released to get this out.

Setup the right env that matches the description of the bug report.

1. Try to se if CodeSonar plugin is compatible with this version
2. Run code analysis of CodeSonar plugin source code

The plugin have trend graphs, and applies thresholds so we need historic data from the analysis result persisted with the builds.

But if we look at the build.xml file inside Jenkins, it contains all data from the parsed xml dataset from the hub, including for example url, notes etc. and an entry for each finding from the analysis.
We don't need to persist all those data, it's massive, so we should only persist the summarized data we need for threshold and graphs.

The test setup for manual testing could benefit from installing and configuring timestamp plugin in Jenkins, so each line in the job get a timestamp so we can see how long time different part of the builds are taking.

Add to the plugin list for Jenkins, and add configuration (if needed) for the job dsl scripts.

does not render the total warning count graph

Read more on: https://praqma.fogbugz.com/f/cases/15062

As a user would like to be able to set a score threshold for what
is reported, and have a Jenkins failure option that is based on absolute
counts at or above the threshold. So, for example, they might choose to
set the threshold at a score of 56. This would count all the CodeSonar
warnings that have the red marker at the far left hand side of the
warning table. Then they would like the Jenkins run to fail if the count
of these issues is greater than a specified absolute value. This is not
a comparison to the count from the previous run, but an absolute value.
For example, 20. Then, if this is feature is implemented the Jenkins
run would list as a failure if there are more than 20 warnings with a
score of 56 or higher. Warnings with lower scores are not a concern for
this scenario.

Proposed solution: Either introduce a parent project field, allowing the tree-structure without project name to complement the project name field, or allow the project name field to include the parent projects in a tree-slash structure like parent1/parent2/project.

This feature is a request based on support issues #18

@kryptag installed a local hub on his MAC, so that means we could it in container.

The last step after that to have hub running is to add a license when starting the first time.
If we could add all the steps before that in Docker it would make the development much more effective.

This issue is about getting the environment up and running and make it possible to analyse a project

1. Send mail to codesonar
2. get login
3. download codesonar
4. follow installation guide
5. request license
6. add license to local codesonar instance
7. Document the proces and make it reproducible

For the test setup we use for large scale project analysis, see test folder, we need to be able to access CodeSonar documentation and download CodeSonar releases.

Grammatech requires an account, earlier where @kryptag developed he used a personal account.

I have created [email protected] as account with Grammatech and await approval so we can use it as a common one.
The mail is an alias for relevant Praqma developers.

Still try the misra on linux kernel, disable the massive memory usage test. There should be a switch.

In case this will not run, Grammatech will supply a runtime.

We will test with generated warnings doubling the numbers until something fails.

Explain the following:

• a default job in Jenkins that will analyse the codesonar plugin from github, the job we did now @kryptag and @MadsNielsen.
• also a few tips and tricks with know error messages like:
  • restarting hub when running out of workers
  • how to start the hub initially
  • how to avoid the src code could not be found with Findbugz by disabling it

How to run debugging attached to running jenkins.

Much smaller, and only need to count lines. Almost.

warning rank seems to be renamed to score

referenced in #15

If CodeSonar is set to restrict access to analysis results so
Anonymous cannot see results, and an appropriate user name and password
is provided to the plugin, the plugin still fails to collect the
information. Based on the http requests made, the login attempt happens
and succeeds, but the request for results doesn't use that session and
instead starts a new anonymous session. When the anonymous session fails
to get results, the named session disconnects. This looks like a need to
keep track of the session from the login for other requests.

It would be easy to use AWS CloudWatch to monitor these instances during execution of our tests.

Resources:

• Easy to add using aws cli command: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html
• Free for 5 minutes update intervals, should be good for us
• small scripts available to monitor "inside" details of the OS, also for Ubuntu: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html

Furthermore, the result should NOT be stored, since the log can get very large in certain situations.

So...wrap it in a BufferedReader, and return immediately when we find a result.

See https://github.com/Praqma/codesonar-plugin/blob/master/test/README.md#wgenpy-project

"ERROR: Step 'Codesonar' failed: Project by the name MC4_APP_STIHL_XMC1100 was not found on the hub Finished: FAILURE"

When calling getRedAlerts or GetYellowAlerts on Analysis model if there are no alerts the application throws a nullPointerException.