edgexfoundry / ci-management Goto Github PK

See https://forum.snapcraft.io/t/fixing-out-of-order-message-output-when-piping-snapcrafts-output-to-file/9176 for full explanation, but basically we should update the docker build script for the snap to use unbuffer so that the log looks a bit better as oftentimes on jenkins logs the error gets output to stderr much earlier than the corresponding output at the same time on stdout, hence one has to do some backtracking when debugging the logs.

Originally filed as edgexfoundry/edgex-go#1018 but now since the snap build scripts are here, migrating to this repo

with the change to go lang 1.12 we want to also change the jjb freestyle job to a Pipeline

As we move to making go 1.12 a requirement, tests are failing due to shell/install_custom_golang.sh installing 1.11.5. In talking to @lranjbar, removing this pre_build_script should resolve the failure for new pull requests coming into master for app functions. #418

Example : edgex-go
US3472

go-mod-messaging and app-functions-sdk-go both have a dependency on zeromq.

• Update the build image to add zeromq

There is an assumption that edgex-go and export-disto have this dependency.

US3612

• PULL THIS IN

US3469

View is no longer needed. Please update the JJB template to remove the view.

The current view named volume should be modified to be named:
edgex-volume

Land new infrastructure for TIG stack

• backup plan if the ELK stack proposed by LF Infrastructure team doesn't work.

US3324

See for example : https://jenkins.edgexfoundry.org/view/Snap/job/edgex-go-snap-california-stage-snap/54/consoleFull

After having pushed the snap up, it fails to release here:

Processing...|                                                                 
22:16:38 Error: no such option: --channel
22:16:38 Build step 'Execute shell' marked build as failure

I think this is because in the snap stage job here: https://github.com/edgexfoundry/ci-management/blob/master/shell/edgex-publish-snap.sh#L14

we have

snapcraft release edgexfoundry $REVISION --channel=$SNAP_CHANNEL

which should be :

snapcraft release edgexfoundry $REVISION $SNAP_CHANNEL

This view is no longer needed. Please update the JJB template to remove the view named:
export

• Add new telegraf and jmeter containers with the ELK stack config

• Need endpoint details from LF Infra team to validate connectivity from sandbox

• DISCUSS OWNER WITH QA-TEST WG

US3327

Modify JJB script for edgex-go job definitions to include edinburgh work stream and remove california. The latter is no longer supported.

Not sure where to put this, but it would be nice if jenkins.edgexfoundry.org had a view category for all of the edgex-ui-* jobs (i.e. https://jenkins.edgexfoundry.org/view/Merge-Jobs/job/edgex-ui-go-master-merge-go/search/?q=edgex-ui&Jenkins-Crumb=c86df3190b6c686ae420ee0de9090c8f)
as currently I have to manually search for them every time I need to see one

When a config file requires a secret for credentials (example: telegraf config to connect to ELK endpoint), how will we manage secrets without committing the config file to GitHub EdgeX repository.
Issue discussed during a working / backlog grooming session with EdgeX team (JP / Trevor / Eric Ball).
We need to know how LF wants to handle secrets for containers so we can figure out how to build the automation that is required to launch the full stack for the EdgeX Blackbox tests.

US3470

JJB changes to convert device-random-go to use go modules
US3474

US3440

Automate the tagging of the go library modules
example: tag = v0.0.1

• go-mod-core
  US3478

The default mirror for alpine updates is very slow at times. Resulting in builds that are greater than 2 hours. I found that it is an issue for other folks here gliderlabs/docker-alpine#279. There isn't much we can do except switch mirrors since we don't have control over them.
I'm investigating this.

Automate the tagging of the go library modules
example: tag = v0.0.1

• go-mod-registry
  US3479

• requires a defined role in Jenkins to allow certain groups to trigger a job manually
• similar to other LF Jenkins projects

US3471

Modify JJB script for device-virtual-go job definitions to include edinburgh work stream

Assigned to: @tmpowers

edinburgh branch / code freeze

https://github.com/edgexfoundry/edgex-ui-clojure

NEW CODE DROP HAS NO BUILD AUTOMATION

US4514

Now that the project has standardized on go 1.11.9 for go repositories, we don't need to maintain a separate go install process for the security-secret-store jobs:

See:

Create the jjb builder macro that could be popped in on the job template. It would be the template level though so it would be a bigger scope than the one repo .

Currently the docker push scripts aren't parameterized by project in the JJB definition. We should refactor these scripts to be more generic. Also look into the logic for pushing to release.

See comments on #431 for the history of this issue

The edgex-ui-go master branch recently switched to using go modules, and now has jenkins jobs failing. See for example: https://jenkins.edgexfoundry.org/job/edgex-ui-go-master-verify-go/46/console from edgexfoundry/edgex-ui-go#85

I think the reason for this is that the default version of go is too old and doesn't support modules by default, and so we need to use the updated version of go that we use for i.e. edgex-go. AFAICT, for the edgex-go jobs this is accomplished in the job definition yaml with this snippet:

Does that mean all we need to do to get this to work on edgex-ui-go is to add that go-root spec to the master stream?

Before the schedule sets on the performance test job, the job has to trigger manually.
Planning set the schedule to "H 4 * * 6". ( The job will run at every Saturday 4 AM UTC time)

With regard to the go services, our current naming convention for Docker images is "docker--go". There are old entries in Docker Hub from previous releases that do not follow this convention. When in a hurry during troubleshooting scenarios, this can cause confusion. We should clean out the old entries.

Example:
OLD -- https://hub.docker.com/r/edgexfoundry/core-command-go/
CURRENT -- https://hub.docker.com/r/edgexfoundry/docker-core-command-go/

• spike to figure out sigul - using a faceless service account (Jenkins user)
• configure Jenkins to use Sigpul
• test / verification of signed tagging process

US3476

The docker tag is generated with the following snip:
grep -Po 'Successfully built \K[^ ]+`

Building Python wheels generates messages with the same format:

Successfully built EdgeXResourceBacnet pyyaml itsdangerous MarkupSafe
Successfully built a24a0fa9733a

Resulting in invalid tags (i.e. "EdgeXResourceBacnet\na24a0fa9733a").

Modify JJB script for app-functions-sdk-go job definitions to include edinburgh work stream

Build automation needed for DevKit USB installer iso image

ON HOLD - The device wg team will need to provide source code for the DevKit USB installer

US3196

Please add a LICENSE file to the root of this repo per https://wiki.edgexfoundry.org/pages/viewpage.action?pageId=21823866

This image has recently been updated; the merge job and release job do not work as they rely on reverse triggers to a maven job only.

Create the jjb that kicks of job similar to the blackbox testing with a telegraf container configured to hit the endpoint.
Suggestion is to simulate docker run hello-world 100xs and see if the data set from the running container is seen in the Grafana dashboard.

US3712

View is no longer needed in Jenkins. Please update the JJB template to remove the view named:
maven-release-california

Cleanup edgexfoundry-holding/edgex-ui-clojure
JIRA:
https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-16511

JJB changes to convert device-modbus-go to use go modules
US3473

Update and synchronize the Maven POM files to have the appropriate versions and dependency references in the Java repos (both the service and dependent libraries) listed below. Java services and libraries on the California branch should have 0.6.0 versions. Java services and libraries on the Master branch should have the 0.7.0 version.

libraries
core-test
core-exception
core-domain
support-domain
device-domain
device-controller
device-scheduling
export-client
core-command-client
export-test
support-notifications-client
export-domain
support-logging-client
core-data-client
core-metadata-client

services
support-rulesengine
device-mqtt
device-bacnet
device-bluetooth
device-modbus
device-fishertechnik
device-snmp
device-sdk
device-virtual

Need to modify edgex-go jobs to use go version 1.11.2

Owner: (Ernesto / JP)

US3329

example: tag = v0.0.1

• go-mod-messaging
  US3477

Root Cause Analysis for the failed arm builds where "no supported platform found in the manifest list" / "no matching manifest for XXX in the manifest list entries"

The California snap builds are failing and considered obsolete.
Need to remove the following from the jjb template
https://github.com/edgexfoundry/ci-management/blob/master/jjb/edgex-go/edgex-go-snap.yaml#L11-L13

SPIKE: Investigate GitHub Pull Request builder plugin - whitelist functionality as a security control for blocking malicious PRs

See for example the last few stage jobs:

• https://jenkins.edgexfoundry.org/view/Snap/job/edgex-go-snap-master-stage-snap/356/console
• https://jenkins.edgexfoundry.org/view/Snap/job/edgex-go-snap-master-stage-snap/357/console
• https://jenkins.edgexfoundry.org/view/Snap/job/edgex-go-snap-master-stage-snap/358/console
• https://jenkins.edgexfoundry.org/view/Snap/job/edgex-go-snap-master-stage-snap/359/console

(and also associated arm jobs which fail the same way).

This is failing on this line in the job:

which I believe is because the output of snapcraft push for some reason doesn't have the required regex in it and so the grep command exits with exit code 1 and since the shell script is run with -e it exits at that point. This is not ideal as we don't get to see what the real output is which would help identify the problem with the snap.

What we should do is use the --release option to snapcraft push and not have to do the two things as separate commands because then we don't need to know what the revision is at all.