opensearch-project / opensearch-ci Goto Github PK
View Code? Open in Web Editor NEWEnables continuous integration across OpenSearch, OpenSearch Dashboards, and plugins.
License: Apache License 2.0
Enables continuous integration across OpenSearch, OpenSearch Dashboards, and plugins.
License: Apache License 2.0
@abhinavGupta16 commented on Thu Feb 03 2022
We are able to deploy the ECR related roles and repositories on Isengard using cdk automatically.
The ecr roles set up trust relationships with existing roles/nodes deployed with jenkins-ci
@abhinavGupta16 commented on Fri Feb 11 2022
Following on offline team discussion, we will add the cdk to deploy ECR as a part of opensearch-ci
. Moving therefore, moving this issue to opensearch-ci
Here is a list of all of the jobs on the Jenkins staging system, lets figure out if we need them or not
Final results are in:
Looks like a version update of jenkins is needed as some plugins are not longer working after the log4j security vulnerability.
From my dev stacks /var/log/jenkins/jenkins.log
Running from: /usr/lib/jenkins/jenkins.war
2022-01-27 00:11:34.261+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated
2022-01-27 00:11:34.274+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated
2022-01-27 00:11:34.278+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @308ms to org.eclipse.jetty.util.log.JavaUtilLog
2022-01-27 00:11:34.317+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
java.lang.SecurityException: class "Log4jHotPatch"'s signer information does not match signer information of other classes in the same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:891)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:661)
at java.lang.ClassLoader.defineClass(ClassLoader.java:754)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:304)
at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)
2022-01-27 00:11:35.233+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2022-01-27 00:11:35.273+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.33.v20201020; built: 2020-10-20T23:39:24.803Z; git: 1be68755656cef678b79a2ef1c2ebbca99e25420; jvm 1.8.0_312-b07
2022-01-27 00:11:35.463+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2022-01-27 00:11:35.487+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2022-01-27 00:11:35.488+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2022-01-27 00:11:35.488+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 660000ms
2022-01-27 00:11:35.719+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME")
2022-01-27 00:11:35.782+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@3aacf32a{Jenkins v2.263.4,/,file:///var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war}
We need to import the certificates for the Jenkins instance into the cdk project and adjust the settings and configuration so a new deployment is all wired up.
Acceptance Criteria:
@abhinavGupta16 Alternatively, would it be better to maybe write the groovy code as a groovy file and import it?
Values can then later be replaced as variables from the props. This would make the groovy code more readable and IDEs would also be able to parse it to verify for errors etc
I did think about that, about using InitFile.fromAsset but how would we change the values once the file is created in the ec2 or even before it? These files contain dynamic values like SG, subnet ids, etc. How should we replace that?
Will look into it if we can use Cfn:Fn:sub
like I used it here
Originally posted by @gaiksaya in #39 (comment)
Currently, no jenkins job will run on jenkins since docker and withAws
is not installed.
Make changes so jenkins jobs with docker agent nodes are able to run.
No response
No response
We are currently deploying jenkins version jenkins-2.263.4
However, the latest LTS has many security fixes including the plugins. Jenkins is smart enough to install latest compatible plugin and hence we can also remove hard coded version of plugins.
Since the configuration is loaded from jenkins.yaml
, the plugin specific version should not matter.
Upgrade to latest LTS version and remove plugin hard coded versions.
No response
The upgrade is not a direct change as jenkins has changed the way it installs on CentOS.
A quick check concluded that changes like https://github.com/opensearch-project/opensearch-ci/blob/main/lib/compute/jenkins-main-node.ts#L213 does not take effect on jenkins.service anymore.
Needs deep dive into configuration.
Everything in GHA is open-source and in repos. The Jenkins scripts and configuration should follow suit. Anyone should be able to reproduce the OpenSearch CI/CD infrastructure after #2 and importing scripts and configuration that would be OSS.
We have those scripts in source control now, but it remains private. Need to OSS those.
Describe the bug
Before PR's are allowed to be merged the OpenSearch repository enforces running a CI task, ./gradlew check
, which runs full unit, integration, smoke, and varying cluster tests. This takes a very long time and consumes a lot of resources. To avoid becoming a bottleneck we introduced a manual start gradle check
that a maintainer can kick off by adding a comment to a PR once a basic review of a PR has been made. This is a manual step.
Expected behavior
No manual interaction from a maintainer other than approving/merging a PR.
Screenshots
Additional context
This is a meta issue tracking all subtasks for enabling Jenkins workflows and fleet monitoring
We already have some system level alarms configured as a part of the current cdk code. CodeBase
Apart from that we need to add below monitoring metrics:
Is your feature request related to a problem? Please describe.
Community wants to see which jobs are being run for CI/CD and logs.
Describe the solution you'd like
We are running a Jenkins instance that is currently private. Make it public.
Is your feature request related to a problem? Please describe.
I would like to execute Continuous Integration on arm64 hosts.
Describe the solution you'd like
https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners to create arm64 at organization level, hence, this can be used by all OpenSearch projects.
Describe alternatives you've considered
Running integration tests manually from ARM64 laptop
Additional context
N/A
The links in here are not accessible
opensearch-project/OpenSearch#1343 (comment)
Click on the links in this comment
opensearch-project/OpenSearch#1343 (comment)
No response
If applicable, add screenshots to help explain your problem.
No response
No response
No response
The useSsl
parameter is always false irrespective of the input parameter value.
set the --parameter useSsl=true
and output the value of useSsl
flag value. The value will be false.
The value of the useSsl
boolean value should be set according to the parameter value.
If applicable, add screenshots to help explain your problem.
No response
No response
No response
http://url should auto redirect to https when SSL is enabled.
currently it does not.
Need to change this line to http:// https://github.com/opensearch-project/opensearch-ci/blob/main/lib/compute/jenkins-main-node.ts#L222
Deploy the cluster using ssl enabled and try http://url. It hangs up
where as https://url works.
Http should redirect to https automatically
If applicable, add screenshots to help explain your problem.
No response
No response
No response
Agent node config and keys setup is done manually currently which needs to be automated.
Secrets Management: The setup should be automated and done using the CI-Config-Stack
as a part of Jenkins deployment via cdk
Agent Node configuration: TBD
From #26
.@gaiksaya
Quick question about jenkins plugin abhinavGupta16 if we do not specify the version of these plugins, won't it install most recent version that can differ in functionality in someway? peterzhuamazon can weigh in here regarding plugins and how important their versioning is (or not important).@abhinavGupta16
The script that ran to install plugins did not have versions for the plugins. This is a good point we should think about..@peterzhuamazon
I dont see this as a problem as we want to update the plugins to the latest versions to patch all vulnerabilities.
However, I also think maintaining a list of versions for stable usage is also beneficial.
You can have 1 template for latest and 1 template for stable. What do you think about this?
Need to document how that system works, interfacing patterns, and guidance as well as on-boarding new jobs.
End to end process.
The jenkins .yml file in the root of the repository needs to be moved into a more portable location so it can be reused by other libraries.
With current set up whenever there is a change with EC2 configuration or change in InitCommands, the stack replaces the EC2 with new one with new configurations. In initial stages it is fine because there is no relevant data. However, once in advance stage the replacement of EC2 will lead to data loss as EC2 is replaced.
Currently looking at the below data that needs to be retained:
We are uploading all logs from (/var/log/jenkins/jenkins.log) to cloudwatch so no need to have that.
I do not have concrete solution for config.xml because of the following concern:
Config.xml stores all the global configurations in Jenkins including env variables, global settings, etc.
If we replace the newly deployed config.xml, we overwrite new changes made by new deployment.
If we keep the new one, we overwrite all the old changes made by the system.
Possible solution:
Pros:
Cons:
Pros:
Cons:
A multi-attach EBS volume that can be attached to the EC2 instance.
Pros:
Cons:
Use EFS to store the data entirely out of EC2 on shared file system.
Mount the EFS at a specific location so that old data and history is accessible.
Pros:
Cons:
We've seen an unexplained update on our staging jenkins environment, we need to be sure that updates are only driven by operator actions.
sh-4.2$ sudo su -
Last login: Sat Dec 11 03:06:17 UTC 2021 on pts/1
[root@ip-10-0-181-89 ~]# yum history
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
ID | Command line | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
5 | install -y adoptopenjdk- | 2021-08-12 18:48 | Install | 1
4 | install java-11-openjdk | 2021-08-11 21:57 | Install | 2
3 | update | 2021-08-11 21:56 | I, O, U | 23 EE
2 | install -y jenkins-2.263 | 2021-08-02 04:54 | Install | 1
1 | -y install git java-1.8. | 2021-08-02 04:54 | Install | 82
history list
[root@ip-10-0-181-89 ~]# rpm -qa | grep jenkins
jenkins-2.289.3-1.1.noarch
Need to make changes to repos to include asset deployment as a part of beta and prod deployments on the ci
The Jenkins instance that runs CI/CD jobs is setup via CDK. There's no reason why that should not be OSS AFAIK so that anyone can deploy a similar instance.
Work checklist:
Open questions:
Create a proposal on how we plan to configure new jobs on public jenkins.
The recommended approach is to auto-configure jenkins workflow when a jenkins file is added to https://github.com/opensearch-project/opensearch-build/tree/main/jenkins
We want to avoid workflow configuration via UI entirely and look at configuration as a code.
Acceptance Criteria:
@abhinavGupta16 commented on Thu Feb 03 2022
We are able to deploy the ECR related roles and repositories on Isengard using cdk automatically.
The ecr roles set up trust relationships with existing roles/nodes deployed with jenkins-ci
@abhinavGupta16 commented on Fri Feb 11 2022
Following on offline team discussion, we will add the cdk to deploy ECR as a part of opensearch-ci
. Moving therefore, moving this issue to opensearch-ci
Sometimes while deploying the stack, jenkins fails to bootstrap correctly.
I see only failed-boot-attempts.txt
in /var/lib/jenkins and copied yaml nothing else.
The txt file only contains date/time of the failure.
The logs does not show anything relevant:
2022-04-06 23:07:00.303+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2022-04-06 23:07:00.330+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2022-04-06 23:07:00.330+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2022-04-06 23:07:00.331+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2022-04-06 23:07:00.578+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME")
2022-04-06 23:07:00.642+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@3aacf32a{Jenkins v2.263.4,/,file:///var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war}
Its a transient issue. Try deploying the stack using the codebase
Jenkins should deploy without any hiccups.
If applicable, add screenshots to help explain your problem.
No response
No response
No response
Currently we need to manually create a S3 bucket and setup all the access logs, vpc flow logs, and such to it, and use it for access logging. Also need to setup all the required policies.
ci-stack
access-logging.ts
file for access logs changesNo response
No response
Plan how we can integrate the public OpenSearch-CI components into the internal infrastructure to manage deployments automatically.
Acceptance Criteria:
The Jenkins DSL for gradle checks should be visible in this repository. Right now it is not executing from DSL and is not public.
This Should be moved into this repository.
Coming from opensearch-project/project-meta#17
A Developer Certificate of Origin is required on commits in the OpenSearch-Project.
See doc.yml for an example workflow. Ensure CONTRIBUTING.md to has a section on the DCO per the project template.
Currently, OpenId Connect for Jenkins Cluster Setup is a manual process which needs to be automated using Jenkins deployment using cdk
More information on Amazon Internal
No response
No response
Add support to create a role which can log into ecr
and push docker images.
Create cdk support to be able to create the below policy and add trust relationship with the agent node and main node.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr-public:BatchCheckLayerAvailability",
"ecr-public:CompleteLayerUpload",
"ecr-public:InitiateLayerUpload",
"ecr-public:PutImage",
"ecr-public:UploadLayerPart"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ecr-public:GetAuthorizationToken",
"sts:GetServiceBearerToken"
],
"Resource": "*"
}
]
}
No response
No response
When we want to describe a system, make a flow chart, or an activity diagram there should be a descriptive language that is used and can be edited without using another tool.
Inline detection via a GithubAction that converts plantuml into images would be great.
Coming from opensearch-project/opensearch-plugins#92.
Each plugin has a short descriptive text blurb that says what it does: this appears in plugin-descriptor.properties (OpenSearch plugins) or package.json (OpenSearch Dashboards plugins), as well as on the project website's "source" page and in the "About" section of every repo. These were created one-by-one over the years as plugins were created, so looking at them all together now it would be hard for somebody new to the project to use these to understand what the plugins do.
Our system should delete the workspace and anything inside of it to prevent disk buildup. It should be done in such a way that jobs do not need to be concerned about managing the system state, such as by explicitly making a cleanWs
call.
Currently, we need to manually change the jenkins.yaml
file on the server and reload the config, or load the config file via the UI manually. We need a mechanism which detects the changes in jenkins.yaml
and loads the changes on jenkins automatically.
Things to lookout for -
jenkins.yaml
can potentially change the security config. We need to be careful for changes on the file.username:passoword
will not work with oidcCreate a job on jenkins that detects the changes in config file on github and reloads it. We could use jenkins API tokens to run the urls, but they’ll will disappear after config reload.
No response
No response
Monitors to be set up:
For the existing instance we are hardcoding the path https://ci.opensearch.org/ci/dbc
set as CDN_PUBLIC_ARTIFACT_PREFIX
in the jenkins environment. Is this the correct way to set this and how can we make sure this is unique for all deployments?
At Amazon there is a hard requirement to log shell actions, for detection of compromised ec2 instances and store audit history. This software agent is not meant for public distribution and as such should not be made available. The software gets updated monthly it should be up to date on new instances. All of the convenient tooling for this system is not available on GitHub repositories and is part of Amazon internal tooling.
Recap; proprietary software, often updated, no existing bridge to this deployment infrastructure.
By creating a stack within Amazon tooling ecosystem, it is trivial to install the components and receive updates automatically. On a periodic basis AMIs could be created with EC2 Image Builder and then SNS message fired that would trigger this stack to update the configured AMI image.
Pro:
Con:
AWS System Manager can be configured to run scripts against existing hosts and new hosts. By using a State Manager association new instance can have scripts executed against them* there will also be a periodic scan of hosts to ensure they are up to date
Pro:
Con:
Add a cron hook into the long lived nodes to download and run the existing script from S3, add a section to the cloud-init for short-lived instanced. Port over the software into a private repository which we were already going to use to manage actual deployment scripts.
Pro:
Con:
Configuring jenkins is a complex process. Currently, we configure jenkins using the UI which cannot be replicated easily. We need a method to be able to load and change the jenkins configuration through a configuration file.
CDK deployment deploys a configuration file which is used to configure jenkins automatically during deployment.
Since IAM is global every new role name and policy name should be unique.
Having rolename hard coded throws error saying the role already exists and hence cannot create new role.
https://github.com/opensearch-project/opensearch-ci/blob/main/lib/compute/jenkins-main-node.ts#L93
Deploy stack in one region and try deploying the same stack in another region
Stacks should deploy in multiple regions
If applicable, add screenshots to help explain your problem.
No response
No response
No response
Load configuration via code using Jenkins Configuration as Code plugin that uses jenkins.yaml to load config. We need load the entire config via jenkins.yaml. This means we need to move current config changes made to config.xml to jenkins.yaml
Originally posted by @gaiksaya in #85 (comment)
Currently, there is no mechanism to verify the json format and checking required fields for JSON put in the secret manager for OIDC. If there is something missing, the deployment will fail and there won't be descriptive error messages helping to debug the issue.
Write a mechanism to verify the JSON coming in from the secret manager for the OIDC credential and fail with descriptive error messages if any field is missing.
No response
No response
Create a basic CDK project for additional contributions.
When we destroy the cdk stack, we are currently required to specify the flag parameters. We should be able to destroy the stack without specifying the parameters.
create a CI-Dev
stack and destroy is using cdk destroy CI-Dev
We should be able to destroy the stack without specifying the parameter flags
If applicable, add screenshots to help explain your problem.
No response
No response
No response
Coming from opensearch-project/project-meta#17
A Developer Certificate of Origin is required on commits in the OpenSearch-Project.
See doc.yml for an example workflow. Ensure CONTRIBUTING.md to has a section on the DCO per the project template.
We have been living with the start gradle check
passphrase for a while now and it is grating on our team of admins.
Lets disabling this feature for a trial period of 1 week and have a way to determine if this is something we should do long term.
Beforehand we need to make sure we have the correct monitoring in place to make sure that our CI fleet is health with the reduced human intervention.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.