devstream-io / devstream Goto Github PK
View Code? Open in Web Editor NEWDevStream: the open-source DevOps toolchain manager (DTM).
Home Page: https://www.devstream.io
License: Apache License 2.0
DevStream: the open-source DevOps toolchain manager (DTM).
Home Page: https://www.devstream.io
License: Apache License 2.0
We should move the compile/build process to a container to ensure that everyone can get consistent products. Local compile/build often introduces problems due to inconsistent environment, such as inconsistent golang versions.
dtm install -f config.yaml --force
Add CLA (Contributor License Agreement) for DevStream
"argocdapp" will be triggered twice:
Steps to reproduce the bug:
A clear and concise description of what you expected to happen.
If applicable, add screenshots to help explain the issue.
Add any other helpful context about the issue here.
Got unexpected EOF error when using dsm to install argocd.
---
tools:
- name: argocd
version: 0.0.1
options:
repo:
name: argo
url: https://argoproj.github.io/argo-helm
chart:
name: argo/argo-cd
release_name: argocd
namespace: argocd
create_namespace: False
./dsm install -f /tmp/config.yaml
Currently, the github actions plugin will create two workflows for Golang, one is for pull request, the other is for the master/main branch.
Maybe we can simplify it, only create one workflow, for example, like this one:
https://github.com/actions/starter-workflows/blob/a85155b04a1b885de7a5021312303fe5c1cb58e4/ci/go.yml
For the main branch, set the number of approving reviews to 2.
Similar to the github actions plugin, we might need one for GitLab CI.
At the moment, we have three plugins:
We need one document for each plugin.
Maybe creating a subfolder named plugins
under docs
, then one document for each plugin?
Each doc should contain a sample config, what parameters are mandatory, sample values, etc.
Making sure the name and architecture match the implementation.
add a progress bar to the process of downloading the plugin
// TODO
/kind proposal
/assign @daniel-hutao I'm very interested in this feature.
At the moment, the workflows for github actions plugin are hard-coded and stored in variables.
Evaluate the possibility to download (http, git api, etc) from here instead of storing everything as a variable.
If downloading is possible, we need to handle retries/exponential backoff, so we might need to dig into some open-source go modules for these capabilities.
Only research is OK, maybe a small MVP, no need to implement right away.
I think that we could create a new plugin
package to store the code with all the built-in plugins and the pluginengine.
The pkg tree now is like below:
internal/pkg/
argocd
argocdapp
backend
configloader
githubactions
planmanager
pluginengine
pluginmanager
statemanager
version
The packages with the build-in plugins will be more and more. Maybe putting them under one package will make it clearer. Just like below:
internal/pkg/
plugin/
argocd
argocdapp
githubactions
backend
configloader
planmanager
pluginengine
pluginmanager
statemanager
version
/kind proposal
/cc @merico-dev/devstream
Maybe we need to change the install command to create/apply because the uninstall operation can be triggered during the execution of the install command, so the install command is confusing.
In addition, the delete operation may require more consideration. If the user accidentally specifies an empty configuration file, after executing the dtm
command, all installed components will be uninstalled, which may not be what they want. We may need to provide a separate command for the uninstall action, such as the delete
command, to let users know the real result of their command action.
I consider solving the first problem mentioned above first - rename the install command.
Then modify the planmanager
so that create/apply
does not trigger the uninstall action.
Finally, add a separate uninstall/delete
command entrance.
/kind proposal
/assign @daniel-hutao
/cc @merico-dev/devstream
see the discussion below for more information.
https://github.com/merico-dev/stream/pull/46#discussion_r773565726_
At the moment, the github actions plugin only can create workflows for Golang. We need to be able to support Python and Nodejs (we use them ourselves, so these two have higher priority than other programming languages).
For Python, maybe we can create some simple workflow like this one:
https://github.com/actions/starter-workflows/blob/a85155b04a1b885de7a5021312303fe5c1cb58e4/ci/python-app.yml
More research in github actions starter workflows is welcome.
For Node, it's a bit more complicated because some use node alone, some use it with webpack, some use it with gulp, etc.
Examples:
At the moment, the Reinstall
and Uninstall
are mocks.
We want to implement them.
Reinstall: kubectl apply again
Uninstall: kubectl delete crd xxx
Make the makefile aware of version changes.
The version at the Makefile now is Hardcored:
// ...
build: fmt vet ## Build dtm & plugins locally.
go mod tidy
go build -buildmode=plugin -trimpath -gcflags="all=-N -l" -o plugins/githubactions_0.0.1.so ./cmd/githubactions/
go build -buildmode=plugin -trimpath -gcflags="all=-N -l" -o plugins/argocd_0.0.1.so ./cmd/argocd/
go build -buildmode=plugin -trimpath -gcflags="all=-N -l" -o plugins/argocdapp_0.0.1.so ./cmd/argocdapp/
go build -trimpath -gcflags="all=-N -l" -o dtm ./cmd/devstream/
// ...
sed
command in the first line at the build stage
(make build) to render this variable into the version.go file./cc @merico-dev/devstream Do you think this idea is suitable?
Failed to uninstall argocdapp.
These resources should be cleaned up by the operator
, but at this time the operator has been deleted.
It should be solved after we have the dependency manager.
~/MyGolangProjects/stream/demo main ./dtm install -f ./empty.yaml
2021/12/29 22:31:26 succeeded to initialize States
2021/12/29 22:31:26 added a change: argocdapp -> uninstall
2021/12/29 22:31:26 added a change: githubactions -> uninstall
2021/12/29 22:31:26 added a change: argocd -> uninstall
2021/12/29 22:31:26 changes count: 3
2021/12/29 22:31:26 processing progress: 1/3
2021/12/29 22:31:26 processing: argocdapp -> uninstall
2021/12/29 22:31:26 application.argoproj.io "hello" deleted
2021/12/29 22:31:26 processing progress: 2/3
2021/12/29 22:31:26 processing: githubactions -> uninstall
2021/12/29 22:31:26 language is go-1.17
2021/12/29 22:31:30 deleting github actions Workflow pr-builder.yml...
2021/12/29 22:31:31 github actions Workflow pr-builder.yml removed
2021/12/29 22:31:31 deleting github actions Workflow master-builder.yml...
2021/12/29 22:31:32 github actions Workflow master-builder.yml removed
2021/12/29 22:31:32 processing progress: 3/3
2021/12/29 22:31:32 processing: argocd -> uninstall
2021/12/29 22:31:32 uninstalling argocd helm chart
2021/12/29 22:31:32 uninstall: Deleting argocd
2021/12/29 22:31:34 Starting delete for "argocd-redis" Service
2021/12/29 22:31:34 Starting delete for "argocd-repo-server" Service
2021/12/29 22:31:34 Starting delete for "argocd-application-controller" Service
2021/12/29 22:31:34 Starting delete for "argocd-server" Service
2021/12/29 22:31:34 Starting delete for "argocd-dex-server" Service
2021/12/29 22:31:34 Starting delete for "argocd-redis" Deployment
2021/12/29 22:31:34 Starting delete for "argocd-application-controller" Deployment
2021/12/29 22:31:34 Starting delete for "argocd-server" Deployment
2021/12/29 22:31:34 Starting delete for "argocd-dex-server" Deployment
2021/12/29 22:31:34 Starting delete for "argocd-repo-server" Deployment
2021/12/29 22:31:34 Starting delete for "argocd-dex-server" RoleBinding
2021/12/29 22:31:34 Starting delete for "argocd-application-controller" RoleBinding
2021/12/29 22:31:34 Starting delete for "argocd-server" RoleBinding
2021/12/29 22:31:34 Starting delete for "argocd-dex-server" Role
2021/12/29 22:31:34 Starting delete for "argocd-server" Role
2021/12/29 22:31:34 Starting delete for "argocd-application-controller" Role
2021/12/29 22:31:34 Starting delete for "argocd-server" ClusterRoleBinding
2021/12/29 22:31:34 Starting delete for "argocd-application-controller" ClusterRoleBinding
2021/12/29 22:31:34 Starting delete for "argocd-application-controller" ClusterRole
2021/12/29 22:31:34 Starting delete for "argocd-server" ClusterRole
2021/12/29 22:31:34 Starting delete for "argocd-tls-certs-cm" ConfigMap
2021/12/29 22:31:34 Starting delete for "argocd-cm" ConfigMap
2021/12/29 22:31:34 Starting delete for "argocd-gpg-keys-cm" ConfigMap
2021/12/29 22:31:34 Starting delete for "argocd-rbac-cm" ConfigMap
2021/12/29 22:31:34 Starting delete for "argocd-ssh-known-hosts-cm" ConfigMap
2021/12/29 22:31:34 Starting delete for "argocd-secret" Secret
2021/12/29 22:31:34 Starting delete for "argocd-dex-server" ServiceAccount
2021/12/29 22:31:34 Starting delete for "argocd-application-controller" ServiceAccount
2021/12/29 22:31:34 Starting delete for "argocd-server" ServiceAccount
2021/12/29 22:31:34 purge requested for argocd
2021/12/29 22:31:34 release uninstalled, response: &{0x1400093c000 }
2021/12/29 22:31:34 === all plugins' Install/Uninstall/Reinstall process are succeeded ===
2021/12/29 22:31:34 === END ===
~/MyGolangProjects/stream/demo main kgpo
NAME READY STATUS RESTARTS AGE
hello-75f9fddff6-l6kjj 1/1 Running 0 6h19m
/kind bug
Maybe it could be the same as the argocd plugin and we can implement a helm install for Jenkins in K8s.
Maybe we could reuse some part of the code (helm install), in which case, a new go module is appreciated.
statemanager delete a State when uninstall failed.
state file hold the State with uninstall failed
sync.Map is like a Go map[interface{}]interface{}
but is safe for concurrent use by multiple goroutines without additional locking or coordination.
sync.Map is more reliable and efficient than default build-in Map + Mutex.
ConcurrentMap
type to package sync.Map & keyType & valueTypeStates
to StatesMap
, States & State are a bit confusingConcurrentMap
with keyType string
& valueType *State
to redefine StatesMap
We need to provide a mechanism to support users to use in an air-gapped or a bad network environment.
Maybe we could provider a shell script like offline-tool.sh
& text file like resource.txt
. These 2 file could:
resources.txt
maintain all resources dsm needed when deploy a new env. Users can customize the content of this file to decide that they need to download all or part of the resources.offline-tool.sh
can download all resources listed in resources.txt
file structure may like below:
stream
print some logs when dtm init -f xxx.yaml
with no error
For users behind the GFW without a VPN, it's not possible to build or run using the default/official registry sources for Docker images, go modules, helm charts, and so on.
Add documentation about how to set:
goproxy.cn
)so that users behind the GFW can work and use DevStream without any problem.
Extra CLI params might be needed.
Any change should not affect the current default behavior for non-GFW-blocked users.
Related to #36 which also needs to be implemented.
The Makefile now is:
fmt: ## Run go fmt against code.
go get golang.org/x/tools/cmd/goimports
goimports -local="github.com/merico-dev/stream" -d -w cmd
goimports -local="github.com/merico-dev/stream" -d -w internal
go fmt ./...
Maybe we can re-choose a base image containing goimports
or customize our own base image with more tools.
Adding a "action calculation" feature to determine the next action based on the current States of the environment and the input Config.
When the state management function is integrated into the main branch #28 , we need to use the States in the next step, that is, an algorithm is needed to determine the specific actions each time the dsm is executed.
The general idea is as follows:
type ActionFunc func(tool *config.Tool) (bool, error)
type Plan struct {
Changes []*Change
}
type Change struct {
Tool *config.Tool
Action ActionFunc
}
func MakePlan(smgr statemanager.Manager, cfg config.Config) *Plan {...}
// ......
func TestMakePlan() {
// ......
plan := MakePlan(smgr, cfg)
for _, change := range plan.Changes {
change.Action(change.Tool)
}
}
githubactions supports multi-repo
As it's been discussed at https://github.com/merico-dev/stream/pull/20#discussion_r757267479, we need to support cross-platform build, but it's not very urgent.
We could support linux-amd64 darwin-amd64 windowns-amd64 first, then support other arch like "arm i386, mips64le, ppc64le, s390x ...
What are you trying to do, , how do you want to do it, how would you do it differently?
Is it something you currently cannot do? Is this related to an issue?
A clear and concise description of what you would like.
Please link related issues here, if any.
A clear and concise description of any alternative solution or feature you've considered.
Add any other helpful context or screenshots about the feature request here.
Add the right amount of unit tests, so that when the CI build passes, we are certain that the quality is good enough to be merged into main.
A local state file as the first implementation.
Remembering which plugins are already executed, which are not, which are executed but failed.
When re-run, only execute those failed/not executed ones.
For example, we have installed the v1 version of argocd, and the configuration file is changed to v2 now. Planmanager needs to be able to recognize this change and give a corresponding upgrade plan
// TODO
/kind proposal
invalid memory address or nil pointer dereference at getFileSHA()
2021/12/28 13:18:02 language is go-1.17
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x12bbfc7e0]
goroutine 1 [running]:
github.com/merico-dev/stream/internal/pkg/githubactions.(*GithubActions).getFileSHA(0x1400006a7e0, {0x12bc03c3c, 0xe})
github.com/merico-dev/stream/internal/pkg/githubactions/githubactions.go:125 +0x1a0
github.com/merico-dev/stream/internal/pkg/githubactions.(*GithubActions).DeleteWorkflow(0x1400006a7e0, 0x140001c65d0)
github.com/merico-dev/stream/internal/pkg/githubactions/githubactions.go:80 +0x60
github.com/merico-dev/stream/internal/pkg/githubactions.Uninstall(0x14000113c10)
github.com/merico-dev/stream/internal/pkg/githubactions/uninstall.go:17 +0x218
github.com/merico-dev/stream/cmd/githubactions.Plugin.Uninstall({0x0, 0x0}, 0x14000113c10)
github.com/merico-dev/stream/cmd/githubactions/main.go:27 +0x34
github.com/merico-dev/stream/internal/pkg/pluginengine.Uninstall(0x14000113bf0)
github.com/merico-dev/stream/internal/pkg/pluginengine/plugin.go:42 +0xd4
github.com/merico-dev/stream/internal/pkg/planmanager.(*Plan).Execute(0x14000113b60)
github.com/merico-dev/stream/internal/pkg/planmanager/plan.go:61 +0x3b0
main.installCMDFunc(0x10348b980, {0x1400006a3e0, 0x0, 0x2})
github.com/merico-dev/stream/cmd/devstream/install.go:45 +0x318
github.com/spf13/cobra.(*Command).execute(0x10348b980, {0x1400006a3c0, 0x2, 0x2})
github.com/spf13/[email protected]/command.go:860 +0xa04
github.com/spf13/cobra.(*Command).ExecuteC(0x10348bc00)
github.com/spf13/[email protected]/command.go:974 +0x7b8
github.com/spf13/cobra.(*Command).Execute(0x10348bc00)
github.com/spf13/[email protected]/command.go:902 +0x30
main.main()
github.com/merico-dev/stream/cmd/devstream/main.go:28 +0x30
I've fixed this bug at #74 but it's reopen at #76 by "resolve conflict".
This is not a criticism of @lfbdev
Just remind us all that we should pay more attention at "avoid confict"/"resolve conflict"/"code review"
/kind bug
/cc @merico-dev/devstream
argocdapp installation failed with no error log output
2021/12/29 11:22:31 release installed successfully: argocd/argo-cd-3.29.4
2021/12/29 11:22:31 === plugin argocd process done ===
2021/12/29 11:22:31 processing progress: 3/3
2021/12/29 11:22:31 processing: argocdapp -> install
2021/12/29 11:22:32 === plugin argocdapp process failed ===
2021/12/29 11:22:32 === some errors occurred during plugins Install/Uninstall/Reinstall process ===
2021/12/29 11:22:32 exit status 1
2021/12/29 11:22:32 === END ===
/kind bug
At the moment, the plugins must be placed in a specific directory and must be compiled/built by users before using the CLI tool.
We can build the CLI and the plugins and publish them into GitHub releases.
For more information, check out this discussion here:
https://github.com/merico-dev/stream/discussions/26
init
Command to dsm
Add some commands like dsm init -f config.yaml
, which:
.dsm/plugins/
) then download the needed plugins with the correct version from GitHub releases public URLThe idea is simple: only download when necessary.
Maybe I'm missing something here, please have a look @lfbdev @daniel-hutao
There could be a possibility to improve: md5sum before download to ensure the local plugin is the desired one (no need to implement now).
If dsm init
isn't executed, but dsm install
is executed, either a hint should be printed out to let the user to init it first, or the init will be called automatically before install is called.
Add UT for the init/download part, especially the download logic part because it will be changed probably in the future.
For documentation, please write a Chinese version (no need to be put in the repo) and I will translate it into English.
check md5sum of existing local plugins, then decide if a download is needed or not;
if a local plugin exists, but the version differs, delete local, then download the correct version.
Some logs like below:
now downloading plugin: githubactions_0.0.1-rc1.so ,version: 0.0.1-rc1
Try to make it more beautiful.
/kind bug
/assign @lfbdev
At the moment, in the argocdapp plugin, we create the ArgoCD app CRD by using kubectl apply -f
, which is not ideal.
Do some research and see if it's possible to create the CRD by using ArgoCD Golang SDK/API.
Only research (and maybe a test-drive is needed, no need to implement immediately)
Current situation:
cmd/devstream/install.go
; the internal/pkg/pluginengine
doesn't do much except load the so and execute the corresponding interfacestream/internal/pkg/planmanager/result.go
which doesn't seem right according to the namecmd/devstream
package and into the internal/pkg/pluginengine
packageTake argocdapp as an example. If this plugin has been uninstalled, the uninstall operation will fail at this time.
Handle the error like below
Error from server (NotFound): error when deleting "./app.yaml": applications.argoproj.io "hello" not found
/kind proposal
CLI rename from dsm
to dtm
Packages and directories renaming:
download
to pluginmanager
or similar (for example plugindownloader
, so that it's consistent with statemanager
(functions-verb-er);
plugin
to pluginengine
config
to configloader
plan
to planmanager
@lfbdev @daniel-hutao, please think about it and add more to the list. We can create a test plan for each plugin in the future based on the list.
What should happen when install is called without calling init first?
Shall we run the init anyway for the user, or only a warning should be given and let the user take control of the whole process?
We can refer to other open-source tools, like Terraform init/apply.
At the moment, the Reinstall
and Uninstall
are mocks.
We want to implement them.
Reinstall: helm upgrade --install
Uninstall: helm delete
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.