kubedl-io / kubedl Goto Github PK
View Code? Open in Web Editor NEWRun your deep learning workloads on Kubernetes more easily and efficiently.
Home Page: https://kubedl.io/
License: Apache License 2.0
Run your deep learning workloads on Kubernetes more easily and efficiently.
Home Page: https://kubedl.io/
License: Apache License 2.0
for KubeDL users, they can directly manipulate jobs via kubectl
or arena
in terminal, however, it has learning curve for kubernetes beginners, if possible dashboard can be a more user-kindly way to manage and manipulate jobs.
For example, user may want to apply different docker registry prefix of kubedl image, it varies from different deploy regions and should be configurable, is it considerable to integrate with helm chars ?
setup training jobs in cron style (e.g. linux cron-tab expression) and trigger scheduling jobs regularly is a common practice, KubeDL
should design a special workload to control cron training jobs.
Model lineage and versioning to track the history of a model natively in CRD: when the model is trained using which data and which image, each version of the model, which version is runnng etc.
remove the CMD ["/backend-server"] in the main dockerfile
some implementation in service.go and pod.go can be shared among the job types
https://bestpractices.coreinfrastructure.org/en
For CNCF requirements
https://github.com/alibaba/kubedl/blob/master/config/crd/bases/kubeflow.org_testjobs.yaml
I think the test job CRD does not need to exist in the installation folder?
pod/job eviction status may should be aggregated and shown on JobStatus, as well as metrics reporting.
The purpose is to document what metrics exists, and what does the metrics mean.
There's no point in documenting how to add a metrics
https://github.com/alibaba/kubedl/blob/cd837b271037a9a11afeab75b3a0dca31465d5fc/docs/metrics.md#L10
When installing the kubedl, I want to change the value of the max-reconciles. However, I do not find the corresponding value in the values.yaml. Should I try to use the --set
flag or to change the source code?
In the previous ControllerInterface
design, the UpdateJobStatus
method is as follows:
// UpdateJobStatus updates the job status and job conditions
UpdateJobStatus(job interface{}, replicas map[ReplicaType]*ReplicaSpec, jobStatus *JobStatus) error
when its comes to decide whether a job should restart or not, we check the pod exit code and phase,in tf-operator implementation restart, a restart
bool is used and can be passed to updateSingleJobStatus
, but now limited by interface design, we can only determine whether to restart by simply compare restartpolicy, which may cause a bug. In another way we can recompute the restart
flag by iterating all pods again, but it quite inefficient.
so I propose to change the UpdateJobStatus
signature to:
UpdateJobStatus(job interface{}, replicas map[ReplicaType]*ReplicaSpec, jobStatus *JobStatus, restart bool) error`
and the restarting mechanism is required by all workload controllers.
It's seems in v1.16+ k8s cluster, KubeDL can not reconcile jobs successfully, and the log shows Pods
and Services
can be created, but when update job status to api-server, reflected in code as follows:
r.Status().Update(context.Background(), jobCpy)
it always throw and error says can not found xxx job
. But in normal v1.14 k8s cluster it works fine. This issue reproduces in AliCloud v1.16.2 cluster and minikube v1.17.0 cluster, and the root cause may be in v1.16+ k8s, CRD has moved into GA state, and its CRD-related components has some improvements.
ref: https://book-v1.book.kubebuilder.io/basics/status_subresource.html
In these controller-runtime
based operator implementations, the work queue is well encapsulated and be transparent to operator developers. But in some scenarios, we still need some informations exposed by work queue, e.g:
we rely the NumRequeues(jobKey)
method of work queue to retrieve the previous number of retries of this job, and determine whether retry times exceed the backoff limit.
However, we can't get this information because that work queue is encapsulated inside controller-runtime. I've tried to open an issue kubernetes-sigs/controller-runtime#686 to report this problem, but it seems still not elegant.
so I propose to add a new filed named BackoffCounts
in common JobStatus
definition, to record backoff times directly in status:
type JobStatus struct {
// ...
// BackffCounts record retry times for this job.
BackoffCounts int32
// ...
}
for now, we create a FakeQueue
to adapt the JobController, it actually does nothing... if the backoff info can be exposed by this new field, the FakeQueue can be removed.
xdljobs 每个work 代码运行日志打印不是每个step打印一次,而是运行完成后一次性返回所有的打印日志,有人碰到过这是什么情况吗
For example, if I mistype Worker
to worker
when submit a TFJob, it can not be successfully reconciled, so it is more user-friendly to ignore the case of replica types.
KubeDL expects to integrate natively with the training and serving CRD to enable the full automation with controller, and support serving frameworks like TFServing
and Nvidia Triton
.
What would you like to be added:
Run the docker image build for kubedl daily
Why is this needed:
the number of workers in each controller are set to 1 defaulted, but not configurable.
follow prometheus standard metrics namings https://prometheus.io/docs/practices/naming/#metric-names
such as:
kubedl_jobs_created_total
The UT doesn't pass
What happened:
kubedl manager startup fails like below
I0723 23:28:46.174606 1 controllers.go:42] workload Inference controller has started.
2021-07-23T23:28:46.174Z INFO setup setting up storage backends
F0723 23:28:46.174685 1 client.go:32] get clientMgr fail, clientMgr is nil
What you expected to happen:
How to reproduce it:
Anything else we need to know?:
Environment:
kubectl version
):cat /etc/os-release
):uname -a
):the sig-scheduling community implement gang-scheduling semantic by extending scheduling-framework, which is called coscheduling
and code is maintained @ https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling
KubeDL can implement a gang-scheduling plugin based on coscheduling
.
KubeDL auto tuning for inference tries to figure out the best configurations for inference container with respect to its container resources or running parameteres .
Here we can return the default image, instead of return empty image
func (h *KubeDLHandler) GetImageConfig() *ImageConfig {
cm := &v1.ConfigMap{}
err := h.client.Get(context.Background(), types.NamespacedName{
Namespace: constants.KubeDLSystemNamespace,
Name: constants.KubeDLConsoleConfig,
}, cm)
if err != nil {
glog.Infof("get image config error: %v", err)
return &ImageConfig{}
}
Hello kubeDL team,
I'm one of the co-chairs of the CNCF TAG-Runtime. I'm reaching out and I think it would be great for you to present/discuss the project at one of our meetings. For example, a general overview of the project or/and a demo.
Feel free to add it to our agenda or reach out to me (raravena80 at gmail.com)
Thanks!
A security disclosure process would be nice per CNCF requirements!
Create a dockerfile for the fornt and backend server container
As we all know, volcano
scheduler is developed based on kube-batch v1alpha2, and currently it's recommended by community instead of kube-batch, volcano is being actively contributed while kube-batch has not been updated for quite a long time.
And a main motivation of upgradation is that: dependencies of kube-batch stays on k8s v1.13 and it's been stale, and conflicts with other k8s dependencies in higher version, volcano is free from these trouble.
When running the manager:
go run ./main.go -enable-leader-election=true
I met this error log:
2020-10-12T13:12:47.578+0800 ERROR setup unable to start manager {"error": "LeaderElectionID must be configured"}
github.com/go-logr/zapr.(*zapLogger).Error
/Users/qijun/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
main.main
/Users/qijun/works/kubedl/main.go:77
runtime.main
/usr/local/Cellar/go/1.14.5/libexec/src/runtime/proc.go:203
exit status 1
We found an data-consistency anomaly between local informer cache
and etcd
in case of etcd slow watch
, we can reproduce this in a kubernetes cluster with unstable networking, for example:
job-1
failed;job-1
as Failed
and apiserver response with success;Running
, which is unexpected;the reason is that READ requests(get/list resources) hits local informer cache
by default, but the cached object will only be updated after update event broadcasted from etcd, however, if slow watch happened and event broadcasting delayed, the local cache will be stale.
Controller Local-Informer ApiServer
failed running running
1. |--------------------------------------> failed
2. | <-------------------> running |
3. | failed <-------- |
4. | <------------------> failed |
@SimonCqk can you add a documentation for the cron controller
support using custom kaniko image with configmap
The example file doesn't exist in the repo, make sure the example job works in a mini cluster
currently, the operator wakes up periodically to check sum all the jobs running, this is inefficient.
/Users/qiukaichen/go/bin/controller-gen "crd" rbac:roleName=manager-role webhook paths="./apis/..." paths="./controllers/..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/xdljobs.xdl.kubedl.io created
customresourcedefinition.apiextensions.k8s.io/xgboostjobs.xgboostjob.kubeflow.org created
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "pytorchjobs.kubeflow.org" is invalid: spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "tfjobs.kubeflow.org" is invalid: spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified
make: *** [install] Error 1
Dear, all
When I used kubedl in managing distributed pytorch jobs, I found that sometimes the output data of a pytorch job cannot be fully saved to the storage system, however, the pytorch job will still claim itself as successful.
To understand that, I check the detailed logs of each worker.
Logs show that after the master role of the pytorch job gets completed, the whole pytorch job becomes terminated, regardless of the running pods of the other pytorch workers. I think it should be the reason for the incomplete output data.
It seems to be a bug in my view. Could you please take a check?
Thanks,
Wencong
In the practice of our prod environment, we found a portion of scenarios relies on scheduling job replicas in stages, otherwise there will be severe exceptions, for example:
Worker
s step into Running
phase before Master
(Master
pod may hanging on pulling images or some other reasons), it will crash immediately because Worker
cannot ping Master
successfully, then Job goes to Failed
.Launcher
step into Running
phase before Worker
s (Worker
pod may hanging on pulling images), Launcher
will exit unexpected because kubectl exec
command can not reach target container in each Worker
, similarly, Job finally failed.in addition, DAG scheduling can also improve efficiency in certain scenarios, for example: schedule PS
before Worker
for TFJob
to reduce the duration of worker-stall;
Would like to join the group, it'll be appreciated if you could update the qr code.
for backward compatibility with kubeflow.org
api group, KubeDL implements enhanced features (e.g. hostnetwork, tensorboard...) through annotations
, now we'd replenish these features to job.spec
which can be more native.
and it's better KubeDL can detect which workload CRDs are installed into cluster, then enable its corresponding controller automatically.
https://github.com/alibaba/kubedl/blob/f0062f116f14a402ebc6568e8039512284855efe/go.mod#L37
CVE-2020-8565 CVE-2020-8563 CVE-2020-8566 CVE-2020-8564
Recommended upgrade version:v1.18.15
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.