kubedl-io / kubedl Goto Github PK

for KubeDL users, they can directly manipulate jobs via kubectl or arena in terminal, however, it has learning curve for kubernetes beginners, if possible dashboard can be a more user-kindly way to manage and manipulate jobs.

For example, user may want to apply different docker registry prefix of kubedl image, it varies from different deploy regions and should be configurable, is it considerable to integrate with helm chars ?

setup training jobs in cron style (e.g. linux cron-tab expression) and trigger scheduling jobs regularly is a common practice, KubeDL should design a special workload to control cron training jobs.

Model lineage and versioning to track the history of a model natively in CRD: when the model is trained using which data and which image, each version of the model, which version is runnng etc.

remove the CMD ["/backend-server"] in the main dockerfile

some implementation in service.go and pod.go can be shared among the job types

https://bestpractices.coreinfrastructure.org/en

For CNCF requirements

https://github.com/alibaba/kubedl/blob/master/config/crd/bases/kubeflow.org_testjobs.yaml

I think the test job CRD does not need to exist in the installation folder?

pod/job eviction status may should be aggregated and shown on JobStatus, as well as metrics reporting.

The purpose is to document what metrics exists, and what does the metrics mean.

There's no point in documenting how to add a metrics

https://github.com/alibaba/kubedl/blob/cd837b271037a9a11afeab75b3a0dca31465d5fc/docs/metrics.md#L10

When installing the kubedl, I want to change the value of the max-reconciles. However, I do not find the corresponding value in the values.yaml. Should I try to use the --set flag or to change the source code?

In the previous ControllerInterface design, the UpdateJobStatus method is as follows:

// UpdateJobStatus updates the job status and job conditions
UpdateJobStatus(job interface{}, replicas map[ReplicaType]*ReplicaSpec, jobStatus *JobStatus) error

when its comes to decide whether a job should restart or not, we check the pod exit code and phase，in tf-operator implementation restart, a restart bool is used and can be passed to updateSingleJobStatus, but now limited by interface design, we can only determine whether to restart by simply compare restartpolicy, which may cause a bug. In another way we can recompute the restart flag by iterating all pods again, but it quite inefficient.

so I propose to change the UpdateJobStatus signature to:

UpdateJobStatus(job interface{}, replicas map[ReplicaType]*ReplicaSpec, jobStatus *JobStatus, restart bool) error`

and the restarting mechanism is required by all workload controllers.

It's seems in v1.16+ k8s cluster, KubeDL can not reconcile jobs successfully, and the log shows Pods and Services can be created, but when update job status to api-server, reflected in code as follows:

r.Status().Update(context.Background(), jobCpy)

it always throw and error says can not found xxx job. But in normal v1.14 k8s cluster it works fine. This issue reproduces in AliCloud v1.16.2 cluster and minikube v1.17.0 cluster, and the root cause may be in v1.16+ k8s, CRD has moved into GA state, and its CRD-related components has some improvements.

ref: https://book-v1.book.kubebuilder.io/basics/status_subresource.html

In these controller-runtime based operator implementations, the work queue is well encapsulated and be transparent to operator developers. But in some scenarios, we still need some informations exposed by work queue, e.g:
we rely the NumRequeues(jobKey) method of work queue to retrieve the previous number of retries of this job, and determine whether retry times exceed the backoff limit.

However, we can't get this information because that work queue is encapsulated inside controller-runtime. I've tried to open an issue kubernetes-sigs/controller-runtime#686 to report this problem, but it seems still not elegant.

so I propose to add a new filed named BackoffCounts in common JobStatus definition, to record backoff times directly in status:

type JobStatus struct {
    // ...
    
    // BackffCounts record retry times for this job.
    BackoffCounts    int32

    // ...
}

for now, we create a FakeQueue to adapt the JobController, it actually does nothing... if the backoff info can be exposed by this new field, the FakeQueue can be removed.

xdljobs 每个work 代码运行日志打印不是每个step打印一次，而是运行完成后一次性返回所有的打印日志，有人碰到过这是什么情况吗

For example, if I mistype Worker to worker when submit a TFJob, it can not be successfully reconciled, so it is more user-friendly to ignore the case of replica types.

KubeDL expects to integrate natively with the training and serving CRD to enable the full automation with controller, and support serving frameworks like TFServing and Nvidia Triton.

What would you like to be added:
Run the docker image build for kubedl daily

Why is this needed:

the number of workers in each controller are set to 1 defaulted, but not configurable.

https://github.com/alibaba/kubedl/blob/cd837b271037a9a11afeab75b3a0dca31465d5fc/pkg/metrics/counter.go#L40

follow prometheus standard metrics namings https://prometheus.io/docs/practices/naming/#metric-names

such as:

kubedl_jobs_created_total

The UT doesn't pass

What happened:
kubedl manager startup fails like below

I0723 23:28:46.174606 1 controllers.go:42] workload Inference controller has started.
2021-07-23T23:28:46.174Z INFO setup setting up storage backends
F0723 23:28:46.174685 1 client.go:32] get clientMgr fail, clientMgr is nil

What you expected to happen:

How to reproduce it:

Anything else we need to know?:

Environment:

• KubeDL version:
• Kubernetes version (use kubectl version):
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

the sig-scheduling community implement gang-scheduling semantic by extending scheduling-framework, which is called coscheduling and code is maintained @ https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling

KubeDL can implement a gang-scheduling plugin based on coscheduling.

KubeDL auto tuning for inference tries to figure out the best configurations for inference container with respect to its container resources or running parameteres .

Here we can return the default image, instead of return empty image

func (h *KubeDLHandler) GetImageConfig() *ImageConfig {
	cm := &v1.ConfigMap{}
	err := h.client.Get(context.Background(), types.NamespacedName{
		Namespace: constants.KubeDLSystemNamespace,
		Name:      constants.KubeDLConsoleConfig,
	}, cm)
	if err != nil {
		glog.Infof("get image config error: %v", err)
		return &ImageConfig{}
	}

Hello kubeDL team,

I'm one of the co-chairs of the CNCF TAG-Runtime. I'm reaching out and I think it would be great for you to present/discuss the project at one of our meetings. For example, a general overview of the project or/and a demo.

Feel free to add it to our agenda or reach out to me (raravena80 at gmail.com)

Thanks!

A security disclosure process would be nice per CNCF requirements!

Create a dockerfile for the fornt and backend server container

As we all know, volcano scheduler is developed based on kube-batch v1alpha2, and currently it's recommended by community instead of kube-batch, volcano is being actively contributed while kube-batch has not been updated for quite a long time.
And a main motivation of upgradation is that: dependencies of kube-batch stays on k8s v1.13 and it's been stale, and conflicts with other k8s dependencies in higher version, volcano is free from these trouble.

When running the manager:

go run ./main.go -enable-leader-election=true

I met this error log:

2020-10-12T13:12:47.578+0800	ERROR	setup	unable to start manager	{"error": "LeaderElectionID must be configured"}
github.com/go-logr/zapr.(*zapLogger).Error
	/Users/qijun/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
main.main
	/Users/qijun/works/kubedl/main.go:77
runtime.main
	/usr/local/Cellar/go/1.14.5/libexec/src/runtime/proc.go:203
exit status 1

We found an data-consistency anomaly between local informer cache and etcd in case of etcd slow watch, we can reproduce this in a kubernetes cluster with unstable networking, for example:

1. one master pod of pytorchjob job-1 failed;
2. pytorch controller update job-1 as Failed and apiserver response with success;
3. another reconciling starts but latest fetched job status is still Running, which is unexpected;

the reason is that READ requests(get/list resources) hits local informer cache by default, but the cached object will only be updated after update event broadcasted from etcd, however, if slow watch happened and event broadcasting delayed, the local cache will be stale.

   Controller               Local-Informer      ApiServer
     failed                    running           running
      
1.   |-------------------------------------->   failed
2.   | <------------------->  running            |
3.   |                        failed  <--------  |
4.   | <------------------>   failed             |

@SimonCqk can you add a documentation for the cron controller

support using custom kaniko image with configmap

The example file doesn't exist in the repo, make sure the example job works in a mini cluster

currently, the operator wakes up periodically to check sum all the jobs running, this is inefficient.

/Users/qiukaichen/go/bin/controller-gen "crd" rbac:roleName=manager-role webhook paths="./apis/..." paths="./controllers/..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/xdljobs.xdl.kubedl.io created
customresourcedefinition.apiextensions.k8s.io/xgboostjobs.xgboostjob.kubeflow.org created
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "pytorchjobs.kubeflow.org" is invalid: spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "tfjobs.kubeflow.org" is invalid: spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified
make: *** [install] Error 1

Dear, all

When I used kubedl in managing distributed pytorch jobs, I found that sometimes the output data of a pytorch job cannot be fully saved to the storage system, however, the pytorch job will still claim itself as successful.

To understand that, I check the detailed logs of each worker.
Logs show that after the master role of the pytorch job gets completed, the whole pytorch job becomes terminated, regardless of the running pods of the other pytorch workers. I think it should be the reason for the incomplete output data.

It seems to be a bug in my view. Could you please take a check?

Thanks,
Wencong

In the practice of our prod environment, we found a portion of scenarios relies on scheduling job replicas in stages, otherwise there will be severe exceptions, for example:

1. for PyTorchJob, if Workers step into Running phase before Master (Master pod may hanging on pulling images or some other reasons), it will crash immediately because Worker cannot ping Master successfully, then Job goes to Failed.
2. for MPIJob, if Launcher step into Running phase before Workers (Worker pod may hanging on pulling images), Launcher will exit unexpected because kubectl exec command can not reach target container in each Worker, similarly, Job finally failed.

in addition, DAG scheduling can also improve efficiency in certain scenarios, for example: schedule PS before Worker for TFJob to reduce the duration of worker-stall;

Would like to join the group, it'll be appreciated if you could update the qr code.

for backward compatibility with kubeflow.org api group, KubeDL implements enhanced features (e.g. hostnetwork, tensorboard...) through annotations, now we'd replenish these features to job.spec which can be more native.

and it's better KubeDL can detect which workload CRDs are installed into cluster, then enable its corresponding controller automatically.

https://github.com/alibaba/kubedl/blob/f0062f116f14a402ebc6568e8039512284855efe/go.mod#L37

CVE-2020-8565 CVE-2020-8563 CVE-2020-8566 CVE-2020-8564

Recommended upgrade version：v1.18.15