This is the Helm chart for the Kubernetes Operator for Apache Spark.
The Operator requires Kubernetes version 1.8 and above because it relies on garbage collection of custom resources. If customization of driver and executor pods (through mounting custom ConfigMaps and volumes) is desired, then the Mutating Admission Webhook needs to be enabled and it only became beta in Kubernetes 1.9.
The chart can be installed by cloning the repo and running:
$ helm install ./fdp-sparkoperator --namespace spark-operatorAlso you could use a released artifact from Github releases instead, when invoking helm install.
Note that you need to use the --namespace flag during helm install to specify the namespace into which you want to install the operator. The namespace can be existing or not. When it's not available, Helm will take care of creating the namespace. Note that this namespace has no relation to the namespace where you would like to deploy Spark jobs (i.e. the setting sparkJobNamespace shown in the table below). They can be the same namespace or different ones.
The following table lists the configurable parameters of the Spark operator chart and their default values.
| Parameter | Description | Default |
|---|---|---|
operatorImageName |
The name of the operator image | lightbend/sparkoperator |
operatorVersion |
The version of the operator to install | 2.1.2-OpenShift-v1beta2-1.0.1-2.4.4-rh-2.12 |
imagePullPolicy |
Docker image pull policy | IfNotPresent |
sparkJobNamespace |
K8s namespace where Spark jobs are to be deployed. It is also the namespace that the operator instance is to manage. | default |
enableWebhook |
Whether to enable mutating admission webhook | true |
enableMetrics |
Whether to expose metrics to be scraped by Premetheus | true |
controllerThreads |
Number of worker threads used by the SparkApplication controller | 10 |
ingressUrlFormat |
Ingress URL format | "" |
installCrds |
Whether to install CRDs | true |
logLevel |
Logging verbosity level | 2 |
metricsPort |
Port for the metrics endpoint | 10254 |
metricsEndpoint |
Metrics endpoint | "/metrics" |
metricsPrefix |
Prefix for the metrics | "" |
nodeSelector |
Node selector for operator pod assignment | {} |
rbac.create |
Whether to create required roles and bindings | true |
resyncInterval |
Informer resync interval in seconds | 30 |
webhookPort |
Service port of the webhook server | 8080 |
resources |
Resources needed for the operator deployment | {} |
enableBatchScheduler |
Whether to enable batch scheduler for pod scheduling | false |
enableResourceQuotaEnforcement |
Whether to enable the ResourceQuota enforcement for SparkApplication resources. Requires the webhook to be enabled by setting enableWebhook to true. |
false |
enableLeaderElection |
Whether to enable leader election when the operator Deployment has more than one replica, i.e., when replicas is greater than 1. |
false |
| Name-related configs | See below for further explanation | |
nameOverride |
Suffix for created ServiceAccount names, and value of app.kubernetes.io/name labels |
<Chart.Name> |
fullnameOverride |
Used when generating fully qualified app name | "" |
serviceAccounts.sparkoperator.create |
Create Spark operator ServiceAccount name using fully qualified app name | true |
serviceAccounts.sparkoperator.name |
ServiceAccount name for the Spark operator | default if not created |
serviceAccounts.spark.create |
Create Spark job ServiceAccount name using release name | true |
serviceAccounts.spark.name |
ServiceAccount name for the Spark jobs | default if not created |
Specify each parameter using the --set key=value[,key=value] argument to helm install.
The Spark operator namespace is set with the --namespace argument to the helm command.
The namespace for the Spark jobs, and the namespace that the Spark operator will watch, is set with the sparkJobNamespace config parameter.
All created resources are given app.kubernetes.io/name labels with a value the same as the chart name, or the nameOverride config parameter if specified.
The metadata.name for all created resources uses a computed fullname prefix and resource-specific suffix. The fullname will be set from the fullnameOverride config value if provided. Otherwise, if the release name (--name Helm argument) contains the nameOverride config value, or the chart name, then the fullname will be set to the release name value. Lastly, it will be derived by concatenating the release name with the nameOverride value or chart name. (e.g. With release name release, chart name fdp-sparkoperator, and neither overrideName or fullnameOverride having values, the fullname will be release-fdp-sparkoperator and the metadata.name for an initialization Job would be release-fdp-sparkoperator-init.)
All created resources are given a helm.sh/chart label. This is derived by concatenating the chart name with the chart version. For example fdp-sparkoperator-0.3.0.
All generated labels and names are truncated to 63 characters to meet Kubernetes constraints.
If the serviceAccounts.sparkoperator.create config value is true, then the ServiceAccount name for the Spark operator uses the value of the serviceAccounts.sparkoperator.name value if provided, or the computed fullname as explained above.
If the serviceAccounts.spark.create config value is true, then the ServiceAccount name for the Spark job uses the value of the serviceAccounts.spark.name value if provided, or the concatenation of the release name and the string -spark if not. If the serviceAccounts.spark.create config value is instead false, then the ServiceAccount name for the Spark job is set to the value of the serviceAccounts.spark.name value if provided, or default if not.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent