Is the reason why it is skipped for ratel

Hi all,

When trying to deploy the chart with the alpha antiAffinity set to hard, I get the following error:
Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(StatefulSet.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution): invalid type for io.k8s.api.core.v1.PodAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution: got "map", expected "array"
Setting it to soft works perfectly fine, so I assume it's just a syntax issue. Setting it to hard on the zeros also works fine.

seems it is not possible to deploy this helm charts without having persistence enabled
what about deploying this on baremetal and want to use local storage?
how can one disable persistence? why is persistence enforced?

Release "dgraph" does not exist. Installing it now.
Error: YAML parse error on dgraph/templates/alpha-statefulset.yaml: error converting YAML to JSON: yaml: line 66: did not find expected key

Error when specifying values.yaml file

helm install dgraph -f values.yaml --namespace dgraph dgraph/dgraph

Error: YAML parse error on dgraph/templates/alpha-statefulset.yaml: error converting YAML to JSON: yaml: line 74: did not find expected key

and

Error: YAML parse error on dgraph/templates/alpha-statefulset.yaml: error converting YAML to JSON: yaml: line 81: did not find expected key

It seems that quite some changes happened since 0.0.4 release, as the name override or the ingress support.

Is it planned to make a new release any soon?

This would be very useful, as maintaining a copy of the chart can be very tedious for users.

I spin up a cluster with req/limit of 6gb mem. After about 10 mins of heavy load, dgraph alphas get oomkilled by kubernetes. When the alpha pods restart, the get oomkilled straight away- and the whole cluster stays in a broken state.

I'm guessing there's some sort of write-ahead-log that dgraph is trying to resume from (from the attached persistent volumes) that is larger than the mem limit given- causing it to instantly get oom killed?

When spec.rules[0].host is absent or set to null, it will default to *. This is used when only one service, such as Dgraph Alpha, is used. Instead of an assigned FQDN for the hostname, the IP address of the load balancer will be used.

This is definitely, not a recommended practice, but for testing or other edge cases, this could be used. The helm install should not error when the host gets set to null through global.alpha_hostname, global.ratel_hostname, global.alpha_grpc_hostname, alpha.ingress.hostname, ratel.ingress.hostname.

When running helm install, alphas have following invalid address errors:

2020/03/24 13:21:01 dgraph3-dgraph-alpha-2.dgraph3-dgraph-alpha-headless.default.svc.cluster.local.:7080 is not valid address
github.com/dgraph-io/dgraph/x.AssertTruef
	/tmp/go/src/github.com/dgraph-io/dgraph/x/error.go:88
github.com/dgraph-io/dgraph/worker.StartRaftNodes
	/tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:84
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run.func4
	/tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:645
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357

But everything are fine in zeros. Is that anything wrong when checking --my parameter for alpha?

I0324 12:46:54.069770       1 pool.go:160] CONNECTING to dgraph003-dgraph-zero-1.dgraph003-dgraph-zero-headless.default.svc.cluster.local.:5080

STEPS TO REPRODUCE

helm template ./charts/dgraph --values -  <<EOF
fullNameOverride: jmdg
global:
  ingress:
    enabled: true
    alpha_hostname: alpha.example.com
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
  ingress_grpc:
    enabled: true
    alpha_grpc_hostname: grpc.example.com
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
EOF

ACTUAL RESULTS

---
# Source: dgraph/templates/global-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: release-name-dgraph-ingress
  labels:
    app: dgraph
    chart: dgraph-0.1.0
    component: alpha
    release: release-name
    heritage: Helm
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - backend:
              service:
                name:  release-name-dgraph-alpha
                port:
                  number: 8080
            pathType: ImplementationSpecific
            path: /*
      host: alpha.example.com
---
# Source: dgraph/templates/global-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: release-name-dgraph-ingress-grpc
  labels:
    app: dgraph
    chart: dgraph-0.1.0
    component: alpha
    release: release-name
    heritage: Helm
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: grpc.example.com
      http:
        paths:
          - backend:
              service:
                name:  release-name-dgraph-alpha
                port:
                  number: 9080
            pathType: ImplementationSpecific
            path: /*
      host: grpc.example.com

EXPTECTED RESULTS

The grpc should only have a single host key, e.g.

# Source: dgraph/templates/global-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: release-name-dgraph-ingress-grpc
  labels:
    app: dgraph
    chart: dgraph-0.1.0
    component: alpha
    release: release-name
    heritage: Helm
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - backend:
              service:
                name:  release-name-dgraph-alpha
                port:
                  number: 9080
            pathType: ImplementationSpecific
            path: /*
      host: grpc.example.com

It'd be useful to have a way to enable TLS in the chart. Ideally, the chart would allow you to enable TLS and configure the tls_client_auth flag (defaulting to REQUIREANDVERIFY), and allow you to provide your own TLS CA and node certificates. It would need a way to provide a client certificate for health checking, to conditionally replace the livenessProbe (so it'd use curl).

It should not always default to the *cluster.local dns name.

I'm editing values.yaml to attach my pvc for alpha pod. But i don't see any mounted volumes in that path. Any idea why there are no errors as such in logs as well?
alpha:
persistence:
enabled: true
claimName: azurefile
storageClass: "azurefile"
accessModes:
- ReadWriteMany
size: 1Gi
annotations: {}
mountPath:
- /mnt/azure

There are no tolerations config for the ratel in the values.yaml in dgraph helm chart.
It cause that the pod of ratel can't been schedule to some node with taint.
I add tolerations config in the values.yaml. And then ,I refer to that tolerations yaml in the file of dgraph/templates/ratel/deployment.yaml like alpha or zero.
as follow:

the helm chart version is dgraph/dgraph:0.2.2 .

root@oldsix:/data2/tmp/dgraph/package# helm search repo dgraph
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
dgraph/dgraph           0.2.2           v23.0.1         Dgraph is a horizontally scalable and distribut...
dgraph/dgraph-lambda    0.0.2           v1.2.4          Serverless platform for running JavaScript on S...
dgraph/ratel            0.1.1           v21.12.0        A Helm chart for Kubernetes 

All.

dgraph v22.0.2 is GA. will there be a new version for it?
if not, which chart version should I use? 0.0.19(tag: v21.12.0) or 0.0.18(tag: v21.03.1)

Ratel currently is used as UI mostly for debugging and experimenting.
Normal operations run through alpha from (presumably) another service in the Cloud.

UI can be considered, within the Cloud / Kubernetes environment, an optional feature to be enabled.

Ratel is also completely unrelated to the deployment of zero and alpha, and it can be used from either localhost or play.dgraph.io without consuming any resource from k8s nodes (or load balancers, which are not cheap for small projects).

Ratel should also point to the port 80 in the service when enabled since kubernetes manages all the port allocation and there is no risk to encounter a busy port (as it might happen when ratel is deployed in docker/docker-compose in a local environment).

The expected behavior would also be to find the UI on the IP of the load balancer (or DNS record pointing to ratel) on the port 80 or 443, rather than 8080, which is instead common for local development.

Some Kubernetes clusters require resources to be defined for all containers running in the cluster. Need to be able to define resources for the initContainers.

Support for path with both ingressClass and annotations

IngressClass Support for path

When ingressClass is set to gce, alb, or nxs, the path should be set to /* with Dgraph, otherwise, / can be used for the path.

Annotation Support for path

Additionally, the annotations of kubernetes.io/ingress.class still needs to be supported if google-load-balancer-controller is not used on GKE.

References

• https://github.com/kubernetes/ingress-gce
• https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#controller_summary

Enabling mTLS on the Alpha nodes breaks the livenessProbe and readinessProbe. A workaround is to run a command probe instead but the best fix IMO is to allow the /health path to be served on HTTP instead.

OVERVIEW

Ratel requires a host key to be specified when creating an ingress. In Kubernetes, if the key host is absent, then the host: * will be added as the key:value pair. This is done should the operator want to deploy a single ALB as the load balancer, but choose to forego any HTTP based routing.

STEPS TO REPRODUCE

helm install ratel --namespace ratel --create-namespace ./charts/ratel/ --values - <<EOF
ingress:
  enabled: true
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
  hosts:
    - paths:
        - path: /*
          pathType: ImplementationSpecific
EOF

EXPECTED RESULTS

If the host key is absent, then it should not be included rendered ingress output.

ACTUAL RESULTS

A host key without a value will be created, which will cause an error and prevent Ratel from being deployed.

Specifically this is what is produced:

# Source: ratel/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: release-name-ratel
  labels:
    helm.sh/chart: ratel-0.1.0
    app.kubernetes.io/name: ratel
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
# ERROR IS HERE
    - host:
# ABOVE CAUSES AN ERROR
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: release-name-ratel
                port:
                  number: 80

Notice the host key with no value.

Ratel is mainly used for access through http from the external world

The apps mostly use the alpha to communicate with Dgraph

I can pick this up, just wanted to know thoughts from someone who knows better

Right now this chart doesn't work.
Looks like we need to serve some XML(?).

The backup full/incremental charts are using the old apiVersion: batch/v1beta1 value which prevents use on k8 version 1.25+. batch/v1 has been stable since 1.21.

• https://github.com/dgraph-io/charts/blob/master/charts/dgraph/templates/backups/cronjob-full.yaml#L15
• https://github.com/dgraph-io/charts/blob/master/charts/dgraph/templates/backups/cronjob-inc.yaml#L15

What are your thoughts on checking the capabilities and automatically changing the version OR using a values override?

{{ if or ($.Capabilities.APIVersions.Has "batch/v1/CronJob") (eq $.Values.global.batch.cronJob.apiVersion "batch/v1") -}}
apiVersion: batch/v1
{{- else -}}
apiVersion: batch/v1beta1
{{- end }}

• Source: https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2692/diffs

If ingress_grpc.enabled=true,some error would happend,like this:

  ingress:
    enabled: true
    ingressClassName: nginx
    hostname: "dgraph.knowdee.com"
    # annotations: {}
    tls: 
    - hosts: 
      - dgraph.knowdee.com
      secretName: dgraph-dgraph-alpha-tls-secret

  ## alpha GRPC ingress resource configuration
  ## This requires an ingress controller to be installed into your k8s cluster
  ingress_grpc:
    enabled: true
    ingressClassName: nginx
    hostname: "grpc-dgraph.knowdee.com"
    annotations: 
      nginx.ingress.kubernetes.io/backend-protocol: GRPC
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
    tls: 
    - hosts: 
      - grpc-dgraph.knowdee.com
      secretName: dgraph-dgraph-alpha-tls-secret

        Helm install failed: error while running post render on files:
        map[string]interface {}(nil): yaml: unmarshal errors:
          line 31: mapping key "apiVersion" already defined at line 2
          line 32: mapping key "kind" already defined at line 3
          line 33: mapping key "metadata" already defined at line 4
          line 41: mapping key "spec" already defined at line 14

it is because there are the extra "-" in the file of dgraph/templates/alpha/ingress.yaml at the line 64.

it should remove the symbol "-".

the helm chart version is dgraph/dgraph:0.2.2 .

root@oldsix:/data2/tmp/dgraph/package# helm search repo dgraph
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
dgraph/dgraph           0.2.2           v23.0.1         Dgraph is a horizontally scalable and distribut...
dgraph/dgraph-lambda    0.0.2           v1.2.4          Serverless platform for running JavaScript on S...
dgraph/ratel            0.1.1           v21.12.0        A Helm chart for Kubernetes 

that all.

case 1:

If the alpha grpc service is accessed using ingress, the nginx(ingress) should be added some annotations, otherwise the grpc service will not be normally accessible by the client. These annotations are as follow:

  ingress_grpc:
    enabled: true
    ingressClassName: nginx
    hostname: "grpc-dgraph.knowdee.com"
    annotations: 
      nginx.ingress.kubernetes.io/backend-protocol: GRPC
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
    tls: 
    - hosts: 
      - grpc-dgraph.knowdee.com
      secretName: dgraph-dgraph-alpha-tls-secret

My java client is :

import io.dgraph.DgraphClient;
import io.dgraph.DgraphGrpc;
import io.dgraph.DgraphProto;
import io.dgraph.Transaction;
import io.grpc.ManagedChannel;
import io.grpc.netty.GrpcSslContexts;
import io.grpc.netty.NettyChannelBuilder;
import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslContextBuilder;
import lombok.extern.slf4j.Slf4j;
import javax.net.ssl.SSLException;
import java.util.HashMap;
import java.util.Map;
@Slf4j
public class Dgraph {
    public static void main(String[] args) throws SSLException {
        SslContextBuilder builder = GrpcSslContexts.forClient();
        SslContext sslContext = builder.build();
        ManagedChannel channel = NettyChannelBuilder.forAddress("grpc-dgraph.knowdee.com", 443)
                .sslContext(sslContext)
                .build();
        DgraphGrpc.DgraphStub stub = DgraphGrpc.newStub(channel);
        DgraphClient dgraphClient = new DgraphClient(stub);
        DgraphProto.Version v = dgraphClient.checkVersion();
        log.info("dgraph version is {}",v.getTag());
        String json = "{    findNames(func: has(name)){            name     }    }";
        Map<String, String> vars = new HashMap<>();
        try (Transaction transaction = dgraphClient.newReadOnlyTransaction()) {
            DgraphProto.Response response = transaction.queryWithVars(json, vars);
            String result = response.getJson().toStringUtf8();
            log.info("result is {}",result);
        }
    }
}

case 2:

the value of type should be type: kubernetes.io/tls not type: Opaque in the file of dgraph/templates/alpha/secret-tls.yaml and dgraph/templates/zero/dgraph/templates/zero .
When I use tls certificate with base64 encode, the Opaque value will cause tls certificate of ingress is invalid .

the helm chart version is dgraph/dgraph:0.2.2 .

root@oldsix:/data2/tmp/dgraph/package# helm search repo dgraph
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
dgraph/dgraph           0.2.2           v23.0.1         Dgraph is a horizontally scalable and distribut...
dgraph/dgraph-lambda    0.0.2           v1.2.4          Serverless platform for running JavaScript on S...
dgraph/ratel            0.1.1           v21.12.0        A Helm chart for Kubernetes 

All.

The aws-alb-ingress-controller no longer exist and has been replaced with aws-load-balancer-controller. The example needs to be updated.