The notebook used for RAG currently writes a python file locally and uses it as an entrypoint to a Ray job using JobSubmissionClient.

Instead of this approach, we can use Ray's interactive remote client to run the steps remotely. This would allow the notebook to be broken down into smaller cells that can run in isolation, as opposed to a single job that executes everything at once.

The trade-off here is that the remote interactivce client may not be as reliable as using a Ray job

[GKE Batch Reference Architecture] Update Kaniko executor images in Cloud Build create and destroy steps.

On the doc: https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#create-gpu-pool-auto-drivers
It looks like this can be add

gpu_driver_installation_config {
        gpu_driver_version = var.gpu_driver_version
}

To the guest_accelerator section of terraform so there isn't a need for a the nvidia_driver_installer resource eliminating the need of the kubernetes module under gke-platform

Make the installation flow for Kueue Grafana dashboard smoother, propagate environment variables from the Cloud Build yaml into the installation script.

Customer need: need to be able to use a preexisting GCP Service Account when running the image builder instance.

For example, what I did:

                                                Instance: compute.Instance{
                                                        Name:        fmt.Sprintf("%s-instance", name),
                                                        MachineType: fmt.Sprintf("zones/%s/machineTypes/%s", req.Zone, req.MachineType
),
+                                                       ServiceAccounts: []*compute.ServiceAccount{
+                                                               &compute.ServiceAccount{
+                                                                       Email: req.ServiceAccount,
+                                                                       Scopes: []string{
+                                                                               "https://www.googleapis.com/auth/devstorage.read_only"
,
+                                                                               "https://www.googleapis.com/auth/logging.write",
+                                                                               "https://www.googleapis.com/auth/monitoring.write",
+                                                                               "https://www.googleapis.com/auth/pubsub",
+                                                                               "https://www.googleapis.com/auth/service.management.re
adonly",
+                                                                               "https://www.googleapis.com/auth/servicecontrol",
+                                                                               "https://www.googleapis.com/auth/trace.append",
+                                                                       },
+                                                               },
+                                                       },

Customer requirement: need to be able to configure the compute instance to run using shielded settings:

What I needed:

+                                                       ShieldedInstanceConfig: &compute.ShieldedInstanceConfig{
+                                                               EnableSecureBoot:          true,
+                                                               EnableVtpm:                true,
+                                                               EnableIntegrityMonitoring: true,
+                                                       },

Ray pod is seeing this permission issue

  Warning  FailedMount  55s (x7 over 88s)  kubelet            MountVolume.SetUp failed for volume "gcs-fuse-csi-ephemeral" : rpc error: code = PermissionDenied desc = the sidecar container failed with error: mountWithArgs: mountWithStorageHandle: fs.NewServer: create file system: SetUpBucket: Error in iterating through objects: Get "https://storage.googleapis.com/storage/v1/b/rag-data-andrewsy/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&matchGlob=&maxResults=1&pageToken=&prefix=&prettyPrint=false&projection=full&startOffset=&versions=false": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation:
  https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to

Following the steps to access JupyterLab, I install ray from terminal with pip install -U "ray[air]" and then run gpt-j-online example. I get the following error:

ConnectionAbortedError: Initialization failure from server:
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/util/client/server/proxier.py", line 703, in Datapath
    if not self.proxy_manager.start_specific_server(
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/util/client/server/proxier.py", line 298, in start_specific_server
    runtime_env_config = job_config.get_proto_runtime_env_config()
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/job_config.py", line 143, in get_proto_runtime_env_config
    return self.get_proto_job_config().runtime_env_info.runtime_env_config
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/job_config.py", line 115, in get_proto_job_config
    pb.py_driver_sys_path.extend(self.py_driver_sys_path)
AttributeError: 'JobConfig' object has no attribute 'py_driver_sys_path'

Create a mutating admission webhook to inject TPU environment variables into pods started by Kuberay operator service.

What are we using this file for?
This path refers to a personal repo & path ray-on-gke/user is no longer being used

Customer requirement: need the compute instance to run with a configurable disk source image.

Reference:

                                                        Disks: []*compute.AttachedDisk{
                                                                &compute.AttachedDisk{
                                                                        AutoDelete: true,
@@ -136,9 +169,10 @@ func GenerateDiskImage(ctx context.Context, req Request) error {
                                                                        DeviceName: fmt.Sprintf("%s-bootable-disk", name),
                                                                        Mode:       "READ_WRITE",
                                                                        InitializeParams: &compute.AttachedDiskInitializeParams{
-                                                                               DiskSizeGb:  req.DiskSizeGB,
-                                                                               DiskType:    fmt.Sprintf("projects/%s/zones/%s/diskTyp
es/%s", req.ProjectName, req.Zone, req.DiskType),
-                                                                               SourceImage: /* NEED THIS TO BE CONFIGURABLE */

Sometimes the Autopilot e2e tests fail because GMP webhook:

Error: Internal error occurred: failed calling webhook "default.podmonitorings.gmp-operator.gke-gmp-system.monitoring.googleapis.com": failed to call webhook: Post "https://gmp-operator.gke-gmp-system.svc:443/default/monitoring.googleapis.com/v1/podmonitorings?timeout=10s": No agent available

  with module.kuberay-monitoring[0].helm_release.gmp-engine,
  on ../../modules/kuberay-monitoring/main.tf line 16, in resource "helm_release" "gmp-engine":
  16: resource "helm_release" "gmp-engine" {

My guess is that this is because the Autopilot cluster has no nodes initially so the webhook it can't serve this request.

We should try to be consistent with documentation style, which maybe warrants adding a style guide. For starters, I've noticed inconsistencies in how <placeholder> values are represented:

<placeholder>
PLACEHOLDER
%placeholder%
<compound placeholder>
<compound-placeholder>
<compound_placeholder>
$PLACEHOLDER

Angle brackets <placeholder> seems to be most widely used (recommended by the k8s docs style guide). Capitalization and spacing might be optional, but my personal preference is for upper case with underscores: <COMPOUND_PLACEHOLDER>

In this README

2. If deploying webhook across multiple namespaces: make install-reflector

Explain what reflector does and why it's needed?

While following the rag app installation, Terraform prints out the following:

Warning: "default_secret_name" is no longer applicable for Kubernetes v1.24.0 and above

It is currently not possible to tear down a successful RAG deployment because the private service connection to CloudSQL requires a 4 day waiting period before being deleted. This is also reflected in our CI tests where deletion never succeeds.

Add reference architecture for a batch processing platform on GKE using Kueue.

When I tried to run JupyterHub backed by a GCSFuse, I ran into this error from JupyterHub:

2024-03-22T18:48:02Z [Warning] MountVolume.SetUp failed for volume "gcs-fuse-csi-ephemeral" : rpc error: code = PermissionDenied desc = failed to get GCS bucket "gcsfuse-admin": googleapi: Error 403: jupyter-sa@<project-id>.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist)., forbidden

It looks like the predefined roles are missing GCS related roles:

In this README, it should be documented that RayClusters must be labeled with app.kubernetes.io/name: kuberay for kuberay-tpu-webhook to act on them. Mine didn't at first and it took me a while to realize without looking at source.

Update google and google-beta Terraform providers to the latest available ones in the GKE Batch Reference Architecture.

We should have a pre-submit test that will check if the files have the correct licensing headers. We can look into reusing this script: https://github.com/kubernetes/kubernetes/tree/master/hack/boilerplate or something else.

Right now the YAMLs use the default K8s namespace. I wanted to deploy the webhook, Issuer, and Certificate to the ray-system namespace. Some documentation on how to manually edit or a way to parameterize would be great.

I got:

│ Error: services "jupyter-ingress" not found
│
│ with module.jupyterhub.data.kubernetes_service.jupyter-ingress,
│ on ../../modules/jupyter/main.tf line 184, in data "kubernetes_service" "jupyter-ingress":
│ 184: data "kubernetes_service" "jupyter-ingress" {

constantly until i updated this data to use data "kubernetes_ingress_v1"

Ray images (especially with GPU support) are very large and can take several minutes to pull. This has resulted in some CI checks failing while waiting for the Ray cluster to be ready.

Customer requirement: need to be able to pass configurable network/subnet when running the compute instance and run with limited permissions.

NOTE: To get this working myself, I needed to update the daisy dependency (I only did this locally - did not push changes). Daisy was assuming I was either creating the network or had the permissions to list subnets. In this environment, I only have permissions to: gcloud compute networks subnets list-usable --project SOME_OTHER_PROJECT (not a full list operation on subnets).

The following link to the GKE docs page appears to be broken

https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/gke-tpu-examples/training/mnist-single-tpu/README.md

Multiple syntax errors on the follow gcloud command:

gcloud container clusters create l4-demo --location ${REGION}
--workload-pool ${PROJECT_ID}.svc.id.goog
--enable-image-streaming --enable-shielded-nodes
--shielded-secure-boot --shielded-integrity-monitoring
--enable-ip-alias
--node-locations=${REGION}-a
--workload-pool=${PROJECT_ID}.svc.id.goog
--labels="ai-on-gke=l4-demo"
--addons GcsFuseCsiDriver

Create more notebook examples of running Ray workloads on GKE (https://docs.ray.io/en/latest/train/examples.html#train-examples)

The RAG notebook spits out some warning from the SQLAlchemy library:

/tmp/ipykernel_71/582989623.py:7: MovedIn20Warning: The ``declarative_base()`` function is now available as sqlalchemy.orm.declarative_base(). (deprecated since: 2.0) (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
  Base = declarative_base()

The kuberay-cluster module always sets the CLOUDSQL_INSTANCE_CONNECTION_NAME environment variable, even when deploying applications/ray without the RAG part.

CLOUDSQL_INSTANCE_CONNECTION_NAME should be optional and only set when needed for RAG

Missing file in this README: https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/jobset/pytorch

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/jobset/main/examples/pytorch/mnist.yaml
error: unable to read URL "https://raw.githubusercontent.com/kubernetes-sigs/jobset/main/examples/pytorch/mnist.yaml", server reported 404 Not Found, status code=404

My GKE cluster is in europe-west4-b. It'd be great to have an equivalent us-docker.pkg.dev/ai-on-gke/kuberay-tpu-webhook/kuberay-tpu-webhook image in Europe to reduce network egress costs. 🙏

Several READMEs recommend editing the terraform variable declarations directly. I do not think this is a best practice for working with terraform variables. The GCP terraform best practices docs provide the following recommendation:

Store variables in a tfvars file
For root modules, provide variables by using a .tfvars variables file. For consistency, name variable files terraform.tfvars.

Don't specify variables by using alternative var-files or var='key=val' command-line options. Command-line options are ephemeral and easy to forget. Using a default variables file is more predictable.

Other relevant variable best practices are defined here, and discourage providing default values for required variables.

TL;DR - the ray-on-gke readme should be updated with the requirements needed for setting up the prerequisite GKE clusters.

• GKE Cluster must have Workload Identity Enabled
• GKE Cluster must have KubeRay Operator deployed

Here are the details on the errors when these prerequisites are NOT met. These are based on a GKE Standard cluster (1.27.3-gke.100)

module.service_accounts.google_project_iam_binding.monitoring-viewer: Creation complete after 7s [id=ie-raycluster-0f2aa542/roles/monitoring.viewer]
╷
│ Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "example-cluster-kuberay" namespace: "" from "": no matches for kind "RayCluster" in version "ray.io/v1alpha1"
│ ensure CRDs are installed first
│ 
│   with module.kuberay.helm_release.ray-cluster,
│   on modules/kuberay/kuberay.tf line 15, in resource "helm_release" "ray-cluster":
│   15: resource "helm_release" "ray-cluster" {
│ 
╵

This is a result of the KubeRay operator not being installed on the cluster

The iam-role-binding will also fail. This is the result of Workload Identity not being enabled.

I'm calling Ray on GKE cluster via Notebook on gpt-j-online.ipynb and prediction is working as expected. I'm trying to get logging results in log explorer as per instructions, so I'm adding this to the filters:

resource.type="k8s_container"
resource.labels.cluster_name=%CLUSTER_NAME%
resource.labels.pod_name=%RAY_HEAD_POD_NAME%
resource.labels.container_name="fluentbit"

(substituting cluster_name and pod_name)

However I don't get any results at all from resource.labels.container_name="fluentbit" container.
Any suggestions?

There are some pre-requsites documented to enable APIs needed for the infrastructure module. Similar to other modules in applications/, these should be auto enabled as part of the module

Follow-up to https://github.com/GoogleCloudPlatform/ai-on-gke/pull/497/files#r1543631746

Our guides mostly use L4 GPUs now, the default profiles in the JupyterHub configuration should reflect this. Currently they allow use of T4s, A100 and TPUs.

Currently the resources per Jupyterhub Profile is

cpu:
    limit: .5
    guarantee: .5
  memory:
    limit: 1G
    guarantee: 1G

And it should be increased

dcgm-on-gke should have a readme file explaining what it is, and providing some context.

Our cloud build presubmit creates a GPU node pool. This node pool creation can fail if there are no GPUs available. If possible, the clusters created as part of the presubmit should only use CPUs.

The CLOUDSQL_INSTANCE_CONNECTION_NAME env var is always set for RayCluster deployments in applications/ray. However, this env var is only needed for the RAG deployment. This env var should only be set when applications/ray is used by the RAG deployment.

1. Currently, https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/applications/rag/README.md mentions that the example uses https://www.kaggle.com/datasets/denizbilginn/google-maps-restaurant-reviews as the input dataset. However, per #385, the example uses a "Netflix shows" dataset.

2. To help users quickly understand what the sample application does, consider mentioning the example dataset upfront in the introductory section of the README. Currently, the dataset info is buried rather deep in the README.

When attempting to use the gke-disk-image-builder tool with hyperdisk, we ran into the following error:

Nov 22 22:58:32 debian google_metadata_script_runner[2040]: startup-script-url: Device /dev/sdb does not exist. Please rerun the tool to try it again.

Steps to Reproduce:

go run ./cli --project-name=$PROJECT_NAME --image-name=triton-2309-py3-hd --zone=$ZONE --gcs-path=gs://$GCS_PATH/ --container-image='nvcr.io/nvidia/tritonserver:23.09-py3' --machine-type=h3-standard-88 --disk-type=hyperdisk-balanced

Customer requirement: need to be able to configure the network tags that the instance runs with for firewall purposes.

We use modules in https://github.com/terraform-google-modules/terraform-google-kubernetes-engine for creating GKE clusters, which does not expose a config for image streaming yet.

We should either propose a fix to allow clusters to enable image streaming with that module or remvoe our depdendency to that module.

Trying to deploy RAG on an existing cluster results in this error:

Step #0 - "Apply blueprint": on .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity/main.tf line 51
Step #0 - "Apply blueprint": 
Step #0 - "Apply blueprint": time="2024-04-02T14:44:14Z" level=error msg="error writing to GCS: error closing temp logfile: googleapi: got HTTP response code 503 with body: Service Unavailable"
Step #0 - "Apply blueprint": Error: Error, failed to create instance because the network doesn't have at least 1 private services connection. Please see https://cloud.google.com/sql/docs/mysql/private-ip#network_requirements for how to create this connection.
Step #0 - "Apply blueprint": time="2024-04-02T14:44:14Z" level=error msg="Error (exit code 1) running \"terraform apply -json /tmp/tfplan-2667148229/plan.out\". Stderr:\n"
Step #0 - "Apply blueprint": Error: Error, failed to create instance because the network doesn't have at least 1 private services connection. Please see https://cloud.google.com/sql/docs/mysql/private-ip#network_requirements for how to create this connection.
Step #0 - "Apply blueprint": error: Error, failed to create instance because the network doesn't have at least 1 private services connection. Please see https://cloud.google.com/sql/docs/mysql/private-ip#network_requirements for how to create this connection.
Step #0 - "Apply blueprint": 
Step #0 - "Apply blueprint": on .terraform/modules/cloudsql.cloudsql/modules/postgresql/main.tf line 54

I think this is because we only create the private service connection when creating new networks, which doesn't execute when trying to run the RAG deployment on an existing cluster

Steps to reproduce:

1. Scale down RAG frontend to 1 replica
2. Send multiple prompts
3. kubectl get pods to see that frontend pod is restarted due to OOM kill.

This is also possible to replicate when running multiple frontend replicas, but it just takes more prompts due to prompts being load balanced across the replicas.

Feature request: Add Dynamic Workload Scheduler (https://cloud.google.com/blog/products/compute/introducing-dynamic-workload-scheduler) example to GKE Batch Reference Architecture. GCP docs have a published example (https://cloud.google.com/kubernetes-engine/docs/how-to/provisioningrequest) showing how to run GPUs for batch workloads using ProvisioningRequest.