ServiceX, a component of the IRIS-HEP DOMA group's iDDS, is an experiment-agnostic service to enable on-demand data delivery along the concepts originally developed for ATLAS but specifically tailored for nearly-interactive, high performance, array based and pythonic analyses context. It provides uniform backend interfaces to data storage services and frontend (client-facing) service endpoints for multiple different data formats and organizational structures.

It is capable of retrieving and delivering data from data lakes, as those systems and models evolve. It depends on Rucio to find and access the data. The service is capable of on-the-fly data transformations to enable data delivery in a variety of different formats, including streams of ROOT data, small ROOT files, HDF5, and Apache Arrow buffers as examples. In addition, ServiceX includes pre-processing functionality for event data and preparation for multiple clustering frameworks (e.g. such as Spark). Eventually, it will be able to automatically unpack compressed formats, potentially including hardware accelerated techniques, and can prefilter events so that only useful data is transmitted to the user.

The entire serviceX stack can be installed using the helm chart contained in this repo. The chart has been deployed to the ssl-hep helm repo. The default values for the chart have been set up so you can run the service with a public xAOD file without any grid credentials.

There is also an included Jupyter notebook for driving the system and getting some simple plots back.

ServiceX requires an installation of Kubernetes. If you have access to a cluster you can use that. Otherwise, you can enable a single node kubernetes cluster on your desktop using Docker-Desktop.

The service is published as a helm chart. You will need a version of helm. The chart has been tested on Helm3 installed on your desktop.

ServiceX will use your CERN grid certificates to communicate with Rucio. In this repo we've included a set of demo values that will just use a public xAOD file so you can try out ServiceX with minimal fuss.

% helm repo add ssl-hep https://ssl-hep.github.io/ssl-helm-charts/
% helm repo update
% helm install -g -f demo-values.yaml ssl-hep/servicex 

The notes at the end give you some helpful tips on interacting with the pods in your deployment. It takes about one minute for RabbitMQ to complete its initialization and all of the other pods are able to register and start up. Note that it is normal for some pods to enter a "Crash Loop Backoff" during this time.

You can check on the status of the deployment with the command

% kubectl get pod

Eventually your setup should resemble this:

NAME                                             READY   STATUS    RESTARTS   AGE
servicex-1579021789-code-gen-7cd998d5b6-nwtgv            1/1     Running   0          49m
servicex-1579021789-did-finder-7c5cbb4575-52wxf          1/1     Running   0          7m53s
servicex-1579021789-minio-78b55bfdf8-mbmmf               1/1     Running   0          49m
servicex-1579021789-preflight-b748b4dfd-qqt89            1/1     Running   4          49m
servicex-1579021789-rabbitmq-0                           1/1     Running   0          49m
servicex-1579021789-servicex-app-98779c79c-cvmqx         1/1     Running   3          49m
servicex-1579021789-x509-secrets-74f4bcc8bb-5kqvb        1/1     Running   0          49m

The name of your release will be different.

The default values for the ServiceX helm chart deploy the following services:

Kubernetes offers some sophisticated ways to expose deployed services for external connections. For this demo, we will just use the simplest possible way expose the services to your desktop: port forwarding.

We will need to expose port 5000 of the REST server so transform requests can be submitted. We will also expose port 9000 of the Minio server so we can browse and download generated files.

The notes at the end of helm deployment provide some nice generic commands to do this. You can also look at the list of pods and then issue commands like this to tell kubernetes to forward their ports. Here are examples based on the pod names shown above. Your pod names will be different:

% kubectl port-forward exegetical-mouse-servicex-app-6fdd5bf7b6-dcwwd 5000:5000
% kubectl port-forward exegetical-mouse-minio-57cbd595b5-77426 9000:9000

Note the port-forward command doesn't return when you run it. It blocks in the terminal so long as the port is being forwarded. You can stop it at any time with ^C. You will need to enter each of these commands in different terminals since port forwarding will be needed during the demo.

We have a notebook that submits a sample request to your serviceX, waits for the transformation to complete and downloads the results. The notebook is found in the examples folder of this repo.

First create a python virtualenv. Note that the notebooks (and package requirements) are currently configured to work for Python 3.

% virtualenv ~/.virtualenvs/servicex_demo
% source ~/.virtualenvs/servicex_demo/bin/activate

Now install the dependent python packages

% pip install -r examples/requirements.txt

Then you can start Jupyter server

% jupyter notebook

Once that launches it should open up a browser window for you. Vist the examples folder of this repo and open up the ServiceXDemo notebook.

Now that you have tried out the demo, you will certainly want to take the training wheels off and run ServiceX against a real dataset.

The DID Finder will need to talk to CERN's Rucio service which requires grid certificates and a passcode to unlock them. If you are a member of the ATLAS experiment, you can follow these helpful instructions on obtaining proper grid certificates.

You install the certs into the cluster as a global kubernetes secret. This is easily done with the ServiceX Command Line Interface (cli).

1. Install the cli with pip install servicex-cli
2. When this completes you can copy the certs into the cluster with servicex init By default, the script will pick up your grid certs from the .globus directory in your home directory. You can override this with the --cert-dir command line option. Also, the cli will create the secret in the default namespace. If you are using a different namespace you can override this with the --namespace option.

ServiceX can deliver transformed datasets to an object store service (Minio) that is optionally installed with this helm chart. An alternative delivery mechanism is streamed arrow tables using Kafka. Full instructions and sample config files are in the kafka directory.

The following table lists the configurable parameters of the ServiceX chart and their default values. Note that you may wish to change some of the default parameters for the dependent rabbitMQ or mino

You can access the REST service on your desktop with

% kubectl port-forward <<app pod>> 5000:5000

You can access a minio browser with:

% kubectl port-forward <minio pod> 9000:9000

Log in to this as miniouser, password is leftfoot1

Once you have exposed port 5000 of the REST app, you can use the included postman collection to submit a transformation request, and check on the status of a running job. You can import the collection from the ServiceX REST App repo

We are still experimenting with various configurations for deploying a scaled-up ServiceX.

It's certainly helpful to beef up the RabbitMQ deployment:

rabbitmq:
  resources:
     requests:
       memory: 256Mi
       cpu: 100m
  replicas: 3

Tip: List all releases using helm list

To uninstall/delete the my-release deployment:

$ helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

ServiceX is a community effort. The architecture lends itself to orchestrating different types of workflows and generating different outputs. Fork us on github and make a contribution!

Please submit issues for bugs and feature requests to this repo.

We manage project priorities with a zenhub board

We coordinate effort on the IRIS-HEP slack. Come join this intellectual hub.

Microservice architectures can be difficult to test and debug. Here are some helpful hints to make this easier.

1. Instead of relying on the DID Finder to locate some particular datafile, you can mount one of your local directories into the transformer pod and then instruct the DID Finder to always offer up the path to that file regardless of the submitted DID. You can use the hostMount value to have a local directory mounted into each transformer pod under /data. You can use the didFinder.staticFile value to instruct DID Finder to offer up a file from that directory.
2. You can use port-forwarding to expose port 15672 from the RabbitMQ pod to your laptop and log into the Rabbit admin console using the username: user and password leftfoot1. From here you can monitor the queues, purge old messages and inject your own messages

This project is supported by National Science Foundation under Cooperative Agreement OAC-1836650. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.