Adding GPU testing to Github workflows about singa HOT 21 OPEN

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

Ok, i will look into it

from singa.

I believe that what you propose in this issue is very important for us.
We need to safeguard our codebase, because a wrong PR could destroy the whole system functionality.
Therefore I believe that GPU testing is necessary.

Here are my opinions on the questions:

1. prefer dedicated server at NUS (not prefer cloud service providers because I think they are too expensive)
2. Linux is enough (ubuntu)
3. For every PR we need to run the test workflow (just like the current test settings for CPU versions, but need to upgrade the test to GPUs)
4. Prefer to keep the test server on so when someone submits a PR it will be checked immediately.
5. Maybe consider in the future to run the examples. I think the priority is to enable GPU build and test first.

from singa.

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

from singa.

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

from singa.

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

Ok, i will look into it

FYI, the different between CPU version and GPU version in building SINGA is just the flag we used in cmake
GPU version: https://github.com/apache/singa/blob/master/tool/docker/devel/ubuntu/cuda10/Dockerfile#L104
cmake -DENABLE_TEST=ON -DUSE_CUDA=ON -DUSE_PYTHON3=ON -DUSE_DNNL=ON ..

CPU version: https://github.com/apache/singa/blob/master/tool/docker/devel/ubuntu/cpu/Dockerfile#L67
cmake -DENABLE_TEST=ON -DUSE_PYTHON3=ON -DUSE_DNNL=ON ..

In GPU version, it include all the GPU test case when we run test/python/run.py
In CPU version, it automatically skiped all the GPU test case

from singa.

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:

First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)

To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.

Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

from singa.

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:

First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)

To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.

Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

@moazreyad thanks a lot for the information, I will consider it. Also I am thinking about if GPU emulator is another option? https://developer.nvidia.com/nvemulate

@naili-xing do you think the github actions can access our server to do gpu test, does it need to connect to nus VPN?

from singa.

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:
First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)
To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.
Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

@moazreyad thanks a lot for the information, I will consider it. Also I am thinking about if GPU emulator is another option? https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

@naili-xing do you think the github actions can access our server to do gpu test, does it need to connect to nus VPN?

I think it needs, if accessing the server from outside of nus, it needs vpn, if access from inside, using nus network, i think there is no needs

from singa.

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

from singa.

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

Hi, @chrishkchris , these two GPU emulators haven't been updated for more than 5 years. And since we have our own machine, the self-hosted GitHub Actions is a better solution.

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

from singa.

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

Hi, @chrishkchris , these two GPU emulators haven't been updated for more than 5 years. And science we have our own machine, the self-hosted GitHub Actions is a better solution.

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

@joddiy OK, then we go for the solution using our own machine gpu for github actions test first.

For our reference: communication between our server and github actions
https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github

from singa.

https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
FYI
"The self-hosted runner uses a HTTPS long poll that opens a connection to GitHub for 50 seconds"

from singa.

@moazreyad I have opened a ticket to ask the infrastructure and hopes they can give us a solution

https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21115?filter=allopenissues

from singa.

@moazreyad
They told us that we can use the API in the following link to add github secrets (to store token like username and password)
https://developer.github.com/v3/actions/secrets/

@joddiy
Could you help test the API to see if we can store some random token, and see if we are able to access what we have stored?
(e.g. you can use postman https://www.postman.com/ to test the restful API)

from singa.

@naili-xing

I think before implementation on the SINGA repo, we fork the repo to test the github actions with our own github account first.

Could you follow the following steps for the test

1. fork the singa repo to your github acoount
2. make the forked repo private (throughout the test to ensure the safety of our server), add me as a collaborator of the forked repo so I can also access it
3. implement the github actions code for gpu test:
   We create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

We evaluate the approach first before implementing it in apache/SINGA repo

from singa.

from singa.

The GPU testing will improve the code coverage and by the way, there is a problem in the current testing workflows: since around two months, the Python tests disappeared from the codecov report in the dev branch and they are never executed in the master branch. Probably a pull request did something that disabled the Python tests. I tried to check quickly, but I think it needs more investigation.

The last codecov report with the Python test is here with code coverage = 70%. The codecov report of the next commit shows no Python test here with code coverage drops to 63.7%.

from singa.

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

If the connection between NUS self-hosted runners and Github servers is not possible because of NUS network restrictions, then another solution can be considered: self-hosting Github itself using Github Enterprise. The cost is $21 per user per month, but there are special discounted licenses for universities. In this case both Github and the Github runners are hosted locally in the same network. (But again I guess we need to configure the DNS of the SINGA github repository to point to the locally hosted github, so we must have some NUS network configuration for external access but may be it is easier?). There is a free trail for Github Enterprise if we want to evaluate it on the NUS server.

Otherwise, we just use a cloud-based solution (from Amazon/Microsoft/Google, ..) to execute the GPU Github workflows. There will be a cost for this cloud service time and resources, but it will also save the cost we need to configure and maintain our own GPU testing pipeline.

from singa.

umm... cloud service will induce recurrent spending, e.g. if on-demand g4dn.xlarge costs ~0.526USD per hour, every month the recurrent cost seems to be around 0.526USD*24(hour)*30(days)=378USD per month
https://aws.amazon.com/cn/ec2/pricing/on-demand/
correct me if I am wrong, and yet we still need to configure and maintain the GPU testing pipeline ourselves

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

If the connection between NUS self-hosted runners and Github servers is not possible because of NUS network restrictions, then another solution can be considered: self-hosting Github itself using Github Enterprise. The cost is $21 per user per month, but there are special discounted licenses for universities. In this case both Github and the Github runners are hosted locally in the same network. (But again I guess we need to configure the DNS of the SINGA github repository to point to the locally hosted github, so we must have some NUS network configuration for external access but may be it is easier?). There is a free trail for Github Enterprise if we want to evaluate it on the NUS server.

Otherwise, we just use a cloud-based solution (from Amazon/Microsoft/Google, ..) to execute the GPU Github workflows. There will be a cost for this cloud service time and resources, but it will also save the cost we need to configure and maintain our own GPU testing pipeline.

from singa.

Hi all, I am the creator of Cirun.io, "GPU Testing" caught my eye.

FWIW I'll share my two cents. I created a service for problems like these, which is basically running custom machines (including GPUs) in GitHub Actions: https://cirun.io/

It is used in multiple open source projects needing GPU support like the following:

• https://github.com/pystatgen/sgkit/
• https://github.com/qutip/qutip-cupy

It is fairly simple to setup, all you need is a cloud account (AWS or GCP) and a simple yaml file describing what kind of machines you need and Cirun will spin up ephemeral machines on your cloud for GitHub Actions to run. It's native to GitHub ecosystem, which mean you can see logs/trigger in the Github's interface itself, just like any Github Action run.

Also, note that Cirun is free for Open source projects. (You only pay to your cloud provider for machine usage)

from singa.

Thank you, but to use Cirun.io we need to install Cirun application in Github. This is not possible because SINGA github account is managed by the Apache foundation that must approve the Cirun application and allow it to be installed in SINGA repo.

from singa.