Release, versioning and continous integration about singa HOT 13 OPEN

Hi @nudles , is it possible to access the machine running jenkins?

from singa.

I have studied how fastai release process works, and semantic versioning and etc.

In my opinion, this topic involves 2 roles:

1. developers code, commit, and send PR to singa:master or singa:dev. For each PR, CI compiles, builds(conda), tests and returns pass/fail.

2. supervisors make release, and decide if the release is a patch, minor, major. There should be some automation here, supervisors then can just run command like $ make release-patch or $ make release-minor.

The automation should do
a) checkout to latest singa:master (the code merged in latest master should pass CI)
b) define new_version <-- semantic versioning
c) git tag new_version && git push new_version
d) build anaconda package and upload to anaconda cloud

If this approach is ok, then we need to add automation part.

from singa.

Hi @nudles , how release is done currently? is this flag "TRAVIS_SECURE_ENV_VARS" (tool/travis/build.sh line 41) used?

if [[ "$TRAVIS_SECURE_ENV_VARS" == "false" ]];
  # install and run unittest
then
  echo "no uploading if ANACONDA_UPLOAD_TOKEN not set"
else
  # turn off debug to hide the token in travis log
  set +x
  # upload the package onto anaconda cloud
  anaconda -t $ANACONDA_UPLOAD_TOKEN upload -u $USER -l main $CONDA_BLD_PATH/$OS/singa-*.tar.bz2 --force
fi

from singa.

Yes. Currently, we use Travis to build CPU package and Jenkins to build GPU package.
If the TRAVIS_SECURE_ENV_VARS is available, the package will be uploaded to the cloud.

from singa.

Yes. Currently, we use Travis to build CPU package and Jenkins to build GPU package.
If the TRAVIS_SECURE_ENV_VARS is available, the package will be uploaded to the cloud.

Currently, how is the "release" build triggered or how is the Env Var TRAVIS_SECURE_ENV_VARS set? for travis is it done by web UI? https://blog.travis-ci.com/2017-08-24-trigger-custom-build

from singa.

The travis for apache/singa does not upload the package because the variable is not set.
In jenkins, we sync the master branch to nusdbsystem/singa, whose travis will do the building and send the package to the cloud.

from singa.

Hi @nudles , could you kindly advise if this PR caters this issue?
#621
usage: https://github.com/apache/singa/blob/d25539e16b6f3551ad6b1e2a74713025e150d30c/tool/release/README.md

from singa.

1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
2. both the CMake and Conda building scripts need to parse the version from the git tag.
3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

from singa.

1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
2. both the CMake and Conda building scripts need to parse the version from the git tag.
3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

All of this can be easily done with Github Actions. The CI in Github can be configured to monitor a given branch. There are many actions available for use that manage the version tags, create docker containers and so on.

from singa.

Hi @moazreyad , thank you for the advice, I will study how github actions works.

from singa.

1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
2. both the CMake and Conda building scripts need to parse the version from the git tag.
3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

Regarding 1, after comparing other framework, like tf.

• For major release, they have alpha and beta, then followed by rc, release candidate.
• And for minor, pre release starts from rc.
• While patch will be released directly.

For torch, similarily:

• major starts from alpha, followed by rc
• minor starts from rc
• patch is released directly.

By right, simply speaking, alpha is first phase of testing internally. beta is testing publicly. rc is almost like stable release unless serious bug is discovered.

from singa.

from singa.

1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
2. both the CMake and Conda building scripts need to parse the version from the git tag.
3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

All of this can be easily done with Github Actions. The CI in Github can be configured to monitor a given branch. There are many actions available for use that manage the version tags, create docker containers and so on.

Is it possible to build the website via Github Action? We need the following steps

1. when there is a new commit to singa-doc, we build the website to generate the html files
2. push the html files to singa-site as a new commit

from singa.