new release page about mate-dev-scripts HOT 44 CLOSED

@raveit65 @sc0w The server was upgraded to Debian 10 yesterday 😃

from mate-dev-scripts.

You can use nginx or caddy or other software, that support reverse proxy, to use https like https://pub.mate-desktop.org/release as the notify server url.

from mate-dev-scripts.

@raveit65 @flexiondotorg

This is the step by step to use the release-server:

1. download the last release-server binary from here:
   https://github.com/yetist/release-server/releases
   then save the release-server in $PATH directory and give it execute permission.
2. download the example config file from here:
   https://github.com/yetist/release-server/blob/master/release-server.toml.example
   and rename it to release-server.toml, put it in /etc/xdg/release-server/<VERSION>/release-server.toml
3. edit the release-server.toml file.
   In the [path] section, to setup where the tarballs saved, these path maybe used in static http server, such as apache, nginx or caddy.
   In the [security] section, there are there item need to set:
   a) api_secret: please use some random characters. Then, goto the travis website, select one repo which want to use auto release, create new hide environment variable named API_SECRET and set it to use the the same value.
   b) allow_repos can simple setup to * for allow any repo, or write every repos like mate-desktop/caja, mate-desktop/mozo etc.
   c) allow_ips can setup to the travis-ci ip address, they can found here: https://docs.travis-ci.com/user/ip-addresses/
4. run release-server in the server.
5. goto the repo which to use github release page, append the notify_servers into .travis.yml.
   The example is here: https://github.com/yetist/mate-common/blob/master/.travis.yml
   The url is provide by the release-server, according the the release-server.toml.

Then, if you create tag for some repo, and push it to the github, the github release page will created, and the tarballs will saved in the custom release server.

You can try it and hope to go well.

from mate-dev-scripts.

The pre-release folder wasn't a public folder. We use it in the past to share tarballs for testing between team members.
And is there a way to upload tarballs by hand? Same what we did here https://release.mate-desktop.org
People with access to this website cold upload a tarball there, which was moved to public download folder internally.

from mate-dev-scripts.

This is two questions:

1. Publish drafts and pre-release versions:
   Github supports the release of drafts, pre-releases, and release versions. To automate them, you need to enable the settings in .travis.yml:

releases:
  draft: false
  prerelease: false
  ...

2. Custom server listing directory:
   The release-server only download tarballs from github release page and save them into some directories on the server.
   Whether to display these directories publicly, or to require authentication to access these directories, can be changed by modifying the http server configuration.

from mate-dev-scripts.

The whole process is like this:

local -> travis -> github release -> http server.

local: git push TAG =>
travis: make dist -> publish to github release -> notify http server =>
http server: download tarballs from github -> save them into static http directory =>
person: visit url, to see the tarballs in http server.

from mate-dev-scripts.

Ok, so the folder stucture is what we used at github release page and doesn't have anything to do with our old pre-relase and release folder to upload tarballs to our server.

then save the release-server in $PATH directory and give it execute permission.

This is /usr/bin ? Or can i put the binary in a home folder of a fresh account?

I think a systemd unit file would be nice and comfortabel for starting the binary.
Hopefully i will find some time over the weekend to configure the server.

from mate-dev-scripts.

Yes, you can set the directory as needed.
You can also use the old method instead of the draft and pre-release provided by github.
The release-server can be placed in any suitable directory as long as systemd can start it.
Here is demo for that:

[Unit]
Description=Release server
After=syslog.target
After=network.target

[Service]
LimitNOFILE=65535
RestartSec=2s
Type=simple
User=www-data
Group=www-data
ExecStart=/usr/bin/release-server
Restart=always

[Install]
WantedBy=multi-user.target

from mate-dev-scripts.

@yetist
Which characters can I use for the api_secret token?
Letters, numbers, upper and lower case, special characters ?

from mate-dev-scripts.

Letters, numbers, upper and lower case is ok, except space and Tab.

from mate-dev-scripts.

One question regarding the new release system. The current system updates an RSS feed when a new release is published:

• https://pub.mate-desktop.org/rss.xml

New feed entries are automatically posted to social media. Where will the RSS feed reside for the new release system?

from mate-dev-scripts.

@raveit65

According to your configuration files, https://pub.mate-desktop.org/releases is used to receive
the notification, but the real service address is http://<IP>:9090/release, Do you setup the reverse proxy so that they access the same service? If not, the notify_server should be set to http://<IP>:9090/release.

I used caddy, the configuration file is as follows:

# vim: set filetype=caddy ts=4 sts=4 : 
mate.zhcn.cc {
        root /srv/http/mate-desktop/
        log /var/log/caddy/mate-desktop.log
        gzip
        index        index.html
        browse
}

post.zhcn.cc {
        log /var/log/caddy/mate-desktop.log
        proxy / http://localhost:9090
        limits 20mb
}

The release-server's config file:

host = "localhost"
port = 9090

So I can use the https://post.zhcn.cc/release as the notify_servers in .travis.yml:

    notify_servers:
      - https://post.zhcn.cc/release

from mate-dev-scripts.

Please check the server with:

$ curl -v -d 'a=b' https://pub.mate-desktop.org:9090/release

If a valid service is tested, it responds {"status":"invalid request."}:

$ curl -v -d 'a=b' http://localhost:9090/release
*   Trying ::1:9090...
* TCP_NODELAY set
* Connected to localhost (::1) port 9090 (#0)
> POST /release HTTP/1.1
> Host: localhost:9090
> User-Agent: curl/7.66.0
> Accept: */*
> Content-Length: 3
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 3 out of 3 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=utf-8
< Date: Sat, 05 Oct 2019 01:20:58 GMT
< Content-Length: 29
< 
* Connection #0 to host localhost left intact
{"status":"invalid request."}

from mate-dev-scripts.

It seems the server is running, but what is the result when you running curl at your working pc? If there is no result, maybe you should check the iptables at server.

from mate-dev-scripts.

So I think the firewall is open at server, and it disable the port, you should check that.

from mate-dev-scripts.

I've open port 9090 on the firewall. The other thing to consider is that the server is behind a CloudFlare CDN, mostly to block malicious traffic.

https traffic to mate-desktop.org will be terminated by the CloudFlare CDN. So we need to connect to the release server on a FQDN that is not terminated by the CDN, yerba.mate-desktop.org should work.

from mate-dev-scripts.

@flexiondotorg

If possible, I recommend turning off port 9090, which can be done using a reverse proxy. And I recommend using https without a port to access it.

from mate-dev-scripts.

@yetist Understood 👌 I'll use nginx.

from mate-dev-scripts.

@raveit65 @yetist The release-server is now behind an nginx reverse proxy and available via https://release.mate-desktop.org

from mate-dev-scripts.

@raveit65 @yetist The release-server is now behind an nginx reverse proxy and available via https://release.mate-desktop.org

OK, https://release.mate-desktop.org/release works.

from mate-dev-scripts.

I have closed port 9090 on the server.

I have also shutdown the old release services.

If you like, you can use some authentication and not allow public access to https://pub.mate-desktop.org/draft/ or https://pub.mate-desktop.org/pre-release/.

The config in .travis.yml can control that:

releases:
  draft: false
  prerelease: false

EDIT:
if draft: true, the released tarball will saved into https://pub.mate-desktop.org/draft/, if prerelease: true, the tarball saved into https://pub.mate-desktop.org/pre-release/

from mate-dev-scripts.

@yetist
sucessfull test with

notify_servers:
      - https://release.mate-desktop.org/release

and

allow_repos = [
"*"
]

Edit: with mate-common-1.23.150

from mate-dev-scripts.

@raveit65

Can you test it again to use, please?

allow_repos = [
"raveit65/mate-common"
]

For more infomation, you can stop systemd release-server, and running release-server by hand on server, let's see the output messages.

from mate-dev-scripts.

@raveit65

I think I found the reason about allow_repos, I made a mistake here: https://github.com/yetist/release-server/blob/master/util.go#L28
https: //github.com, after https:, I type a space, this space should be removed.

from mate-dev-scripts.

@raveit65

For release-server 0.1.4, it use the config here: /etc/xdg/release-server/0.1.4/release-server.toml, dont forget to rename the 0.1.3 directory.

from mate-dev-scripts.

@raveit65

The release-server 0.1.4 binary is ready: https://github.com/yetist/release-server/releases/tag/v0.1.4

from mate-dev-scripts.

@yetist
Thanks a lot, you fixed the issue with 0.1.4
mate-common 1.23.152 was done with

allow_repos = [
"raveit65/mate-common"
]

https://pub.mate-desktop.org/releases/1.23/

from mate-dev-scripts.

root@yerba:/home/web-release/release-server-travis-github/0.1.4# ls -l /home/web-release/pub.mate-desktop.org/sources/mate-common/1.23/
total 192
lrwxrwxrwx 1 web-release web-release    78 Sep 26 14:40 mate-common-1.23.0.tar.xz -> /home/web-release/pub.mate-desktop.org/releases/1.23/mate-common-1.23.0.tar.xz
-rw-r--r-- 1 web-release web-release    41 Sep 26 14:40 mate-common-1.23.0.tar.xz.sha1sum
-rw-r--r-- 1 web-release web-release    42 Oct  5 14:38 mate-common-1.23.150.news
-rw-r--r-- 1 web-release web-release 69308 Oct  5 14:38 mate-common-1.23.150.tar.xz
-rw-r--r-- 1 web-release web-release    94 Oct  5 14:38 mate-common-1.23.150.tar.xz.sha256sum
-rw-r--r-- 1 web-release web-release    42 Oct  5 15:45 mate-common-1.23.152.news
-rw-r--r-- 1 web-release web-release 69184 Oct  5 15:45 mate-common-1.23.152.tar.xz
-rw-r--r-- 1 web-release web-release    94 Oct  5 15:45 mate-common-1.23.152.tar.xz.sha256sum
lrwxrwxrwx 1 web-release web-release    78 Sep 26 21:10 mate-common-1.23.1.tar.xz -> /home/web-release/pub.mate-desktop.org/releases/1.23/mate-common-1.23.1.tar.xz
-rw-r--r-- 1 web-release web-release    41 Sep 26 21:10 mate-common-1.23.1.tar.xz.sha1sum
lrwxrwxrwx 1 web-release web-release    78 Sep 28 09:05 mate-common-1.23.2.tar.xz -> /home/web-release/pub.mate-desktop.org/releases/1.23/mate-common-1.23.2.tar.xz
-rw-r--r-- 1 web-release web-release    41 Sep 28 09:05 mate-common-1.23.2.tar.xz.sha1sum
lrwxrwxrwx 1 web-release web-release    78 Sep 30 18:05 mate-common-1.23.3.tar.xz -> /home/web-release/pub.mate-desktop.org/releases/1.23/mate-common-1.23.3.tar.xz
-rw-r--r-- 1 web-release web-release    41 Sep 30 18:05 mate-common-1.23.3.tar.xz.sha1sum

root@yerba:/home/web-release/release-server-travis-github/0.1.4# ls -l /home/web-release/pub.mate-desktop.org/releases/1.23/
<cut>
-rw-rw-r-- 1 release     sftponly       69528 Sep 26 14:36 mate-common-1.23.0.tar.xz
-rw-r--r-- 1 web-release web-release       42 Oct  5 14:38 mate-common-1.23.150.news
-rw-r--r-- 1 web-release web-release    69308 Oct  5 14:38 mate-common-1.23.150.tar.xz
-rw-r--r-- 1 web-release web-release       94 Oct  5 14:38 mate-common-1.23.150.tar.xz.sha256sum
-rw-r--r-- 1 web-release web-release       42 Oct  5 15:45 mate-common-1.23.152.news
-rw-r--r-- 1 web-release web-release    69184 Oct  5 15:45 mate-common-1.23.152.tar.xz
-rw-r--r-- 1 web-release web-release       94 Oct  5 15:45 mate-common-1.23.152.tar.xz.sha256sum
-rw-rw-r-- 1 release     sftponly       69632 Sep 26 21:07 mate-common-1.23.1.tar.xz
-rw-rw-r-- 1 release     sftponly       69584 Sep 28 09:01 mate-common-1.23.2.tar.xz
-rw-rw-r-- 1 release     sftponly       69380 Sep 30 17:59 mate-common-1.23.3.tar.xz
<cut>

@yetist
Do you think it is possible to use symlinks for tarballs in pub.mate-desktop.org/sources/ ?
Same like we did for older releases.

from mate-dev-scripts.

@yetist
Do you think it is possible to use symlinks for tarballs in pub.mate-desktop.org/sources/ ?
Same like we did for older releases.

Done, yetist/release-server@c70ef21

from mate-dev-scripts.

Cool :)
I forgot to ask......
Does the release server handle different directory on server for mate-themes?
They are located under https://pub.mate-desktop.org/releases/themes/ .

from mate-dev-scripts.

Ok, i deleted some posts (mostly mine) to make this topic readable.
So this can be used as a HOWTO.

from mate-dev-scripts.

@raveit65 @flexiondotorg

New version release-server-0.1.6 is ready: https://github.com/yetist/release-server/releases/tag/v0.1.6

It can handle themes and update rss.xml file, rss have some config items, please see the example: https://github.com/yetist/release-server/blob/master/release-server.toml.example.

EDIT:
For mate-themes, I suggest to use

url_prefix = "https://pub.mate-desktop.org/sources/{{.Name}}/{{.ApiVersion}}"

in config file, they has the same rules.

from mate-dev-scripts.

@yetist
You last edit of you post confused me....
I don't understand what i should write for mate-themes in release-server.toml ?
Isn't this line for rss-feed?

url_prefix = "https://pub.mate-desktop.org/sources/{{.Name}}/{{.ApiVersion}}"

from mate-dev-scripts.

0.1.6 version has two lines about 'url_prefix' in example file, after release 0.1.6, I removed the other line.

from mate-dev-scripts.

Got it, {{.mate-themes}}/{{.ApiVersion}} are variables, so no edit is needed.

from mate-dev-scripts.

Please keep this line 'url_prefix = "https://pub.mate-desktop.org/sources/{{.Name}}/{{.ApiVersion}}"' in config file, the variables will use correct name by release-server.

Under the sources directory, they have the same rules, so we don't need to handle the mate-themes here.

from mate-dev-scripts.

I got it running. Stupid typo in release-server.toml.

allow_ips = [
"34.66.25.221","34.66.50.208","34.66.178.120","34.66.200.49","34.68.144.114",
"35.184.96.71","35.184.226.236","35.188.1.99","35.188.73.34","35.192.85.2",
"35.192.136.167","35.192.187.174","35.193.7.13","35.193.14.140","35.202.145.110",
"35.202.245.105","35.224.112.202","104.154.113.151","104.154.120.187","104.198.131.58""
]

Double double-quote at end of line :/

from mate-dev-scripts.

@raveit65

Cool, rss.xml works, screenshot from liferea:

And I think the old scripts should stop to update the rss.xml.

from mate-dev-scripts.

Yes, should be possible to disable old rss script.
Engrampa-1.23.2 was uploaded with caja via old sftp method and without the running new release-server.
That means your rss magic works when we upload tarballs by hand.

from mate-dev-scripts.

@flexiondotorg
Can you please remove rss parts from your release-runner.py script?
I don't want to break it ;)

from mate-dev-scripts.

I just released mate-power-manager 1.23.1 and i don't see a double entry in rss feed.
So, it might be Ok not to remove rss code from release-runner.py script?
I think we need it when uploading tarballs by hand.
Or does the release-server monitor /home/web-release/release.mate-desktop.org/uploads/ directory?
@yetist ^^^

from mate-dev-scripts.

I'd like to finish this topic and work on it. So again my questions.
@yetist
When i upload tarballs by hand via sftp to /home/web-release/release.mate-desktop.org/uploads/,
does your release-server monitor this directory and create a new rss entry?
If true we can remove rss code from old release-runner.py script.

from mate-dev-scripts.

Release-server does not monitor the upload directory, so if you upload file by hand, it will not generate rss and checksum files. The rss content and checksum files are from the github release.

from mate-dev-scripts.

Ok, thanks for clarification.
Than it is better to leave the old release-runner.py script like it is, to monitor the upload directory for manual uploads.
Topic is resolved.

from mate-dev-scripts.