ceph / chacra Goto Github PK
View Code? Open in Web Editor NEWA binary/file REST API to aid in multi-distro|arch|release management
A binary/file REST API to aid in multi-distro|arch|release management
This probably hasn't showed up in production because rabbitmq-server is already installed.
TASK [common : ensure chacra-celerybeat is enabled and running] ****************
fatal: [node1]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to start chacra-celerybeat.service: Unit rabbitmq-server.service not found.\n"}
Currently chacra reads an entire binary into memory and then writes it to a file. Some of Ceph's binaries are huge (ceph-test-dbg is 770MB) and this can easily overwhelm a chacra server so that it is unresponsive (SSH times out, etc).
From chacra/controllers/binaries/__init__.py
:
destination = os.path.join(dir_path, self.binary_name)
with open(destination, 'wb') as f:
try:
f.write(file_obj.getvalue())
except AttributeError:
f.write(file_obj.read())
Instead of prefering f.write(file_obj.getvalue())
, it would be good to prefer the file_obj.read()
iterator (If that really works?) and make that read/write something like 8192 bits at a time.
It currently removes:
But it doesn't remove the actual repository path (where the built repository goes to). This is trivial to fix as the repository model object has the path stored in it.
It uses: contents = request.POST.get('file', False)
which is wrong because it is a PUT request, not a POST
When deploying, to avoid the idiotic database connection error
This should be logged, especially when there are non-zero return codes
Some projects need to have automatic repository (for DEBs) or package (for RPMs) signing.
An unattended signing process requires a number of constraints, primarily is the ability to not require a passphrase (or to have it in clear text somewhere).
The current docs say it can be done but that really disables all repo creation (docs should get updated too)
Automatic repo creation is triggered when a new binary is uploaded. Projects that upload multiple binaries will benefit from not pilling up on repo creation requests.
So that the loop that keeps adding repos to the processing queue will not add repos that are already present in the queue.
As I'm working on teuthology support for shaman/chacra, I've mostly been running tests against centos 7 since there aren't any ubuntu 14.04 builds yet. Today I tried a few tests against 16.04 and am seeing issues.
First, the rest runs. This one uses my wip branch but is using gitbuilder:
http://pulpito.ceph.com/zack-2016-09-09_08:40:06-smoke-jewel-distro-basic-vps/
This run uses shaman:
http://pulpito.ceph.com/zack-2016-09-09_08:39:27-smoke-jewel-distro-basic-vps/
Focusing on one job, it appears there may something wrong with the repos being generated, and package_manager_version
isn't matching what the packages are actually using:
2016-09-09T14:55:05.362 INFO:teuthology.orchestra.run.vpm107:Running: u'echo deb https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/ ubuntu/16.04 main | sudo tee /etc/apt/sources.list.d/ceph.list'
2016-09-09T14:55:05.382 INFO:teuthology.orchestra.run.vpm107.stdout:deb https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/ ubuntu/16.04 main
2016-09-09T14:55:05.383 INFO:teuthology.orchestra.run.vpm107:Running: 'sudo apt-get update'
2016-09-09T14:55:05.666 INFO:teuthology.orchestra.run.vpm107.stdout:Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
2016-09-09T14:55:05.759 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
2016-09-09T14:55:05.808 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 InRelease
2016-09-09T14:55:05.830 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 Release
2016-09-09T14:55:05.839 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
2016-09-09T14:55:05.853 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:05.875 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:05.897 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:05.919 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:05.920 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:10 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
2016-09-09T14:55:05.941 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:05.963 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:05.984 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.006 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.028 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.050 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.072 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.094 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.116 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.138 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.160 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.182 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.204 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.226 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.248 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.270 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.292 INFO:teuthology.orchestra.run.vpm107.stdout:Err:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.292 INFO:teuthology.orchestra.run.vpm107.stdout: 404 Not Found
2016-09-09T14:55:06.314 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.337 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.359 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.397 INFO:teuthology.orchestra.run.vpm107.stdout:Fetched 94.5 kB in 0s (118 kB/s)
2016-09-09T14:55:07.397 INFO:teuthology.orchestra.run.vpm107.stdout:Reading package lists...
2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:W: The repository 'https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 Release' does not have a Release file.
2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:E: Failed to fetch https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/dists/ubuntu/16.04/main/binary-amd64/Packages 404 Not Found
2016-09-09T14:55:07.440 INFO:teuthology.orchestra.run.vpm107.stderr:E: Some index files failed to download. They have been ignored, or old ones used instead.
2016-09-09T14:55:07.441 INFO:teuthology.orchestra.run.vpm107:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=10.2.2-508-g9bfc0cf-1xenial ceph-mds=10.2.2-508-g9bfc0cf-1xenial ceph-common=10.2.2-508-g9bfc0cf-1xenial ceph-fuse=10.2.2-508-g9bfc0cf-1xenial ceph-test=10.2.2-508-g9bfc0cf-1xenial radosgw=10.2.2-508-g9bfc0cf-1xenial python-ceph=10.2.2-508-g9bfc0cf-1xenial libcephfs1=10.2.2-508-g9bfc0cf-1xenial libcephfs-java=10.2.2-508-g9bfc0cf-1xenial libcephfs-jni=10.2.2-508-g9bfc0cf-1xenial librados2=10.2.2-508-g9bfc0cf-1xenial librbd1=10.2.2-508-g9bfc0cf-1xenial rbd-fuse=10.2.2-508-g9bfc0cf-1xenial'
2016-09-09T14:55:07.483 INFO:teuthology.orchestra.run.vpm107.stdout:Reading package lists...
2016-09-09T14:55:07.644 INFO:teuthology.orchestra.run.vpm107.stdout:Building dependency tree...
2016-09-09T14:55:07.645 INFO:teuthology.orchestra.run.vpm107.stdout:Reading state information...
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-mds' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-common' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-fuse' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-test' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'radosgw' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'python-ceph' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs1' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs-java' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs-jni' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'librados2' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'librbd1' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'rbd-fuse' was not found
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 83, in __exit__
File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 101, in next
File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 19, in capture_traceback
File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/task/install.py", line 107, in _update_deb_package_list_and_install
File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/remote.py", line 192, in run
File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/run.py", line 399, in run
File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/run.py", line 166, in wait
CommandFailedError: Command failed on vpm107 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=10.2.2-508-g9bfc0cf-1xenial ceph-mds=10.2.2-508-g9bfc0cf-1xenial ceph-common=10.2.2-508-g9bfc0cf-1xenial ceph-fuse=10.2.2-508-g9bfc0cf-1xenial ceph-test=10.2.2-508-g9bfc0cf-1xenial radosgw=10.2.2-508-g9bfc0cf-1xenial python-ceph=10.2.2-508-g9bfc0cf-1xenial libcephfs1=10.2.2-508-g9bfc0cf-1xenial libcephfs-java=10.2.2-508-g9bfc0cf-1xenial libcephfs-jni=10.2.2-508-g9bfc0cf-1xenial librados2=10.2.2-508-g9bfc0cf-1xenial librbd1=10.2.2-508-g9bfc0cf-1xenial rbd-fuse=10.2.2-508-g9bfc0cf-1xenial'
The CVE is easy, but the python3 work is going to take a bit of effort. Simply bumping Sqlalchemy to 1.3.0
fixes both CVEs.
Update everything to work with Ubuntu 19.04
Is search working? the following URL returns an empty list:
To avoid:
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
From @alfredodeza on May 24, 2016 19:3
Binaries are huge, and for testing purposes there is no need to keep every binary and every repository built.
A configurable value (that defaults to two weeks) should be allowed to destroy a sha1/refs for a given project.
Copied from original issue: ceph/mita#75
Clients like chacractl will almost always want to forcefully try to recreate/create a repository, so the is_queued
flag needs to be reset to False
so that the celerybeat process will pick it up
For example, these two urls should provide the same output:
https://chacra.ceph.com/binaries/ceph/jewel/HEAD/ubuntu/xenial/ https://chacra.ceph.com/binaries/ceph/jewel/HEAD/ubuntu/16.04/
I don't think we care about this for rpm-based distros.
Implement a distinct flavors
endpoint to accommodate different kinds of binaries for the same kind of build.
This should be optional, and the default would be to show the non-distinct flavor (which would help in any instance that doesn't care about build "flavors")
Provide some simple validation for distro and distro_version. For example, if 'centos' is a known and accepted distro and that '7' is a valid distro_version for centos.
Because debug binaries are gigantic and it takes a while to POST them.
[chacractl.api.binaries][INFO ] POSTing file: /extra/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/release/10.2.2/rpm/el7/RPMS/aarch64/ceph-debuginfo-10.2.2-0.el7.aarch64.rpm
[requests.packages.urllib3.connectionpool][INFO ] Starting new HTTPS connection (1): chacra.ceph.com
[requests.packages.urllib3.connectionpool][DEBUG ] "POST /binaries/ceph/jewel/centos/7/arm64/ HTTP/1.1" 502 166
[chacractl.api.binaries][WARNING] 502 -> <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
Traceback (most recent call last):
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/bin/chacractl", line 6, in <module>
main.ChacraCtl()
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/main.py", line 38, in __init__
self.main(argv)
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/decorators.py", line 68, in newfunc
return f(*a, **kw)
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/main.py", line 82, in main
parser.dispatch()
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/tambo/dispatcher.py", line 21, in dispatch
return instance.main()
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/api/binaries.py", line 127, in main
self.post(url, filename)
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/api/binaries.py", line 82, in post
response.raise_for_status()
File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/requests/models.py", line 844, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://chacra.ceph.com/binaries/ceph/jewel/centos/7/arm64/
Just armhf. So the template for DEB distributions needs to be updated.
hosts file is
[test]
chacra1.xsky.com short_hostname=chacra1
deply.yml is
- hosts: test
user: root
roles:
- common
- repos
- statsd
vars:
app_name: "chacra"
use_ssl: true
wsgi_file: wsgi.py
wsgi_callable: application
ansible_ssh_port: 22
binary_root: "/opt/binaries"
repos_root: "/opt/repos"
branch: "master"
development_server: true
purge_repos: True
fqdn: "{{ inventory_hostname }}"
# required for first deploy
api_user: "admin"
api_key: "secret"
# graphite reporting for statsd
graphite_host: "3.shaman.xsky.com"
# required
graphite_api_key: '1234-asdf-1234'
# callbacks
callback_url: "https://3.shaman.xsky.com/api/repos/"
# required for first deploy
callback_user: "admin"
callback_key: "secret"
callback_verify_ssl: False
health_ping: true
health_ping_url: "https://3.shaman.xsky.com/api/nodes/"
# use_letsencrypt: True
# nginx_ssl_cert_path: "/etc/letsencrypt/live/{{ fqdn }}/fullchain.pem"
# nginx_ssl_key_path: "/etc/letsencrypt/live/{{ fqdn }}/privkey.pem"
nginx_ssl_cert_path: "/etc/ssl/certs/{{ fqdn }}-bundled.crt"
nginx_ssl_key_path: "/etc/ssl/private/{{ fqdn }}.key"
combine_deb_repos: False
The deploy command is 'ansible-playbook -vvv -i hosts deploy.yml'
it's failed with
TASK [common : populate the database for chacra] ***************************************************************************************************************************************************************************************
task path: /opt/chacra/src/chacra/deploy/playbooks/roles/common/tasks/postgresql.yml:81
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/commands/command.py
<chacra1.xsky.com> ESTABLISH SSH CONNECTION FOR USER: root
<chacra1.xsky.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d61856f8b9 chacra1.xsky.com '/bin/sh -c '"'"'ALEMBIC_CONFIG=/opt/chacra/src/chacra/alembic-prod.ini /usr/bin/python && sleep 0'"'"''
<chacra1.xsky.com> (1, '\n{"changed": true, "end": "2021-08-06 16:34:13.372460", "stdout": "==> LOADING ENVIRONMENT\\nbasepath=/opt/chacra/src/chacra,module_name=prod", "cmd": ["/opt/chacra/bin/pecan", "populate", "/opt/chacra/src/chacra/prod.py"], "failed": true, "delta": "0:00:02.304443", "stderr": "Traceback (most recent call last):\\n File \\"/opt/chacra/bin/pecan\\", line 8, in <module>\\n sys.exit(CommandRunner.handle_command_line())\\n File \\"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\\", line 96, in handle_command_line\\n runner.run(sys.argv[1:])\\n File \\"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\\", line 91, in run\\n self.commands[ns.command_name]().run(ns)\\n File \\"/opt/chacra/lib/python2.7/site-packages/chacra/commands/populate.py\\", line 34, in run\\n self.load_app()\\n File \\"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\\", line 162, in load_app\\n return load_app(self.args.config_file)\\n File \\"/opt/chacra/lib/python2.7/site-packages/pecan/core.py\\", line 213, in load_app\\n set_config(config, overwrite=True)\\n File \\"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\\", line 253, in set_config\\n config = conf_from_file(config)\\n File \\"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\\", line 183, in conf_from_file\\n *imp.find_module(module_name, [basepath])\\n File \\"/opt/chacra/src/chacra/prod.py\\", line 5, in <module>\\n from prod_db import sqlalchemy\\nImportError: No module named prod_db", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "/opt/chacra/bin/pecan populate /opt/chacra/src/chacra/prod.py", "removes": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2021-08-06 16:34:11.068017", "msg": "non-zero return code"}\n', '')
fatal: [chacra1.xsky.com]: FAILED! => {
"changed": true,
"cmd": [
"/opt/chacra/bin/pecan",
"populate",
"/opt/chacra/src/chacra/prod.py"
],
"delta": "0:00:02.304443",
"end": "2021-08-06 16:34:13.372460",
"invocation": {
"module_args": {
"_raw_params": "/opt/chacra/bin/pecan populate /opt/chacra/src/chacra/prod.py",
"_uses_shell": false,
"chdir": null,
"creates": null,
"executable": null,
"removes": null,
"stdin": null,
"warn": true
}
},
"msg": "non-zero return code",
"rc": 1,
"start": "2021-08-06 16:34:11.068017",
"stderr": "Traceback (most recent call last):\n File \"/opt/chacra/bin/pecan\", line 8, in <module>\n sys.exit(CommandRunner.handle_command_line())\n File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 96, in handle_command_line\n runner.run(sys.argv[1:])\n File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 91, in run\n self.commands[ns.command_name]().run(ns)\n File \"/opt/chacra/lib/python2.7/site-packages/chacra/commands/populate.py\", line 34, in run\n self.load_app()\n File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 162, in load_app\n return load_app(self.args.config_file)\n File \"/opt/chacra/lib/python2.7/site-packages/pecan/core.py\", line 213, in load_app\n set_config(config, overwrite=True)\n File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 253, in set_config\n config = conf_from_file(config)\n File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 183, in conf_from_file\n *imp.find_module(module_name, [basepath])\n File \"/opt/chacra/src/chacra/prod.py\", line 5, in <module>\n from prod_db import sqlalchemy\nImportError: No module named prod_db",
"stderr_lines": [
"Traceback (most recent call last):",
" File \"/opt/chacra/bin/pecan\", line 8, in <module>",
" sys.exit(CommandRunner.handle_command_line())",
" File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 96, in handle_command_line",
" runner.run(sys.argv[1:])",
" File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 91, in run",
" self.commands[ns.command_name]().run(ns)",
" File \"/opt/chacra/lib/python2.7/site-packages/chacra/commands/populate.py\", line 34, in run",
" self.load_app()",
" File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 162, in load_app",
" return load_app(self.args.config_file)",
" File \"/opt/chacra/lib/python2.7/site-packages/pecan/core.py\", line 213, in load_app",
" set_config(config, overwrite=True)",
" File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 253, in set_config",
" config = conf_from_file(config)",
" File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 183, in conf_from_file",
" *imp.find_module(module_name, [basepath])",
" File \"/opt/chacra/src/chacra/prod.py\", line 5, in <module>",
" from prod_db import sqlalchemy",
"ImportError: No module named prod_db"
],
"stdout": "==> LOADING ENVIRONMENT\nbasepath=/opt/chacra/src/chacra,module_name=prod",
"stdout_lines": [
"==> LOADING ENVIRONMENT",
"basepath=/opt/chacra/src/chacra,module_name=prod"
]
}
RUNNING HANDLER [common : restart app] *************************************************************************************************************************************************************************************************
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
chacra1.xsky.com : ok=31 changed=7 unreachable=0 failed=1
For systemd when deploying.
I make a run.py just like "shaman/config/run.py" to run chacra.
After the Jenkins build finished with success, the Repos in shaman and chacra is NULL. Would you tell me why is this or how to start (setup) chacra. Thank you. @andrewschoen @alfredodeza
The following is part of log:
cat /home/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/trusty/DIST/trusty/repo-extra.json
{
"version":"12.0.0-810-gda06227",
"package_manager_version":"12.0.0-810-gda06227-1trusty",
"build_url":"http://jenkins.star.com.cn/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty/97/",
"root_build_cause":"MANUALTRIGGER",
"node_name":"phy-10.118.203.10-ubuntu-trusty",
"job_name":"ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty"
}
cat /home/ubuntu/.chacractl
url = "http://192.168.34.252/"
user = "admin"
key = "secret"
ssl_verify = False
curl -X POST -H Content-Type:application/json --data @/home/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/trusty/DIST/trusty/repo-extra.json -u admin:secret http://192.168.34.252/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/extra/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1198 100 771 100 427 15724 8708 --:--:-- --:--:-- --:--:-- 16062
{"project_name": "ceph", "extra": {"build_url": "http://jenkins.star.com.cn/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty/97/", "root_build_cause": "MANUALTRIGGER", "version": "12.0.0-810-gda06227", "node_name": "phy-10.118.203.10-ubuntu-trusty", "job_name": "ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty", "package_manager_version": "12.0.0-810-gda06227-1trusty"}, "distro_version": "trusty", "path": null, "flavor": "default", "size": 0, "needs_update": true, "sha1": "da06227c7fdd71936fc152e29551268e78541e21", "is_queued": false, "is_updating": false, "type": "deb", "modified": "2017-02-27 17:09:20.166290", "signed": false, "ref": "master", "archs": ["x86_64"], "distro": "ubuntu"}+ /tmp/venv.Dp0IvzuZV8/bin/chacractl repo update ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default
[requests.packages.urllib3.connectionpool][DEBUG ] Starting new HTTP connection (1): 192.168.34.252
[requests.packages.urllib3.connectionpool][DEBUG ] http://192.168.34.252:80 "HEAD /repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/update HTTP/1.1" 200 0
[chacractl.api.repos][INFO ] POST: http://192.168.34.252/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/update
[requests.packages.urllib3.connectionpool][DEBUG ] Starting new HTTP connection (1): 192.168.34.252
[requests.packages.urllib3.connectionpool][DEBUG ] http://192.168.34.252:80 "POST /repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/update HTTP/1.1" 200 771
[chacractl.api.repos][INFO ] needs_update: True
[chacractl.api.repos][INFO ] project_name: ceph
[chacractl.api.repos][INFO ] is_queued: False
[chacractl.api.repos][INFO ] is_updating: False
[chacractl.api.repos][INFO ] extra: {u'build_url': u'http://jenkins.star.com.cn/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty/97/', u'root_build_cause': u'MANUALTRIGGER', u'version': u'12.0.0-810-gda06227', u'node_name': u'phy-10.118.203.10-ubuntu-trusty', u'job_name': u'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty', u'package_manager_version': u'12.0.0-810-gda06227-1trusty'}
[chacractl.api.repos][INFO ] ref: master
[chacractl.api.repos][INFO ] modified: 2017-02-28 09:00:31.405657
[chacractl.api.repos][INFO ] distro_version: trusty
[chacractl.api.repos][INFO ] sha1: da06227c7fdd71936fc152e29551268e78541e21
[chacractl.api.repos][INFO ] signed: False
[chacractl.api.repos][INFO ] path: None
[chacractl.api.repos][INFO ] flavor: default
[chacractl.api.repos][INFO ] distro: ubuntu
[chacractl.api.repos][INFO ] type: deb
[chacractl.api.repos][INFO ] archs: [u'x86_64']
[chacractl.api.repos][INFO ] size: 0
echo Check the status of the repo at: http://192.168.34.254/api/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/
Check the status of the repo at: http://192.168.34.254/api/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/
update_build_status completed ceph ubuntu trusty x86_64
state=completed
project=ceph
distro=ubuntu
distro_version=trusty
distro_arch=x86_64
submit_build_status POST completed ceph ubuntu trusty x86_64
http_method=POST
state=completed
project=ceph
distro=ubuntu
distro_version=trusty
distro_arch=x86_64
cat
cat
SHAMAN_URL=http://192.168.34.254/api/builds/ceph/
curl -X POST -H Content-Type:application/json --data @/home/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/trusty/DIST/trusty/build_status.json -u admin:secret http://192.168.34.254/api/builds/ceph/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 772 100 2 100 770 136 52441 --:--:-- --:--:-- --:--:-- 55000
{}+ rm -fr /tmp/install-deps.12694
Notifying upstream projects of job completion
Finished: SUCCESS
Using a similar solution for the callback system (see issue #110) implement a "ping" request to a remote destination so that when an instance fails it can be removed from rotation.
Configurable options should look like:
ping_frequency
(in seconds). Note it is critical to allow enough time in between checks on the receiving end to avoid being marked as down.ping_url
: the remote host and endpoint that receives the requestping_retries
: Retry count for failed ping requests (non 200 responses back)The code should determine what constitutes a working system for its various systems: database, rabbitmq, http)
From @alfredodeza on May 24, 2016 18:59
Whenever a repository is fully built and ready to be consumed, it would be nice to be able to define a URL to get a POST
with metadata information about the repository:
This would help if a separate system is in charge of understanding what has been built. For testing purposes it is crucial to be able to know what distro versions at what sha1 are ready to be consumed.
Copied from original issue: ceph/mita#74
If shaman gives out a chacra node that has little disk space builds fail when trying to upload the binaries. We need some way to remove chacra nodes from the pool that are not capable of storing anymore binaries or repos.
At some point when those repos are auto deleted we'll need a mechanism for adding them back to the pool as well.
Currently there are tasks for creating deb and rpm repos and for polling but they all exist in async/__init__.py
and it is becoming impossibly large.
These tasks should get separated, cleaned up, and use the proper Celery helpers to get them initialized and configured properly.
From @alfredodeza on May 24, 2016 18:55
This presents a problem because the current URL structure is:
project/ref/distro/distro_version/architecture/
So modifying this to accommodate sha1 (backwards incompatible) would look like
project/ref/sha1/distro/distro_version/architecture/
The other option would be to not make sha1 part of the url, but part of the API. That is, ref
would need to be told about the sha1
it is receiving, but this sounds problematic as the correctness would fully rely on the client and not the system. It would be tricky to ensure separation of sha1's across the board.
For example, when building a repository, code would need to be adapted to not mix different binaries from different sha1's.
I am not sure what is the best answer here, some users may not even want to have a sha1 in the url and that would be cumbersome to make it optional.
Copied from original issue: ceph/mita#73
We need documentation for adding new chacra nodes to shaman.
We can now build Windows binaries that are currently distributed via a zip file. Since this extension doesn't map to an RPM nor DEB repo, there is no easy way to access the artifacts via chacra repo url.
(chacra)vagrant@vagrant:~$ pip freeze --version chacractl
argparse==1.2.1
chacractl==0.0.8
requests==2.10.0
tambo==0.2.0
wsgiref==0.1.2
(chacra)vagrant@vagrant:~$ chacra/bin/chacractl binary create calamari/1.4.7/centos/7/x86_64/ calamari-server-1.4.7.rpm
[requests.packages.urllib3.connectionpool][INFO ] Starting new HTTPS connection (1): chacra.ceph.com
/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
[requests.packages.urllib3.connectionpool][DEBUG ] "HEAD /binaries/calamari/1.4.7/centos/7/x86_64/calamari-server-1.4.7.rpm/ HTTP/1.1" 404 0
[chacractl.api.binaries][INFO ] POSTing file: /home/vagrant/calamari-server-1.4.7.rpm
[requests.packages.urllib3.connectionpool][INFO ] Starting new HTTPS connection (1): chacra.ceph.com
/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
[requests.packages.urllib3.connectionpool][DEBUG ] "POST /binaries/calamari/1.4.7/centos/7/x86_64/ HTTP/1.1" 405 56
[chacractl.api.binaries][WARNIN] 405 -> {"message": "POST requests to this url are not allowed"}
Traceback (most recent call last):
File "chacra/bin/chacractl", line 6, in <module>
main.ChacraCtl()
File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/main.py", line 38, in __init__
self.main(argv)
File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/decorators.py", line 68, in newfunc
return f(*a, **kw)
File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/main.py", line 82, in main
parser.dispatch()
File "/home/vagrant/chacra/local/lib/python2.7/site-packages/tambo/dispatcher.py", line 21, in dispatch
return instance.main()
File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/api/binaries.py", line 136, in main
self.post(url, filepath)
File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/api/binaries.py", line 82, in post
response.raise_for_status()
File "/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/models.py", line 844, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 405 Client Error: Method Not Allowed for url: https://chacra.ceph.com/binaries/calamari/1.4.7/centos/7/x86_64/
(chacra)vagrant@vagrant:~$
For our chacra gitbuilder nodes we do not want to combine debian versions into one repo. We need to remove this configuration from all test chacra nodes: https://github.com/ceph/chacra/blob/master/deploy/playbooks/roles/common/templates/prod.py.j2#L116
In the example we found it also mixed two different sha1s into the same repo: https://2.chacra.ceph.com/r/ceph/master/b8c203fb64c80fe52a4d17562ddef39371347d42/ubuntu/xenial/flavors/default/pool/main/c/ceph/
The code will only create the file if it doesn't exist, no way to force it to create one if modifying the templating.
So that it can be used similarly to the logging
module, configurable with a secret key and automatic host detection.
These should fire to statsd that will end up relaying to a central Graphite instance.
Things to do:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.