1、introduce

Promgen is a configuration file generator for Prometheus. Promgen is a web application written with Django and can help you do the following jobs.

Create and manage Prometheus configuration files
Configure alert rules and notification options

all in one：192.168.3.134.

2、install redis

docker run --rm --net=host  --name redis -d redis

3、install mysql

docker run --rm  --name mysql --net=host -e MYSQL_ROOT_PASSWORD=secret -d percona/percona-server:latest

create db：

mysqladmin -uroot -h 192.168.3.134 -p create  promgen

4、install prometheus

cd /root
mkdir prometheus

cat > /root/prometheus/prometheus.yml<EOF
global:
  scrape_interval:     15s
  evaluation_interval: 15s

  external_labels:
    cluster_name: 'promtheus'

rule_files:
- "/etc/prometheus/promgen.rule.yml"

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093

scrape_configs:
  - job_name: 'promgen'
    file_sd_configs:
      - files:
        - "/etc/prometheus/promgen.json"

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
    file_sd_configs:
      - files:
        - "/etc/prometheus/blackbox.json"
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      - source_labels: []
        regex: .*
        target_label: __address__
EOF
chmod 777 -R prometheus


docker run --name prometheus --rm -d --net=host -v $(pwd)/prometheus:/etc/prometheus:rw  prom/prometheus:latest

5、initial

mkdir -p ~/.config/promgen
chmod 777 ~/.config/promgen
docker run --rm -it -v ~/.config/promgen:/etc/promgen/ line/promgen:v0.30 bootstrap

input：

  mysql://root:[email protected]/promgen
  redis://192.168.3.134:6379/0

configs need to modify:

cat ~/.config/promgen/promgen.yml

prometheus:
  url: http://192.168.3.134:9090/

6、migrate & createsuperuser

docker run --rm -v ~/.config/promgen:/etc/promgen/ line/promgen migrate
docker run --rm -it -v ~/.config/promgen:/etc/promgen/ line/promgen:v0.30 createsuperuser

7、 boot promgen web

docker run --rm -d  --net=host -v ~/.config/promgen:/etc/promgen/ line/promgen:v0.30

8、register msg

#docker exec 59e0b45181ed  /usr/local/bin/promgen register Default 192.168.3.134 9090

here the docker id is promgen web docker id last step.

9、boot monitor service

docker run --net=host -d  --rm -v ~/.config/promgen/:/etc/promgen/ -v /root/prometheus/:/etc/prometheus/:rw line/promgen:v0.30 worker --queues 192.168.3.134

10、others

left for future about install blackbox/altermanager and so on.

Hi Folks,

Finding hard to configure via a dashboard. Any help which is easier to setup things. ?

I use docker-compose up -d to create promgen service, it works well. But when I execute docker-compose run web createsuperuser, mysql outputs error like this:
[root@registry promgen]# docker-compose run web createsuperuser
Starting promgen_mysql_1 ... done
Starting promgen_redis_1 ... done

You have 25 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, promgen, sessions, sites, social_django.
Run 'python manage.py migrate' to apply them.

Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 101, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py", line 411, in _query
rowcount = self._do_query(q)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py", line 374, in _do_query
db.query(q)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py", line 277, in query
_mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1146, "Table 'promgen.auth_user' doesn't exist")

I use myql 5.5 in docker-compose,yaml, the output of docker-compose is like this:
[root@registry promgen]# docker-compose ps
promgen_alertmanager_1 /bin/alertmanager --config ... Up 0.0.0.0:9093->9093/tcp
promgen_base_1 /docker-entrypoint.sh sh - ... Exit 1
promgen_blackbox_1 /bin/blackbox_exporter --c ... Up 9115/tcp
promgen_mysql_1 docker-entrypoint.sh mysqld Up 3306/tcp
promgen_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp
promgen_redis_1 docker-entrypoint.sh redis ... Up 6379/tcp
promgen_web_1 /docker-entrypoint.sh prom ... Up 0.0.0.0:8080->8000/tcp
promgen_worker_1 /docker-entrypoint.sh work ... Up 8000/tcp

Dear Developer,

Can provide a simple tutorial on how to use promgen?
Also How Can We Configure Promgen When Prometheus is on a different server from Promgen?

Hi all,

I am trying to feed PushProx's service discovery into Promgen. (For those new to it, PushProx is a proxy that serves as a messaging proxy between Prometheus, and hosts behind an inaccessible network setup.)

So far I have both PushProx and Promgen working (yay!), and now I'm trying to sort out how to make Promgen aware of the hosts reported by PushProx's service discovery (SD).

PushProx publishes the targets it's aware of in (an admittedly barebones) JSON format, e.g.

[{"targets":["mybox.domain.com"],"labels":null}]

(The above is exposed via PushProx's /clients endpoint, but the dev-recommended way to use it is to copy the above via a cronjob somewhere file_sd_configs can read it.)

What would be a recommended way to automatically import the above targets into Promgen? I'm aware I could bypass Promgen and simply use Prometheus to absorb these targets, but I really want Promgen to be aware of them so my team can use it to configure rules and alerts. And I'm also aware we could skip the PushProx SD and enter the targets manually into Promgen... but I'm also hoping that won't be necessary as the manual effort won't be negligible due to the expected number of hosts we will be adding/updating.

Any and all advice will be welcome. Thanks!

not really understand why its happened
my DATABASE_URL
mysql://promgen:xxxxxxxxxxxxxxx@localhost/promgen

ubuntu@prometheus:~/.config/promgen$ docker run --rm -v ~/.config/promgen:/etc/promgen/ line/promgen migrate
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 164, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2002, 'Can\'t connect to local MySQL server through socket \'/run/mysqld/mysqld.sock\' (2 "No such file or directory")')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/promgen", line 11, in <module>
    load_entry_point('promgen', 'console_scripts', 'promgen')()
  File "/usr/src/app/promgen/manage.py", line 20, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 361, in execute
    self.check()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 390, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 64, in _run_checks
    issues = run_checks(tags=[Tags.database])
  File "/usr/local/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/local/lib/python3.6/site-packages/django/core/checks/database.py", line 10, in check_database_backends
    issues.extend(conn.validation.check(**kwargs))
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 9, in check
    issues.extend(self._check_sql_mode(**kwargs))
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 13, in _check_sql_mode
    with self.connection.cursor() as cursor:
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 164, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2002, 'Can\'t connect to local MySQL server through socket \'/run/mysqld/mysqld.sock\' (2 "No such file or directory")')

(Edited to format as code block)

Is there any way to install promgen inside simple VM, without docker?

HI developer：
I install Promgen for a development environment
python3 -m venv /path/to/virtualenv --ok
source /path/to/virtualenv/bin/activate --ok
pip install -e ./tmp/promgen --ok
pip install mysqlclient --ok
promgen bootstrap --error

(virtualenv) [root@python-rh7 tmp]# promgen bootstrap
Traceback (most recent call last):
File "/path/to/virtualenv/bin/promgen", line 11, in
load_entry_point('promgen', 'console_scripts', 'promgen')()
File "/path/to/virtualenv/lib64/python3.6/site-packages/pkg_resources/init.py", line 570, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/path/to/virtualenv/lib64/python3.6/site-packages/pkg_resources/init.py", line 2751, in load_entry_point
return ep.load()
File "/path/to/virtualenv/lib64/python3.6/site-packages/pkg_resources/init.py", line 2405, in load
return self.resolve()
File "/path/to/virtualenv/lib64/python3.6/site-packages/pkg_resources/init.py", line 2411, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
ModuleNotFoundError: No module named 'promgen.manage'

If I'm correct, Promgen use database to store the rule and configuration. So if we want to manage promgen rule as code it's impossible.
Please advice me in this case @kfdm
Thank you so much,

Hi, I recently updated to v0.25 promgen from my custom branch and im running into issues with static files not being served. See the demonstration below, where the css files are not being served.

I looked if I can get anything useful out of the log, but all I can get is:

web_1           | [05/Mar/2018 13:08:01] "GET /login/?next=/ HTTP/1.1" 200 5341
web_1           | [05/Mar/2018 13:08:02] "GET /static/css/bootstrap.min.css HTTP/1.1" 404 4529
web_1           | [05/Mar/2018 13:08:02] "GET /static/css/bootstrap-theme.min.css HTTP/1.1" 404 4535
web_1           | [05/Mar/2018 13:08:02] "GET /static/css/bootstrap-datetimepicker.min.css HTTP/1.1" 404 4544
web_1           | [05/Mar/2018 13:08:02] "GET /static/css/bootstrap-switch.min.css HTTP/1.1" 404 4536

I checked in the container whether the static files are being linked to via the ~/.cache/promgen/ (as i found in settings.py) and they seem they arent being exposed through there:

$ docker exec -it 3c6beb49d869 /bin/bash
bash-4.3# ls /root/.cache/
pip
bash-4.3# ls ~/.cache/
pip

My Dockerfile:

FROM line/promgen:v0.25

RUN chmod a+w /etc/prometheus
RUN chown -R promgen /etc/prometheus

USER root

COPY docker/wait-for-it.sh /
COPY docker/bootstrap.sh /

i had issues with some volumes inside docker to share configuration files, so thats what the RUN and USER are about, then i have some scripts that help me with automatic deployment. The USER root could be the culprit, however it seems unlikely that I would run into permission issues because I am running the container as root.

And finally for completeness, my docker-compose.yml:

version: '2'

services:
  config:
    image: ${AWS_DOCKER_REPOSITORY}/${PROMETHEUS_REPOSITORY}:${CONFIG_IMAGE}-$SHA1
    mem_limit: 16000k

  web:
    image: ${AWS_DOCKER_REPOSITORY}/${PROMETHEUS_REPOSITORY}:${PROMGEN_IMAGE}-$SHA1
    entrypoint: "/wait-for-it.sh"
    environment:
      PROMGEN_USER: ${PROMGEN_USER}
      PROMGEN_EMAIL: ${PROMGEN_EMAIL}
      PROMGEN_PASSWORD: ${PROMGEN_PASSWORD}
      CELERY_BROKER_URL: ${CELERY_BROKER_URL}
      DATABASE_URL: ${DATABASE_URL}
      SECRET_KEY: ${SECRET_KEY}
      PROMGEN_PORT: ${PROMGEN_PORT}
    links:
      - mysql
      - prometheus
      - redis
    command: ["-t", "0", "mysql:3306", "--", "sh", "/bootstrap.sh"]
    ports:
      - "8000:8000"
    mem_reservation: 256M
    volumes_from:
      - config:rw

  worker:
    image: ${AWS_DOCKER_REPOSITORY}/${PROMETHEUS_REPOSITORY}:${PROMGEN_IMAGE}-$SHA1
    environment:
      PROMGEN_USER: ${PROMGEN_USER}
      PROMGEN_EMAIL: ${PROMGEN_EMAIL}
      PROMGEN_PASSWORD: ${PROMGEN_PASSWORD}
      CELERY_BROKER_URL: ${CELERY_BROKER_URL}
      DATABASE_URL: ${DATABASE_URL}
      SECRET_KEY: ${SECRET_KEY}
    links:
      - mysql
      - prometheus
      - redis
    command: worker -l info --queues localhost,celery
    mem_reservation: 64M
    volumes_from:
      - config:rw

  mysql:
    image: mysql
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    volumes:
      - /opt/data/mysql/:/var/lib/mysql
    mem_reservation: 64M

  redis:
    image: redis
    mem_reservation: 64M

I run promgen by docker-compose.yaml. But login the web, what is username and password?

prometheus version is 2.0

call method check_rules in prometheus.py fail

def check_rules(rules):
    with tempfile.NamedTemporaryFile(mode='w', encoding='utf8') as fp:
        logger.debug('Rendering to %s', fp.name)
        # Normally we wouldn't bother saving a copy to a variable here and would
        # leave it in the fp.write() call, but saving a copy in the variable
        # means we can see the rendered output in a Sentry stacktrace
        rendered = render_rules(rules)
        fp.write(rendered)
        fp.flush()

        try:
            subprocess.check_output([
                settings.PROMGEN['rule_writer']['promtool_path'],
                'check-rules',
                fp.name
            ], stderr=subprocess.STDOUT)
        except subprocess.CalledProcessError as e:
            raise Exception(rendered + e.output.decode('utf8'))

error message

promtool: error: expected command but got "check-rules", try --help

Admin could create lots of users, yet each user must only view or update there own services/projects.

HI developer：
When I use the test rules, the backstage is wrong, 500, how do you solve it in general? How do you use promgen to generate Prometheus rules?

From the alertmanager log, it shows 403 when sending notify to Promgen.

level=error ts=2019-12-08T09:57:48.364Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 403: http://172.24.72.111:8082/alert"

Hi team,
I upgrade the latest Promgen and do some test here. When I creating the new rule and run test rule here. The result is 404 not found.

[05/Sep/2019 07:27:45] "GET /proxy/v1/alerts HTTP/1.1" 200 53892
[05/Sep/2019 07:27:45] "GET /proxy/v1/silences HTTP/1.1" 200 3079
Not Found: /rule/0/test
[05/Sep/2019 07:27:48] "POST /rule/0/test HTTP/1.1" 404 1759

Can you have a look?
It not happened in the old rule.

Dear All,

Currently i have created an installation guide for Promgen Web on CentOS 7, how can i install and configure Promgen Worker On CentOS 7?

@kfdm

so long not see. haha

docker pull line/promgen:v0.31
docker pull line/promgen:master
docker pull line/promgen:latest

do not work well.

visit /static/*.css 404

Hi,

promgen looks exactly what i need to run prometheus since I want to change the configurations of alerts and nodes dynamically. I've started with taking your docker-compose that you have in your project and altering it to include both the mysql and redis databases (later on I can move them outside of this docker). I can successfully start the docker containers and all of them seem to be running just fine, but I'm having issues configuring promgen.

I add a exporter successfully to the UI and the test functionality confirms it working, however the Prometheus server does not get it added to its targets. Further when I attempt to add a alert rule i get internal server errors (and nothing visible in the promgen log). What am I doing wrong?

PS: i ran the DB initialization scripts manually (i plan to run them automatically later).

The prometheus server has been added:

The compose (WIP) used:

version: '2'
services:
  redis:
    image: redis
    container_name: redis
    network_mode: bridge
    restart: always
    ports:
      - 6379:6379

  mysql:
    image: mariadb
    container_name: mysql
    network_mode: bridge
    restart: always
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: pypromgen
      MYSQL_USER: promgen
      MYSQL_PASSWORD: promgen

  base:
    build: .
    image: promgen
    network_mode: bridge
    command: echo ""
    environment:
      CELERY_BROKER_URL: redis://redis:6379/0
      DATABASE_URL: mysql://promgen:promgen@mysql:3306/pypromgen
      SECRET_KEY: asdf

  prep:
    extends: base
    container_name: prep
    command:  sh -c "promgen migrate"
    links:
      - mysql
      - redis
    environment:
      DEBUG: 1

  web:
    extends: base
    container_name: web
    command: promgen runserver 0.0.0.0:8000
    links:
      - mysql
      - redis
      - prep
      - nodeexporter
    ports:
      - "8000:8000"
    environment:
      DEBUG: 1

  worker:
    extends: base
    container_name: worker
    command: worker -l info --queues localhost,celery
    links:
      - prometheus
      - redis
    volumes:
      - ./docker:/etc/prometheus

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    network_mode: bridge
    links:
      - alertmanager
      - blackbox
    ports:
      - "9090:9090"
    volumes:
      - ./docker:/etc/prometheus

  blackbox:
    image: prom/blackbox-exporter
    container_name: blackbox
    network_mode: bridge
    volumes:
      - ./docker:/etc/prometheus

  nodeexporter:
    image: prom/node-exporter
    container_name: nodeexporter
    network_mode: bridge
    ports:
      - 9100:9100

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    network_mode: bridge
    links:
      - web
    ports:
      - "9093:9093"
    volumes:
      - ./docker/alertmanager.yml:/etc/alertmanager/config.yml

Validation of redis URLs such as redis://redis:6379/0 (i.e. without a tld part) fails during bootstrap process with the error: Enter a valid URL.
When using docker this URL will often lack a tld part.

The root cause is the Django URLValidator utility.
It seems to be directed more at end user input on a web page than for validating admin input.

Ticket #25418 tracks progress on the topic in the Django Project. Ticket #9202 is also related.

I would suggest either removing validation or replacing it with something more meaningful, for example a simple RegEx or an actual check which tests whether the URL's target is reachable.

What are your thoughts on the subject?

promgen.models.Host will be refactored so that a Host model is unique, but can belong to multiple Farm objects. This will help correct our metrics and make it easier to build a host report

I saw lots of code using urljoin, if we let the second argument starts with '/', urljoin will simply take the second argument as the path of the URL.

TestsCode

>>> urljoin("http://abc:9090/prometheus/",  "/aa/bb" )
'http://abc:9090/aa/bb'
>>> urljoin("http://abc:9090/prometheus",  "/aa/bb" )
'http://abc:9090/aa/bb'

Hi Paul,

I was trying to overwrite an alert for a single host, but got a "Page not found" error. I had seen this before, so I updated via docker to the (then) most recent version, 0.40, but the issue remains.

I noticed the resulting URL is to a non-existing service (e.g. http://my_server:8000/service/26 ) so maybe that is the issue... Note that attempting to add an overwrite to a project doesn't yield the error, it's only for a host.

I recorded a 40-sec video: https://www.loom.com/share/8976856bf2034428b24ab92df7030db4

(I'd like to take it down after you no longer need it, please let me know)

Thanks again,
Saul

Hello,
Thank you for great work.
I was looking for ways to make easy to simplify management of prometheus.

Are there any way to add settings to monitor specific port, e.g. 2181 port, on every node of farm at once using promgen?
I monitor zookeeper, kafka and other clusters with prometheus and so there are cases where I want to add settings for port monitoring with one operation.

I tried to use promgen, it looks like I need to regist target URL one by one using HTTP Checks form .
If there is any way, I am glad if you tell me that.

Thanks,

I just create a issue #210 and fixed the ServiceList filter codes.

But I found that the ProjectList, RulesList (/rule), UrlsList, NotifiorsList will display all the entries by default.
It's not very safe because we have lots of users to manage different projects. We should ensure that each user could only have the update privilege of there own projects,rules, urls etc.

When trying to register a service with an already existing name the view crashes (500: Internal Server Error).

traceback.txt

Steps to reproduce:

1. Create shard: "Shard 1"
2. Create shart: "Shard 2"
3. Register service in "Shard 1" with name "Default Service"
4. Register service in "Shard 2" with name "Default Service"

Interestingly this issue does not occur when creating projects with the same name. Here, an appropriate error message is displayed.

The err msg is the followning.

worker_1 | [2017-09-14 09:47:18,643: INFO/MainProcess] Received task: promgen.prometheus.write_config[21b64bc9-a37e-4fac-a03f-777a6cd2e17f]
worker_1 | [2017-09-14 09:47:18,647: ERROR/ForkPoolWorker-1] Task promgen.prometheus.write_config[21b64bc9-a37e-4fac-a03f-777a6cd2e17f] raised unexpected: PermissionError(13, 'Permission denied')
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 367, in trace_task
worker_1 | R = retval = fun(*args, **kwargs)
worker_1 | File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 622, in protected_call
worker_1 | return self.run(*args, **kwargs)
worker_1 | File "/usr/src/app/promgen/prometheus.py", line 130, in write_config
worker_1 | with atomic_write(path, overwrite=True) as fp:
worker_1 | File "/usr/local/lib/python3.5/contextlib.py", line 59, in enter
worker_1 | return next(self.gen)
worker_1 | File "/usr/local/lib/python3.5/site-packages/atomicwrites/init.py", line 149, in _open
worker_1 | with get_fileobject() as f:
worker_1 | File "/usr/local/lib/python3.5/site-packages/atomicwrites/init.py", line 166, in get_fileobject
worker_1 | delete=False, **kwargs)
worker_1 | File "/usr/local/lib/python3.5/tempfile.py", line 549, in NamedTemporaryFile
worker_1 | (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
worker_1 | File "/usr/local/lib/python3.5/tempfile.py", line 260, in _mkstemp_inner
worker_1 | fd = os.open(file, flags, 0o600)
worker_1 | PermissionError: [Errno 13] Permission denied: '/etc/prometheus/tmpluryoyb'

Hello,

we recently started using the promgen to manage the configuration for Prometheus and Alertmanager. for our use case we monitor 500+ AWS EC2 instances in our AWS account.

in our current deployment we use ec2_sd_config to discover the new EC2 instances and with ec2_metadata we categorise the instances.

we do not see any option in promgen through which we can leverage the ec2_sd_config service discovery.

if there is already a feature in promgen then do let us know.

This was pointed out in #61 and should be fixed to remove confusion.

https://github.com/line/promgen/blob/master/promgen/templates/promgen/rule_detail.html#L71
https://github.com/line/promgen/blob/master/promgen/templates/promgen/rule_block.html#L14

If we put a expression like {{$instance}} in label value, the html will not render. So we suggest it in a
code block.

-                    <li class="label label-primary">{{k}}:{{v}}</li>
+                    <li class="label label-primary">
+                       <code v-pre>{{k}}:{{v}}</pre></code>
+                    </li>

All Prometheus and Alertmanager routes will be refactored under the /api/ route and correspond to promgen.proxy module

hill，i run promgen by docker compose,and the project running seems normally.but do not know how to configure the dashboard.finding the project lack of a detailed user manual.can you supply more detail user manual or a detail configuration demo.very thanks

On a linux mint installation, I run docker-compose up on a freshly cloned repo, and get many errors from standard_init_linux.go about exec user process.

Console output:

Starting promgen_base_1 ... 
promgen_redis_1 is up-to-date
promgen_mysql_1 is up-to-date
Starting promgen_base_1 ... done
Starting promgen_web_1  ... done
Starting promgen_alertmanager_1 ... done
Starting promgen_prometheus_1   ... done
Starting promgen_worker_1       ... done
Attaching to promgen_redis_1, promgen_mysql_1, promgen_blackbox_1, promgen_base_1, promgen_web_1, promgen_alertmanager_1, promgen_prometheus_1, promgen_worker_1
base_1          | standard_init_linux.go:190: exec user process caused "no such file or directory"
prometheus_1    | level=info ts=2019-08-19T22:13:08.699Z caller=main.go:293 msg="no time or size retention was set so using the default time retention" duration=15d
prometheus_1    | level=info ts=2019-08-19T22:13:08.699Z caller=main.go:329 msg="Starting Prometheus" version="(version=2.12.0, branch=HEAD, revision=43acd0e2e93f9f70c49b2267efa0124f1e759e86)"
web_1           | standard_init_linux.go:190: exec user process caused "no such file or directory"
worker_1        | standard_init_linux.go:190: exec user process caused "no such file or directory"
prometheus_1    | level=info ts=2019-08-19T22:13:08.699Z caller=main.go:330 build_context="(go=go1.12.8, user=root@7a9dbdbe0cc7, date=20190818-13:53:16)"
prometheus_1    | level=info ts=2019-08-19T22:13:08.699Z caller=main.go:331 host_details="(Linux 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 6b675e3f84a0 (none))"

Following steps from https://hub.docker.com/r/kfdm/promgen/ also results in standard_init_linux.go:190: exec user process caused "no such file or directory" error at the end of the first Docker domand.

Is there a missing step in the installation guide for proper configuration?

the variable should be PROMGEN_CONFIG_DIR not just CONFIG_DIR - this actually makes the current docker image at the master tag unusable

related changes in
1ef8f4d#diff-8183575a528efbf0c406468637fb6253

hill

I was trying to get up and running to do some testing and ran into some issues. I started with:

joey@mbp : ~/Desktop/projects/promgen > docker-compose up -d
...
Creating promgen_redis_1 ... done
Creating promgen_blackbox_1 ... done
Creating promgen_base_1 ... done
Creating promgen_mysql_1 ... done
Creating promgen_web_1 ... done
Creating promgen_alertmanager_1 ... done
Creating promgen_prometheus_1 ... done
Creating promgen_worker_1 ... done

docker-compose run web createsuperuser

joey@mbp : ~/Desktop/projects/promgen > docker-compose run web createsuperuser

Starting promgen_redis_1 ... done
Starting promgen_mysql_1 ... done

You have 34 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, authtoken, contenttypes, promgen, sessions, sites, social_django.
Run 'python manage.py migrate' to apply them.
...
django.db.utils.ProgrammingError: (1146, "Table 'promgen.auth_user' doesn't exist")

I logged into the promgen_web_1 container and attempted to run manage.py migrate, but I ran into other issues, most notably that it was getting stuck importing celery because it appears to be importing https://github.com/line/promgen/blob/master/promgen/celery.py rather than the celery library.

How do we setup promgen in a non docker environment?
There is a dev setup in the readme, but I am afraid that its not a recommended for a near production setup.
Also is python3 a requirement for promgen, can we run this on 2.7 ?

Freshly installed promgen.

my config.yml:

---
prometheus:
  url: http://127.0.0.1:9090/
db:
  logging: true
  dsn: sqlite://./db
config_writer:
  path: /tmp/prom.json
  notify:
    - http://localhost:9090/api/v1/config
rule_writer:
  rule_path: /tmp/prom.rule
  promtool_path: /usr/bin/promtool
  notify:
    - http://localhost:9090/api/v1/rule/
alert_senders:
  - module: Ikachan
    url: http://ikachan.localhost/
  - module: MailNotify
    smtp_server: smtp.localhost
  - module: Webhook
  - module: Alerta
    alerta_url: http://alerta.localhost
password: sea_monitoring
# Default Exporter Settings
# Examples from https://github.com/prometheus/prometheus/wiki/Default-port-allocations
default_exporters:
  node: 9100
  nginx: 9113
  mysqld: 9104
  apache: 9117

[ERROR] [Promgen] SQLite3::ConstraintException: columns type, params are not unique: INSERT INTO `server_directory` (`type`, `params`) VALUES ('Promgen::ServerDirectory::DB', '{}')
2017-01-26 18:48:31 - Sequel::UniqueConstraintViolation - SQLite3::ConstraintException: columns type, params are not unique:
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:108:in `step'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:108:in `block in each'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:107:in `loop'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:107:in `each'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:158:in `to_a'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:158:in `block in execute'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:95:in `prepare'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:134:in `execute'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/adapters/sqlite.rb:189:in `block (2 levels) in _execute'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/database/logging.rb:48:in `log_connection_yield'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/adapters/sqlite.rb:189:in `block in _execute'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/database/connecting.rb:251:in `block in synchronize'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/connection_pool/threaded.rb:105:in `hold'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/database/connecting.rb:251:in `synchronize'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/adapters/sqlite.rb:180:in `_execute'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/adapters/sqlite.rb:151:in `execute_insert'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/dataset/actions.rb:986:in `execute_insert'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sequel-4.35.0/lib/sequel/dataset/actions.rb:338:in `insert'
	/root/promgen/lib/promgen/repo/server_directory_repo.rb:49:in `insert'
	/root/promgen/lib/promgen/web/route/server_directory_route.rb:43:in `block in <class:Web>'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/nulllogger.rb:9:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/static.rb:124:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/lint.rb:49:in `_call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/lint.rb:37:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/showexceptions.rb:24:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/chunked.rb:54:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/content_length.rb:15:in `call'
	/usr/local/rvm/gems/ruby-2.3.3/gems/rack-1.6.4/lib/rack/handler/webrick.rb:88:in `service'
	/usr/local/rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/webrick/httpserver.rb:140:in `service'
	/usr/local/rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/webrick/httpserver.rb:96:in `run'
	/usr/local/rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/webrick/server.rb:296:in `block in start_thread'
10.250.250.43 - - [26/Jan/2017:18:48:31 +0000] "POST /server_directory/register HTTP/1.1" 500 574802 0.0531

@kfdm

I am trying to setup promgen for our prometheus environment, however I am unable to find out a standard procedure about installing and configuring promgen.

I have downloaded the code https://github.com/line/promgen/archive/master.zip and can start promgen in a python venv by running "#promgen runserver" but unable to understand how I am going to deploy worker in each prometheus server and start promgen worker and integrate the same with promgen web running in promgen:8000.

I am very much interested to deploy promgen and running it, so any help will be highly appreciated.

Thanks & regards,
Suman

I tried to use the api of the document to get the services or projects exporter or rules configurations ,but failed . I just want to use api to get service and project exporters and rules ,and prometheus use cron to pull configurations. thank you.

Hi Team,
Currently, promgen configuration only support some params. I have some configuration need migrate to Promgen like:

scrape_configs:
   - job_name: 'test_application'
     metrics_path: 'prometheus'
     params:
       token: ['xxxxxx']
     file_sd_configs:
     - files:
       - file-sd/jennifer_node*.json

jennifer_node*.json have this content

  {
    "targets": [
      "http://atrkadas-stream1.travel-dev.dev.jp.local/healthcheck.html",
      "http://atrsrm-api.travel-dev.dev.jp.local/healthcheck.html"
    ],
    "labels": {
      "type": "internal",
      "platform": "cloud"
    }

Can we add some params like token, type, platform .... ?
It should be useful.
Thank you,

Dear All,

Is it possible to install promgen on Centos 7?

What are the prerequisite?

Based on what i understand it requires Python 3, REDIS, MySQL to be installed

What are the steps to configure, install and run?

Hi Team,
I tried to install Promgen via docker-compose and manual as well. Anyway, In both of way, I can not use TestQuery function.
It always show
jquery.min.js:4 POST http://127.0.0.1:8080/rule/0/test 500 (Internal Server Error)

Log file from docker-compose

> web_1           | [02/Aug/2019 06:25:32] "POST /rule/0/test HTTP/1.1" 500 16836
web_1           | Internal Server Error: /rule/0/test
web_1           | Traceback (most recent call last):
web_1           |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
web_1           |     response = get_response(request)
web_1           |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
web_1           |     response = self.process_exception_by_middleware(e, request)
web_1           |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
web_1           |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
web_1           |   File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
web_1           |     return view_func(*args, **kwargs)
web_1           |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 71, in view
web_1           |     return self.dispatch(request, *args, **kwargs)
web_1           |   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/mixins.py", line 52, in dispatch
web_1           |     return super().dispatch(request, *args, **kwargs)
web_1           |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 97, in dispatch
web_1           |     return handler(request, *args, **kwargs)
web_1           |   File "/usr/src/app/promgen/views.py", line 1241, in post
web_1           |     result = util.get(url, {'query': query}).json()
web_1           |   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 897, in json
web_1           |     return complexjson.loads(self.text, **kwargs)
web_1           |   File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
web_1           |     return _default_decoder.decode(s)
web_1           |   File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
web_1           |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
web_1           |   File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
web_1           |     raise JSONDecodeError("Expecting value", s, err.value) from None
web_1           | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

Can you have a look?
Thank you
CC: @kfdm

This will correspond to routes in promgen.rest module. The goal here is to move all Promgen specific APIs out of promgen.view and fully embrace django-restframeowrk

hill, i run promgen by docker,and the project running seems normally.but do not know how to create the promgen.json

Hi,
I made a mistake in the configuration, I removed the docker deleted the image and cler all the docker cache, but when im running this command again, I receiving that SECRET_KEY , DATABASE_URL , CELERY_BROKER_URL are exists.
how can I change these parameters

please advise

ubuntu@prometheus:~$ docker run --rm -it -v ~/.config/promgen:/etc/promgen/ line/promgen bootstrap
Bootstrapping Promgen
Setting SECRET_KEY exists
Setting DATABASE_URL exists
Setting CELERY_BROKER_URL exists

Hi!

I was able to get the latest release of promgen (v0.38) running via Virtualenv with this guide.

I am now trying to deploy a worker with the below command, as per these instructions:

promgen register promshard prometheus001 9090

But I get the below error:

Unknown command: 'register'. Did you mean register-server?

Is there perhaps an updated deployment guide I should be looking at?

Thanks!

Installation Promgen On CentOS 7 (Updated - 11 Jan 2018)

This documentation outlines the steps required to install, configure and run promgen application on CentOS 7.4 minimal server.

Section 1 - System Components Installation

Step 1: Install the IUS repository

The IUS repository provides newer versions of some software in the official CentOS and Red Hat repositories. The IUS repository depends on the EPEL repository.

# yum -y install https://centos7.iuscommunity.org/ius-release.rpm

Step 2: Installation of Git, Python 3.6 and All Requirements

# yum install python36u python36u-devel python36u-setuptools git gcc libxml2-devel libxslt-devel libffi-devel graphviz openssl-devel redhat-rpm-config -y

Step 3: Install PIP

easy_install-3.6 pip

Step 4: Create Python Symlink

# sudo ln -s /usr/bin/python3.6 /usr/bin/python3

Step 4: Check Python 3 Version

# python3 --version

Section 2 - MySQL Database Configurations

Step 1: Add MariaDB Yum Repository

Create the MariaDB repo file by issuing the following command

# sudo vi /etc/yum.repos.d/MariaDB.repo

Paste the following contents in MariaDB.repo and save the file

[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

Step 2 - Install MariaDB Server, Client and Development Tools

# sudo yum install MariaDB-server MariaDB-devel MariaDB-client -y

Once the installation is complete, we'll start the MariaDB daemon with the following command:

# systemctl start mariadb

To ensure that the MariaDB daemon is started, issue the following command

# sudo systemctl status mariadb

Finally after ensuring that MariaDB is able to start successfully, we will use the systemctl enable command to ensure that MariaDB will start at boot.

# sudo systemctl enable mariadb

Step 3 - Securing the MariaDB Server

MariaDB includes a security script to modify some of the less secure default options for items such as remote root logins and sample users. Issue the following command to execute the security script:

# sudo mysql_secure_installation

The script provides a detailed explanation for every step. The first prompts asks for the root password, which hasn't been set so we'll press ENTER as it recommends. Next, we'll be prompted to set that root password, which we'll do.

Then, we'll accept all the security suggestions by pressing Y and then ENTER for the remaining prompts, which will remove anonymous users, disallow remote root login, remove the test database, and reload the privilege tables.

Step 4 - Create the promgen Database

Login as root user with the password set earlier at step 2

# mysql -u root -p

Create promgen database

> CREATE DATABASE promgen;

Exit MySQL command utility

> quit

Section 3 - Deploying Promgen Application

Step 1: Install the Required Python Packages

# pip3 install mysqlclient

Step 2: Clone the Git Repository

# git clone https://github.com/line/promgen.git

Step 3: Create promgen Setting Directory

Issue the following command to create the reuqired folder to store promgen settings

# sudo mkdir -p ~/.config/promgen

After creation of the folder, we configure the folder permission to allow for write access

# sudo chmod 777 ~/.config/promgen

Finally for ease of access, create a symlink to promgen setting directory from /promgen_cfg

# sudo ln -sf ~/.config/promgen /promgen_cfg

Step 4: Generate The Required Configuration Files (CELERY_BROKER_URL, DATABASE_URL, promgen.yml, SECRET_KEY)

# promgen bootstrap

Step 5: Install promgen from setup.py

# python3 setup.py install

Step 6: Run Database Migrations

# promgen migrate

Step 7: Create a Super User**

# promgen createsuperuser

Step 8: Collect Static Files

# promgen collectstatic

Step 9: Test the Application

promgen runserver 0.0.0.0:8000

Section 4 - Web Server Deployment

A simple WSGI front end using gunicorn will be used for this guide and also Nginx will be the web server. We will also utilize supervisord to enable service persistence.

Step 1: Nginx Web Server Installation

Install Nginx Using Yum Command

yum install nginx -y

Once the installation is complete, we'll start the nginx daemon with the following command:

# systemctl start nginx

To ensure that the nginx daemon is started, issue the following command

# sudo systemctl status nginx

Finally after ensuring that nginx is able to start successfully, we will use the systemctl enable command to ensure that nginx will start at boot.

# sudo systemctl enable nginx

Once nginx is installed, save the following configuration to /etc/nginx/sites-available/promgen.conf. Be sure to replace host name with the domain name or IP address of your installation.

Content of promgen.conf

server {
    listen 80;
    server_name host name;

  access_log /var/log/nginx/promgen.access_log main;
  error_log  /var/log/nginx/promgen.error_log;

      location /static/ {
        alias /pg_static/;
    }

  location / {
    proxy_pass http://127.0.0.1:8000;
    proxy_set_header X-Forwarded-Host $server_name;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Create a symlink in the sites-enabled directory to the configuration file you just created.

# cd /etc/nginx/conf.d/
# sudo ln -s /etc/nginx/sites-available/promgen.conf

Restart the nginx service to use the new configuration

# sudo systemctl nginx restart

Step 2: Configure promgen static file location

Create a folder /pg_static/ and copy all the files from promgen collectstatic to this folder. Grant nginx read permission to this folder

# sudo cp -r /root/.cache/promgen/* /pg_static/
# sudo chown -R nginx /pg_static/*
# sudo chmod 755 /pg_static/* 

Step 3: Installation of gunicorn package

pip3 install gunicorn

Save the following configuration in the root promgen installation path as gunicorn_config.py (e.g. /promgen/gunicorn_config.py per our example installation).

Content of gunicorn_config.py

command = '/usr/bin/gunicorn'
bind = '127.0.0.1:8000'
workers = 3

Step 4: Installation of supervisord

# sudo yum install supervisord -y

Once the installation is complete, we'll start the supervisord daemon with the following command:

# systemctl start nginx

To ensure that the supervisord daemon is started, issue the following command

# sudo systemctl status supervisord

Finally after ensuring that supervisord is able to start successfully, we will use the systemctl enable command to ensure that supervisord will start at boot.

# sudo systemctl enable supervisord

Once the service is verified to run correctly, add the following to /etc/supervisord.conf

[program:promgen]
command = gunicorn -c /promgen/gunicorn_config.py "promgen.wsgi:application" 

Step 5: Restart the required services to take effect all the changes

# sudo systemctl supervisord restart
# sudo systemctl nginx restart

Step 6: Use promgen

On any web browser, navigate to http://hostname-of-promgen

Congratulation, promgen is successfully installed and configured

Hello.
I tried to get promgen installed yesterday, but should've looked ahead in the readme, because when I started the process I didn't have MySQL or Redis. Well, did the MySQL part, but ran out of time, and so had no broker url to pass to the installation, and I had the smart idea of Ctrl+C'ing out of it, expecting that if I just do sudo docker run --rm -it -v ~/.config/promgen:/etc/promgen/ line/promgen bootstrap again tomorrow, it'll work.
Nope. Now it spits out a gigantic nasty stack trace to me, so I think I broke a lot of things. Help?
Stack trace in question:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 274, in get_new_connection
conn = Database.connect(**conn_params)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/init.py", line 86, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py", line 204, in init
super(Connection, self).init(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2002, 'Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (2 "No such file or directory")')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/bin/promgen", line 11, in
load_entry_point('promgen', 'console_scripts', 'promgen')()
File "/usr/src/app/promgen/manage.py", line 20, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/init.py", line 364, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/site-packages/django/core/management/init.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 327, in execute
self.check()
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 359, in check
include_deployment_checks=include_deployment_checks,
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 346, in _run_checks
return checks.run_checks(**kwargs)
File "/usr/local/lib/python3.5/site-packages/django/core/checks/registry.py", line 81, in run_checks
new_errors = check(app_configs=app_configs)
File "/usr/local/lib/python3.5/site-packages/django/core/checks/model_checks.py", line 30, in check_all_models
errors.extend(model.check(**kwargs))
File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 1284, in check
errors.extend(cls._check_fields(**kwargs))
File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 1359, in _check_fields
errors.extend(field.check(**kwargs))
File "/usr/local/lib/python3.5/site-packages/django/db/models/fields/init.py", line 913, in check
errors = super(AutoField, self).check(**kwargs)
File "/usr/local/lib/python3.5/site-packages/django/db/models/fields/init.py", line 219, in check
errors.extend(self._check_backend_specific_checks(**kwargs))
File "/usr/local/lib/python3.5/site-packages/django/db/models/fields/init.py", line 322, in _check_backend_specific_checks
return connections[db].validation.check_field(self, **kwargs)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/validation.py", line 49, in check_field
field_type = field.db_type(self.connection)
File "/usr/local/lib/python3.5/site-packages/django/db/models/fields/init.py", line 644, in db_type
return connection.data_types[self.get_internal_type()] % data
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 35, in get
res = instance.dict[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 174, in data_types
if self.features.supports_microsecond_precision:
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 35, in get
res = instance.dict[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/features.py", line 53, in supports_microsecond_precision
return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 35, in get
res = instance.dict[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 385, in mysql_version
with self.temporary_connection() as cursor:
File "/usr/local/lib/python3.5/contextlib.py", line 59, in enter
return next(self.gen)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 591, in temporary_connection
cursor = self.cursor()
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 254, in cursor
return self._cursor()
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 229, in _cursor
self.ensure_connection()
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 274, in get_new_connection
conn = Database.connect(**conn_params)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/init.py", line 86, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py", line 204, in init
super(Connection, self).init(*args, **kwargs2)
django.db.utils.OperationalError: (2002, 'Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (2 "No such file or directory")')