Acra-based example projects: check how easy it is to deploy data security in typical web infrastructures.
Home Page: https://www.cossacklabs.com/acra/
License: Apache License 2.0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 23394 100 23394 0 0 57946 0 --:--:-- --:--:-- --:--:-- 58778
## Selected demo project: python-mysql
== Work in current directory
Resources that will become available after launch:
* Container with environment prepared for the python example. Folder with
example scripts will be mounted to container, so you will be able to
modify these scripts without stopping docker compose.
Run example with zones (write, read):
docker exec -it python_python_1 \
python /app/extended_example_with_zone.py --data=data.json
docker exec -it python_python_1 \
python /app/extended_example_with_zone.py \
--print --zone_id=$ZONE_ID
where $ZONE_ID - zone id, printed on write step
Before using AcraServer without zones, open `python/acra-server-config/acra-server.yaml` and change
`zonemode_enable: true` value to `false` and
`encryptor_config_file: encryptor_config_with_zone.yaml` to `encryptor_config_without_zone.yaml`.
Run example without zones (write, read):
docker exec -it python_python_1 \
python /app/extended_example_without_zone.py --data=data.json
docker exec -it python_python_1 \
python /app/extended_example_without_zone.py --print
* Web interface for MySQL - see how the encrypted data is stored:
http://$HOST:8080
Default user/password: [email protected]/test
* MySQL - also you can connect to DB directly:
mysql://$HOST:3306
Default admin user/password: test/test
* Prometheus - examine the collected metrics:
http://$HOST:9090
* Grafana - sample of dashboards with Acra metrics:
http://$HOST:3000
* Jaeger - view traces:
http://$HOST:16686
where are HOST is the IP address of the server with running Acra
Engineering Demo. If you run this demo on the same host, from
which you will connect, use "localhost".
Press any key to continue...HEAD is now at 89f6bdd Pre release (#546)
== Pull fresh images
>> COSSACKLABS_ACRAENGDEMO_VCS_URL="https://github.com/cossacklabs/acra-engineering-demo" COSSACKLABS_ACRAENGDEMO_VCS_BRANCH="master" COSSACKLABS_ACRAENGDEMO_VCS_REF="bde646a4fd05ac47321ec307c4cbef50010c5a71" COSSACKLABS_ACRAENGDEMO_BUILD_DATE="2023-07-04T01:33:08Z" COSSACKLABS_ACRA_VCS_URL="https://github.com/cossacklabs/acra" COSSACKLABS_ACRA_VCS_BRANCH="0.93.0" COSSACKLABS_ACRA_VCS_REF="89f6bdd3334f3b92b6a3e56b58ce2588de1c7609" docker-compose -f /Users/amri.hidayat/Documents/GitHub/acra-engineering-demo/python-mysql/docker-compose.python-mysql.yml pull
[+] Pulling 24/24
✔ mysql Skipped - No image to be pulled 0.0s
✔ python Skipped - No image to be pulled 0.0s
✔ grafana 3 layers [⣿⣿⣿] 0B/0B Pulled 25.3s
✔ f2aa67a397c4 Pull complete 9.5s
✔ e93fe8a31a9e Pull complete 18.1s
✔ fa279080e10e Pull complete 18.1s
✔ phpmyadmin Skipped - No image to be pulled 0.0s
✔ jaeger 2 layers [⣿⣿] 0B/0B Pulled 12.9s
✔ a83feccf3672 Pull complete 5.8s
✔ d518ddfa1466 Pull complete 5.8s
✔ acra-keymaker_client 1 layers [⣿] 0B/0B Pulled 24.8s
✔ b1a534294bbb Pull complete 17.6s
✔ prometheus 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 22.5s
✔ f70adabe43c0 Pull complete 7.6s
✔ b91f96ddd9e3 Pull complete 9.0s
✔ a8f5bee8d104 Pull complete 13.1s
✔ 9262e07c8788 Pull complete 13.5s
✔ 25fe26899079 Pull complete 13.5s
✔ ef1d242494d2 Pull complete 13.5s
✔ 92cb2b46cbde Pull complete 14.2s
✔ ef791356f009 Pull complete 14.5s
✔ ed46b860f1bb Pull complete 15.3s
✔ acra-server 1 layers [⣿] 0B/0B Pulled 28.6s
✔ f5da3757e651 Pull complete 21.4s
== Starting docker-compose
>> COSSACKLABS_ACRAENGDEMO_VCS_URL="https://github.com/cossacklabs/acra-engineering-demo" COSSACKLABS_ACRAENGDEMO_VCS_BRANCH="master" COSSACKLABS_ACRAENGDEMO_VCS_REF="bde646a4fd05ac47321ec307c4cbef50010c5a71" COSSACKLABS_ACRAENGDEMO_BUILD_DATE="2023-07-04T01:33:08Z" COSSACKLABS_ACRA_VCS_URL="https://github.com/cossacklabs/acra" COSSACKLABS_ACRA_VCS_BRANCH="0.93.0" COSSACKLABS_ACRA_VCS_REF="89f6bdd3334f3b92b6a3e56b58ce2588de1c7609" docker-compose -f /Users/amri.hidayat/Documents/GitHub/acra-engineering-demo/python-mysql/docker-compose.python-mysql.yml up --build
[+] Building 75.8s (37/49)
=> [python internal] load build definition from python.dockerfile 0.0s
=> => transferring dockerfile: 2.35kB 0.0s
=> [python internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [python internal] load metadata for docker.io/library/alpine:3.15.0 2.5s
=> [mysql internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [mysql internal] load build definition from mysql-playground.dockerfile 0.0s
=> => transferring dockerfile: 1.63kB 0.0s
=> [mysql internal] load metadata for docker.io/library/mariadb:10.7.1 2.5s
=> [mysql internal] load build context 0.0s
=> => transferring context: 3.54kB 0.0s
=> [mysql 1/6] FROM docker.io/library/mariadb:10.7.1@sha256:5a8009369ee57c85b6f4a08406147bd9c505cde6b8250d16a27d2a5febfdead7 0.0s
=> [python internal] load build context 0.0s
=> => transferring context: 955B 0.0s
=> CACHED [python 1/23] FROM docker.io/library/alpine:3.15.0@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 0.0s
=> [python 2/23] RUN echo 'root:!' | chpasswd -e 0.6s
=> CACHED [mysql 2/6] COPY _common/ssl/mysql/mysql.crt /tmp.ssl/mysql.crt 0.0s
=> CACHED [mysql 3/6] COPY _common/ssl/mysql/mysql.key /tmp.ssl/mysql.key 0.0s
=> CACHED [mysql 4/6] COPY _common/ssl/ca/ca.crt /tmp.ssl/ca.crt 0.0s
=> [mysql 5/6] RUN chown -R mysql:mysql /tmp.ssl && chmod 0400 /tmp.ssl/mysql.key 0.6s
=> [python 3/23] RUN apk update 1.5s
=> [mysql 6/6] COPY _common/mysql-playground/mariadb-ssl.cnf /etc/mysql/mariadb.conf.d/ 0.0s
=> [mysql] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:d40b98e414ef7fb0e73df8e4de6400694ab76a97a54d91fe47ab68c9d482dddf 0.0s
=> => naming to docker.io/library/python-mysql-mysql 0.0s
=> [phpmyadmin internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [phpmyadmin internal] load build definition from phpmyadmin.dockerfile 0.0s
=> => transferring dockerfile: 1.52kB 0.0s
=> [phpmyadmin internal] load metadata for docker.io/library/phpmyadmin:5.1.1 1.3s
=> [python 4/23] RUN apk add --no-cache bash python3 py3-pip mariadb-dev mariadb-client 16.1s
=> [phpmyadmin 1/6] FROM docker.io/library/phpmyadmin:5.1.1@sha256:48e667339e129195358b61d47be4f65a84e513663604d8d64070e3e17ab9f6c9 0.0s
=> [phpmyadmin internal] load build context 0.0s
=> => transferring context: 4.09kB 0.0s
=> CACHED [phpmyadmin 2/6] COPY ./_common/ssl/acra-client/acra-client.crt /tmp.ssl/acra-client.crt 0.0s
=> CACHED [phpmyadmin 3/6] COPY ./_common/ssl/acra-client/acra-client.key /tmp.ssl/acra-client.key 0.0s
=> CACHED [phpmyadmin 4/6] COPY ./_common/ssl/ca/ca.crt /tmp.ssl/root.crt 0.0s
=> CACHED [phpmyadmin 5/6] COPY ./_common/phpmyadmin/config.user.inc.php /etc/phpmyadmin/config.user.inc.php 0.0s
=> [phpmyadmin 6/6] RUN chown -R www-data:www-data /tmp.ssl 0.2s
=> [phpmyadmin] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:770c466e8ac77e899277afd3237d23a54b4caaccab7a5629531db5225e704f58 0.0s
=> => naming to docker.io/library/python-mysql-phpmyadmin 0.0s
=> [python 5/23] RUN pip3 install --no-cache-dir --upgrade pip 4.6s
=> [python 6/23] RUN ln -s /usr/bin/python3 /usr/bin/python 0.3s
=> [python 7/23] RUN apk add gcc python3-dev musl-dev libxml2-dev git alpine-sdk rsync 44.3s
=> [python 8/23] RUN mkdir -p /usr/local/sbin 0.2s
=> [python 9/23] RUN echo -e '#!/bin/sh\n\nexit 0\n' > /usr/local/sbin/ldconfig 0.2s
=> [python 10/23] RUN chmod +x /usr/local/sbin/ldconfig 0.2s
=> ERROR [python 11/23] RUN cd /root && git clone --depth 1 -b stable https://github.com/cossacklabs/themis 5.3s
------
> [python 11/23] RUN cd /root && git clone --depth 1 -b stable https://github.com/cossacklabs/themis:
0.236 Cloning into 'themis'...
5.247 fatal: unable to access 'https://github.com/cossacklabs/themis/': Could not resolve host: github.com
------
failed to solve: process "/bin/sh -c cd /root && git clone --depth 1 -b stable https://github.com/cossacklabs/themis" did not complete successfully: exit code: 128
Running git clone --depth 1 -b stable https://github.com/cossacklabs/themis on its own is fine. So I guess it's how the docker runtime resolve https ?
This is referenced in many places, but it’s not in repo
After i run the suggested command
curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \ bash -s -- python-mysql-postgresql
and hit any key to proceed i get
Cloning into '/var/folders/xg/hbss0xms2pl4ztxzhs92hd200000gn/T/tmp.tVATPUDJUb/acra-engineering-demo/acra'...
remote: Enumerating objects: 1041, done.
remote: Counting objects: 100% (1041/1041), done.
remote: Compressing objects: 100% (908/908), done.
remote: Total 1041 (delta 147), reused 576 (delta 60), pack-reused 0
Receiving objects: 100% (1041/1041), 1.42 MiB | 4.64 MiB/s, done.
Resolving deltas: 100% (147/147), done.
fatal: reference is not a tree: e1773020a881a39d126724bd3992b9b436173fa0
After running the suggested command line $ curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \ bash -s -- python-searchable
i get tons of errors such as these
python-searchable-acra-server-1 | time="2023-10-25T13:23:30Z" level=info msg="Starting service acra-server [pid=1]" version=0.93.0
python-searchable-acra-server-1 | time="2023-10-25T13:23:30Z" level=info msg="Validating service configuration..."
python-searchable-acra-server-1 | time="2023-10-25T13:23:30Z" level=info msg="Load encryptor configuration from /config/searchable.yaml ..."
python-searchable-acra-server-1 | time="2023-10-25T13:23:30Z" level=error msg="Can't read config for encryptor" code=507 error="read /config/searchable.yaml: is a directory"
python-searchable-acra-server-1 | time="2023-10-25T13:23:30Z" level=error msg="Can't load encryptor config" error="read /config/searchable.yaml: is a directory"
python-searchable-acra-server-1 exited with code 1
and this
python-searchable-postgresqlweb-1 | 2023-10-25 13:25:19,888: ERROR pgadmin: 400 Bad Request: The CSRF session token is missing.
python-searchable-postgresqlweb-1 | Traceback (most recent call last):
python-searchable-postgresqlweb-1 | File "/venv/lib/python3.9/site-packages/flask_wtf/csrf.py", line 257, in protect
python-searchable-postgresqlweb-1 | validate_csrf(self._get_csrf_token())
python-searchable-postgresqlweb-1 | File "/venv/lib/python3.9/site-packages/flask_wtf/csrf.py", line 95, in validate_csrf
python-searchable-postgresqlweb-1 | raise ValidationError('The CSRF session token is missing.')
python-searchable-postgresqlweb-1 | wtforms.validators.ValidationError: The CSRF session token is missing.
python-searchable-postgresqlweb-1 |
python-searchable-postgresqlweb-1 | During handling of the above exception, another exception occurred:
python-searchable-postgresqlweb-1 |
python-searchable-postgresqlweb-1 | Traceback (most recent call last):
python-searchable-postgresqlweb-1 | File "/venv/lib/python3.9/site-packages/flask/app.py", line 1948, in full_dispatch_request
python-searchable-postgresqlweb-1 | rv = self.preprocess_request()
python-searchable-postgresqlweb-1 | File "/venv/lib/python3.9/site-packages/flask/app.py", line 2242, in preprocess_request
python-searchable-postgresqlweb-1 | rv = func()
python-searchable-postgresqlweb-1 | File "/venv/lib/python3.9/site-packages/flask_wtf/csrf.py", line 225, in csrf_protect
python-searchable-postgresqlweb-1 | self.protect()
python-searchable-postgresqlweb-1 | File "/venv/lib/python3.9/site-packages/flask_wtf/csrf.py", line 260, in protect
python-searchable-postgresqlweb-1 | self._error_response(e.args[0])
python-searchable-postgresqlweb-1 | File "/venv/lib/python3.9/site-packages/flask_wtf/csrf.py", line 303, in _error_response
python-searchable-postgresqlweb-1 | raise CSRFError(reason)
python-searchable-postgresqlweb-1 | flask_wtf.csrf.CSRFError: 400 Bad Request: The CSRF session token is missing.
The db is up (and i can access it) but no tables were created.
I'm trying to follow these steps https://github.com/cossacklabs/acra-engineering-demo#example-4-transparent-encryption-zones-python-app-mysql-deprecated-will-be-removed but I encounter an issue on Insert data using updated config.
Error i receiedd when running: root@ubuntu-s-1vcpu-2gb-intel-sgp1-01:~/acra-engineering-demo# docker exec -it python-mysql-python-1 python3 extended_example_with_zone.py --host=acra-server --port=9393 --data=data.json
data: [{'token_i32': 1234, 'data_i32': '1234', 'token_i64': 645664, 'data_i64': '645664', 'token_str': '078-05-1111', 'token_bytes': 'byt13es', 'token_email': '[email protected]', 'data_str': 'John Wed, Senior Relationshop Manager', 'masking': '$112000', 'searchable': '[email protected]'}, {'token_i32': 1234, 'data_i32': '1234', 'token_i64': 645664, 'data_i64': '645664', 'token_str': '078-05-1112', 'token_bytes': 'byt13es2', 'token_email': '[email protected]', 'data_str': 'April Cassini, Marketing Manager', 'masking': '$168000', 'searchable': '[email protected]'}, {'token_i32': 1234, 'data_i32': '1234', 'token_i64': 645664, 'data_i64': '645664', 'token_str': '078-05-1117', 'token_bytes': 'byt13es3', 'token_email': '[email protected]', 'data_str': 'George Clooney, Famous Actor', 'masking': '$780000', 'searchable': '[email protected]'}]
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
self.dialect.do_execute(
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute
cursor.execute(statement, parameters)
File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 153, in execute
result = self._query(query)
File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 558, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 822, in _read_query_result
result.read()
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1200, in read
first_packet = self.connection._read_packet()
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 739, in _read_packet
packet_header = self._read_bytes(4)
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 795, in _read_bytes
raise err.OperationalError(
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/extended_example_with_zone.py", line 146, in <module>
write_data(args.data, connection)
File "/app/extended_example_with_zone.py", line 104, in write_data
connection.execute(
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute
return meth(
File "/usr/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement
ret = self._execute_context(
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context
return self._exec_single_context(
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context
self._handle_dbapi_exception(
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
self.dialect.do_execute(
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute
cursor.execute(statement, parameters)
File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 153, in execute
result = self._query(query)
File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 558, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 822, in _read_query_result
result.read()
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1200, in read
first_packet = self.connection._read_packet()
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 739, in _read_packet
packet_header = self._read_bytes(4)
File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 795, in _read_bytes
raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: INSERT INTO test (data_str, masking, token_i32, data_i32, token_i64, data_i64, token_str, token_bytes, token_email) VALUES (%(data_str)s, %(masking)s, %(token_i32)s, %(data_i32)s, %(token_i64)s, %(data_i64)s, %(token_str)s, %(token_bytes)s, %(token_email)s)]
[parameters: {'data_str': b'John Wed, Senior Relationshop Manager', 'masking': b'$112000', 'token_i32': 1234, 'data_i32': b'1234', 'token_i64': 645664, 'data_i64': b'645664', 'token_str': '078-05-1111', 'token_bytes': b'byt13es', 'token_email': '[email protected]'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
This is continuing the issue i encountered #59.
After few times running the insert command:
root@ubuntu-s-1vcpu-2gb-intel-sgp1-01:~/acra-engineering-demo# docker exec -it python-mysql-python-1 python3 extended_example_with_zone.py --host=acra-server --port=9393 --data=data.json
data: [{'token_i32': 1234, 'data_i32': '1234', 'token_i64': 645664, 'data_i64': '645664', 'token_str': '078-05-1111', 'token_bytes': 'byt13es', 'token_email': '[email protected]', 'data_str': 'John Wed, Senior Relationshop Manager', 'masking': '$112000', 'searchable': '[email protected]'}, {'token_i32': 1234, 'data_i32': '1234', 'token_i64': 645664, 'data_i64': '645664', 'token_str': '078-05-1112', 'token_bytes': 'byt13es2', 'token_email': '[email protected]', 'data_str': 'April Cassini, Marketing Manager', 'masking': '$168000', 'searchable': '[email protected]'}, {'token_i32': 1234, 'data_i32': '1234', 'token_i64': 645664, 'data_i64': '645664', 'token_str': '078-05-1117', 'token_bytes': 'byt13es3', 'token_email': '[email protected]', 'data_str': 'George Clooney, Famous Actor', 'masking': '$780000', 'searchable': '[email protected]'}]
i checked the phymyadmin for this database's table it shows no results:
I noticed that the example uses this file to perform insert & select. On this file, I don't see any reference to acrawriter that perform the encryption into an acrastruct. Am I missing something ?
The docker command that executes the insert (as per the tutorial: https://github.com/cossacklabs/acra-engineering-demo/#24-insert-data-using-updated-config):
docker exec -it python-mysql_python_1 \ python3 extended_example_without_zone.py --host=acra-server --port=9393 --data=data.json
Could you assist me on this ?
In "Example 4. Transparent encryption, Zones, Python app, MySQL" , I am having error "mysql_1 | 2023-06-20 20:30:30 178 [Warning] Aborted connection 178 to db: 'test' user: 'test' host: '172.20.0.5' (Got an error reading communication packets)" in following commands
docker exec -it python-mysql_python_1
python3 extended_example_with_zone.py --host=acra-server --port=9393 --generate_zone
docker exec -it python-mysql_python_1
python3 extended_example_without_zone.py --host=acra-server --port=9393 --data=data.json
How to resolve this?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.