When I install data_pipeline, it gave me the following issue:
kafka-utils 1.7.1 has requirement kafka-python<1.5.0,>=1.3.2, but you'll have kafka-python 0.9.5 which is incompatible.
Cannot uninstall 'pyOpenSSL'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

and when installing mrjob I got the following:
kafka-utils 1.7.1 has requirement kafka-python<1.5.0,>=1.3.2, but you'll have kafka-python 0.9.5 which is incompatible.

Anyway to update kafka-utils?

when system USER variable is in uppercase, the docker build will fail and complain：

docker build -f Dockerfile-opensource -t replication-handler-opensource-dev-UPPERCASE .
invalid argument "replication-handler-opensource-dev-UPPERCASE" for t: Error parsing reference: "replication-handler-opensource-dev-UPPERCASE" is not a valid repository/tag
See 'docker build --help'.
make: *** [cook-image] Error 125

GitHub flagged the following security vulnerability affecting this repository.

https://nvd.nist.gov/vuln/detail/CVE-2018-18074

CVE-2018-18074
moderate severity
Vulnerable versions: <= 2.19.1
Patched version: 2.20.0
The Requests package through 2.19.1 before 2018-09-14 for Python sends an HTTP Authorization header to an http URI upon receiving a same-hostname https-to-http redirect, which makes it easier for remote attackers to discover credentials by sniffing the network.

Hi guys,

when running the 'make -f Makefile-opensource test' command, I get the below errors:
2017-02-03 16:39:08,079 - replication_handler.batch.base_parse_replication_stream:115: ERROR Shutting down because of exception
Traceback (most recent call last):
File "/root/mysql_streamer-master/replication_handler/batch/base_parse_replication_stream.py", line 108, in run
self.process_event(replication_handler_event)
File "/root/mysql_streamer-master/replication_handler/batch/base_parse_replication_stream.py", line 140, in process_event
self.current_event_type = self.handler_map[event_class].event_type
KeyError: <class 'mock.mock.MagicMock'>

I find that the reason is because the 'handler_map' function dosen't handle the class MagicMock event type in base_parse_replication_stream.py, could someone can fix it?
<MagicMock name=u'ReplicationStreamRestarter().get_stream().next().event' id='140589236637520'>

And now we don't have docker environment, can I deploy it in linux environment directly?
Do you have some detail deployment document? Please share it.

Thank you.

Hi there,

Any installation guide on Centos 7?

since mysql_streamer can capture every change on MySQL, and send the operations to Kafka, so it can be used to log all the operations and then can be used to restore the database at a specific timestamp based on the operations in Kafka.

Obtaining file:///code (from -r /code/requirements-opensource.txt (line 5))
Running setup.py (path:/code/setup.py) egg_info for package from file:///code

Downloading/unpacking MySQL-python==1.2.5 (from -r /code/requirements-opensource.txt (line 8))
Cannot fetch index base URL https://pypi.yelpcorp.com/simple/
Running setup.py (path:/code/virtualenv_run/build/MySQL-python/setup.py) egg_info for package MySQL-python

Downloading/unpacking PyMySQL==0.6.7 (from -r /code/requirements-opensource.txt (line 9))
Cannot fetch index base URL https://pypi.yelpcorp.com/simple/
Downloading/unpacking PyStaticConfiguration==0.10.2 (from -r /code/requirements-opensource.txt (line 10))
Cannot fetch index base URL https://pypi.yelpcorp.com/simple/
Downloading PyStaticConfiguration-0.10.2.tar.gz
Running setup.py (path:/code/virtualenv_run/build/PyStaticConfiguration/setup.py) egg_info for package PyStaticConfiguration

Downloading/unpacking PyYAML==3.11 (from -r /code/requirements-opensource.txt (line 11))
Cannot fetch index base URL https://pypi.yelpcorp.com/simple/
Running setup.py (path:/code/virtualenv_run/build/PyYAML/setup.py) egg_info for package PyYAML

Downloading/unpacking SQLAlchemy==1.0.12 (from -r /code/requirements-opensource.txt (line 12))
Cannot fetch index base URL https://pypi.yelpcorp.com/simple/
Running setup.py (path:/code/virtualenv_run/build/SQLAlchemy/setup.py) egg_info for package SQLAlchemy

https://pypi.yelpcorp.com/simple/ is down ?

Is geogrid an internal package? Was trying to run the integration tests (make -f Makefile-opensource itest) and I'm getting this error:

No distributions at all found for geogrid==1.0.8 (from -r /code/requirements-opensource.txt (line 20))

Hi,

I am trying to create docker image from Dockerfile present in mysql_streamer But it is giving me following errors :

Obtaining file:///code (from -r /code/requirements.txt (line 5)) Collecting MySQL-python==1.2.5 (from -r /code/requirements.txt (line 8)) Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003747360>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/mysql-python/ Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003747398>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/mysql-python/ Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x00000000037473d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/mysql-python/ Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003747408>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/mysql-python/ Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003747440>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/mysql-python/ Downloading MySQL-python-1.2.5.zip (108kB) Collecting PyMySQL==0.6.7 (from -r /code/requirements.txt (line 9)) Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x000000000366f5c8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/pymysql/ Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x000000000366f600>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/pymysql/ Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x000000000366f638>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/pymysql/

Any idea what this error might be?

Thanks and Regards,

Chinmay Prabhu

Hi,

Encountering

error while running 'make -f Makefile-opensource test' command. Can you help us resolve the issue.
/bin/sh: 1: tox: not found make: *** [test] Error 127

centos run “make -f Makefile-opensource test” throws error：
OPEN_SOURCE_MODE=True FORCE_AVOID_INTERNAL_PACKAGES=True tox -c tox-opensource.ini
/bin/sh: tox: command not found
make: *** [test] Error 127

Could you point me to a stable release/commit? Thanks in advance!

I know the name says, it is a mysql streamer. But since MariaDb was forked from MySQL, I am wondering if it can be made to work with even MAriaDB? Also what version of MySQL is supported, only 5.5?

Hi
while running the make -f Makefile-opensource test' command am getting below exception and its cleaning all the setup. Any leads on this will be helpful.

gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/_mysql.o -L/usr/lib -L/usr/lib64 -lmysqlclient -lpthread -lm -lrt -ldl -lpython2.7 -o build/lib.linux-x86_64-2.7/_mysql.so

/usr/bin/ld: cannot find -lmysqlclient

collect2: error: ld returned 1 exit status

error: command 'gcc' failed with exit status 1

Failed building wheel for MySQL-python
SLOW!! no wheel found after building (couldn't be wheeled?): MySQL-python==1.2.3
Installing collected packages: MySQL-python, PyMySQL, PyStaticConfiguration, PyYAML, SQLAlchemy, boto, contextdecorator, data-pipeline, dateglob, enum34, future, futures, ipaddress, iso8601, kafka-python, kazoo, mccabe, mrjob, mysql-replication, pep8, ply, py, pycrypto, pyflakes, pysubnettree, python-dateutil, pytz, requests, send-nsca, setproctitle, simplejson, six, sqlparse, testing.mysqld, thriftpy, vmprof, wheel, yelp-clog, replication-handler, swagger-zipkin, cryptography, cached-property, pyramid-zipkin, data-pipeline-avro-util, pyopenssl, bravado, cffi, yelp-avro, frozendict, kafka-utils, psutil, filechunkio, testing.common.database, py-zipkin, idna, pyasn1, pyramid, fido, bravado-core, twisted, yelp-bytes, crochet, pycparser, requests-futures, retrying, fabric, PasteDeploy, repoze.lru, translationstring, WebOb, zope.deprecation, venusian, hupper, zope.interface, service-identity, jsonschema, swagger-spec-validator, Automat, constantly, incremental, yelp-encodings, paramiko, attrs, pyasn1-modules, functools32, rfc3987, strict-rfc3339, webcolors, ecdsa
Running setup.py install for MySQL-python
building 'mysql' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Dversion_info=(1,2,3,'final',0) -D__version_=1.2.3 -I/usr/include/mysql -I/usr/include/python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o -fPIC -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing
gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/_mysql.o -L/usr/lib -L/usr/lib64 -lmysqlclient -lpthread -lm -lrt -ldl -lpython2.7 -o build/lib.linux-x86_64-2.7/_mysql.so
/usr/bin/ld: cannot find -lmysqlclient
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
Complete output from command /mnt/data/mysql_streamer/mysql_streamer-master/venv/docs/bin/python2.7 -c "import setuptools, tokenize;file='/mnt/data/mysql_streamer/mysql_streamer-master/venv/docs/build/MySQL-python/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-zrNOEa-record/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/data/mysql_streamer/mysql_streamer-master/venv/docs/include/site/python2.7:
running install

running build

running build_py

running build_ext

building '_mysql' extension

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Dversion_info=(1,2,3,'final',0) -D__version__=1.2.3 -I/usr/include/mysql -I/usr/include/python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o -fPIC -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing

gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/_mysql.o -L/usr/lib -L/usr/lib64 -lmysqlclient -lpthread -lm -lrt -ldl -lpython2.7 -o build/lib.linux-x86_64-2.7/_mysql.so

/usr/bin/ld: cannot find -lmysqlclient

collect2: error: ld returned 1 exit status

error: command 'gcc' failed with exit status 1

Cleaning up...
Command /mnt/data/mysql_streamer/mysql_streamer-master/venv/docs/bin/python2.7 -c "import setuptools, tokenize;file='/mnt/data/mysql_streamer/mysql_streamer-master/venv/docs/build/MySQL-python/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-zrNOEa-record/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/data/mysql_streamer/mysql_streamer-master/venv/docs/include/site/python2.7 failed with error code 1 in /mnt/data/mysql_streamer/mysql_streamer-master/venv/docs/build/MySQL-python
Storing debug log for failure in /root/.pip/pip.log

Something went wrong! Sending 'venv/docs' back in time, so make knows it's invalid.
ERROR: InvocationError: '/mnt/data/mysql_streamer/mysql_streamer-master/bin/venv-update venv= /mnt/data/mysql_streamer/mysql_streamer-master/venv/docs install= -r /mnt/data/mysql_streamer/mysql_streamer-master/requirements.d/docs.txt -r /mnt/data/mysql_streamer/mysql_streamer-master/requirements-opensource.txt'
pre-commit installed: PyYAML==3.12,aspy.yaml==0.2.2,cached-property==1.3.0,functools32==3.2.3.post2,jsonschema==2.6.0,nodeenv==1.1.2,pre-commit==0.13.3,venv-update==1.0.1,virtualenv==15.1.0,wheel==0.29.0,wsgiref==0.1.2
pre-commit runtests: PYTHONHASHSEED='2193082984'
pre-commit runtests: commands[0] | /mnt/data/mysql_streamer/mysql_streamer-master/bin/venv-update venv= /mnt/data/mysql_streamer/mysql_streamer-master/venv/pre-commit install= -r /mnt/data/mysql_streamer/mysql_streamer-master/requirements.d/pre_commit.txt

virtualenv venv/pre-commit
Keeping valid virtualenv from previous run.
pip install --find-links=file:///root/.cache/pip-faster/wheelhouse venv-update==1.0.1
Requirement already satisfied (use --upgrade to upgrade): venv-update==1.0.1 in ./venv/pre-commit/lib/python2.7/site-packages
Cleaning up...
pip-faster install --upgrade --prune -r requirements.d/pre_commit.txt
Requirement already up-to-date: pre-commit>=0.4.2 in ./venv/pre-commit/lib/python2.7/site-packages (from -r /mnt/data/mysql_streamer/mysql_streamer-master/requirements.d/pre_commit.txt (line 1))
Cleaning up...
pre-commit runtests: commands[1] | pre-commit
Traceback (most recent call last):
File "/mnt/data/mysql_streamer/mysql_streamer-master/venv/pre-commit/bin/pre-commit", line 7, in
from pre_commit.main import main
File "/mnt/data/mysql_streamer/mysql_streamer-master/venv/pre-commit/lib/python2.7/site-packages/pre_commit/main.py", line 7, in
import pre_commit.constants as C
File "/mnt/data/mysql_streamer/mysql_streamer-master/venv/pre-commit/lib/python2.7/site-packages/pre_commit/constants.py", line 3, in
import pkg_resources
File "/mnt/data/mysql_streamer/mysql_streamer-master/venv/pre-commit/lib/python2.7/site-packages/pkg_resources/init.py", line 48, in
import six
ImportError: No module named six
ERROR: InvocationError: '/mnt/data/mysql_streamer/mysql_streamer-master/venv/pre-commit/bin/pre-commit'
__________________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________________
ERROR: py27: commands failed
ERROR: docs: commands failed
ERROR: pre-commit: commands failed
make: *** [test] Error 1

Unit tests are failing (make -f Makefile-opensource test) given data_pipeline==0.9.1 is no longer available.

I tried forking + creating a release tag for this version, but then using that version fails as some other dependencies are no longer available for the requested version in the yelp repo.

Upgrading data_pipeline to 0.9.5 also fails due to unmet dependencies/conflicts:

Error: unmet dependency: cryptography<=1.3.4 (from data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: frozendict==0.5 (from data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: pyramid-zipkin>=0.16.0 (from data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: swagger-zipkin>=0.1.0 (from data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: version conflict: yelp-clog 2.2.10 (venv/py27/lib/python2.7/site-packages) <-> yelp-clog==2.2.11 (from data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: filechunkio (from mrjob==0.4.3->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 21))
Error: unmet dependency: testing.common.database>=1.1.0 (from testing.mysqld->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 37))
Error: unmet dependency: backports.ssl-match-hostname>=3.5 (from docker-py>=1.7.0,<2->docker-compose==1.6.2->-r /home/ubuntu/mysql_streamer/requirements.d/dev.txt (line 5))
Error: unmet dependency: docker-pycreds>=0.2.1 (from docker-py>=1.7.0,<2->docker-compose==1.6.2->-r /home/ubuntu/mysql_streamer/requirements.d/dev.txt (line 5))
Error: unmet dependency: functools32 (from jsonschema>=2.5.1,<3->docker-compose==1.6.2->-r /home/ubuntu/mysql_streamer/requirements.d/dev.txt (line 5))
Error: unmet dependency: backports.ssl-match-hostname (from websocket-client>=0.32.0,<1.0->docker-compose==1.6.2->-r /home/ubuntu/mysql_streamer/requirements.d/dev.txt (line 5))
Error: unmet dependency: bravado-core>=4.2.2 (from bravado>=8.4.0->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: crochet>=1.4.0 (from bravado>=8.4.0->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: fido>=2.1.0 (from bravado>=8.4.0->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: twisted>=14.0.0 (from bravado>=8.4.0->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: yelp-bytes (from bravado>=8.4.0->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: fabric>=1.8.0,<1.11.0 (from kafka-utils>0.3.3->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: requests-futures>0.9.0 (from kafka-utils>0.3.3->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: retrying (from kafka-utils>0.3.3->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))
Error: unmet dependency: cryptography>=1.3 (from pyopenssl==16.0.0->data-pipeline==0.9.5->-r /home/ubuntu/mysql_streamer/requirements-opensource.txt (line 11))```

I tried to resolve some of the conflicts but it was getting complicated quite quickly. Any suggestions on how to proceed?

• Configuration documentation on MySQL servers
• Configuration documentation on Kafka cluster/server

.. etc.