google / crmint Goto Github PK

Reliable data integration & processing for advertisers

Home Page: https://google-marketing-solutions.github.io/crmint/

License: Other

Python 69.57% Mako 0.04% Shell 1.96% TypeScript 17.60% JavaScript 0.28% HTML 4.99% Dockerfile 0.84% SCSS 0.67% Sass 0.99% HCL 3.06%

appengine-python google-cloud-platform google-bigquery google-cloud-storage google-adwords google-analytics

crmint's Introduction

CRMint

Make reliable data integration and data processing with Google easy for advertisers.

Status	Coverage	Description
		Testing the reliability of our deployment tool
		Testing the core of CRMint backend engine
	-	Testing our frontend
	-	Building docker images for our services
	-	Testing our Terraform plan

Deploy CRMint

Follow the tutorial built into Cloud Shell:

Ambition

CRMint was created to make advertisers' data integration and processing easy, even for people without software engineering background.

It has simple and intuitive web UI that allows users to create, edit, run, and schedule data pipelines consisting of data transfer and data processing jobs.

This is not an official Google product.

crmint's People

Contributors

Stargazers

Watchers

crmint's Issues

"Don't create table if doesn't exist"

In StorageToBQImporter

Double negatives confuse me :-)
Does this mean "Require table to already exist" ?
Or is it slightly different?

Deadline exceeded while waiting for HTTP response

I have a pipeline job that has been failing for the last three days the following error:

Unexpected error: ConnectionError: ('Connection aborted.', HTTPException('Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/bigquery/v2/projects/PROJECT_NAME/jobs',))

This job has been running every day for many months without an issue and has remained unchanged, has there been some change to BigQuery that could cause this?

Make the link to predicted prices in the BigQuery ML tutorial open in a new window

The link to the predicted values, in step 5 of "Let's predict some prices" doesn't launch in a new window.

Consider making this launch in a new window.

Also, consider adding the project and dataset details to make them explicit in the URL, e.g.

https://console.cloud.google.com/bigquery?d=predictions

https://console.cloud.google.com/bigquery?project=my-project&p=my-project&d=predict_realestate_brasil&t=predictions&page=table

Deploy CRMint in Google Cloud, Applying DB migrations / Sowing DB seeds

I'm using the docs from: https://github.com/google/crmint/wiki/Deploy-CRMint-on-Google-Cloud-Platform to deploy CRMint in Google Cloud (master branch). On the second command:

cd crmint
bin/app cloud setup
bin/app cloud deploy

.. at the end of deploy i've got some errors on this steps:

==> Applying DB migrations

2018/10/22 17:32:28 Listening on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp for crmint-bmind3:europe-west1:crmintapp
2018/10/22 17:32:28 Ready for new connections
2018/10/22 17:32:29 New connection for "crmint-bmind3:europe-west1:crmintapp"
Traceback (most recent call last):
2018/10/22 17:32:29 Client closed local connection on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/crmint-bmind3/backends/lib/flask/__main__.py", line 15, in <module>
    main(as_module=True)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 513, in main
    cli.main(args=args, prog_name=name)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 380, in main
    return AppGroup.main(self, *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 257, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/flask_migrate/cli.py", line 132, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/tmp/crmint-bmind3/backends/lib/flask_migrate/__init__.py", line 244, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/tmp/crmint-bmind3/backends/lib/alembic/command.py", line 254, in upgrade
    script.run_env()
  File "/tmp/crmint-bmind3/backends/lib/alembic/script/base.py", line 427, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/tmp/crmint-bmind3/backends/lib/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/tmp/crmint-bmind3/backends/lib/alembic/util/compat.py", line 135, in load_module_py
    mod = imp.load_source(module_id, path, fp)
 File "migrations/env.py", line 104, in <module>
    run_migrations_online()
  File "migrations/env.py", line 88, in run_migrations_online
    connection = engine.connect()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2102, in connect
    return self._connection_cls(self, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 90, in __init__
    if connection is not None else engine.raw_connection()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2188, in raw_connection
    self.pool.unique_connection, _connection)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
    e, dialect, self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
    exc_info
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
    return fn()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 342, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 788, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 529, in checkout
    rec = pool._do_get()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1284, in _do_get
    return self._create_connection()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 347, in _create_connection
    return _ConnectionRecord(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 474, in __init__
    self.__connect(first_connect_check=True)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 671, in __connect
    connection = pool._invoke_creator(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/strategies.py", line 106, in connect
    return dialect.connect(*cargs, **cparams)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/default.py", line 412, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/tmp/crmint-bmind3/backends/lib/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'crmintapp'@'cloudsqlproxy~35.205.214.149' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)

==> Sowing DB seeds

2018/10/22 17:32:30 New connection for "crmint-bmind3:europe-west1:crmintapp"
Traceback (most recent call last):
2018/10/22 17:32:30 Client closed local connection on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/crmint-bmind3/backends/lib/flask/__main__.py", line 15, in <module>
    main(as_module=True)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 513, in main
    cli.main(args=args, prog_name=name)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 380, in main
    return AppGroup.main(self, *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 257, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/flask_tasks.py", line 25, in db_seeds
    database.load_fixtures(logger_func=click.echo)
  File "/tmp/crmint-bmind3/backends/core/database.py", line 60, in load_fixtures
    general_setting = models.GeneralSetting.where(name=setting).first()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2895, in first
    ret = list(self[0:1])
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2687, in __getitem__
    return list(res)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2995, in __iter__
    return self._execute_and_instances(context)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3016, in _execute_and_instances
    close_with_result=True)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3025, in _get_bind_args
    **kw
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3007, in _connection_from_session
    conn = self.session.connection(**kw)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1035, in connection
    execution_options=execution_options)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
    conn = engine.contextual_connect(**kw)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2123, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
    e, dialect, self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
    exc_info
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
    return fn()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 400, in connect
    return _ConnectionFairy._checkout(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 788, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 529, in checkout
    rec = pool._do_get()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1193, in _do_get
    self._dec_overflow()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1190, in _do_get
    return self._create_connection()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 347, in _create_connection
    return _ConnectionRecord(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 474, in __init__
    self.__connect(first_connect_check=True)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 671, in __connect
    connection = pool._invoke_creator(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/strategies.py", line 106, in connect
    return dialect.connect(*cargs, **cparams)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/default.py", line 412, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/tmp/crmint-bmind3/backends/lib/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'crmintapp'@'cloudsqlproxy~35.205.214.149' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)

once deployed in app engine, I see this:

Any clue about this issue? Any tips on how to deploy crmint in app engine ?

Also, in Cloud Shell i've tried to reset the statuses of all jobs and pipelines:

bin/app cloud reset

==> Reseting statuses of jobs and pipelines.

2018/10/22 17:42:51 New connection for "crmint-bmind3:europe-west1:crmintapp"
Traceback (most recent call last):
2018/10/22 17:42:51 Client closed local connection on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/crmint-bmind3/backends/lib/flask/__main__.py", line 15, in <module>
    main(as_module=True)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 513, in main
    cli.main(args=args, prog_name=name)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 380, in main
    return AppGroup.main(self, *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 257, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/flask_tasks.py", line 30, in reset_pipelines
    for pipeline in Pipeline.all():
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy_mixins/activerecord.py", line 64, in all
    return cls.query.all()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2843, in all
    return list(self)
 File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2995, in __iter__
    return self._execute_and_instances(context)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3016, in _execute_and_instances
    close_with_result=True)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3025, in _get_bind_args
    **kw
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3007, in _connection_from_session
    conn = self.session.connection(**kw)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1035, in connection
    execution_options=execution_options)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
    conn = engine.contextual_connect(**kw)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2123, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
    e, dialect, self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
    exc_info
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
    return fn()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 400, in connect
    return _ConnectionFairy._checkout(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 788, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 529, in checkout
    rec = pool._do_get()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1193, in _do_get
    self._dec_overflow()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1190, in _do_get
    return self._create_connection()
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 347, in _create_connection
    return _ConnectionRecord(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 474, in __init__
    self.__connect(first_connect_check=True)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 671, in __connect
    connection = pool._invoke_creator(self)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/strategies.py", line 106, in connect
    return dialect.connect(*cargs, **cparams)
  File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/default.py", line 412, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/tmp/crmint-bmind3/backends/lib/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/tmp/crmint-bmind3/backends/lib/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'crmintapp'@'cloudsqlproxy~35.205.214.149' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)

Thank you

Implement product lists for Enhanced Ecommerce in Measurement Protocol hits

It would be great if we were able to send MP hits with several products and product lists as described in Enhanced Ecommerce Tracking.

Write a test suite for the frontend

The frontend is currently without a test suite. Please write a test suite for the frontend.

Customer Match Worker

We need to be able to use emails instead of user_id's to create smart (RFMed etc.) lists in Google Ads.

Deploy Guide: Open in Cloud Shell button doesn't handle digits correctly

The Open Cloud Shell button doesn't always activate when digits are present in the Cloud Project Name. Seen in the deploy guide at Deploy CRMint Quickstart.

Works: crmint-3
Fails: crmint-3n
Fails: crmin-n3

GA2BQ Importer - Google analytics API error

I started getting the following error today when using the GA2BQ importer:

It seems like the domain name of the GA API calls must change from "analytics.googleapis.com" to "analyticsreporting.googleapis.com" (according to this and this )

Is anybody else having similar issues?

Error from gcloud in deployment guide

When running the deployment guide the final step to launch CRMint in a browser fails:

garan@cloudshell:~/crmint (my-project-name)$ gcloud app browse
Traceback (most recent call last):
  File "/google/google-cloud-sdk/lib/gcloud.py", line 20, in <module>
    from __future__ import absolute_import
ImportError: No module named __future__
garan@cloudshell:~/crmint (my-project-name)$

Update frontend to Angular 8

CRMint is currently using Angular 4.4.6. To help with maintainability and to improve frontend performance, an update to the latest version of Angular (8.0 as of 2019-06-21) can be carried out.

Starting]

Implement Google Ads Authentication

Google Ads Authentication is not yet implemented and so we cannot generate workers that use this to access the Adwords API

Implement importing of offline app conversions

Be able to take data from BigQuery table and make HTTP requests for each app conversion

Scheduled pipelines getting stuck at 'running'

Recently I have upgraded my CRMint installation and added several new pipelines, since this change I have experienced pipelines getting stuck on 'running' for several hours, requiring manual stopping.

If I stop the pipelines and restart them manually, they complete successfully.

Here are some screenshots from the jobs:

Deploy CRMint in Google Cloud, DB migrations / Sowing DB seeds ERROR

==> Applying DB migrations

Applying database migrations: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/coral-mission-244416/backends/lib/flask/__main__.py", line 15, in <module> main(as_module=True) File "/tmp/coral-mission-244416/backends/lib/flask/cli.py", line 513, in main cli.main(args=args, prog_name=name) File "/tmp/coral-mission-244416/backends/lib/flask/cli.py", line 380, in main return AppGroup.main(self, *args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 717, in main rv = self.invoke(ctx) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/flask/cli.py", line 257, in decorator return __ctx.invoke(f, *args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/flask_migrate/cli.py", line 132, in upgrade _upgrade(directory, revision, sql, tag, x_arg) File "/tmp/coral-mission-244416/backends/lib/flask_migrate/__init__.py", line 244, in upgrade command.upgrade(config, revision, sql=sql, tag=tag) File "/tmp/coral-mission-244416/backends/lib/alembic/command.py", line 276, in upgrade script.run_env() File "/tmp/coral-mission-244416/backends/lib/alembic/script/base.py", line 475, in run_env util.load_python_file(self.dir, "env.py") File "/tmp/coral-mission-244416/backends/lib/alembic/util/pyfiles.py", line 90, in load_python_file module = load_module_py(module_id, path) File "/tmp/coral-mission-244416/backends/lib/alembic/util/compat.py", line 216, in load_module_py mod = imp.load_source(module_id, path, fp) File "migrations/env.py", line 104, in <module> run_migrations_online() File "migrations/env.py", line 88, in run_migrations_online connection = engine.connect() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2193, in connect return self._connection_cls(self, **kwargs) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 103, in __init__ else engine.raw_connection() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2293, in raw_connection self.pool.unique_connection, _connection File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect e, dialect, self File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/util/compat.py", line 399, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect return fn() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 303, in unique_connection return _ConnectionFairy._checkout(self) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 760, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 492, in checkout rec = pool._do_get() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/impl.py", line 238, in _do_get return self._create_connection() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 308, in _create_connection return _ConnectionRecord(self) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 437, in __init__ self.__connect(first_connect_check=True) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 639, in __connect connection = pool._invoke_creator(self) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/default.py", line 451, in connect return self.dbapi.connect(*cargs, **cparams) File "/tmp/coral-mission-244416/backends/lib/MySQLdb/__init__.py", line 81, in Connect return Connection(*args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/MySQLdb/connections.py", line 193, in __init__ super(Connection, self).__init__(*args, **kwargs2) sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2002, "Can't connect to local MySQL server through socket '/tmp/cloudsql/coral-mission-244416:us-central1:crmintapp' (2)") (Background on this error at: http://sqlalche.me/e/e3q8)

Anyone can help me? I don't know waht to do.

Tests and other scripts seem to be missing some dependencies

Trying to run the following commands for a local dev setup (MacOS) after navigating to backends directory, following the wiki guide. Output below:

python runtests.py $GOOGLE_CLOUD_SDK

Traceback (most recent call last):
  File "runtests.py", line 98, in <module>
    result = main(args.sdk_path, args.test_path, args.test_pattern)
  File "runtests.py", line 60, in main
    import dev_appserver
ImportError: No module named dev_appserver

python run_ibackend.py

Traceback (most recent call last):
  File "run_ibackend.py", line 16, in <module>
    import appengine_config
  File "/Users/ayman/projects/crmint/backends/appengine_config.py", line 20, in <module>
    from google.appengine.ext import vendor
ImportError: No module named google.appengine.ext

python run_jbackend.py

Traceback (most recent call last):
  File "run_jbackend.py", line 18, in <module>
    from jbackend.app import create_app
  File "/Users/ayman/projects/crmint/backends/jbackend/app.py", line 21, in <module>
    from core.database import init_engine
  File "/Users/ayman/projects/crmint/backends/core/database.py", line 15, in <module>
    from sqlalchemy import create_engine
ImportError: No module named sqlalchemy

All dependencies install in the virtualenv already.

Starting]

Add Notification for old versions of CRMint

Hey team,

As discussed can we please add a notification for the users who deployed old versions of CRMint to notify them that there is a new version to upgrade to?

Thanks,

Commands running flask tasks with underscores in task names fail with error

bin/app cloud reset fails with error:

Error: No such command "reset_pipelines".

Global and pipeline variables are not being saved.

Pipeline settings are not being saved when pipeline variables are specified. Same happens when saving global variables on the Settings tab. "500 Internal Server Error" is seen in the network panel in browser dev tools.

Smart Notifications Feature (perhaps as a worker)

Some pipelines require complex rules fo notifications that make sense - i.e. when a certain pipeline is scheduled to run hourly because the fresh source data file arrives at various intervals - such pipeline crushes multiple times a day and its fine as long as it works once within last 24 h.

Table schema details for BQToMeasurementProtocol

Wanting to document this - what should the table schema be? What aspects of MP are supported?

No warning if the worker class name doesn't exist

To replicate this bug, upload a pipeline with a worker class name that is not available in your particular instance:

either a worker from dev branch, while your instance was deployed from master branch
or the worker is a private implementation from another user

There might be other cases leading to the same issue.
It could be fixed with a pre-check before running the pipeline.

Cheers

Deploy CRMint in Google Cloud, Creating MySQL user

I'm using the docs from: https://github.com/google/crmint/wiki/Deploy-CRMint-on-Google-Cloud-Platform to deploy CRMint in Google Cloud (master branch). On the second command:

# Configures the GCP project with the resources needed.
bin/app cloud setup

from google cloud console I get this issue:

==> Creating MySQL user
ERROR: (gcloud.sql.users.create) Positional argument deprecated_host has been removed. Use --host instead.
Usage: gcloud sql users create USERNAME --instance=INSTANCE, -i INSTANCE [optional flags]
  optional flags may be  --async | --help | --host | --password

I've double checked, and in SQL Cloud instance the user "crmintapp" is missing. Anyway, manually I created the user for mysql with all credentials from /script/variables/stages/<my_project_id>.sh and job done.

as a fix, in script/cloud/setup.sh this line:

   gcloud sql users create $db_username % --instance $db_instance_name \
	    --password $db_password --project $project_id_gae --quiet

should be replaced by:

gcloud sql users create $db_username --host % --instance $db_instance_name \
	    --password $db_password --project $project_id_gae --quiet

Thank you !

Add troubleshooting hints to quickstart tutorial

It would be nice to include some troubleshooting hints into the quickstart tutorial.

E.g. we could add under "Run and check evaluation metrics":
4. In case the pipeline fails, click on the "Logs" tab and read the error message displayed. Additionally, click on "Load newest" to ensure the latest error messages have been loaded.

Slow deployment

I've analyzed a little bit why the deployment of the frontend was super slow.

One of the main culprit appears to be the angular cli compilation: takes a crazy amount of time, and a huge amount of memory, making the Cloud Shell VM unresponsive for 20 minutes sometimes.

First, to avoid too much memory to be consumed, let's limit node to 512MB of ram with: node --max-old-space-size=512 ./node_modules/.bin/ng ....

With the --prod flag

$ time node --max-old-space-size=512 ./node_modules/.bin/ng build --prod 
...
real    18m40.182s
user   15m35.933s
sys     0m5.094s

Without optimization

$ time node --max-old-space-size=512 ./node_modules/.bin/ng build
...
real    0m32.093s
user   0m23.547s
sys     0m1.458s

Since CRMint is not meant for high-through put access. I would suggest to drop the --prod flag.
What do you think @Slony?

Cheers

GAToBQImporter - All values are NULL in BigQuery

Hello team. I have just tested CRMINT and is an amazing tool for marketing automation in Google Cloud (we plan to use it and even develop some workers eg: connect to sftp client, grab all files and store them in Google Storage).

I have a little issue with: "GAToBQImporter".

I've setup the pipeline with a job that is using "GAToBQImporter" and everything works just fine (got the message with "Pipeline succeeded", all logs are great in stackdriver, in bigquery, got all rows (3759 - i've compared CRMINT results vs APIv4 GA vs Query Explorer) but all values are NULL). Am I'm doing something wrong?

My schema for BigQuery "output" table is:
[
{
"mode": "NULLABLE",
"name": "field1",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "field2",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "field3",
"type": "STRING"
}
]

Sample data inserted via CRMINT with GAToBQImporter in BigQuery output table:

[
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
}
..
..
..
..
]

Job setup:
"jobs": [
{
"hash_start_conditions": [],
"worker_class": "GAToBQImporter",
"params": [
{
"description": null,
"value": "122591566",
"label": null,
"is_required": false,
"type": "string_list",
"name": "view_ids"
},
{
"description": null,
"value": "2018-08-01",
"label": null,
"is_required": false,
"type": "string",
"name": "start_date"
},
{
"description": null,
"value": "2018-08-05",
"label": null,
"is_required": false,
"type": "string",
"name": "end_date"
},
{
"description": null,
"value": false,
"label": null,
"is_required": false,
"type": "boolean",
"name": "day_by_day"
},
{
"description": null,
"value": "ga:users",
"label": null,
"is_required": false,
"type": "string_list",
"name": "metrics"
},
{
"description": null,
"value": "ga:source\nga:date",
"label": null,
"is_required": false,
"type": "string_list",
"name": "dimensions"
},
{
"description": null,
"value": "",
"label": null,
"is_required": false,
"type": "string",
"name": "filters"
},
{
"description": null,
"value": true,
"label": null,
"is_required": false,
"type": "boolean",
"name": "include_empty_rows"
},
{
"description": null,
"value": "BMIND",
"label": null,
"is_required": false,
"type": "string",
"name": "bq_project_id"
},
{
"description": null,
"value": "CRMINT",
"label": null,
"is_required": false,
"type": "string",
"name": "bq_dataset_id"
},
{
"description": null,
"value": "output",
"label": null,
"is_required": false,
"type": "string",
"name": "bq_table_id"
}
],
"id": "1de006a76d7c40489bb740b0157fbb5c",
"name": "GA REPORT TO BQ"
},
],

Cheers

Create GAToBQImporter friendly error if no row in report

"Fetch data day by day"

An option in GAToBQImporter - what does this mean?

500 error if SQL request is too long

Hi,

Thanks for CRMint, it is a great tool and we start using it. Could be our future tool for data operations.

I have this 500 error when I try to save a BQQueryLaucher job.
when I looked into the trace of the error in the logs, I saw that my SQL request was too long :

DataError: (_mysql_exceptions.DataError) (1406, "Data too long for column 'value' at row 1") [SQL: u'INSERT INTO params (created_at, updated_at, name, type, pipeline_id, job_id, is_required, description, label, value, runtime_value) VALUES (now(), now(), %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (u'query', u'sql', None, 33L, 0, None, u'Query', '#standardSQL\r\nWITH\r\n  scope_jour AS (\r\n  SELECT\r\n    *\r\n  FROM\r\n    -- `engie-b2c-cloud.ga_flat_DEV.ga_sessions_20190404_vincent`),\r\n   ... (137443 characters truncated) ... tic42,\r\n  Statistic43,\r\n  Statistic44,\r\n  Statistic45,\r\n  Statistic46,\r\n  Statistic47,\r\n  Statistic48,\r\n  Statistic49,\r\n  Statistic50', None)] (Background on this error at: http://sqlalche.me/e/9h9h)

Can we make something about this problem ?

UI link is a shortlink, not just a hyperlink

When trying to add a BQToMeasurementProtocol the measurement protocol batch size field has a short-link, as opposed to a link to the page on developers.google.com to launch in a new tab. "https://goo.gl/7VeWuB"

Keep pipeline variables ordering in the UI as the one in the JSON template file

Commenter job generates meaningless log message

When a Commenter job runs with "Finish successfully" flag unchecked, it fails with a meaningless message: Execution failed: WorkerException: "False" is unchecked: … in a pipeline log.

Can't start a single job when it's start conditions are not met

Starting a single job fails when statuses of it's parent jobs do not meet it's starting conditions.

The starting conditions should not be taken into account when a single job is being started.

GAAudiencesUpdater

Is there somewhere I can find more about the format of "GA audience JSON template"

Move stage configurations

Hey team,

As discussed, can we please move stage configurations from Cloud Shell file-system to a shared environment like Google Cloud Storage?

Thanks,

Drive scope missing in BigQuery client

When querying a table that has an external data source in Google Drive, BQQueryLauncher worker fails with the following error:

Access Denied: BigQuery BigQuery: No OAuth token with Google Drive scope was found.

"None" instead of a param name is displayed in logs when param's value is incorrect

Bad job param "None": … shown in logs when the param's value is invalid. Log message should look like this: Bad job param "BQ Project ID": …

Pipelines hang and an error displayed instead of logs

All pipelines hang upon start with the first jobs in state running and the rest of the jobs in state waiting. Red error is always shown in Logs tab of web UI.

Debug mode for Measurement Protocol hits

When sending Measurement Protocol hits to Google Analytics, sometimes it's very difficult to troubleshoot discrepancy between the number of rows in a BigQuery table and the number of hits observed in Google Analytics.

It would be great to implement debug mode in BQToMeasurementProtocol worker and use Validation Service to identify invalid hits.

Move Cloud deploy scripts to main repo

It would be nice to have quick-start scripts in the main CRMint repo, wouldn't it?

Pipeline silently fails to start when there is a syntax error in a job param expression

A pipeline won't start and will show nothing in logs if there is a syntax error in a job param expression, e.g. missing parentheses like this:

{% today('%Y-%m-%d' %}

Some files have root ownership during local setup

I've set up the project on my Linux machine, and files like instance/config.py or data/service_account.json are created with root as the owner.

So, to manually fix it I had to run $ sudo chown <me>:<me> backends/instance/config.py.

It could also be explained in a Linux set-up section in the Wiki.

Cheers

Worker class for DML queries

I think there's a nice feature that is missing. I couldn't find any workers available, which would allow someone to execute DML queries (e.g. DELETE). It would be really useful, as such a worker would allow someone to clear a BQ table, before importing new GA data with the GAToBQImporter.

I already tried to use the BQQueryLauncher worker class with a DELETE statement, but I'm getting the following error:

Job: XXXX, Worker Class: BQQueryLauncher
Execution failed: WorkerException: Cannot set destination table in jobs with DML statements

Cannot stop a pipeline that is stucked in "running" mode

If a pipeline is ever stuck, for any underlying reason, it's impossible to stop it and get back a idle status.

We should make sure that the stopping method is robust enough to reset the pipeline status.

Cheers

Failed to start job if the starting condition is not set to `success`

The pipeline was failing if we only select the name of the parent job, without specifying success or failure.

This is a regression, probably need a test to replicate that bug.
I'm on it.

Cheers

Add a worker to check if the files matching a pattern exist in Google Cloud Storage

It would be nice to have a worker that can check if the files matching given filename pattern exist in Cloud Storage.

Checking if existing files have total size greater than a given threshold would be helpful as well.

Pipeline visualization disappears when adding broken dependencys

As a user I am able to add conflicting dependencies to a job, which then makes the pipeline page a blank screen. I can navigate to the jobs screen and see the job with multiple dependencies, but when I click to edit it I also get an empty screen. If I delete the job, then this fixes the issue and the pipeline appears again, although without the job that I broke!

It would be better if there was an error/blocker to stop a user applying broken dependencies initially.

google / crmint Goto Github PK

crmint's Introduction

CRMint

Deploy CRMint

Ambition

crmint's People

Contributors

Stargazers

Watchers

Forkers

crmint's Issues

Recommend Projects

Recommend Topics

Recommend Org