google / crmint Goto Github PK
View Code? Open in Web Editor NEWReliable data integration & processing for advertisers
Home Page: https://google-marketing-solutions.github.io/crmint/
License: Other
Reliable data integration & processing for advertisers
Home Page: https://google-marketing-solutions.github.io/crmint/
License: Other
Bad job param "None": …
shown in logs when the param's value is invalid. Log message should look like this: Bad job param "BQ Project ID": …
Recently I have upgraded my CRMint installation and added several new pipelines, since this change I have experienced pipelines getting stuck on 'running' for several hours, requiring manual stopping.
If I stop the pipelines and restart them manually, they complete successfully.
Here are some screenshots from the jobs:
Some pipelines require complex rules fo notifications that make sense - i.e. when a certain pipeline is scheduled to run hourly because the fresh source data file arrives at various intervals - such pipeline crushes multiple times a day and its fine as long as it works once within last 24 h.
When a Commenter
job runs with "Finish successfully" flag unchecked, it fails with a meaningless message: Execution failed: WorkerException: "False" is unchecked: …
in a pipeline log.
I'm using the docs from: https://github.com/google/crmint/wiki/Deploy-CRMint-on-Google-Cloud-Platform to deploy CRMint in Google Cloud (master branch). On the second command:
cd crmint
bin/app cloud setup
bin/app cloud deploy
.. at the end of deploy i've got some errors on this steps:
==> Applying DB migrations
2018/10/22 17:32:28 Listening on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp for crmint-bmind3:europe-west1:crmintapp
2018/10/22 17:32:28 Ready for new connections
2018/10/22 17:32:29 New connection for "crmint-bmind3:europe-west1:crmintapp"
Traceback (most recent call last):
2018/10/22 17:32:29 Client closed local connection on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/tmp/crmint-bmind3/backends/lib/flask/__main__.py", line 15, in <module>
main(as_module=True)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 513, in main
cli.main(args=args, prog_name=name)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 380, in main
return AppGroup.main(self, *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/flask_migrate/cli.py", line 132, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/tmp/crmint-bmind3/backends/lib/flask_migrate/__init__.py", line 244, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/tmp/crmint-bmind3/backends/lib/alembic/command.py", line 254, in upgrade
script.run_env()
File "/tmp/crmint-bmind3/backends/lib/alembic/script/base.py", line 427, in run_env
util.load_python_file(self.dir, 'env.py')
File "/tmp/crmint-bmind3/backends/lib/alembic/util/pyfiles.py", line 81, in load_python_file
module = load_module_py(module_id, path)
File "/tmp/crmint-bmind3/backends/lib/alembic/util/compat.py", line 135, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 104, in <module>
run_migrations_online()
File "migrations/env.py", line 88, in run_migrations_online
connection = engine.connect()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2102, in connect
return self._connection_cls(self, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 90, in __init__
if connection is not None else engine.raw_connection()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2188, in raw_connection
self.pool.unique_connection, _connection)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
e, dialect, self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
exc_info
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
return fn()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 342, in unique_connection
return _ConnectionFairy._checkout(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 788, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 529, in checkout
rec = pool._do_get()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1284, in _do_get
return self._create_connection()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 347, in _create_connection
return _ConnectionRecord(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 474, in __init__
self.__connect(first_connect_check=True)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 671, in __connect
connection = pool._invoke_creator(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/strategies.py", line 106, in connect
return dialect.connect(*cargs, **cparams)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/default.py", line 412, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/tmp/crmint-bmind3/backends/lib/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/MySQLdb/connections.py", line 193, in __init__
super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'crmintapp'@'cloudsqlproxy~35.205.214.149' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
==> Sowing DB seeds
2018/10/22 17:32:30 New connection for "crmint-bmind3:europe-west1:crmintapp"
Traceback (most recent call last):
2018/10/22 17:32:30 Client closed local connection on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/tmp/crmint-bmind3/backends/lib/flask/__main__.py", line 15, in <module>
main(as_module=True)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 513, in main
cli.main(args=args, prog_name=name)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 380, in main
return AppGroup.main(self, *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/flask_tasks.py", line 25, in db_seeds
database.load_fixtures(logger_func=click.echo)
File "/tmp/crmint-bmind3/backends/core/database.py", line 60, in load_fixtures
general_setting = models.GeneralSetting.where(name=setting).first()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2895, in first
ret = list(self[0:1])
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2687, in __getitem__
return list(res)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2995, in __iter__
return self._execute_and_instances(context)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3016, in _execute_and_instances
close_with_result=True)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3025, in _get_bind_args
**kw
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3007, in _connection_from_session
conn = self.session.connection(**kw)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1035, in connection
execution_options=execution_options)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
conn = engine.contextual_connect(**kw)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2123, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
e, dialect, self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
exc_info
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
return fn()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 400, in connect
return _ConnectionFairy._checkout(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 788, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 529, in checkout
rec = pool._do_get()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1193, in _do_get
self._dec_overflow()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1190, in _do_get
return self._create_connection()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 347, in _create_connection
return _ConnectionRecord(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 474, in __init__
self.__connect(first_connect_check=True)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 671, in __connect
connection = pool._invoke_creator(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/strategies.py", line 106, in connect
return dialect.connect(*cargs, **cparams)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/default.py", line 412, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/tmp/crmint-bmind3/backends/lib/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/MySQLdb/connections.py", line 193, in __init__
super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'crmintapp'@'cloudsqlproxy~35.205.214.149' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
once deployed in app engine, I see this:
Any clue about this issue? Any tips on how to deploy crmint in app engine ?
Also, in Cloud Shell i've tried to reset the statuses of all jobs and pipelines:
bin/app cloud reset
==> Reseting statuses of jobs and pipelines.
2018/10/22 17:42:51 New connection for "crmint-bmind3:europe-west1:crmintapp"
Traceback (most recent call last):
2018/10/22 17:42:51 Client closed local connection on /tmp/cloudsql/crmint-bmind3:europe-west1:crmintapp
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/tmp/crmint-bmind3/backends/lib/flask/__main__.py", line 15, in <module>
main(as_module=True)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 513, in main
cli.main(args=args, prog_name=name)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 380, in main
return AppGroup.main(self, *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/flask/cli.py", line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/flask_tasks.py", line 30, in reset_pipelines
for pipeline in Pipeline.all():
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy_mixins/activerecord.py", line 64, in all
return cls.query.all()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2843, in all
return list(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 2995, in __iter__
return self._execute_and_instances(context)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3016, in _execute_and_instances
close_with_result=True)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3025, in _get_bind_args
**kw
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/query.py", line 3007, in _connection_from_session
conn = self.session.connection(**kw)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1035, in connection
execution_options=execution_options)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
conn = engine.contextual_connect(**kw)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2123, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
e, dialect, self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
exc_info
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
return fn()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 400, in connect
return _ConnectionFairy._checkout(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 788, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 529, in checkout
rec = pool._do_get()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1193, in _do_get
self._dec_overflow()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 1190, in _do_get
return self._create_connection()
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 347, in _create_connection
return _ConnectionRecord(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 474, in __init__
self.__connect(first_connect_check=True)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/pool.py", line 671, in __connect
connection = pool._invoke_creator(self)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/strategies.py", line 106, in connect
return dialect.connect(*cargs, **cparams)
File "/tmp/crmint-bmind3/backends/lib/sqlalchemy/engine/default.py", line 412, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/tmp/crmint-bmind3/backends/lib/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/tmp/crmint-bmind3/backends/lib/MySQLdb/connections.py", line 193, in __init__
super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'crmintapp'@'cloudsqlproxy~35.205.214.149' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
Thank you
It would be nice to have quick-start scripts in the main CRMint repo, wouldn't it?
The frontend is currently without a test suite. Please write a test suite for the frontend.
All pipelines hang upon start with the first jobs in state running
and the rest of the jobs in state waiting
. Red error is always shown in Logs
tab of web UI.
To replicate this bug, upload a pipeline with a worker class name that is not available in your particular instance:
There might be other cases leading to the same issue.
It could be fixed with a pre-check before running the pipeline.
Cheers
Trying to run the following commands for a local dev setup (MacOS) after navigating to backends directory, following the wiki guide. Output below:
python runtests.py $GOOGLE_CLOUD_SDK
Traceback (most recent call last):
File "runtests.py", line 98, in <module>
result = main(args.sdk_path, args.test_path, args.test_pattern)
File "runtests.py", line 60, in main
import dev_appserver
ImportError: No module named dev_appserver
python run_ibackend.py
Traceback (most recent call last):
File "run_ibackend.py", line 16, in <module>
import appengine_config
File "/Users/ayman/projects/crmint/backends/appengine_config.py", line 20, in <module>
from google.appengine.ext import vendor
ImportError: No module named google.appengine.ext
python run_jbackend.py
Traceback (most recent call last):
File "run_jbackend.py", line 18, in <module>
from jbackend.app import create_app
File "/Users/ayman/projects/crmint/backends/jbackend/app.py", line 21, in <module>
from core.database import init_engine
File "/Users/ayman/projects/crmint/backends/core/database.py", line 15, in <module>
from sqlalchemy import create_engine
ImportError: No module named sqlalchemy
All dependencies install in the virtualenv already.
When trying to add a BQToMeasurementProtocol the measurement protocol batch size field has a short-link, as opposed to a link to the page on developers.google.com to launch in a new tab. "https://goo.gl/7VeWuB"
Hi,
Thanks for CRMint, it is a great tool and we start using it. Could be our future tool for data operations.
I have this 500 error when I try to save a BQQueryLaucher job.
when I looked into the trace of the error in the logs, I saw that my SQL request was too long :
DataError: (_mysql_exceptions.DataError) (1406, "Data too long for column 'value' at row 1") [SQL: u'INSERT INTO params (created_at, updated_at, name, type, pipeline_id, job_id, is_required, description, label, value, runtime_value) VALUES (now(), now(), %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (u'query', u'sql', None, 33L, 0, None, u'Query', '#standardSQL\r\nWITH\r\n scope_jour AS (\r\n SELECT\r\n *\r\n FROM\r\n -- `engie-b2c-cloud.ga_flat_DEV.ga_sessions_20190404_vincent`),\r\n ... (137443 characters truncated) ... tic42,\r\n Statistic43,\r\n Statistic44,\r\n Statistic45,\r\n Statistic46,\r\n Statistic47,\r\n Statistic48,\r\n Statistic49,\r\n Statistic50', None)] (Background on this error at: http://sqlalche.me/e/9h9h)
Can we make something about this problem ?
Hey team,
As discussed, can we please move stage configurations from Cloud Shell file-system to a shared environment like Google Cloud Storage?
Thanks,
We need to be able to use emails instead of user_id's to create smart (RFMed etc.) lists in Google Ads.
Is there somewhere I can find more about the format of "GA audience JSON template"
It would be great if we were able to send MP hits with several products and product lists as described in Enhanced Ecommerce Tracking.
In StorageToBQImporter
Double negatives confuse me :-)
Does this mean "Require table to already exist" ?
Or is it slightly different?
Starting a single job fails when statuses of it's parent jobs do not meet it's starting conditions.
The starting conditions should not be taken into account when a single job is being started.
The link to the predicted values, in step 5 of "Let's predict some prices" doesn't launch in a new window.
Consider making this launch in a new window.
Also, consider adding the project and dataset details to make them explicit in the URL, e.g.
https://console.cloud.google.com/bigquery?d=predictions
vs
Pipeline settings are not being saved when pipeline variables are specified. Same happens when saving global variables on the Settings tab. "500 Internal Server Error" is seen in the network panel in browser dev tools.
I have a pipeline job that has been failing for the last three days the following error:
Unexpected error: ConnectionError: ('Connection aborted.', HTTPException('Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/bigquery/v2/projects/PROJECT_NAME/jobs',))
This job has been running every day for many months without an issue and has remained unchanged, has there been some change to BigQuery that could cause this?
It would be nice to have a worker that can check if the files matching given filename pattern exist in Cloud Storage.
Checking if existing files have total size greater than a given threshold would be helpful as well.
Wanting to document this - what should the table schema be? What aspects of MP are supported?
Google Ads Authentication is not yet implemented and so we cannot generate workers that use this to access the Adwords API
It would be nice to include some troubleshooting hints into the quickstart tutorial.
E.g. we could add under "Run and check evaluation metrics":
4. In case the pipeline fails, click on the "Logs" tab and read the error message displayed. Additionally, click on "Load newest" to ensure the latest error messages have been loaded.
When querying a table that has an external data source in Google Drive, BQQueryLauncher
worker fails with the following error:
Access Denied: BigQuery BigQuery: No OAuth token with Google Drive scope was found.
I've set up the project on my Linux machine, and files like instance/config.py
or data/service_account.json
are created with root
as the owner.
So, to manually fix it I had to run $ sudo chown <me>:<me> backends/instance/config.py
.
It could also be explained in a Linux set-up section in the Wiki.
Cheers
Hello team. I have just tested CRMINT and is an amazing tool for marketing automation in Google Cloud (we plan to use it and even develop some workers eg: connect to sftp client, grab all files and store them in Google Storage).
I have a little issue with: "GAToBQImporter".
I've setup the pipeline with a job that is using "GAToBQImporter" and everything works just fine (got the message with "Pipeline succeeded", all logs are great in stackdriver, in bigquery, got all rows (3759 - i've compared CRMINT results vs APIv4 GA vs Query Explorer) but all values are NULL). Am I'm doing something wrong?
My schema for BigQuery "output" table is:
[
{
"mode": "NULLABLE",
"name": "field1",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "field2",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "field3",
"type": "STRING"
}
]
Sample data inserted via CRMINT with GAToBQImporter in BigQuery output table:
[
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
},
{
"field1": null,
"field2": null,
"field3": null
}
..
..
..
..
]
Job setup:
"jobs": [
{
"hash_start_conditions": [],
"worker_class": "GAToBQImporter",
"params": [
{
"description": null,
"value": "122591566",
"label": null,
"is_required": false,
"type": "string_list",
"name": "view_ids"
},
{
"description": null,
"value": "2018-08-01",
"label": null,
"is_required": false,
"type": "string",
"name": "start_date"
},
{
"description": null,
"value": "2018-08-05",
"label": null,
"is_required": false,
"type": "string",
"name": "end_date"
},
{
"description": null,
"value": false,
"label": null,
"is_required": false,
"type": "boolean",
"name": "day_by_day"
},
{
"description": null,
"value": "ga:users",
"label": null,
"is_required": false,
"type": "string_list",
"name": "metrics"
},
{
"description": null,
"value": "ga:source\nga:date",
"label": null,
"is_required": false,
"type": "string_list",
"name": "dimensions"
},
{
"description": null,
"value": "",
"label": null,
"is_required": false,
"type": "string",
"name": "filters"
},
{
"description": null,
"value": true,
"label": null,
"is_required": false,
"type": "boolean",
"name": "include_empty_rows"
},
{
"description": null,
"value": "BMIND",
"label": null,
"is_required": false,
"type": "string",
"name": "bq_project_id"
},
{
"description": null,
"value": "CRMINT",
"label": null,
"is_required": false,
"type": "string",
"name": "bq_dataset_id"
},
{
"description": null,
"value": "output",
"label": null,
"is_required": false,
"type": "string",
"name": "bq_table_id"
}
],
"id": "1de006a76d7c40489bb740b0157fbb5c",
"name": "GA REPORT TO BQ"
},
],
Cheers
A pipeline won't start and will show nothing in logs if there is a syntax error in a job param expression, e.g. missing parentheses like this:
{% today('%Y-%m-%d' %}
CRMint is currently using Angular 4.4.6. To help with maintainability and to improve frontend performance, an update to the latest version of Angular (8.0 as of 2019-06-21) can be carried out.
As a user I am able to add conflicting dependencies to a job, which then makes the pipeline page a blank screen. I can navigate to the jobs screen and see the job with multiple dependencies, but when I click to edit it I also get an empty screen. If I delete the job, then this fixes the issue and the pipeline appears again, although without the job that I broke!
It would be better if there was an error/blocker to stop a user applying broken dependencies initially.
bin/app cloud reset
fails with error:
Error: No such command "reset_pipelines".
If a pipeline is ever stuck, for any underlying reason, it's impossible to stop it and get back a idle
status.
We should make sure that the stopping method is robust enough to reset the pipeline status.
Cheers
I think there's a nice feature that is missing. I couldn't find any workers available, which would allow someone to execute DML queries (e.g. DELETE). It would be really useful, as such a worker would allow someone to clear a BQ table, before importing new GA data with the GAToBQImporter.
I already tried to use the BQQueryLauncher worker class with a DELETE statement, but I'm getting the following error:
Job: XXXX, Worker Class: BQQueryLauncher
Execution failed: WorkerException: Cannot set destination table in jobs with DML statements
I'm using the docs from: https://github.com/google/crmint/wiki/Deploy-CRMint-on-Google-Cloud-Platform to deploy CRMint in Google Cloud (master branch). On the second command:
# Configures the GCP project with the resources needed.
bin/app cloud setup
from google cloud console I get this issue:
==> Creating MySQL user
ERROR: (gcloud.sql.users.create) Positional argument deprecated_host has been removed. Use --host instead.
Usage: gcloud sql users create USERNAME --instance=INSTANCE, -i INSTANCE [optional flags]
optional flags may be --async | --help | --host | --password
I've double checked, and in SQL Cloud instance the user "crmintapp" is missing. Anyway, manually I created the user for mysql with all credentials from /script/variables/stages/<my_project_id>.sh and job done.
as a fix, in script/cloud/setup.sh this line:
gcloud sql users create $db_username % --instance $db_instance_name \
--password $db_password --project $project_id_gae --quiet
should be replaced by:
gcloud sql users create $db_username --host % --instance $db_instance_name \
--password $db_password --project $project_id_gae --quiet
Thank you !
When running the deployment guide the final step to launch CRMint in a browser fails:
garan@cloudshell:~/crmint (my-project-name)$ gcloud app browse
Traceback (most recent call last):
File "/google/google-cloud-sdk/lib/gcloud.py", line 20, in <module>
from __future__ import absolute_import
ImportError: No module named __future__
garan@cloudshell:~/crmint (my-project-name)$
I've analyzed a little bit why the deployment of the frontend was super slow.
One of the main culprit appears to be the angular cli compilation: takes a crazy amount of time, and a huge amount of memory, making the Cloud Shell VM unresponsive for 20 minutes sometimes.
First, to avoid too much memory to be consumed, let's limit node to 512MB of ram with: node --max-old-space-size=512 ./node_modules/.bin/ng ...
.
With the --prod flag
$ time node --max-old-space-size=512 ./node_modules/.bin/ng build --prod
...
real 18m40.182s
user 15m35.933s
sys 0m5.094s
Without optimization
$ time node --max-old-space-size=512 ./node_modules/.bin/ng build
...
real 0m32.093s
user 0m23.547s
sys 0m1.458s
Since CRMint is not meant for high-through put access. I would suggest to drop the --prod
flag.
What do you think @Slony?
Cheers
When sending Measurement Protocol hits to Google Analytics, sometimes it's very difficult to troubleshoot discrepancy between the number of rows in a BigQuery table and the number of hits observed in Google Analytics.
It would be great to implement debug mode in BQToMeasurementProtocol
worker and use Validation Service to identify invalid hits.
An option in GAToBQImporter - what does this mean?
The Open Cloud Shell button doesn't always activate when digits are present in the Cloud Project Name. Seen in the deploy guide at Deploy CRMint Quickstart.
Works: crmint-3
Fails: crmint-3n
Fails: crmin-n3
Be able to take data from BigQuery table and make HTTP requests for each app conversion
The pipeline was failing if we only select the name of the parent job, without specifying success
or failure
.
This is a regression, probably need a test to replicate that bug.
I'm on it.
Cheers
==> Applying DB migrations
Applying database migrations: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/coral-mission-244416/backends/lib/flask/__main__.py", line 15, in <module> main(as_module=True) File "/tmp/coral-mission-244416/backends/lib/flask/cli.py", line 513, in main cli.main(args=args, prog_name=name) File "/tmp/coral-mission-244416/backends/lib/flask/cli.py", line 380, in main return AppGroup.main(self, *args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 717, in main rv = self.invoke(ctx) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/flask/cli.py", line 257, in decorator return __ctx.invoke(f, *args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/flask_migrate/cli.py", line 132, in upgrade _upgrade(directory, revision, sql, tag, x_arg) File "/tmp/coral-mission-244416/backends/lib/flask_migrate/__init__.py", line 244, in upgrade command.upgrade(config, revision, sql=sql, tag=tag) File "/tmp/coral-mission-244416/backends/lib/alembic/command.py", line 276, in upgrade script.run_env() File "/tmp/coral-mission-244416/backends/lib/alembic/script/base.py", line 475, in run_env util.load_python_file(self.dir, "env.py") File "/tmp/coral-mission-244416/backends/lib/alembic/util/pyfiles.py", line 90, in load_python_file module = load_module_py(module_id, path) File "/tmp/coral-mission-244416/backends/lib/alembic/util/compat.py", line 216, in load_module_py mod = imp.load_source(module_id, path, fp) File "migrations/env.py", line 104, in <module> run_migrations_online() File "migrations/env.py", line 88, in run_migrations_online connection = engine.connect() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2193, in connect return self._connection_cls(self, **kwargs) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 103, in __init__ else engine.raw_connection() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2293, in raw_connection self.pool.unique_connection, _connection File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect e, dialect, self File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/util/compat.py", line 399, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect return fn() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 303, in unique_connection return _ConnectionFairy._checkout(self) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 760, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 492, in checkout rec = pool._do_get() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/impl.py", line 238, in _do_get return self._create_connection() File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 308, in _create_connection return _ConnectionRecord(self) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 437, in __init__ self.__connect(first_connect_check=True) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/pool/base.py", line 639, in __connect connection = pool._invoke_creator(self) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/tmp/coral-mission-244416/backends/lib/sqlalchemy/engine/default.py", line 451, in connect return self.dbapi.connect(*cargs, **cparams) File "/tmp/coral-mission-244416/backends/lib/MySQLdb/__init__.py", line 81, in Connect return Connection(*args, **kwargs) File "/tmp/coral-mission-244416/backends/lib/MySQLdb/connections.py", line 193, in __init__ super(Connection, self).__init__(*args, **kwargs2) sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2002, "Can't connect to local MySQL server through socket '/tmp/cloudsql/coral-mission-244416:us-central1:crmintapp' (2)") (Background on this error at: http://sqlalche.me/e/e3q8)
Anyone can help me? I don't know waht to do.
Hey team,
As discussed can we please add a notification for the users who deployed old versions of CRMint to notify them that there is a new version to upgrade to?
Thanks,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.