I installed the CI rpms of the framework as built from the master branch on July 2. I installed the new framework rpm and left the decisionengine_modules RPM as it was.. but in this particular case this is a pure Framework issue.
The main change of the framework is the configuration change to the .jsonnet format.
We initially got this dump:
2020-07-08 20:16:11,925 - root - TaskManager - 13588 - SourceProxy - ERROR - Exception running source SourceProxy : operator does not exist: text = text[]
LINE 4: ...kmanager_id=156 AND foo.generation_id=1424 AND key=ARRAY['aw...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/decisionengine/framework/taskmanager/TaskManager.py", line 298, in run_source
data = src.worker.acquire()
File "/usr/lib/python3.6/site-packages/decisionengine/framework/modules/SourceProxy.py", line 152, in acquire
self._get_data(data_block, k_in))
File "/usr/lib/python3.6/site-packages/decisionengine/framework/modules/SourceProxy.py", line 94, in _get_data
data = data_block.get(key)
File "/usr/lib/python3.6/site-packages/decisionengine/framework/dataspace/datablock.py", line 296, in get
return self.getitem(key, default=default)
File "/usr/lib/python3.6/site-packages/decisionengine/framework/dataspace/datablock.py", line 363, in getitem
self.generation_id, key)
File "/usr/lib/python3.6/site-packages/decisionengine/framework/dataspace/dataspace.py", line 127, in get_dataproduct
return self.datasource.get_dataproduct(taskmanager_id, generation_id, key)
File "/usr/lib/python3.6/site-packages/decisionengine/framework/dataspace/datasources/postgresql.py", line 241, in get_dataproduct
return self._select_dictresult(q, (taskmanager_id, generation_id, key))[0]
File "/usr/lib/python3.6/site-packages/decisionengine/framework/dataspace/datasources/postgresql.py", line 445, in _select_dictresult
sql_query, values, cursor_factory=psycopg2.extras.RealDictCursor)
File "/usr/lib/python3.6/site-packages/decisionengine/framework/dataspace/datasources/postgresql.py", line 337, in _select
colnames, res = self.__query(query_string, values, cursor_factory)
File "/usr/lib/python3.6/site-packages/decisionengine/framework/dataspace/datasources/postgresql.py", line 349, in __query
cursor.execute(query_string, values)
File "/usr/local/lib/python3.6/site-packages/DBUtils/SteadyDB.py", line 605, in tough_method
result = method(*args, **kwargs) # try to execute
File "/usr/local/lib64/python3.6/site-packages/psycopg2/extras.py", line 248, in execute
return super(RealDictCursor, self).execute(query, vars)
psycopg2.errors.UndefinedFunction: operator does not exist: text = text[]
LINE 4: ...kmanager_id=156 AND foo.generation_id=1424 AND key=ARRAY['aw...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
2020-07-08 20:16:12,828 - root - TaskManager - 13588 - MainThread - ERROR - Error occured during initial run of sources. Task Manager AWS_Calculations_with_source_proxy.jsonnet exits
There is nothing involved from the modules here--just the base framework and the SourceProxy.py which is also part of the source framework.
This is the configuration that calls it
"AWSJobLimits": {
"module": "decisionengine.framework.modules.SourceProxy",
"name": "SourceProxy",
"parameters": {
"channel_name": "channel_aws_config_data",
"Dataproducts": [
[
"aws_instance_limits",
"Job_Limits"
]
],
"retries": 3,
"retry_timeout": 20
},
"schedule": 360
}
In the process of converting the config files Vito noticed that this source originally
had a tuple in the config.
Originally the old style config looked like this:
"AWSJobLimits" : {
"module" : "decisionengine.framework.modules.SourceProxy",
"name" : "SourceProxy",
"parameters": {"channel_name": "channel_aws_config_data",
"Dataproducts":[("aws_instance_limits", "Job_Limits")],
"retries": 3,
"retry_timeout": 20,
},
"schedule": 360,
}
The purpose of a tuple in the config was to indicate that the data block channel_aws_config_data / aws_instance_limits should be brought into this channel with the name Job_Limits.
This is the only place in the whole set of modules where this is done.
Vito modified the code to take a tuple or a list but that had no effect, we are still getting the same error as before.
So the error is not caused by the config changes or code changes that Vito is making.