loopbackio / loopback-connector-cassandra Goto Github PK
View Code? Open in Web Editor NEWCassandra connector for the LoopBack framework.
License: Other
Cassandra connector for the LoopBack framework.
License: Other
Im working with loopback 4 and cassandra,
Model, controller, repository and datasource are created successfully, Also loopback 4 server started without any error.
At the time of post or get a method form model im getting error msg "unconfigured Table tablename". Table already created in database manually.
Unhandled error in GET /testmodels: 500 ResponseError: unconfigured table Testmodel at FrameReader.readError (D:\version2\lb4\explore\node_modules\cassandra-driver\lib\readers.js:326:15) at Parser.parseBody (D:\version2\lb4\explore\node_modules\cassandra-driver\lib\streams.js:194:66) at Parser._transform (D:\version2\lb4\explore\node_modules\cassandra-driver\lib\streams.js:137:10) at Parser.Transform._read (_stream_transform.js:190:10) at Parser.Transform._write (_stream_transform.js:178:12) at doWrite (_stream_writable.js:415:12) at writeOrBuffer (_stream_writable.js:399:5) at Parser.Writable.write (_stream_writable.js:299:11) at Protocol.ondata (_stream_readable.js:693:20) at Protocol.emit (events.js:198:13) at addChunk (_stream_readable.js:288:12) at readableAddChunk (_stream_readable.js:269:11) at Protocol.Readable.push (_stream_readable.js:224:10) at Protocol.Transform.push (_stream_transform.js:151:32) at Protocol.readItems (D:\version2\lb4\explore\node_modules\cassandra-driver\lib\streams.js:109:10) at Protocol._transform (D:\version2\lb4\explore\node_modules\cassandra-driver\lib\streams.js:32:10)
make sure jugller/test/include.test.js passes
I use cassandra 3.10
When I create record with loopback API, server show error Error: unconfigured table UserEvent
Check it please !
Hi guys,
I am trying to use integrate loopback-connector-cassandra with no success.
First issue I stumbled upon is how models should be configured.
According to the documentation, they should be configured similar to this:
zipCodes = db.define('zipCodes', {
state: String,
zipCode: Number,
});
But loopback configures them totally differently! The common structure is:
{
"name": "AppUser",
"base": "User",
"idInjection": true,
"options": {
"validateUpsert": true
},
"properties": {
},
"validations": [],
"relations": {},
"acls": [],
"methods": {}
}
How exactly they should be configured?
I also tried to create tables in Cassandra using this code ( I tried both methods autoupdate & automigrate ):
var server = require('./server');
var ds = server.dataSources.cassandra;
var models = []; // will be aplied to all models
ds.autoupdate(models, function(er) {
if (er) throw er;
console.log('Loopback tables [' - models - '] created in ', ds.adapter.name);
ds.disconnect();
});
My datasource name is "cassandra". However, code just prints:
NaN 'cassandra'
Do you have any basic example of fully configured Loopback project with cassandra datastore? I believe it will help a lot.
Thank you in advance.
Best,
CASS_SORTABLE = db.define('CASS_SORTABLE', {
patBool: {type: Boolean, id: 2},
str: String,
patStr: {type: String, id: true},
num: Number,
patNum: {type: Number, id: 1},
yearMonth: {type: String, index: true},
}, {
cassandra: {
clusteringKeys: ['str', 'num DESC'],
},
});
var ROW
CASS_SORTABLE.create({
str: cassTestString + '10',
num: 10,
patBool: true,
patNum: 100,
patStr: cassTestString + '100',
}, function(err, m) {
ROW = m;
});
ROW_1.updateAttributes({
str: ROW_1.str,
num: ROW_1.num,
patBool: ROW_1.patBool,
patNum: ROW_1.patNum,
patStr: ROW_1.patStr,
yearMonth: '2018-06'}, function(err, rows) {
});
The update failed w/ error
Uncaught AssertionError: expected ResponseError {
name: 'ResponseError',
info: 'Represents an error message from the server',
message: 'PRIMARY KEY part str found in SET part',
code: 8704,
query: 'UPDATE "CASS_SORTABLE" SET "str"=?,"num"=?,"yearMonth"=? WHERE "patStr"=?'
} to not exist
The operation is expected to succeed.
ROW.updateAttributes called these codes,
SQLConnector.prototype.updateAttributes = function(model, id, data, options, cb) , which is called by
DataAccessObject.prototype.updateAttributes.
In DataAccessObject.prototype.updateAttributes, it calls
invokeConnectorMethod(connector, 'updateAttributes', Model,
[getIdValue(Model, inst), Model._forDB(context.data)],
options, updateAttributesCallback);
getIdValue(Model, inst)
doesn't consider the composite primary key.
The ideal fix is in juggler to get an object from getIdValue(Model, inst)
instead of a string value. But before that, a workaround can be in SQLConnector.prototype.updateAttributes
, to put primary keys inwhere
and remove them from data.
Cassandra loopback connector uses incorrect syntax for schema migration.
Issue 1 - adding a column
Error: no viable alternative at input 'NOT' (ALTER TABLE table1 ADD newCol [TEXT] NOT...)
Correct syntax: ALTER TABLE table1 ADD col1 TEXT;
Issue 2 - dropping multiple columns
Error: mismatched input ',' expecting EOF (ALTER TABLE table1 DROP col1[,]...)
Correct syntax: ALTER TABLE table1 DROP (col1, col2);
Cassandra table is succussfully migrated with the correct syntax below:
ALTER TABLE table1 ADD col1 TEXT;
ALTER TABLE table1 DROP (col1, col2);
https://github.com/strongloop/loopback-connector-cassandra/blob/master/lib/migration.js#L180
https://github.com/strongloop/loopback-connector-cassandra/blob/master/lib/migration.js#L270
https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlAlterTable.html
I wanna store an array of strings.
I tried to use this in Loopback 4, but all I get in Cassandra is a simple text field:
@property.array(String, {
default: [],
jsonSchema: {
uniqueItems: true,
type: 'string',
format: 'uri-reference',
errorMessage: 'An array of file URLs is expected',
},
})
attachments?: string[];
How to create an array?
TBD - will be filled by the team.
Respected Sir,
I am using strongloop with data source CASSANDRA. The test connection is success. I can try to add one new row in the DB but getting error code 8192 as a response from server. See attached screenshot for your reference.
Can you please help?. Thanks in advance for your help.
Thanks,
Bhavin Bhalodia.
Travis CI is failing due to Connection fails: NoHostAvailableError: All host(s) tried for query failed. First host tried, 127.0.0.1:9042: Error: connect ECONNREFUSED 127.0.0.1:9042. See innerErrors. It will be retried for the next request.
Jenkins has more random failures but one the consistent ones are:
Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
errors (sometimes happens on other Node versions, but most often Node 12)1) cassandra custom tests discoverPrimaryKeys:
Uncaught AssertionError: expected Array [ 'num', 'patBool', 'patNum', 'patStr', 'str' ] to equal Array [ 'patNum', 'patBool', 'patStr', 'str', 'num' ] (at '0', A has 'num' and B has 'patNum')
+ expected - actual
[
- "num"
+ "patNum"
"patBool"
- "patNum"
"patStr"
"str"
+ "num"
]
discoverPrimaryKeys
failure (could sort the array to fix)#71 is a quick fix. It is ok to merge first. I'll use this ticket to add ut later.
Hi there,
How do I execute CQL queries directly with loopback-connector-cassandra?
Thanks,
I am trying update user password and I am using updateAttribute function.
I am seeing an interesting issue. It does update the field but it also sets the error. I am see this error
It tries to delete accesstoken but in accesstoken key is id. If I change the schema to change primary_key to user_id then other functions will start throwing errors;
message: 'Some partition key parts are missing: id',
info: 'Represents an error message from the server',
code: 8704,
coordinator: 'local:9042',
query: 'DELETE FROM "AccessToken" WHERE "userId" IN (?)' }
user.updateAttribute('password', password, function(error, response) {
if (error) {
return callback(error);
}
console.log(response);
return callback(null, {userId: response.id});
});
migration tables that have multiple drop columns do not work.
Steps to reproduce
`
if (actualFieldNotPresentInModel(actualField, model)) {
sql.push('DROP ' + self.escapeName(actualField.column));
}
`
it generates sql like
ALTER TABLE "abc" DROP "column3", DROP "column4", DROP "column5"
and that will fail with
line 1:42 mismatched input ',' expecting EOF (ALTER TABLE "abc" DROP "column3"[,]...),
while from below cql, it should be DROP column | ( column_list )
https://docs.datastax.com/en/dse/6.0/cql/cql/cql_reference/cql_commands/cqlAlterTable.html
DROP ("column3", "column4", "column5")
https://docs.datastax.com/en/dse/6.0/cql/cql/cql_reference/cql_commands/cqlAlterTable.html
https://github.com/strongloop/loopback-connector-cassandra/blob/eb176bb71475276031e298d005aa05b4810f5a9a/lib/migration.js#L274
We are wondering about the status of the project because the datasource is not available even after installing loopback-connector-cassandra with npm install <local-directory> (so it's not related to issue #2).
Also the link for the official documentation at http://docs.strongloop.com/display/LB/Cassandra+connector (from README.md) is dead and searching for "cassandra" in the strongloop documentation yields nothing.
Could you clarify the status of the project please ?
design question: do we expose the TOKEN to the C* connector API, or can we completely hide in under the hood and support pagination?
At the moment, our connectors are installing juggler 3.x to run the test suite. When we make a change to juggler@3, cis-jenkins detects downstream dependency and triggers CI build of connectors with the modified juggler version. When we make a change to juggler's master, no downstream builds are triggered.
In the past, we were maintaining multiple branches to test connectors against different juggler versions. E.g. loopback-2.x containing code from "master" but installing juggler 2.x for testing. This is rather impractical, because we don't have bandwidth to update those other branches with changes made to master.
Let's rework the connector test setup to execute tests from both juggler versions 3.x and 4.x, as we are already doing in other connector repositories.
The reference pull request to draw inspiration from: loopbackio/loopback-connector-mongodb#519. See also loopbackio/loopback-connector-postgresql#376.
v1.0.0 assumes all connector calles aredone in one keyspace.
Note: USE query in lib/migration.js/createTable
I am using Cassandra datasource in loopback 3, and trying to store values in list datatype column of a table(keyspace) of Cassandra datasource. But while creating an entry from loopback explorer, we are getting TypeError: Not a valid list value, expected Array obtained '["Morning","Afternoon","Evening"]'
Here is my loopback model:
{
"name": "Time",
"base": "PersistedModel",
"idInjection": true,
"options": {
"validateUpsert": true
},
"properties": {
"options": {
"type": "array",
"format": "list"
}
},
"validations": [],
"acls": [],
"methods": {}
}
Tried debugging loopback-cassandra-datasource and saw options converted to string in params.
Data passed from loopback explorer:
{
"options": ["Morning","Afternoon","Evening"]
}
Data logged from beforeRemote hook of the model:
{ options:
[ 'Morning',
'Afternoon',
'Evening' ]
}
package.json
"dependencies": {
"loopback": "3.26.0",
"loopback-boot": "2.28.0",
"loopback-component-explorer": "6.4.0"
"loopback-connector-cassandra": "1.7.1"
}
The typeof options is object. I was debugging loopback cassandra connector and found typeof options as string in encoder.js of cassandra-driver node module. We have migrated from MongoDB to Cassandra and didn't face any issues for MongoDB driver. To what type or which format do I need to set for options property? Please let me know if any additional information required. Thank you in advance.
Create a model which uses Date type. Try to save with milliseconds. Check inside Cassandra how the value was saved. It loses milliseconds.
Save manually a date in Cassandra using milliseconds. Retrieve it from loopback 4, you will see milliseconds are set to 0.
The problem is inside cassandra.js
When you save the date it uses the function:
function dateToMysql(date) {
return date.getUTCFullYear() + '-' +
fillZeros(date.getUTCMonth() + 1) + '-' +
fillZeros(date.getUTCDate()) + ' ' +
fillZeros(date.getUTCHours()) + ':' +
fillZeros(date.getUTCMinutes()) + ':' +
fillZeros(date.getUTCSeconds());
function fillZeros(v) {
return parseInt(v) < 10 ? '0' + v : v;
}
}
As you can see millisecons are not being taking into account.
This could be a possible solution:
function dateToMysql(date:Date) {
function fillZeros(v, n = 2) {
return ('0'.repeat(n-1) + v).slice(-n);
}
return date.getUTCFullYear() + '-' +
fillZeros(date.getUTCMonth() + 1) + '-' +
fillZeros(date.getUTCDate()) + ' ' +
fillZeros(date.getUTCHours()) + ':' +
fillZeros(date.getUTCMinutes()) + ':' +
fillZeros(date.getUTCSeconds()) + '.' +
fillZeros(date.getUTCMilliseconds(),3);
}
When you retrieve a Date it uses function:
Cassandra.prototype.fromColumnValue
case 'Date':
// Cassandra allows, unless NO_ZERO_DATE is set, dummy date/time entries
// new Date() will return Invalid Date for those, so we need to handle
// those separate.
if (val == '0000-00-00 00:00:00') {
val = null;
} else {
val = new Date(val.toString().replace(/GMT.*$/, 'GMT'));
}
val.toString() get rid of millisecons, val is actually a Date, you can return it without any conversion. Return val without any conversion solves the issue.
To connect to certain cloud providers they require "localDataCenter" this is missing in both cassandra.js and requires an update to cassandra-connector in package.json.
I added to following and it now allows me to connect.
"cassandra-driver": "3.5.0"" needs to be updated to ^4.0.0
reference "https://docs.datastax.com/en/developer/nodejs-driver/4.6/upgrade-guide/"
and 'function generateOptions(settings)' needs the following line added.
"clientOptions.localDataCenter = settings.localDataCenter;"
Whitch is the status of the project?. I tried install the package from npm and anything was download, and not exist any commit from many time.
I have and cassandra prjoect and would like to use loopback with this connector ...
I am unable to save basic user for authentication loopback 3.
form.js:128:32)',
message: 'IN predicates on non-primary-key columns (property) is not yet supported',
info: 'Represents an error message from the server',
code: 8704,
coordinator: 'localhost:9042',
query: 'SELECT "model","property","accessType","permission","principalType","principalId","id" FROM "ACL" WHERE "model"=? AND "property" IN (?,?) AND "accessType" IN (?,?) ALLOW FILTERING' }
Unhandled error for request POST /RMUser: Error: IN predicates on non-primary-key columns (property) is not yet supported.
Model config below.
"User": {
"dataSource": "db",
"public": false
},
"AccessToken": {
"dataSource": "db",
"public": false
},
"ACL": {
"dataSource": "cassandra",
"public": false
},
"RoleMapping": {
"dataSource": "cassandra",
"public": false
},
"Role": {
"dataSource": "cassandra",
"public": false
},
"RMUser": {
"dataSource": "cassandra",
"public": true,
"options": {
"emailVerificationRequired": true
}
}
I used automigrate to create tables. It created table like this. Which is incorrect for the query below
CREATE TABLE nfeeds."ACL" (
id uuid PRIMARY KEY,
"accessType" text,
model text,
permission text,
"principalId" text,
"principalType" text,
property text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
npm i loopback-connector-cassandra
Results in an empty folder in node_modules/
. Can you publish up a new version?
create user defined type and support GeoPoint type as many of other DB connectors do.
Imported tests from loopback-datasource-juggler failed. They passed with [email protected]. But failed w/ [email protected].
Failed tests are in manipulation.tests.js it.only('should have updated password hashed with replaceAttributes'
. The error occurred in dao.js, see following code snippet,
function replaceCallback(err, data) {
if (err) return cb(err);
context.data = data; // This is new code. Data is undefined as update in cassandra returns nothing.
if (typeof connector.generateContextData === 'function') {
context = connector.generateContextData(context, data);
}
var ctx = {
Model: Model,
hookState: hookState,
data: context.data,
isNewInstance: false,
options: options,
};
Model.notifyObserversOf('loaded', ctx, function(err) {
if (err) return cb(err);
if (ctx.data[pkName] !== id && !Model._warned.cannotOverwritePKInLoadedHook) { // Here error is thrown!!!
Model._warned.cannotOverwritePKInLoadedHook = true;
In lib/migration.js/createTable, it tries to create keyspace if not exists with predetermined parameters such as:
' WITH REPLICATION = { \'class\' : \'SimpleStrategy\',' +
' \'replication_factor\' : \'3\' }';
make it client definable
Type format should be like:
Tuple<String, String, Number> ==> TUPLETEXT, TEXT, INT>
"cassandraConn": {
"host": "192.168.4.21",
"port": 9042,
"database": "emsv2_testing",
"password": "",
"name": "cassandraConn",
"user": "",
"connectTimeout": 30000,
"readTimeout": 30000,
"connector": "cassandra"
}
automigration create database name 'undefined', create tables within 'undefined' database. when posting data showing error,
{
"error": {
"statusCode": 500,
"name": "NoHostAvailableError",
"message": "Connecting after shutdown is not supported",
"info": "Represents an error when a query cannot be performed because no host is available or could be reached by the driver.",
"innerErrors": null,
"stack": "NoHostAvailableError: Connecting after shutdown is not supported\n at Client._connectCb (/home/rd/enms/node_modules/cassandra-driver/lib/client.js:397:21)\n at Client.promiseWrapper (/home/rd/enms/node_modules/cassandra-driver/lib/utils.js:410:13)\n at Client.connect (/home/rd/enms/node_modules/cassandra-driver/lib/client.js:384:31)\n at Array.utils.series.next (/home/rd/enms/node_modules/cassandra-driver/lib/client.js:854:18)\n at next (/home/rd/enms/node_modules/cassandra-driver/lib/utils.js:714:17)\n at Object.series (/home/rd/enms/node_modules/cassandra-driver/lib/utils.js:698:3)\n at Client._innerExecute (/home/rd/enms/node_modules/cassandra-driver/lib/client.js:853:9)\n at Client.handler (/home/rd/enms/node_modules/cassandra-driver/lib/client.js:487:10)\n at Client.promiseWrapper (/home/rd/enms/node_modules/cassandra-driver/lib/utils.js:410:13)\n at Client.execute (/home/rd/enms/node_modules/cassandra-driver/lib/client.js:485:31)\n at Cassandra.executeSQL (/home/rd/enms/node_modules/loopback-connector-cassandra/lib/cassandra.js:238:15)\n at /home/rd/enms/node_modules/loopback-connector/lib/sql.js:651:10\n at /home/rd/enms/node_modules/loopback-datasource-juggler/lib/observer.js:259:9\n at doNotify (/home/rd/enms/node_modules/loopback-datasource-juggler/lib/observer.js:155:49)\n at Cassandra.ObserverMixin._notifyBaseObservers (/home/rd/enms/node_modules/loopback-datasource-juggler/lib/observer.js:178:5)\n at Cassandra.ObserverMixin.notifyObserversOf (/home/rd/enms/node_modules/loopback-datasource-juggler/lib/observer.js:153:8)"
}
}
my cassandra database is alive..
How can we specify multiple contactPoints? Are multiple contactPoints possible?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.