jkminder / data2neo Goto Github PK

Runtime can be optimised. Firstly we can circumvent code executions for resources that do not require to be passed through the factories. Secondly we can leverage batch processing by adding a buffer. Maybe the work of committing the graph can be offloaded to a separate process.

Note: It needs to be thoroughly checked what the impact is on dependency issues. Committing to the graph will be asynchronous. Any query in wrappers to the graph, must be aware of this.

The library should natively support the adding of error handlers.

Hi all,

I have had a really weird error when providing the yaml file to the converter, and it truly took me some time to find out the reason.

The error looked as follows:

ValueError: The arguments list has not the same number of opening and closing brackets: "MPs.city")_birt

And I knew it was coming from one of the following lines in the yaml file:

    IF_NOT_EMPTY(NODE("City", "Location"), "BirthPlace_City") city_birth:
        + name = MPs.BirthPlace_City         
    IF_NOT_EMPTY(IF_REL_NOT_EXISTS(RELATION(person, "BORN_IN", city_birth)), "BirthPlace_City"):

but I could not understand the reason. I was suspecting the wrappers, and their order, might be causing the issue, but I could not find a pattern. Then, I finally realize it was all a conflict with

    IF_NOT_EMPTY(NODE("City", "Location"), "city") city:
        + name = EXTR_CITY(MPs.city)

It seems it all boils down to some conflict in naming between city and city_birth, because now the following is working:

    IF_NOT_EMPTY(NODE("City", "Location"), "BirthPlace_City") birth_city:
        + name = MPs.BirthPlace_City         
    IF_NOT_EMPTY(IF_REL_NOT_EXISTS(RELATION(person, "BORN_IN", birth_city)), "BirthPlace_City"):

Therefore, I believe this is a bug that needs to be solved, as it should not be happening, and it took way a lot of time to find out.

Thank you so much!

Example:

ENTITY("Session"):
    NODE("Year") year:
        + year = YEAR_FROM_DATE(Session.StartDate)
        
    REQUIRED(RELATION(sessionnode, "DURING", year)):
        - name = Session.SessionName
        - type = Session.TypeName
        - date_start = DATE(Session.StartDate)
        - date_end = DATE(Session.EndDate)

    IF_END_DIFFERENT(NODE("Year")) year_end:
        + year = YEAR_FROM_DATE(Session.StartDate)
        
    IF_END_DIFFERENT(REQUIRED(RELATION(sessionnode, "DURING", year_end))):
        + name = Session.SessionName
        - type = Session.TypeName
        - date_start = DATE(Session.StartDate)
        - date_end = DATE(Session.EndDate)

The config parser/compiler has problems here due to how identifiers (year and year_end) are parsed. The identifier year_end is sometimes parsed first as year (not always though, not yet sure why). Which gives weird parser errors (it identifies the substring year first-> "year"_end). Requires rewrite of compiler in config_parser.py.

Temporary Solution:
Make sure your identifiers non-overlapping. E.g. year and yend instead of year and year_end.

The progress bar should be updated after the work is done. Otherwise it can happen that it gets stuck before reaching 100% which confuses users.

The ResourceIterator should work based on the native python __next__ and __iter__ functions instead of the next and reset_to_first functions.

Replace py2neo dependency and move to native neo4j driver.

https://github.com/neo4j-contrib/py2neo

For the config below only the last (BBI) is created.

ENTITY("LegislativePeriod"):
    NODE("Legislative Period") lp:
        + uid = LegislativePeriod.LegislativePeriodNumber
        - date_start = DATE(LegislativePeriod.StartDate)
        - date_end = DATE(LegislativePeriod.EndDate)
    # add the one time source creations
    NODE("Source"):
        + name = "Online DB"
        - abbrev = "ODB"
        - description = "Public OData database of swiss national parliament."
        - url = "https://ws.parlament.ch/odata.svc/"
    
    NODE("Source"):
        + name = "Amtliche Sammlung"
        - abbrev = "AS"

    NODE("Source"):
        + name = "Bundesblatt"
        - abbrev = "BBI"

If one ads identifiers all of them are created

ENTITY("LegislativePeriod"):
    NODE("Legislative Period") lp:
        + uid = LegislativePeriod.LegislativePeriodNumber
        - date_start = DATE(LegislativePeriod.StartDate)
        - date_end = DATE(LegislativePeriod.EndDate)
    # add the one time source creations
    NODE("Source") odb:
        + name = "Online DB"
        - abbrev = "ODB"
        - description = "Public OData database of swiss national parliament."
        - url = "https://ws.parlament.ch/odata.svc/"
    
    NODE("Source") as:
        + name = "Amtliche Sammlung"
        - abbrev = "AS"

    NODE("Source") bbi:
        + name = "Bundesblatt"
        - abbrev = "BBI"

The pipeline should better communicate which types are allowed/which are converted. Maybe add warnings if a type is converted to string.

E.g. np.int64 is not working with some neo4j versions. Because its of type numbers.Number it is not converted to string and passed along to neo4j, which causes the crash.
TypeError: Neo4j does not support JSON parameters of type int64

When creating a Converter instance, I only found a way to supply the conversion-YAML as a file. Is it possible to supply the YAML from a str variable as well?

For example:

my_yaml = """... Multiline YAML statements here ..."""

converter = Converter(from_variable=my_yaml, my_iterator, my_graph)

The from_variable arg is just an example of a possible implementation. Another idea could, e.g., be a dedicated ConverterFromText class, etc.

Supplying the YAML from a Python variable would be useful, for example, when generating the YAML dynamically.

If no graph entity is defined, the parser crashes.

ENTITY("entity"):

For consistency reasons, the "RELATION" keyword should be renamed to "RELATIONSHIP" (in neo4j its also called RELATIONSHIP).

The function rel2graph.neo4j.match_nodes fails when matching multiple attributes.

It seems to happen that with very large databases and many workers, neo4j throws variations of the following errors. There is also the suspicion that this looses some relations. Mainly happens if many workers throw many new relations (or matches) at the server. Could not yet be reproduced in toy setting.

Transaction failed and will be retried in 1.0245680547582983s (ForsetiClient[transactionId=333733, clientId=2845] can't acquire UpdateLock{owners=ForsetiCli ent[transactionId=333390, clientId=2844], ForsetiClient[transactionId=333733, clientId=2845], refCount=2} on NODE_RELATIONSHIP_GROUP_DELETE(106064) because holders of that lock are waiting for ForsetiClient[transactionId=333733, clientId=2845]

Transaction failed and will be retried in 0.8898980770768736s (ForsetiClient[transactionId=1199721, clientId=42911] can't acquire ExclusiveLock{owner=ForsetiClient[transactionId=1199388, clientId=42920]} on NODE_RELATIONSHIP_GROUP_DELETE(192231) because holders of that lock are waiting for ForsetiClient[transactionId=1199721, clientId=42911].