Git versioning integration about ground HOT 15 CLOSED

Do we also want ground to manage an API where a user/application could POST and it would write a message to one of grounds kakfa topics? For other types of webhooks.

from ground.

I also need to add in a python script for setting up the github webhook. For the script the user would provide:

• repo URL
• credentials
• ground github webhook API URL

from ground.

This is how I am POSTing each version/commit

{
    "tags": {
        "branch": {
            "key": "branch",
            "value": "refs/branches/master",
            "type": "string"
        },
        "commit": {
            "key": "commit",
            "value": "8464793044eff62c794f88c06ea286e427efc8b9",
            "type": "string"
        },
    },
    "structureVersionId": null,
    "reference": null,
    "nodeId": "Nodes.ground",
    "parameters": {}
}

Do you think this is the best way to do it?
Should nodeId be repo id (153213) instead of repo name? repo names are not unique and could cause problems

from ground.

I'm facing a problem with the nodes/versions/ API, I'm listing the parent versions in the URL and when I get to 30~ versions the API starts taking a long time and then just not working. I'm using neo4j as my backend. If I don't list and parents in the URL then it works fine and doesn't slow down. I'll look into the API and see if I can figure out what might be causing it.

from ground.

It looks like it is a problem when getting the DAG. If the path along the versions is too long it causes problems
https://github.com/ground-context/ground/blob/master/ground-core/src/main/java/edu/berkeley/ground/api/versions/neo4j/Neo4jVersionHistoryDAGFactory.java#L45

from ground.

Oh, I see. That's using a Neo4j feature for recursive queries basically. We might be able to do things a little faster if we create an index on VersionSuccessor in Neo4j. Do you want to try that?

from ground.

I've used gremlin more than cypher so I'm still trying to figure out how to create an index to help the query go faster. It looks like its easy to create an index on a property type but that doesn't really help us.

from ground.

Hm, what makes you say that creating an index on the VersionSuccessor property type will not help?

Do you think you could set up a Neo4j server in the cloud somewhere with this data, so we can both play with the data to figure out how to make this go faster?

from ground.

I guess I thought VersionSuccessor was a relationship not a property type.
I ran CREATE INDEX ON :NodeVersion(VersionSuccessor) yesterday and it didn't improve performance.

from ground.

Hm. I think that creates an index on the VersionSuccesor property of the vertices labeled NodeVersion. It's not clear to me why a path of length 50 has such a big performance hit. Could you try doing something like [:VersionSuccessor*..100] (only find paths of length up to 100).

I'm going to look into this a little bit more, but if Neo4j can't handle these queries, then we might have to drop it and go back to Postgres...

from ground.

Ok I think it might be working. I'll test it more

from ground.

Still not working, but this query might work for getting similar information. It's faster but it doesn't include the starting node. I'll see I can get the query to include it

MATCH (a:NodeVersion { node_id : 'Nodes.ground' }) -[r:VersionSuccessor] - (b:NodeVersion {node_id : 'Nodes.ground'})
RETURN DISTINCT r

Nodes.ground is the initial node

from ground.

MATCH (a)-[r:VersionSuccessor]-(b:NodeVersion {node_id : 'Nodes.ground'})
WHERE a.id='Nodes.ground'
OR a.node_id='Nodes.ground'
RETURN DISTINCT r

This will include the original node.

from ground.

Interesting. This works recursively?

from ground.

I don't think the new one does it recursively. It gets all relationships that come from or go to a node_id "Nodes.ground", with label "VersionSuccessor". Then it combines them all to get a path.

When I ran the old query:

MATCH (a:Node {id: 'Nodes.engine' })-[e:VersionSuccessor*..20]->(b:NodeVersion)
RETURN e

It returned 9736 rows, so I think it is finding every possible path.

from ground.