formallanguageconstrainedpathquerying / cfpq_data Goto Github PK

With a total check of data correctness.

Add sets of vertices indexes for multiple-source querying for chunk sizes of 1, 10, 20, 50 and 100 to RDF, MemoryAliases and Java graphs.

The chunks are available on Google Drive.

Automate reference values updating and publishing. Fix its description: https://github.com/JetBrains-Research/CFPQ_Data#reference-values

Software Heritage project collects information about open-source software. Can CFPQ be useful to analyze collected data?

It could be very useful when you add information about the number of vertices and the number of edges for each graph.

Petteri Sevon, Lauri Eronen. Subgraph Queries by Context-free Grammars

Create logo and use it on website.

It would be cool to have an opportunity to get the graph with only edges or vertices of interest. For example, Enzyme only with broaderTransitive relationship edges.

We should establish releases of CFPQ_data as freeze points for graphs and queries.

1. Tag on GitHub.
2. I'm not sure, how we should freeze RDFs, which currently store in Google Drive. Maybe, we should move it to git-lfs. The reason, why RDFs currently store not in git-lfs is that git-lfs has a limit on upload-download operations and it may be a problem for users. Maybe we should just freeze RDFs in a specific folder in Google Drive and fix references to download. But in this case, each release will be in a separate branch. Or master should be for releases only. The dev branch for development.
3. Release automation: tag creation, branch creation (or not), documentation regeneration, site update, artifacts freezing.

Make it possible to generate synthetic graphs to RDF or triples without conversion.

Add documentation for developer.
Include:

• Pre-commit
• Test pipeline
• Docs deploy
• Package deploy
• Guideline

We should establish releases of CFPQ_data as freeze points for graphs and queries.

1. Tag on GitHub.
2. I'm not sure, how we should freeze RDFs, which currently store in Google Drive. Maybe, we should move it to git-lfs. The reason, why RDFs currently store not in git-lfs is that git-lfs has a limit on upload-download operations and it may be a problem for users. Maybe we should just freeze RDFs in a specific folder in Google Drive and fix references to download. But in this case, each release will be in a separate branch. Or master should be for releases only. The dev branch for development.
3. Release automation: tag creation, branch creation (or not), documentation regeneration, site update, artifacts freezing.

• Fix #27
• Merge #24
• Fix #32
• Fix #20
• Fix #30
• Fix #33
• Fix #2
• Fix #26

• It is very difficult to look at the number of vertices, edges, etc. when it exceeds hundreds of thousands
• It would be very helpful to add a three-digit separator
• It is better to indicate the size of the graph in at least kilobytes

Link to download Geospecies doesn't work in README.

Proposal

The data architecture at the moment looks like this:

• 1.0.0 -- Release version of cfpq_data
  • MemoryAliases
    • graphs -- Separate rdf files, each packed in its own .tar.gz archive
  • RDF
    • graphs -- Separate rdf files, each packed in its own .tar.gz archive

I suggest to do the following data architecture:

• Release version of cfpq_data
  • csv file for each graph

With CSV headers [from,label,to]

Pros

• The proposed architecture simplifies the manipulation of graphs in S3
• Loading will be faster
• CSV is simpler and more convenient to represent labeled graphs than RDF

Cons

• It may take more disk space than .tar.gz archives
• CSV format is not as rich as RDF
• No other data format provided

To simplify the search for the desired function and improve the auto-generation of documentation, I propose to linearize the reference page https://jetbrains-research.github.io/CFPQ_Data/reference/index.html

Maybe add test release for each PR on master on Test PYPI?

Default prefix for reversed edges should be "_r".

cc #67

Add PR template for grammars similar with PR template for adding graphs.

Do it using Sphinx or switching to other engine.

Is it possible to replace GTgraph (which is currently used to generate synthetic graphs) with generators from networkx.

The following graphs are taken from DaCapo benchmark suite.

All required data is uploaded to Google Drive.

• Graph name: graphs/graph_name
• Graph triplets: graphs/graph_name/data.csv
• Edge Statistics: graphs/graph_name/edge_stats.csv
• Node and Edge counts: graphs/graph_name/graph_stats.csv (columns 1 and 2 respectively)

Canonical grammar:
Alias → PointsTo PointsTo_r
PointsTo → (assign | load_f Alias store_f )* alloc

• Understanding Data Science Lifecycle Provenance via Graph Segmentation and Summarization
• ProvBench site
• ProvBench repo with data
• How to get provenance info from Bio2RDF
• ProvGen: generating synthetic PROV graphswith predictable structure

• Add data (both graphs and grammars) from Jyothi Vedurada. Batch Alias Analysis
• Add data and query form "Demand-driven Alias Analysis for C" (Xin Zheng and Radu Rugina). We use data from this article in our work. Ilya Kirillov can help to extract graphs from C code. A number of serialized graphs.

Example: memory aliases g2_ma

• Add repository structure description
• Add data set description
  • Graphs
  • Queries
• Add reference values for algorithms correctness checking
• Add a list of works which use this data set

This will be great if it is possible to get a representation of the graphs in text form and the cnf grammars after initialization. This will make it easier to run cfpq algorithms and there will be no need to convert formats. All the necessary scripts are already written, so it will be easy to implement.

Hello,

I want to get a graph from a dataset in a txt file. To do this, I wrote a script like this:

import cfpq_data
g = cfpq_data.graph_from_dataset("generations")
path = cfpq_data.graph_to_txt(g, "test.txt")

But in the test.txt file I get something like this:

'N1a0fdccf1e5747358d788032136b2d3a' 'http://www.w3.org/1999/02/22-rdf-syntax-ns#first' 'N5b1227cd9d554e6b8376780df8c0ac4b'
'N1a0fdccf1e5747358d788032136b2d3a' 'http://www.w3.org/1999/02/22-rdf-syntax-ns#rest' 'http://www.w3.org/1999/02/22-rdf-syntax-ns#nil'
'N5b1227cd9d554e6b8376780df8c0ac4b' 'http://www.w3.org/2002/07/owl#someValuesFrom' 'Nf6e96ad96dee4cc380a425a39571412b'
'N5b1227cd9d554e6b8376780df8c0ac4b' 'http://www.w3.org/2002/07/owl#onProperty' 'http://www.owl-ontologies.com/generations.owl#hasChild'
...

Please tell me where I am wrong. Thank you

Current list of synthetic graphs should be analyzed and updated.

• Add grammar support to the CFPQ_Data package (downloading, other utilities)
• Add grammar templates to website
• Add grammar instances for existing graphs to the storage
• Update other website parts wrt the new grammars