calpano / graphinout Goto Github PK

Add test file which tries to use all DOT features, (or use multiple files), then verify via e.g. mocks that data from file ended up correctly in Gio stream (calls to GioWriter)

Look at existing Graph6Reader

• Try to find a streaming way without sending data first to a JGraphT graph
• Sparse6 format
• Digraph6 format

add validation to the GioNode/GioEdge/etc. objects themselves

maybe a ValidatingGioWriter is not needed. -> is required to check interplay of objects

Every stream of valid/creatable GioObjects should be handled by GioWriter,
delegates to GraphMLWriter -- and the ValidatingGraphmlWriter catches all wrongly nested/ordered elements.

similar to ValidatingXmlWriter,

ensure that the structure of GrapML as defined in the spec/DTD is respected.
Check for the order of elements according to DTD.

DTD: http://graphml.graphdrawing.org/specification/dtd.html

• let them both extends AutoCloseable and do .. whatever needs to be done?
• in our impl: implement close correclty
• - [ ] in our code: use try-with-resources to close them

• ValidatingGioWriter implements GioWriter
• each method does checks, as far as possible regarding the gio stream, and delegates to base.
• ensure GioData.key refers to defined GioKey.id
• ensure GioEdge.Endpoint.node has been defined in a GioNode.id (do at the end of graph)
• ensure GioEdge.Endpoint.port has been defined in a GioNode.Port.name (do at the end of graph)
• ensure all IDs used in Gio are completely unique in document

Test reader-dot in-depth:

• for each .dot sample file -- each testing only one feature of DOT (as short as possible)
  • Read to GioWriter
  • Validate calls to GioWriter contain all data from source file
• test subgraph feature
• test directed / undirected edges
• ports in DOT syntax: how are they handled by JGraphT DOTImporter?

If the XmlWriterImpl receives start & end element with no other content in between,
it should emit a self-closing tag.

E.g. instead of
<node id="node1"></node>
it should generate
<node id="node1" />

• sparse6Reader
• digraph6Reader

Spec: https://users.cecs.anu.edu.au/~bdm/data/formats.txt

• text based
• only printable ASCII characters
• one line per graph

graph6

undirected graphs
ext: ".g6"
support: no loops, no multi-edges

sparse6

undirected graphs
more space-efficient for large, sparse graphs
ext: ".s6"
support: loops, multi-edges

digraph6

directed graphs
ext: ".d6"

Usage

Format is used, e.g., by "House of Graphs", a graph test data repository.
E.g. here https://houseofgraphs.org/meta-directory/nut

See also https://houseofgraphs.org/help#format_g6

TODO

Decide if JGraphT can be used (EPL 2.0, many dependencies)

Use

in reader-tgf test to read all .tgf resources in the test.

• find DOT text files from the wild internet
• integrate into GioReader interface
• have a Junit test which reads TGF and write GraphML (depends on GraphML stuff, of course)
• have more JUnit tests at least for simple cases

Me, trying to convince them to export GraphML:
hyperbard/hyperbard#8

• 1)
  Download file from https://keybase.pub/mjdilworth/yed-aws-palettes/AWS%20-%20Developer%20Tools.graphml put in reader-graphml /test/resources as "AWS - Developer Tools.graphml".
  Next add a "AWS - Developer Tools.graphml.meta" file with the single line "url-info=(the URL above)" in it.

• 2)
  Similar for
  https://git.flexsim.com/sdk/flexsim-module-sdk/-/blob/61a9f90ad6083efe3917f3b889c919d0126579ba/Documentation/Classes/svg/OnClickEngine.graphml

• 3) https://zivgitlab.uni-muenster.de/sciebo-rds/sciebo-rds/-/tree/gitlab-ci-changes-in-folder/docs/static/images

Download the graphml files sing the download button ON the page, dont download the HTML PAGE :-)

• 4)
  The snippet from
  http://kb.yworks.com/article/78/

https://en.wikipedia.org/wiki/DOT_(graph_description_language)

https://www.graphviz.org/doc/info/lang.html

• find DOT XML files from the wild internet
• integrate into GioReader interface
• have a Junit test which reads TGF and write GraphML (depends on GraphML stuff, of course)
• have more JUnit tests at least for simple cases

https://en.wikipedia.org/wiki/Trivial_Graph_Format

• find TGF files from the wild internet
• have a Junit test which reads TGF and write GraphML (depends on GraphML stuff, of course)
• add some simple tests (parse several files and verify validation errors are found)
• Adapt to new finding: edges are optional, even empty graph is valid

Using JGraphT, in the reader-jgrapht project

Samples files:

https://userpages.uni-koblenz.de/~ist/GXL/examples/instance/gxl/simpleExample/content.html
https://github.com/Bioconductor/RBGL/tree/master/inst/XML
https://git.litislab.fr/bgauzere/py-graph/-/tree/0cc89ae026108889b986aba01e3f999d12d37a2b/datasets/monoterpenoides
https://gitlab.liris.cnrs.fr/aekiouche/kite-project/-/tree/a5310781f18e28c19ba98941226bd01228bea915/data/Kite_graphs_xml

DTD
https://userpages.uni-koblenz.de/~ist/GXL/index.php

Nested ports are not correctly writtein to graphml

• Create graphml test file to have nested ports
• Use generic AbstractReaderTest to verify

• write all incoming data to GioWriter
• respect XML namespaces, if they are defined, e.g. if root element defined "ns0:svg-something..." we needs this pair "ns0:svg-something" in the resulting XML -> can be used as custom extended attributes
• default to GraphML namespace, if none defined
• Use any custom attributes, such as "yfile.type" -- and don't skip them, in fact, make sure they are in the resulting graphMl
• parse non-Graphml-XML (such as SVG) as String content into surrounding element (e.g. in data/key)
• use streaming SAX parsing

• add GioWriter.data( GioData data )
• remove GioElementWithData
• adapt all tests & verifiers to this change

stream-based, like the "inverse" of SAX

• eg. write metadata
• write node
• write edge

Test reader-tgf in depth:

• for each .tgf sample file:

  • Read to GioWriter and produce GraphML/XML in memory (via MemoryOutputSink)
  • Validate produced content via DTD validation (or XMLSchema if that is easier, for GraphML both is available)
  • Read produced graphml via reader-graphml (base project needs a dependency on reader-graphml, scope test), then write the read graphml to graphml.xml again

• read once: file.tgf =reader-tgf (->gio->graphml)=> file.graphml

• read again: file.graphml =reader-graphml (->(gio->graphml)=> file2.graphml

• now file.graphml should be byte-identical to file2.graphml (no need to use real files, just in-memory byte-buffers)

download all files from https://dgit.cs.uni-saarland.de/Nikolai/cpraa/-/tree/master/AFs
and as sample files.

put them all in a directory /AFs within /test/resources.

add one file ".meta" file with the single line "url-info=https://dgit.cs.uni-saarland.de/Nikolai/cpraa/-/tree/master/AFs"

• streaming processing

first iteration:
simply stream incoming content errors to a Logger.log and render them somewhat readable
(no stack, no file, no ..)
we can map our Level ERROR to ERROR, WARN to WARN

remember: all normal logging in the code happens in all normal levels as usual to Logger as well

no central messages

--Using AWS Lambda.-- -> Contabo or similar

Functionality:

• validate Graphml
• parse anything into Graphml

GET /status
response: "OK"

POST /validate (upload file)

POST /read (upload file)

• optional queryParam "inputType" with values "json", "tgf", "dot", "graphml"
  we can auto-detect input type -> GioEngine

• result is a zip file named "result.zip" which contains
  -- result.graphml
  -- log.txt

• create a first swagger doc for the API

• create initial impl with at least "GET /status"

  • creating streaming zip response in memory, no files written to disk

• add more functionality

Add test with e.g. 3 nodes and 2 edges (with and without label), verify that this goes into Gio-stream -> mock

Using JGraphT, in the reader-jgrapht project

Sample files:

https://gexf.net/data/hello-world.gexf
https://github.com/Man-UP/GroupMyContacts/blob/master/dataCrawlerBirthMonth.gefx
https://github.com/Man-UP/GroupMyContacts/blob/master/dataCrawlerBirthDay.gexf
https://github.com/gephi/gephi-toolkit-demos/tree/master/src/main/resources/org/gephi/toolkit/demos

• endGraph now with GioLocator

no unused imports

all files auto-formatted

all interface types

• documented
• tested

"most interface methods documented, where not clear"
each major feature has a JUnit test (each reader-read, reader find validation error)