For increasing graph lisility, here are some enhancements concerning thumbnails program

1. non graphical files
   Writing file name in a empty bitmap
   General syntax :
   "convert -pointsize 36 -fill red -background white label:<file_name> ../thumbnails/<index_name>.png"
   including subdirectories names of main directory
   but with discrete and tasteful colors

2. graphical files
   Adding files name label to thumbnail
   General syntax to add to present convert command:
   "label:'<initial_filename>' -gravity Center -append"
   <initial_filename> does not contain directory path, other option stays unchanged.

Informations on closed issue #17

3. using extracted data from documents (like PDF files)
   Such data must be considered as metadata from source document. We can be led to be able to compute clusters from such metadata.
   First example exists in MyLib: metadata are vectors of words extracted from PDF which are content.
   The need is to preserve relation between document name, metadata name and thumbnail name.

Actually, this relation is preserved with filenames themselve. Vectors and thumbnails are produced by external tools. That implies : ensuring that graph program allows ever to work with thumbnails directories (that means "with real filenames) or adapt thumbnails program including "convert" scripts.

For "prep-data" that means the ability to store source document, vector name and thumbnail presentation (see joined document in french on pivot format definition).

This issue replace existing issues #17 #38 #39 (which are closed)

roadmap LT documentare.pdf

Review the thumbnail computing process for adapting it to docker implementation (main problem : troubleshooting with path and naming of content and thumbnails)

We should be able to compute clusters from metadata extracted from content.
First example exists in MyLib: metadata are vectors of words extracted from PDF which are content.
In NCD process we have no more relations between documents name in PDF and vector files except filenames (without termination).
The only way to display a readable graph is to have thumbnails of PDF inside.
Because these are "external" relations, the only way to create thumbnails is to execute an external program and the only way to add these thumbnails in graph is to use option -d in graph program.
So, maintain directories options in coming evolution for allowing external thumbnails.

Current version of the API supports requests which provide a directory as a parameter to load data: we load all files from the provided directory.

BUT, we would like to provide a new way to load data, with a request which will look like this:

{
 [
  { "id": "34838434" ,
    // Here the data will be loaded from a file resource
    "file": "dat/pdf/bill.pdf"
  },
  ...
  { "id": "45435345" ,
    // Here the data are already present as a Base 64 string
    "bytes": "Sm9UT3Bo..."
  },
  ...
 ]
}

So compared to current version (input directory only), it will give us the following possibilities:

• work only on a subpart of a directory
• do not work with files, but provide bytes directly when data are small

ClusteringGraphBuilder class needs to be refactored, since it seems to be difficult to understand the main steps.

Add 'sloop' boolean parameter in the ClusteringParameter class.

• if scut != 0 => scutStart = scut
• else scutStart = 3
  The test ClusteringParametersTest needs to be updated

We want to generalize the use of a tag prefix to improve NCD results (suggested tag is "JoTophe").

So we suggest to:

• remove the tag from raw files generated by the LineDetection segmentation tool
• in NCD, add this tag as prefix for all input files

This single change will provide a new version which can be tested alone.

I suggest to display a functionnal example on our server with:

• default values for clustering parameters,
• true directoires (for avoiding "mismatched" directories, create a kind of junk under /data-mylib, for example : /data-mylib/example)
• when launching server with SimdocServer: optional parameter indicating root directory for shared data (i.e. "/data-mylib" or other name depending client application, then by testing API with swagger, one can better understand used directories tree).

To be discussed.

There is perhaps a problem with memory management:

1. even with largest memory (managed with -Xmx java option), java heap error on large directories containing raw pictures (1500 files of 1,8 MB jpeg files).
2. ncd process can work at 100% memory in some cases, without java heap error, why?

(directory "/Claudia/NewYork" on both servers z620 and z820 under home directory of jyig5563)

Need of harmonization in naming options in command lines

to be discussed.

LineDetection

no option -> can be maintained
[possible upcoming enhancement allowing to input a directory of pictures ou a multipage picture]

NCD

-file1 and -file2 -> change to -d1 and -d2
[possible upcoming enhancement allowing to compare a file with a directory or two files]
-simDocJsonGz -> change to -simdoc
-help -> change to -h or -help indifferently

PrepClustering

-f -> change to -json (what need this parameter? a compressed json file!)
-help -> change to -h or -help indifferently

SimClustering

-h -> change to -h or -help indifferently
-i -> change to -json
-simDocJsonGz -> change to -simdoc
-scut and -sdScut -> change -scut as optional parameter with optional standard deviation value (default 2)
-ccut and -tileCcut -> change to -ccut as optional parameter with optional quantile value (default 75)
-sdArea -> change to -sdarea as optional parameter with optional value (default 2)
-sdQ -> change to -sdq as optional parameter with optional value (default 2) <- previous mismatch typing value 3 (sorry!)
[explanation : use of KNN allows to avoid Area and equilaterality computing with a short value of k]

Graph

-h -> change to -h or -help indifferently
-i -> change to -json

A very strong improvement has been noticed during tests by using raw image format. The reasons are:

1. true bitmap used for measurement (no dependence with file coding specifications)
2. blocking the length of byte sequences by the use of a fixed separation word (the "JoTophe" word)

One can also foresee that a simple Base64 coding of this raw sequence of bytes will occurs the same improvement (to be discussed).

Therefore, using raw or Base64 format represents a good way ahead to improve NCD.

The general format of measurement sequence should be "JoTophe"& or "JoTophe"&Base64()

This evolution should be made in two steps :

1. raw format application for all kinds of pictures
2. encapsulation in a Base64 coding with consequences in models (filesystem and json modes) convergence (a new document model story to define).

No change in NCD, PrepClustering and SimClustering. generalization of dictionary principle to keep relation between measured content and data representation for measurement : changes in thumbnails (preparing an import module) and graph.

• Code review to clean up current impl of ncd remote & simdoc-server
• Clustering API: must rely on Tasks now as the Distance service
• add services discovery

Having a discussion on API spec

Despite correct results, need to straighten things out with Voronoï process in SimClustering
Technical target : becoming progressively independent of GraphViz).
Functional : Voronoï's networks are useful to increase clustering from triangulation : if a cluster candidate has many areas with short distances, one can suspect many clusters inside (important for OCR and other application needing a thin and accurate segmentation)

See message below.

jyig5563@g-z820-cm:~/NY_ref$ java -jar ../Bastet/ncd-remote-1.40.1.jar -j1 bytes-data.json

[NCD REMOTE - Start]
Assumes j2 = j1
J - Mem 6% 0.5/7.0GBNo suitable constructor found for type [simple type, class com.orange.documentare.core.prepdata.PreppedBytesData]: can not instantiate from JSON object (missing default constructor or creator, or perhaps need to add/enable type information?)
at [Source: java.io.FileInputStream@149494d8; line: 2, column: 3]

Do you have any estimation on the time consuming increase of NCD due to raw format choose compared to png ? (not so important question, but a good indice for use definitions)

avoiding hidden files in processing

cannot handle swagger query for demo

Just exclude these distances in similarity program like in -scut but with the only criterion of distance >=1

1. mode of correction : same decision test as in -scut parameter (exclude triangle) and same actions (if no more edges connected to a node, flag it as singleton), normally, we should have no change to operate in other programs.
2. observations : a triangle containing an edge with length > 1 is in all cases a source of ambiguity for clustering particularly with noisy data (images). In theory, a distance equal to 1 appears when byte-string are completely different, that explain the choice to exclude all distances >=1
3. functional usage : no new parameter to define, the cause of this problem comes from length of byte strings after computing run-length and is corresponding to an exception due to limit conditions of distance measurement (equation right, but wrong data)
   This is a kind of "palliative surgery". When this problem of run-length will be corrected, this test will become "dead code".

Using a "KNN" approach in SimClustering

Actually, SimClustering is computing clusters on all distances matrix computed by NCD. This give good results on coherent alignments in bytes sequences, occuring a large interval of distances values betwwen 0 and 1. Typical coherent alignment is found with text files or DNA sequences.

The problem of misaligned sequences, like pictures is that occurs difficulties to evaluate common parts between sequences with the consequency of computed distances included in a short interval, near from 1.

One can put the hypothesis that we have very short common parts between two pictures, and this property generates clusters including too much non representative data, corresponding to a noise in measurement.

The idea consists of using just firsts nearest distances in SimClustering, eliminating, by the way, the noise generated by non representative data : this will not increase intervals of distances, but will just compute clusters on really relevant distances.

The other advantage of this KNN use in clusters computing is that SimClustering will work on a reduced set of data.

The sequence of algorithms

NCD

The distances matrix computing program will not be modified.

PrepClustering

Other parameters will not be changed.

In command line, a parameter ("-k") and a value (the number of retained neighbours) will filter the distances matrix, retaining k+1 firt neighbours. The +1 operation is related to clusters boundaries which are situated between k and k+1 neighborhood of patterns.

PrepClustering will compute triangles where we find :

• vertex1 : current pattern
• vertex2 : first neighbour of vertex1
• vertex3 : first neighbour of vertex2

And 3 segments whose length are distances and joining :

• vertex1-vertex2
• vertex2-vertex3
• vertex3-vertex1

By retaining only k+1 neighbours, we will just write existing triangles in knn intervals. In fact, if vertex3 is not found in filtered matrix, that means that vertex3 and vertex2 (which is, in fact, the k+1 neighbour of vertex1) will not belong to current cluster, because distance represented by vertex3-vertex1 is outside knn limit.

SimClustering

There is, normally, no modifications to do here. SimClustering will compute Heron's and equilarerality formulas, but on a reduced set of triangles. pattern eliminated by this filter will be considered as singletons.

Question and remarks to discuss :

• Are Heron's and equilaterality formulas computed in SimClustering or am I wrong ?
• This evolution should be developed in a "separate branch", because it needs a validation through experience with a large set of data.

[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080
Service http://g-z440-cm:8080 can not handle more tasks
Request to http://g-z440-cm:8080 failed: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.BufferedReader@714c002a; line: 1, column: 2]
Request to http://g-z440-cm:8080 failed: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.BufferedReader@6d86613c; line: 1, column: 2]
Request to http://g-z440-cm:8080 failed: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.BufferedReader@5232be6d; line: 1, column: 2]
Request to http://g-z440-cm:8080 failed: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.BufferedReader@658e1d75; line: 1, column: 2]
Request to http://g-z440-cm:8080 failed: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.BufferedReader@2e04eaed; line: 1, column: 2]
Request to http://g-z440-cm:8080 failed: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.BufferedReader@301e6f87; line: 1, column: 2]
[SUCCESS 55%] from http://g-z440-cm:8080 took 229s (1.41s/elem)
[POST REQUEST] http://g-z440-cm:8080
[SUCCESS 55%] from http://g-z440-cm:8080 took 229s (1.42s/elem)
[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080
[POST REQUEST] http://g-z440-cm:8080

the tool will be used this way: java -jar ImageToRaw.jar -d path/to/input/image/directory

=> it will generate a raw output directory which will contain the initial images converted to our raw image file format, ie: image matrix as a byte array, with the '\n' character at the end of each row.

make filenames management reliable ( and other non alphanumerical characters).

(despite the fact that we recommand "normalized names" input)

an useful enhancement : using the ability of imagemagick to transform ASCII files to bitmap.

General syntax :
"convert -pointsize 36 -fill red -background white label:<file_name> ../thumbnails/<index_name>.png"
including subdirectories names of main directory

but with discrete and tasteful colors

If using simple text files (without termination), on sfdp script we obtain this example of message :
Warning: No or improper image="./platypus.png" for node "c18_platypus_92".

That means that Graph build systematically a list of thumbnails from filenames before exporting .dot file. (conditional test missing?)

Display help message rather than a system error message.

Most applications name duplicate file name with [filename (n)]

That's occurring a problem in program similarity when computing temporary graph for Voronoï process by the fact that "(" and ")" characters are incompatible (reserved markups) in dot files.

Consequence : similarity program is stopped at Voronoï computing.

Applying the same rules than in final graph building for temporary graphs?

[Thumbnails - Start]
Can not create thumbnail, provided image '/Users/jyig5563/Desktop/plib/./pdf/1Is6Kqopl9ErAdhgrS9R8eXAWm0zZz.PDF' is invalid
Niels:plib jyig5563$

guilty file joined

1hmzZzIxvPcKymqyDBZx6avrXnO3wzZz.PDF

For increasing graph lisility, here are two enhancements using imagemagick's convert properties

1. non graphical files
   Adding file name in a empty bitmap
   General syntax :
   "convert -pointsize 36 -fill red -background white label:<file_name> ../thumbnails/<index_name>.png"
   including subdirectories names of main directory
   but with discrete and tasteful colors

2. graphical files
   Adding files name label to thumbnail
   General syntax to add to present convert command:
   "label:'<initial_filename>' -gravity Center -append"

<initial_filename> does not contain directory path, other option stays unchanged.

Joined : produced exemples

Non graphical:

Graphical:

Am I right if I suppose that we cannot launch more than one program ncd-remote ?

Add a canceling of eventual distant running ncd processes at ncd-remote starting.

"kNearestNeighboursThreshold=2147483646" : Was ist das?

To ease deployment and run, we could package simdoc servers (UI & backends) in a docker image or several docker images (compose).

In case of PDF files used in NCD, no thumbnails produced. Add this point to thumbnails conversion.

Having a meeting to plan this emerging problem.

The aim of this functionality is to find new clusters inside the set of singletons produced by similarity program. A further evolution will consist to study a fusion methods of clusters which will be the recall stategy of our clustering method, by using multisets (one cluster = one multiset).

• First step : copying singletons in a dedicated directory. This function should be optional for debugging and testing new ncd.
• Second step : including a "-enroll" parameter in similarity program which will compute new clusters from all singletons. In this case, similarity should work on existing distance matrix. This relaunch of similarity will use same parameters than in first launch.
• Third step : complete integration of this functionality (docker).

take "mnvw clean install" in account in build documentation

Need for reviewing processing :

• if mounted directory and local subdirectory are different : symbolic links are containing the path of docker container.
• detected in source-symlinks : "1" symbolic link ever missing
• Impossible have a coherent graph (wrong link between node and thumbnail). Below, messages displayed when executing graphviz script (concerning all thumbnails)

Warning: No such file or directory while opening /home/jyig5563/data/outpdf/./safe-working-dir/0.png
Warning: No or improper image="/home/jyig5563/data/outpdf/./safe-working-dir/0.png" for node "c1_0_99"

This functionality will allow users to select part of document before processing.

• blocks will be processed from data computed by linedetection program. Blocks could be paragraphs, columns or non textual information. These blocks are pointable by a user click and sequence of pointing will be remained. Pointed text block will be sent to clustering process.
• blocks could be also pointed by user tracing a bounding box. For example, for non textual blocks like pictures). In this case, the user should have the ability to indicate a "block type"
  - if image or graphical : erasing all segmentation inside the block to produce a non textual node in json description,
  - if text : storing this new text block in model before sending it to clustering process).

problem if data to be clustered directory rights are not idle : no treatments done but ncd-remote keeps running

jyig5563@g-z620-4lfq:/home/tesfaces/png

There are not converted pictures to ./thumbnails directory (trace.txt joined)

traces.txt

The test is based on the "anatomie" source image. We suggest to keep only half of the image to speed up the test.

Here are some explanations for needed tools to evaluate similarity measurement using base 64 information transcoding.

The purpose

Base 64 is a generic information coding based on a alphabet composed of 64 bytes that will allow to free 64 other bytes to improve similarity distance by replacing run length with a more efficient algorithm. An other way ahead will consist of applying "combinatorial pattern matching" algorithm to evaluate direct byte sequences segmentation (delimiters will be coded of the 64 released bytes after alphabet compression).

This purpose means operating a reversible alphabet compression for data coding.

A first step for this experiment consists on evaluating measurement based on current algorithm, of similarities of information coded in base64 which would become a pivot format if good results obtained.

Globally, we have just to add a base 64 transcoding option inside data preparation.

For pictures

• We need to preserve linearity of bitmap, so, before base64 coding, a raw format transcoding could be necessary. If possible, maintain the -raw option.

For other content

• Generic (raw PDF, for example): add base64 coding facility.

• Work on metadata (vector of extracted words for example): transcode in base 64 extracted data and preserve alignment with source files through canonical filename (i.e. filename without termination) for permitting thumbnails computation in graph program with -src option.

Is that OK ?

Thumbnail 0.png not produced. Message :

Exception in thread "main" com.orange.documentare.core.comp.nativeinterface.NativeException: Command line 'convert /home/jyig5563/lfw/lfw-deepfunneled/Aaron_Peirsol/._Aaron_Peirsol_0004.jpg[0] -thumbnail x300 -background white -alpha remove -polaroid -0 /home/jyig5563/lfw/thumbnails/0.png > /home/jyig5563/lfw/thumbnails/0.png.log' returned error code 1
at com.orange.documentare.core.comp.nativeinterface.NativeInterface.launchCmd(NativeInterface.java:59)
at com.orange.documentare.core.comp.nativeinterface.NativeInterface.launch(NativeInterface.java:48), etc...

All other thumbnails OK, traces of NCD joined
trace.txt