Part of the #54: create an internal library for community detection.

Small internal library in Python (a function or class) for community detection using IGraph, based on the algorithm from graph.py#detect_communities().

Should include a very basic tests (but not really testing IGraph implementation itself).

Setup release automation if needed and produce a first release.

• tag
• release notes
• GH release
• CI automation on tag
• push image to DockerHub (handled separately under #33 )

Show links to the files when it's found duplicates.

• Add reference hash to DB.
• For Github https://github.com/<repo>/blob/<ref_hash>/<file_path>
• For Gitlab - same URL schema is the same as github
• For BitBucket, according to docs 
https://bitbucket.org/<repo>/src/<ref_hash>/<file_path>

It should be able to work with all "formats". Siva (regular and buckets), bare, regular.
It also should provide different output depends on the number of repositories.

At least

• URL to Cassandra/ScyllaDB
• limit on the number of processed repositories (only for ./hash)

should be exposed though configuration.

CLI https://github.com/src-d/gemini#cli-arguments should be used to set Apache Spark configuration properties, as some commands do not use Spark (./hash or ./report)

Part of #54: small app to read connected components, do community detection and pretty-print the results.

A CLI app in Python, that reads the connected components from Parquet created in #58, does community detection using the library in #59 and prints the output to STDOUT, in a format consistent \w current duplicate output format.

A call to this app should also be included in ./report, right after a duplicate detection, so for a end user whole process will look like a single application.

Not mandatory, but might be a good idea to use some kind of simple templating library for formatting the output, so templates can be shared between JVM and Python.

Problem

tests sometimes fail with the following output.

How to reproduce

Run ./sbt test many times, and it'll fail sometimes. It is difficult to reproduce in the CI (because it takes more time), but I could see there that error a couple of times.

Using the tip of dpordomingo/gemini::reproduce-issue-22, I created a gist with the 3 scenarios I found:

Test succeeds

No data is stored in the test_hashes_duplicates

• report1-succeeding.txt -> output of the passing tests
• report1-succeeding-db.txt -> content of the testing keyspaces. Both keyspaces are full of data

Test fails

No data at all is stored in the test_hashes_duplicates

• report2-failing.txt -> output of the failing test
• report2-failing-db.txt -> content of the testing keyspaces. There is no data in test_hashes_duplicates

Partial data is stored in the test_hashes_duplicates

• report3-failing.txt -> output of the failing test
• report3-failing-db.txt -> content of the testing keyspaces. Only the half of the data retrieved by engine were saved in test_hashes_duplicates (the srcd/borges repo is hashed, but erizocosmico/borges repo is lost.

More info

It requires more investigation, but it could be related when how data is stored during the tests before the test cases start running.

Currently, in the beforeAll section, it is created two keyspaces (test_hashes_duplicates for tests needing duplicates, test_hashes_uniques for tests needing no duplicates)
This is done in two stages:

1. engine returns a DataFrame containing all the files in the given siva files,
2. the DataFrame is stored in Cassandra <-- (I think something fails at this point)

After the beforeAll the test cases run, and (sometimes) the ones asserting that there are duplicate files in test_hashes_duplicates fails.

It can be seen in the logs, that in all situations the DataFrame is populated with the right contents (1,2,3).
But then:

• when no data is stored at all, at the save stage, there is always outputed Wrote 0 rows to test_hashes_duplicates.blob_hash_files
• when half of the data is stored, at the save stage, there is one row that says: Wrote 47 rows to test_hashes_duplicates.blob_hash_files 47 rows
• when everything go well, at the save stage, there are some rows that say Wrote n rows to test_hashes_duplicates.blob_hash_files, and the sum of all rows sum 80 33 rows + 47 rows

Hints

• Why when duplicate data is partially saved into test_hashes_duplicates, the missing files are from github.com/erizocosmico/borges.git? (here)
• Why DataFrame is always populated with right data, but sometimes its data is not properly saved?

Part of the #54: update Scala implementation of the report to write a Connected Components of similar-files graph.

TODO:

• query a DB for candidates graph.py#find_connected_components()
• detect connected components
• define Parquet schema and write the results on disk

Umbrella issue for updating current implementation to use "similarity" from Apollo:

• Query DB, detect connected components, write Parquet in Scala #58
• Internal library in python (function/class) for community detection \w IGraph #85
• App in Python, reading Parquet, doing community detection using the library above and printing the output #60

After seeing some users face issues running gemini for the first time, I created a new Dockerfile image

See the code here: https://gist.github.com/campoy/d48de3f6aef356e641df36a1fea6a778

Once we have this, running gemini is as easy as:

docker run -v $(pwd)/siva:/repositories campoy/gemini:quickstart

I wonder if this should be part of this repo and where it should be added.

There is a limit for a job in Spark. It's 2GB. We need to investigate how to change it if possible and how it will affect spark. (the limit was introduced for some reason)

If somebody else will look at it, here is a tip. It looks like the limit is not from Spark actually, but JVM. I can be wrong. JFYI

Exception:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 415 in stage 1.0 failed 4 times, most recent failure: Lost task 415.3 in stage 1.0 (TID 1072, 10.2.15.79, executor 8): java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:869)
	at tech.sourced.siva.SivaReader.getEntry(SivaReader.java:42)
	at tech.sourced.engine.provider.RepositoryObjectFactory$$anonfun$genSivaRepository$1.apply(RepositoryProvider.scala:209)

Engine 0.5.1 was released, including bugfixes needed for #42

It also may include adapting to minor API changes.

Document in README the resources, needed to successfully process 1k, 2k, 10k, 100k and whole PGA of the .siva files.

So good start would be

• document the known configuration of the cluster we use internally
• running Gemini hash, documenting how long does it take to finish,
• what CPU-load, Mem, IO-thoughtput workload it creates on that cluster (i.e from Sysdig Dashbord, to get access file an issue)
• see which resource is a bottleneck
• try to optimize, in order to utilize that resource better (i.e in case of throughput - have more executor JVMs running on the same machine)
• see if we are hit and can help with some Engine issues

TODO:

• add/document sbt task with linter and code style
• integrate linter in CI

As well noted in #48 (review)

I would be nice if all of us use the same configuration for code style.

Part of the #53

Add query.py#query() logic to Gemini Scala API:

• no need to handle args.id case
• need to agree with ML team on a way to read WMH params like htnum and band_size that must be the same, as used for hashing
• use a DB from a Apollo hash run
• use vocabulary in .asdf (OrderedDocumentFrequencies), built by Apollo hash \w --docfreq

Part of the #53

We have one that does hash and then looks for duplicates. We need to update it to test similarity too.

I suggest such a plan:

• Amend current test dataset to include similar file(s)
• Populate DB table hashtables with fixture created using apollo with that dataset
• Use docFreq&params from apollo also
• Run bblfsh server & fe server
• Check query output and make sure both duplicates and similar files appear in the output

We have scripts that launch docker images, used by Travis CI. To skip them we look for STYLE_CHECK env var, but that is not the only case we should skip anymore.

if [[ -z "${STYLE_CHECK}" ]]; then

From this thread #83 (comment)

@carlosms

Maybe we should find a better way to skip docker images? For instance we will have at some point lint for python, scala, and other possible new tests that need a different env.
Would it work if in travis.yml we moved the before_script: inside each matrix entry? It would be more verbose, but easier to spot where we are launching docker.
I'm not sure about the best option, but it's something worth looking into.

@smacker

current check is copy-past for a script we already use. But I agree is need something better. Maybe also use docker-compose as proposed by smola.

And from this other thread #83 (comment)

@smola

I understand this is not strictly related to this PR, since you were already doing this previously with scripts/start_docker_db.sh, but you might consider using docker-compose for this in the future.

It can be easily pre-installed in Travis (see here) and it's a single file with more concise syntax to define Docker dependencies. You can choose to start all of them with up or specific up bblfsh.

@smacker

yes. We are going to have 3 docker images (db, bblfsh, feature extraction) in dependencies for gemini. It makes perfect sense to use docker-compose in the feature. (actually, I already do locally)

Add LICENSE file + to readme. See: src-d/guide licensing policy.

One of the Scala libraries for CLI parsing can be used, i.e https://github.com/scopt/scopt

This is an umbrella issue for Feature Extractors (FE) implementation:

• Define gRPC Service, messages in src/main/proto/*.proto #52
• Generation scripts: server in python src/main/python/generated.py, #57
  Python CLI app src/main/python/main.py configures the port
• Generation scripts: client in scala src/main/scala #63
• Docker file \w feature extractor (used on CI) #73
• Implement FE service: call sourced.ml.extractros #73
• Add missed weight parameter to extractors #79
• Integration Tests: add CI profile, that for a given UAST checks the response is not failing #81

Umbrella issue for adding perf measurements to every command for file-based similarity:

• Add CLI flag for ./query ./report to print time for each stage
• Instrument FE, exposing one endpoint \w json
• Instrument Apache Spark hashing job using org.apache.spark.groupon.metrics.UserMetricsSystem to expose to Spark JSON endpoint

Umbrella issue to update current implementation to use a :similarity" notion from Apollo:

• UAST extraction #68
• talk to FE gRPC, to extract features #69
• Find/write CPU WMH implementation in Scala (tests!) #78
• Write query.py#query() logic on Scala side #89
• Update ./query CLI output to include similar files #92
• Integration test with all components (except hash, use cql fixture for it)

Right now, only tests do not rely on having existing schema in DB.

Each command should be changed (hash, query, report) to do the same and create schema, instead of failing if it does not exist.

This command will output the duplicate files among all the hashed files:

• <file_path> | <repo> | <sha1>
• show the <sha1> only when command is ran with --verbose mode
• add a lik to the file in GH (based in HEAD) -> https://github.com/<repo>/blob/HEAD/<file_path>

To streamline first user experience:

• add simple Docker compose script for dependencies: DB and FE - it will enable part of #84
• make CI use it (so only appropriate containers are started - as suggested in #91)

Initial work duplicate code detection was done under this DesignDoc.

Next steps should be solicited in a new improvement proposal DesignDoc.

Currently, we have 2 flags: group-by and condensed. Both of them require Cassandra and can't work with ScyllaDB. We think condensed flag is quite confusing. So we propose to keep only group-by flag.

Same as with Scala in #50 - as soon as we have src/main/python - it would be nice to document and add https://github.com/google/yapf support.

Right now we have a profile on CI that does Integration Tests with Apache Spark in local mode.

In order to be able to catch more trickier issues i.e \w runtime classpath collision, we need test it also in Apache Spark cluster mode Standalone configuration.

TODOs:

• add new profile on CI \w INTEGRATION_TESTS=true
• same test scenario, as in local mode, except for
• add starting Apache Spark Master and Workers manually
• run Gemini using, using the above with MASTER="spark://127.0.0.1:7077" ./hash ...

After #49, it would be nice to have auto-formatter

TODO

• add/configure scalafmt support
• document IDE configuration http://scalameta.org/scalafmt/#IntelliJ as default InteliJ one can produce a bit different output

As well noted in #48 (review)

I would be nice if all of us use the same configuration for code style.

Right now, Gemini has 3 use cases:

1. local, for developer on pre-configured environment, using shell scripts
2. local, for a first-time user, #84 (comment)
3. k8s \w Apache Spark cluster, Dockerfiles: Gemini \w Spark, Feature Extractors

Current release artifacts are only .tar file for 1 and a Docker file.

This issue is about changing the release process to accommodate recent changes on file-level similarity so:

• add .sh for starting a feature extractor process on local machine
• in ./report, check if Python is available
• make sure Dockerfile for FE re-uses the shell scripts (as much as possible)
• add publishing Docker container for FE to the release process
  (this will enable testing Gemini on 3rd use-case)

this way, a new release should accommodate all 1 and 3 case from above.

As discussed in #63 (comment) , this project is governed by GPL3 but include few (well identified, .proto) files under BSD-like licenses, thus we need to make sure we keep copyright notice/disclamer and redistribute it as a part of the final released artefacts.

To do so, we just need to:

• make sure LICENSE file does not miss any details of all third-party licenses from external resources like src/main/proto, etc
• update the release automation https://github.com/src-d/gemini/blob/master/scripts/release.sh to include LICESE file to the distributed archive.

To reconcile metadata schema and use the same one between Apollo/Gemini we need to:

• Apollo: bucket meta by path src-d/apollo#21
• Apollo: unify formatting \w Gemini URLFormatter, bucket by ref_hash/commit src-d/apollo#22
• Gemini: refactoring, for consistent naming in code i.e ref_hash->commit #48
• Gemini: change default keyspace/column names in schema to match Apollo ones
• Gemini: add Go query to the CI

Part of the #53

Add UAST extraction using https://github.com/bblfsh/client-scala to the query API

Right now, a number of feature extractors is already supported and used by default in Gemini.

This task is about:

• adding one more extractor from sourced-ml - GraphletBagExtractorand
• change Gemini defaults to use 3 extractors: Graphlet, Identifier and Literals

Let's decide about tags/branch on which it should happen later.

The current proposal is to build images from 2 branches:

• master
• experimental

Part of #53 and #55

For both, query and hashing similar files we need to have a CPU-based WeightedMinHash implementation in JVM, to avoid depending on MinHashCUDA lib and GPU.

Here are few reference implementations of this algorithm:

• Go impl
• Python impl
• C++ CUDA impl

It might make sense to research if such library already exists in JVM ecosystem, and if not - it might make sense to implement one in Java as part of Gemini codebase. This way it can be later used from Java, Scala, Clojure, etc eventually, as a standalone library.

Correctness verification is of paramount importance, so we would need to have some tests, and may be a reference data \w hashes of some sets, produced by one of the implementations from above.

Umbrella issue for adding hashing similar files using Apache Spark:

• UAST, feature extraction (same args as in query)
• Tokenize, Vectorize: wBOW, tf/idf - defining a pipeline
  • generate docFreq.json and params.json
  • vectorize: file->set
• Use CPU-based WMH implementation in Scala from #67 to hash every set
• Take results of hash pipeline, create hash tables, write to ScyllaDB \w schema as in Apollo #111
• Test correctness: Apollo and Gemini produces same results (use same seed if needed)

After #95, update the README structure, add

• "Easy Way" a quickstart for Gemini \w containers
• "Hard Way" as discussed in #84, a separate file under /docs/, with description on how to run it on Apache Spark cluster \w Scylla/Cassandra with all the details
• Apply suggestions from https://github.com/src-d/guide/blob/master/engineering/documentation.md

This change can bee small - it does not need to be "complete" in documentation coverage or full conformance \w doc guide from above, but it has to setup the right "structure" of our user-facing documentation, that will be improved later.

Right now, after #90 (comment) ./report command (at least on macOS) produces

WARN 11:57:50 org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner (FileSystem.java:2995) - exception in the cleaner thread but it will continue to run
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:2989)
	at java.lang.Thread.run(Thread.java:748)

It is may or may not be related to HADOOP-12829 - the goal is to find out and fix it.

As we have src/main/python, it would be nice to have something like a https://www.pylint.org/ integrated in CI.

For query and report tools:

• add -v flag, producing more verbose debug output
• replace println() with proper slf4j logging (same as Apache Spark uses) on different levels INFO/DEBUG

The Spark Application logging will be handled in different issue.

Most probably something is wrong with documentation.

Right now test use EmbeddedCassandra to speed up CI \wo Docker.

But proper INTEGRATION_TESTS=true profile would not be possible \wo Docker in before_script anyway, so it would make sense to just rely on it in both Dev and CI flows (as we do in other projects)

Part of the #53

Update ./query CLI output to include not only duplicates, but also a similar files.

Same as in report #60, it may (or may not yet) make sense to start using some kind of templating.

On local ./sbt run as well as on CI, sometimes test become flaky with org.eclipse.jgit.errors.MissingObjectException.

Example:

ERROR 15:13:13,185 org.apache.spark.internal.Logging$class (Logging.scala:91) - Exception in task 0.0 in stage 1.0 (TID 2)
org.eclipse.jgit.errors.MissingObjectException: Missing commit 4aa29ac236c55ebbfbef149fef7054d25832717f
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:164)
	at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:903)

CI

• https://api.travis-ci.org/v3/job/319232107/log.txt from this run, but same commit passes here
• TODO add more

This might, or might not have to do something either with Engine behaviour, or with test fixtures that we have.

This issue is about investigating the reason and if needed, filing appropriate issues elsewhere to fix it.

For ./hash
migrate to logger, provided by Apache Spark
output: mute meaningless Spark logs (though log4j.config)

This would most probably mean to refactor Gemini class to have logging dependency injected (thorough constructor or .set...()), as it will be instantiated by the client (Spark/non-spark ones) in a different ways.

When we have identical files in repository, only one is written in DB and duplicates won't appear in report because the primary key is the same for them.

Steps to reproduce:

smacker at Maxims-MacBook-Air in ~/tmp/testrepo on master*
$ ls
CONTRIBUTING.md file.py         file_2.py

file_2.py is copy of file.py.

Run engine to collect files:

+--------------------+--------------------+---------------+--------------------+----+
|         commit_hash|           file_hash|           path|       repository_id|name|
+--------------------+--------------------+---------------+--------------------+----+
|06e561f1a7d6db4f3...|c4e5bcc8001f80acc...|      file_2.py|file:///Users/sma...|HEAD|
|06e561f1a7d6db4f3...|eaf26a547aa54cde7...|CONTRIBUTING.md|file:///Users/sma...|HEAD|
|06e561f1a7d6db4f3...|c4e5bcc8001f80acc...|        file.py|file:///Users/sma...|HEAD|
+--------------------+--------------------+---------------+--------------------+----+

Check what we have in DB after hash:

cqlsh:hashes> select * from blob_hash_files;

 blob_hash                                | repo                               | file_path
------------------------------------------+------------------------------------+-----------------
 eaf26a547aa54cde7079567d832ac05880eb6bd2 | file:///Users/smacker/tmp/testrepo | CONTRIBUTING.md
 c4e5bcc8001f80acc238877174130845c5c39aa3 | file:///Users/smacker/tmp/testrepo |       file_2.py

(2 rows)

The longest task is integration. Most of the time takes:

• install python. We can improve it a bit, installing some deps from apt instead of building
• docker for FE. We can remove it now when we have python in CI
• Make build. We can try to use travis cache to improve it

also check that we don't run services when they aren't necessary

Part of the #53

Call Feature Extractors using gRPC client, to get feature from the UASTs.