mondego / oreo Goto Github PK

Hi,
When I run controller.py, nothing is generated in the candidates folder (I have repeated the same program 10 times so there are clones!), and the log file says that number of clone pairs is 0. Also the files candidates.file and query,file are empty for all shards.
What can cause this?
Here is the content of the log file in SCC_LOGS/NODE_1/scc.log:
2019-04-14 21:49:36,401 INFO c.m.i.SearchManager [main] user.dir is: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector
2019-04-14 21:49:36,402 INFO c.m.i.SearchManager [main] root dir is:/mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/
2019-04-14 21:49:36,402 INFO c.m.i.SearchManager [main] reading Q values from properties file
2019-04-14 21:49:36,403 INFO c.m.i.SearchManager [main] propertiesPath: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/NODE_1/sourcerer-cc.properties
2019-04-14 21:49:36,416 INFO c.m.i.SearchManager [main] Query path:/mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/input/dataset
2019-04-14 21:49:36,416 INFO c.m.i.SearchManager [main] level1ShardSegmentString String is : 30,60,120,300,600
2019-04-14 21:49:36,418 INFO c.m.m.Shard [main] shard Shard [id=1, minSizeToSearch=1, maxSizeToSearch=30, minBagSizeToIndex=1, maxBagSizeToIndex=50, indexPath=1, subshardsSize=0] created
2019-04-14 21:49:36,418 INFO c.m.m.Shard [main] shard Shard [id=2, minSizeToSearch=31, maxSizeToSearch=60, minBagSizeToIndex=19, maxBagSizeToIndex=100, indexPath=2, subshardsSize=0] created
2019-04-14 21:49:36,418 INFO c.m.m.Shard [main] shard Shard [id=3, minSizeToSearch=61, maxSizeToSearch=120, minBagSizeToIndex=37, maxBagSizeToIndex=200, indexPath=3, subshardsSize=0] created
2019-04-14 21:49:36,418 INFO c.m.m.Shard [main] shard Shard [id=4, minSizeToSearch=121, maxSizeToSearch=300, minBagSizeToIndex=73, maxBagSizeToIndex=500, indexPath=4, subshardsSize=0] created
2019-04-14 21:49:36,418 INFO c.m.m.Shard [main] shard Shard [id=5, minSizeToSearch=301, maxSizeToSearch=600, minBagSizeToIndex=181, maxBagSizeToIndex=1000, indexPath=5, subshardsSize=0] created
2019-04-14 21:49:36,418 INFO c.m.m.Shard [main] shard Shard [id=6, minSizeToSearch=601, maxSizeToSearch=500000, minBagSizeToIndex=361, maxBagSizeToIndex=833333, indexPath=6, subshardsSize=0] created
2019-04-14 21:49:36,418 INFO c.m.i.SearchManager [main] Number of Top level shards created: 6
2019-04-14 21:49:36,418 INFO c.m.i.SearchManager [main] action: search
threshold: 6
QLQ_THREADS: 1 QBQ_THREADS: 4 QCQ_THREADS: 10 VCQ_THREADS: 1 RCQ_THREADS: 1

2019-04-14 21:49:36,418 INFO c.m.i.SearchManager [main] initializing channels
2019-04-14 21:49:36,420 INFO c.m.i.SearchManager [main] action: search
threshold: 6
BQ_THREADS: 1
SBQ_THREADS: 1
IIQ_THREADS: 1

2019-04-14 21:49:36,421 INFO c.m.i.SearchManager [main] METRICS_ORDER_IN_SHARDS created: 1
2019-04-14 21:49:36,421 DEBUG c.m.i.SearchManager [main] NODE_1 MAX_TOKENS=500000 MIN_TOKENS=1
2019-04-14 21:49:36,423 INFO c.m.u.Util [main] creating directory: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/NODE_1/output6.0
2019-04-14 21:49:36,424 INFO c.m.i.SearchManager [main] Query File: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/1/query.file
2019-04-14 21:49:36,431 INFO c.m.i.SearchManager [main] creating indexes for /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/1/candidates.file
2019-04-14 21:49:36,432 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 68 micro seconds
2019-04-14 21:49:36,432 INFO c.m.i.SearchManager [main] indexes created
2019-04-14 21:49:36,433 INFO c.m.i.SearchManager [main] initializing channels
2019-04-14 21:49:36,433 INFO c.m.i.SearchManager [main] shutting down channels
2019-04-14 21:49:36,433 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 14 micro seconds
2019-04-14 21:49:36,433 INFO c.m.i.SearchManager [main] shutting down QLQ, 1555278576433
2019-04-14 21:49:36,433 INFO c.m.i.SearchManager [main] shutting down QBQ, 1555278576433
2019-04-14 21:49:36,433 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,433 INFO c.m.i.SearchManager [main] shutting down QCQ, 1555278576433
2019-04-14 21:49:36,433 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,433 INFO c.m.i.SearchManager [main] shutting down VCQ, 1555278576433
2019-04-14 21:49:36,433 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 9 micro seconds
2019-04-14 21:49:36,434 INFO c.m.i.SearchManager [main] shutting down RCQ, 1555278576434
2019-04-14 21:49:36,434 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,434 INFO c.m.i.SearchManager [main] COMPLETED LINES: -1
2019-04-14 21:49:36,434 INFO c.m.i.SearchManager [main] Query File: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/2/query.file
2019-04-14 21:49:36,439 INFO c.m.i.SearchManager [main] creating indexes for /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/2/candidates.file
2019-04-14 21:49:36,439 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 20 micro seconds
2019-04-14 21:49:36,439 INFO c.m.i.SearchManager [main] indexes created
2019-04-14 21:49:36,439 INFO c.m.i.SearchManager [main] initializing channels
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] shutting down channels
2019-04-14 21:49:36,440 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 12 micro seconds
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] shutting down QLQ, 1555278576440
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] shutting down QBQ, 1555278576440
2019-04-14 21:49:36,440 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] shutting down QCQ, 1555278576440
2019-04-14 21:49:36,440 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 12 micro seconds
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] shutting down VCQ, 1555278576440
2019-04-14 21:49:36,440 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] shutting down RCQ, 1555278576440
2019-04-14 21:49:36,440 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] COMPLETED LINES: -1
2019-04-14 21:49:36,440 INFO c.m.i.SearchManager [main] Query File: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/3/query.file
2019-04-14 21:49:36,446 INFO c.m.i.SearchManager [main] creating indexes for /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/3/candidates.file
2019-04-14 21:49:36,446 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 17 micro seconds
2019-04-14 21:49:36,446 INFO c.m.i.SearchManager [main] indexes created
2019-04-14 21:49:36,446 INFO c.m.i.SearchManager [main] initializing channels
2019-04-14 21:49:36,447 INFO c.m.i.SearchManager [main] shutting down channels
2019-04-14 21:49:36,447 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 20 micro seconds
2019-04-14 21:49:36,447 INFO c.m.i.SearchManager [main] shutting down QLQ, 1555278576447
2019-04-14 21:49:36,447 INFO c.m.i.SearchManager [main] shutting down QBQ, 1555278576447
2019-04-14 21:49:36,447 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,447 INFO c.m.i.SearchManager [main] shutting down QCQ, 1555278576447
2019-04-14 21:49:36,447 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,447 INFO c.m.i.SearchManager [main] shutting down VCQ, 1555278576447
2019-04-14 21:49:36,447 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,447 INFO c.m.i.SearchManager [main] shutting down RCQ, 1555278576447
2019-04-14 21:49:36,447 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,447 INFO c.m.i.SearchManager [main] COMPLETED LINES: -1
2019-04-14 21:49:36,448 INFO c.m.i.SearchManager [main] Query File: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/4/query.file
2019-04-14 21:49:36,453 INFO c.m.i.SearchManager [main] creating indexes for /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/4/candidates.file
2019-04-14 21:49:36,453 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 15 micro seconds
2019-04-14 21:49:36,453 INFO c.m.i.SearchManager [main] indexes created
2019-04-14 21:49:36,453 INFO c.m.i.SearchManager [main] initializing channels
2019-04-14 21:49:36,454 INFO c.m.i.SearchManager [main] shutting down channels
2019-04-14 21:49:36,454 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 13 micro seconds
2019-04-14 21:49:36,454 INFO c.m.i.SearchManager [main] shutting down QLQ, 1555278576454
2019-04-14 21:49:36,454 INFO c.m.i.SearchManager [main] shutting down QBQ, 1555278576454
2019-04-14 21:49:36,454 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 13 micro seconds
2019-04-14 21:49:36,454 INFO c.m.i.SearchManager [main] shutting down QCQ, 1555278576454
2019-04-14 21:49:36,454 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,454 INFO c.m.i.SearchManager [main] shutting down VCQ, 1555278576454
2019-04-14 21:49:36,454 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,454 INFO c.m.i.SearchManager [main] shutting down RCQ, 1555278576454
2019-04-14 21:49:36,454 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,454 INFO c.m.i.SearchManager [main] COMPLETED LINES: -1
2019-04-14 21:49:36,455 INFO c.m.i.SearchManager [main] Query File: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/5/query.file
2019-04-14 21:49:36,461 INFO c.m.i.SearchManager [main] creating indexes for /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/5/candidates.file
2019-04-14 21:49:36,462 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 16 micro seconds
2019-04-14 21:49:36,462 INFO c.m.i.SearchManager [main] indexes created
2019-04-14 21:49:36,462 INFO c.m.i.SearchManager [main] initializing channels
2019-04-14 21:49:36,462 INFO c.m.i.SearchManager [main] shutting down channels
2019-04-14 21:49:36,462 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 16 micro seconds
2019-04-14 21:49:36,463 INFO c.m.i.SearchManager [main] shutting down QLQ, 1555278576462
2019-04-14 21:49:36,463 INFO c.m.i.SearchManager [main] shutting down QBQ, 1555278576463
2019-04-14 21:49:36,463 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 14 micro seconds
2019-04-14 21:49:36,463 INFO c.m.i.SearchManager [main] shutting down QCQ, 1555278576463
2019-04-14 21:49:36,463 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,463 INFO c.m.i.SearchManager [main] shutting down VCQ, 1555278576463
2019-04-14 21:49:36,463 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 18 micro seconds
2019-04-14 21:49:36,463 INFO c.m.i.SearchManager [main] shutting down RCQ, 1555278576463
2019-04-14 21:49:36,463 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 12 micro seconds
2019-04-14 21:49:36,463 INFO c.m.i.SearchManager [main] COMPLETED LINES: -1
2019-04-14 21:49:36,463 INFO c.m.i.SearchManager [main] Query File: /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/6/query.file
2019-04-14 21:49:36,468 INFO c.m.i.SearchManager [main] creating indexes for /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/index/6/candidates.file
2019-04-14 21:49:36,469 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 26 micro seconds
2019-04-14 21:49:36,469 INFO c.m.i.SearchManager [main] indexes created
2019-04-14 21:49:36,469 INFO c.m.i.SearchManager [main] initializing channels
2019-04-14 21:49:36,469 INFO c.m.i.SearchManager [main] shutting down channels
2019-04-14 21:49:36,469 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 12 micro seconds
2019-04-14 21:49:36,469 INFO c.m.i.SearchManager [main] shutting down QLQ, 1555278576469
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down QBQ, 1555278576469
2019-04-14 21:49:36,470 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down QCQ, 1555278576470
2019-04-14 21:49:36,470 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 10 micro seconds
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down VCQ, 1555278576470
2019-04-14 21:49:36,470 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down RCQ, 1555278576470
2019-04-14 21:49:36,470 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 11 micro seconds
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] COMPLETED LINES: -1
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down channels
2019-04-14 21:49:36,470 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 7 micro seconds
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down QLQ, 1555278576470
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down QBQ, 1555278576470
2019-04-14 21:49:36,470 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 7 micro seconds
2019-04-14 21:49:36,470 INFO c.m.i.SearchManager [main] shutting down QCQ, 1555278576470
2019-04-14 21:49:36,470 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 8 micro seconds
2019-04-14 21:49:36,471 INFO c.m.i.SearchManager [main] shutting down VCQ, 1555278576471
2019-04-14 21:49:36,471 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 7 micro seconds
2019-04-14 21:49:36,471 INFO c.m.i.SearchManager [main] shutting down RCQ, 1555278576471
2019-04-14 21:49:36,471 INFO c.m.m.ThreadedChannel [main] time taken for shutdown: 7 micro seconds
2019-04-14 21:49:36,471 INFO c.m.i.SearchManager [main] signing off NODE_1
2019-04-14 21:49:36,474 DEBUG c.m.i.SearchManager [main] lock obtained? sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid]
2019-04-14 21:49:36,479 INFO c.m.i.SearchManager [main] NODES COMPLETED SO FAR: 1
2019-04-14 21:49:36,480 INFO c.m.i.SearchManager [main] Total run Time: 80870 micors
2019-04-14 21:49:36,480 INFO c.m.i.SearchManager [main] number of clone pairs detected: 0
2019-04-14 21:49:36,480 INFO c.m.i.SearchManager [main] moving /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/NODE_1/output6.0 to /mnt/c/Users/Feri73/Desktop/oreo-FSE_Artifact/clone-detector/backup_output/1/NODE_1
2019-04-14 21:49:36,490 INFO c.m.i.SearchManager [main] completed on NODE_1

Is there a truly working version of Oreo?
The original artifact is full of null pointers, and the current version comes with null pointers in the SearchManager (418).
I debugged and simplified lots of stuff but to be honest, the repository is a mess.

Now I am at the point where the clone detector searchers for the index
17:21:24.681 [main] ERROR com.mondego.indexbased.SearchManager - $rootPATH/index/1/candidates.file (No such file or directory)exiting

Which script or where is the "index" generated?

Thanks

Hi,
When I finished the command:
python controller.py 1
It actually generate candidate pairs. However there is no candidatesList.txt. So when I continue run:
./runPredictor.sh
The out_9900 file "found 0 candidate files". And of course, the clonepairs_type2_9900.txt is empty.

So I wonder if there is something that I miss when running the code.

when i run the following command(w10(64) python3.6(32) ):
$ python controller.py 1
there was an error :

D:\anaconda\envs\oero-master32\python.exe H:/oreo-master/clone-detector/controller.py
search will be carried out with 1 nodes
loading previous run state
H:\oreo-master\clone-detector\scriptinator_metadata.scc doesn't exist, creating one with state EXECUTE_1
previous run state 0
running new command H:\oreo-master\clone-detector\execute.sh 1
Traceback (most recent call last):
File "H:/oreo-master/clone-detector/controller.py", line 231, in
controller.execute()
File "H:/oreo-master/clone-detector/controller.py", line 58, in execute
command_params, self.full_file_path("Log_execute_1.out"), self.full_file_path("Log_execute_1.err"))
File "H:/oreo-master/clone-detector/controller.py", line 172, in run_command
p = subprocess.Popen(cmd, stdout=fo, stderr=fe, universal_newlines=True)
File "D:\anaconda\envs\oero-master32\lib\subprocess.py", line 707, in init
restore_signals, start_new_session)
File "D:\anaconda\envs\oero-master32\lib\subprocess.py", line 992, in _execute_child
startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application 。

Process finished with exit code 1

Where can I find the code implementing the Siamese architecture?