_______________________________________
< SamSepi0l's AMoC 2021 Hackathon Entry >
---------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Sometime Github refuses to render jupyter notebooks; use this nbviewer page to see the notebooks in the browser with no live python kernel.
To run the dashboard, make sure you've run the data download targets from the Makefile, and make sure you're running a container with host networking because the Panel Server chooses a random port. (network in Makefile target)
The following is taken verbatim from the Hackathon description:
Bristol Cyber Security Group is hosting a cybercrime hackathon. Groups from select institutions are invited to create teams of 4 to tackle a three-day intensive hackathon.
The hackathon is aligned with our AMoC project (Advanced Modelling of Cyber Criminal Careers), which aims to tackle financially-motivated cybercrime on Darknet Markets by gaining a better understanding of cyber offenders, their motives, the interventions effective at disrupting their activities, the different roles within cybercriminal networks and how these evolve over time. Key outputs from the project will contribute to a software toolkit designed to assist law enforcement in their cybercrime investigations.
Teams will compete against each other on two data-driven cybercrime investigation tasks exploiting timestamped textual observations of underground cryptomarkets:
- Longitudinal visualisation: Create an innovative visualisation representing the datasets over time, with particular attention to how significant events might affect the nature of posting on the forums.
- Author re-identification: Train a competitive classifier to re-identify individuals' accounts across different forums, drawing on stylometric and temporal features.
-
Lukas Gienapp: Affilitation
-
Christopher Akiki: Affiliation 1 / Affilitation 2
Local output of tree -s --charset X . :
Tree
.
|-- [ 4096] data
| |-- [ 575721195] base.parquet
| |-- [ 9994088576] body_bertweet_embeddings.npy
| |-- [ 6662725760] body_embeddings.npy
| |-- [ 6662725760] category_embeddings.npy
| |-- [ 268] download.sh
| |-- [ 4096] hackathon_dataset
| | |-- [ 4096] fora
| | | |-- [ 80475006] abraxas.csv
| | | |-- [ 59372422] agora.csv
| | | |-- [ 13651267] alphabay.csv
| | | |-- [ 532069] andromeda.csv
| | | |-- [ 27654884] blackbank.csv
| | | |-- [ 39761017] bmr.csv
| | | |-- [ 1969455] bungee54.csv
| | | |-- [ 3901035] cannabisroad2.csv
| | | |-- [ 3164247] cannabisroad3.csv
| | | |-- [ 10165330] crackingarena.csv
| | | |-- [ 12338244] crackingfire.csv
| | | |-- [ 215033] darkbay.csv
| | | |-- [ 158679] darknetheroes.csv
| | | |-- [ 2441620] diabolus.csv
| | | |-- [ 167220] dogeroad.csv
| | | |-- [ 267554251] evolution.csv
| | | |-- [ 18226] greyroad.csv
| | | |-- [ 2221113] hackhound.csv
| | | |-- [ 117507] havana.csv
| | | |-- [ 454402] hydra.csv
| | | |-- [ 8364072] kingdom.csv
| | | |-- [ 766236] kiss.csv
| | | |-- [ 18774] mrniceguy.csv
| | | |-- [ 42652560] nucleus.csv
| | | |-- [ 884519] outlawmarket.csv
| | | |-- [ 1935573] panacea.csv
| | | |-- [ 30481136] pandora.csv
| | | |-- [ 5037959] pbf.csv
| | | |-- [ 398731] revolver.csv
| | | |-- [ 444288998] silkroad2.csv
| | | |-- [ 544526908] silkroad.csv
| | | |-- [ 40346538] thehub.csv
| | | |-- [ 22266519] themajesticgarden.csv
| | | |-- [ 617373] therealdeal.csv
| | | |-- [ 85842] tom.csv
| | | |-- [ 1390316] torbazaar.csv
| | | |-- [ 307867] torescrow.csv
| | | |-- [ 17882] tortuga.csv
| | | |-- [ 7229472] utopia.csv
| | | `-- [ 833656446] webkill.csv
| | |-- [ 592443] hackathon_tasks_overview.html
| | |-- [ 2868111] hackathon_tasks_overview.pdf
| | |-- [ 4096] measure_impact
| | | `-- [ 990714] fora_registedusers.csv
| | `-- [ 4096] reidentification
| | |-- [ 240999] test_dataset.csv
| | |-- [ 261243] test_dataset.json
| | |-- [ 938646] train_negative_users.csv
| | |-- [ 1017494] train_negative_users.json
| | |-- [ 25657] train_positive_users.csv
| | `-- [ 27783] train_positive_users.json
| |-- [ 4096] ngrams
| | |-- [ 34761549] bigrams.json
| | |-- [ 18799932] fourgrams.json
| | `-- [ 17436096] trigrams.json
| |-- [ 6662725760] subject_embeddings.npy
| `-- [ 4096] tsne_XY
| |-- [ 8434055] abraxasforums_perplexity_10_x_y.parquet
| |-- [ 2639180] agoraforums_perplexity_10_x_y.parquet
| |-- [ 1021387] blackbankmarketforums_perplexity_10_x_y.parquet
| |-- [ 105751] bungee54forums_perplexity_10_x_y.parquet
| |-- [ 183134] cannabisroad2_perplexity_10_x_y.parquet
| |-- [ 153972] cannabisroad3_perplexity_10_x_y.parquet
| |-- [ 10490] dogeroadforums_perplexity_10_x_y.parquet
| |-- [ 15847006] evolutionforums_perplexity_10_x_y.parquet
| |-- [ 1924] greyroadforums_perplexity_10_x_y.parquet
| |-- [ 132947] hackhound_perplexity_10_x_y.parquet
| |-- [ 30442] kissforums_perplexity_10_x_y.parquet
| |-- [ 22803] outlawmarketforums_perplexity_10_x_y.parquet
| |-- [ 71196] panaceaforums_perplexity_10_x_y.parquet
| |-- [ 1527708] pandoraforums_perplexity_10_x_y.parquet
| |-- [ 192278] projectblackflagforums_perplexity_10_x_y.parquet
| |-- [ 21831] revolverforums_perplexity_10_x_y.parquet
| |-- [ 1828187] thehubforums_perplexity_10_x_y.parquet
| |-- [ 814783] themajesticgardenforums_perplexity_10_x_y.parquet
| |-- [ 34099] therealdealforums_perplexity_10_x_y.parquet
| |-- [ 2264] tortugaforums_perplexity_10_x_y.parquet
| `-- [ 4042036] webkill_perplexity_10_x_y.parquet
|-- [ 4096] dockerfiles
| |-- [ 4096] gpu
| | `-- [ 755] Dockerfile
| `-- [ 4096] vanilla
| `-- [ 1368] Dockerfile
|-- [ 4096] docs
| |-- [ 489] commands.rst
| |-- [ 7878] conf.py
| |-- [ 256] getting-started.rst
| |-- [ 441] index.rst
| |-- [ 5112] make.bat
| `-- [ 5596] Makefile
|-- [ 11357] LICENSE
|-- [ 2231] Makefile
|-- [ 4096] models
|-- [ 4096] notebooks
| |-- [ 2361] 2.0-lg-embeddings.ipynb
| |-- [ 3183370] dashboard.ipynb
| |-- [ 17720] document_embeddings_with_LMs.ipynb
| |-- [ 13984] hackathon_tasks_overview.ipynb
| |-- [ 4509773] kl_evolution.ipynb
| |-- [ 9480] ngrams.ipynb
| |-- [ 6884] nucleus_size_series.ipynb
| |-- [ 21050] tsne.ipynb
| |-- [ 3118021] umap.ipynb
| `-- [ 2260] users.ipynb
|-- [ 2758] README.md
|-- [ 4096] references
|-- [ 4096] reports
| `-- [ 4096] figures
`-- [ 4096] src
|-- [ 4096] data
| `-- [ 987] generate-base-dataset.py
|-- [ 0] __init__.py
|-- [ 4096] __pycache__
| `-- [ 1348] utils.cpython-36.pyc
`-- [ 1283] utils.py
foo@bar:/amoc-hackathon$ make vanilla-docker-build-imagefoo@bar:/amoc-hackathon$ make vanilla-docker-run-jupyterfoo@bar:/amoc-hackathon$ make gpu-docker-build-imagefoo@bar:/amoc-hackathon$ make gpu-docker-run-jupyterfoo@bar:/amoc-hackathon$ make docker-run-rapidsfoo@bar:/amoc-hackathon$ make download-basefoo@bar:/amoc-hackathon$ make download-dashboard-datasetfoo@bar:/amoc-hackathon$ make download-embeddings-bodyProject based on the cookiecutter data science project template. #cookiecutterdatascience
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent