https://observablehq.com/@d3/bar-chart-race-explained

• - Download all clinicaltrialsrepository
• - Parsed data
• - Create working db (sqlite)

1. Bulk download and folders setup
2. Parse all xml files. XML_to_JSON_File.
3. JSON file to CSV for dataframe exploration
4. ML Pipeline

Questions:

• How to summarize data
• Show relations?
• Algorithm with main trials

• Solve first tooltips with secondary arrays

Charts to complement all lines:

• Number of total trials by year
• Trials by phase (7)

https://www.essycode.com/posts/create-sparkline-charts-d3/

Reindexing
https://chrisalbon.com/python/data_wrangling/pandas_dataframe_reindexing/
https://stackoverflow.com/questions/19324453/add-missing-dates-to-pandas-dataframe

Function to extract entities from files. Check if done in preprocessing or back-end. Applicable to any dataframe (clinical trials, pubmed citations)

Videos_ https://www.youtube.com/playlist?list=PLBmcuObd5An4UC6jvK_-eSl6jCvP1gwXc

Libraries:

• SciSpacy https://allenai.github.io/scispacy/

• https://github.com/jakelever/kindred (Spacy Kindred)

• NLP PRE https://github.com/NIHOPA/NLPre

• Saber https://spacy.io/universe/project/saber

• Text visualization https://github.com/JasonKessler/scattertext

• FastText (facebook)

• Visualize relations: https://allenai.github.io/hierplane/#getstarted

• T-SNe https://www.datacamp.com/community/tutorials/introduction-t-sne

• Context: Medical Subject Headings (MeSH) terms are added through an algorithm to every file on clinicaltrials.gov.
• Test ways to extract this information, with n numbers of fields, using notebook and limited number of random files.
• Why: key information to map diseases and networks.

(check paper)

https://dzone.com/articles/kibana-tutorial-getting-started-part-1

https://stackoverflow.com/questions/40149767/d3-js-add-label-to-bar-chart

CancerMine data:
https://www.biorxiv.org/content/10.1101/364406v1.full
http://bionlp.bcgsc.ca/cancermine/
https://zenodo.org/record/3265657

Branch: column_index

App.py line 91:
df_timeline = df_timeline.set_index('year_submitted')
instead of: df_timeline = df_timeline.set_index('year_submitted').reset_index()

JS data variables in Result.html line 103...

Digital Ocean model:
https://www.digitalocean.com/community/tutorials/how-to-build-and-deploy-a-flask-application-using-docker-on-ubuntu-18-04

Related info:
https://dev.to/riverfount/dockerize-a-flask-app-17ag

• Add progress bar
• Add main function at the end
• Add dialogue/clear feedback
• Revise tags parsed [add more info on condition and sponsors and centers involved]

• Fix tooltips to show intended data (not record position or "undefined)
  Info: transforming data into arrays to be used in d3 functions (http://www.jeromecukier.net/2012/05/28/manipulating-data-like-a-boss-with-d3/)

Current state:
Issue might be: df['verification_year'] = df['verification_date'].dt.year, in notebook script 3

Check cells under ## Change date types, extract years

Background:

• When creating the DB, a trial 'recruiting' and updated within the last 3 years is labeled as "recruiting". If not, is considered as not recruiting.

• Conflicting results with sample db show trials older than 3 years old labeled as recruiting. Numbers don't match between timeline and phase dataset.

• Check results in full database and review process if needed.

Related: #68

Related to #51

Test queries
Create indexes
Connect it to Flask App

https://realpython.com/flask-by-example-part-2-postgres-sqlalchemy-and-alembic/

Options:

https://github.com/gijswobben/pymed/blob/master/examples/simple_search/main.py
https://github.com/jakelever/pubrunner
https://titipata.github.io/pubmed_parser/
https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Download Pubmed files
ftp://ftp.ncbi.nlm.nih.gov/pubmed/

• Transformations: prepare dataset for analysis
• Pipeline: series of analysis

Check issue #46

Going fast with sqlite and python
https://charlesleifer.com/blog/going-fast-with-sqlite-and-python/

Full text search
https://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/

Flask extension:
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-x-full-text-search/page/2
https://pythonhosted.org/Flask-WhooshAlchemy/

https://www.coursera.org/specializations/tensorflow-in-practice

Add progress bar.
Check structure: main function at the end

Transformations
Standarized transformations

ML classification
Test several models for unsupervised classification using conditions labels and keywords?

Extract value from data
Pending

• Quick review of scripts and flow

• Current problem: data is updated for timeline when checking 'Recruiting' radio button, and data is updated, but color is not. Check if that's the case for phase too.

• Buttons filter data for timeline

• Buttons filter data for phase chart

• Colors updated accordingly to dataset

Example: https://bl.ocks.org/patiencehaggin/ffc6522a844cea9f0712282ba6d3cbeb

Stack overflow https://stackoverflow.com/questions/23227991/how-to-add-in-zero-values-into-a-time-series-in-d3-js-javascript

Test it with a selection of trials (see use cases #43 )

First: implement TF-IDF similarity
https://www.datacamp.com/community/tutorials/recommender-systems-python

https://www.scikit-yb.org/en/latest/api/text/tsne.html

Use KNN to classify search results in groups, using several indicators (ie, tf-id score, phase, year, citations?, subsequent studios with same component?)

An introduction to clustering: https://towardsdatascience.com/an-introduction-to-clustering-algorithms-in-python-123438574097

Analyzing time series
https://towardsdatascience.com/analyzing-time-series-data-in-pandas-be3887fdd621

Backend returns two json files:

• Phase data with all results and recruiting only results
  Reformat data in app.py
  Connect phase chart to new columns

• Timeline data with all results and recruiting only results
  Reformat data in app.py
  Connect timeline chart to new columns

• Change datatypes and optimize memory before exporting cleaned_data as csv (code in 4th notebook).
• Check memory usage in cleaned dataframe before exporting to csv
• Check memory usage in cleaned dataframe once imported in ML Pipeline.

Flask app with d3 visualizations

References
https://realpython.com/web-development-with-flask-fetching-data-with-requests/

Info: https://aact.ctti-clinicaltrials.org/snapshots

• Docs: https://dash.plot.ly/getting-started
• Tutorial (Datacamp): https://www.datacamp.com/community/tutorials/learn-build-dash-python
• Medium post: https://medium.com/a-r-g-o/using-plotlys-dash-to-deliver-public-sector-decision-support-dashboards-ac863fa829fb

Urls:

• Index
• /result?

https://www.datacamp.com/courses/introduction-to-time-series-analysis-in-python

Connect graphs to new data sources, refactor their structure.

Related issues:
#48
#52

Process for each table:

• Create pre-defined table in postgres.
• Connect flask to postgres.
• Search within table: input> term, return > pre-defined table

Issue #65

Limited to pre-defined use cases

• Dialogue: All files versus Sample. If Sample, define number of files
• Add feedback and progress bar

Filtering will be done in the script 4: ML transformations & pipeline

https://www.statnews.com/2019/07/25/ai-expert-writing-code-save-son/
https://ncats.nih.gov/translator/about
https://www.statnews.com/2019/07/18/clinical-trials-birds-eye-view-drug-development/

• Import JSON file, clean and rename columns
• Add dates columns
• Select data since 2008
• Export data as csv

https://observablehq.com/@d3/d3-group
(years, phases)

Check if done in preprocessing (better) or in back-end. To be used for Trials / Papers explorer?

Library: https://github.com/jakelever/kindred (Spacy Kindred)