A fake news detector developed as part of the Leaders Prize: Fact or Fake News? competition and SFU Special Research Project (CMPT 415) course. The goal was to implement an AI algorithm that correctly labels a given claim as true, partly true or false.

The program takes as input a metadata file with information about a series of claims and an empty label field for each claim. The output is a predictions file with a rating of true, partly true or false in the label field for each claim.

The dataset consists of claims and associated metadata downloaded from 9 fact-checking websites: politifact.com, snopes.com, washingtonpost.com, weeklystandard.com, africacheck.org, factscan.ca, factcheck.afp.com, polygraph.info, factcheck.org.

An example of a claim is shown below:

"claim": "\"We have $91 billion going to Puerto Rico. We have $29 billion to Texas and $12 billion to Florida for the hurricanes.\"",
"claimant": "Donald Trump",
"date": "2019-03-28", 
"label": 1, 
"related_articles": [60922, 41290, 28742],
"id": 123

where claim is the statement to be evaluated on truthfulness, claimant is the entity who made the claim, date is when the claim was made, label is the truth rating of the claim (0: false, 1: partly true, 2: true), related_articles is a list of article IDs that point to the names of text files containing the articles, and id is the unique identifier for each claim.

Submission scores were calculated using the macro average F1 score of the outputted truth ratings. The formula is defined as follows:

where P is precision and R is recall, defined as:

The precision and recall for each class are defined as:

where TP is the number of true positives, FP is the number of false positives, and FN is the number of false negatives.