Retrieval of json files about xai-scholar HOT 3 CLOSED

Hi :) thank you for your interest.

The jsons are retrieved using the semanticscholar python package in a 5-step process described in the blogpost and paper. This process included manual steps (i.e., I looked at papers by myself and curated them), so I can't give you a code snippet that does everything automatically.

Step 1 and step 4 were the automatic steps, and are very simple to do in a few lines of code. The process for the retrieval in these steps is as follows:

Step 1: Search semanticscholar for every keyword, then verify if the title or abstract contains at least two keywords.
Step 4: Go over your list of papers, and for every paper they cite or reference, retrieve its json file and verify if the title or abstract contains at least two keywords.

I uploaded a notebook that contains the code for both steps in commit ebb3c59 . The file is Retrieval.ipynb. Hope that helps!

@Dipankar1997161

from xai-scholar.

@alonjacovi thanks for the response, I was actually going through the Semantic scholar Api and now checked the code you gave.
There is an argument in the API as "year", you didn't use it in your retrieval and you got till 2022 and 14 papers from 2023.

Even if we don't use it we get all papers till date?

from xai-scholar.

@Dipankar1997161 I executed my retrieval in December 2022. Some papers have the year 2023 because SemanticScholar recorded their year of publication before they were "officially" published. If you retrieve papers from the API, you should get all papers that match your query regardless of year. But this is related to the SemanticScholar API that I have no control over, so any further details you may have to ask them :)

Closing this issue for now as it seems the question has been answered, feel free to reopen if necessary

from xai-scholar.