petermr / semanticclimate Goto Github PK

Conversion of IPCC documents into semantic form

License: Apache License 2.0

Python 0.01% Jupyter Notebook 2.95% HTML 97.02% JavaScript 0.01% CSS 0.02%

semanticclimate's Introduction

semanticClimate

Most other subjects have highly heterogeneous data without semantics and this holds back the creation of knowledge. There is a pressing need to make knowledge about climate available to mitigate the effects of gaseous emissions.

IPCC

An important resource is the UN’s IPCC reports, published about every five years. In 2021-2022 AR6, with 10,000 pages, was released. #semanticClimate is a group of young Indian science students who are developing tools and community protocols to make IPCC AR6 semantic.

UNFCCC

The UNFCCC publishers annual reports, mainly based on the UN's COP meetings. We have scraped and analysed most of these from the last >25 years.

Goals

to convert the IPCC documents from PDF into (a) HTML (b) XML
extract terms and explore their use and meaning
link terms to Wikidata and create AMI-dictionaries
create new structiures for navigation, search, display

What #SemanticClimate does

We develop tools to liberate knowledge from locked PDFs and host events everybody gets a chance to explore the content in these reports through our tools. Our Technical Strategy Page gives an overview of the tools.

Upcoming Events

Check out our Events page for details about upcoming hackathons we host and other events we are part of.

How to get involved

We are looking for volunteers/funders to:

run more events
develop the code (open an Issue/PR to get started)
develop the content (start a Discussion thread at https://github.com/petermr/petermr/discussions/)
- dictionaries
- semantified chapters

Events

OpenAccess Week

Hackathon planned for 2022-10-24 to 2022-10-28

see WG3 repository

Discussions

We are using the Github Discussions tool to keep a narrative of our work. Currently we focus on individual chapters of the IPCC/AR^/WG3 report.

WG3/Chapter06: #22
WG3/Chapter08: #21

semanticclimate's People

Contributors

Stargazers

Watchers

Forkers

kaartik7 enakshi-1998 ambrineh pg-pi3gey priti-chahal stroemphi mandeepumra emanuelfaria xenophiliamee egonw rajarshi0 poulomig99 semanticclimate parijat-03

semanticclimate's Issues

pyamihtml documentation is out of date

pyamihtml==0.0.6 has out of date docs for argparse

Examples (# foo is a comment):
  pyamihtml        # runs help
  pyamihtml -h     # runs help
  pyamihtml PDF -h # runs PDF help
  pyamihtml PDF --infile foo.pdf --outdir bar/ # converts PDF to HTML
  pyamihtml PROJECT --project foodir/ # converts all PDF in foodir to CTrees
  pyamihtml IPCC --pdf2html file/ # converts pdf file to html 

----------------------------------------

positional arguments:
  {HTML,IPCC,PDF}  subcommands

optional arguments:
  -h, --help       show this help message and exit
  -v, --version    show version 0.0.6

run:
        pyamihtml <subcommand> <args>
          where subcommand is in   {DICT,GUI,HTML,PDF,PROJECT} and args depend on subcommand

Needs expanding to new commands and editing of subcommands

Collect together OA Week 22 content links

OA Week - https://github.com/petermr/semanticClimate/blob/pmr1/ipcc/ar6/OA_WEEK_2022.md

semanticClimate - https://github.com/petermr/semanticClimate/blob/main/README.md

Instructions CoLab - https://github.com/petermr/semanticClimate/tree/main/outreach/climate_knowledge_hunt_hackathon/Hackathon_Notebook

Video about hakathon - Post-Hackthon Pitch to SDG Accelerator (12 min.): https://youtu.be/0MGXic-uEQY

Other events page - https://github.com/petermr/semanticClimate/blob/main/events.md

how to improve PDF to HTML conversion

Currently the Semantic Climate project converts PDFs to HTML.

The content is the IPPC Climate report AR6 and we need to improve is markup for further semantic annotation, resuse, and presentation. From a typesetting perspective and freeing us from descructive reliance on PDF (note we can get PDF like results in a non-descructive way using Vivliostyle) - that's me @mrchristian I would like to produce HTMl that could be rendered in Vivliostyle better than this.

The output needs improvement. Currently it contained a number of elements which may not be needed, e.g., page numbers, inline styles, etc.

The objective would be to improve the output with tooling that can integrate with the current workflow.

The suggestion would be to create a way to evaluate the process by collating information on the issue:

Current tooling
Condition of the source PDFs
Problems with outputs
List of parts and markup that we need to retain their integrity
Define what we want in out target outputs
Do we want other output formats for richer markup and other interoperability
List and evaluate tools
Consult experts in the field: pandoc, le-tex, fidus, vivlio, css-rocks, etc

This research can be conducted in a wiki page on the Semantic Climate repository.

Here are sample files:

PDF source - Chapter 8 https://github.com/petermr/semanticClimate/blob/main/ipcc/ar6/wg3/Chapter08/fulltext.pdf

HTML full text - Chapter 8 https://github.com/petermr/semanticClimate/blob/main/ipcc/ar6/wg3/Chapter08/fulltext.html

Tasks

Link to current PDF to HTML tooling.
Consult Single Source Publishing Community https://github.com/singlesourcepub/community/discussions and others: le-tex, pandoc, css rocks?

Things to do

Creating CTree structure for Chapters

OA Week TDL - To do list (Just some suggestions)

Guide for the current workflow
Guide for making a dictionary
List IPCC Report chapters and have a way to organise volunteers
Invitation and instructions for Chapter Champions
Invitation and instructions for volunteers
Coordinate on recruitment
Organisers test workflow and give feedback
Translation of key parts of reports into non-EN language
Translation of instructions (notebooks, these pages)
Setup documentation and knowledge management process an infra
Form organising team and assign roles, and tasks, infra