Giter VIP home page Giter VIP logo

dsfsi / gov-za-multilingual Goto Github PK

View Code? Open in Web Editor NEW
4.0 3.0 0.0 1.35 GB

The data set contains cabinet statements from the South African government. Data was scraped from the governments website: https://www.gov.za/cabinet-statements

Home Page: https://doi.org/10.5281/zenodo.7635167

License: MIT License

Makefile 1.02% Python 27.57% Jupyter Notebook 68.73% Shell 2.68%
africannlp machine-learning nlproc south-africa african-language-data-liberation-front africanlp corpus dataset language dsfsi-datasets

gov-za-multilingual's Introduction

The South African Gov-ZA multilingual corpus

Automated Sentence Alignment of Cabinet Speeches CI

Github: https://github.com/dsfsi/gov-za-multilingual

Zenodo: DOI

Arxiv Preprint: arXiv

View main dataset on Datasette

Give Feedback 📑: DSFSI Resource Feedback Form{:target="_blank"}

About Dataset

The data set contains cabinet statements from the South African government, maintained by the Government Communication and Information System (GCIS). Data was scraped from the governments website: https://www.gov.za/cabinet-statements

The datasets contain government cabinet statements in 11 languages, namely:

Language Code Language Code
English (eng) Sepedi (nso)
Afrikaans (afr) Setswana (tsn)
isiNdebele (nbl) Siswati (ssw)
isiXhosa (xho) Tshivenda (ven)
isiZulu (zul) Xitstonga (tso)
Sesotho (sot)

The dataset contains the full data in a JSON file (/data/govza-cabinet-statements.json), as well as CSV’s split by each language, eg: “govza-cabinet-statements-en.csv” for english. The dataset does not contain special characters like unicode or ascii.

Please see the data-statement.md for full dataset information. (TODO)

Number of Aligned Pairs with Cosine Similarity Score >= 0.65

src_lang trg_lang num_aligned_pairs
nbl ven 18984
nso ssw 18697
zul ssw 18563
xho ssw 18387
xho zul 18145
xho nso 18110
xho tso 17954
ssw tso 17880
zul tso 17789
zul nso 17630
nso tso 17617
tsn tso 16681
xho tsn 16571
xho eng 16537
zul tsn 16482
tsn ssw 16386
nso tsn 16179
nbl sot 16163
zul eng 16149
tso eng 16068
afr xho 16065
ssw eng 15721
afr ssw 15610
afr nso 15388
nso eng 15257
afr zul 14998
afr tso 14936
afr eng 14549
tsn eng 14544
sot ven 14098
afr tsn 12605
afr sot 8834
sot nso 8077
xho sot 7489
afr nbl 6621
sot tso 6586
nso ven 6367
nbl nso 6342
zul sot 5975
sot ssw 5811
afr ven 5776
sot tsn 5450
nbl xho 5213
sot eng 5212
nbl ssw 4655
ssw ven 4588
ven tso 4578
xho ven 4559
nbl tso 4465
nbl zul 3868
ven eng 3670
nbl eng 3616
zul ven 3606
nbl tsn 3369
tsn ven 3267

Disclaimer

This dataset contains machine-readable data extracted from online cabinet statements from the South African government, provided by the Government Communication Information System (GCIS). While efforts were made to ensure the accuracy and completeness of this data, there may be errors or discrepancies between the original publications and this dataset. No warranties, guarantees or representations are given in relation to the information contained in the dataset. The members of the Data Science for Societal Impact Research Group bear no responsibility and/or liability for any such errors or discrepancies in this dataset. The Government Communication Information System (GCIS) bears no responsibility and/or liability for any such errors or discrepancies in this dataset. It is recommended that users verify all information contained herein before making decisions based upon this information.

Authors

  • Vukosi Marivate - @vukosi
  • Matimba Shingange
  • Richard Lastrucci
  • Isheanesu Joseph Dzingirai
  • Jenalea Rajab

Citation

Paper

Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora

@inproceedings{lastrucci-etal-2023-preparing, title = "Preparing the Vuk{'}uzenzele and {ZA}-gov-multilingual {S}outh {A}frican multilingual corpora", author = "Richard Lastrucci and Isheanesu Dzingirai and Jenalea Rajab and Andani Madodonga and Matimba Shingange and Daniel Njini and Vukosi Marivate", booktitle = "Proceedings of the Fourth workshop on Resources for African Indigenous Languages (RAIL 2023)", month = may, year = "2023", address = "Dubrovnik, Croatia", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.rail-1.3", pages = "18--25" }

Dataset

Vukosi Marivate, Matimba Shingange, Richard Lastrucci, Isheanesu Joseph Dzingirai, Jenalea Rajab. The South African Gov-ZA multilingual corpus, 2022

@dataset{marivate_vukosi_2023_7635168, author = {Marivate, Vukosi and Shingange, Matimba and Lastrucci, Richard and Dzingirai, Isheanesu and Rajab, Jenalea}, title = {The South African Gov-ZA multilingual corpus}, month = feb, year = 2023, publisher = {Zenodo}, version = {1.0}, doi = {10.5281/zenodo.7635168}, url = {https://doi.org/10.5281/zenodo.7635168} }

Licences

gov-za-multilingual's People

Contributors

actions-user avatar idzingirai avatar lastrucci01 avatar matimbarose avatar peaceaz avatar vukosim avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

gov-za-multilingual's Issues

Sentence Alignment Action crashes

From the output of the action:
image

I have a hunch in a recent update of panda's they deprecated the append function.
I think it must be pd.concat() now.

Fix broken Action: Fetch cabinet statement

Description: Github action Fetch cabinet statement failing

Traceback (most recent call last):
  File "main.py", line 6, in <module>
    cabinent_statements = helper_functions.get_cabinent_statements_urls(date) #read cabinet statements until date
  File "/home/runner/work/gov-za-multilingual/gov-za-multilingual/src/gov_cab_statements_scrape/helper_functions.py", line 60, in get_cabinent_statements_urls
    statement['title'] = row.find('td').find('a').text
AttributeError: 'int' object has no attribute 'find

Looks like a simple fix, maybe they changed something in the website structure

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.