michigan-com / summarizer Goto Github PK

View Code? Open in Web Editor NEW

This project forked from indigoresearch/textteaser

31.0 31.0 4.0 353 KB

Python text summarizer

Home Page: https://pypi.python.org/pypi/summarizer

License: MIT License

Python 100.00%

summarizer's People

Contributors

Stargazers

Watchers

Forkers

prabhatcs krachid sachinkalsi adriangarciadiaz

summarizer's Issues

How to install macOSX Monterey M1 chip?

 pip install bert-extractive-summarizer
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting bert-extractive-summarizer
  Using cached bert_extractive_summarizer-0.9.0-py3-none-any.whl (24 kB)
Collecting transformers
  Using cached transformers-4.12.5-py3-none-any.whl (3.1 MB)
Requirement already satisfied: spacy in /opt/homebrew/lib/python3.9/site-packages (from bert-extractive-summarizer) (3.2.0)
Collecting scikit-learn
  Using cached scikit_learn-1.0.1-cp39-cp39-macosx_11_0_arm64.whl
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-3.0.0-py3-none-any.whl (14 kB)
Requirement already satisfied: numpy>=1.14.6 in /opt/homebrew/lib/python3.9/site-packages (from scikit-learn->bert-extractive-summarizer) (1.21.4)
Requirement already satisfied: joblib>=0.11 in /opt/homebrew/lib/python3.9/site-packages (from scikit-learn->bert-extractive-summarizer) (1.1.0)
Requirement already satisfied: scipy>=1.1.0 in /opt/homebrew/lib/python3.9/site-packages (from scikit-learn->bert-extractive-summarizer) (1.7.3)
Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (0.8.2)
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (59.4.0)
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (1.0.1)
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (3.3.0)
Requirement already satisfied: jinja2 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (3.0.3)
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (2.0.6)
Requirement already satisfied: blis<0.8.0,>=0.4.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (0.7.5)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (4.62.3)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (3.0.6)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (2.0.6)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (2.26.0)
Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (2.4.2)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (1.0.6)
Requirement already satisfied: typer<0.5.0,>=0.3.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (0.4.0)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (3.0.8)
Requirement already satisfied: pathy>=0.3.5 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (0.6.1)
Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (8.0.13)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (1.8.2)
Requirement already satisfied: packaging>=20.0 in /opt/homebrew/lib/python3.9/site-packages (from spacy->bert-extractive-summarizer) (21.3)
Requirement already satisfied: regex!=2019.12.17 in /opt/homebrew/lib/python3.9/site-packages (from transformers->bert-extractive-summarizer) (2021.11.10)
Collecting sacremoses
  Using cached sacremoses-0.0.46-py3-none-any.whl (895 kB)
Requirement already satisfied: pyyaml>=5.1 in /opt/homebrew/lib/python3.9/site-packages (from transformers->bert-extractive-summarizer) (6.0)
Collecting huggingface-hub<1.0,>=0.1.0
  Using cached huggingface_hub-0.2.1-py3-none-any.whl (61 kB)
Collecting tokenizers<0.11,>=0.10.1
  Using cached tokenizers-0.10.3.tar.gz (212 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: filelock in /opt/homebrew/lib/python3.9/site-packages (from transformers->bert-extractive-summarizer) (3.4.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/homebrew/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers->bert-extractive-summarizer) (4.0.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/homebrew/lib/python3.9/site-packages (from packaging>=20.0->spacy->bert-extractive-summarizer) (3.0.6)
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /opt/homebrew/lib/python3.9/site-packages (from pathy>=0.3.5->spacy->bert-extractive-summarizer) (5.2.1)
Requirement already satisfied: idna<4,>=2.5 in /opt/homebrew/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (2.10)
Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/homebrew/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (2.0.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/homebrew/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (1.26.7)
Requirement already satisfied: certifi>=2017.4.17 in /opt/homebrew/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (2021.10.8)
Requirement already satisfied: click<9.0.0,>=7.1.1 in /opt/homebrew/lib/python3.9/site-packages (from typer<0.5.0,>=0.3.0->spacy->bert-extractive-summarizer) (8.0.3)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/homebrew/lib/python3.9/site-packages (from jinja2->spacy->bert-extractive-summarizer) (2.0.1)
Requirement already satisfied: six in /opt/homebrew/lib/python3.9/site-packages (from sacremoses->transformers->bert-extractive-summarizer) (1.16.0)
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/ns/x4n6sdd92zb6j9xm0dvr6yzm0000gn/T/tmp99lqhsje
       cwd: /private/var/folders/ns/x4n6sdd92zb6j9xm0dvr6yzm0000gn/T/pip-install-zec9hksx/tokenizers_08b1c60d38e84b40968212eb6158fa87
  Complete output (51 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-11-arm64-3.9
  creating build/lib.macosx-11-arm64-3.9/tokenizers
  copying py_src/tokenizers/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers
  creating build/lib.macosx-11-arm64-3.9/tokenizers/models
  copying py_src/tokenizers/models/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/models
  creating build/lib.macosx-11-arm64-3.9/tokenizers/decoders
  copying py_src/tokenizers/decoders/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/decoders
  creating build/lib.macosx-11-arm64-3.9/tokenizers/normalizers
  copying py_src/tokenizers/normalizers/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/normalizers
  creating build/lib.macosx-11-arm64-3.9/tokenizers/pre_tokenizers
  copying py_src/tokenizers/pre_tokenizers/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/pre_tokenizers
  creating build/lib.macosx-11-arm64-3.9/tokenizers/processors
  copying py_src/tokenizers/processors/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/processors
  creating build/lib.macosx-11-arm64-3.9/tokenizers/trainers
  copying py_src/tokenizers/trainers/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/trainers
  creating build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  copying py_src/tokenizers/implementations/byte_level_bpe.py -> build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  copying py_src/tokenizers/implementations/sentencepiece_unigram.py -> build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  copying py_src/tokenizers/implementations/sentencepiece_bpe.py -> build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  copying py_src/tokenizers/implementations/base_tokenizer.py -> build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  copying py_src/tokenizers/implementations/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  copying py_src/tokenizers/implementations/char_level_bpe.py -> build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  copying py_src/tokenizers/implementations/bert_wordpiece.py -> build/lib.macosx-11-arm64-3.9/tokenizers/implementations
  creating build/lib.macosx-11-arm64-3.9/tokenizers/tools
  copying py_src/tokenizers/tools/__init__.py -> build/lib.macosx-11-arm64-3.9/tokenizers/tools
  copying py_src/tokenizers/tools/visualizer.py -> build/lib.macosx-11-arm64-3.9/tokenizers/tools
  copying py_src/tokenizers/__init__.pyi -> build/lib.macosx-11-arm64-3.9/tokenizers
  copying py_src/tokenizers/models/__init__.pyi -> build/lib.macosx-11-arm64-3.9/tokenizers/models
  copying py_src/tokenizers/decoders/__init__.pyi -> build/lib.macosx-11-arm64-3.9/tokenizers/decoders
  copying py_src/tokenizers/normalizers/__init__.pyi -> build/lib.macosx-11-arm64-3.9/tokenizers/normalizers
  copying py_src/tokenizers/pre_tokenizers/__init__.pyi -> build/lib.macosx-11-arm64-3.9/tokenizers/pre_tokenizers
  copying py_src/tokenizers/processors/__init__.pyi -> build/lib.macosx-11-arm64-3.9/tokenizers/processors
  copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.macosx-11-arm64-3.9/tokenizers/trainers
  copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.macosx-11-arm64-3.9/tokenizers/tools
  running build_ext
  running build_rust
  error: can't find Rust compiler
  
  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
  
  To update pip, run:
  
      pip install --upgrade pip
  
  and then retry package installation.
  
  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Can I know Reference? You have used

This Summarizer is Good and gives Fast Result. Can i get reference papers for this Work ? ( or this work on Page rank algorithm)?

Thanks in Advance.

Scale summary length

Scale summary length based on article length

Abbrev following a sentence break

http://brevity.detroitnow.io/?articleId=73880126

They will be honored in November in Washington. U.S. Secretary of Education Arne Duncan, in a videotaped message to the schools, said the award recognizes “your students’ accomplishments and the hard work and dedication that went into their success.”

http://hometownlife.com/story/news/local/2015/10/13/school-brief/73880126/

Abbreviation ending in a sentence with a smart quote

Ford also has said it will move production of the Ford Focus and Ford C-Max compact cars outside the U.S. “If the UAW represented a greater share of the U.S. auto industry, it would help them gain greater leverage at the bargaining table, but those gains would still be tempered by the global automotive competitive reality,” Dziczek said.

Wow this is really good

This is not an issue, but an accolade. Compared to the painstaking debugging and lackluster results of pytextrank, this 1) just worked 2) with no debugging 3) from a standard text file 4) and produced a superior summary that 5) was scalable on the number of sentences and 6) very fast by comparison.

Thanks for making this available! Cheers!

Single smart quote screwing up tokenizer

http://brevity.detroitnow.io/?articleId=75996714

http://www.detroitnews.com/story/sports/college/university-michigan/2015/11/18/baxter-end-um-msu-game-fluke-flukes/75996714/

And he says, ‘Oh, I’m going to get to?’ And I said, ‘Of course you are.’ And he says, ‘Well, good, that’s why I came to Michigan.

Custom abbreviation following a sentence break causes error

http://brevity.detroitnow.io/?articleId=75983520

The person spoke on condition of anonymity because the team hasn't announced the firing. J.B. Bickerstaff will take over on an interim basis and Chris Finch will be the associate head coach.

PyPI Classifiers Not Working?

Hi! I can see that you have a good list of Classifiers for PyPI including all the good stuff, like version compatibility and license. But, that stuff doesn't appear on PyPI. If there's been a push to PyPI since those classifiers were added, perhaps it's failing because they're Byte-strings instead of strings? Just a guess.

Thanks!

No., Gov., Sgt. abbreviations

Add these to our abbrev_types list

2 if statements in _sbs function must included in the for loop.

Hi,

In file summarizer\summarizer.py, Line 77 and 80
function _sbs:
These two statements have to included in the Line 73 for-loop.

The score is not being computed properly.

Thanks, Bhargava

Period is getting removed during tokenization?

https://api.michigan.com/v1/article/75995706/

Goaltender Petr Mrazek turned aside 15 Ovechkin shots the last time the teams played (Nov. 10), a 1-0 Red Wings victory.\nThere’s not much of a plan or strategy, said Mrazek, other than being ready for the challenge.

turns into:

Goaltender Petr Mrazek turned aside 15 Ovechkin shots the last time the teams played (Nov. 10), a 1-0 Red Wings victory There’s not much of a plan or strategy, said Mrazek, other than being ready for the challenge.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.