didasy / tldr Goto Github PK

View Code? Open in Web Editor NEW

124.0 6.0 20.0 66 KB

Text summarizer for golang using LexRank

License: MIT License

Go 100.00%

nlp golang go lexrank tldr summarizer summarization pagerank

tldr's Introduction

tldr

When you are too lazy to read the entire text

What?

tldr is a golang package to summarize a text automatically using lexrank algorithm.

How?

There are two main steps in lexrank, weighing, and ranking. tldr have two weighing and two ranking algorithm included, they are Jaccard coeficient and Hamming distance, then PageRank and centrality, respectively. The default settings use Hamming distance and pagerank.

Is This Fast?

Test it yourself, my system is [email protected] with single channel 4GB RAM using Ubuntu 15.10 with kernel 4.5.0

$ go test -bench . -benchmem -benchtime 5s -cpu 4
BenchmarkSummarizeCentralityHamming-4	    2000	   6429340 ns/op	  401204 B/op	    3551 allocs/op
BenchmarkSummarizeCentralityJaccard-4	     200	  30036357 ns/op	 3449461 B/op	   12543 allocs/op
BenchmarkSummarizePagerankHamming-4  	    1000	   7015008 ns/op	  420665 B/op	    3731 allocs/op
BenchmarkSummarizePagerankJaccard-4  	     200	  31066764 ns/op	 3469629 B/op	   12737 allocs/op

So, not bad huh?

Installation

go get github.com/didasy/tldr

Example

package main

import (
	"fmt"
	"io/ioutil"
	"github.com/didasy/tldr"
)

func main() {
	intoSentences := 3
	textB, _ := ioutil.ReadFile("./sample.txt")
	text := string(textB)
	bag := tldr.New()
	result, _ := bag.Summarize(text, intoSentences)
	fmt.Println(result)
}

Testing

To test, just run go test, but you need to have gomega and ginkgo installed.

Dependencies?

tldr depends on pagerank package, and you can install it with go get github.com/alixaxel/pagerank.

License?

Check the LICENSE file. tldr: MIT.

Have fun!

tldr's People

Contributors

Stargazers

Watchers

tldr's Issues

Rank is empty on text with repeated sentences

Try to summarize next text
bag.Summarize("Hello, world. Hello, world. Hello, world.", 1)
and got error
Ranks is empty

I think the summarize function need return Hello, world.
Why this happens?

multiple-value bag.Summarize() in single-value context

I ran into this issue when I used your code provided in README.md

package main

import (
	"fmt"
	"io/ioutil"
	"github.com/JesusIslam/tldr"
)

func main() {
	intoSentences := 3
	textB, _ := ioutil.ReadFile("./sample.txt")
	text := string(textB)
	bag := tldr.New()
	result := bag.Summarize(text, intoSentences)
	fmt.Println(result)
}

But in your code Summarize function returns two values, i.e. result and error.
For me modifying the line result := bag.Summarize(text, intoSentences) worked. Please check and modify README.md file.

Difference between createSentences and createOriginalSentences

Hi, I have a sentence tokenizer that I want to use instead of the one used in this package, but I'm a little confused as to what I need to replace to get it working properly.

It looks like createOriginalSentences returns []string whereas createSentences returns [][]string.

It'd be nice to have one function to replace the sentence tokenizer so anyone can swap it out.

Unicode support

I tried to use this lib on non-english text and it crashes.

I noticed this:
https://github.com/JesusIslam/tldr/blob/master/tldr.go#L192

Could you change it to use golang's unicode library to determine is it alpha numeric instead?

http://golang.org/pkg/unicode/#IsDigit
http://golang.org/pkg/unicode/#IsLetter

Thanks!

Strange check in summarize function

In function

func (bag *Bag) Summarize(text string, num int) (string, error)

checking number of sentences for output

if num > (len(bag.Ranks)-1) || num < 1 {
	num = 1
}

Why len(Ranks)-1?

May be need:

if num > len(bag.Ranks) || num < 1 {
	num = 1
}

Empty TL;DR Results?

hi @JesusIslam, Not sure if you are still maintaining this library, but as I don't understand the algorithms used I wanted to ask you if you would have any insights as to why on some inputs (from RSS/Atom feeds) the result is an empty string?

My use-case here is to take RSS/Atom feeds, feed into this library to produce 1-2 sentences and use that as a "summary" with a "Read more" link.

Thanks :)

Error on compilation

When running "go install" on the corresponding project, it reports this error:

./main.go:14: multiple-value bag.Summarize() in single-value context

Error to summarizer

it's my code

package main

import (
    "fmt"
    "github.com/JesusIslam/tldr"
    "strings"
)

func main() {
    intoSentences := 3
    words := "loremdkasmsdma skdmlaskldmsla mdaskdlsamskdlasm..............tsk"
    leng := strings.Count(words, ".")
    if leng > 3 {
        bag := tldr.New()
        result := bag.Summarize(words, intoSentences)
        fmt.Println(result)
    }
}

when I run this code I get

panic: runtime error: slice bounds out of range