postmodern / wordlist.rb Goto Github PK

Add support for reading 7zipped wordlists.

7za e -so wordlis.txt.7za wordlist.txt

This could go into another gem or possibly ronin-wordlists or ronin-web. The wordlist gem should stay focused on just building wordlists from text that its given.

Remove the Wordlist::Runners CLI classes and bin script. The wordlist library should be a pure-library. Other CLI functionality could be moved into ronin-wordliss.

Add support for creating 7zipped wordlist archives.

Similar to wordlist operators, we should add wordlist modifier classes and methods.

tr

wordlist.tr('e','3') (Modifier::Translate): similar to String#tr.

sub

wordlist.sub(/.../,'...') (Modifier::Sub): similar to String#sub.

gsub

wordlist.gsub(/.../,'...') (Modifier::GSub): similar to String#gsub.

capitalize

wordlist.capitalize (Modifier::Capitalize): similar to String#capitalize.

upcase

worlist.upcase (Modifier::Upcase): similar to String#upcase.

downcase

wordlist.downcase (Modifier::Downcase): similar to String#downcase.

Add support for enumerating over .gz, .bz2, xz, and maybe even .zip wordlists. Use the zcat, bzcat, xzcat commands to read each line from the wordlist; as they are technically faster than doing it in Ruby and does not add additional dependencies.

The class should be named TXT to indicate that it reads plain-text wordlist files.

Wordlist::Lexer uses \s in it's regexs which cannot match unicode spaces, such as unicode character 160. Use [[:space:]] instead.

After 1.0.0 has been released, port to Crystal.

Add operator methods and Operator classes for manipulating wordlists.

Concat

wordlist1 + wordlist2 (Operator::Concat): enumerates over wordlist1, then enumerates over wordlist2.

Product

wordlist1 * wordlist2 (Operator::Product): enumerates over every combination of words from wordlist1 + words from wordlist2

Power

(wordslist1 * wordlist2) ** 3 (Operator::Power): only available on Operator::Product, simply repeats the wordlist product multiple times. Equivalent to wordlist1 * wordlist2 * wordlist2 * wordlist2.

Union

wordlist1 | wordlist2 (Operator::Union): enumerate over each unique word in both wordlists.

Intersect

wordlist1 & wordlist2 (Operator::Intersect): enumerate over each word that exists in both wordlists.

Refactor Wordlist::List to accept an Enumerable list of words. This will allow us to convert Arrays or Ranges of Strings into a wordlist, which has the usual wordlist operator/modifier methods.

user_names_wordlist * Wordlist::List.new("000".."999")

Add a top-level Wordlist.open method which determines the type of file by File.extname and picks the appropriate list class to read the wordlist file (ex: GZip vs TXT).

Release it!

Switch to using require_relative to load other wordlist/ files to improve load-times.

Add support for building a .txt wordlist and then run zip on it; because zip does not support streaming input in to an arbitrary file name inside of the zip archive.

Add an option that only allows lowercase words.

Add a --max-length option to filter words by max length.

Instead of:

$ zcat foo.txt.gz 

it needs to be run as:

$ zcat < foo.txt.gz 

The -P option does not combine the wordlist's own words as specified.

$ cat test.txt 
hello
world
iam
testing
aprogramm
but
ialso
love
ruby
that are
just
some
stupid
words
$ wordlist test.txt -P 3
hello
world
iam
testing
aprogramm
but
ialso
love
ruby
that are
just
some
stupid
words

$ wordlist --version
wordlist 1.0.1

Fixed Lexer's regexs to ignore anything that isn't a ASCII or unicode letter (aka \p{L}).

Add a --min-length option to filter words by minimum length.

Fix the regexes in Lexer to handle words containing unicode characters (ex: umlaut, accents, etc).

With the following code:

require 'wordlist/builders/website'
ws = Wordlist::Builders::Website.build( file, {
   :host => 'ronin-ruby.github.com'
})

The following error results:

<path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/session_cache.rb:78:in `[]': undefined method `[]' for nil:NilClass (NoMethodError)
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/agent.rb:683:in `prepare_request'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/agent.rb:512:in `get_page'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/agent.rb:578:in `visit_page'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/agent.rb:249:in `run'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/agent.rb:231:in `start_at'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/agent.rb:184:in `site'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/agent.rb:204:in `host'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/spidr-0.4.1/lib/spidr/spidr.rb:89:in `host'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/wordlist-0.1.0/lib/wordlist/builders/website.rb:240:in `build!'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/wordlist-0.1.0/lib/wordlist/builder.rb:83:in `block in build'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/wordlist-0.1.0/lib/wordlist/builder.rb:55:in `call'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/wordlist-0.1.0/lib/wordlist/builder.rb:55:in `initialize'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/wordlist-0.1.0/lib/wordlist/builders/website.rb:207:in `initialize'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/wordlist-0.1.0/lib/wordlist/builder.rb:81:in `new'
    from <path>/.rvm/gems/ruby-1.9.3-p194/gems/wordlist-0.1.0/lib/wordlist/builder.rb:81:in `build'

I spent some time trying to trace the issue here but could not figure it out. I also assume that this lib is deprecated since the wordlist code migrated over to ronin-support, but the builders class is still really nice. Is this a user error?

Add support for reading zipped wordlists (ex: unzip -p file.txt.zip file.txt).

Parsers should become a proper String Tokenizer. It should be capable of scrubbing input text of unwanted punctuation, then splitting the input text into individual words.

Add a Wordlist::List.[] method for initializing a list with literal words:

Wordlist::List["foo", "bar", "baz"]