Default installation is not working.

$ go get github.com/assetnote/commonspeak2

github.com/assetnote/commonspeak2/command/deletedfiles

go/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:33:17: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:34:16: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:35:14: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:36:14: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
go/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:37:18: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)

github.com/assetnote/commonspeak2/command/routes

go/src/github.com/assetnote/commonspeak2/command/routes/routes.go:37:17: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/routes/routes.go:38:16: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/routes/routes.go:39:14: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/routes/routes.go:40:14: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
go/src/github.com/assetnote/commonspeak2/command/routes/routes.go:41:18: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)

github.com/assetnote/commonspeak2/command/subdomains

go/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:32:17: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:33:16: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:34:14: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
go/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:35:18: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)

github.com/assetnote/commonspeak2/command/wordswithext

go/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:31:17: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:32:16: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
go/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:33:14: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
go/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:34:18: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)

See the data/sql/ folder for examples of how we currently create SQL queries for rails routes.

See the command/ folder for examples of how we modify the routes returned to produce better quality data.

Create SQL queries for NodeJS and Tomcat, create appropriate filtering and substitution methods to improve quality of output.

when i try to generate rails routes wordlists using this command: ./commonspeak2 --project XXXX --credentials XXXXX routes --frameworks rails -l 100000 -o rails-routes.txt, it doesnt return the expected result, all it returns is:

INFO[0000] Generated SQL template for Rails routes. Mode=Routes
INFO[0000] Executing BigQuery SQL... this could take some time. Framework=rails Mode=Routes
id
staff_id
event_id
smart_proxy_id
key
loggable_id
forum_id
topic_id
conversation_id
index
record
account_id
page
invitation_token
start
end
date
project_id
nonprofit_id
user_id
agent_id
discussion_id
feed_uid
commit_id
key_id
paper_uid
group_id
auth_token
category_id
prison_id
recording_id
confirmation_token
comment_id
host_id
parent_slug
back_id
call_id
1
activation_code
days
new_id
issue_id
govtrack_id
step
provider
name
action
User
body
permalink
legislative_term
paper
slug
component_1
component_2
component_3
loggable_type
repo
address
year
month
locale
token
tags
username
archetype
link
check
apiv
query
filename
day
feed
article_name
controller
status
secret
user
code
tab
service
text
Filemanager
currency
model
platform
idp
color
url_token
admin_key
client
subscription
letter
ministry
filter
url_start
url_end
session
application
newsletter
format
category_slug
organization
INFO[0009] Total rows extracted 0. Framework=rails Mode=Routes Source=Github

FATA[0046] Error executing BigQuery SQL. Error="googleapi: Error 400: Unrecognized name: origin at [17:16], invalidQuery" Mode=Subdomains Source=httparchive

Hello
I get this error "FATA[0020] Error executing BigQuery SQL. Error="googleapi: Error 400: Unrecognized name: origin at [17:16], invalidQuery" Mode=Subdomains Source=httparchive" when commonspeak tries to fetch subdomains from httparchive specifically, how can i fix this?

Command: ./commonspeak2 --project XXXXX --credentials XXXXX subdomains -o subdomains.txt

While I love the idea of having premise wordlists, I wonder if keeping them in this same repo might clutter things up/make it too big.

Would it make more sense to have a separate repo (like commonspeak2-wordlists or similar) to keep those in?

We need a GitHub pages setup for the commonspeak2-wordlists repo which contains a list of wordlists in a JSON schema, including a list of direct download locations.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x82f66c]

goroutine 1 [running]:
github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery.(*JobIDConfig).createJobRef(0xc4200d7d40, 0x0, 0x0)
/go/src/github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery/job.go:154 +0x3c
github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery.(*Query).newJob(0xc4200d7d40, 0x9f0668, 0xa34e20, 0xc4201ee990)
/go/src/github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery/query.go:294 +0x6c
github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery.(*Query).Run(0xc4200d7d40, 0xa34e20, 0xc4201ee990, 0x0, 0x0, 0x0)
/go/src/github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery/query.go:277 +0xb5
github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery.(*Query).Read(0xc4200d7d40, 0xa34da0, 0xc4200a4050, 0x4, 0xc4200c6280, 0x34)
/go/src/github.com/assetnote/commonspeak2/vendor/cloud.google.com/go/bigquery/query.go:302 +0x43
github.com/assetnote/commonspeak2/command/wordswithext.query(0x0, 0xa34da0, 0xc4200a4050, 0xc4200ca0b0, 0xae, 0x1, 0x1, 0x0)
/go/src/github.com/assetnote/commonspeak2/command/wordswithext/helper.go:32 +0x90
github.com/assetnote/commonspeak2/command/wordswithext.CmdStatus(0xc4200bcf20, 0xc4200b6600, 0xc4200bcf20)
/go/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:88 +0x8c0
github.com/assetnote/commonspeak2/vendor/github.com/urfave/cli.HandleAction(0x8d4120, 0x9f0610, 0xc4200bcf20, 0x0, 0xc4200b6600)
/go/src/github.com/assetnote/commonspeak2/vendor/github.com/urfave/cli/app.go:501 +0xc8
github.com/assetnote/commonspeak2/vendor/github.com/urfave/cli.Command.Run(0x9b9361, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x9d11be, 0x3c, 0x0, ...)
/go/src/github.com/assetnote/commonspeak2/vendor/github.com/urfave/cli/command.go:165 +0x47d
github.com/assetnote/commonspeak2/vendor/github.com/urfave/cli.(*App).Run(0xc42009a1c0, 0xc4200ac000, 0xc, 0xc, 0x0, 0x0)
/go/src/github.com/assetnote/commonspeak2/vendor/github.com/urfave/cli/app.go:259 +0x6e8
main.main()
/go/src/github.com/assetnote/commonspeak2/main.go:36 +0x175

Hello,
While compiling commonspeak2 from source I am getting the following error:

github.com/assetnote/commonspeak2/command/deletedfiles
gotools/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:33:17: c.GlobalBool undefined (type *cli.Context has no fi eld or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:34:16: c.GlobalBool undefined (type *cli.Context has no fi eld or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:35:14: c.GlobalBool undefined (type *cli.Context has no fi eld or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:36:14: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
gotools/src/github.com/assetnote/commonspeak2/command/deletedfiles/deleted.go:37:18: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
github.com/assetnote/commonspeak2/command/routes
gotools/src/github.com/assetnote/commonspeak2/command/routes/routes.go:37:17: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/routes/routes.go:38:16: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/routes/routes.go:39:14: c.GlobalBool undefined (type *cli.Context has no field or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/routes/routes.go:40:14: c.GlobalString undefined (type *cli.Context has no field o r method GlobalString)
gotools/src/github.com/assetnote/commonspeak2/command/routes/routes.go:41:18: c.GlobalString undefined (type *cli.Context has no field o r method GlobalString)
github.com/assetnote/commonspeak2/command/subdomains
gotools/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:32:17: c.GlobalBool undefined (type *cli.Context has no f ield or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:33:16: c.GlobalBool undefined (type *cli.Context has no f ield or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:34:14: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
gotools/src/github.com/assetnote/commonspeak2/command/subdomains/subdomains.go:35:18: c.GlobalString undefined (type *cli.Context has no field or method GlobalString)
github.com/assetnote/commonspeak2/command/wordswithext
gotools/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:31:17: c.GlobalBool undefined (type *cli.Context has no fiel d or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:32:16: c.GlobalBool undefined (type *cli.Context has no fiel d or method GlobalBool)
gotools/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:33:14: c.GlobalString undefined (type *cli.Context has no fi eld or method GlobalString)
gotools/src/github.com/assetnote/commonspeak2/command/wordswithext/words.go:34:18: c.GlobalString undefined (type *cli.Context has no fi eld or method GlobalString)

It looks like some changes has been made on httparchive

INFO[0000] Generated SQL template for HackerNews.        Mode=Subdomains
INFO[0000] Generated SQL template for HTTPArchive.       Mode=Subdomains
INFO[0000] Executing BigQuery SQL... this could take some time.  Mode=Subdomains Source=hackernews

INFO[0022] Total rows extracted 74160.                   Mode=Subdomains Silent=false Source=hackernews Verbose=false
INFO[0022] Executing BigQuery SQL... this could take some time.  Mode=Subdomains Source=httparchive
FATA[0025] Error executing BigQuery SQL.                 Error="googleapi: Error 400: Unrecognized name: origin at [17:16], invalidQuery" Mode=Subdomains Source=httparchive

FATA[0011] Error executing BigQuery SQL. Error="googleapi: Error 403: Access Denied: BigQuery BigQuery: Error getting metadata for external code resource, please verify you have provided a valid path and/or that you have access to the resource: gs://commonspeak-udf/URI.min.js, accessDenied" Mode=WordsWithExt Source=httparchive

./commonspeak2 --project crunchbox-160315 --o subdomains.txt
Incorrect Usage. flag provided but not defined: -o

NAME:
Commonspeak 2 - Generate wordlists using BigQuery by analysing datasets that evolve constantly.

USAGE:
commonspeak2 [global options] command [command options] [arguments...]

VERSION:
0.1.0

AUTHOR:
Assetnote

COMMANDS:
ext-wordlist Generate wordlists based on extensions provided by the user.
subdomains Generates a list of subdomains from all available BigQuery public datasets.
routes Generate wordlists based on routes extracted from popular frameworks.
help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
--project value, -p value The Google Cloud Project to use for the queries.
--credentials value, -c value Refer to: https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-install-go [credentials.json]
--verbose Enable verbose output.
--silent, -s If this is set to true, the results will be written to a file but not to STDOUT.
--test, -t If this is set to true, Commonspeak2 will execute queries against smaller, testing datasets.
--help, -h show help
--version, -v print the version
04:02:00 root kali /root/Desktop/commonspeak2_0.1.4_Linux_x86_64

./commonspeak2 --project crunchbox-160315 --o /root/Desktop/alive.txt

Incorrect Usage. flag provided but not defined: -o

NAME:
Commonspeak 2 - Generate wordlists using BigQuery by analysing datasets that evolve constantly.

USAGE:
commonspeak2 [global options] command [command options] [arguments...]

VERSION:
0.1.0

AUTHOR:
Assetnote

COMMANDS:
ext-wordlist Generate wordlists based on extensions provided by the user.
subdomains Generates a list of subdomains from all available BigQuery public datasets.
routes Generate wordlists based on routes extracted from popular frameworks.
help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
--project value, -p value The Google Cloud Project to use for the queries.
--credentials value, -c value Refer to: https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-install-go [credentials.json]
--verbose Enable verbose output.
--silent, -s If this is set to true, the results will be written to a file but not to STDOUT.
--test, -t If this is set to true, Commonspeak2 will execute queries against smaller, testing datasets.
--help, -h show help
--version, -v print the version

Hi,

I'm getting this error when getting a subdomain list with this command:

commonspeak2 --project crunchbox-160315 --credentials creds.json subdomains -o subdomains.txt

and this as a result:

INFO[0000] Generated SQL template for HackerNews.        Mode=Subdomains
INFO[0000] Generated SQL template for HTTPArchive.       Mode=Subdomains
INFO[0000] Executing BigQuery SQL... this could take some time.  Mode=Subdomains Source=hackernews
FATA[0001] Error executing BigQuery SQL.                 Error="googleapi: Error 403: Access Denied: Project crunchbox-160315: User does not have bigquery.jobs.create permission in project crunchbox-160315., accessDenied" Mode=Subdomains Source=hackernews

How to fix this?