nlnwa / gowarc Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
The index command can be ignored in this issue as there should be another issue that is about unifying the logic for index and serve indexing.
Is your feature request related to a problem? Please describe.
Our current testdata set has a warc file with version 0.17 which is not supported in the refactor
Allow end user to specify regex (or other format?) to ignore certain files when watching directories
A heisenbug which causes gowarc to panic at db AddBatch()
Currently unknown, but indexing many warcs increase the odds
All warcs are either indexed or some report errors i.e invalid file content
Some warcs causes gowarc to panic at random
Panic stack trace
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa0a59c]
goroutine 23 [running]:
github.com/nlnwa/gowarc/pkg/index.(*Db).AddBatch.func1(0xc517e22980, 0x450001, 0xc517e22980)
/build/pkg/index/db.go:253 +0x8c
github.com/dgraph-io/badger/v2.(*DB).Update(0xc0001ce800, 0xc78fa4bf50, 0x0, 0x0)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/txn.go:696 +0x94
github.com/nlnwa/gowarc/pkg/index.(*Db).AddBatch(0xc0001d7ab0, 0xc4fd83e000, 0x2711, 0x2711)
/build/pkg/index/db.go:251 +0x100
github.com/nlnwa/gowarc/pkg/index.NewIndexDb.func1(0xc0001d7ab0, 0xc0000e8e40)
/build/pkg/index/db.go:79 +0x71
created by github.com/nlnwa/gowarc/pkg/index.NewIndexDb
/build/pkg/index/db.go:77 +0x31a
Currently there is lacking documentation of what the config can do. It would help to describe exactly this in the readme for now.
Running warc index ./testdata/IAH-20080430204825-00000-blackbook.warc
results in an runtime error where the program tries to dereference invalid memory.
./warc index ./testdata/IAH-20080430204825-00000-blackbook.warc
in rootdump:
Using config file: /home/aksel/Projects/gowarc/config.yaml
Format: { <nil>}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xa124b5]
goroutine 1 [running]:
github.com/nlnwa/gowarc/cmd/warc/cmd/index.runE(0xc0000d4dc0, 0x0, 0x0)
/home/aksel/Projects/gowarc/cmd/warc/cmd/index/index.go:88 +0xd5
github.com/nlnwa/gowarc/cmd/warc/cmd/index.NewCommand.func1(0xc00012f900, 0xc000122060, 0x1, 0x1, 0x0, 0x0)
/home/aksel/Projects/gowarc/cmd/warc/cmd/index/index.go:77 +0x6f
github.com/spf13/cobra.(*Command).execute(0xc00012f900, 0xc000122020, 0x1, 0x1, 0xc00012f900, 0xc000122020)
/home/aksel/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00012ef00, 0xc000000180, 0xc00018ff78, 0x411905)
/home/aksel/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x30b
github.com/spf13/cobra.(*Command).Execute(...)
/home/aksel/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
/home/aksel/Projects/gowarc/cmd/warc/main.go:26 +0x2b
logrus allows changing log level with log.SetLevel(*level*)
(see doc).
gowarc should expose this feature through the config file and command argument in root to allow users to change it for each session.
-log-level=warn
it seems like some of the command fields are copy paste and are completely unused.
The following changes should be done:
Describe the bug
Gowarc panics when loading certain files
Additional context
Stack trace on Jira
Create a tagged alpha release so that other go projects can start using gowarc as a dependency without relying on commit hashes
Create unit tests for all cases that can be unit tested in pkg
Since neither README or a wiki page describe the config, the example config should utilize all possible fields in some way.
There is no handling of deleting files. There are currently two issues with this
Preferrably there should also be a command to remove removed indices from the db
Panic log:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xa0f2bd]
goroutine 226 [running]:
github.com/nlnwa/gowarc/pkg/index.indexFile(0xc000176e00, 0xc0003ac090, 0x25)
/home/aksel/Projects/gowarc/pkg/index/indexworker.go:108 +0xfd
github.com/nlnwa/gowarc/pkg/index.(*indexWorker).worker(0xc00025e810, 0x7)
/home/aksel/Projects/gowarc/pkg/index/indexworker.go:75 +0x108
created by github.com/nlnwa/gowarc/pkg/index.NewIndexWorker
/home/aksel/Projects/gowarc/pkg/index/indexworker.go:58 +0x13a
This will make it easier to experiment in the future with alternative index components
Is your feature request related to a problem? Please describe.
Linting of incoming code changes
Describe the solution you'd like
Update CI to mirror changes in gowarcserver PR 25
Describe alternatives you've considered
Additional context
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.