Panic about datareader HOT 6 CLOSED

Can you provide more context on this? What were you doing when this happened? Are you using the latest versions of everything?

from datareader.

I was using the binary that was included in the git repository. I started to attempt to debug but the line #'s weren't matching up in sas7bdat.go for the slice panic, so I built the binary from a recent checkout and this is what I now get.

panic: runtime error: slice bounds out of range [:12408] with capacity 12288

goroutine 1 [running]:
github.com/kshedden/datareader.(*SAS7BDAT).processByteArrayWithData(0xc0000e4000, 0x2fe8, 0x90, 0x30, 0x4ea720)
        /home/mezimm00/go/src/github.com/kshedden/datareader/sas7bdat.go:1208 +0x73f
github.com/kshedden/datareader.(*SAS7BDAT).readline(0xc0000e4000, 0x1f40, 0x1f40, 0xc000122000)
        /home/mezimm00/go/src/github.com/kshedden/datareader/sas7bdat.go:823 +0x304
github.com/kshedden/datareader.(*SAS7BDAT).Read(0xc0000e4000, 0x3e8, 0x13, 0x13, 0x0, 0x0, 0x1d)
        /home/mezimm00/go/src/github.com/kshedden/datareader/sas7bdat.go:670 +0x33e
main.doConversion(0x515d20, 0xc0000e4000)
        /home/mezimm00/datareader/cmd/stattocsv/main.go:31 +0x95a
main.main()
        /home/mezimm00/datareader/cmd/stattocsv/main.go:144 +0x307

from datareader.

I was trying to do this dev/fix in my branch, but I'm also realizing that's messing stuff up (go path/mod/etc) which I don't want to fix/address at the moment. Still working on it.

from datareader.

Using a go.mod file inside datareader/cmd/stattocsv forces using "this" repository on go build. Doing that, the stacktrace is now:

panic: runtime error: slice bounds out of range [:12408] with capacity 12288

goroutine 1 [running]:
github.com/kshedden/datareader.(*SAS7BDAT).processByteArrayWithData(0xc00019e000, 0x2fe8, 0x90, 0x30, 0x4ea720)
        /home/mezimm00/datareader/sas7bdat.go:1208 +0x73f
github.com/kshedden/datareader.(*SAS7BDAT).readline(0xc00019e000, 0x1f40, 0x1f40, 0xc000126000)
        /home/mezimm00/datareader/sas7bdat.go:823 +0x304
github.com/kshedden/datareader.(*SAS7BDAT).Read(0xc00019e000, 0x3e8, 0x13, 0x13, 0x0, 0x0, 0x1d)
        /home/mezimm00/datareader/sas7bdat.go:670 +0x33e
main.doConversion(0x515e40, 0xc00019e000)
        /home/mezimm00/datareader/cmd/stattocsv/main.go:31 +0x95a
main.main()
        /home/mezimm00/datareader/cmd/stattocsv/main.go:144 +0x307

I can't provide the file due to data sensitivity and I'm not familiar with the actual format of the file, so I'm not sure if this is an issue with my data file or the reader, but I do know I can hack the reader to not panic :)

from datareader.

Seems like when this happens, it needs to read more from the file and put it in sas.cachedPage -- I admit I don't know how to do that. I can hack it not to panic and read as much as is there, but then the columnDataLengths and columnDataOffsets all get messed up right afterwards and those panic. Those values appear to be coming from the file contents itself? So if the file is saying there's more data but there's not enough data in the page, either more data needs to be read or it's a corrupt file. Help @kshedden :)

from datareader.

For lack of a better term, can a record/row "span" "pages" in sas7bdat files? If so, this patch seemed to fix it. This is not pretty I don't think but seemed to get the job done although I admit that I don't know which record(s) in the file were causing this issue nor do I know what they should be, so I don't have much to compare against for accuracy.

See #20

from datareader.