Comments (20)
Enry already has a test with nil content, as you can see here. It's for the general function GetLanguage
, but GetLanguagesByModeline
is the first strategy to run, so it should raise a fail if any problem.
Anyway, I added a couple of cases with nil
and []byte{}
as content to the test TestGetLanguagesByModeline
and they pass without problems.
There is a weird thing, the function where the process is raising a panic (bytes.Count
) works well with nil
or []byte{]
content, demonstration here
For the moment, I couldn't reproduce yet the panic in local.
Keep on it.
from enry.
@ajnavarro how can I reproduce it?
from enry.
I would try to send to GetLanguagesByModeline
an empty or nil content
param.
from enry.
Eventually I couldn't reproduce the panic, so I didn't check if #130 fix the problem at all.
About the previous comment, some other things should be known about this issue:
bytes.Count()
function implementation depends on the platform it's running on. You can found these differences under$GOROOT/src/bytes
in filesbytes.go
,bytes_generic.go
,bytes_amd64.go
,bytes_s390x.go
- Currently travis it's using go version
1.8
, I changed it to1.9.x
to use the latest minor release of this version. Maybe it would fix something. - Another weird behavior from
bytes.Count()
,sep
argument it's always required except ifs
is nil. In that case, the function returns1
, check out here
Anyway, PR #130 try to fix this checking for empty or nil content before call bytes.Count()
function
from enry.
I can reproduce it too and it blocks us.
panic: runtime error: slice bounds out of range
goroutine 17 [running, locked to thread]:
bytes.Count(0x7f23e02849b0, 0xa, 0x0, 0x1c42003cc20, 0x1, 0x20, 0x0)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/bytes/bytes.go:62 +0x21d
gopkg.in/src-d/enry%2ev1.getHeaderAndFooter(0x7f23e02849b0, 0xa, 0x0, 0x66, 0x6, 0x7f23b80bc5f0)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:161 +0x9d
gopkg.in/src-d/enry%2ev1.GetLanguagesByModeline(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x7f23b99cdc08, 0x0, 0x0, 0x0, 0x0, ...)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:142 +0x5c
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x7f23b8ef9a73, 0xc, 0x1c420020de0)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:126 +0x129
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x1c42003ce48, 0x1c420016f00)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55
main.GetLanguage(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x1c42001a1d8, 0x1c4200007e0)
/home/travis/build/src-d/enry/shared/enry.go:11 +0x55
main._cgoexpwrap_f7db11756761_GetLanguage(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x0, 0x0)
command-line-arguments/_obj/_cgo_gotypes.go:58 +0x9a
panic: runtime error: slice bounds out of range
I will try to reproduce it because I am desperate and cannot work forward.
from enry.
I've just released v1.6.0, which includes #130. Check if the error is solved with this release.
from enry.
@ajnavarro I think it makes sense to allow enry crash and just skip the corresponding file, recording it to the errors list together with stderr output. E.g. we used to do it in our Python pipeline.
from enry.
@abeaumont I confirm that there is no panic with the latest release.
from enry.
great, closing then.
from enry.
Hi,
it looks like that this issue still exists. It appears quite rarely, so it should be tricky to debug.
src-d/sourced-ce#243
should a new issue be created? Or reopen this?
@abeaumont , @mcarmonaa , what do you think?
from enry.
@EgorBu You need to prove that Engine uses the updated version. When I used to have similar issues, I ran Engine locally, without a container, and replaced libenry.so in the engine's JAR with the manually built binary. And for sure in that case you need to prove that your patched JAR is used - sometimes Maven decides to overwrite it.
from enry.
@vmarkovtsev , from src-d/sourced-ce#243 (comment) - it should be enry 1.6.1
from enry.
The issue can be reopened if needed. Do you have a file sample causing the crash? Does the problem also happen when enry cli is used?
from enry.
@abeaumont, unfortunately - no. Because it rarely happens. Most of the times - everything works well, and sometimes it fails on the same dataset.
If you may tell which information could be useful for debugging - I may try to find the way how to collect it
from enry.
Even if frequency is low, it must happen with some file... that would be best information for debugging. Apart from that, an updated stack trace would also help.
from enry.
Do you want stack trace like in src-d/sourced-ce#243 (comment) ? Or something else?
BTW - does some kind of DEBUG
version of enry
exist? I may try to modify jar
as @vmarkovtsev mentioned in #129 (comment)
from enry.
@EgorBu yes, that one. Apparently you're passing an empty filename and that's causing the problem. I'll fix enry to avoid that crash, you may want to workaround it by filtering files without name in the meantime.
from enry.
Interesting that it works most of the times if it always should be empty
from enry.
We can hit two different problems here: breaking on empty filenames and the engine probabilistically supplying empty file names due to some crazy race or whatever.
from enry.
sounds reasonable.
What do you think, @ajnavarro? Can it happen?
from enry.
Related Issues (20)
- oniguruma - free memory (avoid potential memory leaks) HOT 10
- benchmark: refactor to be usable on CI for releases
- CLI app: refactor and improve performance
- Add IsGenerated function to the API HOT 1
- CLI: -mode=line reports bad results HOT 2
- enry - linguist single file report logic mismatch HOT 3
- Release: use non-EOLed JDK version HOT 1
- Different tokenisation results between oniguruma and RE2 HOT 12
- CLI: inconsistent path filtering HOT 1
- Reports gitignore as vendor HOT 2
- IsVendor could be changed to use a single regexp HOT 1
- Make build not working in golang:alpine container HOT 3
- import is a program, not an importable package HOT 4
- cannot find package "github.com/src-d/enry/v2" HOT 3
- Clarify installation instruction HOT 1
- Language detection accuracy measurements
- Not handling extensions HOT 2
- Get color from "parent language" if there is no definition for the language itself HOT 6
- Is this and all other srcd projects dead? HOT 13
- How to get a report of a whole directory when using it as Go module
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from enry.