Sparse file in archive triggers "OSError: [Errno 28] No space left on device: '...' " during scan about extractcode HOT 8 OPEN

Would this be with https://github.com/moby/moby/archive/refs/tags/v20.10.5.tar.gz ?

from extractcode.

And I had not seen your attachment! so I am good wih the code link you provided.

So I think this is sparse tarball issue which is 60GB extracted and only 5KB otherwise, unextracted, and not even compressed

Can you test scanning that one single 5KB file?
https://github.com/moby/moby/blob/363e9a88a11be517d9e8c65c998ff56f774eb4dc/vendor/archive/tar/testdata/gnu-sparse-big.tar

And can you detail what pre-processing you apply on this. Did you call extractcode first?
Is there a place where I can see your code? I am always interested in how ScanCode is integrated!

from extractcode.

Note that this could be related to nexB/scancode-toolkit#2431 where @Angi2412 and @avishmehta68710 both mentioned having a similar issue or at least this comment nexB/scancode-toolkit#2431 (comment) references the same code:

I could observe the same behaviour and error message with this file: moby v20.10.5.

from extractcode.

Yes, I apply extractcode first! Unfortunately, there is no place where you can see my code, but I can provide a minimalexample.py:

  from scancode import cli
  import multiprocessing
  import logging
  from extractcode import extract
  
  def main():
  
      extract_log = extract.extract(location='gnu-sparse-big.tar', recurse=True)
      i = 0
      for e in extract_log:
          print(e)
  
      rc, results = cli.run_scan(
          'gnu-sparse-big.tar-extract', license=True, copyright=True,
          return_results=True, processes=14, verbose=True, quiet=False,
          timeout=46800)
  
      print(rc, results)
  
  if __name__ == "__main__":
      main()

As you proposed, I tested the one single 5KB file and the exact same Error occurs:

(docker logs) (Linux and Windows with docker)

Depending on the machine I work on

• I either experience the same Issue as in nexB/scancode-toolkit#2431 (Linux with docker or run minimalexample.py as standalone python script)
• or there appears the Issue "OSError: [Errno 28] No space left on device" as I mentioned above (Windows with docker, as well as Linux with docker)

from extractcode.

So this is a sparse file issue. The short term workaround may be to ignore the gnu-sparse-big.tar file entirely.

from extractcode.

Thank you very much! After further analysis, I also came to the conclusion that this is a sparse file issue.

from extractcode.

I still may want to keep this open for now and transfer the issue to extractcode... as we may want to have a special processing for sparse files

from extractcode.

Done... now in extractcode!

from extractcode.