subprocess PIPE error about python-soundfile HOT 15 CLOSED

The problem is that the pipe is not seekable, and soundfile expects something seekable.

We guard against this quite thoroughly by checking each object whether it has a seek method, and refrain from trying to call it if there is none. However, the pipe object does have a seek method, but throws an error if you try to use it. This is unfortunate.

This means that any use of seek and tell should probably be regarded as unsafe, and should be wrapped in a try/catch. Interestingly enough, the same goes for write. Pipes do have a write method, but throw an error when the write method is actually used. This will require some re-architecting.

from python-soundfile.

Ok, i tried too seek and it worked without errors. But the seek method tricket me by returning the buffer size...

sp.stdout.seek(0, 2)
Out[]: 4686

Ok what would be a good way fixing it? Does libsndfile need to know the exact file length?

Maybe it turns out not to bee solvable withoud reading all data to cache if i want to have a normal seekable soundfile object?

from python-soundfile.

seek(0, 2) is expected to return the buffer size, as you are seeking from SEEK_END.

Interestingly though, my sp.stdout.seek simply throws an error. What operating system are you running? This should be solvable within PySoundFile.

from python-soundfile.

As far as work-arounds are concerned, you can try to use the file descriptor corresponding to your "pipe" file object:

sf = soundfile.SoundFile(sp.stdout.fileno())

But note that this will not work on Windows if you use our packaged DLL (nor with the DLL from the official libsndfile installer), you'd have to use a DLL that's compiled with the same compiler as the Python interpreter itself (see http://www.mega-nerd.com/libsndfile/api.html#open_fd).

Another work-around would be to use a named pipe (e.g. with os.mkfifo()).
Sadly, this won't work on Windows either, because this is only supported for Unix-like systems.

from python-soundfile.

Another thing you can try is to call ffmpeg outside of Python:

ffmpeg -i test.mp3 -f wav - | python myscript.py

Within myscript.py, you can then use '-' as file name which tells libsndfile to use the standard input.

As a slight variation, you can pass the "file name" to your script:

ffmpeg -i test.mp3 -f wav - | python myscript.py -

I tried a similar thing once (see https://github.com/bastibe/PySoundFile/pull/68#issuecomment-69583755) and it works on Linux, but I have no idea if this works on Windows.

from python-soundfile.

Thanks for the replies!
Actually i'm running a windows system but have the demand running on linux as well. In case of running the software on a linux server.
So the .fileno() and named pipe is not a workaround.
The abstraction layer piping in the system console is not the way i want to design the software.
My temporary workaround is like following snippet:

import subprocess
from io import BytesIO
from soundfile import SoundFile


sp = subprocess.Popen(
    ['ffmpeg', '-i', 'test.mp3', '-f', 'wav', '-'],
    stdout=subprocess.PIPE)
bio = BytesIO(sp.stdout.read())
sf = SoundFile(bio)

The caveats of this method are:

• Simmilar fast to using temp-file on windows (BytesIO is quite slow).
• All data needs do be converted and read.

But it works and the harddisk is used more economical...

from python-soundfile.

@bastibe wrote:

This should be solvable within PySoundFile.

Is there something we can do?

If not, I think we can close this issue.

from python-soundfile.

I believe we can not seek if the file is not seekable. This limits us to sequential reads, but that might be a fair tradeoff in many circumstances.

from python-soundfile.

That would be great! Because my RAM is overflowing permanently with the BytesIO solution :-)
if i can help, just say me what to do...

from python-soundfile.

I'm seconding this issue. Basically I'm just trying to pipe audio data from sox:

import subprocess, soundfile as sf

p = subprocess.Popen('sox -n -t au -c 1 - synth 5 sine 440'.split(), stdout=subprocess.PIPE)
audiofh = sf.SoundFile(p.stdout)

# Read input incrementally
audiofh.read(100, dtype='int32')
# and so on...

The approach mentioned above about asserting whether a file is seekable by looking for a seek method seems wrong:

The problem is that the pipe is not seekable, and soundfile expects something seekable.

Why? Is that really required for reading input?

We guard against this quite thoroughly by checking each object whether it has a seek method, and refrain from trying to call it if there is none. However, the pipe object does have a seek method, but throws an error if you try to use it. This is unfortunate.

I wouldn't see it as unfortunate - this is by design! All io objects in Python implement a base interface (IOBase) that includes a seek() and a seekable() method. Why wouldn't you use seekable?

But note that this will not work on Windows if you use our packaged DLL (nor with the DLL from the official libsndfile installer), you'd have to use a DLL that's compiled with the same compiler as the Python interpreter itself

Yikes! Thanks for that important note. Indeed, it doesn't work. I wonder if using the cibuildwheel would address this since it uses the recommended python build environment when generating wheels. Just a thought.

I hope to have some time soon to commit to working on this; personally I find it useful to be able to read from a non-seekable file object.

from python-soundfile.

@tgarc

Thanks for the nicely reproducible code example!

I still don't think there is anything the soundfile module can do about that.

The problem is, if I'm not mistaken, that libsndfile insists on seeking when using the "virtual file I/O" mode (http://www.mega-nerd.com/libsndfile/api.html#open_virtual), which is what the soundfile module uses when you pass it a file-like object.

Libsndfile expects us to pass a pointer to some seek() function and it seems to unconditionally call this function (even when opening a RAW file). This causes an error when reading from a pipe file object (OSError: [Errno 29] Illegal seek), which I guess is what you experienced.

  typedef struct
  {    sf_vio_get_filelen  get_filelen ;
       sf_vio_seek         seek ;
       sf_vio_read         read ;
       sf_vio_write        write ;
       sf_vio_tell         tell ;
  } SF_VIRTUAL_IO ;

I even went out on a limb and tried to pass a NULL pointer for sf_vio_seek, but this resulted in an error (Error : bad pointer on SF_VIRTUAL_IO struct.).

So the problem isn't the soundfile module checking for a .seek() method or certain Python file objects having a .seek() method that throws. The problem is that libsndfile stubbornly insists on using seek() even if it wouldn't actually need to.
You should probably open an issue there?

When changing your example to use the file descriptor of the pipe, it works fine!

>>> audiofh = sf.SoundFile(p.stdout.fileno())
>>> audiofh.read(100, dtype='int32')
array([          0,   123617807,   246825651,   369214931,   490379759,
         609918308,   727434144,   842537542,   954846776,  1063989388,
...
       -1913421940, -1854127826, -1788684744, -1717309727, -1640239481,
       -1557729600, -1470053716, -1377502594, -1280383169, -1179017523], dtype=int32)

Except probably on Windows ...

Yikes!

Yikes indeed.

What you can try, however, is to persuade Christoph Gohlke to provide a wheel for the soundfile module on his page http://www.lfd.uci.edu/~gohlke/pythonlibs/.
Currently it's listed under "Other useful packages and applications not currently available on this page".
The Gohlke libraries are typically compiled with a Microsoft compiler, so if you are lucky, file descriptors work with it.

Or you could try to build a libsndfile DLL on Appveyor, this might work, too?

from python-soundfile.

from python-soundfile.

Thank you, @mgeier, for your explanation.

I wonder if we could provide libsndfile with a fake seek function that does nothing.

from python-soundfile.

I wonder if we could provide libsndfile with a fake seek function that does nothing.

I've tried doing that by returning 0 for vio_tell and vio_seek but no dice. libsndfile still is using seek under the hood so instead of giving an error, SoundFile.read() just always returns an empty array.

Using the file descriptor is a good workaround though, I just wonder how difficult it would be to distribute a properly compiled portaudio...

This issue is closed from my perspective since it's related to libsndfile and nothing can really be done from the pysoundfile side. For those interested there is some discussion on the libsndfile repo about this here.

from python-soundfile.

@tgarc Thanks for finding and joining the discussion on the libsndfile repo. I hope the problem with "virtual file IO" will be fixed there!

from python-soundfile.