PortAudio, DirectSound and AVCapture bindings
-
Good news everyone,
This week I did a binding for PortAudio in an attempt to get cross-platform audio to luapower. It might be a dead end though because it turns out PortAudio has a very bad design: it's callback-based and the callback is called from a different thread. Bummer. It also has a blocking API but it doesn't work with all backends. 2xBummer. I might have to bite the bullet once again and start from scratch (I really wish I didn't for this one). To that effect, I did a quick binding+demo for DirectSound, check it out - it plays psychedelic robot jazz :)
Then yesterday I started working on a cross-platform video capturing API to get video from webcams and such. I started with OSX since it has the easiest API, so if you have a Mac, you can already see your moving face on a nw window.
Anyway, if anyone wants to join me, here's a todo list for ya (I'll start working on these in order - I think):
- videoinput project:
- bind DirectShow in winapi/dshow.lua
- use it in videoinput_winapi.lua (the Windows backend of videoinput)
- bind Video4Linux (make the
v4l
orvideo4linux
project) - use it in videoinput_v4l.lua (the Linux backend of videoinput)
- audioio project:
- maybe find a better name for it first
- design a portable I/O API
- make the Windows backend audioio_dsound.lua based on DirectSound API
- make the OSX backend audioio_cocoa.lua based on AV Foundation API
- bind ALSA (make the
alsa
project) - make the Linux backend audioio_alsa.lua based on ALSA API
Btw, if anyone knows of a good portable audio I/O library please share it. It might save me a lot of trouble. Same for video input (I did look a bit at ffmpeg's libavdevice but it's undocumented and I hate compiling big C projects, so I dunno...).
- videoinput project:
-
Hey, audio processing got me interested.
What are the possibilities for the backends?
Are we talking access to raw microphone input and so on?As for libraries, https://github.com/andrewrk/libsoundio sounds interesting.
-
Geez how did I missed that? Thanks a lot LuaStoned! Looks like it's written by someone who really knows what he's doing (which would be refreshing after working with X11 for a month). It's a bit sad that it doesn't work on the Greatest OS of All Time (which is WinXP of course), but I guess we'll have to leave that behind someday (or add in a DirectSound backend). I'll definitely try it out and report back on this thread.
-
Oh, 1.0.0 was released a month ago that would explain why google turned down on me...
-
Well, it seems like libsoundio has the same problem as PortAudio - the callback is called from a different thread, which means you can't just use a luajit callback, you need to use a thread along with all the troubles it brings. I suppose this is to avoid latency that a locked queue would introduce, but then how do you push data in the callback from a different thread anyway? There's something I'm missing here...
Here's the codeif anyone's interested to play with it and making it work (using threads): https://github.com/luapower/libsoundio
I also asked the author about it: https://github.com/andrewrk/libsoundio/issues/28
-
Update: turns out this is not as bad as I was imagining. libsoundio has a thread-safe ring buffer which can be used to transfer data between threads and luastate can be used for the callbacks. The demo works now, so I think I can go on with the binding.
-
Update: Bindings, build scripts and an output demo for libsoundio are available for OSX (CoreAudio backend) and Windows (WASAPI backend). Linux/ALSA requires ALSA 1.0.27 (so Ubuntu 14+) (there's a conversation about that), I'm not sure how to fix this yet to get better compatibility without forking libsoundio.
TODO:
- compile the PulseAudio backend (Ubuntu 10 pulse-dev package is broken)
- compile the JACK backend
- ALSA backend (with better backwards compat if possible)
- some issues with setting sample rates on OSX and Windows
-
Hey, just saw all your news / the issue on the libsoundio GitHub page.
Going to play with the demo as soon as I can, will report back!
-
Cool. Here's another update:
- OSX only works with 44100Hz and 32bit float formats
- Windows reports a single sample rate range on my built-in realtek (8000-140K or something) but in reality only works with 44.1- 58KHz sample rates (and it doesn't raise errors on invalid sample rates).
- couldn't make ALSA work on Ubuntu 14 x64 with any of the reported cards on my virtualbox (either device.probe_error or stream.layout_error is set). Might be a linking issue, will look into it.
- also note the updated compatibility table
My conclusion is that libsoundio is a bit green right now and not very backwards compat. It has a great API and it supports Pulse and Jack for those who want it (not compiled in yet though) it so I'll keep maintaining the lib but for my purposes I have to look for something else. Either finish the PortAudio binding or make a pure Lua binding based on DirectSound, CoreAudio and ALSA (for which I have to do bindings too). Haven't decided yet which way is best. Feedback welcome.
-
Pure Lua bindings always sound good but it seems like a bunch of work for supporting all different platforms.
So far I've only generated sound output but that seems to work pretty fine, will report back once I work with input.
-
Ok, don't forget to report the platforms (OS version and arch), sample rates and formats which work and those which don't.