PortAudio, DirectSound and AVCapture bindings

[cosmin]

Good news everyone,

This week I did a binding for PortAudio in an attempt to get cross-platform audio to luapower. It might be a dead end though because it turns out PortAudio has a very bad design: it's callback-based and the callback is called from a different thread. Bummer. It also has a blocking API but it doesn't work with all backends. 2xBummer. I might have to bite the bullet once again and start from scratch (I really wish I didn't for this one). To that effect, I did a quick binding+demo for DirectSound, check it out - it plays psychedelic robot jazz :)

Then yesterday I started working on a cross-platform video capturing API to get video from webcams and such. I started with OSX since it has the easiest API, so if you have a Mac, you can already see your moving face on a nw window.

Anyway, if anyone wants to join me, here's a todo list for ya (I'll start working on these in order - I think):

• videoinput project:
  • bind DirectShow in winapi/dshow.lua
  • use it in videoinput_winapi.lua (the Windows backend of videoinput)
  • bind Video4Linux (make the v4l or video4linux project)
  • use it in videoinput_v4l.lua (the Linux backend of videoinput)
• audioio project:
  • maybe find a better name for it first
  • design a portable I/O API
  • make the Windows backend audioio_dsound.lua based on DirectSound API
  • make the OSX backend audioio_cocoa.lua based on AV Foundation API
  • bind ALSA (make the alsa project)
  • make the Linux backend audioio_alsa.lua based on ALSA API

Btw, if anyone knows of a good portable audio I/O library please share it. It might save me a lot of trouble. Same for video input (I did look a bit at ffmpeg's libavdevice but it's undocumented and I hate compiling big C projects, so I dunno...).

[LuaStoned]

Hey, audio processing got me interested.

What are the possibilities for the backends?
Are we talking access to raw microphone input and so on?

As for libraries, https://github.com/andrewrk/libsoundio sounds interesting.

[cosmin]

Geez how did I missed that? Thanks a lot LuaStoned! Looks like it's written by someone who really knows what he's doing (which would be refreshing after working with X11 for a month). It's a bit sad that it doesn't work on the Greatest OS of All Time (which is WinXP of course), but I guess we'll have to leave that behind someday (or add in a DirectSound backend). I'll definitely try it out and report back on this thread.

[cosmin]

Oh, 1.0.0 was released a month ago that would explain why google turned down on me...

[cosmin]

Well, it seems like libsoundio has the same problem as PortAudio - the callback is called from a different thread, which means you can't just use a luajit callback, you need to use a thread along with all the troubles it brings. I suppose this is to avoid latency that a locked queue would introduce, but then how do you push data in the callback from a different thread anyway? There's something I'm missing here...

Here's the codeif anyone's interested to play with it and making it work (using threads): https://github.com/luapower/libsoundio

I also asked the author about it: https://github.com/andrewrk/libsoundio/issues/28

[cosmin]

Update: turns out this is not as bad as I was imagining. libsoundio has a thread-safe ring buffer which can be used to transfer data between threads and luastate can be used for the callbacks. The demo works now, so I think I can go on with the binding.

[cosmin]

Update: Bindings, build scripts and an output demo for libsoundio are available for OSX (CoreAudio backend) and Windows (WASAPI backend). Linux/ALSA requires ALSA 1.0.27 (so Ubuntu 14+) (there's a conversation about that), I'm not sure how to fix this yet to get better compatibility without forking libsoundio.

TODO:

• compile the PulseAudio backend (Ubuntu 10 pulse-dev package is broken)
• compile the JACK backend
• ALSA backend (with better backwards compat if possible)
• some issues with setting sample rates on OSX and Windows

[LuaStoned]

Hey, just saw all your news / the issue on the libsoundio GitHub page.
Going to play with the demo as soon as I can, will report back!

[cosmin]

Cool. Here's another update:

• OSX only works with 44100Hz and 32bit float formats
• Windows reports a single sample rate range on my built-in realtek (8000-140K or something) but in reality only works with 44.1- 58KHz sample rates (and it doesn't raise errors on invalid sample rates).
• couldn't make ALSA work on Ubuntu 14 x64 with any of the reported cards on my virtualbox (either device.probe_error or stream.layout_error is set). Might be a linking issue, will look into it.
• also note the updated compatibility table

My conclusion is that libsoundio is a bit green right now and not very backwards compat. It has a great API and it supports Pulse and Jack for those who want it (not compiled in yet though) it so I'll keep maintaining the lib but for my purposes I have to look for something else. Either finish the PortAudio binding or make a pure Lua binding based on DirectSound, CoreAudio and ALSA (for which I have to do bindings too). Haven't decided yet which way is best. Feedback welcome.

[LuaStoned]

Pure Lua bindings always sound good but it seems like a bunch of work for supporting all different platforms.
So far I've only generated sound output but that seems to work pretty fine, will report back once I work with input.

[cosmin]

Ok, don't forget to report the platforms (OS version and arch), sample rates and formats which work and those which don't.