PortAudio, DirectSound and AVCapture bindings


  • Good news everyone,

    This week I did a binding for PortAudio in an attempt to get cross-platform audio to luapower. It might be a dead end though because it turns out PortAudio has a very bad design: it's callback-based and the callback is called from a different thread. Bummer. It also has a blocking API but it doesn't work with all backends. 2xBummer. I might have to bite the bullet once again and start from scratch (I really wish I didn't for this one). To that effect, I did a quick binding+demo for DirectSound, check it out - it plays psychedelic robot jazz :)

    Then yesterday I started working on a cross-platform video capturing API to get video from webcams and such. I started with OSX since it has the easiest API, so if you have a Mac, you can already see your moving face on a nw window.

    Anyway, if anyone wants to join me, here's a todo list for ya (I'll start working on these in order - I think):

    • videoinput project:
      • bind DirectShow in winapi/dshow.lua
      • use it in videoinput_winapi.lua (the Windows backend of videoinput)
      • bind Video4Linux (make the v4l or video4linux project)
      • use it in videoinput_v4l.lua (the Linux backend of videoinput)
    • audioio project:
      • maybe find a better name for it first
      • design a portable I/O API
      • make the Windows backend audioio_dsound.lua based on DirectSound API
      • make the OSX backend audioio_cocoa.lua based on AV Foundation API
      • bind ALSA (make the alsa project)
      • make the Linux backend audioio_alsa.lua based on ALSA API

    Btw, if anyone knows of a good portable audio I/O library please share it. It might save me a lot of trouble. Same for video input (I did look a bit at ffmpeg's libavdevice but it's undocumented and I hate compiling big C projects, so I dunno...).


  • 11
    Posts
    3239
    Views
    Log in to reply


  • Hey, audio processing got me interested.

    What are the possibilities for the backends?
    Are we talking access to raw microphone input and so on?

    As for libraries, https://github.com/andrewrk/libsoundio sounds interesting.



  • Geez how did I missed that? Thanks a lot LuaStoned! Looks like it's written by someone who really knows what he's doing (which would be refreshing after working with X11 for a month). It's a bit sad that it doesn't work on the Greatest OS of All Time (which is WinXP of course), but I guess we'll have to leave that behind someday (or add in a DirectSound backend). I'll definitely try it out and report back on this thread.



  • Oh, 1.0.0 was released a month ago that would explain why google turned down on me...



  • Well, it seems like libsoundio has the same problem as PortAudio - the callback is called from a different thread, which means you can't just use a luajit callback, you need to use a thread along with all the troubles it brings. I suppose this is to avoid latency that a locked queue would introduce, but then how do you push data in the callback from a different thread anyway? There's something I'm missing here...

    Here's the codeif anyone's interested to play with it and making it work (using threads): https://github.com/luapower/libsoundio

    I also asked the author about it: https://github.com/andrewrk/libsoundio/issues/28



  • Update: turns out this is not as bad as I was imagining. libsoundio has a thread-safe ring buffer which can be used to transfer data between threads and luastate can be used for the callbacks. The demo works now, so I think I can go on with the binding.



  • Update: Bindings, build scripts and an output demo for libsoundio are available for OSX (CoreAudio backend) and Windows (WASAPI backend). Linux/ALSA requires ALSA 1.0.27 (so Ubuntu 14+) (there's a conversation about that), I'm not sure how to fix this yet to get better compatibility without forking libsoundio.

    TODO:

    • compile the PulseAudio backend (Ubuntu 10 pulse-dev package is broken)
    • compile the JACK backend
    • ALSA backend (with better backwards compat if possible)
    • some issues with setting sample rates on OSX and Windows


  • Hey, just saw all your news / the issue on the libsoundio GitHub page.
    Going to play with the demo as soon as I can, will report back!



  • Cool. Here's another update:

    • OSX only works with 44100Hz and 32bit float formats
    • Windows reports a single sample rate range on my built-in realtek (8000-140K or something) but in reality only works with 44.1- 58KHz sample rates (and it doesn't raise errors on invalid sample rates).
    • couldn't make ALSA work on Ubuntu 14 x64 with any of the reported cards on my virtualbox (either device.probe_error or stream.layout_error is set). Might be a linking issue, will look into it.
    • also note the updated compatibility table

    My conclusion is that libsoundio is a bit green right now and not very backwards compat. It has a great API and it supports Pulse and Jack for those who want it (not compiled in yet though) it so I'll keep maintaining the lib but for my purposes I have to look for something else. Either finish the PortAudio binding or make a pure Lua binding based on DirectSound, CoreAudio and ALSA (for which I have to do bindings too). Haven't decided yet which way is best. Feedback welcome.



  • Pure Lua bindings always sound good but it seems like a bunch of work for supporting all different platforms.
    So far I've only generated sound output but that seems to work pretty fine, will report back once I work with input.



  • Ok, don't forget to report the platforms (OS version and arch), sample rates and formats which work and those which don't.


11
Posts
3239
Views
Log in to reply

Internal error.

Oops! Looks like something went wrong!