Permission to integrate a FFI binding on LuaPower's github repo

  • Hello, I've been working for about a month or more in a plain FFI binding for sockets. (Which is partially functional because I haven't met much people using Linux that could test it), but your site is very noticeable and I think it's the only way for the people to find it, I'm going to maintain it myself, so I wanted to ask you for your permission to keep it on LuaPower github's repository. Is that possible?

    This is the actual code:

  • 23
    Log in to reply

  • Wow this is mindblowing, especially the replacement to malloc. It might take me some time to replace all that but for now I just replaced whatever was easy to replace. If I want to allocate new memory I should use this...

    cbuf = rb.cbuffer{size = (The size of the buffer)}

    Edit: So I was expecting that this replacement would be correct, but it's pretty confusing.
    function TUDPStream:Read(Buffer, Size)

    local NewBuffer --, PrevBuffer
    local Size = math.min(Size, self.RecvSize)
    if Size > 0 then
        --ffi.copy(Buffer, self.RecvBuffer, Size)
        self.RecvBuffer:read(Buffer, 0, 0, Size) -- This would copy 'Size' bytes into the buffer
        if Size < self.RecvSize then
            NewBuffer = rb.cbuffer{size = self.RecvSize - Size} -- The bytes we read must be removed from the begining of the buffer
            self.RecvBuffer:read(NewBuffer, Size + 1, 0, self.RecvSize - Size)
            --PrevBuffer = ffi.string(self.RecvBuffer, self.RecvSize)
            --ffi.copy(NewBuffer, PrevBuffer:sub(Size + 1), self.RecvSize - Size)
            self.RecvBuffer = NewBuffer
            self.RecvSize = self.RecvSize - Size
            self.RecvBuffer = rb.cbuffer{size = 0}
            self.RecvSize = 0
    return Size


    I assumed I would no longer needed to run Code tags are weird :O

    Also, is zero the first index? If so, then I guess it's wrong to make the receive buffer start from "Size + 1"

  • Oh, that was just an example of what you could do to avoid allocations. You don't have to use that necessarily, you can make your own, or use some other allocation-avoiding scheme. Anyway, if you want to use that particular ring buffer implementation, I just added some documentation for it. Hope it helps. Note that you can't just replace malloc() with a ring buffer, it's a bit more complicated than that. The algorithm is explained better than I could ever explain it on wikipedia.

  • Anyway, I don't want to sidetrack you or anything. Remember, if performance is not your goal, don't worry about it. Performace optimization can be a time sink and can be done at later stages anyway and it's best served after benchmarking. I'd focus more on the API, in fact I'd write the API first :)

  • On the other hand I just remembered that TUDPStream is a ffi structure and that means it needs to work with cdata fields. I must have broken it again.

  • The tcp and udp parts of the library are now fully functional, at least for Windows. I did most of the things you told me to do, except some here.

    • in luapower all modules follow the lowercase_sometimes_with_underscores convention rather than CamelCase. Again, I don't mind it, but users would probably feel for the inconsistency.
      I considered that I should keep the actual functions that were written like this for backwards compatibility, I know some people that were using my library before. (Hence why it's called bnet, lol)

    • % 256 and / 256 are slow; consider replacing them with rshift(8), lshift(8) but that's not needed either, just use a cast if you want to write a whole int32 at a time (same with ServerIP calculation, you can just cast h_addr_list to an int32 and tonumber() it).
      I couldn't really get how to use those two functions, I'm not really used that much into the bit library, but at least bit.lshift replaced Shl well.

    However the rest of the library looks alike LuaSocket's api, I still need to finish the socket.dns.* api (which I couldn't understand very well from the original documentation), and finally I need to complete the documentation, which I guess I'm going to copy-paste from the functions I ported and the ones I made. (Yes, I'm very bad at explaining plus it would be good to keep the original documentation)

    With this progress, is there a possibility that this could appear on the site and the repo?

  • I couldn't really get how to use those two functions

    • x % 256 is, 0xff)
    • x / 256 is bit.rshift(x, 8)
    • h_addr_list is defined as a char**, but in reality it's a in_addr** and in_addr is a int32, so if you define it correctly you can get addresses directly with h_addr_list[0][i], no need for bit operations.

    The doc should be something like this:

    tagline: networking library
    platforms: mingw32, mingw64
    <warn>Work in progress! To be released when?</warn>
    ## `local bnet = require'bnet'`
    Networking library, API compatible with LuaSocket 2.0.2.
    ## API
    ------------------------- ----------------------------------------------------
    ------------------------- ----------------------------------------------------
    ## Notes

    Please read the publishing guideline before publishing. Especially you might want to include in the doc how your lib is different from luasocket, since luasocket is already included and users might get confused if your library just reads "API compatible with luasocket" -- you should probably explain why should they choose bnet over luasocket. Also, please consider adding a test or demo file. Thanks.

  • Actually, since it's API compatible with luasocket 2.0.2 you could include luasocket's test suite and demos instead.

    Btw, luasocket is currently at v3.0:

  • I changed with the one you gave me and I'm now using the bit functions you told me (I added a slightly better description too),
    I added a WHAT file on /csrc/bnet/,
    Now instead of using the cdefs on bnet.lua, I added bnet_h.lua,
    I added my name on bnet.lua and the license,
    I tested four of the libraries from the original LuaSocket (the rest of them were using different ones I wasn't supposed to port like HTTP, FTP, SMTP, etc),

    About h_addr_list, I might change it later because the code is already functional and I might break something trying to change this (I don't expect to miss this important detail but if I do it now the stable version will probably stop being stable).

  • I think you should install the module to your luapower tree first to get a sense of the overlaid directory structure. You can do that with mgit:

    mgit clone

    You will notice that your repo files don't get cloned into their own directory but they share the root directory with all other modules. That's why it's important to name your files such that don't clash with the files of other repos. So you can't have a directory called "test" in there and you can't have a .gitignore file in the root directory either. Consider merging the test files into bnet_test.lua. If they're not automated tests and are interactive tests instead, consider naming the file bnet_demo.lua. If they are automated tests but you want to have the ability to run them separately as well, you can put them in a table and dispatch on arg[1] so you can run them individually from command line.

  • I merged the test files into bnet_test.lua, and removed the .gitignore file.

    One of the tests is interactive but it's actually easier to use the web browser to test it, I pointed that out on the script.

  • Looks great. Thank you.

    The website pulls the repo every hour so you have to wait a bit to see the updated docs. I can increase that time if needed.

  • Few more notes on the code and more unsolicited advice :)

    • are you sure for i = 0, n_read do shouldn't be for i = 0, n_read-1 do ?
    • not sure why would a socket library have a time/sleep API in it (I know luasocket has it but I think that's lame, they should've released a time library separately) but anyway, here's a desktop-portable implementation:
    • you don't want to iterate arrays with pairs(), use ipairs() instead, especially if you want to add the results to another array -- adding the keys of a contiguous array in random order is really painful to the poor memory allocator
    •"unsigned int", value) gives you a constant int32 -- doing math with that results in int64 values which may go in the heap! There's seldom a need to allocate scalars with Consider using plain Lua numbers instead (use math.floor() to simulate integer math if you have to).
    • in socket.protect() you wanna say that if there's no error, then return all the values right? well, you may not want people to see that implementation :) here's a better one:
    function socket.protect(func)
       local function pass(ok, ...)
          if ok then return ... end
          return nil, ...
       return function(...)
          return pass(pcall(func, ...))

    There's more that I could say but I don't want you to run away screaming :) So let's talk API instead. A good I/O library IMHO should have:

    • ability to work with any file descriptors the OS supports not just sockets
    • ability to select the polling mechanism (or at least choose a good one, which is not select())
    • a cdata stream interface
    • non-blocking dns lookup -- this should probably be a separate module/package
    • no Internet protocols and hi-level stuff (these are just parsers/formatters anyway and they don't belong in a I/O library -- they should be separate libs that hook into any API that gives you read() and write()) But I'm going into rant mode so I'd better stop before it's too late :)

  • It's fine, I'm taking all the notes you're giving to me since I'm not even a computer scientist. About the first point, I copied that select function from BlitzMax's wrapper, so I'm not sure if I should substract a unit, but you're probably right. About the "unsigned int", this is just for the timeout, apparently assigning a normal integer to math.huge returned negative values so I just turned it into a "unsigned int" rather than a "int", I would use Lua numbers but the values returned from socket.udp() and socket.tcp() are actually cdata objects so I needed to give them some class.

  • Well it's gonna be tough until you get a good knack of C types and conversion rules, integer vs floating point math, bit ops, and an intuition of performance characteristics of various Lua patterns (how tables are implemented, static vs dynamic memory allocation) and even a few LuaJIT and ffi insights as well. There's a lot of ground to cover, so don't worry if you don't do a good job at first -- but be willing to scrap a lot of code and even start again from scratch potentially many times until you get it right, if you want to make a good lib (and yourself a better programmer in the process).

    Here's some of my notes (not nearly enough of course):

    There's more tips on the Lua wiki, and of course, the PiL book which I highly recommend.

  • Hey, out of curiosity how do you update the actual page on I was trying to design the documentation file but it doesn't seem to apply changes on the site (and it doesn't know when the last commit was), do you need a webhook or do you do that yourself manually?

  • The website pulls the repo every hour so you have to wait a bit to see the updated docs. I can increase that time if needed. You can also add a webhook to

  • Hey, github isn't pushing the webhook anymore because of this.
    /home/cosmin/website/www/actions.lua:1100: attempt to call field 'exec' (a nil value) stack traceback: /home/cosmin/website/www/actions.lua:1100: in function 'handler' /home/cosmin/website/www/app.lua:167: in function </home/cosmin/website/www/app.lua:149> [C]: in function 'xpcall' /home/cosmin/website/nginx/../www/ngx.lua:20: in function 'try_call' /home/cosmin/website/nginx/../www/ngx.lua:27: in function </home/cosmin/website/nginx/../www/ngx.lua:1>
    Just saying.

  • Thanks for reporting. I fixed that but didn't test (will test later today).

Log in to reply

Internal error.

Oops! Looks like something went wrong!