Permission to integrate a FFI binding on LuaPower's github repo


  • Hello, I've been working for about a month or more in a plain FFI binding for sockets. (Which is partially functional because I haven't met much people using Linux that could test it), but your site is very noticeable and I think it's the only way for the people to find it, I'm going to maintain it myself, so I wanted to ask you for your permission to keep it on LuaPower github's repository. Is that possible?

    This is the actual code:
    https://github.com/Starkkz/luajit-bnet


  • 23
    Posts
    9612
    Views
    Log in to reply


  • I haven't met much people using Linux that could test it

    You could install an Ubuntu in a Virtualbox. That's how I test on Linux.



  • Alright, I did the adjustments you said but I still need to write a decent documentation about it.

    About what is it, it's supposed to be originally a socket library for another programming language but I just tried to do the right ports from it into FFI because the API seemed a way too friendly. But I guess this port I'm doing right now will be LuaSocket-api compatible as much as BNet-api compatible.

    http://github.com/Starkkz/bnet



  • Please rename socket.lua -> bnet.lua. This is because luapower packages are unzipped or cloned over the same directory, so they share a common namespace. This is the reason for the strict naming conventions.

    Just a few comments on the code:

    • don't use module() - I don't mind it but most Lua users do. Use the local M = {}; function M.foo() end; [...more functions...] return M pattern if you can.
    • in luapower all modules follow the lowercase_sometimes_with_underscores convention rather than CamelCase. Again, I don't mind it, but users would probably feel for the inconsistency.
    • don't use malloc() unless you allocate big chunks (bitmaps etc.); ffi.new() can be faster and better integrated with the gc (allocation sinking; reporting the actual size to the gc).
    • ffi.new() calls ffi.fill() implicitly for fixed-size types.
    • you might want read(), write() and select() calls to be alloc-free. Consider using a ring buffer (also string concatenation and ffi.string() are out). LuaJIT 2.1 will sink some of the allocations for you but you have to test that empirically as there's no general guideline on when it bails out (such guideline would probably be very complex).
    • there's bit.lshift for Shl (works up to 32bit)
    • % 256 and / 256 are slow; consider replacing them with rshift(8), lshift(8) but that's not needed either, just use a cast if you want to write a whole int32 at a time (same with ServerIP calculation, you can just cast h_addr_list to an int32 and tonumber() it).

    [I'm not writing these to discourage you btw. And if your lib is not meant for performance use cases, none of the above matter anyway. In any case, reimplementing luasocket in pure Lua was something I've been wanting to do for a long time. It's nice that someone actually started doing it].



  • Wow this is mindblowing, especially the replacement to malloc. It might take me some time to replace all that but for now I just replaced whatever was easy to replace. If I want to allocate new memory I should use this...

    cbuf = rb.cbuffer{size = (The size of the buffer)}
    Right?

    Edit: So I was expecting that this replacement would be correct, but it's pretty confusing.
    function TUDPStream:Read(Buffer, Size)

    local NewBuffer --, PrevBuffer
    local Size = math.min(Size, self.RecvSize)
    if Size > 0 then
        --ffi.copy(Buffer, self.RecvBuffer, Size)
        self.RecvBuffer:read(Buffer, 0, 0, Size) -- This would copy 'Size' bytes into the buffer
    
        if Size < self.RecvSize then
            NewBuffer = rb.cbuffer{size = self.RecvSize - Size} -- The bytes we read must be removed from the begining of the buffer
            self.RecvBuffer:read(NewBuffer, Size + 1, 0, self.RecvSize - Size)
    
            --PrevBuffer = ffi.string(self.RecvBuffer, self.RecvSize)
            --ffi.copy(NewBuffer, PrevBuffer:sub(Size + 1), self.RecvSize - Size)
            --C.free(self.RecvBuffer)
    
            self.RecvBuffer = NewBuffer
            self.RecvSize = self.RecvSize - Size
        else
            --C.free(self.RecvBuffer)
            self.RecvBuffer = rb.cbuffer{size = 0}
            self.RecvSize = 0
        end
    end
    return Size
    

    end

    I assumed I would no longer needed to run C.free. Code tags are weird :O

    Also, is zero the first index? If so, then I guess it's wrong to make the receive buffer start from "Size + 1"



  • Oh, that was just an example of what you could do to avoid allocations. You don't have to use that necessarily, you can make your own, or use some other allocation-avoiding scheme. Anyway, if you want to use that particular ring buffer implementation, I just added some documentation for it. Hope it helps. Note that you can't just replace malloc() with a ring buffer, it's a bit more complicated than that. The algorithm is explained better than I could ever explain it on wikipedia.



  • Anyway, I don't want to sidetrack you or anything. Remember, if performance is not your goal, don't worry about it. Performace optimization can be a time sink and can be done at later stages anyway and it's best served after benchmarking. I'd focus more on the API, in fact I'd write the API first :)



  • On the other hand I just remembered that TUDPStream is a ffi structure and that means it needs to work with cdata fields. I must have broken it again.



  • The tcp and udp parts of the library are now fully functional, at least for Windows. I did most of the things you told me to do, except some here.

    • in luapower all modules follow the lowercase_sometimes_with_underscores convention rather than CamelCase. Again, I don't mind it, but users would probably feel for the inconsistency.
      I considered that I should keep the actual functions that were written like this for backwards compatibility, I know some people that were using my library before. (Hence why it's called bnet, lol)

    • % 256 and / 256 are slow; consider replacing them with rshift(8), lshift(8) but that's not needed either, just use a cast if you want to write a whole int32 at a time (same with ServerIP calculation, you can just cast h_addr_list to an int32 and tonumber() it).
      I couldn't really get how to use those two functions, I'm not really used that much into the bit library, but at least bit.lshift replaced Shl well.

    However the rest of the library looks alike LuaSocket's api, I still need to finish the socket.dns.* api (which I couldn't understand very well from the original documentation), and finally I need to complete the documentation, which I guess I'm going to copy-paste from the functions I ported and the ones I made. (Yes, I'm very bad at explaining plus it would be good to keep the original documentation)

    With this progress, is there a possibility that this could appear on the site and the repo?



  • I couldn't really get how to use those two functions

    • x % 256 is bit.band(x, 0xff)
    • x / 256 is bit.rshift(x, 8)
    • h_addr_list is defined as a char**, but in reality it's a in_addr** and in_addr is a int32, so if you define it correctly you can get addresses directly with h_addr_list[0][i], no need for bit operations.

    The doc should be something like this:

    ---
    tagline: networking library
    platforms: mingw32, mingw64
    ---
    
    <warn>Work in progress! To be released when?</warn>
    
    ## `local bnet = require'bnet'`
    
    Networking library, API compatible with LuaSocket 2.0.2.
    
    ## API
    
    ------------------------- ----------------------------------------------------
    ------------------------- ----------------------------------------------------
    
    ## Notes
    

    Please read the publishing guideline before publishing. Especially you might want to include in the doc how your lib is different from luasocket, since luasocket is already included and users might get confused if your library just reads "API compatible with luasocket" -- you should probably explain why should they choose bnet over luasocket. Also, please consider adding a test or demo file. Thanks.



  • Actually, since it's API compatible with luasocket 2.0.2 you could include luasocket's test suite and demos instead.

    Btw, luasocket is currently at v3.0: https://github.com/diegonehab/luasocket



  • I changed bnet.md with the one you gave me and I'm now using the bit functions you told me (I added a slightly better description too),
    I added a WHAT file on /csrc/bnet/,
    Now instead of using the cdefs on bnet.lua, I added bnet_h.lua,
    I added my name on bnet.lua and the license,
    I tested four of the libraries from the original LuaSocket (the rest of them were using different ones I wasn't supposed to port like HTTP, FTP, SMTP, etc),

    About h_addr_list, I might change it later because the code is already functional and I might break something trying to change this (I don't expect to miss this important detail but if I do it now the stable version will probably stop being stable).



  • I think you should install the module to your luapower tree first to get a sense of the overlaid directory structure. You can do that with mgit:

    mgit clone https://github.com/starkkz/bnet
    

    You will notice that your repo files don't get cloned into their own directory but they share the root directory with all other modules. That's why it's important to name your files such that don't clash with the files of other repos. So you can't have a directory called "test" in there and you can't have a .gitignore file in the root directory either. Consider merging the test files into bnet_test.lua. If they're not automated tests and are interactive tests instead, consider naming the file bnet_demo.lua. If they are automated tests but you want to have the ability to run them separately as well, you can put them in a table and dispatch on arg[1] so you can run them individually from command line.



  • I merged the test files into bnet_test.lua, and removed the .gitignore file.

    One of the tests is interactive but it's actually easier to use the web browser to test it, I pointed that out on the script.



  • Looks great. Thank you.

    https://luapower.com/bnet

    The website pulls the repo every hour so you have to wait a bit to see the updated docs. I can increase that time if needed.



  • Few more notes on the code and more unsolicited advice :)

    • are you sure for i = 0, n_read do shouldn't be for i = 0, n_read-1 do ?
    • not sure why would a socket library have a time/sleep API in it (I know luasocket has it but I think that's lame, they should've released a time library separately) but anyway, here's a desktop-portable implementation: https://luapower.com/time
    • you don't want to iterate arrays with pairs(), use ipairs() instead, especially if you want to add the results to another array -- adding the keys of a contiguous array in random order is really painful to the poor memory allocator
    • ffi.new("unsigned int", value) gives you a constant int32 -- doing math with that results in int64 values which may go in the heap! There's seldom a need to allocate scalars with ffi.new(). Consider using plain Lua numbers instead (use math.floor() to simulate integer math if you have to).
    • in socket.protect() you wanna say that if there's no error, then return all the values right? well, you may not want people to see that implementation :) here's a better one:
    function socket.protect(func)
       local function pass(ok, ...)
          if ok then return ... end
          return nil, ...
       end
       return function(...)
          return pass(pcall(func, ...))
       end
    end
    

    There's more that I could say but I don't want you to run away screaming :) So let's talk API instead. A good I/O library IMHO should have:

    • ability to work with any file descriptors the OS supports not just sockets
    • ability to select the polling mechanism (or at least choose a good one, which is not select())
    • a cdata stream interface
    • non-blocking dns lookup -- this should probably be a separate module/package
    • no Internet protocols and hi-level stuff (these are just parsers/formatters anyway and they don't belong in a I/O library -- they should be separate libs that hook into any API that gives you read() and write()) But I'm going into rant mode so I'd better stop before it's too late :)


  • It's fine, I'm taking all the notes you're giving to me since I'm not even a computer scientist. About the first point, I copied that select function from BlitzMax's wrapper, so I'm not sure if I should substract a unit, but you're probably right. About the "unsigned int", this is just for the timeout, apparently assigning a normal integer to math.huge returned negative values so I just turned it into a "unsigned int" rather than a "int", I would use Lua numbers but the values returned from socket.udp() and socket.tcp() are actually cdata objects so I needed to give them some class.



  • Well it's gonna be tough until you get a good knack of C types and conversion rules, integer vs floating point math, bit ops, and an intuition of performance characteristics of various Lua patterns (how tables are implemented, static vs dynamic memory allocation) and even a few LuaJIT and ffi insights as well. There's a lot of ground to cover, so don't worry if you don't do a good job at first -- but be willing to scrap a lot of code and even start again from scratch potentially many times until you get it right, if you want to make a good lib (and yourself a better programmer in the process).

    Here's some of my notes (not nearly enough of course):

    https://luapower.com/luajit-notes
    https://luapower.com/api-design
    https://luapower.com/lua-tricks

    There's more tips on the Lua wiki, and of course, the PiL book which I highly recommend.



  • Hey, out of curiosity how do you update the actual page on luapower.com? I was trying to design the documentation file but it doesn't seem to apply changes on the site (and it doesn't know when the last commit was), do you need a webhook or do you do that yourself manually?



  • The website pulls the repo every hour so you have to wait a bit to see the updated docs. I can increase that time if needed. You can also add a webhook to https://luapower.com/github



  • Hey, github isn't pushing the webhook anymore because of this.
    /home/cosmin/website/www/actions.lua:1100: attempt to call field 'exec' (a nil value) stack traceback: /home/cosmin/website/www/actions.lua:1100: in function 'handler' /home/cosmin/website/www/app.lua:167: in function </home/cosmin/website/www/app.lua:149> [C]: in function 'xpcall' /home/cosmin/website/nginx/../www/ngx.lua:20: in function 'try_call' /home/cosmin/website/nginx/../www/ngx.lua:27: in function </home/cosmin/website/nginx/../www/ngx.lua:1>
    Just saying.


23
Posts
9612
Views
Log in to reply

Internal error.

Oops! Looks like something went wrong!