bundle single EXE / winapi_demo / linking problems


  • I want to create a single self-contained EXE of 'winapi_demo" and used this command line:

    \Lua\luapower> ./mgit bundle -a --all -m --all -M winapi_demo -o win_api.exe

    It seems everything compiles fine but I get a whole bunch of "multiple references" & "undefined references" errors when linking. Looks like this:

    .a bin/mingw64/vararg.a bin/mingw64/vorbis.a bin/mingw64/vorbisenc.a bin/mingw64/vorbisfile.a bin/mingw64/xxhash.a bin/mingw64/z.a -Wl,--no-whole-archive -lgdi32 -lmsimg32 -lopengl32 -lwinmm -lws2_32
    bin/mingw64/sodium.a(stream_salsa20_amd64_xmm6.o):fake:(.text+0x0): multiple definition of crypto_stream_salsa20' bin/mingw64/sodium.a(stream_salsa20_ref.o):stream_salsa20_ref.c:(.text+0x0): first defined here bin/mingw64/sodium.a(stream_salsa20_amd64_xmm6.o):fake:(.text+0x80): multiple definition ofcrypto_stream_salsa20_xor_ic'
    bin/mingw64/sodium.a(xor_salsa20_ref.o):xor_salsa20_ref.c:(.text+0x0): first defined here
    bin/mingw64/terra.a(treadnumber.o):treadnumber.c:(.text+0x7a0): multiple definition of lj_strscan_scan' bin/mingw64/luajit.a(lj_strscan.o):lj_strscan.c:(.text+0x7f0): first defined here bin/mingw64/terra.a(treadnumber.o):treadnumber.c:(.text+0x1140): multiple definition oflj_strscan_num'
    bin/mingw64/luajit.a(lj_strscan.o):lj_strscan.c:(.text+0x1330): first defined here
    bin/mingw64/terra.a(treadnumber.o):treadnumber.c:(.rdata+0x40): multiple definition of lj_char_bits' bin/mingw64/luajit.a(lj_char.o):lj_char.c:(.rdata+0x0): first defined here bin/mingw64/soundio.a(os.o):os.cpp:(.text+0x11): undefined reference to__imp_CoInitializeEx'
    bin/mingw64/soundio.a(os.o):os.cpp:(.text+0x3b): undefined reference to __imp_CoUninitialize' bin/mingw64/soundio.a(wasapi.o):wasapi.cpp:(.text+0x10b9): undefined reference to__imp_CoTaskMemFree'

    Any idea what's wrong?


  • 17
    Posts
    3085
    Views
    Log in to reply


  • Hi Robert,

    From just looking at the errors:

    • sodium.a wasn't built right (it's a WIP, and you don't need it for that demo)
    • terra.a includes luajit (maybe it shouldn't -- in any case, you don't need it for that demo)
    • soundio.a needs directshow (linking to ole32.dll and oleout32.dll should fix it)


  • Hi Cosmin, thanks for the feedback.

    "sodium.a wasn't built right (it's a WIP, and you don't need it for that demo)" - Well, I didn't do anything special.

    • Why is it included? I thought that bundle will only include things necessary for the script to be bundeled.
    • What do now? How can I remove it?

    "terra.a includes luajit (maybe it shouldn't -- in any case, you don't need it for that demo)"

    • Ok, what to do now?

    "soundio.a needs directshow (linking to ole32.dll and oleout32.dll should fix it)"

    • Ok, how to do it?
    • Shouldn't these libs be added automatically? Or am I wrong here?


  • I thought that bundle will only include things necessary for the script to be bundeled.

    Yes, but that's because of how linkers work with static libs (they choose only what's necessary and discard the rest). The script itself includes all the static libs wuth -a --all.

    What do now? How can I remove it?

    Instead of passing --all you pass only the libs that you need.

    Shouldn't these libs be added automatically? Or am I wrong here?

    The script adds a few common windows libs to the cmdline as you can see in the output, but it doesn't add everything (Windows has thousands of libs). You can add these with the -d switch (type mgit bundle to see all the switches).



  • Oh, alternatively, you can remove the libs that you don't need from your luapower installation (mgit remove <package>) so that you can continue to use --all.



  • "Yes, but that's because of how linkers work with static libs (they choose only what's necessary and discard the rest). The script itself includes all the static libs wuth -a --all."

    Ah, that was my missing link. I thought it works like this:

    1. Scan the sources and collect all REQUIRE (with meta data bout necessary link libs etc.) and create something like an internal "assembled single big script"

    2. Bundle this script with all necessary libs etc. into an EXE. And here the compiler linker will only include used stuff from the provided libs.

    Wouldn't this workflow be possible? IMO that would be a nice toolchain.

    If anyone want to do it, let me know, I'm open to sponsor such a thing.



  • Yes, it's possible but you need to parse the PE/ELF/MACH file formats to get the binary dependency tree. Such feature could be added to https://luapower.com/luapower which already does a lot of metadata extraction and then the bundle script would just use that information.

    (I'm not sure it's worth the effort though, I mean when you make an app, you know what libs you use, the --all switch is mostly useful to check for errors and conflicts between libs as you found out yourself)



  • Hmm... still trying to make my way through BUNDLE.

    D:\develop\Lua\luapower> ./mgit bundle -v -m "dasm.lua dasm_x64.lua nw.lua" -M nw_demo -o nw.exe
    Bundle parameters:
    Platform: mingw (mingw64)
    Modules: bundle_loader dasm.lua dasm_x64.lua nw.lua
    Static libs: luajit
    Dynamic libs: gdi32 msimg32 opengl32 winmm ws2_32
    Main module: nw_demo
    Icon: csrc/bundle/luapower.ico
    Compiling modules...
    icon csrc/bundle/luapower.ico
    manifest bin/mingw32/luajit.exe.manifest
    Linking nw.exe...
    g++ .bundle-tmp/mingw64/_icon.o .bundle-tmp/mingw64/_manifest.o .bundle-tmp/mingw64/bundle_loader.lua.o .bundle-tmp/mingw64/dasm.lua.o .bundle-tmp/mingw64/dasm_x64.lua.o .bundle-tmp/mingw64/nw.lua.o .bundle-tmp/mingw64/csrc/bundle/bundle.c_nw_demo.o .bundle-tmp/mingw64/csrc/bundle/luajit.c_nw_demo.o -o nw.exe -static -static-libgcc -static-libstdc++ -Wl,--export-all-symbols -Wl,--whole-archive bin/mingw64/luajit.a -Wl,--no-whole-archive -lgdi32 -lmsimg32 -lopengl32 -lwinmm -lws2_32
    Done.
    D:\develop\Lua\luapower> ./nw.exe
    D:\develop\Lua\luapower\nw.exe: D:\develop\Lua\luapower\dasm.lua:99: cannot resolve symbol 'dasm_init': The specified procedure could not be found.

    Do I have to find out by trial & error, which modules I need to add (-m switch), which static libs (-a switch) and which dynmic libs (-d switch)? How can I derive what I need from my source?

    I try to bundle the nw_demo.lua script. How can I find out what modules it needs etc.?



  • Ok, managed to get a starting EXE with this line:

    ./mgit bundle -v -a 'dasm_x86 z png freetype pixman cairo' -M nw_demo -o nw.exe

    Seems I only need to add the static libs required.

    However, clicking wild in the window I get this crash:

    D:\develop\Lua\luapower> ./nw.exe
    D:\develop\Lua\luapower\winapi\spi.lua:217: Invalid system-wide (SPI_
    ) parameter.

    stack traceback:
    [C]: in function 'error'
    D:\develop\Lua\luapower\winapi\util.lua:102: in function 'checknz'
    D:\develop\Lua\luapower\winapi\spi.lua:217: in function 'SystemParametersInfo'
    D:\develop\Lua\luapower\nw_winapi.lua:1278: in function 'wheel_scroll_chars'
    D:\develop\Lua\luapower\nw_winapi.lua:1283: in function 'handler'
    D:\develop\Lua\luapower\winapi\basewindowclass.lua:470: in function <D:\develop\Lua\luapower\winapi\basewindowclass.lua:457>
    [C]: in function 'xpcall'
    D:\develop\Lua\luapower\winapi\basewindowclass.lua:81: in function <D:\develop\Lua\luapower\winapi\basewindowclass.lua:80>
    [C]: in function 'DispatchMessage'
    D:\develop\Lua\luapower\winapi\basewindowclass.lua:119: in function 'ProcessMessage'
    D:\develop\Lua\luapower\winapi\basewindowclass.lua:135: in function 'MessageLoop'
    D:\develop\Lua\luapower\nw_winapi.lua:82: in function 'run'
    D:\develop\Lua\luapower\nw.lua:235: in function 'run'
    D:\develop\Lua\luapower\nw_demo.lua:36: in function 'm'
    .\bundle_loader.lua:58: in function <.\bundle_loader.lua:8>
    [C]: at 0x00403331
    D:\develop\Lua\luapower\nw.exe: D:\develop\Lua\luapower\winapi\basewindowclass.lua:85: cannot convert 'nil' to 'int'
    D:\develop\Lua\luapower\cairo.lua:69: refcount of cdata<struct _cairo >: 0x0283caf0 is 43245167, should be 0
    stack traceback:
    [C]: in function 'error'
    D:\develop\Lua\luapower\cairo.lua:69: in function 'free'
    D:\develop\Lua\luapower\nw.lua:1484: in function '_backend_free_bitmap'
    D:\develop\Lua\luapower\nw_winapi.lua:1337: in function '_free_bitmap'
    D:\develop\Lua\luapower\nw_winapi.lua:227: in function 'handler'
    D:\develop\Lua\luapower\winapi\basewindowclass.lua:470: in function <D:\develop\Lua\luapower\winapi\basewindowclass.lua:457>
    [C]: in function 'xpcall'
    D:\develop\Lua\luapower\winapi\basewindowclass.lua:81: in function <D:\develop\Lua\luapower\winapi\basewindowclass.lua:80>
    [C]: in function 'DestroyWindow'
    D:\develop\Lua\luapower\winapi\window.lua:118: in function <D:\develop\Lua\luapower\winapi\window.lua:116>



  • When I try to create a really standalone EXE it seems that I have to include the *.lua modules as well. Is that correct?

    At the moment I use a trail & error approach to find out which lua modules to add. Is there are more structured way to do it?

    D:\develop\Lua\luapower> ./mgit bundle -v -a 'dasm_x86 z png freetype pixman cairo' -m 'nw.lua nw_winapi.lua glue.lua box2d.lua time.lua cbframe.lua cbframe_x86_h.lua dynasm.lua' -M ../nw_test -o ../nw_test.exe
    Bundle parameters:
      Platform:       mingw (mingw64)
      Modules:        bundle_loader nw.lua nw_winapi.lua glue.lua box2d.lua time.lua cbframe.lua cbframe_x86_h.lua dynasm.lua
      Static libs:    luajit dasm_x86 z png freetype pixman cairo
      Dynamic libs:   gdi32 msimg32 opengl32 winmm ws2_32
      Main module:    ../nw_test
      Icon:           csrc/bundle/luapower.ico
    Compiling modules...
      icon            csrc/bundle/luapower.ico
      manifest        bin/mingw32/luajit.exe.manifest
    Linking ../nw_test.exe...
    g++ .bundle-tmp/mingw64/_icon.o .bundle-tmp/mingw64/_manifest.o .bundle-tmp/mingw64/bundle_loader.lua.o .bundle-tmp/mingw64/nw.lua.o .bundle-tmp/mingw64/nw_winapi.lua.o .bundle-tmp/mingw64/glue.lua.o .bundle-tmp/mingw64/box2d.lua.o .bundle-tmp/mingw64/time.lua.o .bundle-tmp/mingw64/cbframe.lua.o .bundle-tmp/mingw64/cbframe_x86_h.lua.o .bundle-tmp/mingw64/dynasm.lua.o .bundle-tmp/mingw64/csrc/bundle/bundle.c_../nw_test.o .bundle-tmp/mingw64/csrc/bundle/luajit.c_../nw_test.o -o ../nw_test.exe -static -static-libgcc -static-libstdc++ -Wl,--export-all-symbols -Wl,--whole-archive bin/mingw64/luajit.a bin/mingw64/dasm_x86.a bin/mingw64/z.a bin/mingw64/png.a bin/mingw64/freetype.a bin/mingw64/pixman.a bin/mingw64/cairo.a -Wl,--no-whole-archive -lgdi32 -lmsimg32 -lopengl32 -lwinmm -lws2_32
    Done.
    

    But running the EXE out of the luapower folder gives this:

    D:\develop\Lua\luapower> ..\nw_test.exe
    D:\develop\Lua\nw_test.exe: module 'cbframe_x86' not found:
            no field package.preload['cbframe_x86']
            no file 'D:\develop\Lua\cbframe_x86.lua'
            no file 'D:\develop\Lua\cbframe_x86\init.lua'
            no file 'D:\develop\Lua\clib\cbframe_x86.dll'
            no symbol 'Blua_cbframe_x86'
            no symbol 'luaopen_cbframe_x86'
            no file 'D:\develop\Lua\cbframe_x86.dasl'
            no file 'D:\develop\Lua\cbframe_x86\init.dasl'
    D:\develop\Lua\luapower>


  • Yes, you have to include the .lua modules too. The full list of dependencies is on the website, and you can also use the luapower command to get them.

    I understand that you're trying to get to an exe as quick as possible so that you can validate that the thing works and seeing that you have to know exactly what you need to put in is frustrating. But the thing is that bundling is really a late step in the dev process -- by the time you need to bundle your app, you'll know your dependencies very intimately.



  • I finally made it!! And used UPX to compact the whole thing. Now it's about 2MB with cairo etc. Very cool!

    However, some more questions:

    1. I'm now including the Lua modules via -b not -m and assume these will include them as (pre-compiled?) binary. Is this right, or what is the difference?

    2. Would it be possible to get back my Lua script out of the EXE bundle? Or is everything pre-compiled (tokenized) inside the bundle?



    1. -b is for including binaries that you then load via the bundle module. You won't be able to just require() those.
    2. .lua modules (added with -m) are compiled so you can't get their source code (but you can -m them and also -b them to get the source code).


  • @cosmin said:

    1. -b is for including binaries that you then load via the bundle module. You won't be able to just require() those.

    Ok. Not sure that I understand everything but I now used -b and it seems to work as well.

    1. .lua modules (added with -m) are compiled so you can't get their source code (but you can -m them and also -b them to get the source code).

    I don't get the part in (...) So, when I use -m AND -b for *.lua files, the code is included?



  • -m compiles the module and you can load it via require()
    -b adds any file as a binary blob which you can then load as a string via bundle.load()



  • I'm still having problems as it looks. When I create an exe and let it run it works if luapower folder is below the EXE:

    nw_demo.exe
    luapower/...
    

    Same for winapi_demo.lua I used this line to bundle it:

    ./mgit bundle -v -a 'dasm_x86 z png freetype pixman cairo' -m 'nw.lua nw_winapi.lua glue.lua bo
    x2d.lua time.lua pp.lua ffi_reflect.lua cbframe.lua cbframe_x86.dasl cbframe_x86_h.lua dynasm.lua dasm.lua dasm_mm.lua b
    itmap.lua winapi.lua winapi/*.lua cairo.lua cairo_h.lua' -M ../winapi_demo -o ../winapi_demo.exe
    

    But when I copy the EXE to some other place and run it, I get:

    C:\Users\robby\Desktop> .\winapi_demo.exe
    LuaJIT 2.1.0-beta2 -- Copyright (C) 2005-2016 Mike Pall. http://luajit.org/
    JIT: ON SSE2 SSE3 SSE4.1 fold cse dce fwd dse narrow loop abc sink fuse
    >
    

    It looks like there are some files loaded when the EXE is run, but I don't know wich ones.



  • Ok, found it out. The problem was that the main module given after the -M switch (without .lua extension), needs to be included in the -m list of Lua modules (with .lua extension) that get compiled into the EXE.


17
Posts
3085
Views
Log in to reply

Internal error.

Oops! Looks like something went wrong!