Compiling ICU with Visual Studio 2013

In my previous post on how to build Qt on Windows, I explained how to build Qt for Windows, using Visual Studio 2010 and prebuilt ICU libraries. If we want to build Qt with Visual Studio 2013, we’ll need to build ICU ourself, and here’s how :

  • Download ICU sources from here : http://site.icu-project.org/download
  • Unzip in C:\ (you will have a C:\icu folder with C:\icu\readme.html (among others))
  • Go into C:\icu\source\common\unicode, edit platform.h and add the following somewhere at the beginning. It’s needed if you want to build Qt :
#define U_CHARSET_IS_UTF8 1
  • Go into C:\icu and create a build.bat file, with the following content and run it :
@echo off

::
:: setup Visual Studio 2013 environment for x64 builds
::
call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" x86_amd64

::
:: build
::
msbuild source\allinone\allinone.sln /m /target:Build /property:Configuration=Release;Platform=x64
msbuild source\allinone\allinone.sln /m /target:Build /property:Configuration=Debug;Platform=x64

This will build ICU in x64. You can run the same script again and remove the “x86_amd64″ argument to the vcvarsall.bat script, and change the Platform property to Win32 to build ICU in x86.

Building Qt from sources on Windows

Building Qt is not in itself difficult. But building Qt on Windows, and with QtWebkit can become quite the challenge. So here is a little tutorial on how I did it.

Configuration

First of all, you’ll need a lot of stuff correctly installed. Keep in mind that the paths to the following tools should appear in your PATH env var. Some installers will update it for you, for others you’ll need to add it personally.

So the first thing you’ll want to install is this great tool : Rapid Env Editor. Be careful though, the big Download button on the download page is NOT the one you want to click ! You don’t have to go to the download section, you have download links in the header on the upper right :) (I made the mistake, thus this warning)

Now the list of stuff you to install :

Now, time to get the sources / libraries :

  • Qt (Get the source zip)
  • ICU (Get the package corresponding to your compiler. Currently only Visual Studio 2010, you’ll need to build from sources to get a version corresponding to Visual Studio 2013)

Extract Qt and ICU somewhere on your disk. I recommend you to put that in a short path, such as C:\Qt and C:\ICU. See Troobleshooting section for the reason why :)

Building

Once everything’s installed and the paths to the tools in your PATH env var, you’re ready to build.

Go into the Qt sources root folder, then create a file name build_Qt_x64.bat (for instance) and copy the following into it :

@echo off

::
:: Remember the source path (where the Qt sources are, e.g. where this file is)
::
set sourcepath=%cd%

::
:: The following part should be updated to reflect your build environment
::

:: this will setup Visual Studio so that we can use it in command line
call "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86_amd64

:: where we want to install Qt
set installpath=C:\Qt_x64

:: set the path where icu's lib was installed
set icupath=C:\icu
set icusuffix=64

::
:: Setup the configuration
::

set configuration= -opensource -confirm-license -debug-and-release
set configuration= %configuration% -prefix "%installpath%"
set configuration= %configuration% -c++11 -mp -opengl desktop
set configuration= %configuration% -icu -I "%icupath%\include" -L "%icupath%\lib%icusuffix%"
set configuration= %configuration% -no-compile-examples -nomake tests -nomake examples -no-accessibility
set configuration= %configuration% -skip qtwebkit-examples

::
:: Cleanup previous install
::

if exist "%installpath%" ( echo Removing "%installpath%" )
if exist "%installpath%" ( rmdir /Q /S "%installpath%" )
if not exist "%installpath%" ( echo Creating "%installpath%" )
if not exist "%installpath%" ( mkdir "%installpath%" )

::
:: Update the path with a few access to dlls, tools, etc.
::

set path=%installpath%\qtbase\lib;%path%
set path=%icupath%\bin%icusuffix%;%path%
set path=%sourcepath%\GnuWin32\bin;%path%

::
:: Configure.
::

pushd "%installpath%"
"%sourcepath%\configure" %configuration% -platform win32-msvc2010

::
:: And build
::

:: note : on Windows, the configure script exit, so the following needs to be done manually. It's here for reference.
nmake
nmake install
nmake clean

popd

Now you can build by opening a command line on the root folder, and by typing the following :

build_Qt_x64.bat
nmake
nmake install
nmake clean

The first line is quite fast (a few minutes) and the second can take a whole night :) The 2 last are quite long too, but it’s reasonable (a few hours max)

Troubleshooting

Hopefully everything will work well, but here are a few problems that you might encounter. I’ll update this part if needed.

fatal error U1095

One of the compilation command line is too long. The solution (which worked for me) is to ove the Qt and ICU to short folders, such as (for instance) C:\Qt and C:\ICU

fatal error U1077

This has to do with ICU. You shouldn’t get this if you’re building with Visual Studio 2010. If you’re using Visual Studio 2012 or 2013, see this post about building ICU (thanks Mihai :p)

Release x64 crashes when all other versions work

This one I discovered very recently, and updated the build script. It’s due to a bug in the Visual Studio 2010 compiler for x64, when you use link time code generation (the -ltcg option) So the solution is to not use this option :)

References

Scope based execution of arbitrary code with C++11

Imagine the following peace of code (which is really simple for the sake of the example :)

for (entity in entities)
{
    glPushMatrix();

    // do some stuff

    glPopMatrix();
}

The basic idea here is that you need to maintain the integrity of a states’ stack in a loop. But what happens if you have some tests that allow to break, continue or return :

for (entity in entities)
{
    glPushMatrix();

    // do some stuff

    if (entity->NeedToDoOtherStuff() == false)
    {
        continue;
    }

    // do some other stuff

    glPopMatrix();
}

Wrong. You didn’t pop the matrix, everything get’s corrupted, so you need to add a glPopMatrix() right before continue. And the more of those, the more glPopMatrix you need to add. And if you have more states to push / pop, you need to duplicate those too.

This scenario happens a lot in real applications. If you allocate some temporary data to work with, you need to cleanup before exiting, and if you have many exit points (error checks, etc.) you end up duplicating cleanup code everywhere, which is error prone and fastidious, or using things like goto’s.

Here is a little templated class which solves this problem using 2 features of C++11 : lambdas and functional.

class Cleanup
{
public:
    template< typename T >
    Cleanup(const T & code)
    {
        m_Code = code;
    }
    ~Cleanup(void)
    {
        m_Code;
    }
private:
    std::function< void (void) > m_Code;
};

This class can be used like this :

for (entity in entities)
{
    glPushMatrix();
    Cleanup popMatrix([]() { glPopMatrix(); });

    // do you stuff, your continue's, break's, etc. without breaking
    // things anymore :)
}

As soon as the Cleanup instance goes out of scope, it will execute the code that you gave it. I find this particularly useful :

  • no longer crashes when you add an exit point, special case, etc. and forget to copy / paste the cleanup code.
  • the cleanup code is defined right at the top of the used scope, which in my experience makes it really hard to forget updating it, if you need to add more states, more intermediate data that need cleanup, etc.

 

Userfriendly Qt5 Types in Visual Studio Debugger

If you’re using Visual Studio 2012 or 2013 Express with Qt, you can’t use the Qt Addin, but you might still want to have user friendly Qt types showing in your debugger. Luckily this is quite easily achieved :

  1. Go on qt-labs git repository here and download the qt5.natvis file.
  2. Make sure you have the following folder : <MyDocuments>\Visual Studio 2013\Visualizers and if not, create it.
  3. Copy the qt5.natvis file in it, and restart Visual Studio.
  4. Done.

Build boost as static libs for Visual Studio 2012

Following is a small script that I use to build a few static Boost libraries in 64 bits for Visual Studio 2012, and with iterator debugging turned off.

:: build the bjam and b2 exe
if not exist "b2.exe" (
    bootstrap.bat
)

:: build boost with visual 2012 toolset.
::   - static lib
::   - multithreaded shared runtime
::
:: it only builds signals and filesystem
b2.exe -a -d0 -j4 toolset=msvc-11.0 address-model=64 variant=debug link=static threading=multi runtime-link=shared define=_HAS_ITERATOR_DEBUGGING=0 --with-signals --with-filesystem
b2.exe -a -d0 -j4 toolset=msvc-11.0 address-model=64 variant=release link=static threading=multi runtime-link=shared define=_HAS_ITERATOR_DEBUGGING=0 --with-signals --with-filesystem

Multiple commands in Sublime Build System

A little note on the build system integrated in Sublime. If you have multiple commands to execute during a build (for instance minifying your js code, and generating the documentation) your can do it quite easily on Windows, but it took me a while to find the equivalent on Mac, so here is the relevent (simplified version) part of my build system:

"windows": {
    "shell": true,
    "cmd": [
        "python", "build.py", "compress", "&;"
        "python", "build.py", "doc"
    ]
},
"osx": {
    "shell": false,
    "cmd": [
        "sh", "-c", "python build.py compress && python build.py doc"
    ]
}

Notice the difference between Windows and OSX. On Windows, you simply add an additional “&;” parameter which will separate the commands, but on OSX (I guess it would be the same on Linux) it doesn’t work. The only way I found to execute multiple commands was to directly call sh with the -c option and put all my commands in a single string. The commands are then separated by a && sequence.

Oh and also shell must be true on Windows, but false on OSX, else nothing runs.

Application-wide shortcuts with wxWidgets (on Windows)

Another programming note, this time about wxWidgets and application wide shortcuts. Before going further, I want to state that I’m using the MSW version of wxWidgets. So maybe Gtk or other implementations don’t suffer from the same problem at all. In any case, keep in mind that I’m talking about the MSW implementation of wxWidgets :)

In wx, you can have “window wide” shortcuts through accelerator tables or system wide shortcuts through RegisterHotKey. Problems with accelerator tables is this one: you have a main window with such an accelerator table. Then you add a tool frame, make it floatable. Then the shortcuts stop working if your tool frame is floating and has the focus. It works fine if the tool frame is docked though … And system wide shortcuts should not be used for obvious reasons.

On top of that, there is a big flaw in the way events get processed in wxWidgets. Let’s say you want to add a shortcut on the keyboard key ‘t’ (for translation, for instance) Hell begins. Why ? Because now you can’t enter the letter ‘t’ in any text control of your application: the accelerator table catches it before the text control and process it as a shortcut …

Some might say “don’t you ‘t’, use ‘alt+t’. Yeah right: 3DS Max uses ‘w’ for translation, and you can still write a ‘w’ in the script editor window, obviously… And I don’t want to make a list of all the software that use single letters with no modifier as shortcuts, and still allow using those letter in scripting window, that would be too long. Just to name a few: 3DS Max, Maya, Blender, Photoshop, Visual Studio… well, basically every professional software out there.

Anyway, there is no way to have an application wide shortcut that doesn’t screw text controls, and work also when using floating panels, so enough writing, let’s add such a system !

The first thing is to add a shortcut system to your application:

class Application : public wxApp
{
public:

    //! application init
    virtual bool OnInit();

    //! application uninit
    virtual int OnExit();

    //! register an application wide shortcut
    template< typename T >
    bool RegisterShortcut(int modifier,
                          int key,
                          int id,
                          const T& fctor);

private:

    //! this is where we will filter events before wx can process them
    void OnCharHook(wxKeyEvent& evt);

    //! callback type for application wide events
    typedef boost::function CallbackType;

    //! a shortcut
    struct Shortcut
    {
        //! ctor
        Shortcut(int m, int k, int id, const CallbackType& c);
        //! modifier
        int Modifier;
        //! key code
        int KeyCode;
        //! the wx id
        int ID;
        //! callback
        CallbackType Callback;
        //! equality operator
        bool operator == (const SShortcut& o);
        //! equality operator with an accelerator entry
        bool operator == (const wxAcceleratorEntry& value);

    };

    //! the registered application wide shortcuts
    std::vector m_Shortcuts;
};

template
bool
Application::RegisterShortcut(int modifier,
                              int key,
                              int id,
                              const T& fctor)
{
    // check if the shortcut is not already registered
    Shortcut shortcut(modifier, key, id, fctor);
    if (std::find(m_Shortcuts.begin(), m_Shortcuts.end(), shortcut) != m_Shortcuts.end())
    {
        return false;
    }
    m_Shortcuts.push_back(shortcut);
    return true;
}

inline
Application::Shortcut::SShortcut(int m, int k, int id, const SCallback& c)
    : Modifier(m)
    , KeyCode(k)
    , ID(id)
    , Callback(c)
{
}

inline
bool
Application::Shortcut::operator == (const Shortcut& o)
{
    return o.Modifier == Modifier && o.KeyCode == KeyCode;
}

inline
bool
Application::Shortcut::operator == (const wxAcceleratorEntry& value)
{
    return KeyCode == tolower(value.GetKeyCode()) && Modifier == value.GetFlags();
}

The idea is pretty simple: I created a SShortcut structure which hold a key code, a modifier, a wx id and a callback object. Thanks to boost::function and boost::bind and the templated function to register a shortcut, you can set this to almost anything: a free function, a method of an object, even a lambda if your compiler supports it (well, at least in Visual Studio 2010, it works)

Now the interesting part: how we handle those to avoid conflicts with text controls, and how do we manage to catch those shortcuts even if the focused window is a floating frame ? Like this:

bool
Application::OnInit()
{
    // bind on the char hook event. This event is sent before any
    // other, so we can catch it to intercept our shortcuts
    Bind(wxEVT_CHAR_HOOK, &Application::onCharHook, this);
}

bool
Application::OnExit()
{
    Unbind(wxEVT_CHAR_HOOK, &Application::onCharHook, this);
}

void
Application::OnCharHook(wxKeyEvent& evt)
{
    // dummy shortcut used for search
    Shortcut s(evt.GetModifiers(),
               evt.GetKeyCode(),
               -1,
               CallbackType());
    std::vector::iterator shortcut
        = std::find(m_Shortcuts.begin(), m_Shortcuts.end(), s);

    // this is where we do the job: if we found a shortcut, we need
    // to make sure of a few things. First, that the application has
    // the focus. Then, that the focused control is not a text editing
    // one.
    wxWindow * focused = wxWindow::FindFocus();
     if(shortcut != m_Shortcuts.end()
       && focused
       && !dynamic_cast< wxTextCtrl * >(focused)
       && !dynamic_cast< wxStyledTextCtrl * >(focused))
    {
        // ok, we're clear, we can execute the callback of the shortcut !
        wxCommandEvent e;
        e.SetId(shortcut->ID);
        shortcut->callback(e);
    }
    else
    {
        // let go of the event. If a text control has the focus, let
        // it handle it, etc.
        evt.Skip();
    }
}

So now, imagine that you have an application with a floating panel, in this floating panel you can just do that:

Application * app = static_cast< Application * >(wxApp::GetInstance());
app->RegisterShortcut(wxACCEL_NORMAL,
                      (int)'T', 
                      AnID,
                      boost::bind(&PanelClass::OnTranslate, this, _1));

And here you have it. ‘t’ is now a shortcut (in my case for ‘translation’) This shortcut will work from anywhere in the application, even from a floating frame, and more importantly, will still let you type the letter ‘t’ in text controls.

Note: AnID is the event id which will be sent to the OnTranslate method of the PanelClass class.

Going Further

I tried to keep this article short, so I simplified a lot of things. One of those is how I do when I try to register a shortcut that is already registered by some other part of my application. Their are a lot of ways to take care of that: store the wxWindow pointer which registered the shortcut, then in the OnCharHook you can check if the focused window is one of those, you can also add a simple priority order to the registration. With those 2 you’ll be able to call the correct shortcut whatever the situation.

There is also 1 pitfall with this approach: when you don’t want to make a shortcut application wide, the simplest way is to use accelerators. But with this method, if a focused panel has an accelerator on the letter ‘t’, it’s still the application wide shortcut that will be called. To overcome this, you need to get the focused window, go up its hierarchy, and for each window, get the accelerator table and test it against you current event. This part is a bit hacky on Windows, unfortunately. The “cleanest” way I found was to simply store in the wxAcceleratorTable the array of entries used to create the table (I had to modify the sources of wx) Maybe I’ll write another post about that.

Add scripting to your tool with Lua

It’s been a while since I last wrote a programming article! I’m working on a big visual fx tool for my company, and I recently started to think that it would be nice to be able to script it. For example, create a small script to convert all texture paths of the effect from absolute to relative, etc.

And I found out that it’s really simple. I decided to use Lua which I (re)discovered through the wonderfull Premake project. So first thing first, go to Lua’s website and download it. Then just create a simple console application, and add all the .c files found in lua/src to your project, except lua.c and luac.c !

For my purpose, I just need to be able to execute some Lua, and access my tool’s functionalities from it. The first step is really simple:

// init lua
lua_State* lua = luaL_newstate();
// load the default libs (io, etc.)
luaL_openlibs(lua);

// then you can load and execute an existing script:
luaL_dofile(lua, "test.lua");
// or directly execute something:
luaL_dostring(lua, "io.write(\"Hello World!\")");

// and close
lua_close(lua);

Yes, it’s that simple.

Then the second part is allowing a Lua script to call a C function. This is also pretty simple:

// our function. All functions that will be called from Lua
// need to have the same signature.
int test(lua_State* lua)
{
    // since lua allow calling a function with variable arguments,
    // get the number of arguments it was called with.
    int num_args = lua_gettop(lua);

    // do some tests (we could also just print a message and return)
    // note: arguments are retrieved from the lua state object. Since the
    // very first value is the number of arguments, the arguments start
    // at index 1.
    assert(num_args >= 1 && num_args <= 2);
    assert(lua_isnumber(lua, 1));

    // get the first parameter
    int a = lua_tointeger(lua, 1);

    // check if we need to print
    if(num_args == 2 && lua_isboolean(lua, 2) && lua_toboolean(lua, 2))
    {
        std::cout << "result: " << a * a << std::endl;
    }

    // push as many return values that you want, and return the number of
    // values that you pushed. In our case it's only 1, but it could be
    // really powerful: need to return a list of selected items ? Just
    // push all of them and return the number of items.
    lua_pushnumber(lua, a * a);
    return 1;
}

// somewhere else, start Lua
lua_State* lua = luaL_newstate();
luaL_openlibs(lua);

// push our function on top of the lua state object
lua_pushcfunction(lua, testFunction);
// then make this function globally accessible through the name "test"
lua_setglobal(lua, "test");

// call our function. That's all :)
luaL_dostring(lua, "test(10, true)");
luaL_dostring(lua, "io.write(test(10))");

// close lua
lua_close(lua);

Now if you want to go further, here are a few links:

  • Reference: The Lua 5.2 reference manual.
  • LuaJIT: From what I understood, this is a replacement of the default Lua compiler/VM that is a lot faster and more efficient. I didn’t have time to investigate but it seems really good. Only drawback is that it’s Lua 5.1 only.

Petzl RocTrip China 2011

Une fois n’est pas coutume, de l’escalade ! Excellente vidéo, des images superbes, montée avec beaucoup d’humour, et une bande son particulièrement à mon goût :) Ca donne envie tout ça … :p

[youtube]http://www.youtube.com/watch?v=_STfWe3u53Q[/youtube]

Weeee !!!

Une petite vidéo de ce qu’il se passe les samedi après-midi au Parc Olympique de Montréal :) Pas de son, pas de montage, juste quelques clips mis bout à bout.

[vimeo]http://vimeo.com/35464747[/vimeo]