Eina_Log customizations

Introduction to Eina_Log:

EFL provides a powerful logging system called “eina_log”, it can help developers and QA engineers a lot by providing key features:

  • Levels/Priorities with option to disable at compile time: similar to syslog, one can specify levels such as critical, error, warning, information or debug. An added bonus is that you can compile-out messages above certain levels, removing the function call overhead and the strings from the binary.
  • Logging domains: one can create multiple domains aside from the global/default one, this allows for easy filtering when debugging or looking for errors. Let’s say you want debug Edje, just export EINA_LOG_LEVELS="edje:4"
  • Comprehensive, Colorful default formatting: it will automatically turn on colors if your $TERM is supported (also works on Windows). Output includes PID, log domain, file name, line number and even thread ID if that’s different from the main thread. Some of the components can be toggled with environment variables.
The benefits of writing with eina_log instead of adding printf() is that you can more easily filter the output and you can keep the debug information there, without extra impacts. Often people that add printf() will forget them and pollute everybody’s else output, eventually clashing with user-output (if printf()/stdout and not fprintf()/stderr). Or they remove and when someone else run into similar bugs and needs to walk the code and think all over again on where to add logs to figure out the problem. Then it’s best that the developer who writes the code leaves proper debug messages with key variables/values, for future debug help.

Although widely documented and with examples at http://docs.enlightenment.org/auto/eina/group__Eina__Log__Group.html some still don’t use it due lack of knowledge, particularly on how to customize the output. They end writing new systems, such as Tizen did with “dlog“.

Customizing Eina_Log output:

The application can specify its own logging function with eina eina_log_print_cb_set(function, function_data). By default it will use eina_log_print_cb_stderr(), but eina_log_print_cb_stdout() and eina_log_print_cb_file() are available. We’ll explore the options by writing our own functions (all source hosted at http://barbieri-playground.googlecode.com/svn/efl-tests/eina-log/)

Simple Log to Stderr:

Let’s start with a simple test that just prints to stderr without fancy formatting. Each line is the log level followed by the actual message:

#include <Eina.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    fprintf(stderr, "%d: ", level);
    vfprintf(stderr, fmt, args);
    putc('\n', stderr);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}

Output:

$ gcc -o eina-log-simple eina-log-simple.c `pkg-config --cflags --libs eina`
$ ./eina-log-simple
1: Hi there: 1234

It starts as usual initializing Eina, then sets our function to be used as print callback, then uses the standard macro that logs to the global domain. Our function is also pretty simple and easy to understand.

Simple Log just our domain to Stderr:

How about if we just want the simple output for our application domain, leaving all others with Eina’s standard eina_log_print_cb_stderr()? This was required by edje_cc tool. It’s just about using the const Eina_Log_Domain *d parameter!

#include <Eina.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
   if ((!d->name) || (strcmp(d->name, "mydomain") != 0))
     eina_log_print_cb_stderr(d, level, file, fnc, line, fmt, NULL, args);
   else
     {
        fprintf(stderr, "%d: ", level);
        vfprintf(stderr, fmt, args);
        putc('\n', stderr);
     }
}

int main(int argc, char *argv[])
{
   int log_domain;

   eina_init();
   eina_log_print_cb_set(_my_log_cb, NULL);

   log_domain = eina_log_domain_register("mydomain", NULL);

   EINA_LOG_ERR("Hi there: %d", 1234);
   EINA_LOG_DOM_ERR(log_domain, "Just for domain: %x", 0xff00);

   eina_shutdown();
   return 0;
}

Output:

$ ./eina-log-simple-domain
ERR: eina-log-simple-domain.c:34 main() Hi there: 1234
1: Just for domain: ff00

As one can see, just “mydomain” is printed in a simpler way, other domains are still using Eina’s standard output.

Log to Syslog:

If you’re writing a key component of some platform, such as the phone dialer, you may want to log it to syslog. It may also be useful in some embedded systems. That’s as simple:

#include <Eina.h>
#include <syslog.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    int priority;
    switch (level) {
     case EINA_LOG_LEVEL_CRITICAL:
        priority = LOG_CRIT;
        break;
     case EINA_LOG_LEVEL_ERR:
        priority = LOG_ERR;
        break;
     case EINA_LOG_LEVEL_WARN:
        priority = LOG_WARNING;
        break;
     case EINA_LOG_LEVEL_INFO:
        priority = LOG_INFO;
        break;
     case EINA_LOG_LEVEL_DBG:
        priority = LOG_DEBUG;
        break;
     default:
        priority = level + LOG_CRIT;
    }
    vsyslog(priority, fmt, args);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}

Output:

Oct 27 16:20:18 solid eina-log-syslog[25582]: Hi there: 1234

As simple as you wondered, just noticed that we are converting Eina’s level to syslog priorities.

Log to Systemd’s Journal

Systemd will be used in Tizen, then it would be nice to have Eina to log to its powerful Journal system. Lennart wrote a nice article systemd for Developers III where he explains how to do it and the benefits. Looking at sd-journal.h we can see how to do it:

#include <Eina.h>
#include <systemd/sd-journal.h>

/* http://0pointer.de/blog/projects/journal-submit */
/* gcc -o test test.c `pkg-config --libs --cflags eina libsystemd-journal` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    char filestr[PATH_MAX + sizeof("CODE_FILE=")];
    char linestr[128];
    int priority;
    switch (level) {
     case EINA_LOG_LEVEL_CRITICAL:
        priority = LOG_CRIT;
        break;
     case EINA_LOG_LEVEL_ERR:
        priority = LOG_ERR;
        break;
     case EINA_LOG_LEVEL_WARN:
        priority = LOG_WARNING;
        break;
     case EINA_LOG_LEVEL_INFO:
        priority = LOG_INFO;
        break;
     case EINA_LOG_LEVEL_DBG:
        priority = LOG_DEBUG;
        break;
     default:
        priority = level + LOG_CRIT;
    }

    snprintf(filestr, sizeof(filestr), "CODE_FILE=%s", file);
    snprintf(linestr, sizeof(linestr), "CODE_LINE=%d", line);
    sd_journal_printv_with_location(priority, filestr, linestr, fnc, fmt, args);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}

Output (json-pretty)

$ ./eina-log-journal
$ journalctl -o json-pretty
{
        "__CURSOR" : "s=3c7eaf66e4734bb286e54efd20bb9f8d;i=645;b=c6681f0bd53d41d4ba682e12021095e7;m=23487887b;t=4cd0e8d5c0bf3;x=ab7f934fdb40485f;p=system.journal",
        "__REALTIME_TIMESTAMP" : "1351362291698675",
        "__MONOTONIC_TIMESTAMP" : "9471232123",
        "_BOOT_ID" : "c6681f0bd53d41d4ba682e12021095e7",
        "MESSAGE" : "Hi there: 1234",
        "PRIORITY" : "3",
        "CODE_FILE" : "eina-log-journal.c",
        "CODE_LINE" : "50",
        "CODE_FUNC" : "main",
        "_TRANSPORT" : "journal",
        "_PID" : "25655",
        "_UID" : "500",
        "_GID" : "500",
        "_SOURCE_REALTIME_TIMESTAMP" : "1351362291698325",
        "_MACHINE_ID" : "b84a5d286a7bfe411836d69100004cf6",
        "_HOSTNAME" : "solid"
}

See that we can get our file, line and function to be saved by the journal. Just awesome! :-)

econnman-1 released

Today I’m releasing the first release of “econnman”, the EFL user interface for ConnMan.

Some time ago raster added a ConnMan module to e17, but then the API broken during 0.5 transition and I rewrote that module. With ConnMan-1.0 the API was changed yet again and our module stopped working.

Lucas De Marchi took the task to revive the module and he is still finishing this task… let’s say Enlightenment’s internal widget set (e_widget) is something that can get everyone bored to death! Also the new ConnMan API requires an Agent interface to reply with passwords, usernames and other input that the manager may need. This code was not written before, so these new API and GUI must be written.

Until Lucas can finish his work I took a weekend to play with my favorite language: Python. I was away from Python-EFL for a while, so I was wanting to write some code using it. Python-DBus is super-simple (compare it to C!), so the EConnMan in Python-EFL was a nice thing to play.

We’ll cooperate efforts: E17 module will only provide the services list and allow to connect and disconnect. It will also provide an Agent, so we can provide passwords. Everything else that is related to configuration will be left to stand-alone “econnman” application. This standalone will do the tweaks and show details such as IP and MAC addresses, proxy, tethering and others.

In less than two days I had the base of the code I’m releasing now. It supports everything I need (IOW: it does not support VPN, Bluetooth-PAN or 3G/Cellular), but it’s easy to extend to support new features…. so patches are welcome!

My delay to have it released was that I wanted to improve theme (both default and detourious) so it look better, as you can see in the screenshots. It’s still not perfect, particularly the “Segment Control” used to show mutually exclusive options (DHCP, Static, Off) is damn ugly!

It also ships with an optional agent (–agent/-a command line option) that can be used until E17 gets a proper module with agent support.

Download:

Future:

Some people want IPv6, Proxy and VPN support. Those are nice ideas, but I personally don’t need them. So I’d like some people that use them to step up and do the code… or at least push me to do them! :-P

Some people want to rewrite it in C and while I can sponsor the idea, it will be much more work for no real gain. With a Python version working one can do the conversion easily, even the helper structures and callbacks can be known beforehand…

 

kmod announcement (and how to help testing it!)

introduction and motivation

At ProFUSION a common topic is how to optimze the system. Not just the speed, but also what and how it is done. Not just for our embedded systems, but also our desktops and laptops.

These discussions led us to be involved in projects that consider this goal, such as EnlightenmentConnMan, oFono and systemd. With great projects, come great people and thus enlightening discussions on how to improve things. From these discussions Lennart Poettering and Kay Sievers put together a A Plumber’s Wish List for Linux, with one of the items being of special interest for ProFUSION as it had the potential of also helping embedded systems and speeding up or boots:

* module-init-tools: provide a proper libmodprobe.so from
module-init-tools:
Early boot tools, installers, driver install disks want to
access information about available modules to optimize
bootup handling.

One of the reasons for this is that udev will search /sys/devices for all “modalias” files and call “/sbin/modprobe -bv $ALIAS“, however many of these calls evaluate to an empty list, thus an useless execution of program that will open /lib/modules/`uname -r`/modules.*, load resources, search for something that is not there and return. This could be way cheaper if done inside udev workers. [note: Lucas did measure and noticed 2.5x speedups, stay tuned for benchmarks]

kmod

Then our developer Lucas De Marchi proposed to do the libmodprobe and we funded it. After some discussions with Lennart and Kay, it was decided to rename it to kmod for shortness and good namespace. The initial goal was to achive a 1:1 replacement of module-init-tools as a library to be used by udev and systemd-modules-load and we are close to it with our second release done today! While we miss depmod tool (planned for v3), our kmod-modprobe should be fully functional and if this proves to be true, the logic will move to libkmod to be used by udev.

we need your testing!

To ensure kmod-modprobe does what it is supposed to do, we need extensive testing, there is where we need you! Try it on as many systems as possibles and let us know. To do this we recommend:

$ git clone git://git.profusion.mobi/kmod.git
$ cd kmod
$ ./autogen.sh
$ ./configure --enable-logging \
              --enable-debug --enable-tools \
              --enable-zlib # if you have module.ko.gz
$ make all
$ sudo make install
$ sudo mv /sbin/modprobe /sbin/modprobe-bkp
$ sudo ln -s /usr/bin/kmod-modprobe /sbin/modprobe
$ ls -1 /sys/module/ > ~/original-modules
$ reboot
$ ls -1 /sys/module/ > ~/kmod-modules
$ diff ~/original-modules ~/kmod-modules

current users and feedback

Being a new project in a critical area of Linux system, we expected lots of criticism and rejection by people, but interestingly enough after Lucas’ announcement and LWN article the feedback was highly positive! We even had some testers and people to help with ideas and experience.

Among the people that joined the project is the current maintainer of module-init-tools package Jon Masters, which announced that kmod replaces module-init-tools. He is helping a lot with his knowledge and cases from Red Hat Enterprise Linux, also providing modules from non-trivial platforms such as s390. Last but not least he provided insights to improve module handling on Linux, particularly replacing modules with better alternative as required in enterprise systems.

With Jon’s blessing we’ll use the same mailing list linux-modules@vger.kernel.org and our git will move to kernel.org soon.

People from ARCH and Debian were also interested and even created packages for it! These guys were extremely helpful to test cases such as gzip modules, 64bits modules with i386 user space and so on. They are: Tom Gundersen (tomegun), Dave Reisner (falconindy) and Marco d’Itri (Md).

next steps

Based on our TODO, we have the following ideas for next steps:

  • Finish libkmod-elf.c to provide information required by depmod. I’m working on this in my depmod branch;
  • tools/kmod-depmod.c: create a 1:1 compatible tool to generate /lib/modules/`uname -r`/modules.*;
  • libkmod should export parsed configuration to be used by kmod-modprobe.c –dumpconfig;
  • create kmodmock library to be LD_PRELOAD’ed to redirect some syscalls such as init_module(), delete_module() and open() of /sys and /proc. I’ve started a branch for this some time ago as “unittest” branch but stopped due lack of ELF support at the time;
  • create unittests and measure coverage. Given kernel modules are user-input they may be broken files and libkmod-elf.c needs to be extensively tested to avoid segmentation fault due out-of-boundaries access. This mean not trusting null terminated string in .strtab section and so on.
  • create optimized modules.archive that would contain optimized search index and all modules compressed independently, but in the same file. This would save file access and could provide information we don’t have today, like the uncompressed size of gzipped modules.

PythonBrasil[7] = Excellent!

Last week from 29-Sep to 01-Oct we had the amazing 7th PythonBrasil conference, for the first time in São Paulo.

Since I’ve start to use Python in 2002 I loved the language, but after getting introduced to the PythonBrasil community in 2004 I’ve boosted my development skills, got some friends and even my first job (INdT-Recife) was a kind recommendation from Osvaldo Santana in 2005.

By 2005 we had the 1st PythonBrasil Conference, then called PyConBrasil, here in Campinas with the help of UNICAMP and our amazing non-stop contributor Rodrigo Senra. It was very cool, I even presented a talk there… and it motivated me to go to following conferences in 2006 and 2007 as well.

However if starting to work at INdT reduced my spare time since late 2006, after ProFUSION was born in 2008 I had no time to participate in the lists or even go to conferences. What a shame!

I couldn’t see how shameful it was until I did this PythonBrasil in 2011. I’ m yet to see a conference with so kind people. People still remembered me and I was ashamed when I couldn’t remind their names… although they did remember mine (NOTE TO CONFERENCE: bigger names next year!) Some would even let me know they still use Eagle-Py, something that I already forgot about. And people I had closer contact before were willing to talk as if we had met last week. Amazing.

During these talks I’ve catch up with Rodrigo Senra, Luciano, Osvaldo, Erico, Marco André, Sidnei, Fernando and many more I couldn’t remember. However one of the talks was very special: talked to Gustavo Niemeyer about Go programming language. That’s right, people were so kind and open we had a keynote about Go, and we talked a lot afterwards without problems! :-P  Gustavo showed me some nice details about the language and my mind is now burning! I must do Enlightenment Foundation Libraries (EFL) + Go = EGO, a perfect GUI tool.

But I was of use… not just a leecher! I went there to present 3 talks (slides in Portuguese!):

Last but not least, I’d like to thank everyone that did this amazing conference possible!

Introducing EWS to help EFL on FB and PS3

Lots of systems do not support multi-window on their own, consider standard Linux Frame Buffer (FB) or the PlayStation3. This makes it cumbersome as you’re restricted to a single window,  likely you’ll have to rewrite your apps to behave well in this scenario.

Traditionally this problem is worked around by toolkits like Qt that provides QWS, a great help for developers. Considering QWS I decided to help our PS3 friends with something similar which I shamelessly called EWS (Ecore + Evas Single Process Windowing System).

Ecore and Evas already made 99% of this work: they abstract rendering, abstract windowing and so on. There is even support for “inlined windows” in Elementary, these are rendered to buffers that are visible inside other windows. Actually the first version of EWS was a 30min hack in Elementary called “eland” (making fun of wayland). But as not everyone would use Elementary it was better to move it lower in the stack: Ecore_Evas.

Ecore_Evas is a glue layer between Ecore and Evas that takes care to configure actual windows and setup Evas on them, to capture events from various sources and feed them to canvas, etc. If you cope with its API you’ll work with most EFL libraries and applications out-of-the-box.

Ecore_Evas_EWS was created as a new engine that builds on top of others. It will use a backing-store engine like X11, FB, DirectFB, WinCE or PS3 to create an internal Ecore_Evas. Every new window created with ecore_evas_ews_new() is rendered to an Evas using the buffer engine and its pixels end as the image source in the backing-store engine, displaying it. This is very similar to existing ecore_evas_buffer feature ecore_evas_object_image_new(), however it will handle more things automatically.

Among the new features are events to allow window management. Things like “window created”, “window moved” and “window resized” are placed in the main loop for interested peers, they may handle these and decorate the windows, offer window management (close, maximize, …). To exemplify that I’ve added support in Elementary with basic controls such as move, maximize, restore and close. Unfortunately it looks  ugly like hell and then I’m not posting any screenshot or screencast ;-) Let’s wait for people doing themes to make a great work.

The work is far from complete, bugs remain,  optimizations could be applied for OpenGL… if you consider the rules “20/80″, 80% of the code was written in 20% of the time, now we’ll take 80% to make it work for sure :-D  But it’s not bad considering it took me 2 days,  1421 lines in ecore_evas_ews.c, 543 in elu_ews_wm.c and 317 in ews.edc…

That’s why I keep saying that EFL is an amazing technology to build products. It runs fast, it’s slick and it does not get in the way. Give it a try! :-)

EFL portability shines: native PS3 support!

While most projects praises portability with fancy things like “gchar” (Gtk), fancy names like “LightHouse” (Qt) or completely new abstraction concepts, The Enlightenment Project always choose a simple approach to use POSIX and fill the gaps as required. This requires much less work and yields the same or better results, as we could see in the past and now being the first toolkit to be ported to PlayStation3 (native, no Linux)!

As announced by the the developer, KaKaRoTo, in our official news the port was done on top of our existing SDL backend in a couple of days, then being migrated to native PlayStation3 using unofficial SDK PSL1GHT for jailbroken systems. There is still no hardware acceleration, yet it runs fine a physics game at Full HD.

Right now we have port to MacOS and Windows, including WinCE. Given our scarce resources we do not lag behind Qt or Gtk in portability front. Basically what we have is one brave developer called Vincent Torri that is a Math professor and do it on his spare time!

Vincent could do it by keeping our approach: use POSIX and fill the gaps. He created “Evil”, a library to provide missing functions to Windows. Kakaroto then created “Escape” to do the same for PSL1GHT. Seems this approach works fine ;-)

DesktopSummit 2011 – Berlin

I had the pleasure to attend the DesktopSummit 2011, a great event that happened in Berlin from 5-12 of August 2011. As I’ve mentioned in an earlier blogpost my focus was to highlight Enlightenment’s opinion that performance matters and that we need broader standards in freedesktop.org. Of course I explained a bit about our history and the current status of E17. The presentation file can be downloaded here.

All in all the event was great, not just due the talks but mostly due the friends and side-talks. Being at these events for a few years I managed to know lots of people from different projects. Not being in a major side (read: KDE or GNOME) I have the gift of free-transit among these fields… which is pleasant as I can gather ideas from both. [There is no hard or official barriers between them, but the psychological blocks peers from talking to each other and this is quite bad]

I came there sponsored by my company (ProFUSION embedded systems) to represent the system that provides a big bucket of our consulting, training and development services: Enlightenment.  I had the pleasure to engage into endless conversations with Enlightenment developers and community leaders (Cedric, Boris, Michael, Philippe…) We discussed a lot how we could broader EFL usage, bring more users, lower the barrier to new developers with easier to use tools and languages and of course how we could get Raster to release E17.

I also tried to learn from other people of technologies as well.  Before I was an Enlightenment hacker I did use and develop with Qt/KDE and already knew some icons such as Aaron Seigo, Thiago Macieira, Helio Castro and Sebastian Kügler, which I managed to meet again. Of course my ex-coworkers at INdT and KDE fanatics Artur Duque de Souza, Renato Chencarek were there. And I was introduced to Daker Fernandes Pinheiro, from INdT. We discussed QML, Qt, MeeGo and lots of optimization and API designs. Quite productive!

Being part of Maemo since 2006, attended some GUADEC and hacked Gtk/GNOME for fun and work I got to know some icons there as well. I’ve talked to Lennart Poettering, Marcel Holtmann and Marc-André Lueau, people that I’ve worked together in a way or another and that are always open minded to discussions. I’ve also touched base with some people like Lucas Rocha and Zeeshan Ali. It is interesting to know what these guys are doing for Linux Desktop (and mobile) infrastructure and their vision for GNOME.

My presentation went quite well, I was bit anxious and nervous in such a huge room that held it, but I guess people understood the history of Enlightenment, why we created the “Foundation Libraries” (instead of using Gtk or Qt),  our special care with performance and why it will always pay off. I did some heavy critics to FreeDesktop.Org that generated positive feedback from Thiago Macieira (Qt/KDE) and people from other desktops (XFCE/LXDE). Last but not least I’ve made it clear Enlightenment has serious problems to manage community, we’re quite bully, and did thank the guys like Philippe Caseiro and other french dudes that are trying to solve this issue.

All in all a great event, with great people! Looking forward to be in next desktopsummit as well!

ANN: LightMediaScanner 0.4.4 released

I’m proud to announce LightMediaScanner 0.4.4 was released and I’d like to take some time to remind you of this awesome project ;-)

While 0.4.4 is no big leap ahead of 0.4.3 I’m doing this post more to make clear people that the project is still alive and it may be the right tool for you, particularly if you need fast media scanning on your desktop or embedded device.

Introduction (History)

LightMediaScanner (LMS) was born in 2007 during the development of Canola2 media player for Nokia N800 Maemo device. We did a previous try at media scanners for Canola1 and did learn a lot, that would be the foundation of LMS.

We had two major problems to solve:

  1. scan large amounts of media as fast as possible
  2. don’t mess with interactivity.

The first problem had more qualifiers, like the media being unreliable (broken files), be removed without prior notification (mmc removed when system was off), slow discs (sd/mmc), small amount of available RAM, low-end CPU and so on.

Traditional approaches were slow and would bring your device to unusable state for minutes, even after the scan process finished as they consumed too much memory that would swap your applications and services. They would also be naive with file scanning and would trash filesystem’s cache with useless data.

Existing Solutions Problems

At that time we had Maemo’s media scanner that was awful and Tracker was being developed to solve its problems. However Tracker was immature and had other focus, more like being fancy with new standards and trying to solve all the diseases of the world… while we need to be fast in some specific cases.

To highlight technical problems, let’s consider JPEG scanners. Everything one Media Player application would need is the  title, author/artist, date, width, height, orientation and maybe geolocation. These are few bytes, particularly because Title and Author are missing in most cases. We should open the file, get these bytes, register these and close the file. Every system at the time used libexif to do so, however it would copy all the header to memory prior to scanning. Given that most files contain thumbnails in the header, it would mean useless malloc + read of around 10-50kb, to get dozen bytes and exit. This trashes the VM and disc. Moreover, some libraries would open the file themselves and would not fadvise as POSIX_FADV_DONTNEED, thus the filesystem would keep it in cache for further access that never happen! Similar situations existed for PNG, Mp3 and others.

LightMediaScanner Solution

We decided to go with a 2 process solution to avoid crashes and hang-ups on the main process. This would be safe and we could kill the slave process if it took too long to report on a given file. Later on we introduced threaded and single process scanning that would be useful in some cases.

We also had dynamic loadable plugins (shared objects) with the media parsers to be used given matching extensions. The plugins can be selected by the user at runtime. Yes, we would not scan every file with every parser, we did trust the extension as a way to save work and it worked fine, and fast.

The main process just scans the filesystem and  pipes the files to the slave (thread or process). This slave would open the file, get the data to populate a structure to be written to database, then close the file and report back the progress. If the slave takes too long to reply or dies, it’s respawned with the next file in the queue.

The database uses SQLite3 and a configurable number of files are processed in one SQL transaction, thus we’d not hit the filesystem with writes that often. The database access is provided with helper functions that uses pre-compiled statements. However we only provide the insert/update methods, as the fetch (select) are up to the application to use (maybe using their own ORM framework).

Some metadata formats did not specify encoding (ASCII was assumed) and fallbacks can be applied with the charsets. It will use iconv in a priority list to figure out what works.

Just the Indexer!

LightMediaScanner is not a daemon-service, it does not talk DBus neither requires you to speak  RDF or SPARQL. If you want, you can create one with it, but it does not provide this stuff out of the box. Basically you call it to check for media and then access the database yourself, using the SQLite3 directly or using some ORM, doing caches or not, it’s up to you.

The database tables are quite simple and try to provide common useful relations. It’s not designed to cover all weird cases in the world (like multiple artists per song or which lens did you use with your camera).

Using it!

Get the code from our GIT or use the release tarball, see http://people.profusion.mobi/cgit.cgi/lightmediascanner.git/. It uses standard autoconf/automake to build, so should be familiar to most developers.

It hard-depends only on SQLite3 and iconv (most libC ships with it). Optional dependencies are libvorbis, libmp4v2 and libflac.

You can call the scanner from C, C++ or Python using our bindings. Comprehensive example can be found at src/bin/test.c

References: previous post about lightmediascanner 0.2.0 release

Blog recovered

Hi all,

After a while not even opening my blog, yesterday I did two posts. While the administrator interface seemed fine, readers quickly notified that it was showing lots of spam in the regular view which I confirmed using Chromium’s private browsing. Investigations led to dozen administrator accounts in WordPress database, then I decided to reinstall from scratch. Unfortunately yesterday was a busy day and I could barely stay at the computer to do so.

Anyway, this morning I restored my blog and I’ll try to keep it updated :-D I also changed the comments rule, instead of requiring people to register, I instead opted to close comments after 14 days, since most spammers seems to look for pages with reasonable pagerank and new pages do not have them that soon.

ESC Brazil – RealTime Linux with RT_PREEMPT

The first Embedded Systems Conference Brazil was held at São Paulo on 24 to May 25 2011 and ProFUSION was there to do a technical talk.

Our contributor Lucas De Marchi proposed the talk “Usando Linux como Sistema de Tempo Real” (Using Linux as an Real Time Operating System). He did his master thesis on that topic for 2 years at Politecnico di Milano, so he knows a lot about the topic.

Unfortunately Lucas is one of those geeks that try to live a real-world life and plays football (soccer for americans) and in the week before he broke his foot… Ouch!

So yours truly was elected to present the talk for him. I already did play with realtime Linux in the past and I knew the concepts from past experiences, thus I accepted the challenge and did the talk. According to the attendees it was a good talk!

People got amazed with Linux’s Real Time capabilities and the ability to mix it with regular (“fair”) tasks, being able to change a process from FAIR to RT during runtime, no recompile was needed! This was demonstrated in a Freescale’s i.MX31 running music player running with regular priority and ping flood and a heavy CPU task, the music skipped and it was clear the deadlines were not being met. Running chrt and raising the priority above Kernel Threads that handle interruptions immediately fixed the skips. We also ran a software oscilloscope at desktop that plotted the achieved deadlines and it confirmed what we were listening.

If you’re not aware, RT_PREEMPT is a patchset (from pre-git days, not a git tree called -rt) that Ingo Molnar and Thomas Gleixner run to improve Real-Time behavior for the Linux kernel. As concepts are tested and patches mature they are moved into mainline tree, benefiting everyone using it, even your desktop running PulseAudio now! The major trick of this work was to handle interruptions in Kernel Threads instead of immediately as it used to be. These threads are RT themselves at priority 50 and if you want you can have a regular user process thread at a higher priority and it will run before the kernel. This was the case with the pingflood, with a regular kernel the music could skip due kernel stopping the playback process to handle the network interruption.

Last but not least, if you’re not familiar with RT have something clear in your mind. Real Time doesn’t mean Real Fast. It just mean you have guarantees to execute things when they should and this is mainly done by means of a preemptive kernel. In some cases RT will turn things slower, as you may switch contexts more often due preemption, but you’ll know you’ll be on time (as opposed to batch processing). Linux being a General Purpose Operating System (GPOS) was not designed with such thing in mind, but it was successfully converted into one!