Since I’ve start to use Python in 2002 I loved the language, but after getting introduced to the PythonBrasil community in 2004 I’ve boosted my development skills, got some friends and even my first job (INdT-Recife) was a kind recommendation from Osvaldo Santana in 2005.

By 2005 we had the 1st PythonBrasil Conference, then called PyConBrasil, here in Campinas with the help of UNICAMP and our amazing non-stop contributor Rodrigo Senra. It was very cool, I even presented a talk there… and it motivated me to go to following conferences in 2006 and 2007 as well.

However if starting to work at INdT reduced my spare time since late 2006, after ProFUSION was born in 2008 I had no time to participate in the lists or even go to conferences. What a shame!

I couldn’t see how shameful it was until I did this PythonBrasil in 2011. I’ m yet to see a conference with so kind people. People still remembered me and I was ashamed when I couldn’t remind their names… although they did remember mine (NOTE TO CONFERENCE: bigger names next year!) Some would even let me know they still use Eagle-Py, something that I already forgot about. And people I had closer contact before were willing to talk as if we had met last week. Amazing.

During these talks I’ve catch up with Rodrigo Senra, Luciano, Osvaldo, Erico, Marco André, Sidnei, Fernando and many more I couldn’t remember. However one of the talks was very special: talked to Gustavo Niemeyer about Go programming language. That’s right, people were so kind and open we had a keynote about Go, and we talked a lot afterwards without problems!  Gustavo showed me some nice details about the language and my mind is now burning! I must do Enlightenment Foundation Libraries (EFL) + Go = EGO, a perfect GUI tool.

But I was of use… not just a leecher! I went there to present 3 talks (slides in Portuguese!):

• Tudo que você sempre quis saber sobre Bindings
• Python em sistemas embarcados: Sim ou Não?
• Canvas-2D extremamente rápido usando EFL

Last but not least, I’d like to thank everyone that did this amazing conference possible!

While 0.4.4 is no big leap ahead of 0.4.3 I’m doing this post more to make clear people that the project is still alive and it may be the right tool for you, particularly if you need fast media scanning on your desktop or embedded device.

Introduction (History)

LightMediaScanner (LMS) was born in 2007 during the development of Canola2 media player for Nokia N800 Maemo device. We did a previous try at media scanners for Canola1 and did learn a lot, that would be the foundation of LMS.

We had two major problems to solve:

1. scan large amounts of media as fast as possible
2. don’t mess with interactivity.

The first problem had more qualifiers, like the media being unreliable (broken files), be removed without prior notification (mmc removed when system was off), slow discs (sd/mmc), small amount of available RAM, low-end CPU and so on.

Traditional approaches were slow and would bring your device to unusable state for minutes, even after the scan process finished as they consumed too much memory that would swap your applications and services. They would also be naive with file scanning and would trash filesystem’s cache with useless data.

Existing Solutions Problems

At that time we had Maemo’s media scanner that was awful and Tracker was being developed to solve its problems. However Tracker was immature and had other focus, more like being fancy with new standards and trying to solve all the diseases of the world… while we need to be fast in some specific cases.

To highlight technical problems, let’s consider JPEG scanners. Everything one Media Player application would need is the  title, author/artist, date, width, height, orientation and maybe geolocation. These are few bytes, particularly because Title and Author are missing in most cases. We should open the file, get these bytes, register these and close the file. Every system at the time used libexif to do so, however it would copy all the header to memory prior to scanning. Given that most files contain thumbnails in the header, it would mean useless malloc + read of around 10-50kb, to get dozen bytes and exit. This trashes the VM and disc. Moreover, some libraries would open the file themselves and would not fadvise as POSIX_FADV_DONTNEED, thus the filesystem would keep it in cache for further access that never happen! Similar situations existed for PNG, Mp3 and others.

LightMediaScanner Solution

We decided to go with a 2 process solution to avoid crashes and hang-ups on the main process. This would be safe and we could kill the slave process if it took too long to report on a given file. Later on we introduced threaded and single process scanning that would be useful in some cases.

We also had dynamic loadable plugins (shared objects) with the media parsers to be used given matching extensions. The plugins can be selected by the user at runtime. Yes, we would not scan every file with every parser, we did trust the extension as a way to save work and it worked fine, and fast.

The main process just scans the filesystem and  pipes the files to the slave (thread or process). This slave would open the file, get the data to populate a structure to be written to database, then close the file and report back the progress. If the slave takes too long to reply or dies, it’s respawned with the next file in the queue.

The database uses SQLite3 and a configurable number of files are processed in one SQL transaction, thus we’d not hit the filesystem with writes that often. The database access is provided with helper functions that uses pre-compiled statements. However we only provide the insert/update methods, as the fetch (select) are up to the application to use (maybe using their own ORM framework).

Some metadata formats did not specify encoding (ASCII was assumed) and fallbacks can be applied with the charsets. It will use iconv in a priority list to figure out what works.

Just the Indexer!

LightMediaScanner is not a daemon-service, it does not talk DBus neither requires you to speak  RDF or SPARQL. If you want, you can create one with it, but it does not provide this stuff out of the box. Basically you call it to check for media and then access the database yourself, using the SQLite3 directly or using some ORM, doing caches or not, it’s up to you.

The database tables are quite simple and try to provide common useful relations. It’s not designed to cover all weird cases in the world (like multiple artists per song or which lens did you use with your camera).

Using it!

Get the code from our GIT or use the release tarball, see http://people.profusion.mobi/cgit.cgi/lightmediascanner.git/. It uses standard autoconf/automake to build, so should be familiar to most developers.

It hard-depends only on SQLite3 and iconv (most libC ships with it). Optional dependencies are libvorbis, libmp4v2 and libflac.

You can call the scanner from C, C++ or Python using our bindings. Comprehensive example can be found at src/bin/test.c

References: previous post about lightmediascanner 0.2.0 release

• Ardy, a tool that brings together EFL and Arduino using Python
• Free.fr, the second biggest ISP in France opened up the development of their Freebox HD set-top box using Enlightenment Foundation Libraries and Mozilla JavaScript library. This is pretty amazing as it’s the biggest deployment of EFL out there, an uncertain number that ranges from 2 to 3 million devices.

Yesterday we started to see some announcements of companies backing Enlightenment Foundation Libraries development. Of course, INdT was pioneer in that since it was decided to use it for Canola2. Later on I created my own company and we officially support EFL as GUI alternative (together with Clutter, GTK and Qt), being the first company to do that.

While there are speculations about which company is it, what I can assure you is that this company is serious and is not alone. ProFUSION itself worked on EFL on behalf of various clients and you may expect another press release about a big French internet and telecom company deploying a massive number of units with EFL pre-installed. Not accounting various community driven projects that choose it and E17 as its base platform, such as OpenMoko and OpenInkpot.

Bottom line? While EFL does not get the same amount of marketing and visibility as Qt and GTK counterparts, it is playing fine enough to be considered to ship in dozen million devices in the next year. Why don’t you consider it for your project? Be open minded and try it out

We even managed to convince Mariana to do the initial talk!

The product is based on free software Canola2 platform, which we support as well. While some parts of it were made available over these months, including our optimized thumbnailer “Ethumb”, some parts will be available later, under the same license. It’s not an issue as you’re unlikely to get it from Volkswagen and running it BMW, what matter for clients is the product as a whole, not just software interface.

Work done so far is not just a face lift of Canola, it goes deeply in optimizations, platform roots and changes plugins as well. The roadmap includes fast UPnP server and control point, as well as GPS/maps and other car useful services.

Theme is just a demo, we call it “ProFUSION theme” as we use our own colors and font. Clients will get an unique look and feel as well as custom changes, such as integrating with real panel keys, text to speech and voice recognition systems.

Read more at http://profusion.mobi/node/17.

As for freedom, free software, mobile media centers and specially Canola2: as announced previously at Maemo community, Canola2 is now opensource (GPLv3)! That’s amazing news, specially to me as I have Canola2 as my baby and would like to have more people involved into its development. It’s not just a great end-user software, it’s an amazing Python platform where you can build all kind of rich user interface.

As you might know, ProFUSION is working with INdT to improve Canola2 and we plan to keep supporting it, starting with some instructions and scripts, see our post for more details. As we want to build a community around it, don’t miss the mailing list and our IRC channel #canola at irc.freenode.net.

My talk Python enabling mobile media centers will tell you all how Python made it possible to finish Canola2 in record time and how it does not suck performance wise in mobile devices as the Nokia N800, N810 and it is even acceptable on 770! I’ll quickly cover how painful development of first version in C was, how we profiled, tools we used to write Python-EFL bindings and more.

For my beloved Brazilian friends, I plan to present it (or a similar talk) at Bossa Conference ’09 and possible present it in Portuguese at PyCon-Brazil later this year.

I still remember myself talking to Mark Shuttleworth about possibility of Qt going LGPL and I was saying that it would never become LGPL since it was an excellent thing for Nokia, keeping adversaries away.

It turned out that I was wrong… “never say never!” they say. Nokia is seems so confident, or Motorola so non-intimidating, that it believes that doing the right thing and moving its product license to more commercial friendly will bring more developers and thus more applications.

Mark was wondering about GNOME goals could be delivered on top of Qt if this was LGPL. Well, in my opinion it is possible, but very unlikely. I dare to say GTK will get going along Qt and it will never go away. It’s about passion, not technology there.

Talking about technology and this concerns ProFUSION, I really like Qt, always did. As most of you know I hack using Qt, GTK and EFL for a long time, Qt is the easiest to use, largest and most complete library out there. And following their progress with 4.x versions you can see they’re heading the right direction, heading where EFL or MacOS libraries are today. ProFUSION will now be able to recommend Qt to a broader range of clients, those that wanted LGPL licenses to avoid licensing fees.

I’m still surprised! I expect Nokia keep doing these great surprises, the next being the next internet tablets (with $99 developer program ) and maybe a Linux phone later this year!

The integrated code is very flexible, it make use of the recently introduced “size hints” and also postpone heavy calculations to pre-render time with calculate smart callback. Table has three modes: regular, homogeneous based on table size and homogeneous based on largest minimum item size. Box, since it just represent a sequence of items, is more extensible and allows you to specify a layout function, we provide some like vertical, horizontal, stack, homogeneous based on box, homogeneous based on the largest minimum item size, etc… but you can easily write your “snake layout” and use it. If you need more option details than “size hints”, you can extend the class and implement options_* virtuals.

These code were integrated by me, but not totally written. Gustavo Lima, from ProFUSION, wrote the box for their sequence_box.c (it was relicensed to E’s BSD with permission) and Rasterman wrote table for his elementary “toolkit for mobiles” els_table.c.

On the pleasure and work side, I’ll fly to The Netherlands next Tuesday so I can attend ELC-E 2008 where I’ll present a talk about Rich Graphical User Interfaces on mobile systems, covering Evas, Edje and the new kids on the block Elementary and Guarana.

Last but not least, due trip and other stuff to do I’ll not be able to integrate table and box into Edje soon. If you always wanted to help E17 and EFL, now it’s your chance! See my mail to the list and start hacking, I can reply to you by mail and IRC (when I’m online). We will also need Python bindings for those, so patches to python-evas and python-edje are also welcome!