Portable C++: Unpacking integers from binary buffers

As C++ code is so close to the metal, we often make dodgy assumptions that hurt portability. One of the ‘simplest’ problems that I’ve seen repeatedly is packing and unpacking binary data.

The C++ works hard to eliminate definitions that would tie us into a particular hardware architecture, and this area invites a desire to throw caution to the wind and make assumptions as to exactly what’s going on.

The new college grad (and old-hat that views this as all theoretical anyway) might write:

There’s a handful of problems here:

  1. We’re assuming bit size of ‘int’ – it may be anywhere from 8 to 64 bits on common platforms.
  2. We’re assuming that we’re safe to read a char aligned buffer to an integer.
  3. We’re assuming the buffer is packed with appropriate byte order for our processor.
  4. We’re breaking the strict aliasing rule.

Can we write a new version of the function to take care of these challenges? Well, with a little care:

This version was tuned to work with GCC 5 and higher. This function is highly portable – it should operate on any architecture providing 8-bit chars and 32-bit int32. Indeed, the C++ standard definitions for conversion to/from std::uint32_t even handle the mode of twos complement arithmetic vs not. Using bit-shifts and or defines the exact expected behavior of the construction of the 32 bit integer.

And there was much rejoicing… sortof… There’s many a blog¬†post out there that support this method of formatting.

Now, let’s say that this particular call is fairly performance critical (perhaps we’re doing some pixel or image manipulation – use your imagination). In my application, I was processing large data files. Modifying from the first style to the second fixed issues with ARM portability, but slowed down performance.

Most compilers see the above pattern and recognize – “hey, I can just load a 32bit word and return, no harm / no foul.” Sadly, Visual C++ does not. No combination of optimization flag and type manipulation get the optimizer to recognize the pattern. Even GCC is fairly sensitive in situations where it can (hence the std::uint8_t casts throughout). To faciliate portability and performance on all my desired targets, the end result was using std::memcpy to a temporary integer. The ARM compiler happily recognizes we may be accessing unaligned memory, and all the other toolchains optimize away the memcpy to a simple load. Of course, now we’re back to handling byte order again. Ugh!

At the end of the day, maybe the grouch has it right – just worry about the processor you’re running on (hopefully just 1). It’s all fun and games until you find yourself porting to that random platform you’d never worry about.

Blinkt! by Pimoroni

Searching for a way to carefully control illumination of a project, I discovered “Blinkt!” by Pimoroni. I was hoping for something that would easily connect to a Raspberry Pi 3 with a minimum of fuss.

The Blinkt! connects straight to the GPIO pins on the Raspberry Pi 3, and can also work on an extension board or off a ribbon cable. A warning: many cases have ‘ribbing’ along the side that will interfere with mounting the Blinkt! directly on the Pi – NONE of my cases were compatible without the ribbon cable.

Pimoroni provides an easy to use Python software library, which worked directly as advertised on Rasbian. The examples were easy to modify and get started. For my use-case, I wanted to create a flashlight with hue control. While these LEDs are very bright on there own, they don’t compare to a higher power utility light or flash light – understandable as they are being powered off the Pi. The array was surprisingly bright, but only just barely able to perform the functions I was hoping.

My goal was to provide carefully controlled lighting for creation of photographic inputs for a source project. The Blinkt! did it’s job fairly well:

One can find cheaper LEDs and ‘bigger setups’, but the simplicity and price are hard to beat for someone that’s searching for an item that’s get up and go. For this project, the $8 spent on this board was well worth it.

Changing world of open source

Back when I started college (2001), I remember a world where tech was, well, ‘fun’. I was a die-hard free software guy willing to put up with far-to-much in the interests of ‘tweaking’ my computer. Running Linux was like drive a custom made hot-rod. I knew every piece of the system and was happy to tweak it all day long. I followed slashdot, freshment, linux games, and laughed out loud while reading userfriendly.

Perhaps today I’m simply waxing nostalgic, but damn if I don’t miss those days. I’m not sure where the old ‘hacker’ ethos has gone too, but now that I’ve had some time to settle into a professional career, and gain some semblance of free-time… Well, damned if I didn’t look and it just¬†feels missing.

I noticed over the years as some sites closed their doors (linuxgames.com) or renamed and then became read-only (freshmeat.net). Out of habit, I’ve still kept tabs on slashdot. Over the years though, it’s gone from featuring front-page articles about postgresql and Ipv6 on FreeBSD, to a front-page dominated by highly political click-bait trash.

This site used to have a fairly amazing google page-rank for multiple subjects, at least before I let it get snatched by a domain crasher and go fallow for years. I hope there are others out there that enjoy hacking in some community somewhere – I’m looking.