Tech That!

The rantings of a mad computer scientist

Installing Windows 8.1 to USB drive from Linux

Linux is great – we all know this. However, as most Linux enthusiasts painfully experience more often than they’d like, Windows is still occasionally needed for certain applications. While WINE and VirtualBox VMs takes care of most use-cases, there is still the occasional game or application that you’d really just rather run in a plain old Windows environment with no layers between the OS and the hardware. Having an external drive you can boot Windows directly from is perfect for this purpose, and as MIT offers Windows 8.1 Enterprise for free to its students, I figured I’d take a stab at it.

So, with Windows 8, Microsoft introduced Windows to Go which (in theory) lets you install the operating system onto an external drive and plug it into any machine. The big change from Windows 7 was that WtG installs drivers beyond what is needed for the initial system, and thus you will (again, in theory) end up with way fewer crashes and errors than before. Unfortunately, WtG only works with a select few “certified devices“, and so despite being the proud owner of a brand new MyDigitalSSD OTG USB 3.0 SSD drive which should run Windows 8.1 without any problems whatsoever, I was stuck with an empty drive and no new games to play.

Luckily for me, Microsoft has released something called the Windows Assessment and Deployment Kit, which eases the job of administrators for big Windows deployments. The Windows ADK includes a tool called ImageX that lets users manipulate and deploy Windows installation images, and ingramator over at the Neowin forums has found that this tool can be used to install Windows 8.1 directly to a disk without going through the setup program, thus bypassing the checks for “certified” hardware! This sounds perfect – if only you could run ImageX from Linux.

Okay, so not quite perfect – we need to run Windows in order to install Windows… Well, time to download a (free and legal!) Windows VM image and see if we can’t get the job done from there. I’ll wait while you set it up…

So you’ve got your shiny new Windows VM up and running, you’ve downloaded the 8.1 Enterprise ISO and GImageX (GUI for ImageX), you’ve mounted the ISO in the VM so you can get to the install.wim file, and you’re about to launch GImageX for the first time. Then you realize that, damn, you’ve forgotten that you can’t see your USB from inside the VM. Oh well, just add it under USB devices and be on our way. “New device recognized” – yeah yeah, come on already. “Device failed to start”. What? So, it turns out that although VirtualBox technically “supports” USB 3.0 devices, most of them won’t actually work properly with a Windows guest OS (see the comments here). So close, and yet so far… There has to be something clever we can do! This is Linux after all! Can’t we just fake it?

I’m glad you asked. See, VirtualBox has this internal feature called “raw host hard disk access” which lets you expose a drive or partition from the host OS directly to the guest OS. This way, the drive won’t be seen as a USB device by the Windows guest, but rather as just another internal disk. Any writes or modifications made to the disk inside the guest VM will be written directly to the real disk. Perfect! Okay, so you plug in your USB 3.0 storage device, find its device path (let’s say it’s /dev/sdb), and then do:

VBoxManage internalcommands createrawvmdk -filename /path/to/file.vmdk -rawdisk /dev/sdb

Permission denied? Oh, of course:

sudo VBoxManage internalcommands createrawvmdk -filename /path/to/file.vmdk -rawdisk /dev/sdb

All we then have to do is mount /path/to/file.vmdk as a SATA drive in the VM, and — “permission denied” again?

sudo chmod 777 /path/to/file.vmdk

That should do it, right? I guess not. It turns out you (perhaps obviously) need to change the permissions of /dev/sdb in order to let VirtualBox manage the device completely.

sudo chmod 777 /dev/sdb

Ah, yes, that did it. Back to Windows. First, we create a new, empty partition (remember to set it as active/bootable!) on our new disk. Next, we open GImageX, go to the “Apply” tab, select install.wim from the mounted 8.1 ISO as the source, select our new partition as the destination, make sure the checkboxes are unchecked, and then click Apply. This process takes a little while (it’s installing the OS), so go get a cup of tea while you wait.

After this process is done, open an Administrator command prompt (Windows key -> type cmd -> Ctrl + Shift + Enter), go to your new WtG drive (type X: and press enter), navigate to System32 (cd \Windows\System32) and run

bcdboot.exe X:\Windows /s X: /f ALL

And that’s it. You should now be able to turn off the VM, reboot your machine and boot directly into Windows 8.1 from your USB drive. The first time it will ask you some installation questions and such, but after that you’re home free! Now go play some games or be productive in Lightroom or something, and then we can go back to hacking on Linux in a couple of hours.


Permanently changing the MAC address of a Broadcom Android 4.x device

Wi-Fi devices all have a (mostly) unique address burned into them when they are manufactured so that a meaningful media access control (MAC) protocol can be developed. This MAC address needs to be unique on your network so that your Wi-Fi access point knows which device is sending it data, and which device it should reply to.

In most deployments, it should not be necessary to ever change your MAC, but there are some use-cases in which this ability can come in handy. First of all, you might want to change your MAC for privacy reasons (so people can’t track your machine). Second, some networks apply (very ineffective) access control based on a list of “permitted” MAC addresses. Finally, some ISPs restrict Internet access to only specific MACs, and being able to change it gives you the freedom to use any device as your modem.

You may have noticed that I said the MAC is burned into the hardware, which begs the question: how are you able to change it? Well, your operating system reads the MAC from the card when it boots, but the OS can choose to put a different MAC into the data packets it sends out. This is often referred to as MAC spoofing. Exactly how you get it to do that though depends on the exact operating system you are running. Today, I will focus on Android.

If you head over to XDA or Google and do a quick search for “Change Android MAC”, you will get a long list of forum threads and apps dealing with this issue. Most of them go something like this:

  1. Root your device (Google it)
  2. Install busybox (Google it)
  3. Run adb shell
  4. Run su to get a root shell
  5. Disconnect from any wireless networks
  6. ifconfig wlan0 down
  7. ifconfig wlan0 hw ether 00:11:22:33:44:55
  8. ifconfig wlan0 up
  9. Connect to your network again and your new MAC will be used

Perfect! Done.

Well, not quite.. Firstly, this solution will only work on some devices. Some devices will seem to succeed, but will keep using the old MAC when you start using the network, and others will successfully change the MAC, but then fail to connect to any networks. Furthermore, this change is temporary. If you turn on Airplane Mode, reboot your device, or in some cases even turn WiFi off and then on again, the MAC will return to its default value again.

Today, I am hoping to show you a more permanent approach that I believe should work on any device using a Broadcom chipset, but that also takes some more effort to pull off. SOME OF THE STEPS BELOW INVOLVE FLASHING YOUR DEVICE, WHICH MIGHT VOID YOUR WARRANTY AND BRICK YOUR DEVICE! I TAKE NO RESPONSIBILITY IF YOU GO AHEAD WITH THIS. YOU HAVE BEEN WARNED. Also, before going ahead with anything below, make a note of your current MAC address so that you can restore it if something goes wrong. It is shown under Settings -> About phone -> Status -> Wi-Fi MAC address.

First, install Cyanogenmod on your device (or some other non-vendor Android, stock works fine too). Next, make sure your device is rooted and that you have adb installed and working. Now, run adb shell followed by su to drop you into a root shell. Before we begin exploring the long way of doing this, we’re going to see if you can take a little shortcut. If you have a file called /persist/wifi/.macaddr, just type echo -n "001122334455" > /persist/wifi/.macaddr, and you are done! This only exists on some devices (I suspect only on Nexus), but if it does, it saves you a lot of work.

If you’re still reading, I assume that the above shortcut did not work for you, and so we need to take a few more steps. You see, it turns out that there is an option in one of the very commonly used Android drivers that must be explicitly set in order for the MAC address to be changeable. There is a file, /efs/wifi/, that contains your device’s MAC address, but without this particular driver option, changing the content of that file will not change the active MAC.

The default CM11 kernels do not appear to have this turned on (at least the Samsung Galaxy S2/i9100 one doesn’t), and so you will have to find one that does. In particular, you want to see if drivers/net/wireless/bcmdhd/dhd_sec_feature.h has the line #define READ_MACADDR in it (a patch might look like this). If you happen to have a Galaxy S2/i9100, you can use the DorimanX kernel which already has it enabled. Flashing this new kernel should be as simple as using adb push to send the kernel’s .zip file to the device, rebooting into recovery with abd reboot recovery and then installing the kernel .zip from there.

After flashing your shiny new kernel, your phone should boot normally and look just the same. The difference is that the kernel will now read /efs/wifi/ when the driver starts, and it will use that MAC address! This file is on the /efs partition, meaning that any changes will survive a reboot, so by executing echo -n "00:11:22:33:44:55" > /efs/wifi/, we have now permanently changed the MAC of your device! Try rebooting and then going into Settings -> About phone -> Status again; lo’ and behold, your MAC has changed.

Hope this has been helpful, and please leave feedback if you find a working kernel for another device, if you get an error of some kind, or if you have ideas about how any of the steps can be made clearer.

Running Dungeon Keeper on Linux

After reading this great post on how in-app purchases are ruining the gaming industry, I felt an urge to play Dungeon Keeper again. Since I run Linux, however, playing games is not always such a trivial affair. WineHQ (unhelpfully) told me that Dungeon Keeper does not run very well under wine 1.5, but given that 1.5 was released early last year, I figued I’d give it another shot.

I headed over to Good Old Games who provide DRM-free versions of old games, and bought Dungeon Keeper Gold edition for just below $6. After downloading the installer, I ran it through wine, and lo’ and behold, the installer ran without problems. At the last screen, I pressed “Launch game”, and to my surprise the game also started without a hitch. No stuttering in video or audio. I started the single player campaign, and nothing seemed to be wrong at all. So much for old WineHQ entries.

Then I quit the game to see whether it would still work when I started it “normally”. I quickly discovered that there was no “normal” way to start the game. The game is run through DOSBox, and GOG have provided Windows shortcut files (.lnk) for launching the game with the appropriate options. Unfortunately, wine doesn’t let you launch .lnk files, and just running DOSBox gave me nothing more than a shell (which, to be fair, was to be expected). Time to start digging.

It turns out that .lnk are part binary, part text, and by catting the file, you do see what commands it would run when executed on Windows. I’ll save you the details, but you essentially need to pass DOSBox two configuration files provided by GOG: dosboxDK.conf and dosboxDK_single.conf. You also need to run this command from inside the DOSBOX/ directory! Being in the game’s root directory will not work, and the game won’t start.

Realizing that DOSBox is also supported natively on Linux, I started to wonder whether I could avoid wine alltogether.. Seems silly to start wine to start DOSBox to start the game if I don’t have to. So, I replaced wine DOSBox.exe with just dosbox, and everything worked just as well! In fact, it turns out that DOSBox has already been tested with Dungeon Keeper, so this shouldn’t have come as a surprise I suppose.

Finally, to clean up, I extracted the Dungeon Keeper icon files and created a Desktop Entry file so that I can now start the game directly from my launcher:Dungeon Keeper launcher

Hope this has been useful, now go and be evil!

Incremental plotting in gnuplot

If you have ever had to plot something, chances are you’ve come across gnuplot. It’s a very versatile plotting tool, and once you get used to the syntax and quirks, you quickly start using it from everything from throwaway data visualization to plots included in papers.

One use-case for gnuplot that there is currently no built-in support for, is handling incremental plotting where you have a data file that some other process is writing to, and you want your plot to update so that it always shows the latest data (or scale so it always shows all of it!). There are many ways of accomplishing this, most of which are based on having something like this in your plot file:

load "< while [ 1 ]; do echo 'replot'; sleep .1s; done"

load is telling gnuplot to continue reading commands from the following source, and the < at the beginning of the source string tells it that it should execute what succeeds it as a shell command. Without asking questions, gnuplot does so, causing it to enter an infinte loop. Each time the script outputs “replot”, gnuplot goes ahead and replots, but since the input source isn’t empty yet, it continues reading (blocking the application in the process). This certainly works, but it has the unfortunate side-effect of completely locking up the interface so you cannot interact with your data at all. Zooming and panning will simply not work. It is also somewhat inconvenient that script will never terminate, even when your data file is no longer being written to.

I therefore propose the following solution instead. First, you create a little bash script that generates your gnuplot script and puts it in a variable (let’s call it script). Then, at the end of your script, you add the following code:

script="plot 'data-file.dat'"

mkfifo $$.gnuplot-pipe
gnuplot -p <$$.gnuplot-pipe & pid=$! exec 3>$$.gnuplot-pipe
echo "$script" >&3

trap 'running=0' SIGINT
while [[ $running -eq 1 && $(lsof "$1" | wc -l) -gt 0 ]]; do
	echo "replot" >&3
	sleep .5s

exec 3>&-
rm $$.gnuplot-pipe
wait $pid

This code uses a named pipe to feed data to gnuplot so that we can continue sending commands to gnuplot after launching it. We also use another little-known trick to prevent echo from closing the FIFO file once it has written to it. Because we launched gnuplot as a background process, we can now run our own while loop that sends replot messages to gnuplot without blocking the main plotting thread! We can also be somewhat smarter about this by only sending replot commands as long as the file is still held open (the lsof command helps us with this), and we can gracefully terminate if the user sends us a SIGINT (presses ^C). Finally, we close and remove the FIFO file, and wait for gnuplot to terminate. Shiny!

So, there you have it, incremental data plotting in gnuplot without blocking the interface.

Generating combinations for N-dimensional searches

Problems involving searches are very common in computer science, and they are often N-dimensional, meaning you are looking for a *combination* of values that yields some outcome. For example, you might be looking for all pairs of numbers whose sum is 50. This is trivial to express in most modern programming languages, and would usually be done with a nested for loop like this:

for (int a = 0; a < 50; a++)
  for (int b = 0; b < 50; b++)
    if (a+b == 50) printf("%d + %d == 50\n", a, b);

Say that you now wanted to find all triples of numbers that sum to 50. You might be tempted to add another nested for loop as such:

for (int a = 0; a < 50; a++)
  for (int b = 0; b < 50; b++)
    for (int c = 0; c < 50; c++)
      if (a+b+c == 50) printf("%d + %d + %d == 50\n", a, b, c);

While this is still legible, it should become apparent that this won’t scale very well as you add more dimensions. More importantly though, this approach also requires you to know in advance the number of numbers you want to add. If you were to build a program that let the user specify how many numbers she wanted to use, this simply wouldn’t work.

I had exactly this problem, and came up with a pretty neat solution. It has probably been done before elsewhere, but I figured I’d document it for future reference.

The following code lets you find all combinations of N variables with values between 0 and S. By multiplying the value with a fraction, you can also extend this to non-integral numbers.

int x, n, i;
int vars[N];
double ns = pow(S, N);

for (n = 0; n < ns; n++) {
  x = n;
  for (i = 0; i < N; i++) {
    vars[i] = x % S;
    x /= S;
   * vars[0..S] now holds one of the
   * combinations of values for the
   * N variables

We can use this to implement the example I gave above of letting the user specify the number of numbers to sum:

int N = /* get user's number */
int S = 50;
int x, n, i;
int *vars = calloc(N, sizeof(int));
double ns = pow(S, N);

for (n = 0; n < ns; n++) {
  int sum = 0;

  x = n;
  for (i = 0; i < N; i++) {
    vars[i] = x % S;
    x /= S;

    sum += vars[i];

  if (sum == 50) {
    for (i = 0; i < N; i++) {
      if (i == 0)
        printf("%d", vars[i]);
        printf(" + %d", vars[i]);
    printf(" = 50\n");


While it is certainly longer, it is also much more flexible. If the values of N and S are known at compile-time, we can also let the compiler unroll the loops completely for us to avoid the runtime termination checks!

Game Dev Tycoon on Linux

Yesterday Game Dev Tycoon went viral with its decision to punish pirates in an unusual, but quite ingenious way. Helping their popularity was probably the fact that the game is available for both Windows, Mac and Linux, and so users were downloading it like crazy! Unfortunately, the Linux version seems to be riddled with bugs at the moment, and for some users getting the game to run in the first place seems to be causing massive headaches.

I can’t help much with the in-game bugs (that I know the developers are hard at work on), but getting it to run on Linux turns out to be easier than expected. The only in-game bug I have a fix for is the one that makes the game hang when trying to move out of the garage – see the bottom of this post for the workaround.

Game Dev Tycoon is built using node-webkit, which essentially lets the developers build the game using web tech, while still being able to provide a binary to users. The way node-webkit packages applications is that the source code and resources are put in a zip file, and this zip file can then be passed as an argument to the nw binary, which will extract it and run the code using node and webkit (as the name implies). To make installation for end-users “easier”, the node-webkit documentation suggests that a binary is generated like this:

cat /usr/bin/nw app.nw > app && chmod +x app

which allows a user to simply run ./app. The problem with this is that this means the user is locked in to using the nw binary as compiled on the developer’s computer, which turns out to be a 32-bit Ubuntu system. This isn’t great for 64-bit users and users on bleeding-edge distributions like Arch Linux, who then have to install all sorts of legacy libraries (lib32-gconf) and ugly symlink hacks ( -> to make it run.

So, how can we do better? Well, the node-webkit documentation does say

In general, it’s recommended to offer a .nw for download, and optionally “merged” versions for the platforms where this makes things simpler.

but the developers didn’t do this. Is all lost? No. As it turns out, the unzip program is quite smart, and can extract zip files even if they’re appended onto another file! This means we can simply run

tar xzvf game-dev-tycoon.tar.gz && unzip gamedevtycoon

And all the source files of the game will be made available to us in the current directory! Now, all we have to do is install/compile node-webkit ourselves (so no 32/64-bit problems and up-to-date libraries) and run

nw ./

and the game will run as advertised with no more hassle or hacks. Yay!

I notified the developers that the code could be extracted in this way, and the reply I got was

Yes, we are aware of this.
If people don’t abuse this, there isn’t an issue.

Finally some game developers who understand the name of the game!

For those of you running Arch Linux, I’ve written a PKGBUILD which does a cleaned-up version of this and made it available in the¬†AUR and on GitHub.

Update: many Linux users are reporting an issue with not being able to move out of the garage (the loading bar just runs forever). The fix has been posted by several people on the GDT forums, and it is simply due to Linux having case-sensitive filenames. The workaround is simply to run

mv images/superb/level2Desk.png images/superb/level2desk.png

after running unzip, and everything should be working smoothly.

Why don’t people ask questions?

You probably know me. Or someone like me. The guy who always raises his hand in class or who has a question or comment regarding whatever idea is being discussed. You might find people like me annoying. I’ve never understood why there seems to be so few of us though. Ever since I was a kid, I’ve wanted to question things; to figure out why or how. Unfortunately, it seems most people don’t. Or rather, it seems as though people are afraid of asking questions. Whether it is from fear of showing that they don’t know or from not wanting to take up other people’s time or something else entirely, I’ve never been able to understand. I do know, however, that it disturbs me.

There are many quotes on questions, like “better to ask and be thought a fool for five minutes than to remain a fool forever” and the likes, but to me, there is something deeper than that. Asking questions often reveals a lot more information than just the answer to the question, almost regardless of the situation. At university, questions can not only reveal errors in the material, but they can spawn discussions on aspects of the topic that suddenly makes it so much clearer why things have to be the way they are. If you strive to understand, questions tend to give a deeper understanding of what’s going on than trying to fill the gaps yourself.

Questions are important in the professional world as well, whether to make clients think properly through the problem they are presenting you with so you don’t go solving the wrong problem or to your peers to come up with the best solution to a problem. I’ve lost count of the number of times a solution has been accepted simply because no-one asked questions at the time it was proposed, only for it to later turn out that it was a terrible solution, and several people in the group saw that it was. This is not because the person with the original idea was an idiot, but rather because there was no feedback process. Without questions, without questioning, ideas are never refined. They need critique and thought, things that are incurred by asking questions.

Indeed, even in personal relationships, questions are important. Asking others questions about themselves serves both to show them that you care, but also for you to learn more about that person. And not only whatever the answer to the question is; you often learn a great deal more about a person simply from digging a little bit. One question often turns into a deeper conversation. Even asking others questions about yourself can be hugely rewarding; what better way of getting a better understanding of how people perceive you than to ask? Many might find this difficult because they are afraid of what they might learn, but if you truly want to improve yourself as a person, then asking is the best way of finding out where you should start.

So, given all the benefits asking questions have, why are there so few people like me? Why is it that there’s “that one person always raising his hand” or that “that one person always questioning people’s ideas”? Why is it that during lectures, I’m the only one who asks questions nearly every time? Why is it that such a low fraction of people account for such a large fraction of the questions? I refuse to believe that other people aren’t curious or have doubts about things. My deepest fear is that people just don’t care. That they don’t care how a particular project turns out or whether they get a complete understanding of whatever is being taught in a lecture. They just want to get by. This is a terrible, terrible attitude, one that inhibits innovation and exploration. Come on people, raise your hands and speak your mind! Develop. Learn. Explore. Question. Care! Or go do something else.

Coding is boring, engineering isn’t

I’ve always considered myself a programmer, a coder. I came up with ideas, or other people did, and I wrote the code needed to make them work. Only recently have I realized that not only is programmer not sufficient to describe what I do, it is not even what I want to be doing.

This realization started to dawn on me after reading a (quite interesting) article by Patrick McKenzie I was referred to by a friend. I can highly recommend having a read through if you haven’t seen it already. The article goes into some depth about what working with software development entails, and, amongst other things, points out that what we do as developers is far more than just writing code. In fact, most of the hard work we do has little to do with code. Anyone can code. Being able to write code does not mean you can develop good software. In fact, if all you can do is write code you should definitely not be creating software!

What I’ve come to realize is that not only is coding not what I do most of, nor the most important skill I have, but coding is my least favorite part of the development process. Writing code is boring. Figuring out what the client wants is interesting (albeit frustrating at times). Determining how to create the best experience for the users is interesting. Coming up with a good technical design for you application, database and APIs is interesting. Writing for-loops, ifs, variable assignments and functions is not. The implementation, while important – without the code you have nothing but an idea – is simply not interesting. Nor is it challenging. It is just something that has to be done to complete the design you have in your head.

Now, I’m not saying there is not an art to programming? Far from it. What I am saying is that the art of programming lies in the thinking you do before writing the code, not in the writing itself. To be a good developer, you need to be more than just a programmer, you need to be an engineer – you have to discuss, plan, predict, design, evaluate and re-design; you have to think. Engineering is about solving problems, thinking deeply about every aspect of a system and figuring out the best way of doing things. That is the part of development I love, and the part I consider art.

Handling swipe and click with JavaScript

So, you’ve got a list with overflow:hidden, and you want to make it scrollable with touch?
And perhaps you don’t want to include a library such as iScroll, you just want simple scrolling with as little code as possible.
Perhaps you even stumbled across the fairly simple solution proposed by Chris Barr on his blog, so you start using it (replacing all .pageY with .pageX for horizontal scrolling), and your content scrolls like never before!

You then try to click one of your elements. And nothing happens. You try, and try, and try to no avail. Then you give up, throw your hands in the air, yell “I knew it wasn’t that easy”, and use iScroll.

Before I hand you the solution on a (somewhat) silver platter, let me tell you what’s causing the problem. Notice the “event.preventDefault();” Chris has in his event handlers? These are what break your onclick handler (or href), since they tell the browser not to do whatever it would normally have done when, say, a touch is initated on a clickable element. Unfortunately, they also have to be there, otherwise your phone would, helpful as it is, try to scroll the entire page.

So, how do we solve this? Well, it’s quite simple really. Think about it. What is a click? It’s a tap. And what’s a tap? It’s a touch motion that does not include any movement.

Without further ado, the solution can be found in this gist

The only additions to Chris’ code is that we monitor whether touchmove has been triggered since the last call to touchstart. If it hasn’t then it was a click.
What you decide to put inside the if in touchend is what you’d normally but in your onclick handler. If you’re using jQuery, you would typically use $(‘a’) or something similar to get the element that the onclick event would normally have triggered on.


Update 2012-04-26: Changed the code to delay handling the click in case the user just paused while swiping.