BMNick.com

The Underappreciated Count Sort

2011-10-07T00:00:00+00:00

Seeing as I need to get a full time job at some point, I've started getting back into the process of having interviews. For a software person, this means getting a lot of technical questions, usually including some basic algorithm optimization questions. I have a secret weapon for these that too few people know about - the count sort.

The count sort is an unstable sorting algorithm that operates in O(n) time by consuming a huge amount of memory and creating a new, intermediate representation of the data. It is primarily useful for long lists of mostly repetitive integer data over a short range of values. As a warning, it is very inefficient with memory in certain cases, but for interview situations, this is unlikely to be important. However, if you want to pull this algorithm out in an interview, knowing the downsides would certainly be a good idea.

The Algorithm

So now that I've talked it up a bunch, I might as well explain what it actually does. Count sort iterates over a list of integers to find the maximum and minimum value. After these have been determined, it creates an array sized to fit all elements. It loops over the original list again, incrementing the value in the intermediate array at the index of the original list value. Finally, the sorted list is reconstructed from the intermediate array and the original list and intermediate array are destroyed. Pseudocode should make this much more easily understood

def count_sort(unsorted_list)
    min = int_max, max = int_min

    for(element in unsorted_list)
        if element < min
            min = element
        if element > max
            max = element

    counts = new Array[max-min]

    for (element in unsorted_list)
        counts[element - min]++

    sorted_list = new list

    for (int i = 0; i < counts.size; i++)
        while counts[i] > 0
            sorted_list.append(i + min)
            counts[i]--

The big advantage to this algorithm is that most of the time in interviews you are only asked to optimize runtime, and this is a wonderful way to do that. Unfortunately, it's not usually as useful in real life due to the space it takes up.

One other advantage is that sometimes you just need the number of occurrences of item in an array, and you can save the reconstruction step. This was the case in both of the interview questions where the count sort really helped me to impress. I'm recounting them here to give you a look at count sort in the wild.

The Palantir Question

This question was asked to me very recently by Palantir at the RIT career fair. It was just a one off question while I was speaking to someone to get a general idea of my technical skills. As a sidenote, the count sort let me beat their recruiters best solution time of O(N log N) with an O(2.5N) solution.

The question was, roughly, as follows. I am given a list of integers, each doubled, but not necessarily all numbers from 0..N. For example, I might have gotten a list of 1, 1, 5, 5, 8, 8, 14, 14, etc. Now, my interviewer deletes one of item from one of these pairs and shuffles the array around. I am told to find which value he deleted.

As you might have guessed, in comes the count sort. I do the standard count sort setup, reading through the array to get the min and max and fill in the intermediate array. Instead of recreating the original list, I simply loop over the intermediate array searching for the first element with a count of 1. This gives me one loop to get the minimum and maximum, one to build the counts, and roughly half to loop over the counts array(assuming that the sequence is mostly full).

The Google Question

This question was asked to me by Google in an on-campus interview with them last spring. It started off as a simple algorithm to get to some solution for this problem, then moved into optimization hunting. I'm detailing my final answer, which may be slightly incomplete. I may also make some slight errors in recounting the exact details of the question.

The question is slightly complicated. You are given a collection of popsicle sticks, and told to join them together. For each combination, you are charged the sum of the lengths. Minimize the cost to you and do so as fast as possible.

The solution to the first problem took some time to work out, but it boils down to "join the 2 smallest available sticks first". This means we need something sorted, and we need something that has a fast sorted instertion time(we need to keep joining the parts we already have). Additionally, I was told to use a magical data structure that was like an array for random access, but dynamically resized itself like a linked list. I'm not really one to look a gift horse in the mouth, so…

First thing for the solution - you need to build your count array. Since we have that magical data structure, you don't even need to get the min and max, so that's one less iteration over the list. Now that we have that intermediate built up, we can start iterating over it joining things together. To do this, I stepped through the array, decrementing the first 2 values I found, adding them, adding that value to the accumulated cost, and then incrementing the count at the sum. I then started from where I was in the array and stepped forward, doing the whole thing over again.

Unfortunately, in the situation where I was given 2 popsicle sticks, of length 1 and 1000000, this is a horribly inefficient implementation. However, in the average case with mostly small values, it is one of the most efficient solutions available.

Conclusion

The count sort is a rarely mentioned algorithm in most classes, at least here at RIT. There are good reasons for that, but when you need a sort that is very very fast and you don't need to worry about the memory usage, the count sort is a great solution.

Of course, now that I've let the secret out, I'll have to start finding other ways to be unusually successful at algorithm interviews...

Thank You Steve

2011-10-05T00:00:00+00:00

I hate that I have to write this. Some part of me knew it was coming, but I still wish it didn't need to be so soon. Steve was an inspiration for me. I was a late convert to Apple, but after discovering what Apple products were really like I switched completely. Steve was a big part of making those wonderful things.

I wish I could say something more meaningful, but I'm not exactly a word smith. Steve Jobs was a visionary, and we will miss him.

And now, I need to go do my best to make something beautiful. I can think of no better tribute.

The greatest addition to language in recent memory!

2011-05-01T00:00:00+00:00

There's been a lot of talk lately about the use of "That's what she said" in conversation and determining appropriate uses for it. Using this recent progress in the important field, I have created what I think it the culmination of the research for Mac users - an autoresponder for iChat! You no longer need to think to get that's what she said responses to all your friends and work colleagues!

All you need to do is put twss.rb somewhere, change the path in twss_autoresponder.scpt, and run twss_autoresponder as an applescript on message recieved in iChat!

Ruby C Extensions

2011-03-13T00:00:00+00:00

I've been interested in using C extensions with ruby for a while, so I decided to do a bit of exploration in order to see what it's actually like. I decided first of all to put together a simple extension that does some floating point multiplication operations in C. I threw in a loop too, so I could so some significant benchmarking after the extension actually exists.

Writing C Extensions

Ruby is a great language, but I will admit that, at times, it can be a bit slow. Also, some things have C APIs but no ruby API yet. What do you do if you want to deal with this? You're going to need to use C, but that doesn't mean that you'll need to work with pure C for the whole project. That's where ruby C extensions come into play. These C extensions allow you to use C alongside ruby in order to work with other codebases or drop down to C for speed.

My opinion on using this for optimization is pretty simple. Write in in readable, simple ruby first, then start benchmarking it. If it's too slow, look at your overall algorithm first. Then try optimizing in ruby. If it's still too slow, then pull what you can out to a C extension. And if it's still too slow, you have a problem, in that you need to drop to ASM or start throwing more hardware at it.

So, first lets go over actually writing the C extension, then we can explore what this does for speed. Writing C extensions sounds very scary, but really isn't if you already know C. A typical C extension consists of some number of .c and .h files, and an extconf.rb file. The extconf.rb file will handle generating a makefile which will package everything up to a format usable by ruby.

For the purposes of this post, I have a very simple extconf.rb to show off. This should show you the basics. You can add conditional compilation and other fun toys in this, but that's beyond the scope of this discussion. I'll cover that if people are interested at another time. This will get all C code in the CExt folder, generate a makefile for it, and bundle it up in a .so or .bundle, depending on your OS. Then you'll be able to get at it using require_relative '/path/to/CExt/CExt'. Anyway, here's extconf.rb!

Very simple, right? Once we have some actual C code written, compiling it is going to be as easy as 'ruby extconf.rb && make'. This will generate the makefile and compile everything in one shot.

Getting to the C part of all this, I'll throw up some C code first, then step through and explain all of it. Here we go!

This looks uglier at first than it actually is. I'll walk through it a step at a time, then we can do some benchmarks.

First we include the ruby header, ruby.h, to get types and methods necessary for building against ruby. If anyone doesn't know how #include works, you might want to skip down to the results, because it's only going to get worse from here on.

Next up, on line 3 we declare a variable to store the module we're in the process of building. Going to be boring until we get something in there, of course. Next up, we declare the methods we're going to be building later. We could do the whole method here, but I prefer doing it this way to make it a little more obvious. In this case, we're creating two methods that return an arbitrary ruby object(the VALUE type) and take in an arbitrary ruby object referring to the calling context, called self. Ruby will pass down self automatically, but C is a little slow, and needs to know what it's getting.

Now we get to a method that, if it doesn't exist, is going to break your whole extension. There must exist a method in one of your C source files called Init_"extension name". Since I unimaginatively called this CExt, I have Init_CExt as my method. This is the method called first when you require the extension. In here, we're doing a few things. First, we're filling in that variable I said would contain the module we're building with a ruby module definition in line 9. rb_define_module simply takes a string of the name of the module to define. If you wanted to create a class instead, the method would simply be rb_define_class("class name", superclass).

Next, I'm defining two singleton methods on the module, floatmul and loop_floatmul. I'm defining these as singleton methods for simplicity, so I don't have to include the module or create a class from it. rb_define_singleton_method takes 4 parameters. The first is the ruby module or class to add these methods to, in this case CExt. Next you provide the name for the ruby version of the method. Then we need a function pointer so ruby knows what to actually run. Finally, we have an integer counting the number of arguments to the function in ruby. The C method will have one more, the self object, when we define it.

Now we can move on to the actual methods I'm defining. I've got a simple method that multiplies the double 2.0 by the double 3.0, then returns the value as a ruby float. The C double type needs to be converted to a ruby Float, which is why we can't just return a double from this method. rb_float_new magically generates the ruby Float we need.

Next is the method I'll be using for benchmarking these things. It defines some doubles, loops through 1000000 times, multiplying the floats, then returns the final value as a ruby float. This could definitely be more interesting, but I just wanted something to chew through some time, to use for benchmarking purposes.

So that's a C extension for ruby. Much more simple than you would think for getting ruby to talk to C, right? Now, to see if it's worth if for performance reasons.

Benchmarking Our Contrived Example

Now these benchmarks don't necessarily mean anything, since they're not stringently conducted and just have representative results shown. However, they demonstrate the results that are typically shown in these sorts of comparisons.

Using the same extension from up above, I ran comparisons 4 ways. I implemented a pure ruby version of the loop, a ruby loop calling down to the single multiplication C method, a ruby wrapper around the C loop method, and a pure C version that just runs through the loop. I'm including the code, then I'll discuss the results a little.

Results

Looking at this, you would expect that the pure C and external.rb would be almost identical, then mixed would be better and pure.rb would be the worst perfomance, right? You would actually be wrong with that assumption. Also, this was run against MRI ruby 1.9.2p136 installed through RVM on a 2.4 GHz Core 2 duo Macbook Pro running OS X 10.6.6. Enough delays, here's the results!

Surprised at all? I hope it's not too different from your expectations! If you have everything put together right, you can run the comparisons yourself and you should get similar results.

Unfortunately, the overhead of calling down to C can get significant under certain circumstances. In this case, that's what kills the mixed.rb test. All the conversions from C doubles to ruby Floats actually causes this to be slower than equivalent ruby code. This is definitely something to think about when you write a C extension. Also of note is that external.rb, which just calls down to the c loop, is 2x slower than purec! Most of this is that the ruby interpreter needs to spin up, load the CExt module, then call out to the native code. However, note the fact that it is still approximately 8x faster than the equivalent pure ruby code.

I hope what you take away from this is that, yes, C extensions can really speed up certain code. However, this does not mean that writing something in C is a silver bullet. Look at what's really happening, speed up your algorithm, and know what different languages do best.

The other reason to look at a C extension is that you can use C to talk to other languages and other libraries, too. So, for example, if you wanted to interface with Redis, and no gems existed for this, you could write your own C wrapper that speaks up to ruby. Luckily, others have already done that work in this particular example, but other libraries exist without ruby APIs.

Wrapping Up

C extensions are definitely a very useful thing, but they aren't always the best choice. As we saw above, writing an extension does in fact have some overhead that pure C does not have. Another issue is redistribution of your application. If you don't write platform-independent C code, which is not the easiest thing, you will start running into issues there. This is one of the big issues people have with ruby on Windows. People will break down to C without accommodating for the fact that they may not have the same resources available on all systems. Using too much *nix specific code means that windows people can't use your product.

Finally, if you have any topics you would be interested in seeing me write about, let me know on twitter. I'm @bmnic if you don't already follow me. No guarantees I'll cover it immediately but I'll add it to my list of things to write about. Send me any comments you have on this to twitter, too.

Running Nesta on Amazon EC2

2011-03-03T00:00:00+00:00

So I finally have my site back up and running, and I've finally trashed the old hacked-together CMS I was running. I decided to go with Nesta on this new incarnation, and I thought some people might enjoy the chance to see how to get it running. Right now, I'm serving Nesta with Unicorn behind nginx on an Amazon EC2 server. I'm also serving up a few random other things from the server so keeping configuration modular was a goal for the configuration. I also wanted to have unicorn run on restarts, so I've got an init script for launching off unicorn. The other complication is that I like having RVM running so I can pull off another version for testing or other apps.

I won't be covering every step of the process, but I will cover the basics at least.

Installing RVM

RVM is pretty easy to use and install, but I'll throw the instructions in here anyway. Following the instructions here will give you the full process, but all you really need to do is run a script and update your .bashrc or .zshrc file. The shell configuration updates will come from the script once you run it. Run the install with:

Installing Unicorn, bundler and wrappers

First off, install a ruby interpreter, setup a gemset and install some of the gems you'll need. At least on a micro instance, this is going to be very slow. I suggest running it under a screen session in case you drop the connection. When I say very slow, I mean upwards of half an hour, at least. Skipping ri and rdoc will speed this up a lot.

Now that we have the interpreter and gems installed, we'll need to make some wrappers to get at these gems from outside of rvm. We could have gotten around this by using a system-wide install, but this works fine for me. We need wrappers for bundler and unicorn. These are the only wrappers we need, since they're the only executables we won't be running interactively.

At this point we have all the gems we need as well as a nice environment to run them under.

Create the Nesta project

Next step is configuring a Nesta project that we can actually serve. Here we're going to create the actual nesta project directory, as well as make a link to it in the /var/www directory. You can use whatever directory you want here, but keep track of it for later. We're also going to create a few directories that will be used later on by unicorn.

The nesta project will also be under git source control for an easier way to deploy changes. Skip the --git part if you choose to handle deployment in another way. You'll also need to make a config.yml file and move any content you have into the appropriate places for nesta. Once you've done that, it's time to get unicorn set up to run.

Configuring Unicorn

First of all, I'll show you the unicorn.rb file, then I'll explain what's going on. I have this file stored in nesta-dir/config/unicorn.rb but again this can be wherever you want to store it.

The first change you might be interested in making is the number of worker processes. This should be the number of cores/processors in your system. I have 1 since I'm only running on a micro instance.

If you're using a different location for your Nesta project, you'll want to change the working directory.

You'll almost definitely want preload the app and having REE specific config won't hurt. If you choose to switch to using REE later it should speed up a little more.

I chose to use a unix socket instead of an actual port on the server for communication. You can also change the pid location here too, but that will need to be set in the update script too. Finally, there's log paths, which you can change up if you like too.

Setting up the init script

The last backend step left is the init script.This is fairly simple, but saves you the bother of restarting unicorn if your server needs to be restarted and automates redeploys. It's mostly self-explanatory, other than changing where certain things point if you didn't follow my layout.

"service nesta start" will spawn the server, "service nesta stop" will kill it from the pid in the pid file, and "service nesta restart" will force a reload of the code. Content changes shouldn't need restarts, but if you update Nesta or do custom patches you'll need to reload it.

You'll also need to update the links for your init script, which you can do manually or with "update-rc.d nesta defaults".

Configuring nginx

Last step is telling nginx how to serve this to the user. I'm using sites-available and sites-enabled so I can serve multiple sites without messing with one giant config file. Throw this config file into /etc/nginx/sites-available and put a symlink to it in /etc/nginx/sites-enabled.

Essentially all this is going to do is pass requests into unicorn unless they exist in the public directory of your nesta project. It knows where unicorn is running from the upstream we declare earlier on.

Finally, you just need to start everything up with your new configurations, so "service nesta start" and "service nginx restart"

Assuming everything worked out well, this should get you a running copy of nesta on your server. Now you're in the same spot as me, needing content and possibly theming. Good luck!