sn.printf.net

ZumoDrive, Tarsnap

2010-03-11T13:00:39+00:00

ZumoDrive made this post: http://blog.zumodrive.com/sometimes-you-have-to-roll-a-hard-six

Colin Percival, maker of Tarsnap, made this reply: http://www.daemonology.net/blog/2010-03-11-zumodrive-rolls-a-hard-six.html

The Hacker News discussion: http://news.ycombinator.com/item?id=1183308

On Point-free Style

2010-03-08T23:34:59+00:00

With a blog posting and then with some people sperging about the posting on Hacker News, I feel it’s time to write a post on the utility of point-free style, or “point-less” style as some call it.

One Hacker News poster, hristov, apparently reporting live from spergatory, had this to say:

First I would like to say that I am only learning Haskell, and have done nothing but toy programs thus far. But the point free style is something that I really do not like about Haskell. It makes code much more complicated for the dubious benefit of saving a couple of keystrokes.For example, take the code from the article:

This is without point free style: sum xs = foldr (+) 0 xs

And this is with: sum = foldr (+) 0

You have saved a total of 4 keystrokes. And made that line of code look really weird and hard to understand.

Thus, now when you look at code you cannot say at first glance how many arguments a function takes. The second line of code looks like sum takes 0 arguments. But you have to look more carefully at the definition of the function, and see that it uses foldr, and know that foldr takes three arguments, but here only two are defined, and then you figure out that someone is using point free style and then you figure out that sum, actually takes one argument that must be a list.

And this of course, is a simple example. Here we have a big clue in that the function is defined with zero arguments, and most functions take at least one argument, therefore the function probably uses the point free notation. But for a more complex and longer function, the point free style can very easily confuse someone reading the code.

So yeah, I think point free style is one of these annoying features of Haskell, that seem to be put in so that users of Haskell can pat themselves on the back and feel like they are very smart. But in the end it does not add much to the language and makes it much less accessible.

I have no idea whether this is a common form of misconfigured worldview. Anyway, the Hacker News poster is right, and the blog poster is a bad person for confusing him. (This shouldn’t be surprising, because blog posters who put a “Conclusion” section in their posts tend to be bad people.) But where the Hacker News poster is right is where he says that using point-free style for top level definitions is pointless. He’s sort of right. He’s right because it is pointless. The definition sum xs = foldr (+) 0 xs is generally more readable to everybody than the definition sum = foldr (+) 0. It does no harm to write out the parameter, and it does make things more clear, because you’re advertising the arity of your function. It helps anybody who’s new to your codebase.

But it’s not as bad as the Hacker News poster makes out, which is why his worldview is misconfigured. It’s not so bad when you see

sum :: Num a => [a] -> a
sum = foldr (+) 0

It’s not! You have the type signature right there. It’s basically standard to put type signatures on all your top-level functions. And when you do that, it’s not so confusing.

Both the blog poster and the Hacker News poster are missing the more useful applications of point-free style, which is for the purpose of passing parameters to higher order functions and for avoiding horrible nests of parentheses.

For example, when solving Project Euler problem 62, the easiest way to tackle it was to chain a bunch of list manipulation operations together.

minimum
. head
. filter ((== 5) . length)
. concat
. map (groupBy ((==) `on` sort) . sortBy (compare `on` sort))
. groupBy ((==) `on` length)
. map (show . (^3))
$ [0..]

We have a ton of point-free examples there. It is natural to use point-free style and function combinators for everything there. If we didn’t, it would look like this:

minimum
 (head
  (filter (\ns -> length ns == 5)
          (concat
           (map (\xs -> groupBy (\x y -> sort x == sort y)
                                (sortBy (\x y -> compare (sort x)
                                                         (sort y))
                                        xs))
                (groupBy (\x y -> length x == length y)
                         (map (\k -> show (k^3)) [0..]))))))

How do you like that? The parenthesis management is horrifying. We could, then, make one small baby step towards point free style, and bring back the big chain of function composition, without touching the parameters we pass to map, groupBy, sortBy, and such.

minimum
. head
. filter (\ns -> length ns == 5)
. concat
. map (\xs -> groupBy (\x y -> sort x == sort y)
                      (sortBy (\x y -> compare (sort x)
                                               (sort y))
                              xs))
. groupBy (\x y -> length x == length y)
. map (\k -> show (k^3))
$ [0..]

An alternate baby step would have been first to make the parameters in point-free style.

minimum
 (head
  (filter ((== 5) . length)
          (concat
           (map (groupBy ((==) `on` sort)
                 . sortBy (compare `on` sort))
                (groupBy ((==) `on` length)
                         (map (show . (^3)) [0..]))))))

But really, we should just go for the best of both worlds.

Conclusion

The blog poster is dumb for making a blog posting about point-free style and demonstrating with top-level functions. The real benefit of point-free style is for making function parameters and avoiding big gobs of syntax. But don’t hate on the blog poster, because he doesn’t say wrong things. The Hacker News poster is excused, because he’s a newbie to Haskell, and he’s reacting to the blog poster’s examples. He also gets a pass because you can tell he’s generally an immature retard, which we know because he says, “You have saved a total of 4 keystrokes,” as if keystrokes was ever the point. Go count your lines of code, son, and when you’re done counting them, then maybe you get to make grandiose generalizations about the users of and an emergent property of a language you don’t understand.

Things I have learned

2010-03-08T19:33:21+00:00

It is hard to debug UI issues, because you can never tell if that white screen you're looking at is really the widget you think it is, or maybe it didn't load and that's the background, or maybe it loaded and it's just colored entirely white, or maybe something white is on top of it, or maybe your code is in an infinite loop and it erased the last UI element, but hasn't loaded the new one.

Argh.

I got the phone taking a picture without any of the camera interface crap, and saved it to memory. Now to take a million photos and save them all.

Flipping Out and Coding

2010-03-05T18:43:07+00:00

The last time I just flipped out and wrote some code at full intensity was at a programming contest two and a half years ago, in my senior year at college. The last time before that was at a job in the summer between sophomore and junior year, connecting up some LabVIEW diagrams. The last time before that was in my first semester, freshman year, when I made a chess engine in a weekend for some class I was taking. Then the last time before that was in 11th grade when I told somebody I could make a tetris game in a day and proceeded to do so in QBASIC. The last time before that was when I made a fancy spreadsheet in Microsoft Works ‘99 to manage an NCAA basketball pool I was running in eighth grade. That was the first program I wrote, and there are no more instances of intense concentration before that, that involved programming.

I didn’t know as much about programming then, and yet somehow I feel that I had more energy to just get stuff done, to just get code down on the computer, to quickly and nonchalantly look up documentation without making a big deal about it, to just care about getting something up and working.

I’m going to devote this weekend to the art of flipping out and getting some code done. Sorry about posting late today. I’m sure the last six hours were miserable for you.

Visualizing Duplicates and Triplicates

2010-03-04T13:00:37+00:00

This is a followup to the previous post, Estimating the number of distinct numbers. In that post, we ask how many distinct values we’ll get when making 100,000 independent picks from the integers between 1 and 1,000,000. The answer we got was approximate, non-rigorous, and 95,163.

I began by documenting my first attempt to solve the problem, which ended up going sorely wrong. We let x be the number of distinct values we’d end up getting. Based on the reasoning that x/1,000,000 is the probability of a given pick being a duplicate, we solved the equation

x = 100,000 − (x/1,000,000)*100,000

which gives the solution

x = 90,909.090909090909090…

This ended up being wrong. But the reasoning sounded so right! What went wrong there? We subtracted the number of duplicates from the number of picks, and wouldn’t that give us the answer?

No. It wouldn’t. Let’s look back.

We correctly surmised that (x/1,000,000) would be the expected proportion of picks that were duplicates, since there were x distinct values, and the probability of any pick hitting a set of size x, out of a set of size 1,000,000, is x/1,000,000.

We then went and applied that proportion to an equation describing the number of distinct numbers picked.

That was wrong. There’s the picks, and there’s the numbers that get picked, and they are two different things. We took a proportion describing the domain, and applied it to the range.

Another way of looking at the problem is by looking at the actual way the proportions relate: When a number is picked twice, 2 of the picks get counted as “picks that ended up being a duplicate number,” while 1 distinct value gets created. Our equation above assumed that 0 distinct values would get created, instead of crediting pairs of duplicate picks for the one distinct value that they produce. Later, when we divided the proportion by 2 (for a different reason), we ended up correcting for this error. The correction would have been exact, if it were not for the presence of triplicates. Here’s a drawing.

As we pick numbers, we start at the bottom and follow the curve upward, counting each new pick we get. Each horizontal row represents some number that has been picked, and as the curve doubles back, that means the number got picked again. Since we have duplicates we’d “turn around” before reaching 100K and go in the opposite direction, covering a few rows again — they represent numbers that got picked twice. The value A describes the number of picks we’d get that end up being duplicates, and B describes the number of distinct values we get. C describes the number of values that end up being picked more than once. C ends up being approximately half of A. It would be exactly half of A, if there were no triplicates. But we have to consider those, and the curve curls back up to represent triplicates. There should probably be another curl, a fraction of a pixel in length, back down, to represent quadruplicates. We’ll approximately calculate the actual number of triplicates we expect, and quadruplicates, and such, tomorrow, since I’m sick of blogging today.

Helping People

2010-02-28T13:00:44+00:00

Look at me. I have a blog. Look, my opinions are on it! Read my opinions. Revel in how witty I am.

I have opinions on a lot of things, and you should listen to them. Hear me talk about programming. Here, me talk about programming languages. Revel in how witty I am. Hear me talk about how you should live your life. Hear me talk about the latest phenomena in modern internet society. These are my opinions, and they are mine.

By all means, write a counter-posting, offering a point-by-point rebuttal to my points. But make a link, pointing back to mine. Don’t have a blog? Well then, nobody cares about you. You don’t exist. You don’t matter. You are nothing. Get a blog. And make a link to mine.

I have insights into the world that you could only dream of understanding. Make a link to my posting on the latest news aggregators.

Post comments about my opinion on the news aggregators. Is this a new username you’ve chosen? You should think twice before having opinions. Build up some cred by not having an opinion about anything. Post factual rebuttals about mistakes of fact, but don’t even think about attacking the general conclusion. That way, more people will listen to you, and so you will be better able to spread your intelligent insights into the populace.

HELPING PEOPLE

is the purpose of having a blog. Your blog must be productive and help people understand the crazy world that we live in!

My blog is not “better” than yours, for how could one design a valid comparison function between blogs? But yours will be improved when you make a link to mine.

I’m going to make more postings, acting like I have new ideas. You will read them and be informed. My posting is not self-aggrandic—it’s a humble quest to increase the probability that wayward net travelers come across something of substance, so that their lives, and mental health, and the accurateness of their opinions, and those of people around them, will be enhanced substantially by listening to what I have to say about programming. Make a link.

My metadiscussion about blogging is more profound than any non-metadiscussion that you may have. I’m now talking about metadiscussion, which makes this level-two metadiscussion. Here is where a normal peon would make a joke about maxing out at level-seventy metadiscussion, but I do not limit my sights to topical references. It is inevitable that the first seventy levels of metadiscussion be followed by a seventy-first, and so on. Here we are, talking about enumerating the levels of metadiscussion. We could now use ordinal numbers to identify the subsequent meta-levels, but in mentioning such an idea, we have surpassed all the ordinal numbers.

I hope you find it worth your time to strive to have a worldview similar to mine.

Writing Numbers

2010-02-27T13:00:54+00:00

In my previous post, I discussed some stuff, and I wrote some numbers down.

1,000,000

95,163

100,000

91,000

90,909.090909090…

95,162.58

Before I inserted the commas into the numbers, I found that it was easy to make mistakes by writing 100000 instead of 1000000. Generally, that was hard to read. However, 91000 isn’t hard to read, and neither are 95162.58 or 95163. So in the previous post I inserted the numbers.

The point of this post is that while previously, I always refrained from putting period separators into numbers, I have now changed my attitude, slightly.

I would also consider using thin spaces as the digit grouping separator, as suggested in Jukka K. Korpela’s epic, Math in HTML (and CSS). However, that would take too much effort.

It would be nice to have a variant of Markdown that took sections bounded by dollar signs, as seen in LaTeX, and converted them to HTML and CSS in the ways that document recommends.

Virtual Currency

2010-02-27T01:19:41+00:00

What's the difference between real currency and virtual currency?

The only thing I can really come up with is that there are laws pertaining to real currency.

I keep waiting for someone to make a decent virtual currency exchange. I think all you would have to do is get some VC funding and get a few major players onboard with you. As soon as a large enough currency allows you to trade for real money or goods, you'll be able to do all your business government tax-free. You'll still have to pay Facebook or Microsoft or Apple or whoever, but not your local or federal government.

It is a brave new world we live in where every company is trying to build their own money system.

Switch!

2010-02-27T01:09:48+00:00

I have switched to primarily developing an app that will let people share sketches in a more... asynchronous way. I now have an app that I can scribble a white line on. Woo!

Hilton has agreed to work with me on our iSketch clone, and I think we'll get a lot accomplished, I'll be doing server dev, marketing, support, and some client-side, and he will do the majority of client-size.

I think that a major mistake many app developers make is to follow these three steps:
1) produce quality app
2) ???
3) profit!

Except when 3 never materializes, they blame anything and anyone for it. "The app store is broken!" "Apple is fucking us again!" "The users don't appreciate quality apps!" "Rargh!" Step 2 is actually marketing. If you took the best product in the world and stuck it on a random shelf at Wal-Mart, you could expect only a handful of sales, at best. This is what you're doing on the App Store.

Estimating the number of distinct numbers

2010-02-26T13:00:02+00:00

In a previous post, I estimated that if you picked a hundred thousand random numbers from one to a million, independently, i.e. “with replacement,” as some call it, you could expect to get 95,163 different values. Maybe sometimes you’ll get more or less (since the outcome is random), but you’ll probably get something close to that number. That’s an approximate guess — I’m not sure if that’s really the closest integer to the true average value — and that’s because I’m an amateur when it comes to math. We might as well chronicle how you’d get to such a value.

This post is going to be about how to come up with approximations for things, when you don’t know the “real” way to get the right answer.

So, if you pick 100,000 numbers from one to a million, and if you pick them independently, how many distinct values do you expect? Well, you’d expect something like 100,000 distinct values in total, with an adjustment for duplicates. So for each value you picked, you’d think that they’d have a 10% chance of being a duplicate of another, since there’s about 99,999 other values for them to collide with, out of a million, and 99,999/1,000,000 ≈ 0.1. So we’d expect 10% of the values to be duplicates, and so we’d expect 90,000 distinct values to be produced.

But wait! We based our estimate of 90,000 on our assumption that our estimate would be about 100,000. Now we have a more accurate estimate, so we could make a new estimate based on that. Since 89,999/1,000,000 ≈ 0.09, we’d expect 9% of the values to be duplicates, and so we’d expect 91,000 distinct values to be produced.

With our new estimate, we could perform an another iteration to produce a more accurate estimate of the number we’re trying to calculate. Or we could set up an equation and solve it. Let’s call our estimate x. If so, we think the probability of a distinct value would be x/1,000,000. That means (x/1,000,000)*100,000 will give the number of picks that are duplicates. The number of distinct values (x) would be equal to the number of picks (100,000) minus the number of duplicates ((x/1,000,000)*100,000), giving us the equation x = 100,000 − (x/1,000,000)*100,000. Simplifying, we have x = 100,000 − x/10, which simplifies to 11x/10 = 100,000, and then x = 1,000,000/11 = 90,909.090909090…

So we’ve estimated that the number of distinct values is 90,909. Is that right? Let’s run a few simulations in Python:

>>> [len(set([random.randint(1, 1000000) for i in xrange(100000)])) for j in xrange(5)]
[95101, 95311, 95268, 95270, 95105]

It looks like we’re way off. You could say we’re off by a factor of two — we expected about twice as many duplicates as we got.

What went wrong? Let’s ask this: What did we assume? We assumed that each pick’s probability of getting a duplicate was equal to the number of total distinct values we’ll end up getting, divided by a million. Is that right, though? (Based on our simulation, apparently not.) We can answer why not by imagining ourselves picking a bunch of random numbers. What’s the chance of the first pick being a duplicate? Zero. What about the 50,000th pick? It would be approximately 50,000/1,000,000, i.e 5%. Then, on our 100,000th pick, the chance of it being a duplicate would be approximately 100,000/1,000,000, i.e. 10%. So the probability goes from 0% up to about 10% in a roughly linear fashion. The average is then 5%. So we can expect 5% of our picks to collide with existing picks, and so we predict 95,000 distinct values.

That’s closer. Of course, with that new estimate, instead of having the proportion go from 0% to 10%, we now can expect the proportion to go from 0% to 9.5%. And so we expect the average to be 4.75%. Taking 4.75% off of 100,000, we get 95,250. We could iterate this again, but let’s just solve an equation, like we did before. The solution to x = 100,000 – x/20 is 2,000,000/21, approximately 95238.

That seems like a reasonable guess. It’s just an estimate, though. Is it too high or is it too low? Right now we’re assuming the rate of getting duplicate elements grows linearly. Of course, it doesn’t. We pick the first 50,000 picks with a lower rate of getting duplicates than the second 50,000 picks, which means our total number of distinct elements will be greater than 95,238/2. We’ve already accounted for that fact — we expect 3/4 of our duplicate elements to be generated from between the 50,001st and 100,000th pick. But we haven’t accounted for the fact that the rate of getting duplicate elements is actually higher than we’d have expected! This means our estimate of 95,238 is an overestimate.

What shall we do then? Let’s calculate the expected number of distinct values after the first 50,000 picks, and then based on the number we get, calculate the expected number of distinct values for the next 50,000 picks.

First we use the same equation as before, but with 50,000 in place of 100,000. Let’s let w be the number of distinct elements after 50,000 picks. Simplifying w = 50,000 – (w/1,000,000)/2*50,000, we get w = 50,000 – w/40, i.e w = 2,000,000/41 ≈ 48,780.5.

Then, let x be the number of distinct elements after the next 50,000 picks. We’re expecting the average collision rate to be the average of x/1,000,000 and 4.87805%. So we have x = 48,780.5 + 50,000 – ((x/1,000,000 + 0.0487805)/2)*50,000. Simplifying that, we have x = 98,780.5 – (x+48,780.5)/40, i.e. 41x/40 = 98,780.5 – 48,780.5/40, i.e. x = (40*98,780.5 – 48,780.5)/41 ≈ 95,181.5.

That should be a better estimate. Let’s do some simulations. Let’s take some averages of 100 simulations.

>>> sum([len(set([g.randint(1, 1000000) for i in xrange(100000)])) for j in xrange(100)])/100.0

95160.339999999997

>>> sum([len(set([g.randint(1, 1000000) for i in xrange(100000)])) for j in xrange(100)])/100.0

95152.179999999993

It seems we’re still overestimating the number of distinct values. After all, each of our iterations produced an overestimate.

I suppose we could break up our approximation into 4 blocks and get a more precise value. Here’s an alternate choice. Set up a differential equation and solve it!

If the number of distinct values we have at a given time is y, then the chance of getting a collision on our next choice is y/1,000,000, and the chance of getting a non-collision is (1,000,000 – y)/1,000,000.

So, that’s our expected rate of change. And so we have the differential equation

y’ = (1,000,000 – y)/1,000,000 = 1 – y/1,000,000.

I’ve completely forgotten how to solve equatons like this, so let’s note that y = K*exp(-t/1,000,000) is the solution family to y’ = -y/1,000,000, and now we can try to tweak that a bit to get a solution to y’ = 1 – y/1,000,000 Just by common sense, we expect a function that decays to 1,000,000, so we can guess y = 1,000,000 + K*exp(-t/1,000,000). Given that there are 0 distinct values after 0 picks, we have the initial condition of y(0) = 0, and so we can solve for K, and thus y = 1,000,000 – 1,000,000*exp(-t/1,000,000).

So, if things go at our “expected rate of change,” we’ll expect them to track along that differential equation. There will be some random noise, but let’s see where this goes.

Plugging in t=100,000, we get 1,000,000 – 1,000,000*exp(-1/10) ≈ 95,162.58.

So that’s how I came up with 95,163, and it seemed close enough to the experimental evidence, so it was good enough for me.

Is it really right, though? Is our method of using a differential equation right? Well, it’s right in the sense that it gives a close answer. It’s a good idea — better than the weird approximations we were using earlier. But the truth is that we’re approximating a discrete process, making 100,000 independent picks, with a continuous function. It’s kind of like the reverse of finding a numerical approximation to the differential equation.

For a more revelatory example, suppose we try using this equation to give us an estimate for the number of distinct values after 1 pick. We see that 1,000,000.0 – 1,000,000*exp(-1/1,000,000) ≈ 0.99999949999619275. Instead of getting the right answer, which is 1, we get an approximation based on the idea that the growth rate of the function falls from 1 to 999,999/1,000,000 on the interval. It’s off by about half a millionth.

What about after two picks? Our formula gives 1.999997999984771. But the right answer is 1.999999. We’re off by one millionth, or two half-millionths — one for the first pick, and one for the next.

Generally speaking, on each timestep, we can expect the continuous approximation to take a half of a millionth off the growth rate, due to the continuous nature of the curve. To compensate for this effect, we might add an extra half-millionth factor to the final estimation. Multiplying 95,162.58 by (1 + 1/2,000,000), we get 95,162.63. That’s not enough to care about.

What if we wanted a really exact answer, though? We might say, okay, we’re going to solve this using Euler’s method, just like we’re supposed to, and we’ll update the expected value each time. We can do that.

>>> y = 0.0
>>> for i in xrange(100000):
...     y += (1000000.0 - y)/1000000.0
...
>>> y
95162.627205942103

Is that the exact, right answer, though? Is that the true expected value? Or have we found another approximation based on incorrect assumptions?

Remember that this is a random process, and that the future rate of change depends on the outcome of current random numbers.

Let’s play a game. Roll a die, and square the value. You might say (and you’d be wrong if you did) that the expected value of a dice roll is 3.5, so the expected value of its square must be 3.5², which is 12.25. You’d be wrong, though, because the expected value is 15.16666… That’s because the numbers around 3.5 spread out nonuniformly when you send them through the squaring operation. If you sent them through a doubling, instead of a squaring, their distances would separate uniformly, and the expected value would remain correct. With squaring, they separate nonuniformly, and the expected value doesn’t.

We can’t just run the expected value through a for loop and get the right expected value at the end. We could probably do some more precise reasoning to adjust for this error. But this blog entry is long enough.

Fixing a build failure

2010-02-26T03:14:16+00:00

I haven’t posted in a while, so I figured I’d just give a sample of the kinds of things I deal with as an Ubuntu developer.

When a Wine developer wants to implement support for playing mp3 files, the smart thing to do is to use a software library, in this case mpg123. Wine is a good example of a large and still growing piece of software with a similarly expanding list of dependencies.

It’s my job to make sure all these bits work well together so that Wine can be built in a complete way for you – without mpg123, Wine would have to be built without full support for mp3s, and sound on some apps would simply break. Sometimes this means packaging software that isn’t in the distribution yet. Usually it means fixing bugs, and here we can see the true strength of the open source approach.

It starts with a bug report

I was ready to put out a Wine release with its new mpg123 support, but Wine wasn’t building on 64 bit. This was a case where the mystery was more interesting because it was only partially broken: 32 bit worked perfectly.

I knew right away that it was probably an Ubuntu-specific issue, although other distributions (especially Debian) might also be affected. I filed a bug upstream to try and nail down the specifics. Everyone shares their relevant expertise here: a Wine developer notes that it’s a mismatched symbol-header problem, I figure out the cause is Ubuntu’s unexpected use of 32-bit headers on 64-bit systems, a community member provides a temporary workaround patch, an mpg123 developer tells why things were the way they were and how Ubuntu+Wine was the first to expose this problem, and Alexandre Julliard says what he’d ultimately like as a best practice.

This sort of collaboration is exactly what Open Source is about. Everyone involved could just go to the same bug tracker and simply get to work – it didn’t matter that all 6 of us were in different continents working for different companies on different projects. Everything is laid bare, easy to see and thus easy to fix.

Getting it done

At the end of the day, getting the fix out to users is what actually matters. It’s really disappointing when great code like mp3 support just doesn’t happens because of something silly like an out of date distribution library.

So, that’s the final step in the chain – the mpg123 developer puts out a new version of mpg123, and then I have to package that up and put it into the latest Ubuntu alpha. This is where Launchpad’s ability to have a single bug exist against multiple packages becomes especially helpful: by filing a launchpad bug I could track the changes I had to make to all 3 packages (mpg123, ia32-libs, and then wine).

Technical details for the interested

It’s worth noting that this bug could have been prevented entirely if Ubuntu had proper multiarch support. Multiarch means the ability to install 32-bit packages directly onto a 64-bit system, rather than having to use special separate 32-on-64 packages like ia32-libs. Multiarch has been a slow, gradual redesign under development in Debian and Ubuntu for years now, and it just barely missed the Lucid cycle.

This leaves us with the current implementation, which can at best be described as “a tremendous hack.” 32-bit libraries from needed packages are manually added to a giant list. Then they’re processed by a script and all thrown into one gigantic, 600+ megabyte package named ia32-libs. This means that any program which needs 32-bit compatibility (such as Wine) has to depend on this massive source package, and any time a single component changes it needs to be manually updated.

However, ia32-libs contains only the library files, not the header files, so when you build a 32-bit application on 64-bit Ubuntu you’re actually building with the header files installed from the 64-bit version of the package. This is normally fine, as most header files are identical across different architectures, however this wasn’t the case with mpg123. So we were using a 32-bit library that had different symbols than the 64-bit header file, and the build then fails with symbol errors.

There are quite a few applications for 64-bit Ubuntu that need to make use of 32-bit libraries for various reasons, however Wine is by far the most dominant. This makes me, the one responsible for Wine, also the official babysitter of ia32-libs and similarly broken packages. Wine is special in a way, though, as unlike other applications there is no way to fix it by just making it 64-bit — Wine will always need 32 bit libraries in order to run 32 bit Windows applications, which will remain incredibly common for at least another decade.

Hide your filesizes, people.

2010-02-24T13:00:03+00:00

I read this BoingBoing post: Tahoe-LAFS: a P2P filesystem that lets you use the cloud without trusting it.

Can we trust it? Can we really trust it? Reading the architecture document for Tahoe-LAFS, I found this text:

In general, anyone who already knows the contents of a file will be in a strong position to determine who else is uploading or downloading it.

…

Also note that the file size and (when convergence is being used) a keyed hash of the plaintext are not protected.

Listen up, folks. A file storage service that reveals file sizes is one that you cannot blindly trust.

Consider the following scenario: You find yourself with a popular pirated movie file, whose size is 2143658709 bytes, and you decide to “securely” share it. The people spying on you can see that you’ve uploaded an encrypted file of this size to the cloud. What are the odds that you happened to coincidentally come across a file that’s exactly the same size? Slim. Sufficiently slim for a subpoena, and bam, you could get busted by the copyright cops.

It gets worse. Suppose you’ve extracted a zip file of some illegal content. Even porn is illegal in some countries. The set of file sizes in that collection is, say, some set of values, and we don’t really care what they are. Call it {x₁, …, x_n}. The evil government watches you one day, and sees that you’ve uploaded a bunch of files, and, in its scan for illegal porn collections, notes that {x₁, …, x_n} is a subset of the files you’ve uploaded. They’ve caught you. With an astronomically small chance of a false positive.

The fact that you can’t hide those file sizes in the noise of other files is somewhat counter-intuitive. For example, suppose you have an illegal file collection with 100 text files. All their file sizes are between one and a million bytes in length. You upload that collection amid 99900 other files with randomly chosen lengths between one and a million. By the way, that’s an absurdly high amount of fluff files to be uploading. So now you’ve uploaded a hundred thousand files. Your small subset is safely obfuscated within all those other files, right? Wrong.

What’s the chance that a given set of 100000 numbers, chosen from one to a million, contains a certain subset of 100 numbers? It’s approximately 1/10¹⁰⁰. That’s right — one over a googol. Each number between one and a million has a 10% chance of being included in the set of 100000 random file sizes. Actually, it’s worse than that, since some file sizes will be repeated. I think the expected number of distinct values is about 95163. That makes it more like 1/10.5¹⁰⁰.

The lesson here is that, if your online upload service knows your file sizes, it cannot be blindly trusted. They’re only useful for keeping secret files that you’ve generated. They’re not useful for secretly storing files whose existence is known to the authorities. When you go about designing the ultimate encrypted peer-to-peer file cloud, please store data opaquely in fixed size blocks, that never get truncated. You can figure out how to do this efficiently. You can have a layer of indirection between uploaded chunks and individual files. And don’t even think about allowing block sizes to be different discrete sizes, such as power of two. You know you’ll mess it up. Be sufficiently paranoid.

Now let’s be even more paranoid. Upload files as rarely as possible. Don’t upload them eagerly. For example, if somebody’s slowly placing a set of illegal movies into a shared directory, eager scanning of a directory and eager uploading of individual files would reveal enough information about individual file sizes to provide an identifiable signature for the entire set. Somebody watching network activity live would notice pauses after each group of chunks that corresponds to a file. This would tell them information about the length of the file, rounded to the nearest whatever-your-chunk-size-is. You’d need to add absurd amounts of random padding to individual files to sufficiently obscure this information. Upload files in groups. It’s safer and less expensive that way. And while you’re at it, don’t just read-a-file-from-disk-and-upload-it, read-a-file-from-disk-and-upload-it, and so on. Are you sure the pauses between files are too small to observe on the network? What if the shared directory you’re uploading is really on a network file system? Are you sure? Read files concurrently and upload them interleavedly. Be sufficiently paranoid.

Follow-ups:

Estimating the number of distinct numbers — on how I got 95163.

Apple’s Review Process Strikes Again

2010-02-23T23:30:38+00:00

For the first time, I have my own idiotic app review rejection story. I have to say, until now I've been one of the lucky ones. My developer app was approved in days, and every app and update I've put through has been approved with no hassles (save for the ...

Want to do ___? Tough.

2010-02-23T23:30:38+00:00

One of the reasons entrenched Windows users find switching to the Mac unappealing is that the interface is so different. Different can be better. But in my opinion, after 6 months developing on the Mac, different is definitely NOT better. I consider myself fairly used to the Mac interface and ...

The most important thing (to me) about the FCC’s investigation of Apple and AT&T

2010-02-23T23:30:38+00:00

You probably heard that the FCC has taken an interest in Apple's rejection of the Google Voice app and requested information from the three companies. Anyone who has read Stromcode over the past 6 or so months knows that this news makes me uncommonly happy, but I haven't posted about ...

Remove Protected Foreign Fonts in Windows 7

2010-02-23T23:30:10+00:00

Windows 7 installs a pile of foreign language fonts by default that clutter up the font selection menu (Aparajita, Batang, etc). There is a "hide" mechanism built into the font manager, but of course, it doesn't actually work (in my case, in Adobe CS4 specifically). If you try to delete one of these fonts manually, you'll get a dialog stating that they are a protected system font and cannot be deleted. Of course unless you're using the languages in question, this isn't true, and it's extremely irritating.

Here's how to remove them. Click on the Start Orb, type in cmd. Right click on the cmd entry in the Menu and choose Run as Administrator

In the cmd window, type:

takeown /R /F C:\Windows\Fonts

This will give you ownership of the Fonts folder (/R flag for recursive). Follow this with:

icacls C:\Windows\Fonts /grant administrators:F /t

This will give the administrators user group full control of the Fonts folder (/T for recursive). Finally, type:

attrib -r -s C:\Windows\Fonts

This will remove the read-only and system attributes of the folder, which causes it to display as a normal folder with font files. You can now go in and delete the fonts you wish. Restart afterwards, as they seem to stick around until you do. Don't delete fonts if you don't know what they do, and don't complain to me if you mess up your system. Good luck!

Django Syntax Highlighting with reStructuredText and Pygments

2010-02-23T23:30:10+00:00

Despite launching my blog only last week using Markdown as my blogging markup language, after reading Eric Holscher's Large Problems in Django, Mostly Solved: Documentation, I was curious if it would be much, if any, work to switch to reStructuredText and still retain the code highlighting support I was getting from Markdown's CodeHilite extension. The actual highlighting comes from Pygments, which is stand alone and should be possible to hook into reStructuredText as well.

Why bother switching? reStructuredText is pretty well loved among the Python community for documentation, and rightfully so. Although ReST is much more extensive than I need for blogging, I see no need to be using two markup languages. Plus, it saves me a module, since I don't have to install markdown.

First, make sure you have Docutils and Pygments installed. I tend to install Python modules with pip:

$ pip install docutils pygments

django.contrib.markup comes with a template filter, |restructuredtext that converts ReST into HTML, and if that works for your purposes, go ahead and use it. I, however, following advice from James Bennett's Practical Django Projects choose to store the rendered content in my database. This way it doesn't have to be parsed every time it is displayed. I peeked into the restructuredtext filter to see how it worked:

def restructuredtext(value):
    try:
        from docutils.core import publish_parts
    except ImportError:
        if settings.DEBUG:
            raise template.TemplateSyntaxError("Error in {% restructuredtext %} filter: The Python docutils library isn't installed.")
        return force_unicode(value)
    else:
        docutils_settings = getattr(settings, "RESTRUCTUREDTEXT_FILTER_SETTINGS", {})
        parts = publish_parts(source=smart_str(value), writer_name="html4css1", settings_overrides=docutils_settings)
        return mark_safe(force_unicode(parts["fragment"]))
restructuredtext.is_safe = True

So, basically it reads in your RESTRUCTUREDTEXT_FILTER_SETTINGS and renders using docutils.core.publish_parts. Personally, it's a pet peeve of mine when I find a template tag that has built in logic that would be useful outside the context of a template tag, but in this case, the template tag function will work fine if we import it and call it from the model:

from django.db import models
from django.contrib.markup.templatetags.markup import restructuredtext

class Entry(models.Model):
    title = models.CharField(max_length=150)
    slug = models.SlugField(unique=True)
    content = models.TextField(editable=False, blank=True)
    content_raw = models.TextField()

    def save(self, *args, **kwargs):
        if self.content_raw:
            self.content = restructuredtext(self.content_raw)
        super(Entry, self).save(*args, **kwargs)

On save, I render whatever was typed into the content_raw to html, then store it in content. content is hidden in the admin (editable=False) and content|safe is used in the templates.

ReST, as documented, now works in my Entry model. However, it knows nothing about code syntax highlighting. The Pygments website has a brief entry on ReST support that merely left me confused. Here's the gist of it: you need to create a "directive" (aka markup tag) to display source code and pygmentize it. Pygments graciously provides the necessary code in the external folder of the download, or you can view it online. I have included it below, taking the liberty to uncomment the "linenos" variant option:

# -*- coding: utf-8 -*-
"""
    The Pygments reStructuredText directive
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    This fragment is a Docutils_ 0.5 directive that renders source code
    (to HTML only, currently) via Pygments.

    To use it, adjust the options below and copy the code into a module
    that you import on initialization.  The code then automatically
    registers a ``sourcecode`` directive that you can use instead of
    normal code blocks like this::

        .. sourcecode:: python

            My code goes here.

    If you want to have different code styles, e.g. one with line numbers
    and one without, add formatters with their names in the VARIANTS dict
    below.  You can invoke them instead of the DEFAULT one by using a
    directive option::

        .. sourcecode:: python
            :linenos:

            My code goes here.

    Look at the `directive documentation`_ to get all the gory details.

    .. _Docutils: http://docutils.sf.net/
    .. _directive documentation:
       http://docutils.sourceforge.net/docs/howto/rst-directives.html

    :copyright: Copyright 2006-2010 by the Pygments team, see AUTHORS.
    :license: BSD, see LICENSE for details.
"""

# Options
# ~~~~~~~

# Set to True if you want inline CSS styles instead of classes
INLINESTYLES = False

from pygments.formatters import HtmlFormatter

# The default formatter
DEFAULT = HtmlFormatter(noclasses=INLINESTYLES)

# Add name -> formatter pairs for every variant you want to use
VARIANTS = {
    'linenos': HtmlFormatter(noclasses=INLINESTYLES, linenos=True),
}


from docutils import nodes
from docutils.parsers.rst import directives, Directive

from pygments import highlight
from pygments.lexers import get_lexer_by_name, TextLexer

class Pygments(Directive):
    """ Source code syntax hightlighting.
    """
    required_arguments = 1
    optional_arguments = 0
    final_argument_whitespace = True
    option_spec = dict([(key, directives.flag) for key in VARIANTS])
    has_content = True

    def run(self):
        self.assert_has_content()
        try:
            lexer = get_lexer_by_name(self.arguments[0])
        except ValueError:
            # no lexer found - use the text one instead of an exception
            lexer = TextLexer()
        # take an arbitrary option if more than one is given
        formatter = self.options and VARIANTS[self.options.keys()[0]] or DEFAULT
        parsed = highlight(u'\n'.join(self.content), lexer, formatter)
        return [nodes.raw('', parsed, format='html')]

directives.register_directive('sourcecode', Pygments)

But what the heck do you do with this? You need to import it somewhere, anywhere really, where it will get loaded. I saved it as rstdirective.py and played it safe by importing it at the top of my settings.py file. Other people import it in the __init__.py of the app using it.

#settings.py
import rstdirective.py

Now you can outline code by using:

.. sourcecode:: python

    import foo #source code!

Or, with line numbers:

.. sourcecode:: python
    :linenos:

    import foo #source code!

The code should be rendered, marked up, and ready for color. Pygments doesn't come with any CSS files, although it does have a tool to create them automatically:

$ pygmentize -f html -S monokai -a .highlight > media/css/pygments.css

Basically: create me CSS under the class .highlight using the theme monokai. You can preview the default Pygments themes here. If you're lazy/confused, richleland has a github repo with the CSS files already generated, you just need to change the class from .codehilite to .highlight. Include the CSS in your template link you would any other CSS file, refresh, and enjoy!

Customizing Django Comments

2010-02-23T23:30:10+00:00

The django.contrib.comments app, like all good apps, comes with many default templates so you can get them up and running in minutes. Unfortunately, by default they will not match your website, since they couldn't possibly know about your site's template structure. Fortunately, because of the way template loading works in Django, it's easy to extend these templates to match.

You may have noticed these lines in your settings.py:

TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.load_template_source',
    'django.template.loaders.app_directories.load_template_source',
)

When Django looks up a template, it will check these modules in order.

django.template.loaders.filesystem.load_template_source will check the directories you specified in the TEMPLATE_DIRS setting. If it doesn't find the associated template, it will move onto django.template.loaders.app_directories.load_template_source, which will look in the templates directories of apps you've listed in INSTALLED_APPS.

Barbara Shaurette recommended overriding any of the templates you wanted to customize by changing each to inherit from your base template, but there's a quicker way that lets you play dumb about how many templates there are (and any updates made to them).

Copy only the base.html template ([DJANGO_DIRECTORY]/contrib/comments/templates/comments/base.html) to your template directory and edit it so that instead of having the page markup included, like this:

templates/comments/base.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>{% block title %}{% endblock %}</title>
</head>
<body>
    {% block content %}{% endblock %}
</body>
</html>

... have it inherit from your base template (or whatever template you want):

{% extends 'base.html' %}
{% block body %}
    {% block content %}{% endblock %}
{% endblock %}

Using this method, by overriding just the top level template, you've customized all the comment templates, without even having to know what they are.

This method will of course work with any application. I just ran in to it specifically using the comments app.

A Fun Math Problem

2010-02-22T19:56:51+00:00

What is the infimum of n²|sin(n)|, over positive integers n?

Is it zero, or is it greater than zero?

Ok, it’s time to spoil the answer.

With some lack of rigor, I think the probability is 1 that the minimum value exists and is greater than zero.

Suppose two people play a number-picking game, where two players pick numbers, and then compare them against one another, infinitely many times. Suppose the first person picks a (real) number uniformly on [0,1] and the second person picks 1/2^k for his kth choice. What’s the expected number of times the first person will pick a number less than the second person does, for the rest of time? The answer is 1 — there’s a 1/2 chance of that happening for k=1, a 1/4 chance for k=2, and so on, so the expected number of times is 1/2 + 1/4 + 1/8 + …, which equals 1.

If we let r(1), r(2), r(3), … be the sequence of random numbers the first person picks, we would therefore expect there to be one value of k for which r(k)/(1/2^k) is less than 1. Maybe there are two, or three such values of k. The probability that there are infinitely many such values is zero.

Now consider |sin(n)|. For small values of ε, and random values of x the probability that |sin(x)| < ε is 2ε/π. That’s because the slope of |sin(_)| is 1 or -1 when it touches zero, and it touches zero at multiples of π. Now let’s ask ourselves: How many times is n²|sin(n)| less than 1? Well, that’s the same as playing a number guessing game where player one chooses |sin(n)| and player two chooses 1/n². And |sin(n)| is effectively a random number function. What’s the probability of player one choosing a value smaller than player two? Approximately 2(1/n²)/π. Or (2/π)/n². If we sum these probabilities, we get (2/π)(1/1² + 1/2² + 1/3² + …) which equals (2/π)(π²/6) which equals π/3. So the expected number of times n²|sin(n)| is less than 1 is about 1.04. This is not accurate — for example, we assumed small values of ε when computing 2ε/π, and early on in the sequence, we have large values of ε, but we would still expect the expected number of times n²|sin(n)| is less than 1 to be finite.

What does this mean? It means that, assuming the fractional parts of multiples of π are random-looking, n²|sin(n)| is less than 1 only for finitely many positive integers n. Which means that of this finite number, there is a minimum value. One that is of course greater than zero. So the infimum is not zero.

I’d say there’s a good chance that sin(1) is the only value in the sequence n²|sin(n)| that is less than 1. If we check the first ten million values in the sequence, we find that sin(1) is the only one so far.

The iPad and Tiling Window Managers

2010-02-19T13:00:17+00:00

Here’s a prediction for you. The iPad will eventually be able to run multiple applications at the same time. Also, it will run them in a tiling window manager.

Does anything else need to be said? Is this not obvious? A tiling window manager would be the ideal way for apps on the iPad to be run concurrently. Turn the iPad sideways, open two apps, have the iPad tell them they’re oriented vertically, and you’re done! This would be the conservative way to do things, that doesn’t let users muck things up by having overlapping windows. So that’s the way I’d expect Apple to go with this.

Here’s a less hopeful prediction: The iPad will eventually let you run two apps side by side, but not in any other configuration.

But some day, the iPad or some other kind of tablet will have a tiling window manager. You just watch.

A Few Flaws in WordPress

2010-02-17T05:36:10+00:00

Now I’d like to make fun of WordPress. WordPress, you see, is not only a great convenience—it’s also a fairly dopey piece of software. First, the default template includes a proclamation that it produces “Valid XHTML” with a link to the W3C validator. As I’ve obnoxiated in a previous posting, it includes some nonstandard attributes (named “role”) for Javascript purposes, and so it fails validation. Well, that’s kind of funny, but it’s not really a big deal.

Another problem is that Chrome keeps crashing when I try to read or administrate this blog. And so I have to use Firefox. I suppose that’s not WordPress’s fault, but it’s peculiar.

Now here’s a serious complaint. You can see the titles of not-yet-published posts. Generally speaking, you can visit posts by id and you’ll get redirected to the permalink page. For example, if you visit http://shrughes.com/?p=17, you’ll get redirected to http://shrughes.com/p/waking-up-in-san-diego/, which contains a published post about waking up in San Diego.

Let’s look at what happens when we guess the post id for a post due to be released in the future. If you visit http://shrughes.com/?p=27, you’ll be redirected to http://shrughes.com/p/an-example-unreleased-post/, which gives a friendly message telling you that this page does not exist.

Right.

I don’t really have a problem with my post titles being released without expecting it, partly because now I know about the problem. I suspect that other bloggers, who have more serious reasons to delay publishing until certain times, might want to keep that information secret. This seem like it would affect anybody who includes post titles in their permalink URLs, and who schedules posts for future publication.

Default Programs Editor 2.6 Update

2010-02-14T20:14:42+00:00

I discovered a couple stupid bugs right after I posted 2.5, so here’s a quick 2.6 update to resolve them.

Version 2.6.1837.1459 (February 14, 2010)

Bugs Fixed:

- Fixed a discrepancy between value data types when saving to a .reg file vs saving to the registry when changing a context menu item name.

- DDE Message was not deleted if the program was running with elevated privileges.

- File type icon would sometimes not be detected correctly when the icon source was the default program.

Crashes Fixed:

- Deleting a context menu item would sometimes result in a crash.

- Inputting invalid character for context menu item name sometimes caused a crash.

- In some cases, opening the autoplay handlers page for some media types caused a crash.

- Saving context menu changes after editing a command label (including implicit edits) caused a crash.

Download now

Default Programs Editor 2.5 Update

2010-02-13T03:38:42+00:00

Default Programs Editor 2.5 was posted this week. The changelog is monstrous; this version has the most bug and crash fixes to date. If you’ve got an idea or gripe, head over to DPE’s UserVoice forum and talk about it!

Version 2.5.1823.2023 (February 9, 2010)

New Features:

+ Added support for DropTarget objects as context menu commands.

+ Added support for DelegateExecute objects as context menu commands.

+ Added support for Dynamic Data Exchange messages invoked with context menu commands.

+ On the Icon page, the raw resource path is now viewable and can be manually changed. It is also now possible to specify a file’s own image in one click.

+ When changing an extension’s file type, a revamped file type selection process shows all file types, even unused ones.

+ Added a checkbox to select/deselect all extensions on the extension selection page for Default Programs associations.

+ Added file path display to the final success page when changes are saved to file.

Bugs Fixed:

- Fixed a bug where the exported command path wouldn’t be a normal string like it should have been.

- Fixed a bug where the wrong context menu item would be detected as the default.

- Fixed blank autoplay handler showing on the media options list after cancelling the creation of a new handler.

- Fixed incorrect default context menu item detection in some cases.

- Fixed a bug where an exported context menu .reg file could have a malformed key path.

- Saving edited file type Descriptions and Icons to file now correctly saves only those changed properties.

- The .reg file comment for Description and Icon changes will now also correctly describe which property is edited.

- The main instruction text on the Delete Extension page now reads appropriately.

- The link button to set a context menu item as default now enables or disables uniformly with the other controls.

- Some resource strings were not resolved properly (such as certain file type descriptions).

- Missing publisher information added to Control Panel uninstall data.

- Burnable Blu-ray discs now show correctly in the Autoplay media list.

- Fixed wrong capitalization for Blu-ray disc entry in the Autoplay media list.

- The behavior of the Done button on the Success page was not uniform for all page paths when launched from the Control Panel.

- The page history is cleared on the Success page to prevent old pages appearing as current.

- Fixed some extensions not being correctly detected as being registered for a Default Program.

- Fixed some extensions with blank descriptions on the Default Programs registration page.

- Additional small internal improvements, optimizations, and interface adjustments.

Crashes Fixed:

- Fixed a crash when saving a new file type without an icon set.

- Saving Autoplay settings after cancelling the creation of a new handler would cause a crash.

- Clicking the Help button would sometimes result in a crash on some systems.

- Attempting to install while another instance of the program was running caused a crash.

- Selecting a new default verb sometimes caused a crash when saving changes to file.

- Viewing certain file type descriptions sometimes caused a crash on the Extension Selection page.

- In certain cases, the program crashed when attempting to add or edit a context menu item where the destination registry key didn’t yet exist.

- Adding a new file type but not entering optional details caused a crash when saving changes to file.

- Fixed a crash after clicking the Next button on the Default Programs selection page.

Holy cow! With the addition of DelegateExecute, DropTarget, and DDE support, Default Programs Editor is now the most powerful and easy to use file association utility on the planet.

Download now at http://defaultprogramseditor.com
Have an idea or feature request? Put it on the feedback forum: http://defaultprogramseditor.uservoice.com

Say Hello – Nasuni Launches Today!

2010-02-09T13:35:03+00:00

The company I’ve worked for since July of last year – Nasuni Corporation (a startup in Massachusetts) has gone live! This is the culmination of a lot of hard, but exceedingly fun and exciting work over the past months.

The Nasuni team is an excellent one – and one I am very, very proud to be a part of. Our product is called the Nasuni Filer – a simple-to-use, versioned, encrypted and cloud-storage backed virtual NAS (network attached storage) server (click here for more information).

Without going into all of the features, our goal in making this was to make cloud storage simple, accessible and secure – and I know we’ve accomplished all three. All you do is download it, boot it and start using it – once you do so you have access to truly unlimited storage. It’s an unlimited filesystem for the cloud. Here’s the elevator pitch:

Nasuni has developed a virtual file server, called the Nasuni Filer, that delivers unlimited file storage and complete file protection for businesses. Working in partnership with leading cloud storage vendors, the Nasuni Filer leverages the vast capacity of the cloud to store and protect company files offsite, while retaining the local functionality and performance of a traditional NAS.

This technology allows businesses to use the cloud provider of their choice as a replacement for traditional primary storage. Snapshots, file versioning, and offsite storage are integrated into the file server itself – ensuring business file are safe and secure at all times. No need to manage complex backup and DR schemes – if the file server is running, files are protected.

We’ve launched the Beta of the product today – anyone can sign up, download and use it. Anyone can give us feedback and suggestions – I encourage all of you who might need something like this to download and give it a try. If you want – go check out the videos we’ve put together showcasing the Filer (and better yet – check out the awesome animated cartoon we have on the front page).

Most of you know that my blog is mainly Python oriented. Suffice it to say, Nasuni – and the Nasuni Filer make use of Python for a wide range of tasks. We use Python, Django and as much of the Python ecosystem as we can to drive everything from the website, to the GUI on the appliance itself – Python is part of the DNA of the company, and it has served us well. Without Open Source and Python – I don’t think it would have been possible to build what we have built in as little time as we have.

We have a strong dedication to not just Python, but open source in general (and a fair number of us will be at PyCon this month). As time progresses, now that we’re exiting stealth mode we plan on possibly open sourcing stuff we feel would benefit the community. Some of us already push patches back where and when we can, but as I said – as time progresses this involvement will only increase.

So not only am I proud to announce the product, be part of this team and to see what we’ve made, I’m also happy to thank so many people in the Python and OSS community which have helped us reach this point.

So go – check it out, let us know what you think.

Threading woo!

2010-01-30T00:01:18+00:00

Fixed an annoying UI bug where UI updates done outside the main thread fail to update until an event propagates into the main thread. Now a user can "login" to the game.

Woo ski trip!

2010-01-27T22:25:03+00:00

I finally got back into development after a long bunch of trips and wedding planning.

Now my app has the beginnings of a UI. All I have to write is 50 other things and it will be working!

Alpha 2 ISO Testing begins

2010-01-13T21:40:28+00:00

So here’s the deal: you want to help contribute to Ubuntu in some small way but don’t want to make much effort or risk your “real” computer. Enter ISO Testing.

Basically, you burn a disk and attempt to install. If it works, you click a box, and if it doesn’t, you click another and file a bug. If the installer finished but turned everything pink and that’s not really your thing, don’t check the box for “serious”.

This is one of the best ways of helping Ubuntu that doesn’t actually involve using it. So if you have a spare computer, partition, or even VM you can basically spend about 30 minutes after downloading and submit a good test report.

To get started:

Go here and make an account and then click about 4 buttons. There’s even a “I’ve started a test so others can try something else” feature.
A fuller explanation is at the wiki page

Benefits:

Once enough people give successful ISO test reports, the Alpha is released and the coolest of us can start general testing. That’s where you actually use the installed system and test all the experimental new features like being slightly less brown. But before that can happen, testers need to know they can install and update without hassle – the daily CDs are frequently unbootable for one reason or another, so that means the alpha CDs need to be good enough to install from.

Also, here is a picture of my cat:

Unladen Swallow: Python 3’s Best Feature.

2010-01-07T02:58:43+00:00

We all know (well – unless you’ve under a rock) about Unladen-Swallow, the semi-Google-Sponsored optimization-focused branch of Python 2.x. Collin, Jeffrey and many others have been working tirelessly on porting the CPython interpreter over to LLVM, applying optimization patches, writing tests, etc all aimed at speeding up real-world operations and code since before last year’s PyCon – at that time, the first release was already working hard, rendering YouTube’s templates. (See the Project Plan for more information)

You probably also saw the 2009Q3 testing/performance results (here), showing a hefty speed increase for various operation. The 2009Q4 will be coming soon.

What you probably don’t know – yet, is that a PEP is coming, proposing Unladen Swallow be merged back to Python-core – specifically, the Python 3k branch. Yeah, that’s right – a PEP to merge Unladen Swallow back into the mothership, but to Py3k only. Talk about a shot in the arm!

Quoting Collin Winter:

Python 3 is the natural place for something like Unladen Swallow to land: Python 3 is clearly the future, and I think improved performance will be a strong selling point in the language’s favor. What we and others have done for Python 2 applies directly to Python 3.

…

One of the things we’ve really focused on with Unladen Swallow is creating a solid foundation that the wider CPython developer community can build on top of. We’ve fixed some nasty problems in the x86-64 JIT, for example, so that the CPython community can focus on the fun stuff — more optimizations! Having the basic JIT infrastructure in place and well-tested opens up a world of opportunities that didn’t exist before.

…

I think PyPy’s focus on research is incredibly valuable. Unladen Swallow has always been about, “what can we achieve right now?” One of our guiding principles has been, Do nothing original, don’t be innovative. PyPy is looking 10 years down the road and thinking, What can we do if we question everything? I think having those different perspectives is important to the community. So, no, I don’t see us in competition (with PyPy) at all.

There’s been a lot of talk about how slow adoption of 3k has been – a backwards incompatible implementation of a widely used language will always have slow(er) adoption then releases of the same language, backwards compatible – but I’d argue that to an extent, part of the reason is that there “haven’t been enough carrots” in the Py3k cart to provide enough of an incentive for projects, libraries, etc to move to it at an accelerated rate.

This, well – this changes things. A significantly faster interpreter, a more rational GIL thanks to Antoine Pitrou – these are the improvements to Python 3 which make it the perfect target to aim for for new projects and libraries. It should drastically help provide the impetus for larger libraries to move over. Things like these are exactly why something like the moratorium make sense, efforts can be focused on the ecosystem of python outside of the syntax, on the interpreter, the standard library, etc.

This is huge; and with a little luck, a lot of discussion and debate – and a ton of work, by the end of 2010, we’ll have not only the best implementation of Python, as a language to date in Python 3 – but the best interpreter as well.

The PEP should go out for review soon, probably within the next two weeks – and more details/information will be coming out at PyCon in February (you’re coming, right?!).

Also see Michael Foord’s post about this news too.

The future of Wine sound

2009-12-24T14:38:23+00:00

Sound in Wine has been a big issue. From a user’s perspective, it didn’t work well. From a technical user’s perspective, there were 3 different drivers to choose from and none of them worked well. From a developer’s perspective, no sound driver would ever work well.

Wine was a victim of the proliferation of sound drivers on Linux.

At one point, Wine had separate sound drivers for ARTS, ESD, OSS, JACK, and ALSA. None of them worked right, even if the user could figure out how to configure Wine to use the “correct” one. Subtle changes to each API and differences between distribution versions made matters even worse.

PulseAudio was supposed to be a solution to problems like this by being the one true Linux Sound System, but in some ways it made the problem worse. Now Wine needed a separate PulseAudio sound driver, and no one wanted to write sound driver code since the whole thing was a big mess to begin with. Worse, the most work had gone into making the ALSA driver better, and there were a few technical reasons why it seemed like a good choice to stick with ALSA and use it as a default.*

So, the decision was made to go with ALSA and improve it a bit, and hope that PulseAudio’s ALSA-compatibility layer would work well enough. That didn’t exactly happen – PulseAudio didn’t actually want to support some “abuses” of the ALSA API, as one dev put it. Some users took to killing PulseAudio outright every time they wanted to use Wine.

Things got a bit political. Was Wine at fault here, or PulseAudio? Should Wine include a PulseAudio driver, even if it was poorly maintained? What would the default be? Also, what the hell were we gonna do about the Mac?

The solution, which in retrospect is pretty obvious

Wine wasn’t the only program facing this problem. Anyone writing or porting a Linux game had to answer the question about how they would play sound – almost none chose to write 7 separate audio drivers. Instead they all did the smart thing, and used a higher level library.

So, that’s the plan – Wine will use a sound library. And then we’ll let them worry about actually talking to whatever sound system happens to be working best. OpenAL is the smart choice here, for much the same reason that OpenGL was the smart choice for graphics. We get Mac compatibility “for free” too.

Now, why didn’t this happen earlier? Well, in fairness to Wine, OpenAL didn’t actually exist in a usable form until Wine already had sound drivers for a few years. A similar thing happened with wikis and distributed version control systems – we eventually saw the light that new open source projects had been following for years, and the project is better because of it.

* As I understand it, a lot of Windows programs expected to be very close to the hardware and at the time many users were having issues with PulseAudio latency.

I want your awesome python snippets.

2009-12-19T14:52:26+00:00

I’m building (and will eventually post someplace) a collection of the cooler Python snippets I dredge up. I’m looking for snippets which are:

Short
New-to-Python accessible
Showcase the best ideas of python – clean, simple, powerful.

The goal is to build up a small pile of code snippets that programming newbies, or programmers from other languages can look at and go “wow, I got to get me some of that”.

Anything is game; feel free to post them in the comments, on pastebin (obviously post the link), etc.