sn.printf.net

Updating the iPhone

2008-11-21T13:47:03+00:00

(pop) I'm updating my (pop) iPhone to firmware version 2.2 for the "improved Safari" features and some of the se(pop)curity fixes and iTunes decides that disabling my fucking phone for an hour isn't enough, it has to annoy (pop) the shit out of me by putting itself on top of all my other windows every few seconds. Thanks a fucking lot, it's not like I wanted to get (pop) any WORK done on MY computer (which is now temporarily Apple's).

Who the fuck coded this? S(pop)omewhere deep in the code, there must be a delay loop that says "hey, it's been 10 seconds, time to be (pop) FUCKING ANNOYING now."

Each of these (pop)s is seriously (pop) another time the window popped up just while I was typing this.

Fucking Apple.

Sequel is Gorgeous

2008-11-19T18:44:45+00:00

My original DataMapper classes:

  class Post
    include DataMapper::Resource
    property :id,       Serial
    property :title,    String
    property :body,     Text
    property :created,  DateTime
    property :author,   String
 
    validates_present :body, :title
 
    has n, :categories, :through => Resource
 
    def self.latest_by_author(name)
      all(:author => name, :order => [:created.desc])
    end
 
    def self.count_by_author(name)
      count(:author => name)
    end
 
    def categories_as_comma_list
      categories.map { |x| x.name }.join ', '
    end
 
    def categories_as_linked_comma_list
      categories.map { |x| "<a href='/archive/#{x.name}'>#{x.name}</a>"}.join ', '
    end
 
  end
 
  class Category
    include DataMapper::Resource
    property :id,       Serial
    property :name,     String
    validates_present :name
 
    has n, :posts, :through => Resource
  end

My new Sequel classes:

  class Post < Sequel::Model
    many_to_many :categories
 
    def self.latest_by_author(name)
      all(:author => name, :order => [:created.desc])
    end
 
    def self.count_by_author(name)
      filter(:author => name).count
    end
 
    def categories_as_comma_list
      categories.map { |x| x.name }.join ', '
    end
 
    def categories_as_linked_comma_list
      categories.map { |x| "<a href='/archive/#{x.name}'>#{x.name}</a>"}.join ', '
    end
 
  end
 
  class Category < Sequel::Model
    many_to_many :posts
  end

Ignoring the migrations and validations, next to nothing has changed. All of my queries work as expected. They’re inefficient because I’m not giving Sequel any hints on how to handle them, but that will hopefully change soon as I migrate to using Sequel’s DSL to specify queries, and stop using models altogether. I’ve been mucking around in the ORM debate and the more I think about it, the more I’d like to use a lightweight language to describe my SQL, rather than hand it all off to a more complex ORM, which is exactly what Sequel provides me.

From Common Lisp to Clojure (Part 1)

2008-11-08T07:39:11+00:00

I recently started tinkering around with Clojure as a test to see if I could replace Java with a Lisp-like language. For those unfamiliar with Clojure, here's a quick intro to what it's all about:

Clojure is a dynamic programming language that targets the Java Virtual Machine. It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.

As a regular user of Common Lisp, it has taken some adjustment to get used to Clojure's syntax choices, but I'm starting to believe that they are for the better and will allow those unfamiliar, or afraid of Lisp, to give it a try. A small example is the new syntax for LET; a function that binds values to variables. The code below in Common Lisp would assign 3 to x, and x + 1 to y:

(let* ((x 3)
       (y (+ x 1)))
  ...)

In this case, we have to use LET*, which evaluates things sequentially (LET does the assignment in parallel). In Clojure, however, LET is always LET*, and a lot of the extra parentheses are removed. The above would be written like the following in Clojure:

(let [x 3
      y (+ x 1)]
  ...)

The []'s were originally introduced in Scheme to make heavy paren code (LET, COND, etc.) easier to read, but was never enforced. I never used the [] helpers while writing Scheme, but I can understand why they might increase readability. Aside from the few syntactic differences, I felt more or less right at home using Clojure, with the added ability of being able to easily interface with Java programs.

Dynamic Java?

Despite Clojure's dynamic nature, we can still drop down to enforce typing when performance is an issue. I've started to switch over to Clojure as my primary language for Project Euler problems, and one of the first things I needed to write was an efficient prime number sieve. I more or less copied the algorithm from Roger Corman's Lisp examples, without the type decoration as I was unsure how to do it in Clojure. I ended up with the following as a result:

(defn unoptimized-sieve [n]
  "Returns a list of all primes from 2 to n"
  (let [root (Math/round (Math/floor (Math/sqrt n)))]
    (loop [i 3
           a (make-array Boolean n)
           result (list 2)]
      (if (>= i n)
        (reverse result)
        (recur (+ i 2)
               (if ( i root)
                 (loop [arr a
                        inc (+ i i)
                        j (* i i)]
                   (if (>= j n)
                     arr
                     (recur (do (aset arr j true) arr)
                            inc
                            (+ j inc))))
                 a)
               (if (not (aget a i))
                 (conj result i)
                 result))))))

Notice the use of Java's java.lang.Math.round/floor/sqrt functions and the Boolean type. The first thing I noticed was how slowly this ran compared to my Common Lisp version. Finding the first 100,000 primes took a little longer than I anticipated.

user> (time (last (unoptimized-sieve 100000)))
"Elapsed time: 6160.188 msecs"                                                                                                                                                       
99991

I knew being able to tag types would increase my runtime dramatically, as it had done with the Common Lisp version, so I went ahead and did just that.

(defn sieve [#^Integer n]
  "Returns a list of all primes from 2 to n"
  (let [root (Math/round (Math/floor (Math/sqrt n)))]
    (loop [#^Integer i 3
           a (make-array Boolean n)
           result (list 2)]
      (if (>= i n)
        (reverse result)
        (recur (+ i 2)
               (if ( i root)
                 (loop [arr a
                        #^Integer inc (+ i i)
                        #^Integer j (* i i)]
                   (if (>= j n)
                     arr
                     (recur (do (aset arr j true) arr)
                            inc
                            (+ j inc))))
                 a)
               (if (not (aget a i))
                 (conj result i)
                 result))))))

A few simple compiler hints (shown in bold) dramatically reduced the running time:

user> (time (last (sieve 100000)))
"Elapsed time: 164.058 msecs"                                                                                                                                                        
99991

Quite the speed bump for very little work if I do say so myself. Now I can finally start solving more Project Euler puzzles!

So... What Do I Think?

I obviously haven't played around with Clojure too much, but here are my current pros and cons.

Pros

On the JVM: For the work that I do, I find myself writing a decent amount of Java code. A great deal of my classes provide code for assignments written in Java and I have to decide between rewriting provided code in a language I like, or using Java. Now, I don't have to make that choice anymore.
Data Structures are Immutable: One of the things I love about Lisp is you can do whatever you want. Program functionally, imperatively, object-oriented-ly, ..., the list goes on and on. But the more I used it, the more I realized I just did everything functionally. Clojure takes a big step for Lisp in forcing immutable data structures and basically forcing the programmer to think functionally. The main benefit lies in concurrency, which I plan on exploring in the near future.

Cons

No Tail Call Optimization: You're probably thinking "how can a functional language not have tail call optimization?" which is exactly what I thought when I read it. Unfortunately the JVM does not provide such a facility, luckily Rich Hickey provided a workaround in the form of RECUR. It essentially is just a goto that rebinds the variables to the new values in the most recent LOOP or DEFN call. In the second RECUR in the example above, (do (aset arr j true) arr) is set to arr, inc is set to inc and (+ j inc) is set to j in the most recent LOOP call, and the forms are re-evaluated. It's kinda hacky but it works, and with any luck the new JVM will have TCO and we'll be all set.
Lack of Robust Documentation: Perhaps I'm a bit spoiled with JavaDocs and the CL HyperSpec, but I do find the Clojure documentation a bit lacking. This is entirely understandable as the language is very new, but it is definitely something that would turn off those new to Lisp. Luckily, with a "Programming Clojure" book on the way, things are looking up in that department as well :).

All in all, I think Clojure is very promising and I hope to see the community thrive. It's slowly winning me over, which isn't too tough once I can work in a great environment for a REPL-based language.

EDIT: an even faster version that uses primitives (courtesy of Rich Hickey in the comments):

user> (time (last (sieve 100000)))
"Elapsed time: 17.963 msecs"                                                                                                                       
99991

Gantty: Ruby and Gantt Charts

2008-11-04T03:25:00+00:00

Well, not just any old Gantt charts. Specifically, Gantt charts created with GanttProject, a multi-platform, open source Project Management application. Charts saved in this format are easily parsed into nice neat lil Ruby objects.

You’ll need GitHub as a gem source if you don’t already have it (gem sources -a http://gems.github.com):

$ sudo gem install ardekantur-gantty

And get ready! It’s like this:

  @gantt = Gantty::Open '~/CorporateProject.gan'
 
  @gantt.tasks.first.name         # => "Research and Development"
  @gantt.tasks.first.start.to_s   # => "2008-01-01"
  @gantt.tasks.first.end.to_s     # => "2008-01-15"
  @gantt.tasks.first.duration     # => 15

You can see what tasks are active for a certain day — the entire reason I wrote this:

  @gantt.tasks.size                               # => 30
 
  # See tasks that are active today
  @gantt.current_tasks.size                       # => 5
 
  # Specify which date to see tasks from
  @gantt.current_tasks(Date.new(2008,1,6)).size   # => 8

Finally, mix it up for some intense data aggregation!

  @gantt.current_tasks.select { |t| t.coordinators.size > 0 }.collect \
    { |task| task.coordinators.collect \
    { |dude| "- #{dude.name} should be working on #{task.name}." }}.flatten.sort
 
  # => [
  #      "- Adam Alpha should be working on Requirements Document.", 
  #      "- Bob Beta should be working on Requirements Document.",
  #      "- Bob Beta should be working on Website.",
  #      "- Ed Epsilon should be working on Website."
  #    ]

Hideous and beautiful, all at once.

Fork it at its GitHub repository! I hope to add some more features over the weekend, like seeing task dependencies, percentage of completion, and resource focus percent on tasks.

My Faith is Restored

2008-11-01T22:25:34+00:00

I neglected to add in my previous post that it wasn’t initially the size of the spawned processes that caused me to reformulate my approach to my website, it was the access time. A dozen or so off-hand measurements averaged the time to display the entire front page (a paragraph of text from a SQLite database) at 45 seconds, and that’s being generous.

I’d like to put together the ideas that allowed me to clear up this issue, as well as solve another problem I was struggling with. This is my way of giving back to the community. As fun as I’m sure it would be to ensconce myself in the cultures, practices, procedures, languages, politics, and other detritus of several large, entrenched open source projects just to solve a single problem… oh wait.

My problem surrounded three central elements. The website in question is a website for my undergraduate design seminar in Computer Science. As the designated administrative lead, my job is to keep everybody connected, organized, and on top of things. To this end, our group agreed on the use of Subversion, Trac, and a central website for any blog posts we feel would contribute to the discussion of our problem domain.

How each person approaches a problem is nuanced, and — if it can be explained and backed up with evidence — does not have to rigidly adhere to one scheme or another. Whereas in the creation of my blog I was satisfied with subdomains, I wanted this project to use subdirectories off of a root domain: less typing to get to the beginning of a URL than the end, less futzing around with Apache, and so on.

Thus:

http://example.com/ would be the blog,
http://example.com/trac would be the Trac landing page, and
http://example.com/svn would be the Subversion repository end-point.

Already, the amount of code involved has exploded (Apache, mod_wsgi, mod_rewrite, and mod_dav_svn, Subversion, Trac, Sinatra, Rack, Phusion Passenger) and no matter how I looked at it, the answer was going to be ugly. When everything was finally set up, the three applications were fighting for breathing room. I could run Trac and SVN together, and my application worked perfectly through Sinatra itself and on my dev machine, but putting them all up on the same site caused the Sinatra app to slow to a crawl on requests. It was when I finally disabled everything else but Passenger did I see the same problem occurring.

Some people call it the ‘Slow First Request’ problem, and recommend several pooling and request options in Passenger’s configuration to mitigate the problem. I didn’t see that as being the same problem, really: There’s no reason why this application couldn’t feel “always on”, ready for team members and potential meetings and presentations.

I began disassembling my solution to narrow down the problem, and in doing so, considered replacements. I initially considered Apache my staple, and therefore beyond reproach, so I began at Passenger. I replaced it with Thin, and attached its instances to Apache with mod_proxy and ilk. Now it ran like a dream! Then I re-enabled all the other sites in Apache.

Now, I’ll admit. I’m not an HTTP guru. I may not fully understand mod_rewrite, I might not completely grasp Apache configuration directives. You may not either. Or maybe you do, and you just can’t fucking wrap your mind around why Apache refuses to redirect everything in its proper place even though everything makes sense to you, at 6:30 in the morning while you nurse a hangover. In this case, the shit I was shoveling finally convinced me to rip out Apache, and it was the best thing I could have done.

I installed lighttpd and immediately fell in love with it. First I configured Trac with it, before stepping back, aghast in horror in the amount of site-specific code filling the lighttpd.conf file, and replaced it with a proxy to Trac’s stand-alone server. Next, I wrote a proxy for Apache to deal with SVN — out of the way, on another port number, since its mod_dav_svn was more adept at handling the htpasswd-specific authentication I required.

And finally, I attached the Thin instances I was running to the configuration file. Now all Lighty did was exactly what it was supposed to: guide traffic to and fro.

### Various Connections
$HTTP["host"] == "example.com" {
proxy.server = (
        "/trac" => (
                ( "host" => "127.0.0.1", "port" => 9090 )
        ),
        "/svn" => (
                ( "host" => "127.0.0.1", "port" => 8080 )
        ),
        "/" => (
                ( "host" => "127.0.0.1", "port" => 3000 ),
                ( "host" => "127.0.0.1", "port" => 3001 ),
                ( "host" => "127.0.0.1", "port" => 3002 )
        )
    )
}

Beautiful. And not an iota more information than is necessary to do its job.

I’m running the Trac daemon with the following options:

/usr/bin/python /usr/bin/tracd -s -p 9090 -b 127.0.0.1 /var/trac/project/ --base-path=/trac -d

The --base-path option ensures tracd doesn’t get confused and try to redirect traffic to http://example.com, instead of http://example.com/trac.

So thanks Hongli, for an actually constructive suggestion. I did consider REE, before I took a step back and walked through everything. Phusion Passenger is a great product and I don’t mean to knock on it, but I do worry that it’s a sort of behemoth when compared to smaller Rack-lite applications that it can run. However, when it comes to understanding exactly why, I’m way out of my league. I’ll let the experts decide how to factor this in, or if my complaint has any merit at all.

Believe it or not, Linus’ Law is not the universal solution to everybody’s problem, nor is one an idiot (around here it’s okay to use bad words) if they don’t immediately succeed in everything they do, the first time through. In my case, the solution was letting every piece of the puzzle fend for themselves, doing what they did best. And I didn’t have to join a single mailing list, or ask for a single commit bit, or write a single specification. On top of that, I’ve still helped the programming community, because now we all know a little bit more about how to handle this scenario.

I’d also like to take the opportunity to laugh at the idea that it was my Ruby application itself that was the bottleneck in this situation. While I admire the kind of omniscience that doesn’t even have to look at code to consider it FAIL, that’s not actually a diagnosis. That’s zealotry disguised as nonchalance.

Seriously Reconsidering My Dedication to Ruby

2008-11-01T11:10:47+00:00

One Ruby web application. Sinatra and Rack, just under 300 lines. SQLite, almost no DB access.

# passenger-memory-stats 
-------------- Apache processes --------------
PID    PPID  Threads  VMSize    Private  Name
----------------------------------------------
2654   1     1        242.0 MB  0.0 MB   /usr/sbin/apache2 -k start
2102   2654  1        363.0 MB  4.3 MB   /usr/sbin/apache2 -k start
2135   2654  1        372.0 MB  5.0 MB   /usr/sbin/apache2 -k start
9454   2654  1        367.1 MB  4.9 MB   /usr/sbin/apache2 -k start
10339  2654  1        364.0 MB  3.8 MB   /usr/sbin/apache2 -k start
10396  2654  1        363.7 MB  15.7 MB  /usr/sbin/apache2 -k start
12279  2654  1        378.5 MB  6.5 MB   /usr/sbin/apache2 -k start
12280  2654  1        362.4 MB  6.3 MB   /usr/sbin/apache2 -k start
13714  2654  1        359.3 MB  5.4 MB   /usr/sbin/apache2 -k start
16431  2654  1        242.0 MB  0.2 MB   /usr/sbin/apache2 -k start
16433  2654  1        242.0 MB  0.4 MB   /usr/sbin/apache2 -k start
### Processes: 11
### Total private dirty RSS: 52.48 MB

--------- Passenger processes ----------
PID    Threads  VMSize    Private  Name
----------------------------------------
2097   13       143.7 MB  0.2 MB   /usr/lib/ruby/gems/1.8/gems/passenger-2.0.2/ext/apache2/ApplicationPoolServerExecutable 0 /usr/lib/ruby/gems/1.8/gems/passenger-2.0.2/bin/passenger-spawn-server  /usr/bin/ruby1.8  /tmp/passenger_status.2654.fifo
2098   2        51.1 MB   1.5 MB   Passenger spawn server
16438  1        120.3 MB  20.5 MB  Rack: /webapps/AppOne
16440  1        120.3 MB  40.2 MB  Rack: /webapps/AppOne

What a fucking joke.

Hooray, Actors!

2008-10-31T14:17:49+00:00

A coworker just dropped an ancient artifact off for me to read:

Actors, concurrency and Kamaelia

2008-10-29T14:57:53+00:00

Recently, I made an offhand comment here about:

I've actually started thinking about/sketching an actor model build on top of MP, using concepts from actors/monitors and things in the ecosystem today

The ensuing comments and discussion were pretty good - but last night Michael Sparks (of Kamaelia) posted a darned nice comment:

Are you aware that a complete mini-axon using generators is tiny and the rest is optimisations and extra stuff that you find useful in real world systems? By tiny, I mean this small:
* http://www.kamaelia.org/MiniAxonFull

A mini-axon using processes would be equally lightweight (shorter probably) and pretty awesome.

Also, it's easy to confuse the two halves of Kamaelia. If you think of Kamaelia as just an actor-type implementation, then it's actually more an actor-like implementation, with STM & an internal SOA system of just over 2000 lines (which is how big Axon actually is, excluding comments & docs), with 80,000 lines of examples...

However, personally I view it as a mechanism for building components which happen to be best used in a concurrent fashion. ie rather than viewing it as "a mechanism for using concurrency", I view it as "OK, assume we have concurrency, how can we use this to assist in building and maintaining systems". Axon also gives you the tools for taking these concurrent systems, and interfacing between concurrent systems and standard code. (http://www.kamaelia.org/AxonHandle)

As a result, I view Axon as a library which provides you with the tools wrapping up idioms useful for building collections of components which be a framework.

Anyway, potato/potato, tomato/tomato - if you like, you like, if you don't, you don't.

I'd love to replace our existing process based stuff btw with a multiprocessing based version though. If I was going to go down this route, I'd follow our mini axon tutorial to do so, largely becauseit's essentially the starting point I took with the multiprocess stuff recently and it worked out pretty well.

Beyond this basic stuff though, I've noted that people generally start talking about co-ordination languages and building up pattern repositories. The interesting intersection between these two which you get if you call things components rather than actors is it becomes natural to create components called a chassis. These chassis often instantiate directly in concrete usable form concepts that you'd normally refer to as a pattern - Pipeline, Graphline, Carousel, Backplane, Seq, TPipe, etc.

On a random note, you may want to check out MASCOT "Modular Approach to Software Construction, Operation and Test". I heard about it late last year, and it appears to have the same sort of architecture as Kamaelia. Interestingly (to me) it makes the same key decision - when you send a message outside your component, you don't know who is going to receive it. This then enables (and requires) a higher level system for connecting components together. The upshot is highly reusable components. This doesn't entirely surprise me - my ethos came from recognising that asynchronous hardware systems & network systems look strikingly similar... (cf http://www.slideshare.net/kamaelian/sociable-so...)

Anyway, reference for MASCOT: http://async.org.uk/Hugo.Simpson/ - skip down to the end of the page for this PDF: http://async.org.uk/Hugo.Simpson/MASCOT-3.1-Man... I was really pleased to be pointed at MASCOT, largely because it showed a large number of other domains where the same basic model has been used for well over 30 years... Just with non-existent exposure, and slightly different metaphors. Though we, like it, also have mechanisms for automatically visualising systems, with a 1:1 correspondence. Beyond that this also gives us a model that matches Edward Lee's "The Problem with threads" - we'd released running code long before that paper was published :-)

Anyway, I'm glad that you're looking at what we've done. If you use it, that'd be great, and I'll happily merge anything you'd like to have a life. (the only comment I'd make there is metaphors and accessibility count - this is surely the point of python? :-) If you don't take what we use etc but it helps you solidify your thoughts to "No, I don't want that, I want this", then likewise, I'm equally glad. If you do that, I'd love to know what you do try, since I like to merge best practice concurrency ideas into Axon :-)

I'd *REALLY* suggest looking at MASCOT though. Really made my Christmas last year when I was pointed at it.

We're currently having lots of fun using concurrency, primarily by allowing it to make our lives easier, and forgettable about :) It'd be nice to see something similar on top of multiprocessing (which we'll do if you don't, but it'd be great if you did - but I'd understand if your view was that you prefer a pure actor (sender knows receiver) model.

Originally posted as a comment by Michael Sparks on jessenoller.com comments using Disqus.

I wanted to pull that comment out and showcase what Michael has to say, and in some way, respond. First, yes - I am looking at Kamaelia (from here on out, I'm going to call it "Kam" - I keep transposing the ae). I actually ran through the mini-axon tutorial, and when I have time, I'm trying to tease apart the internals to better understand it.

Fundamentally, I agree with you (Michael) about the aspects of making concurrency easier (and safer). Right now, I think Kam is a pretty darned good start - for a framework *grin*.

When I made the comment I made, I didn't think it would get the response it got. There has been a dog-pile of discussions about concurrency best practices/etc and in fact, there's a discussion still going on on the python-list about concurrency stuff going on right now.

Personally, I only work on the concurrency stuff part-part time - this includes my minor work on Python-Core. My day job is a test engineer - while I am building highly concurrent (and distributed) tests and I use multiprocessing and threading daily, it's not my full time pursuit. I am passionate about it, and I am passionate about improving python as a language, and library - and if I can do it as a day job and open source it, or get company time to do it, by golly I will.

When I said "I want to build an actor model" - I was not necessarily talking about doing an implementation for python-core. I'm a big believer in learning-through-implementation - so when I said "build x", not only did I mean "build something for the world" - I also meant "build something for my own benefit" so I can deep-dive into the concepts, problems etc.

This is why I am adverse to jumping in and simply "using" a framework - not because I don't think it does something exceedingly well - trust me, I am an opportunistic developer - if I can find a library that does what I need *right now* - I'll use it.

That being said - I am exploring Kamaelia, and yes hopefully I can steal some time to actually do an implementation of the process-based stuff with multiprocessing. I want to explore everything in the ecosystem today - my discussions with Adam Olsen around this stuff (and around python-safethread) and others has made me really want to explore solutions that help everyone, and take the best ideas and concepts and rolls it into something worthy of Python core.

As I have said before - I believe there is room within Python as a language, and CPython as an implementation of that language - for a virtual buffet of helpful libraries for concurrency and distributed systems. Right now, we have threading, async* and multiprocessing. There is plenty of room to grow. Maybe one day I can steal time to grab more of the concepts from java.util.concurrent and propose them via a PEP. Heck - maybe we can work as a group to propose an actor/monitor implementation for Python-Core.

So - personally, and by way of responding to Michael in a more concrete way: I'm, personally looking at anything I can to learn more about implementations and strategies. If something were to come out of it, and I felt strongly enough to propose inclusion in core, I would write and post a PEP - and not run in blindly.

I've got more than enough stuff to work on in addition to the day job and being a Dad, and yard work. Oh and the day job, which doesn't involve nearly as much distributed-and-concurrent systems in Python as I'd like :).

Heck - thinking about it we'd need a good messaging implementation too. I'll put that on the pile too.

Quick Rant (slightly off topic):

Also, can we stop talking about the damned GIL? Yes, you need locks, No, you probably don't care about the GIL. Stop yammering about how "broken" CPython is because of it- CPython is an implementation, not the final one and not the only one. If the GIL really gets you excited, either drop to a C module, use multiprocessing or something else. The GIL is here to stay for some time - either propose a PEP (and a Patch) that doesn't break CPython or hush. Enough bike-shedding - discussion is great, especially when something comes out of it, but constantly berating/lamenting things is just a bike shed. The shed is purple, now move on. Purple!

Required Reading ( in addition to Michael's links):

Fixing Slideshow (S9) and Syntax Highlighting within Markdown

2008-10-28T05:07:36+00:00

In another attempt to fill a blog entry title with as many buzzwords as I can, this diff can be applied to the code in your Slide Show library in order to take the suggestion of a previous developer and use CGI to unescape the HTML that comes back from syntax highlighting code, as opposed to manually trying to hit every element that might have been escaped.

Index: slideshow.rb
===================================================================
--- slideshow.rb	(revision 42)
+++ slideshow.rb	(working copy)
@@ -6,6 +6,7 @@
 require 'fileutils'
 require 'ftools'
 require 'hpricot'
+require 'cgi'
 require 'uv'
 
 
@@ -388,10 +389,7 @@
           if Uv.syntaxes.include?(lang)
             code = e.inner_html.sub(/^\s*#!\w+/, '').strip
 
-            code.gsub!( "&lt;", "<" )
-            code.gsub!( "&gt;", ">" )
-            code.gsub!( "&amp;", "&" )
-            # todo: missing any other entities? use CGI::unescapeHTML?
+            code = CGI::unescapeHTML(code)
             logger.debug "code=>#{code}<"
 
             code_highlighted = Uv.parse( code, "xhtml", lang, opts.code_line_numbers?, opts.code_theme )

Guido answers your questions…

2008-10-27T18:36:27+00:00

Via GVR's blog he's put answers up on the ask-a-google-engineer page for the top 20 questions.

I'm seriously laughing here:

Q: "Why google choose python as the main programming?"
A: Incorrect this is.

Oh, and yay multiprocessing (see: this question) but man is there a lot of work to do. I've actually started thinking about/sketching an actor model build on top of MP, using concepts from actors/monitors and things in the ecosystem today

October Issue of Python Magazine is live.

2008-10-27T13:50:54+00:00

Another information rich issue of Python Magazine is live! This time, it's the October (stunning, I know) issue. The cover story is on "Versioning your database with sqlalchemy-migrate" - and of course, there's a short article from me on "SSH Programming with Paramiko". You can see the run down for the article here.

A lot of good people put a lot of work into the magazine - Doug Hellmann doesn't let any of us slack off, and keeps us on target. If you don't already have a subscription - you should get one.

Additionally, if you like writing - or even if you're new to writing, if you have and idea for an article please submit it!

Also remember, the people behind Python Magazine are having a conference in november - PyWorks is happening in Atlanta, GA on November 12-14th. I'll be doing a talk there on threading/multiprocessing.

Updated pyjavaproperties

2008-10-24T14:07:42+00:00

I just pushed a minor update for PyJavaProperties out - this just adds a simple list as an index of the keys as they're parsed, and appends new keys on the internal index. Next, I want to keep original comments and whitespace.

Nose-testconfig version .5 uploaded.

2008-10-23T13:10:30+00:00

Fixes a minor issue with python config file parsing.

Next up, hierarchical YAML files!

Backport of 2.6 Multiprocessing to 2.4/2.5

2008-10-23T13:01:11+00:00

Thanks to the work of Christian Heimes and Skip Montanaro with a supporting role by me, the 2.6 version of Multiprocessing (pyprocessing) has been forked/adjusted for compatibility with Python 2.4/2.5.

You can see the pypi package here, and the google code project here.

We chose to do the back port for a variety of reasons - having the new API for the package makes it easier to jump into 2.6 if you're like me: a heavy pyprocessing user, not to mention that during the migration into python-core, a lot of bugs were fixed.

Check it out, and feel free to file bugs. We're planning on back porting bug fixes from python-core to the project, and applicable bug fixes from the google code project into python-core.

Blogging Disaster!

2008-10-21T23:10:12+00:00

Disaster has stuck my blog! Today, while clearing out my spam (which I seem to get a lot of these days) it seems a possible bug in the comment interface of Habari all but one of my comments has been deleted. I've checked and I don't have a recent enough backup, and I stupidly didn't have the undelete plugin activated!

So if you have left a comment on the site and it's gone, sorry!

Now off to backup...!

YotD Update

2008-10-19T02:30:17+00:00

I've slowed down my posting quite a bit, and the reason is the game I'm working on is coming along a lot faster than I thought. I had planned on posting a bunch of update videos, basically just capturing some of my unit tests and so on, partly because I thought ...

Camera Component Update

2008-10-19T02:30:17+00:00

I've updated my XNA camera component quite a bit. Initially, moves were handled by passing in a new position vector, a new lookat vector, and the time required for the camera to get there. This was pretty horrible because it required a lot of calculations for very little payoff. The biggest ...

Fixing .NET Installers

2008-10-19T02:30:17+00:00

Another ridiculous failure on the part of Microsoft to understand how small developers need their tools to work. You've finished your .NET project and are ready to distribute. Visual Studio has spit out two files for you: a setup.exe which makes sure the user has the requirements (the targetted version of ...

More McCain supporters prove why they are stupid.

2008-10-16T10:03:20+00:00

jMaps New Beta Release

2008-10-14T21:31:12+00:00

Hey folks, just a quick update - I've posted some news on the jMaps Google Group about a new release.

There are quite a few underlying changes to the code, including a new build script to allow me to separate the code out for easier reading and will allow for other mapping libraries. The API is also a lot cleaner, and I've provided a couple of details on how to use it.

Fuller docs are coming soon, I just need to get scriptdoc in Aptana to generate better ones with examples, but most of the code is commented.

Please feel free to feedback on anything in the code, either on the group or in the issue tracker.

McCain Supporters must die

2008-10-13T22:27:29+00:00

Ok, My title is a little sensationalist, but seriously as a UK citizen, I am following with great interest the upcoming US elections.

I don't like McCain and his whole campaign. Being a Liberal European Socialist who is pro-choice, and atheist and against the war you would think that would be enough. But then I saw something that made me a little sadder.

I present this and this.

If McCain gets in because of these people - some who cannot even articulate or follow a train of thought for more than 30 seconds - then ALL of America is dead to me.

Obama supporters, PLEASE!! You need to make sure this man, and the people that follow him don't win! Get out there, show people who don't vote or who are on the fence that McCain = Stupidity.

jMaps & JavaScript: The Missing Manual

2008-10-13T21:39:17+00:00

Tonight I received an email from one of the Translators working for O'Reilly on the German translation of JavaScript: The Missing Manual.

Turns out that the book uses jQuery a lot and one chapter features my own library, the jMaps library! It was a bit of a surprise for me to learn this now, especially since the book as been out since June.

jMaps is one of these projects that I had the best of intentions for, but because I haven't done much mapping stuff myself it's been put to one side, but this is certainly a good confidence boost and I hope to soon make some more improvements to the library.

I've had a few ideas already, and I'd love to hear yours too! So please feel free to comment on any ideas or suggestions.

If you would like to try out jMaps, you can download it from here.

Quick Update

I've been playing with some ideas, and here is a sneak preview of one :) Multiple Maps!

A nose plugin to run JUnit tests via Jython.

2008-10-12T14:07:33+00:00

Floating an idea to all of you in pythonland - I've been sketching out a nose plugin that would be able to find and execute JUnit tests in your for your java code base.

The reporting of the result would be unified - using nose-xunit for non-junit tests, you should get back one happy-shiny report for all of your tests.

Anyone have any thoughts out there?

New project: pyjavaproperties

2008-10-11T13:03:52+00:00

I just uploaded a new pypi package for "pyjavaproperties" - this is a fork of this recipe with the authors permission.

I'm working on a small number of improvements, but my real reason to do this was that - 1: I have to work with properties files. 2: I need it up there so it can be easy_installed and 3: It makes sense as a project!

In any case, here's the pypi page: http://pypi.python.org/pypi/pyjavaproperties/0.1

And the google code page: http://code.google.com/p/pyjavaproperties/

Suggestions welcome. I am going to monkey in support for this in nose-testconfig as soon as I have some spare cycles.

Hooray, retirement!

2008-10-10T15:40:52+00:00

I don't talk politics or current events here - mainly because I'd come off a a Lolbertarian whack-job, but I just had a serious "what in the crap?! moment when I checked my 401k:

Personal Rate of Return from 01/01/2008 to 10/09/2008 is -41.9%

This is one of the moments I look at my daughter and think about the future and get really sullen, so instead, I'm going to go back to solving technical problems as I can't even rationally discuss this in my own head.

TestButler update (updated)

2008-10-10T13:42:26+00:00

With the much-appreciated help of Brandon Barry (with whom I just happen to work) - there's been a needed update to the testbutler code base I couldn't get to - some highlights:

Cleaned up the CSS, moved to blueprint for the larger portion of the CSS and the start of jQuery usage for the javascript portions
Templates have been cleaned up/gotten a major facelift
site-media has been cleaned up
Deleted unused code I had in the prototype
Models cleaned up

Overall, it's looking much better. Yes, it's in a production-use now, and I love markdown syntax. You can see some screen shots here.

We ditched the roomba-picture, I need to find someone handy with artwork to maybe make some custom icons/pics for us (I really want a cartoony-robot-butler)

There's a lot I'd like to do, obviously - but first I have to get started on the results trackers and the corresponding nose plugin to feed the results to the system. I figure I am going to use the nose-xunit plugin and some custom XSLT for now.

One thing I need to figure out is if the django-markdown plugin allows for relative %url% links within a block of text so we can cross-link testcases, I may write a custom template tag.

Edit: Additionally, I just committed a change to remove all notion of "component" from the system. We decided that a test case could have any number of components, or none at all, and that it was more logically consistent to track those as tags-in-the-cloud. For example, a given test case might be tagged "gui, regression, smoketest, performance" or "smoke, storage, gui" etc. Being more flexible with sorting and organization was our goal.

<head> Conference

2008-10-10T10:34:51+00:00

I'll be "attending" the <head> conference - which have a 20% discount on ticket prices for TODAY ONLY! Go check it out and support the conference!

Elgg and Open Data Definition

2008-10-09T22:40:38+00:00

Earlier today, I was thumbing through upcoming.org when I saw a local event for something called Elgg. Having checked it out, it seems like a very interesting project - to me it's like Drupal, for social networks.

At work, we have a lot of clients who are looking to social media as their "next big thing" and certainly Elgg looks like it's more than up to the task of creating micro-social networks that allow clients to engage with their customers. I'd be interested in looking at creating some kind of Adobe AIR app that could interface with it for one.

As I browsed the site however, I came across a link to something called the Open Data Definiton. Being interested with anything with "Open" in front of it (OpenID, OAuth) I decided to check it out.

Having read the specs, I have to say I was impressed. The idea is a few steps ahead of a similar idea I had for developer profiles that could be attached to distributed projects on source control like Mercurial or Git. My idea was to have the users profile contain data about open source projects they worked on, their programming skills with self-scoring and peer scoring through a voting mechinisim ("At PHP I score myself a 8, other people score me 6") and other relivent data.

I've posted my suggestion to the google group, and I hope to be able to discuss it at the upcoming meeting.

New version of nose-testconfig uploaded.

2008-10-08T21:12:20+00:00

I fixed a few knits Kumar pointed out, removed some overly aggressive eval'ing of ini file values, and fixed the damned rst docs.

http://pypi.python.org/pypi/nose-testconfig/0.4

I think 4 people in the world are using this - all 4 of them work with me, hooray!

Threads can’t be serialized?!

2008-10-08T18:32:19+00:00

Note the subject is tongue-in-cheek - I realize you can't pickle threading.Thread and multiprocessing.Process, but for some (possibly sick) reason I want to do an implementation for both that implements __getstate__ and __setstate__ and changed the __repr__ for both to not refer to any sort of run-time state.

You don't need to serialize the function thats passed in: the assumption here is that you're serializing them prior to calling start() (because serializing a running thread would be awesome - in that "hey my brain just melted" sort of way).

The best (read, only sane) use case would be to allow threads/processes to be defined in say, a YAML file and generated at parse-time (the client could call .start()) or sending a blob of serialized threads to a remote client and having it call start() to act as a slave.

And that concludes the random thought for the day.

Edit: And in the comments, Ben Hayden posted a link to a version of threading.Thread which can be serialized. You can see it here. I need to make a multiprocessing.Process version of it too, just, well, cause.

sn.printf.net

Updating the iPhone

Sequel is Gorgeous

From Common Lisp to Clojure (Part 1)

Dynamic Java?

So... What Do I Think?

Pros

Cons

Gantty: Ruby and Gantt Charts

My Faith is Restored

Seriously Reconsidering My Dedication to Ruby

Hooray, Actors!

Actors, concurrency and Kamaelia

Fixing Slideshow (S9) and Syntax Highlighting within Markdown

Guido answers your questions…

October Issue of Python Magazine is live.

Updated pyjavaproperties

Nose-testconfig version .5 uploaded.

Backport of 2.6 Multiprocessing to 2.4/2.5

Blogging Disaster!

YotD Update

Camera Component Update

Fixing .NET Installers

More McCain supporters prove why they are stupid.

jMaps New Beta Release

McCain Supporters must die

jMaps & JavaScript: The Missing Manual

A nose plugin to run JUnit tests via Jython.

New project: pyjavaproperties

Hooray, retirement!

TestButler update (updated)

&lt;head> Conference

Elgg and Open Data Definition

New version of nose-testconfig uploaded.

Threads can’t be serialized?!

<head> Conference