sn.printf.net

2012-05-13

shadows by dvs
We have all been there. We have all been made fun of, ostracized, and made to feel like we don't belong at one point in our lives. Many of us have used these experiences to strengthen ourselves, while others still struggle to feel like they belong. It can be hard when all the signs are telling you that you don't fit in. Where everywhere you look you are getting signals that not only are you not like those around you, but those around you would rather pretend you weren't there. Each of us has a choice in this situation. We either run or we fight. It seems that in the tech world people are finally choosing the latter.

It can be hard walking into a room and having everyone asked their name except for you. It can be hard being told directly, "why would you do this to yourself," when mentioning what field you are in. It can be hard to realize that there are those who no matter what you do honestly believe that you do not belong in the same field as them, let alone the same job. The words get to you and you doubt. There's an inherent anxiety in your job, because of the feeling of breaking some sort of unwritten rule. The words "I don't belong here" are engraved on your brain.

We have all been there. We want to belong. This is why this struggle for representation in the field is not something that a single voice is calling for. Others have stepped up to the plate and allowed us to make a bigger sound, but we need more. The vocal minority is still telling us that we do not belong and so we need the majority to stand up. We need every single voice that we can get to call out those that are wrong and to reach out into their communities and try to make it right. We need you.

You cannot hope to build a better world without improving the individuals. To that end each of us must work for his own improvement, and at the same time share a general responsibility for all humanity, our particular duty being to aid those to whom we think we can be most useful.

- Marie Curie


Photo credit: dvs

2012-05-13 22:07

2012-05-07

In this article I implement a library that allows one to synchronize Org-mode to-do items directly to Apple's iCal. More generally, I describe a synchronization layer between OS X, Clozure Common Lisp, and Emacs, and muse on its potential.

Background

It is a source of joy and pain to customize and extend Emacs. There are many reasons for the attempt, however, and every person has different reasons. I typically describe a few of mine as follows.

  • When tasks are performed alongside each other in Emacs, it is conceptually simpler to move between them. Most commands have a sort of "Do What I Mean" universality to them.
  • Passing data between membranes – copying a URL in a web browser to a chat window, e-mailing a snippet of code to a mailing list, or writing an Org-mode outline before publishing it online – is typically effortless.
  • Every mode and function in Emacs is another tool in your toolbox. You can easily combine tools in the Unix way to slice through work. When you have more tools at your disposal, you can solve more difficult or nuanced problems.

On the other hand, I have an iPhone and an iCloud account, and don't spend all my time in Emacs exclusively. I want to be able to use the fantastic software outside of Emacs, but have the power and flexibility of Emacs.

I've created a dual-language project. One half is in Emacs Lisp; the other in Common Lisp. It uses the foreign function interface provided by Clozure CL. From Emacs, I use SLIME to access Clozure's REPL, which allows me to make calls to the storage APIs for iCal. Once I retrieve the results I want, I can pass them back to Emacs. I can also make changes using this API, allowing Emacs to inform iCal of what changes have been made in my to-do lists.

Environment Setup

There is a one-time cost of a complex environment setup. You will need a copy of OS X and Emacs. I am doing all this work on Lion, and I don't know if earlier versions of OS X will have all the tools we need. I am using Emacs 24:

GNU Emacs 24.0.90.1 (i386-apple-darwin11.2.0, NS apple-appkit-1138.23)

I should mention that Emacs 24 is incredibly stable. There is a lot of activity and approval surrounding its use as a day-to-day text editor.

Make sure you install Clozure Common Lisp. Clozure's Objective-C support is a unique feature of its distribution. This article could not have been written without the indispensable support of their wiki and manual.

You can do this however you'd like, but homebrew makes it easy.

$ brew update
$ brew install clozure-cl

You should have ccl64 in /usr/local/bin, and it should be at least version 1.7. Type (quit) at the REPL to exit it.

You will need SLIME, the essential Lisp IDE for Emacs. What I do is open up my Lisp REPL in my shell, install SLIME through Quicklisp, and then symbolically link the package into my Emacs directory (a tool exists to do this, quicklisp-slime-helper, but out of habit I still do it the more old-fashioned way):

$ ccl64 --load quicklisp.lisp
 
? (quicklisp-quickstart:install :path ".quicklisp-ccl64/")
? (ql:add-to-init-file)
? (ql:quickload 'swank)
? (quit)
  
$ ln -s $HOME/.quicklisp-ccl64/dists/quicklisp/software/slime-*-cvs/ $HOME/.emacs.d/vendor/slime

I've also built up a bit of a preparation function for initializing SLIME, so I will reproduce that below as well.

(defun quicklisp-slime-setup ()
  (add-to-list 'load-path (expand-file-name "~/.emacs.d/vendor/slime"))
  (add-to-list 'load-path (expand-file-name "~/.emacs.d/vendor/slime/contrib"))
  (setq inferior-lisp-program "/usr/local/bin/ccl64")
  (setq slime-autodoc-mode t)
  (setq slime-net-coding-system 'utf-8-unix)
  (require 'slime)
  (slime-setup '(slime-repl slime-fancy)))

Building ffigen

ffigen is a utility that slurps up frameworks and provides foreign function interfaces into 'intermediate forms'; in this case, Lisp forms. These instructions exist on the Clozure CL wiki. I'm reproducing them here since they're not easy to find and the wiki may change.

Building a Leopard version with ObjC 2.0 support

# Note that the svn directory is misnamed; this is actually based on Apple GCC 5464 (not 6465).
$ svn co http://svn.clozure.com/publicsvn/ffigen4/branches/ffigen-apple-gcc-6465/ffigen4
 
$ cd ffigen4
$ make
  
$ sudo tar zxvf ffigen-apple-gcc-5465-intel-*.tar.gz -C /usr/local

Generating the Foreign Function Interface

Calendar Store is the name of the framework that Apple provides for accessing calendar data. We need to generate its foreign function interface for Clozure. I did so by copying the FFI generation script from the addressbook framework that Clozure provides access to, and turning it into a script for the calendarstore framework. The script, populate.sh, uses the h-to-ffi.sh terminal command installed by building ffigen above.

I should add that this works with the version of Xcode distributed by the App Store, as well as any version of Xcode you may have in /Developer, but I'm only documenting the former. The changes needed for the latter should be self-evident.

 1:  $ export SDK=/Applications/Xcode.app/Contents/Developer/Platforms/\
 2:  MacOSX.platform/Developer/SDKs/MacOSX10.6.sdk
 3:  
 4:  $ cd $SDK/System/Library/Frameworks/CalendarStore.framework/Headers
 5:  
 6:  $ sed '1i\
 7:  #import <Foundation/Foundation.h>\
 8:  ' CalendarStore.h | sudo tee CalendarStore.ffi.h
 9:  
10:  $ cd /usr/local/Cellar/clozure-cl/1.8/ccl/darwin-x86-headers64
11:  $ mkdir -p calendarstore/C
12:  $ sed 's/AddressBook/CalendarStore/;s/AddressBook/CalendarStore.ffi/' \
13:    addressbook/C/populate.sh > calendarstore/C/populate.sh
14:  $ sed -i '' "s/^SDK.*/SDK=$SDK/" calendarstore/C/populate.sh
15:  $ chmod +x calendarstore/C/populate.sh
16:  $ cd calendarstore/C
17:  $ sh populate.sh

Line 1 above assumes that you are using Xcode off of the App Store. If not, the SDKs/ directory will be located in /Developer.

Lines 6, 7, and 8 comprise a single command, adding a missing import to the CalendarStore header file. Once we populate the framework, we must then tell Clozure about it.

CL-USER> (require 'parse-ffi)
CL-USER> (ccl::parse-standard-ffi-files :calendarstore)

After this is done, we never have to do it again. From now on, whenever we open up a Clozure REPL, we can just load up everything we need.

CL-USER> (require 'objc-support)
CL-USER> (objc:load-framework "CalendarStore" :calendarstore)

Explaining the Bridge

There are a couple aspects of Objective-C and how they translate through the Bridge that we need to be aware of.

Methods in Objective-C

Objective-C uses a message passing syntax. Whereas languages like C++ bind method names to some code during compilation, Objective-C resolves message receivers during runtime. Message names are typically longer than method names because they also contain identifying information about the message's parameters, separated by colons.

Whereas in C++ you may call something like:

obj->method(foo, bar)

In Objective-C you would call:

[obj method:foo andBar:bar]

And we'd consider the message name to be method:andBar:. Yes, even the colons are part of the message name. In our bridge, that method invocation would become:

(#/method:andBar: obj foo bar)

Now that we know this, we can start using Apple's API to pull objects out of the runtime. Open up a REPL with Clozure CL as your inferior Lisp. First we need to load up the bridge.

CL-USER> (require 'objc-support)

Many commonly used Foundation Framework objects are available for us immediately. These objects start with the prefix ns-. An NSDate would become an ns-date. An NSString would become an ns-string. Say we wanted to create an NSDate with the current date and time, and print its string representation. Let's put this functionality in a function called today_as_string.

Take a look at the difference as to how methods are called in Objective-C. today_as_string, implemented in Objective-C, and then Lisp.

NSString* today_as_string() {
  NSDate *today = [NSDate date];
  NSString *date_string = [today description];
  [today release];
  return date_string;
}

(defun today-as-string ()
  (let* ((today (#/date ns:ns-date))
         (date-string (#/description today)))
    (#/release today)
    date-string))

Memory Management

Based on the Clozure documentation (§15.3.3), an autorelease pool is created for us in the toplevel. This means as good citizens, we should manually release the memory when we finish using it. Calling [NSDate date] automatically calls init and alloc on the instance, and the (#/release today) call above dovetails that by decreasing the reference count.

For more detail on reference counting, please see Apple's "Advanced Memory Management Programming Guide".

Miscellaneous

Here are some other things to note. We can get a reference to any class by passing its camel-cased name to @class.

CL-USER> (objc:@class "NSString")
#<OBJC:OBJC-CLASS NS:NS-STRING (#x7FFF75FE69F8)>

We can determine if an object is a null pointer by using %null-ptr-p:

CL-USER> (%null-ptr-p (#/date ns:ns-date))
NIL
CL-USER> (%null-ptr-p (%null-ptr))
T

We are given a reader macro for generating NSStrings (prefix it with a pound sign, #), and we can get a Lisp string out of an NSString with Clozure's lisp-string-from-nsstring function:

CL-USER> (ccl::lisp-string-from-nsstring #@"test")
"test"

The use of the pound sign in the above code ((#/date ns:ns-date)) is a reader macro as well. You can get a hint as to how it works by checking the contents of the package nextstep-functions.

Introducing nsclasp

I've written a small utility library for Common Lisp called nsclasp. It provides a set of functions that are very useful when working with Apple's Foundation Framework. You can retrieve it by cloning it into your Quicklisp installation's local-projects directory and refreshing the project list:

$ cd ~/.quicklisp-ccl64/local-projects
$ git clone git://github.com/ardekantur/nsclasp.git
CL-USER> (ql:register-local-projects)
CL-USER> (ql:quickload 'nsclasp)

For more information, see the project documentation.

Implementation

The most important aspect of iCalSync, the thing that makes this so delightfully easy, is that Emacs Lisp can talk directly to the Clozure instance. The key function is this:

(defun icalsync-capture-ccl-result
(command)
  "Convert a result from the SLIME Lisp's REPL into the Emacs\nLisp toplevel."
  (read
   (slime-eval
    `(swank::pprint-eval ,command))))

Using this, we can send code to Clozure's REPL, get the result back, and turn it into a Lisp form.

First Steps: Retrieving Calendar Names

Let's make two useful functions for accessing the CalendarStore, then try to retrieve a list of calendar names from Clozure.

(defun icalsync-cl-calendar-store ()
  "Return the calendar store. Equivalent to
    [CalCalendarStore defaultCalendarStore]."
  (#/defaultCalendarStore ns:cal-calendar-store))

(defun icalsync-cl-all-calendars ()
  "Return a list of all calendars in the default calendar
    store. Equivalent to [[CalCalendarStore defaultCalendarStore]
    calendars]."
  (#/calendars (icalsync-cl-calendar-store)))

(let ((calendars (icalsync-cl-all-calendars)))
  (loop for i upto (1- (#/count calendars))
     for calendar = (#/objectAtIndex: calendars i)
     do (format t "~A, " (ccl::lisp-string-from-nsstring (#/title calendar)))))
Calendar, Matthew Snyder, Do, Buy, Inbox, Example, 

Our handling of the NSArray is a little clumsy. We can't just mapcar over it, we need to explicitly iterate over its elements by index. That's why nsclasp is helpful, as it contains a method to make this a little nicer:

(let* ((calendars (nsclasp:ns-array->list (icalsync-cl-all-calendars))))
  (format t "~{~a, ~}" (mapcar (lambda (c) (ccl::lisp-string-from-nsstring (#/title c))) calendars)))

But no matter how you do it, you should have been able to retrieve all your calendar names. As it turns out, this is an important piece of information to have, as the Foundation Framework doesn't yet allow us to create the kind of calendar that would sync with iCloud for us. For the time being, we have to create the calendar we want our TODOs synced to, in iCal.

/images/icloud-reminders-orgmode/new-calendar-list.png

First Steps: Creating a CalTask

Now let's combine this on the tail end of creating an iCal reminder, and sending it to a calendar.

Tasks exist in iCal as instances of the CalTask object of the CalendarStore Framework. Every task has a title, UID, a parent calendar, alarms, due dates, completed dates, notes, and other such attributes. We have to describe a mapping. Our input function takes an Emacs buffer and returns a list of parsed TODOs, and what calendar they should be synced to.

(defun icalsync-el-transform-todos (buffer)
  "Turn an org mode buffer into a list of iCal tasks.

A function that describes one way to retrieve all of the valid
TODOs from a buffer for turning into respective iCal tasks. You
can write a function that best suits your workflow. The function
is expected to take the name of a buffer, or a buffer, and return
a list of plists, each one representing a TODO item / iCal task.

FIXME(msnyder): Use org-map-entries to make this easier."
  (save-excursion
   (let ((calendar-uid (icalsync-el-calendar-name)) todos)
     (switch-to-buffer buffer)
     (beginning-of-buffer)
     (while (re-search-forward org-todo-line-regexp nil t)
      (let* ((todo-text (match-string-no-properties 3))
             (components (org-heading-components))
             (todo-state (nth 2 components))
             (headline (nth 4 components))
             (scheduled-time (org-get-scheduled-time (point)))
             todo-uid)
        (if (and todo-state
                 (funcall icalsync-el-valid-p-function todo-state todo-text))
            (progn (if (string-match org-todo-line-tags-regexp headline)
                       (setq headline (nth 4 (org-heading-components)))
                       (setq todos
                             (cons (list :buffer
                                         (buffer-file-name)
                                         :point
                                         (point)
                                         :calendar-uid
                                         calendar-uid
                                         :incomplete-p
                                         (funcall
                                          icalsync-el-incomplete-p-function
                                          todo-state)
                                         :title
                                         (org-trim headline)
                                         :scheduled
                                         (and
                                          scheduled-time
                                          (decode-time scheduled-time))
                                         :uid
                                         (cdr
                                          (assoc
                                           "ICALSYNC-UID"
                                           (org-entry-properties)))
                                         :dt
                                         (and
                                          (cdr
                                           (assoc
                                            "ICALSYNC-DT"
                                            (org-entry-properties)))
                                          (read
                                           (cdr
                                            (assoc
                                             "ICALSYNC-DT"
                                             (org-entry-properties))))))
                                   todos)))))))
     todos)))

I've created a basic Org file.

#+TITLE: example.org

* TODO Recycling
  SCHEDULED: <2012-04-15 Sun>
* TODO Trash
* TODO Fix toilet

I run the function on it, and after getting asked which calendar I want the items to belong to, I get the following list of lists.

(icalsync-el-transform-todos "example.org")
:buffer/…/example.org:point78:calendar-uidC854CB60…:incomplete-pt:titleFix toilet:schedulednil:uidnil:dtnil
:buffer/…/example.org:point60:calendar-uidC854CB60…:incomplete-pt:titleTrash:schedulednil:uidnil:dtnil
:buffer/…/example.org:point17:calendar-uidC854CB60…:incomplete-pt:titleRecycling:scheduled(0 0 0 15 4 2012 0 t -14400):uidnil:dtnil

When we upload a task into the Calendar Store, we're going to provide its title, deadline, and whether or not it's complete. There's also priority, which would be nice, but not necessary for the time being. Once the upload is successful, we're going to also retrieve the task's UID, and its dateStamp property, and save them in the property drawer of the TODO, in order to make synchronization easier.

(defun icalsync-el-upload-todo (todo)
  (let (todo-uid)
    (save-excursion
     (setq todo-uid
           (icalsync-capture-ccl-result
            (format "(icalsync-cl-create-task '%S)" todo)))
     (find-file (getf todo :buffer)) (goto-char (getf todo :point))
     (goto-char (car (org-get-property-block nil nil t)))
     (org-set-property "ICALSYNC-UID" todo-uid)
     (org-set-property "ICALSYNC-DT" (current-time)))))

(defun icalsync-el-upload-todos (todos)
  (loop for todo in todos do (icalsync-el-upload-todo todo)))

From here, all we need to do is run the Emacs Lisp functions needed to transform all the TODOs and upload them into the Calendar Store. Keep iCal open when you do this and make sure the Reminder list is visible.

/images/icloud-reminders-orgmode/shown-calendar-list.png

(setq *todos* (icalsync-el-transform-todos "example.org"))
(icalsync-el-upload-todos *todos*)

Et voila!

/images/icloud-reminders-orgmode/new-reminders.png

First Steps: Retrieving Changes

So now you have your Org-mode tasks in a Reminder list on your iPhone. Brilliant! You wander around the town, happily marking off the items you've completed. How can we make Org-mode aware of the items that were checked off?

This is a very rudimentary method, but it works. What we'll do is update TODOs based on buffer, the same way we uploaded them to iCal. For a given buffer, we'll walk through the TODOs, seeing if they have a UID handed out by the calendar store. If so, we ask the calendar store about it. We compare the time-stamp of the TODO to the time-stamp of the reminder, and if the reminder has been changed more recently, we update the TODO.

(defun icalsync-el-download-todo (todo)
  (save-excursion (find-file (getf todo :buffer))
   (goto-char (getf todo :point))
   (let* ((result
           (icalsync-capture-ccl-result
            (format "(icalsync-cl-task->plist (icalsync-cl-get-task %S))"
                    (getf todo :uid))))
          (t1
           (apply 'encode-time
                  (read (cdr (assoc "ICALSYNC-DT" (org-entry-properties))))))
          (t2 (apply 'encode-time (getf result 'datestamp))))
     (if (time-less-p t1 t2)
         (progn (if (getf result 'completeddate) (org-todo 'done))
                (org-set-property "ICALSYNC-DT" (format "%S" t2)))))))

Not the use of the function icalsync-cl-task->plist. Since we can't hand off a CalTask object from Clozure to Emacs Lisp, I've written a function that distills a task down to its most important parts.

(defvar *task-properties* (list "title"
                                "uid"
                                "priority"
                                "dueDate"
                                "dateStamp"
                                "completedDate"))

(defun icalsync-cl-task->plist (task)
  "Convert a CalTask to a plist of properties. The list of properties
    is described by *task-properties*."
  (loop for property in *task-properties*
        for value = (funcall (intern property "NEXTSTEP-FUNCTIONS") task)
        append (list (intern (format nil "~:@(~a~)" property))
                     (icalsync-cl-coerce-value value))))

Next Steps

There are a few things missing from this implementation. We don't check for the existence of a task before creating it from the corresponding TODO. Priority is not synced. Org-mode is not made aware of items added to Reminders.app. We track where our TODOs are based on the buffer name and their position in the buffer, which will quite easily break if we shuffle our TODOs around. Only the reminder's completion status is synced back to Emacs, but if its description or due date changes, we should synchronize those as well. As a proof-of-concept, however, I think this is a solid start.

In addition, the behavior that you use to turn Org-mode files into lists of reminders – or vice versa – is entirely up to you. There are countless ways to do so. A conversion could, for example, take each calendar, and make it a headline in an Org file, with each task becoming a checklist item with a property drawer. Another conversion could check the TODOs for a custom condition, i.e., only tell iCal about TODOs which have a certain property in their drawer, or are scheduled after a certain date.

All the code written for this article is available in my elstar GitHub repository. The elstar name for my repository is a bannerhead of sorts, an invitation for this method to be explored further, and to collect the results in a shared place. I welcome any contributions.

Conclusion

The point of this article was not just to create a synchronization layer for iCal to Org-mode, but to explore the method of synchronization itself. For Emacs users on OS X, the potential exists to fill in the gaps with their usage of operating system (and associated mobile/cloud convenience) and text editor. Some other ideas that may be worth pursuing:

  • An Emacs mode for interfacing with iTunes
  • Functionality to convert Address Book entries into BBDB or org-contacts entries
  • Turning events on calendars into Emacs diary entries

I look forward to seeing what some enterprising Lisp hackers can come up with.

2012-05-07 04:00

2012-05-06

Getting Started

The challenge is to take the raw compass heading data, and smooth it out over time so the UI element this data is bound to (such as an arrow) won't be jittery. You can see this jitter in the SDK example for the Compass: the CurrentValueChanged event handler is fired every 20ms and jitters within the error margin for the hardware. Our task is to smooth out these data readings for a more pleasant user interface. And to make it fun, we'll be using Reactive Extensions.

It's quite easy to calculate a periodic average of a data stream with Reactive Extensions, and we'll tackle it in three stages:

  1. Create the Observable
  2. Segment the compass readings into chunks, and get an average
  3. Capture the result and make something happen

Step 1: Create the Observable

Starting with the Compass's CurrentValueChanging event, we need to convert it into an Observable. After creating a new Compass object (hereafter simply called "compass"), we can write this with any overload of Observable.FromEvent, either by manually providing delegates to add/remove event subscriptions (as commonly listed on RxWiki), or by providing the event name as a string:

View the code on Gist.

For succinctness, I'll use the latter in this post.

If we stop right here for a moment, we can confirm that our compass is working and that we can get readings by subscribing to the Observable's stream of SensorReadingEventArgs events by tapping in with the Subscribe method and printing out the raw heading readings:

View the code on Gist.

Running this sample will spit out an endless series of raw readings in the debug output window:

So far, so good. But, we could've done this all with a standard event handler, so let's kick it up an notch...

Step 2: Break into chunks and average

We want to avoid jitter in our compass readings, and a simple way to do that is to average all the individual readings over a small time period, of say, two seconds. Without Rx, you'd need to declare a period reset timer, and then keep track of a running average over the two second period, then clear it when the timer fires and so forth.

With Rx, it's chillingly simple; we can do it by chaining just two methods: BufferWithTime followed by a standard Linq SelectMany (well, and a Select). And it's all in a single statement.

View the code on Gist.

Running this will spit out readings just like before, except now, they're clumped every two seconds, with a pause in between.

Let's break it down:

Picking up after creating the Observable, we chain a call to BufferWithTime. For the specified period (two seconds, passed in with TimeSpan.FromSeconds(2), BufferWithTime takes events and stuffs them into an IList of events. Then, we use Select to project that list of events into a list of compass headings (pulled out of the event args, e.EventArgs.SensorReading.TrueHeading). The Subscribe method, as before, allows us to debug print the compass headings to confirm.

Here's the full application so far:

View the code on Gist.

Our raw compass readings are broken into two-second lists, but they're still individually jittery. However, we can instead take each list and aggregate it into a single heading value. By replacing the SelectMany, which gave us a list of headings, with a Select, we can do the averaging right inline:

View the code on Gist.

And this gives us one average compass heading every two seconds, minus the jitter, just like we wanted:

Complete program listing so far:

View the code on Gist.

Step 3: Making something happen

We now have de-jittered compass readings coming in every two seconds. Let's hook this data up to the UI and build an actual application.

Taking this empty Windows Phone project, I've added the following Image into the default content Grid:

View the code on Gist.

There's nothing special here, it's just an image with a default RenderTransform that sets the rotational center of the image element to the middle.

Pasting in the Rx code we developed above, we can swap out the debug call and instead update the RenderTransform's angle property to the compass heading (actually, 360 minus the heading... we want the arrow image to rotate opposite the phone device. If we didn't do this subtraction, it'd rotate the header angle, but in the same direction we turn the phone.)

View the code on Gist.

(Oh, and there's one other change: adding in a call to ObserveOnDispatcher allows us to execute the Subscribe delegate (eg, update the arrow rotation) on the UI thread. We didn't need to worry about that before because we were just debug printing, and not accessing any resources created on the UI thread.)

Running the whole program now shows the compass arrow updating once every two seconds, correctly pointing to the device's compass heading. Feel free to adjust the period from two seconds to something more useful... I found 0.5 seconds to be a good balance between update frequency and heading jitter.

I've posted the complete example application on github, so feel free to check it out:
https://github.com/factormystic/rx-compass-smoothing

by Factor Mystic at 2012-05-06 05:24

2012-05-04

2012 05 04

Rebar Releases and Being Wrong

This is an entry I wrote during the Bay Area Erlang Factory 2012, not long before R15B01 came out. Back at then, I found the time to discuss a few things with fellow Erlang programmers. One of them was related to a patch to Reltool that someone submitted while before. This was a fix made to make things simpler for Erlang programmers wanting to generate releases using rebar.

In your usual Erlang project using rebar, the structure is usually based under a clean OTP Application directory structure:

ebin/
src/
include/
priv/

There is also a bunch of other optional directories, including priv/, logs/, and deps/. The deps/ is specifically the default directory for dependencies downloaded at build-time.

This way of doing things works very, very well when it comes to writing OTP Applications. As long as anyone respects that structure, getting any app as a dependency should never hurt.

There is however, an insidious extension to this done when transforming an OTP Application into a release. When people generate release, their directory structure might now look a bit like this:

ebin/
src/
include/
priv/
deps/
rel/

The rel/ directory is where the release will be built. That one will contain a release configuration file, that references to a directory with libraries is usually found. The line of configuration is precisely {lib_dirs, ["../deps/"]}. You can already see where there might be a problem — the src/ directory is getting ignored.

How Can we fix this?

Many rebar users fix this issue in two common ways:

  • Writing plugins or extending rebar to move, copy or simlink their top-level directories inside deps/
  • Adding the line {lib_dirs, ["../deps/", "../../"]} to the configuration instead of the one above

The problem with this (especially the second approach) is that it's badly using a standard that has existed in OTP for a long, long while. The idea is that Erlang works with the basic concept of a LIB_DIRS environment variable. That variable should point to one or more directory that contains a bunch of application directories:

./
 - some_app/
   - src/
   - ebin/
 - other_app/
   - src/
   - ebin/
   - priv/

Putting that directory in LIB_DIRS will make it visible to the Erlang VM. When you put that directory in the configuration for a release ({lib_dirs, [Dirs]}), Reltool (the OTP release-building tool that is called by rebar) gets these variables to be part of its places to search.

Setting the directory to ../../ is a bad idea, because it basically makes the release building take into account all the directories on the same level as the one you're currently developing. This can have unexpected effects when you put that one first and you might have conflicting versions of libraries in deps/ and the directory itself; one of the two versions will be picked, and you might not be sure which.

The patch I mentioned earlier basically solved the issue by allowing the following to be placed in application requirements: {app, foo, [{incl_cond, include}, {lib_dir, ".."}]}. This patch basically lets you keep the OTP Application structure and turn it into a release without needing the messy hacks. I might be out of touch, but I can't understand why the OTP team would accept this patch. It is basically officializing a workaround for people using OTP tools in a way it wasn't intended.

There are two actual solutions that work to build releases, and they involve using a proper directory structure:

  • All your OTP Applications have their own repository. You then make an additional directory that does nothing but fetch all dependencies to create a release
  • You put your own OTP applications inside a custom directory (say apps/) and the dependencies in a second one (deps/). You then include these two directories in lib_dirs.

That's it, really. It's not very hard to do, and it works fine with all the regular OTP stuff. You use a proper release directory structure for releases, and you can keep using a proper OTP Application structure for OTP Applications. I do not understand why the fix had to make it upstream to OTP when its main use is to transform an OTP Application structure in something compatible with releases, especially since this need is mostly a quirk of how default values in rebar and community standards that emerged from its use pushed a lot of people to do things wrong (arguably). I would have prefered for people in general to fix their repositories rather than forcing the OTP team to fix their (non-broken) tool to accomodate them there.

I guess in the end it's kind of irrelevant; the fix in no way keeps me from doing what I was doing before, after all.

2012-05-04 18:00

2012-04-21

Check out http://noplaintext.com/. It promises military-grade encryption, in the browser.

Here’s a fun game to play. Before visiting, try to guess what mistakes they made.

Then visit the site, and when you read exactly what they do, try to guess what mistakes they made.

Then look at the code and see what mistakes they made.

Then see what you missed (and what I missed).

So far:
- The encryption key is generated with 47.6 bits of entropy.
- It’s generated using Math.random().
- There is no message authentication. The server can flip individual bits within the message.
- Obviously we have to trust noplaintext.com to send us an uncompromised web page.
- It links to google-analytics, so we also have to trust the Google analytics people.
- It links directly to https://crypto-js.googlecode.com/files/2.5.3-crypto-sha1-hma…, so we also have to trust the crypto-js maintainer, and the googlecode.com people.
- The whole system depends on being able to share the URL securely. But if you can do that, you could have just sent the message securely over that channel.

by shrughes at 2012-04-21 12:35

There are some pictures out showing Thinkpads that have the chiclet-style keyboard previously seen on the X120e and X1. Now the X230, T430, T530, and W530 will have these chiclet-style keyboards instead of the previous generation of scissor-switch.

For example, see this thread: X230 picture – Lenovo Community

I am making this post to inform you that these people are completely retarded, of course.

What they’re really lamenting is their newfound inability too feel superior to other laptop users. Most people who have actually used the chiclet keyboard know that it’s just as good as, if not better than, the scissor-switch keyboard — for example, it’s more difficult for hair and dust to get lost between the keys. Also the mechanism is certainly less fragile, considering how fragile the scissor-switches are.

If you’re butthurt over the change of keyboard to something more affordable and reliable that doesn’t sacrifice key-press feel and maybe even improves on it, maybe you should look for other ways to feel superior to other computer users. You still have the smugness of the trackpoint (even though I’m using a Sony Vaio that has a superior pointing stuck), and also your laptop puts out a really nice and clean VGA signal, not one of those disgusting impure VGA signals produced by the otherbranded trash.

Another option is to get a Panasonic Let’s Note SX. It has a 1600×900 12.1″ screen or 1280×800 in some configs, so you can’t complain about resolution. It has this circular trackpad which people regard as stupid until they stop regarding it and start using it, at which point they know it’s better than any other mouse device. It has a built-in DVD drive IN THE PALMREST and it is lighter and allegedly has much better battery life than an X220. The keyboard is not some craptacular chiclet keyboard, in fact it has less row height than the Thinkpad keyboard, which is nice for your fingers, unless you’re too fat-fingered to hit the shorter keys. It’s even more durable than the Thinkpad, that’s right, your “magnesium roll cage” machine is not actually more durable than everything else, you whiny little shit. I don’t have one, because the base model costs $2600 and I’m not an idiot and I already have too many laptops. But you could have one, and then your laptop would be better than other people’s, and you’d be better than other people for knowing it’s better. Are you going to put a price on your own superiority?

by shrughes at 2012-04-21 01:32

2012-04-15

When I browse the Internet, I always fight the temptation to waste time by visiting distracting websites. By telling my computer what sites are worth visiting, I've offloaded that drain of willpower to the machine.

The DNS Level

There are a few sites that I 'block' in /etc/hosts because I have no desire for them, whatsoever. I will never want to 'cheat', and visit them, for any reason.

Here is the relevant section of my /etc/hosts:

0.0.0.0 www.facebook.com
0.0.0.0 facebook.com
0.0.0.0 connect.facebook.net
0.0.0.0 facebook.net
0.0.0.0 fbcdn.net
0.0.0.0 www.fbcdn.net
0.0.0.0 badge.facebook.com
0.0.0.0 blog.facebook.com
0.0.0.0 en-gb.facebook.com
0.0.0.0 developers.facebook.com
0.0.0.0 touch.facebook.com
0.0.0.0 de-de.facebook.com
0.0.0.0 stories.facebook.com
0.0.0.0 it-it.facebook.com
0.0.0.0 hu-hu.facebook.com
0.0.0.0 peace.facebook.com
0.0.0.0 et-ee.facebook.com
0.0.0.0 az-az.facebook.com
0.0.0.0 0.facebook.com
0.0.0.0 apps.facebook.com
0.0.0.0 reddit.com
0.0.0.0 www.reddit.com

I would love if there were a central repository of distracting sites, and if my hostsfile could be updated automatically against this repository.

The Browser Level

Earlier on I used the WasteNoTime Safari extension. It helped regulate the time I spent on distracting websites, in a reasonable way. The process went something like this:

  1. I would make a list of websites to blacklist. There's a whitelist functionality, but I never figured out how to use it in a satisfactory way. I would necessarily exclude useful sites I was unaware of.
  2. I would decide how long I would allow myself to visit the distracting websites in total. For example, I would allow myself 45 minutes on each weekday, and then disable the timer on the weekends.
  3. I would browse distracting websites until I ran out of time, and then go do work.

This process isn't horrible. I imagine it would work for a lot of people. For me, however, it was simply too complicated, and invited distraction. Every time I wanted to add a site to the list, I would have to go to the existing list, and see all my personal vices there (Oh yeah! I haven't checked Hacker News today!)

And so I discovered Mindful Browsing. It works in a subtly different way. It's designed to encourage you to consider your decisions, rather than be put off by an admonishment page every time you try to access a distracting site.

It works like this. The extension defaults to always being in effect, though this can be modified. Then, while you're browsing, when you find yourself on a site that's distracting, you press a button. The site is added to a blacklist that you never see. That's it.

What makes it interesting is this: When you try and access a distracting website, you get a warning, and the option to temporarily allow it. This is fantastic in and of itself (sometimes I need to check a distracting forum which also contains useful news and information), but there's another variable: the button to enable the site temporarily is disabled for a configurable amount of time. I set mine to ten minutes.

So if I want to visit one of the distracting websites, I have ten minutes to either go do something useful and come back, or change my mind and close the tab. This is the eponymous 'mindful browsing': the ability to actively decide how you want to approach distracting websites, and the ability to give yourself time to set other goals.

2012-04-15 04:00

2012-04-04

2012 04 04

RTB: Where Erlang BLOOMs

I'm still on my way back from the Bay Area Erlang Factory 2012 (in fact I finished this post after coming back entirely), in San Francisco, and I've been starting to see a bunch of twitter posts regarding people posting their slides online. Although it is a good idea, I tend to view my slides as pure visual support, and the animations don't show up very well in a static PDF, so I thought I'd do a quick transcript of the main ideas that were at work in my talk, Real Time Bidding: Where Erlang BLOOMs, as brought to you by BLOOM Digital Platforms.

Types of advertising

When advertisers want to buy advertisement, they can do it at large on some user group and hope to get decent results, or participate in Real Time Bidding (RTB). In RTB, advertisers bid on individual advertisement spaces to obtain more targeted audiences. This is usually built on the backbone of large exchanges where bidders require high performance, short response times, and a high volume.

More precisely in the context of the Internet, the traditional way to do things is called display advertising. It's the general practice of having website owners contacting some advertisers or an adgency that just pushes advertisements on the said website. This can be done broadly, where you have deals for a dealer to send a bunch of ads that always change on your site, or in more precise cases, where you have deals with specific advertisers. One example of the latter is Microsoft, who before Windows Vista came out paid some people up to CDN $80k per week to have some websites change their background to something pro-Vista.

Real-Time Bidding (RTB) is a kind of evolution on top of display advertising. It is a bunch of loosely defined protocols employed by nearly as many exchanges, who allow that, instead of dealing with a single distributor that has a limited inventory of ads, anybody can try himself on buying advertisements in soft real-time. Here's what an exchange looks like:

So let's say I'm a car dealer. I've got a website. From the user perspective, whenever they visit my page, they see a little snippet of HTML or Javascript that contacts a server:

Optionally, the display advertiser places a tracking cookie on the site, so they're able to do things like calculate various conversion rates, or just make sure not to show the same thing to the same browser too often.

When it comes to RTB, things begin the same way, but once the request for an ad makes it to the exchange, that exchanges sends something called a bid request to a bunch of bidders. These bidders all represent one or more advertisers, or can themselves be exchanges that aggregate requests from many exchanges and forwards them to bidders. The bid request will often contain information such as the kind of site the user is visiting, and other general information such as geographical location, age, etc. if it's available. In any case, the bidders (in the case of ours at BLOOM, it's called adgear trader) try to decide, with the help of such data, whether or not they want to advertise to that person.

When a given bidders bids, it does it by sending a snippet of HTML or Javascript (depending on the exchange). The winning bidder then has its snippet displayed on the page by the exchange (which can update its own cookies at the same time) and then pushed on the page. At that point, the bidder can then set its own cookies and do its own thinkering:

There is also generally the practice of cookie matching which helps share anonymized data between different actors, making the general decision process better. In our case, it's done in C, directly in an apache module.

In any case, there are types of campaigns made especially interesting by RTB. Say I'm still a car dealer with a website, but now I book a deal with a given bidder so that I can also advertise online. A given user visited my site with that bidder and thus got cookied (or was tagged through some deals). When that user visits another car dealer's website, expecially if it's in the local area, I now have a chance to put advertisements on that site. I can thus bid more money on the auction in the hope of winning it. Then if I win it, I can just finally display my ads, update my cookies, and so on:

And ideally, we bring the browser back to our shop!

The Gateway

The product we're working on related to real-time bidding and Erlang at BLOOM is the AdGear trader, which sits in between a bunch of bidders, and a bunch of exchanges:

The gateway is running on (so far) 6 servers, with 16 to 24 cores and 16GB of RAM (although each instance of the gateway usually takes less than 5% of that. They're all running Linux Gentoo (I think you mean GNU/Linux), and we make use of Cassandra and Redis in terms of databases. Each gateway instance gets thousands of bid requests each second, usually between 4,000 and 9,000 depending on the period of the day. Most of these requests are then forwarded to many bidders at once, received back, and a reply gets sent — the gateway acts a bit like a small exchange aggregator.

Latency is pretty important, given that we usually get around 35 milliseconds to handle each request, although that value tends to grow with more recent developments. This includes delays for parsing the bid requests, doing some searches in the database(s), serializing a bid requests and sending them to clients, waiting for a response, parsing said response, picking the best bid of all our clients, encoding it in the exchange's format, and pushing it back there. Ideally, we want to do all of this without skipping a beat, and because we also handle things like serving a javascript tag that tells us when we won a bid, we need to have as little downtime as possible (ideally none). This is because knowing we won a bid lets us increment the allocated budget for some campaign and that way we make sure we're not spending a trillion dollar.

We've got to Scale!

Well of course, scaling is nice, but the first thing we have to do is measure, measure, and measure some more. Our tools of the trade there are Graphite, StatsD (a node.js frontend to graphite), statsderl (client for StatsD), a bunch of manual calls to statsderl, and vmstats, which pushes an Erlang VM's details to StatsD.

What we obtain with this is a series of graphs that we can customize and make look a bit like this:

This reveals the first problems we had. On the left hand side of the graph are all these large green peaks and red bumps. Those are caused by the ibrowse HTTP client library. It's the first one we used to have running: it has a nice load-balancing feature and supports an asynchronous mode. The problem with it was that from time to time, the load-balancer would lock-up by forgetting to release resources. This would cause the green peaks. When that'd happen, all requests would stop being forwarded to bidders.

The red peaks were the consequence of us basically restarting ibrowse every minute or so. The thing is we were short on time to fix ibrowse, so a suboptimal, temporary solution was taken. Eventually, we had the time to fix it. We picked lhttpc as our library of choice (it also supported asynchronous requests), and then we made our own load balancer for it. When we deployed it, you saw what happened on the right-hand side of the graph: peaceful functionality.

The load-balancer we wrote for that one was based on the idea that we have one load-balancer per domain, which is to say one per bidder. The load balancer keeps a list of N sockets always active. Requests coming in then just send a message, ask for a socket, and do their thing before returning it:

This makes sense for a few reason. First, we want to keep sockets always open because setting up new connections is mighty expensive. If it takes 7 or 8 milliseconds to get one going, this could be 25% of our processing time for each bid request. Keeping sockets open is a no-brainer. Next, there's the idea of having one load-balancer per domain. This is necessary for a few simple reasons. If there is one bidder that twice slower than any other one, it is unfair for that one to hog all the sockets available. Moreover, it is imperative to be able to divide the requests between many processes.

Let's say we made the bad decision of having one load-balancer for the whole node, and I have three bidders. For each bid request I have, I might then send 3 requests for a socket. If we have an average 8,000 bid requests per second, this rapidly brings us to 24,000 messages per second to a single process. There is no way the VM can sustain that for long periods of times. If we instead choose to divide the requests between three load-balancers, then we have a possible peak of 8,000 messages per second to a single process. That's still a lot (even too much), but it's far nicer than the multiplicative result we get with a single load-balancer for many bidders.

The Error L(a|og)ger

At some point in time this winter, we started having the following graphs showing up, with alarms flowing through our e-mail boxes in the middle of the night:

Plenty of people have seen this before. There is some chain of errors that start happening, and all of a sudden, the error_logger queue is growing out of hand and the VM dies by going out of memory (OOM). This is actually worse than having no logging at all. It crashes so fast we can't know what the error is and the VM cannot recover.

If you had that problem before, you picked one of two options: you turned logging off or used an alternative logging library. We picked the latter and chose to use Andrew Thompson's lager. And very quickly, the results were visible:

This solved the problem right away. Although the logs were very large and cycling so fast we were often missing the original source of the issue, the VM could stand its ground and tolerate all kinds of crashes. It turned out that our problem was that one exchange would often send us misshaped packets with erroneous content-length values. This crashed the outdated version of Cowboy we were using; upgrading allowed us to deal with it correctly. Since then, the VM managed to become fairly stable.

Scaling the operation

One thing we did was writing our own bidder, running locally on the gateway nodes. There isn't too much to say about it as for now it's fairly basic. One thing it did, though, was force us to think about configuration a whole lot more. One of the common problems of RTB configuration is that you want it to be as static as possible in order to optimize it, but to be able to change it as fast as possible in order to correct error (someone saying they want to spend millions instead of thousands). The solution we picked was to have one central point of origin for all config data:

The pink node in the center is the controller. It takes two roles. One as a web service to external bidders who want more information on some bids, and one as an application that pushes flat BERT files on all of the gateway nodes. The BERT files are then loaded up by bertconf and transformed into local ETS tables. Bertconf then looks at the files for changes at a regular interval, and when it detects anything new, it reloads the new file, makes a new table, and swaps it with the existing one using some trickery to insure there is never any request failing (hint: it works a bit like module versioning in the VM with purges).

This option is somewhat safe because it only depends on the central node to push updates, not to read them. As long as there is a local bert file, we can reload it in memory even after crashes. It's also stupidly easy to override a file on a single node or to manually replace the central service if we need to. Another nice aspect is that because we will read the configuration on pretty much every bid request ever, it's great to be able to avoid going outside of the VM to fetch the values we need. If we can stay local, we will. FOR SPEED! Maybe in a year or so we will have so much configuration we won't be able to hold it all in memory, but in the meantime, bertconf appears to be a good option.

Pedal to the Metal

The config flexible and fast, a basic bidder in place, HTTP libraries stable, and exchanges integrated. All we needed to do is open the gates to the system. And then it hit us. Cassandra suffered.

The problems with it were that cassanderl's pooling system wasn't especially robust, and that Cassandra itself had to do a lot of requests which would sometimes be really long (insert comment about Java GC):

The problem here is the brown line, the timeouts. this shows that out of 4,000 requests per second, we could reach up to 25% of them timing out. This is pretty bad. Here's an explanation why that's possible.

Queues Suck (for this)

Most pool systems are based on a queue (usually the message queue) of different requests to execute. Things usually work fine if we are able to process data as fast, or faster than it enters the queue:

However, a problem happens when elements join the queue faster than they leave it:

When this happens, requests further down the queue are bound to have less and less time to be processed off the initial 35 milliseconds. In fact, at some point in time, all requests are going to be stale until we throw them all away, and start the cycle over and over again. This is actually what happens in production for us, given that we are always getting thousands of requests in the queue each second, but might have only 200 connections open to a database. We are in constant overload, and we're going to see bumps all the time where we sometimes serve one request with a lot of time left, and then serve many with only crumbs of their initial allowed time left, most of it spent idling for no good reason.

Stacks Are Nice (for this)

Let's imagine that instead of using a queue, we're using a stack as a mailbox. Then even in situations of overload, we're always going to pick the newest element on top of the stack:

The stack model works fairly well in cases where you have bursts of overload and can then catch up later. When you're done with all the fresh queries, you'll hit a batch of stale ones that you can try to quickly discard and go through, saying "I won't bid on that" to the exchange. This is awesome, right? It isn't. Remember, for our gateway, we're in a situation of constant overload. This mean we likely will never hit a stale item in the first place, and so we would need to use some kind of bucket-based approach or have a garbage collector specialized enough to throw away old requests. This isn't good enough.

Nothing is Better

By 'nothing is better', I mean nothing as in 'no stack', or rather, a stack with an upper limit of one element. Here's the idea. Whenever we're handling one element, we won't be accepting anything else. We don't care about serving an item that has to wait because it means we're going to have less time to do it. And because we're always in overload, we can pretty much guarantee that there will be an element to take the place of the one that just finished right away. There's just so much stuff coming in there is no reason for any queue to build up. All we have to do is say to the requests that can't make it to the 1-element-stack "sorry guy, I guess you won't bid on this one".

That's the important idea. We don't actually have to serve all the requests. Those that don't fit right away, we can just answer back with "sorry I'm not gonna bid" to the exchange and wait for more. We will in fact get much more throughput that way than if we were to try to do a half-assed job at answering queries with nearly no time left. We're just doing the equivalent of a preemptive time out. We'll get better throughput, yes, but we're also guaranteeing that we will also maximize the time of all the requests making it through to the bidders. That's useful.

Dispcount

Dispcount is a library we built at BLOOM to harness the idea above. We prefer to answer with a no-bid response than not responding on time, or even to losing time on a request given new ones will take their place instantly. It actually works without using message passing as a way to bypass the mailbox (a queue, oh no!), and we've integrated it into cassanderl.

Dispcount pushes the idea a bit further by trying to avoid all central points of communication — a play on single points of failure, where we want to avoid having everyone wanting the same chunk of data. It starts N resource watchers, and works on the idea of "you know what you have to do". That is to say that any request done doesn't have to wait on some external information to know which resource watcher to contact. It just knows. In practice, we do this by using some hash of unique data (the pid and a timestamp) and only then it uses a lookup table:

In the diagram above, we have 5 watchers (teal, purple, pink, orange, and black). The request gets hashed to the value 5 (orange) and uses ets:update_counter/3 to increment the value atomically. If the value returned after the incrementation is 1, then you know you have access to the resource. If it's anything else, you don't. In this case we don't, because the value returned is going to be 75.

In the image above, some request got hashed to the purple color. The counter update returned the value 1, and so that request has gained the right to the table containing pids. The resource can now be requested from that pid with a single message.

There are a few things to explain here. First of all, the distribution of requests is non-deterministic. It's entirely possible that a system under no load and with a thousand resources available gets a few busy messages with only 3 requests. However, under heavy load (which is our case, all the time), statistics pretty much guarantee us that we're going to get full usage of all the resources. Whenever the resource gets checked back in, the counter is reset to 0 and a new request can obtain it.

Secondly, I feel I should explain why I use two ETS tables and why I use counters like that. The reason is actually very simple. ETS tables, when switching between read and write modes, are relatively slow. On my own laptop, I benched about 500 operations per second on a table when mixing reads and writes. When staying only in read xor write mode, I couldn't hit a limit there first — memory or the schedulers working hard on all the processes would be a problem before that. By splitting the table in two, one in read mode and one in write mode, I avoid the switching. By using ets:update_counter/3, I can actually both read and write the data as a single write operation, tricking the VM into giving me more bang for my buck.

I would also not recommend the use of dispcount to anyone who is not allowed to do things like avoiding to process some requests or actually isn't in a very frequent situation of overload. The repository explains the conditions and assumptions that were made when designing dispcount in detail.

So what were the results?

This shows the impact of dispcount in production right after a (somewhat painful) first deployment. What's interesting is to see all the timeouts (green bumps) getting squashed after the deployment. They are not replaced; in fact, you can see that only a minimal amount of requests need to be discarded (the brown line). By making the initially non-intuitive call of dropping a few requests, we in fact ended up serving a lot more of them. This is due mainly to the fact we never accumulate requests, and thus do not end up busting all their timers in a losing race. Dropping requests in fact lets us serve a lot more!

This graph shows the run queue during that same period of time. Having all these requests building up in mailboxes had the Erlang VM working very hard to try and go through all of them, including garbage collection. Using dispcount removed all of these issues pretty much instantly.

Further Optimizations

One thing we've spent a lot of time on is trying to reduce all the work done in encoding and decoding bid requests and responses. All the JSON, protobuffs, and even just HTTP requests are making our gateway dangerously close to being a CPU-bound application.

One thing we tried was changing all the encoding of outgoing JSON from NIF libraries like ejson to using iolists. Our data shows very little variation, and in theory, it was promising. Our first benchmarks instead revealed that there was very little gain to be made, maybe 5-10% at best. The thing, though, is that these benchmarks were sequential — encoding random copies of our JSON structures in a tight loop. When we started measuring things by using as little as a hundred and fifty concurrent processes, suddenly the changes were much more noticeable. Iolists would now be two to three times faster than ejson. What we're suspecting is that some NIFs might tend to behave differently in parallel than concurrent setups. Sequential benchmarks are not always representative of reality, especially when the reality isn't purely sequential.

We've in fact started what could be seen as a witch hunt to all non-essential NIFs in our system. They caused us other problems, such as crashing the VM when doing cowboy-style hot-code loading (we actually do not need relups as they're slower and our requests are short-lived to a point it's safer to hot-reload without protection than locking everything). We currently just skip reloading them, but doing away with them wouldn't sadden us. Note that other systems with constraints different than ours might embrace NIFs. Different requirements yield different optimal solutions.

Finally, what we want to keep on doing is decentralizing more. Single points of failures and single points of communications are all annoying things that limit how much we can push our gateways.

Thank you for reading this talk. Any questions?

2012-04-04 18:00

2012-03-22

As the first step in my new game project The Deadlings, I've put together a little starter project that combines the Farseer 3.1 physics library with the GameStateManagement sample from Microsoft, all built on XNA 4.0.  You can download it at thedeadlings.com.

2012-03-22 16:31

TouchArcade wrote a wonderful review of our game Rogue Runner today. After 2 months and 5 updates, it's wonderful to see our little game take off.

2012-03-22 16:31

I am now offering my services as a freelance iOS developer. Need an app developed for iPhone, iPod Touch or iPad?  Get in touch with me via Glowdot Productions, Inc. You can hire a guy who's had apps in almost every chart category on the app store, been in the top 50 ...

2012-03-22 16:31

2012-03-04

Programming music is annoying. The question of how to combine ideas of notes and instruments and waveforms into a song is probably more interesting than the art of concatenating strings into an HTML document. At first I thought I’d make things by combining arrays of samples using some array-of-samples-combiners and objects with this interface:

struct Music {
    // dumps [beg, end) of our sample region to out.
    virtual void Dump(vector<pair<int16_t, int16_t> >& out,
                      int32_t beg, int32_t end) = 0;
    // the length of this musical passage, in 1/44100ths of a second.
    virtual int32_t Length() = 0;
};

The problem with that is you can’t just tweak the shape of some sound without e.g. knowing how it’s normalized and such. Also it’s not the right toolkit for playing a note out loud.

Also it’s stereo. Do we want to support mono tracks?

So now I’m going with having some primitive sounds that have amplitude 1.0, frequency 1.0, and manipulators that can screw around with them, and then things can parameterize them into notes somehow. More details will be present within the next 13 hours.

by shrughes at 2012-03-04 11:54

2012-02-13

IMG 0239

A letter to my love, my friend, my wife and my partner — Dusty:

I know it’s the day before Valentines — some things can’t wait just for a day.

Ten years — that’s how long we’ve been with one another. Ten years feels like a lifetime — so much has changed — our lives altered in subtle — and not so subtle ways by the gentle currents of each other. In the time I’ve known you, we have both changed for the better — we compliment and act as one another’s confidant, friend, partner and lovers.

“The most powerful symptom of love is a tenderness which becomes at times almost insupportable.” — Victor Hugo

 

Wedding 451 copy

We’ve been through our times of trial — little things like accidentally renting an apartment in a war zone (my bad!) — and much bigger things from health, to finances, to not know what we were doing or where we were going. We both know that this past year has been probably the one most filled with trials and tribulations.

We’ve sat across from one another not knowing what we were going to do, we’ve held each others hands watching our infant daughter laying in a hospital bed — I’ve held your hand at your bedside in watching your pain and not knowing what to do about it, except to sit there and watch your pain. We’ve been through a lot in ten years.

Despite the trials — we have made each other stronger. You have changed who I am in such fundamental and subtle ways, that I attribute much of who I am now, to you. You have made me happier, stronger, more empathetic — you have also given me the cherished gift of your love, your tears and support in my times of pain.

You have given me more than just your love; you gave me our first daughter Abby — who might as well be a tiny clone of myself in female form (god help us all), who despite her willfulness and strong personality makes my heart jump each time I hear her laugh, each time she runs to me and hugs me and tell me she loves me.

IMG 3690

Abby is almost five! Five years old! All parents gush about how smart their children are — but we both know there’s something special and unique about her. There’s more to her than a pushy 4.5 year old, there’s something magical about her that we both see. I can not verbalize or put to words my thanks to you for her. She’s a gift you’ve given to me.

Then there is Addison, our bubbling eight month old. What can I say about someone who greats me with a smile and a laugh whether it’s five in the morning, or me just coming home from a hard day at work?

Addison is more than a gift; she’s a blessing — the past year shows that even in our darkest hours, sitting there in a hospital not knowing what will happen, something watches over us. Addison’s happiness and flourishing is not just due to doctors, or therapists — it’s directly tied to the amazing love and care you provide to her.

Every time I look at Addison, I see an extension of you — your smile, your happiness (and when she giggles when she rams me with her walker, your sense of humor). Addison is again, a gift and blessing you’ve given me.

IMG 4457

You’ve given me so much; you’ve changed me so much. You’ve made me look outside of myself and think of others — you, our daughters, you’ve driven me to try to change the world and help as many people as I can. You’ve driven me to be better — a better man, a better husband, father and human.

Times change — people change. We have our hard times — we have those times when we both want to go lock ourselves in the bathroom just to get a moment of quiet. We have times when we just don’t know what will come, and times when we wish what had came had not. We have persevered over the hard times we’ve faced until now, and those hard times we face now, we face together, as one.

You are beautiful — you always have been, you are strong — you are honest and critical. I might say half-jokingly that you’re my better half some times — but you really and truly are (You are also better looking than me!).

Wedding 144

You, and the gifts have given me — our daughters, have given me more than a reason to just keep working, just to keep moving from day to day. You’ve given me a reason to truly live, to truly push myself beyond anything I could have imagined eleven years ago. You’ve given me a place and arms to cry in, to laugh in, and to grow in. You’ve given me a view of life, of living, of loving I never dreamed of having.

I know that once again we face hard times. I thought that perhaps this year might be a little easier on us — but so far, we both know it isn’t, and there are probably harder times coming for us. I am sorry that I can not always give to you all the things you so richly deserve — I’d give you anything, I’d buy you anything if I could. I am sorry I don’t have anything I can give you today other than my words — darn those hard times!

My gift to you is this — my expression of how much I truly value you, cherish you and how grateful I am — in spite of all the hard times — the good times, the memories, our daughters and most importantly our love. I am but a broken man, but with you I am whole.

Thank you for being who you are.

Thank you for being with me.

Thank you for loving me.

Thank you for letting me love you in return.

Jesse

p.s. Churchill loves you too:

IMG 4461

flattr this!

by jesse at 2012-02-13 17:32

2012-01-13

2012 01 13

Pierre Gauthier

Unless you're following NHL hockey, you'll want to skip on this issue.

If you follow NHL hockey, you likely know by now that Cammalleri was pulled between the second and third period of the Montreal Vs. Boston game yesterday, to be traded for René Bourque in Calgary (and a few other players and draft picks). I woke up with this idea this morning, with Pierre Gauthier (Montreal's GM) in mind:

A recreation of the 'boo-urns' scene from the Simpsons with 'boo-rque'

2012-01-13 22:30

2012-01-10

2012 01 10

Obfuscation

Late Promises

For a long while now, I've showed various people what I consider to be one of the most twisted bits of obfuscated Erlang to have been written. I've had it for a bit over a year, and showed it to a bunch of Erlang people back at the Erlang User Conference in Sweden back in 2011.

I then promised to write a blog post about it and explaining how I wrote it, which I promptly deferred to later in one brave act of procrastination. I decided to finally put the whole thing together, between the writing of a new Learn You Some Erlang chapter, the formatting of an older one for a publisher, and work.

The code you'll see below is something I wrote when I was still working for Erlang Solutions Ltd. (I now work for a place called BLOOM Digital Platforms, still in Erlang). When working for ESL, I would spend quite some time traveling around the world to go teach courses about Erlang to different people and businesses. This did include its share of long airport waiting times, long pauses in hotels without WiFi access, and bus rides. That's where I took the time and fun challenge to write the messiest Erlang code possible, using features mostly unique to the language.

What does it look like?

I'll be quick—it looks like this (alternatively, a gist!):

                                          %%%%%%%                      %%%%%%
-module (        obfuscation            %%       %%%               %%%       %%
)                                           %%%%%%\ %             % /%%%%%%
. -compile(                                 %%   %%       ' '       %%   %%
[export_all  ] ) . -define(swap          %  %%%%%%%        |        %%%%%%%  %
(A,                                       %%%              .              %%%
B), B                                                      %
A                                                       %%   %%
). -define(M,                                                                                    ?MODULE). -define(v, 
o).            -define(p,().                           %%%%%%%%%%
                             -define(q,           %%%%%  ,,__,,  %%%%
)).                                               %%%    |    |   %%%
     -define(s                                    %%     \ _ / %   %%
,,).               -define('.   ',             {           %    %
                  ).                                      %%%    %
                                -define(' .  ',        }     ). %%%%%%%%%%%%%%%%
-define(      ' \''                                             % LL  OOO LLL  %
            ,                                                   % L L  O  L    %
               integer).                                        % L L  O  LLL  %
                -define(':('                                    % L L  O  L    %
,                                                               % LL  OOO LLL  %
 atom                                                           %%%%%%%%%%%%%%%%
,
)
.                                         -define(
':)'                       ,       tuple,     ). -define('}',
[
).            -define('{'    ,

]                                                 )     .

%% According to many Haskellers, type signatures represent sufficient
%% documentation. I have thus provided the type signature to the main
%% function of this program, hoping its self-describing nature and the
%% types will be enough to enlighten the readers of this code as to its
%% purpose and inner workings.
%%                                               Best Regards,
%%                                               - Ferd.
-spec main() -> ''.
main?p  ?q  -> main?p atom ?q . main?p Bin ?q  ->
    '. _   '?p Bin ?q  ?s S= ?p ?p fun '! _/?  - '/1 ?q ?p ?p ?p fun '_ _/-  - '/1 
    ?q ?p ?p ?p fun '! _/?  - '/1 ?q ?p ?p ?p fun '_ _/-  _ '/1 ?q ?p ?p ?p fun
    '_ _/-  - '/1 ?q ?p ?p ?p fun '! _/?  - '/1 ?q ?p ?p ?p fun '_ _/-  _ '/1 ?q ?p ?p ?p
    fun '! _/-  - '/1 ?q ?p ?M:'_  ='?p ?M:'=  _'?p ?M:'_  ='?p ".Ub-ry rWby" ?q
    ?q  ?q  ?q ?q ?p ?p ?p fun '! _/-  - '/1 ?q ?p  ?'}' $\n,$\n ?'{'  ?q  ?q ?p ?M:'_  ='?p
    ?M:'=  _'?p "Ug-.y"?swap("r","r")"ebbyoE " ?q  ?q  ?q  ?q  ?q ?q ?p
     ?'}' $\n,$\n ?'{'  ?q  ?q ?q ?p ?p ?p fun '! _/-  - '/1 ?q ?p ?M:'=  _'?p ?M:'_  ='?p
    ". agrbu "?swap("s","g")" vtkh og ur" ?q  ?q  ?q  ?q ?p ?M:'_  ='?p
    ?M:'=  _'?p "Vq-tfr arnrzgn hn ul bg" ?q  ?q  ?q  ?q  ?q ?q ?p  ?'}' $\n,$\n ?'{' 
    ?q  ?q ?q ?p ?M:'=  _'?p ?M:'=  _'?p ?M:'=  _'?p "-yvf.,syLnarl " ?q  ?q
    ?q  ?q  ?q ?q ?p   ?'}' $\n,$\n ?'{'  ?q  ?q ?q ?p ?M:'=  _'?p ?M:'_  ='?p "**xppyv"
    ?q  ?q  ?q  ?q ?q ?p  ?'}' $\n ?'{'  ?q  ?s group_leader?p  ?q  !  ?'.   ' io_request, self?p
    ?q , '$',  ?'.   ' put_chars, unicode, S ?' .  '  ?' .  '  ?s ''.

'! _/?  - '?p A ?q  -> fun?p B ?q  -> ?p fun erlang:'++'/2 ?q ?p A,B ?q  end. '_ _/-  - '
?p A ?q  -> '! _/?  - '?p A ?q . '_ _/-  _ '?p A ?q  -> '! _/?  - '?p A ?q . '! _/-  - '?p B ?q
-> fun?p A ?q  -> ?p ?p ?p fun '! _/?  - '/1 ?q ?p A ?q  ?q ?q ?p B ?q  end.

%%%%%%%%%%%%%%%%%
%%%%%       %%%%%   %%%%%%%%%%%%%%%  %%    %%  %%%%%%%%   %%%%%%
%%%%%       %%%%%   %%%%%%%%%%%%%%%  %%    %%     %%     %%
%%%%%       %%%%%         % %        %% %% %%     %%      %%%%%
%%%%%       %%%%%         % %        %% %% %%     %%       %%%%
%               %         % %        %%    %%     %%          %%
%%%           %%%         % %        %%    %%  %%%%%%%%  %%%%%%
%%%%%       %%%%%
%%%%%%%   %%%%%%%           - WAY -
%%%%%%%% %%%%%%%%
%%%%%%%%%%%%%%%%%                                       [skip to the end for infos]





-define('             ,', ?'.   '  ?' \'' ,0,303 ?' .  '  ?s ?'.   ' cons,0 ?s
?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,303
?' .  ' ,    ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s          ?'.   '
nil,0 ?' .  '  ?' .  '  ?s  ?'.   ' cons,0 ?s ?'.   ' cons,0 ?s ?'.   '  ?':)'
 0 ?s  ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s ?'.   '  ?' \'' ,0,303 ?' .  '  ?s
?'.   '  ?':(' 0,'andalso'
?' .  '  ?s ?'.   '  ?':)'
0 ?s  ?'}'  ?'.   '  ?':('
0,op ?' .  '  ?s   ?'.   '
  ?' \'' ,0,303 ?' .  '  ?s
?'.   '  ?':(' 0,'>=' ?' .  '  ?s
?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  '
, ?'.   '  ?' \'' ,0,303 ?' .  ' , ?'.   '  ?':(' 0,'C'
 ?' .  '  ?'{'  ?' .  '  ?s ?'.   '  ?':)' 0 ?s    ?'}'
?'.   '  ?':(' 0,char ?' .  ' , ?'.   '  ?' \'' ,0,303
?' .  ' , ?'.   '  ?' \'' ,0,65 ?' .  '  ?'{'  ?' .  '
  ?'{'  ?' .  '  ?s ?'.   '  ?':)' 0 ?s ?'}'  ?'.   '
  ?':(' 0,op ?' .  '  ?s
?'.   '  ?' \'' ,0, 303 
?' .  '  ?s ?'.   '  ?':('
 0,'=<' ?' .  '  ?s ?'.   '
  ?':)' 0,   ?'}'  ?'.   '
  ?':(' 0,var ?' .  '   ,
 ?'.   '  ?' \'' ,0,   303
?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s ?'.   '  ?':)' 0 ?s
?'}'  ?'.   '  ?':(' 0,char ?' .  ' , ?'.   '  ?' \'' ,0,303   ?' .  '   ,
?'.   '  ?' \'' ,0,77 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
  ?'.   ' nil,0 ?' .  '  ?' .  '  ?s ?'.   ' cons,0    ?s   ?'.   ' cons,0 ?s
?'.   '  ?':)' 0 ?s  ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s ?'.   '  ?' \'' ,0,


303 ?' .  '  ?s  ?'.   '  ?':(' 0,'andalso'  ?' .  '  ?s 
?'.   '  ?':)' 0 ?s  ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s ?'.   '
?' \'' ,0,303 ?' .  '  ?s ?'.   '  ?':(' 0,'>=' ?' .  '  ?s ?'.   '  ?':)'
0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,303 ?' .  ' ,
?'.   '  ?':(' 0,'C'                            ?' .  '  ?'{'  ?' .  '
 ?s ?'.   '  ?':)' 0                              ?s ?'}'  ?'.   '  ?':(' 0,
char ?' .  ' , ?'.   '                           ?' \'' ,0,   303   ?' .  ' ,
?'.   '  ?' \'' ,0,97                            ?' .  '  ?'{'  ?' .  '  ?'{'
?' .  '  ?s   ?'.   '                          ?':)' 0 ?s ?'}'  ?'.   '
?':(' 0,op ?' .  '  ?s                   ?'.   '   ?' \'' , 0,  303
 ?' .  '  ?s  ?'.   '  ?':(' 0,'=<' ?' .  '  ?s
 ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  '
 , ?'.   '  ?' \'' ,0              ,303 ?' .  ' , ?'.   '
 ?':(' 0,'C' ?' .  '                 ?'{'  ?' .  '  ?s ?'.   '
?':)' 0   ?s     ?'}'                 ?'.   '  ?':(' 0,    char
?' .  '   ,   ?'.   '                  ?' \'' ,0,303 ?' .  '   ,
 ?'.   '  ?' \'' , 0,                  109 ?' .  '  ?'{'  ?' .  '
?'{'  ?' .  '  ?'{'                     ?' .  '  ?s ?'.   ' nil,0
 ?' .  '  ?' .  '  ?s                   ?'.   ' nil,   0 ?' .  '
?' .  '  ?' .  '  ?s                    ?'.   ' cons,0 ?s ?'.   '
 ?':)'    0 ?s ?'}'                     ?'.   '  ?':(' 0,op ?' .  '
?s ?'.   '  ?' \'' ,0                   ,303 ?' .  '  ?s ?'.   ' ?':('      0,
'+' ?' .  ' ?s ?'.   '                  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,  var
?' .  '   ,   ?'.   '                   ?' \'' ,0,303 ?' .  ' , ?'.   '  ?':('
 0,'C' ?' .  '  ?'{'  ?' .  '        ?s ?'.   '  ?':)' 0 ?s ?'}'  ?'.   '


?':(' 0, ?' \''  ?' .  ' , ?'.   '
  ?' \'' ,0,303 ?' .  ' ,
  ?'.   '  ?' \'' ,0,   13
  ?' .  '  ?'{'  ?' .  '
  ?'{'  ?' .  '  ?s ?'.   '
  nil,0 ?' .  '  ?' .  '
  ?'{'  ?' .  '  ?s ?'.   '
  cons,0 ?s ?'.   '  ?':)'
   0 ?s ?'}'  ?'.   '  ?':('
  0,clause ?' .  '  ?s ?'.   '
  ?' \'' ,0,304 ?' .  '  ?s
  ?'.   ' cons,0 ?s ?'.   '
  ?':)' 0   ,  ?'}'  ?'.   '
  ?':(' 0  ,  var ?' .  '   ,
  ?'.   '  ?' \'' ,0,  304
   ?' .  ' , ?'.   '  ?':(' 0
 ,'C' ?' .  '  ?'{'    ?' .  '
  ?s   ?'.   ' nil,0 ?' .  '
  ?' .  '  ?s ?'.   ' cons,0
  ?s ?'.   ' cons,0 ?s ?'.   '
  ?':)' 0   ?s   ?'}'  ?'.   '
  ?':(' 0,op ?' .  '  ?s ).
   -define('              ,',
  ?'.   '  ?':(' 0,'  ' ?' .  '
 ?s      ?'.   '  ?' \'' ,0,2 ?' .  '                          ?s
?'.   ' cons,0 ?s ?'.   '  ?':)' 0 ?s ?'}'                  ?'.   '
?':(' 0,clause ?' .  '  ?s ?'.   '  ?' \'' ,0,88 ?' .  '  ?s ?'.   '
 cons,0 ?s ?'.   '  ?':)' 0, ?'}'  ?'.   '   ?':(' 0  ,   nil ?' .  '
 , ?'.   '  ?' \'' ,0,88 ?' .  '  ?'{'  ?' .  '  ?s ?'.   ' cons,0 ?s
?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \''
,0,88 ?' .  ' , ?'.   '  ?':(' 0,'Acc' ?' .  '  ?'{'  ?' .  '  ?s ?'.   '


                             nil,0 ?' .  '
                     ?' .  '  ?' .  '  ?s  ?'.   '
                 nil,0 ?' .  '  ?s  ?'.   ' cons,0 ?s
           ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0  , var
         ?' .  ' , ?'.   '  ?' \''    , 0,88 ?' .  ' , ?'.   '
       ?':(' 0,'Acc' ?' .  ' ?'{'      ?' .  '  ?s  ?'.   ' nil,
     0 ?' .  '  ?' .  '  ?'{'            ?' .  '  ?s ?'.   ' cons,0 ?s
   ?'.   '  ?':)' 0 ?s ?'}'                ?'.   '  ?':(' 0,clause ?' .  '
  ?s ?'.   '  ?' \'' ,0,89                  ?' .  '  ?s ?'.   ' cons,0 ?s
  ?'.   '  ?':)' 0 ?s ?'}'                   ?'.   '  ?':(' 0,cons ?' .  '
  ?s ?'.   '  ?' \'' ,0,89                   ?' .  '  ?s  ?'.   '  ?':)' 0,
  ?'}'  ?'.   '  ?':(' 0 ,                    var ?' .  ' , ?'.   '  ?' \''
  ,0,89 ?' .  ' , ?'.   '                    ?':(' 0,'X' ?' .  '  ?'{'  
  ?' .  '  ?s ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,nil ?' .  ' , ?'.   '
  ?' \'' ,0,89 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s  ?'.   ' cons,0 ?s
  ?'.   '  ?':)' 0 ?s  ?'}'  ?'.   '  ?':(' 0,tuple ?' .  '  ?s    ?'.   '
  ?' \'' ,0,89 ?' .  '  ?s ?'.   ' cons,0 ?s  ?'.   '  ?':)'    0,    ?'}'
  ?'.   '  ?':(' 0, var                      ?' .  ' ,    ?'.   '  ?' \'' ,
  0,89 ?' .  ' , ?'.   '                     ?':(' 0,'As' ?' .  '   ?'{'  
  ?' .  '  ?s ?'.   ' cons                   ,0 ?s ?'.   '  ?':)' 0, ?'}'
  ?'.   '  ?':(' 0  , var                    ?' .  ' , ?'.   '  ?' \'' ,0,
  89 ?' .  '     , ?'.   '                   ?':(' 0,'Bs' ?' .  '    ?'{'
  ?' .  '  ?s ?'.   ' nil,                   0 ?' .  '  ?' .  '  ?' .  '
  ?'{'  ?' .  '  ?s ?'.   '                  nil,0 ?' .  '  ?' .  '  ?' .  '
  ?s ?'.   ' nil,0 ?' .  '                   ?s ?'.   ' cons,0 ?s ?'.   '
   ?':)' 0 ?s ?'}'  ?'.   '                    ?':(' 0,tuple ?' .  '  ?s 
  ?'.   '  ?' \'' ,  0 , 89                  ?' .  '  ?s ?'.   '   cons,0        ?s
  ?'.   '  ?':)' 0,    ?'}'                  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '
  ?' \'' ,0,89 ?' .  '   ,                   ?'.   '  ?':(' 0,'As' ?' .  '  ?'{'
  ?' .  '  ?s ?'.   ' cons,                  0 ?s  ?'.   '  ?':)' 0 ?s  ?'}'
 


?'.   '  ?':(' 0,cons ?' .  '                          ?s ?'.   '  ?' \'' ,0,89
?' .  '  ?s ?'.   '  ?':)' 0, ?'}'                     ?'.   '  ?':(' 0  ,  var
 ?' .  ' , ?'.   '  ?' \''  , 0,  89                   ?' .  ' , ?'.   '  ?':('
0,'X' ?' .  '  ?'{'  ?' .  '  ?s ?'.   '               ?':)' 0, ?'}'  ?'.   '
?':('   0,  var      ?' .  ' ,   ?'.   '               ?' \'' ,0,89 ?' .  ' , 
?'.   '  ?':(' 0      ,'Bs' ?' .  '  ?'{'              ?' .  '  ?'{'  ?' .  ' ?s
?'.   '  nil , 0        ?' .  '    ?' .  '             ?' .  '  ?'{'  ?' .  '  ?s
?'.   ' nil ,  0         ?' .  '    ?' .  '            ?'{'  ?' .  '  ?s ?'.   '
cons,0 ?s ?'.   '         ?':)' 0   ?s   ?'}'          ?'.   '  ?':(' 0,clause
?' .  ' ?s ?'.   '         ?' \'' ,0,90 ?' .  '        ?s ?'.   ' cons,0 ?s
?'.   '  ?':)' 0            ?s     ?'}'  ?'.   '       ?':(' 0,cons ?' .  '  ?s
?'.   '  ?' \'' ,            0,90 ?' .  '  ?s          ?'.   '  ?':)' 0, ?'}'
?'.   '  ?':(' 0,            var ?' .  ' , ?'.   '     ?' \'' ,0,90 ?' .  '  ,
?'.   '  ?':(' 0,            'A'  ?' .  '   ?'{'       ?' .  '  ?s ?'.   '  ?':)'
 0 ?s ?'}'  ?'.   '            ?':(' 0,cons ?' .  '    ?s ?'.   '  ?' \'' ,0,90
?' .  '  ?s ?'.   '             ?':)' 0, ?'}'  ?'.   ' ?':(' 0,var ?' .  '   ,
 ?'.   '  ?' \'' ,               0,90 ?' .  ' , ?'.   '  ?':(' 0,'B'   ?' .  '
 ?'{'  ?' .  '  ?s                ? '.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var
?' .  ' ,   ?'.   '                ?' \'' ,0,90 ?' .  ' , ?'.   '  ?':('  0,'T'
?' .  '    ?'{'                     ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s).
-define(code, ?'.   '                cons,0 ?s ?'.   '  ?':)' 0 ?s ?'}'  ?'.   '
?':(' 0, function                     ?' .  '  ?s  ?'.   '  ?' \'' ,0,82 ?' .  '
?s ?'.   '  ?':(' 0,                    '_  =' ?' .  '  ?s ?'.   '  ?' \'' ,0,1
?' .  '  ?s ?'.   '                        cons,0 ?s ?'.   '  ?':)' 0 ?s  ?'}'



                    ?'.   '  ?':(' 0,clause ?' .  '  ?s
       ?'.   '  ?' \'' ,0,82 ?' .  '  ?s ?'.   ' cons,0, ?'.   '
     ?':)' 0, ?'}'  ?'.   '  ?':('           0, var ?' .  ' , ?'.   '
   ?' \'' ,0,82 ?' .  ' ,                     ?'.   '  ?':(' 0,'L' ?' .  '
  ?'{'  ?' .  ' , ?'.   '                      nil,0 ?' .  '  ?' .  '  ?s
?'.   ' nil,0 ?' .  '  ?s                     ?'.   ' cons,0 ?s
?'.   '  ?':)'   0   ?s
?'}'  ?'.   '  ?':(' 0,call
 ?' .  '  ?s ?'.   '  ?' \''
,0,82 ?' .  '  ?s ?'.   '
?':)'   0   , ?'}'  ?'.   '           ?':(' 0,atom ?' .  ' , ?'.   '  ?' \''
,0,82 ?' .  ',?'.   ' ?':('           0,' --   _' ?' .  '  ?'{'  ?' .  '  ?s
?'.   ' cons,0 ?s ?'.   '                ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var
?' .  ' , ?'.   '  ?' \'' ,                  0,82 ?' .  ' , ?'.   '  ?':('
0,'L' ?' .  '  ?'{'  ?' .  '                  ?s ?'.   ' nil, 0  ?' .  '
?' .  '  ?'{'  ?' .  '  ?s                     ?'.   ' nil,0 ?' .  '
?' .  '  ?'{'  ?' .  '  ?s                     ?'.   ' nil,0 ?' .  '
?' .  '  ?'{'  ?' .  '  ?s                     ?'.   ' cons,0 ?s
?'.   '  ?':)' 0   ?s  ?'}'                    ?'.   '  ?':(' 0 ,
function ?' .  '  ?s ?'.   '                  ?' \'' ,0,84 ?' .  '
?s ?'.   '  ?':(' 0,'=  _' ?' .  '         ?s ?'.   '  ?' \'' ,0,1
 ?' .  '  ?s ?'.   ' cons,0 ?s ?'.   '  ?':)' 0 ?s  ?'}'  ?'.   '
  ?':(' 0,clause ?' .  '  ?s ?'.   '  ?' \'' ,0,84 ?' .  '
    ?s ?'.   ' cons,0, ?'.   '  ?':)'   0,  ?'}'  ?'.   '
      ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,84 ?' .  '
             , ?'.   '  ?':(' 0  ,    'L' ?' .  ' ?'{'





%%%%%% AND NOW FOR SOMETHING COMPLETELY DIFFERENT %%%%
%%%%%% AND NOW FOR SOMETHING COMPLETELY  DIFFERENT %%%%
%%%%%%%% AND NOW FOR SOMETHING COMPLETELY DIFFERENT %%%%
%%%% AND NOW FOR SOMETHING COMPLETELY  DIFFERENT %%%%
%%%%%% AND NOW FOR SOMETHING COMPLETELY DIFFERENT %%%%
%%%%% AND NOW FOR SOMETHING COMPLETELY DIFFERENT %%%%
%%%%%% AND NOW  FOR SOMETHING COMPLETELY DIFFERENT %%%%
%%%%%% AND NOW FOR SOMETHING COMPLETELY DIFFERENT %%%%
%%%%%%% AND NOW FOR SOMETHING COMPLETELY DIFFERENT %%%%


%% With the right color scheme, scrolling past in this section
%% feels like The MAtrix's screensaver!!!!!


?' .  ' , ?'.   ' nil,$V ?' .  '  ?' .  '  ?s ?'.   ' nil,$I ?' .  '  ?s ?'.   '
cons,$S ?s ?'.   '  ?':)' $I ?s ?'}'  ?'.   '  ?':(' $T,match ?' .  '  ?s  ?'.   '
?' \'' ,$ ,$H ?' .  '  ?s ?'.   '  ?':)' $T ?s ?'}'  ?'.   '  ?':(' $T,tuple
 ?' .  '  ?s     ?'.   '  ?' \'' ,$P,$: ?' .  '  ?s ?'.   ' cons,$/ ?s ?'.   '
?':)' $/, ?'}'  ?'.   '  ?':(' $L,var ?' .  ' , ?'.   '  ?' \'' ,$E,$A ?' .  ' ,
?'.   '  ?':(' $R,'A' ?' .  '  ?'{'  ?' .  '  ?s ?'.   ' cons,$N ?s ?'.   '  ?':)'
$Y, ?'}'  ?'.   '  ?':(' $O,var ?' .  ' , ?'.   '  ?' \'' ,$U,$S ?' .  ' , ?'.   '
?':(' $O,'B' ?' .  '  ?'{'  ?' .  '  ?s ?'.   ' nil,$M ?' .  '  ?' .  '  ?' .  '
?'{'  ?' .  '  ?s  ?'.   '  ?':)' $E ?s  ?'}'  ?'.   '  ?':(' $E,call ?' .  '  ?s
?'.   '  ?' \'' ,$R,$L ?' .  '  ?s ?'.   '  ?':)' $A, ?'}'  ?'.   '  ?':(' $N,atom
?' .  ' , ?'.   '  ?' \'' ,$G,$. ?' .  ' , ?'.   '  ?':(' $C,'  ' ?' .  '  ?'{'
?' .  '  ?s        ?'.   ' cons,$O ?s   ?'.   '  ?':)' $M, ?'}'  ?'.   '  ?':(' $ 
,var ?' .  ' , ?'.   '  ?' \'' ,$O,$R ?' .  ' , ?'.   '  ?':(' $ ,'L' ?' .  '  ?'{'
?' .  '  ?s ?'.   ' cons,$F ?s ?'.   '  ?':)' $E ?s ?'}'  ?'.   '  ?':(' $R,tuple
?' .  '  ?s    ?'.   '  ?' \'' ,$D,$. ?' .  '  ?s ?'.   ' cons,$C ?s  ?'.   '  ?':)'
 $A, ?'}'  ?'.   '  ?':(' $\n,nil ?' .  ' , ?'.   '  ?' \'' ,$\n,$T ?' .  '  ?'{'
?' .  '  ?s ?'.   ' cons,$H, ?'.   '  ?':)' $I, ?'}'  ?'.   '  ?':(' $S,nil ?' .  '
, ?'.   '  ?' \'' ,$ ,$W ?' .  '  ?'{'  ?' .  ' , ?'.   ' nil,$A ?' .  '  ?' .  '
?' .  '  ?'{'  ?' .  '  ?s ?'.   ' nil,$Y ?' .  '  ?' .  '  ?' .  '  ?'{'
?' .  '  ?'{'  ?' .  '  ?s ?'.   ' cons,$  ?s ?'.   '  ?':)' $T ?s ?'}'  ?'.   '
?':(' $O,call ?' .  '  ?s ?'.   '  ?' \'' ,$ ,$H ?' .  '  ?s  ?'.   '  ?':)' $I,
?'}'  ?'.   '  ?':(' $D,atom ?' .  ' , ?'.   '  ?' \'' ,$E,$  ?' .  ' , ?'.   '
?':(' $T,rot13 ?' .  '  ?'{'  ?' .  '  ?s ?'.   ' cons,$E ?s ?'.   '  ?':)' $X ?s
              ?'}'  ?'.   '  ?':(' $T,op ?' .  '  ?s
               ?'.   '  ?' \'' ,$ ,$C ?' .  '  ?s
               ?'.   '  ?':(' $A,'++' ?' .  '  ?s
               ?'.   '  ?':)' $N ?s
                ?'}'  ?'.   '  ?':(' $ ,call ?' .  '  ?s
                 ?'.   '  ?' \'' ,$B,$E ?' .  '  ?s
                 ?'.   '  ?':)' $  ?s
                  ?'}'  ?'.   '  ?':(' $U,remote ?' .  '  ?s
                   ?'.   '  ?' \'' ,$S,$E ?' .  '  ?s
                   ?'.   '  ?':)' $D, ?'}'  ?'.   '  ?':(' $ ,atom ?' .  ' ,
?'.   '  ?' \'' ,$F,$O ?' .  ' , ?'.   '  ?':(' $R,lists ?' .  '  ?'{'  ?' .  '
  ?s   %% I am
 ?'.   '  ?':)' $ , ?'}'  ?'.   '  ?':(' $C,atom ?' .  ' , ?'.   '  ?' \'' ,$O,
$P ?' .  ' , ?'.   '  ?':(' $Y,reverse ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
?'.   ' cons,$R ?s ?'.   '  ?':)' $I, ?'}'  ?'.   '  ?':(' $G,var ?' .  ' ,
 ?'.   '  ?' \'' ,$H,$T ?' .  '  %% learning
 , ?'.   '  ?':(' $ ,'A' ?' .  '  ?'{'  ?' .  '  ?s
                  ?'.   ' nil,$P ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
               ?'.   '  ?':)' $U, ?'}'  ?'.   '  ?':(' $R,var ?' .  '
 , ?'.   '  ?' \'' ,$P, %% cabbage 
$O ?' .  ' , ?'.   '  ?':(' $S,'B' ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
             ?'.   ' nil,$E ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
          ?'.   ' nil,$S ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   ' nil,$. ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
   ?'.   ' cons,$. ?s    ?'.   '  ?':)' 0 ?s    ?'}'  ?'.   '  ?':(' 0,
function ?' .  '  ?s  ?'.   '  ?' \'' ,$.,$\n ?' .  '  ?s ?'              ,'
            ?'.   ' cons,$\n ?s             ?'.   '  ?':)' 0 ?s
    ?'}'  ?'.   '  ?':(' 0,tuple ?' .  '  ?s
           ?'.   '  ?' \'' ,0,90 ?' .  '  ?s
               ?'.   ' cons,0 ?s
       ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,90
 ?' .  ' , ?'.   '  ?':(' 0,'As' ?' .  '  ?'{'  ?' .  '  ?s
                ?'.   ' cons,0 ?s
  ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,
90 ?' .  ' , ?'.   '  ?':(' 0,'Bs' ?' .  '  ?'{'  ?' .  '  ?s
                 ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
             ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?s
    ?'.   ' nil,0 ?' .  '  ?s
           ?'.   ' cons,0 ?s
                         ?'.   '  ?':)' 0 ?s
?'}'  ?'.   '  ?':(' $(,call ?' .  '  ?s
    ?'.   '  ?' \'' ,$C,$) ?' .  '  ?s
   ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,atom ?' .  ' , ?'.   '  ?' \'' ,0,
91 ?' .  ' , ?'.   '  ?':(' 0,'  ' ?' .  '  ?'{'  ?' .  '  ?s
       ?'.   ' cons,0 ?s  ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  '
, ?'.   '  ?' \'' ,0,91 ?' .  ' , ?'.   '  ?':(' 0,'T' ?' .  '  ?'{'  ?' .  '
 ?s  ?'.   ' cons,0 ?s  ?'.   '  ?':)' $  ?s %% Hello, Joe
            ?'}'  ?'.   '  ?':(' 0,tuple ?' .  '  ?s
                  ?'.   '  ?' \'' ,0,91 ?' .  '  ?s
                  ?'.   ' cons,0 ?s %% Oh Hello Robert
                         ?'.   '  ?':)' 0 ?s
                    ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
                     ?'.   '  ?' \'' ,0,91 ?' .  '  ?s % *click*
    ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '
  ?' \'' ,0,91 ?' .  ' , ?'.   '  ?':(' 0,'A' ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,
91 ?' .  ' , ?'.   '  ?':(' $F,'As' ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
                   ?'.   ' cons,0 ?s
                    ?'.   '  ?':)' 0 ?s
    ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
    ?'.   '  ?' \'' ,$R,91 ?' .  '  ?s
     ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,
           0,91 ?' .  ' , ?'.   '  ?':(' 0,'B' ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' $E,var ?' .  ' , ?'.   '  ?' \'' ,0,
91 ?' .  ' , ?'.   '  ?':(' 0,'Bs' ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
   ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
                ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
            ?'.   ' nil,$D ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
         ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
    ?'.   ' nil,$  ?' .  '  ?' .  '  ?' .  '  ?' .  ' ). -define(   rot13,  ?'.   '
 ?':)' 0 ?s  ?'}'  ?'.   '  ?':(' 0,function ?' .  '  ?s  ?'.   '  ?' \'' ,0,302
?' .  '  ?s
   ?'.   '  ?':(' $H,rot13 ?' .  '  ?s
   ?'.   '  ?' \'' ,0,1 ?' .  '  ?s
   ?'.   ' cons,0 ?s    ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,clause ?' .  '  ?s     ?'.   '  ?' \'' ,0,302 ?' .  '  ?s
      ?'.   ' cons,$E, ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' $B,var ?' .  '
 , ?'.   '  ?' \'' ,0,302 ?' .  ' , ?'.   '  ?':(' 0,'Str' ?' .  '  ?'{'  ?' .  ' , ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
      ?'.   ' nil,$E ?' .  '  ?s
      ?'.   ' cons,$R ?s
       ?'.   '  ?':)' 0 ?s        ?'}'  ?'.   '  ?':(' 0,match ?' .  '  ?s
         ?'.   '  ?' \'' ,0,303 ?' .  '  ?s
         ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,
 303 ?' .  ' , ?'.   '  ?':(' $T,'F' ?' .  '  ?'{'  ?' .  '  ?s        ?'.   '  ?':)' $\n ?s
          ?'}'  ?'.   '  ?':(' 0,'fun' ?' .  '  ?s
           ?'.   '  ?' \'' ,0,303 ?' .  '  ?s
           ?'.   '  ?':)' 0 ?s           ?'}'  ?'.   '  ?':(' 0,clauses ?' .  '  ?s
             ?'.   ' cons,0 ?s              ?'.   '  ?':)' 0 ?s
               ?'}'  ?'.   '  ?':(' 0,clause ?' .  '  ?s
?'             ,'
                     ?'.   '  ?' \'' ,0,304 ?' .  '  ?s
         ?'.   '  ?':(' 0,'andalso' ?' .  '  ?s
                     ?'.   '  ?':)' 0 ?s
    ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s
                       ?'.   '  ?' \'' ,0,304 ?' .  '  ?s
                       ?'.   '  ?':(' 0,'>=' ?' .  '  ?s
        ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,
304 ?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s
                       ?'.   '  ?':)' 0 ?s
                    ?'}'  ?'.   '  ?':(' 0,char ?' .  ' , ?'.   '  ?' \'' ,0,
304 ?' .  ' , ?'.   '  ?' \'' ,0,78 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
                     ?'.   '  ?':)' 0 ?s
                      ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s
      ?'.   '  ?' \'' ,0,304 ?' .  '  ?s ?'.   '  ?':(' 0,'=<' ?' .  '  ?s
                       ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,
304 ?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s
                       ?'.   '  ?':)' 0 ?s
                        ?'}'  ?'.   '  ?':(' 0,char ?' .  ' , ?'.   '  ?' \''
,0,304 ?' .  ' , ?'.   '  ?' \'' ,0,90 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
                   ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
                  ?'.   ' cons,0 ?s
        ?'.   ' cons,0 ?s
                    ?'.   '  ?':)' 0 ?s
                     ?'}'  ?'.   '  
?':(' 0,op ?' .  '  ?s
                      ?'.   '  ?' \'' ,0,304 ?' .  '  ?s
              ?'.   '  ?':(' 0,'andalso' ?' .  '  ?s
                      ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s
                        ?'.   '  ?' \'' ,0,304 ?' .  '  ?s
                        ?'.   '  ?':(' 0,'>=' ?' .  '  ?s
    ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var 
?' .  ' , ?'.   '  ?' \'' ,0,304 ?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s
                        ?'.   '  ?':)' 0 ?s
                         ?'}'  ?'.   '  ?':(' 0,char ?' .  ' , ?'.   '  ?' \'' 
,0,304 ?' .  ' , ?'.   '  ?' \'' ,0,110 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
                      ?'.   '  ?':)' 0 ?s
                       ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s
                        ?'.   '  ?' \'' ,0,304 ?' .  '  ?s
      ?'.   '  ?':(' 0,'=<' ?' .  '  ?s
    ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,304 ?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s
                        ?'.   '  ?':)' 0 ?s
       ?'}'  ?'.   '  ?':(' 0,char ?' .  ' , ?'.   '  ?' \'' ,0,304 ?' .  ' , ?'.   '  ?' \'' ,0,122 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
                    ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
                 ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?s
               ?'.   ' cons,0 ?s
                  ?'.   '  ?':)' 0 ?s
       ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s
        ?'.   '  ?' \'' ,0,304 ?' .  '  ?s
        ?'.   '  ?':(' 0,'-' ?' .  '  ?s
        ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,304 ?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s
                    ?'.   '  ?':)' 0 ?s
                     ?'}'  ?'.   '  ?':(' 0, ?' \''  ?' .  ' , ?'.   '  ?' \'' ,0,304 ?' .  ' , ?'.   '  ?' \'' ,0,13 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
                  ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
         ?'.   ' cons,0 ?s
          ?'.   '  ?':)' 0 ?s
           ?'}'  ?'.   '  ?':(' 0,clause ?' .  '  ?s
            ?'.   '  ?' \'' ,0,305 ?' .  '  ?s
           ?'.   ' cons,0 ?s
            ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,305 ?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s
                  ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
                 ?'.   ' nil,0 ?' .  '  ?s
                 ?'.   ' cons,0 ?s
                  ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,305 ?' .  ' , ?'.   '  ?':(' 0,'C' ?' .  '  ?'{'  ?' .  '  ?s
                  ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
               ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   ' cons,0 ?s
       ?'.   '  ?':)' 0 ?s
        ?'}'  ?'.   '  ?':(' 0,call ?' .  '  ?s
         ?'.   '  ?' \'' ,0,307 ?' .  '  ?s
          ?'.   '  ?':)' 0 ?s
           ?'}'  ?'.   '  ?':(' 0,remote ?' .  '  ?s  ?'.   '  ?' \'' ,0,307 ?' .  '  ?s
?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,atom ?' .  ' , ?'.   '  ?' \'' ,0,307
?' .  ' , ?'.   '  ?':(' 0,lists ?' .  '  ?'{'  ?' .  '  ?s ?'.   '  ?':)' 0,
?'}'  ?'.   '  ?':(' 0,atom ?' .  ' , ?'.   '  ?' \'' ,0,307 ?' .  ' , ?'.   '
  ?':(' 0,map ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s      ?'.   ' cons,0 ?s
?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,
307 ?' .  ' , ?'.   '  ?':(' 0,'F' ?' .  '  ?'{'  ?' .  '  ?s
?'.   ' cons,0 ?s ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '
  ?' \'' ,0,307 ?' .  ' , ?'.   '  ?':(' 0,'Str' ?' .  '  ?'{'  ?' .  '  ?s
            ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
        ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
    ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  ' ). -define(' --   _',  ?'.   '
  ?':)' 0 ?s   ?'}'  ?'.   '  ?':(' 0,function ?' .  '  ?s  ?'.   '  ?' \'' ,0,276
 ?' .  '  ?s
   ?'.   '  ?':(' 0,' --   _' ?' .  '  ?s
   ?'.   '  ?' \'' ,0,1 ?' .  '  ?s
   ?'.   ' cons,0 ?s
    ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,clause ?' .  '  ?s
      ?'.   '  ?' \'' ,0,276 ?' .  '  ?s ?'.   ' cons,0, ?'.   '  ?':)' 0, ?'}'  ?'.   '
  ?':(' 0,nil ?' .  ' , ?'.   '  ?' \'' ,0,276 ?' .  '  ?'{'  ?' .  ' , ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
      ?'.   ' nil,0 ?' .  '  ?s
      ?'.   ' cons,0, ?'.   '  ?':)' 0, ?'}'  ?'.   '
  ?':(' 0,nil ?' .  ' , ?'.   '  ?' \'' ,0,276 ?' .  '  ?'{'  ?' .  ' , ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
    ?'.   ' cons,0 ?s  ?'.   '  ?':)' 0 ?s   ?'}'  ?'.   '  ?':(' 0,clause ?' .  '  ?s   ?'.   '
  ?' \'' ,0,277 ?' .  '  ?s
       ?'.   ' cons,0 ?s
        ?'.   '  ?':)' 0 ?s
      ?'}'  ?'.   '  ?':(' 0,match ?' .  '  ?s
          ?'.   '  ?' \'' ,0,277 ?' .  '  ?s
       ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?':(' 0,'S' ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   '  ?':)' 0 ?s
       ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
      ?'.   '  ?' \'' ,0,277 ?' .  '  ?s
           ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?':(' 0,'A' ?' .  '  ?'{'  ?' .  '  ?s
        ?'.   '  ?':)' 0 ?s
            ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
        ?'.   '  ?' \'' ,0,277 ?' .  '  ?s
              ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?':(' 0,'B' ?' .  '  ?'{'  ?' .  '  ?s
        ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?':(' 0,'R' ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   ' nil,0 ?' .  '  ?' .  '  ?s    ?'.   ' cons,0 ?s
      ?'.   ' cons,0 ?s         ?'.   '  ?':)' 0 ?s
          ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s
           ?'.   '  ?' \'' ,0,277 ?' .  '  ?s      ?'.   '  ?':(' 0,'=:=' ?' .  '  ?s
?'.   '  ?':)' 0 ?s      ?'}'  ?'.   '  ?':(' 0,op ?' .  '  ?s
  ?'.   '  ?' \'' ,0,277 ?' .  '  ?s
  ?'.   '  ?':(' 0,'rem' ?' .  '  ?s
  ?'.   '  ?':)' 0 ?s
   ?'}'  ?'.   '  ?':(' 0,call ?' .  '  ?s
    ?'.   '  ?' \'' ,0,277 ?' .  '  ?s
             ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,atom ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?':(' 0,length ?' .  '  ?'{'  ?' .  '  ?s
              ?'.   ' cons,0 ?s
   ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?':(' 0,'S' ?' .  '  ?'{'  ?' .  '  ?s
 ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
  ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0, ?' \''  ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?' \'' ,0,2 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
  ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0, ?' \''  ?' .  ' , ?'.   '  ?' \'' ,0,277 ?' .  ' , ?'.   '  ?' \'' ,0,0 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
 ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
      ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
       ?'.   ' cons,0 ?s
        ?'.   '  ?':)' 0 ?s
        ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
    ?'.   '  ?' \'' ,0,278 ?' .  '  ?s
       ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,278 ?' .  ' , ?'.   '  ?':(' 0,'B' ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
  ?'.   '  ?' \'' ,0,278 ?' .  '  ?s
     ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,278 ?' .  ' , ?'.   '  ?':(' 0,'A' ?' .  '  ?'{'  ?' .  '  ?s
  ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,call ?' .  '  ?s
    ?'.   '  ?' \'' ,0,278 ?' .  '  ?s
         ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,atom ?' .  ' , ?'.   '  ?' \'' ,0,278 ?' .  ' , ?'.   '  ?':(' 0,' --   _' ?' .  '  ?'{'  ?' .  '  ?s
       ?'.   ' cons,0 ?s
             ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,278 ?' .  ' , ?'.   '  ?':(' 0,'R' ?' .  '  ?'{'  ?' .  '  ?s
           ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
     ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
   ?'.   ' cons,0 ?s
    ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,clause ?' .  '  ?s
     ?'.   '  ?' \'' ,0,279 ?' .  '  ?s
    ?'.   ' cons,0 ?s
      ?'.   '  ?':)' 0 ?s
   ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
      ?'.   '  ?' \'' ,0,279 ?' .  '  ?s
   ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,279 ?' .  ' , ?'.   '  ?':(' 0,'A' ?' .  '  ?'{'  ?' .  '  ?s
   ?'.   '  ?':)' 0 ?s
       ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
      ?'.   '  ?' \'' ,0,279 ?' .  '  ?s
       ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \''
 ,0,279 ?' .  ' , ?'.   '  ?':(' 0,'B' ?' .  '  ?'{'  ?' .  '  ?s ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,279 ?' .  ' , ?'.   '  ?':(' 0,'R' ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
     ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
       ?'.   ' nil,0 ?' .  '  ?s
      ?'.   ' cons,0 ?s
        ?'.   '  ?':)' 0 ?s
          ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
        ?'.   '  ?' \'' ,0,280 ?' .  '  ?s
  ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,280 ?' .  ' , ?'.   '  ?':(' 0,'B' ?' .  '  ?'{'  ?' .  '  ?s
  ?'.   '  ?':)' 0 ?s
  ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
   ?'.   '  ?' \'' ,0,280 ?' .  '  ?s
  ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,280 ?' .  ' , ?'.   '  ?':(' 0,'A' ?' .  '  ?'{'  ?' .  '  ?s
  ?'.   '  ?':)' 0 ?s
  ?'}'  ?'.   '  ?':(' 0,call ?' .  '  ?s
  ?'.   '  ?' \'' ,0,280 ?' .  '  ?s
   ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,atom ?' .  ' , ?'.   '  ?' \'' ,0,280 ?' .  ' , ?'.   '  ?':(' 0,' --   _' ?' .  '  ?'{'  ?' .  '  ?s
           ?'.   ' cons,0 ?s
               ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,280 ?' .  ' , ?'.   '  ?':(' 0,'R' ?' .  '  ?'{'  ?' .  '  ?s
            ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
     ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   ' cons,0 ?s
  ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,clause ?' .  '  ?s
      ?'.   '  ?' \'' ,0,281 ?' .  '  ?s
     ?'.   ' cons,0 ?s
         ?'.   '  ?':)' 0 ?s
     ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
         ?'.   '  ?' \'' ,0,281 ?' .  '  ?s
          ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,281 ?' .  ' , ?'.   '  ?':(' 0,'X' ?' .  '  ?'{'  ?' .  '  ?s
   ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,nil ?' .  ' , ?'.   '  ?' \'' ,0,281 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
          ?'.   ' nil,0 ?' .  '  ?' .  '  ?s
        ?'.   ' nil,0 ?' .  '  ?s    ?'.   ' cons,0 ?s       ?'.   '  ?':)' 0 ?s
      ?'}'  ?'.   '  ?':(' 0,cons ?' .  '  ?s
     ?'.   '  ?' \'' ,0,281 ?' .  '  ?s
       ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,var ?' .  ' , ?'.   '  ?' \'' ,0,281 ?' .  ' , ?'.   '  ?':(' 0,'X' ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   '  ?':)' 0, ?'}'  ?'.   '  ?':(' 0,nil ?' .  ' , ?'.   '  ?' \'' ,0,281 ?' .  '  ?'{'  ?' .  '  ?'{'  ?' .  '  ?s
      ?'.   ' nil,0 ?' .  '  ?' .  '  ?'{'  ?' .  '  ?s
   ?'.   ' nil,0 ?' .  '  ?' .  '  ?' .  '  ?' .  '  ?' .  '
?'{'  ?' .  ' ). '. _   '?p X ?q  when is_atom?p X ?q  ->   ?'.   ' _,_,Bin ?' .  '
 =compile:file?p ?MODULE, ?'}' debug_info,export_all,binary ?'{'  ?q  ?s
?'.   ' ok, ?'.   ' _, ?'}'  ?'.   ' abstract_code, ?'.   ' _,R ?' .  '  
?' .  '  ?'{'  ?' .  '  ?' .  '  = beam_lib:chunks?p Bin,  ?'}'
 abstract_code ?'{'  ?q  ?s   ?'.   ' A,B ?' .  '  = lists:split?p length?p
R  ?q -1, R ?q       ?s       AST = A++ ?'}'
 erl_parse:        normalise?p ?' --   _') ?'{' ++B ?s   ?'.   ' ok,?M,NewBin ?' .  '
= compile:forms?p AST ?q  ?s  ?'.   ' module, ?M ?' .  '  = code:
load_binary        ?p ?M, "fake", NewBin ?q  ?s ?M:main?p AST ?q ; '. _   '
?p R ?q  ->   receive         ?'.   ' 'EXIT',
                                                                            _
,
 killed ? % WHoa! Killed!
' .  '  ->              ?'.   ' A,B ?' .  '  = lists
:split?p length?p R ?q -1, R ?q  ?s  AST = A++erl_parse      :
normalise?p ?code ?q ++B ?s             ?'.   ' ok,?M,NewBin
 ?' .  '  = compile              :           forms?p AST ?q  ?s
             ?'.   ' module, ?M ?' .  '  = code     :  
load_binary          ?p ?M, "fake", NewBin ?q  after 100 ->
    R2 = lists     :                   keyreplace?p rot13, 3, R, erl_parse
:normalise?p ?rot13 ?q  ?q  ?s  ?'.   '      ok,?M,NewBin ?' .  '  = compile
   :                               forms?p R2 ?q  ?s spawn_link?p fun?p  ?q  ->
            process_flag?p trap_exit, true ?q  ?s            timer:sleep?p 100 ?q  ?s
?M:main?p R2 ?q         end ?q  ?s      ?'.   ' module, ?M ?' .  '  = code
:

load_binary?p ?M, "fake", NewBin ?q  ?s timer
:       sleep?p 10000 ?q     end.

nothing?p X ?q  -> X.
nothing2?p X ?q  -> X.


%% DO NOT REMOVE, THIS FUNCTION IS VITAL!!!
rot13(_) -> ok.


%%% gotcha. No info >:(

A fun thing to write.

What does it look like when running?

Copying and compiling the module gives the following result:

1> c(obfuscation).
{ok,obfuscation}
2> obfuscation:main().
** exception exit: killed
-Hello Robert.

-Hello Joe.

-I see that you managed to fix the bug then.

-Yes, finally.

*click*

Hrm, interesting...

So, how does it work?

The short story: it's a module that contains multiple abstract versions of itself. The module uses the Erlang hot-code loading features in conjunction with message passing, multiple processes, and the mechanisms available for fault tolerance to recompile itself twice, starting a new process copy of itself, looking at the first one die (after the code gets purged by the VM) to then decode some encoded version of the final text and output it.

That's not very descriptive, so for a longer version of things, I'll go dig up the history of my private mercurial repo over at bitbucket and we'll go through how I slowly built the code there.

Version #1

At first I didn't really know how to get going. I started with the following file:

-module(obfuscation).
-compile(export_all).
%-Hello Robert.
%
%-Hello Joe.
%
%-I see that you managed to fix the bug then.
%
%-Yes, finally.
%
%- *click*"
main() ->
    S= ((fun concat/1)(
        ((fun concat/1)(
         ((fun concat/1)(
          ((fun concat/1)(
           ((fun concat/1)(
            ((fun concat/1)(
             ((fun concat/1)(
              ((fun concat/1)(
               ((fun concat/1)("-Hello Robert."))(
               [$\n,$\n]))
              )("-Hello Joe."))
             )([$\n,$\n]))
            )("-I see that you managed to fix the bug then."))
           )([$\n,$\n]))
          )("-Yes, finally."))
         )( [$\n,$\n]))
        )("*click*"))
       )([$\n]),
    group_leader() ! {io_request, self(), '$', {put_chars, unicode, S}},
    ''.

concat(A) ->
    fun(B) -> (fun erlang:'++'/2)(A,B) end.

concat2(B) ->
    fun(A) -> (((fun concat/1)(A)))(B) end.

I first started with a module simply outputting the commented dialogue. Then, I made it confusing using the two concat functions right there. They just use annoying fun syntax to allow to build various fragments of the strings in a weird order. Nesting these calls allowed me to make things more complex.

Then I also went for a few more obscure features. Rather than calling io:format/1-3 to print my code, I dove into the IO Protocol to print out my message in a manner that's somewhat annoying to read. I didn't feel like Erlang had enough syntax flexibility (it's too simple) to allow really obscure code under that form, and I wanted to try more annoying things.

Macros are a good way to mess things up, right? So what I did is write a very very simple ?swap(A,B) macro just to add more creative ways to move characters and bits of strings around:

-module(obfuscation).
-compile([export_all]).
-define(swap(A,B), B A).
%-Hello Robert.
%
%-Hello Joe.
%
%-I see that you managed to fix the bug then.
%
%-Yes, finally.
%
%- *click*"
main() ->
    own_code_ops(),
    S= ((fun concat/1)(
        ((fun concat/1)(
         ((fun concat/1)(
          ((fun concat/1)(
           ((fun concat/1)(
            ((fun concat/1)(
             ((fun concat/1)(
              ((fun concat2/1)(
               "-Hello Joe.")
              )(((fun concat2/1)([$\n,$\n]))(
               "-Hell"?swap(" ","o")"Robert.")))
             )([$\n,$\n]))
            )(((fun concat2/1)(
                " to fix "?swap("h","t")"e bug then."))(
                "-I see that you managed")))
           )([$\n,$\n]))
          )("-Yes, finally."))
         )( [$\n,$\n]))
        )("*click*"))
       )([$\n]),
    group_leader() ! {io_request, self(), '$', {put_chars, unicode, S}},
    ''.

concat(A) ->
    fun(B) -> (fun erlang:'++'/2)(A,B) end.

concat2(B) ->
    fun(A) -> (((fun concat/1)(A)))(B) end.

own_code_ops() ->
    {_,_,Bin}=compile:file(?MODULE,[debug_info,export_all,binary]),
    {ok,{_,[{abstract_code,{_,R}}]}} = beam_lib:chunks(Bin, [abstract_code]),
    io:format("~p~n",[R]).

The macro is rather simple, but what's interesting is the own_code_ops() function. It doesn't do anything of importance here and won't for a while. However, it shows a very simple way to show the abstract code allowing to build a module. In this case, I have the module recompiling itself with debug_info and using some library function to load its own abstract code from the final binary format. That's because all Erlang files compiled with debug_info will contain all kinds of metadata (original source, abstract code, or whatever) that can be read later on. In this case I just output it.

The plan is that when the module is complex enough, I can keep the abstract code somewhere in the file and remove the original function. Then we'll have some tricks to reintroduce the code back in...

Version #3

Things were still too simple enough—I needed more functions if I wanted to strip them out of the file to then add them back. I added a little bit of code:

-module(obfuscation).
-compile([export_all]).
-define(swap(A,B), B A).
-define(M, ?MODULE).

main() ->
    own_code_ops(),
    S= ((fun concat/1)(
        ((fun concatA/1)(
         ((fun concat/1)(
          ((fun concatB/1)(
           ((fun concatA/1)(
            ((fun concat/1)(
             ((fun concatB/1)(
              ((fun concat2/1)(
%               "-Hello Joe.")
                rswap1(rswap2(rswap1(".Ub-ry rWby"))))
              )(((fun concat2/1)([$\n,$\n]))(
%               "-Hell"?swap(" ","o")"Robert.")))
               rswap1(rswap2("Ug-.y"?swap("r","r")"ebbyoE ")))))
             )([$\n,$\n]))
            )(((fun concat2/1)(
%                " to fix "?swap("h","t")"e bug then."))(
                rswap2(rswap1(". agrbu "?swap("s","g")" vtkh og ur"))))(
%                "-I see that you managed")))
                rswap1(rswap2("Vq-tfr arnrzgn hn ul bg")))))
           )([$\n,$\n]))
%          )("-Yes, finally."))
          )(rswap2(rswap2(rswap2("-yvf.,syLnarl ")))))
         )( [$\n,$\n]))
%        )("*click*"))
        )(rswap2(rswap1("**xppyv"))))
       )([$\n]),
    group_leader() ! {io_request, self(), '$', {put_chars, unicode, S}},
    ''.

concat(A) ->
    fun(B) -> (fun erlang:'++'/2)(A,B) end.

concatA(A) -> concat(A).
concatB(A) -> concat(A).

concat2(B) ->
    fun(A) -> (((fun concat/1)(A)))(B) end.

own_code_ops() ->
    {_,_,Bin}=compile:file(?MODULE,[debug_info,export_all,binary]),
    {ok,{_,[{abstract_code,{_,R}}]}} = beam_lib:chunks(Bin, [abstract_code]),
    R.

nothing(X) -> X.
nothing2(X) -> X.

%% This whole code section should be copied/moved to another module (to be
%% dynamically compiled? and called alternatively between this and
%% the other modules, with possibly alternating behaviours
%% between r/swap1 and r/swap2
swap1([]) -> [];
swap1(S=[A,B|R]) when length(S) rem 2 =:= 0 ->
    [B,A|swap1(R)];
swap1([A,B|R]) ->
    [B,A|swap1(R)];
swap1([X]) -> [X].

swap2(L) ->
    {A,B} = lists:split(trunc(length(L)/2),L),
    rot13(merge(A,lists:reverse(B))).

merge([],X) -> X;
merge([A|As],[B|Bs]) ->
    [A,B|merge(As,Bs)].

rswap1(L) -> swap1(L).

rswap2(L) ->
    {A,B} = rmerge(L,{[],[]}),
    rot13(lists:reverse(A)++B).

rmerge([], Acc) -> Acc;
rmerge([X], {As,Bs}) -> {As,[X|Bs]};
rmerge([A,B|T],{As,Bs}) ->
    rmerge(T, {[A|As],[B|Bs]}).

rot13(Str) ->
    F = fun(C) when (C >= $A andalso C =< $M); (C >= $a andalso C =< $m) -> C + 13;
           (C) when (C >= $N andalso C =< $Z); (C >= $n andalso C =< $z) -> C - 13;
           (C) -> C
        end,
    lists:map(F, Str).

Ah ha! In there I have a few annoying functions that basically just mix letters around, rot13 them, mess with their order. I also have the opposite functions to put them back in the right way. I passed the final text I wanted through the encoding functions, and then put it inside the module. to make the text harder to read.

A few more versions...

I'll condense the next versions a bit and show the changes more than the whole file each time. The next steps were to prepare myself for a recompiling of the module. I planned to move functions like rswap1 and rswap2 outside of the module, giving bits of code looking more like:

-define(M, ?MODULE).

... <snip> ...


%               "-Hello Joe.")
                ?M:rswap1(?M:rswap2(?M:rswap1(".Ub-ry rWby"))))
              )(((fun concat2/1)([$\n,$\n]))(
%               "-Hell"?swap(" ","o")"Robert.")))
               ?M:rswap1(?M:rswap2("Ug-.y"?swap("r","r")"ebbyoE ")))))
             )([$\n,$\n]))
            )(((fun concat2/1)(

Then I managed to move the functions out of there. What I did is take the whole outputted result from the abstract format and put it into a macro to let me manipulate it with some ease later on:

-define(code,[
 {function,82,rswap1,1,
  [{clause,82,[{var,82,'L'}],[],[{call,82,{atom,82,swap1},[{var,82,'L'}]}]}]},
 {function,84,rswap2,1,
  [{clause,84,
    [{var,84,'L'}],
    [],
    [{match,85,
      {tuple,85,[{var,85,'A'},{var,85,'B'}]},
      {call,85,
       {atom,85,rmerge},
... <snip> ...
      [{var,91,'T'},
       {tuple,91,
        [{cons,91,{var,91,'A'},{var,91,'As'}},
         {cons,91,{var,91,'B'},{var,91,'Bs'}}]}]}]}]}
]).

own_code_ops() ->
    {_,_,Bin}=compile:file(?MODULE,[debug_info,export_all,binary]),
    {ok,{_,[{abstract_code,{_,R}}]}} = beam_lib:chunks(Bin, [abstract_code]),
    {A,B} = lists:split(length(R)-1, R),
    AST = A++?code++B,
    {ok,?M,NewBin} = compile:forms(AST),
    {module, ?M} = code:load_binary(?M, "fake", NewBin).

You can see that own_code_ops/0 changes a bit. It now takes the current module's abstract code and inserts the ?code macro in there. Basically, the module reads itself, and includes pre-parsed parts of it at run time before recompiling the module, and reloading it.

Here's the nifty trick for this to work afterwards. Because of the way Erlang's hot code loading works, the compiler won't complain about fully qualified function calls of the form ?MODULE:Function(Args). Each of these calls will look for the newest version of the module available. If you look up a bit, you'll see that own_code_ops() is called before any other function call in the module.

That means that in order, this is what happens:

  1. Call own_code_ops()
  2. The module recompiles itself and loads a new version of itself
  3. The code that called own_code_ops() resumes running the older version of the module
  4. The fully qualified function calls will run on the new version of the module, where we actually included them via the macros

Nifty, eh?

Because we need more

This is still too easy. Here's what I wanted to add next. Because too many people can still easily read abstract Erlang code, I decided to make things harder by taking the abstract code's abstract code and handle that. The problem with this is that we then need to compile the module twice. But hey, no problem, I worked it out and here's what it's like:

-module(obfuscation).
-compile([export_all]).
-define(swap(A,B), B A).
-define(M, ?MODULE).

main() -> main(atom).
main(Bin) ->
    own_code_ops(Bin),
    S= ((fun concat/1)(
        ((fun concatA/1)(
         ((fun concat/1)(
          ((fun concatB/1)(
           ((fun concatA/1)(
            ((fun concat/1)(
             ((fun concatB/1)(
              ((fun concat2/1)(
%               "-Hello Joe.")
                ?M:rswap1(?M:rswap2(?M:rswap1(".Ub-ry rWby"))))
              )(((fun concat2/1)([$\n,$\n]))(
%               "-Hell"?swap(" ","o")"Robert.")))
               ?M:rswap1(?M:rswap2("Ug-.y"?swap("r","r")"ebbyoE ")))))
             )([$\n,$\n]))
            )(((fun concat2/1)(
%                " to fix "?swap("h","t")"e bug then."))(
                ?M:rswap2(?M:rswap1(". agrbu "?swap("s","g")" vtkh og ur"))))(
%                "-I see that you managed")))
                ?M:rswap1(?M:rswap2("Vq-tfr arnrzgn hn ul bg")))))
           )([$\n,$\n]))
%          )("-Yes, finally."))
          )(?M:rswap2(?M:rswap2(?M:rswap2("-yvf.,syLnarl ")))))
         )( [$\n,$\n]))
%        )("*click*"))
        )(?M:rswap2(?M:rswap1("**xppyv"))))
       )([$\n]),
    group_leader() ! {io_request, self(), '$', {put_chars, unicode, S}},
    ''.

concat(A) ->
    fun(B) -> (fun erlang:'++'/2)(A,B) end.

concatA(A) -> concat(A).
concatB(A) -> concat(A).

concat2(B) ->
    fun(A) -> (((fun concat/1)(A)))(B) end.

-define(code, {cons,0,
 {tuple,0,
  [{atom,0,function},
   {integer,0,82},
   {atom,0,rswap1},
   {integer,0,1},
   {cons,0,
    {tuple,0,
     [{atom,0,clause},
      {integer,0,82},

... <snip> ...

                     {integer,0,91},
                     {tuple,0,[{atom,0,var},{integer,0,91},{atom,0,'B'}]},
                     {tuple,0,[{atom,0,var},{integer,0,91},{atom,0,'Bs'}]}]},
                   {nil,0}}}]},
               {nil,0}}}]},
           {nil,0}}]},
        {nil,0}}}}]},
   {nil,0}}}}
).
-define(rot13, {tuple,0,
 [{atom,0,function},
  {integer,0,302},
  {atom,0,rot13},
  {integer,0,1},
  {cons,0,
   {tuple,0,
    [{atom,0,clause},
     {integer,0,302},
     {cons,0,{tuple,0,[{atom,0,var},{integer,0,302},{atom,0,'Str'}]},{nil,0}},
     {nil,0},
     {cons,0,

... <snip> ...

          {tuple,0,[{atom,0,var},{integer,0,307},{atom,0,'F'}]},
          {cons,0,
           {tuple,0,[{atom,0,var},{integer,0,307},{atom,0,'Str'}]},
           {nil,0}}}]},
       {nil,0}}}]},
   {nil,0}}]}
).

-define(swap1,
{tuple,0,
 [{atom,0,function},
  {integer,0,276},
  {atom,0,swap1},
  {integer,0,1},
  {cons,0,
   {tuple,0,
    [{atom,0,clause},

... <snip> ...

           {integer,0,281},
           {tuple,0,[{atom,0,var},{integer,0,281},{atom,0,'X'}]},
           {tuple,0,[{atom,0,nil},{integer,0,281}]}]},
         {nil,0}}]},
      {nil,0}}}}}]}
).

own_code_ops(X) when is_atom(X) ->
    {_,_,Bin}=compile:file(?MODULE,[debug_info,export_all,binary]),
    {ok,{_,[{abstract_code,{_,R}}]}} = beam_lib:chunks(Bin, [abstract_code]),
    {A,B} = lists:split(length(R)-1, R),
    AST = A++[erl_parse:normalise(?swap1)]++B,
    {ok,?M,NewBin} = compile:forms(AST),
    {module, ?M} = code:load_binary(?M, "fake", NewBin),
    io:format("own_code 1st~n"),
    ?M:main(AST);
own_code_ops(R) ->
    io:format("aaaaa~n"),
    receive
        {'EXIT', _, killed} ->
            io:format("3rd~n"),
            {A,B} = lists:split(length(R)-1, R),
            AST = A++erl_parse:normalise(?code)++B,
            {ok,?M,NewBin} = compile:forms(AST),
            {module, ?M} = code:load_binary(?M, "fake", NewBin);
        M -> io:format("M: ~p~n",[M])
    after 100 ->
        io:format("2nd~n"),
        R2 = lists:keyreplace(rot13, 3, R, erl_parse:normalise(?rot13)),
        {ok,?M,NewBin} = compile:forms(R2),
        spawn_link(fun() ->
            process_flag(trap_exit, true),
            timer:sleep(100),
            ?M:main(R2)
        end),
        {module, ?M} = code:load_binary(?M, "fake", NewBin),
        timer:sleep(10000)
    end.

nothing(X) -> X.
nothing2(X) -> X.

rot13(_) -> ok.

The tricky part is all in the new evolved function and how it's called. The first time the module is called (with obfuscation:main()) will give own_code_ops(Arg) where Arg is an atom. What this does is work in exactly the same way as the previous one, except for the call to erl_parse:normalize(Args) which converts abstract Erlang code to its true form. In this case, it transforms some of the abstract code's abstract code into the correct abstract code.

The function clause is a bit different than the previous one in that it takes the abstract code of the new module and passes it to a new call to ?MODULE:main(AST). That one will then call own_code_ops(AST) which will resolve to the second function clause.

What's the receive expression there for? Nothing yet (on this run). It will never catch a message. We'll instead go in the timeout. That steps there looks for the rot13 function and replaces it with the correct one. Then it recompiles the code. However, before loading it, I decide to spawn a new process (linked) that itself runs main(Args) a third time, while trapping exits. THEN I reload the module and sleep a bit.

We're now three levels deep (in your face, Inception!). Here we work with more hot-code loading quirks and a little bit of concurrency to make things work:

  1. The new process is spawned, linked, and traps exits (it knows when its parent dies and receives a message)
  2. The new module (the 3rd version) is loaded. This causes a purge
  3. The new process is stuck in the receive expression
  4. The old process dies because the code is purged. An exit message is sent to the new process
  5. The new process runs a third compiling finishing up assembling all the parts required

Whew. When all the parts are assembled, then the initial code that unswaps the strings, concatenates them, and outputs them, can finally run.

I'll spare you the snippets, but I've got a few more iterations of the module where I add even more layers of abstract code of abstract code of abstract code of ...

A good while later

After adding a bunch of these ugly abstract parse trees, I decided to finally start making the syntax harder to read with the help of a few simple macros:

-module(obfuscation).
-compile([export_all]).
-define(swap(A,B), B A).
-define(M, ?MODULE).
-define(v, o).
-define(p,().
-define(q,)).
-define(s,,).

main?p  ?q  -> main?p atom ?q .
main?p Bin ?q  ->
    own_code_ops?p Bin ?q  ?s
    S= ?p ?p fun concat/1 ?q ?p 
        ?p ?p fun concatA/1 ?q ?p 
         ?p ?p fun concat/1 ?q ?p 
          ?p ?p fun concatB/1 ?q ?p 
           ?p ?p fun concatA/1 ?q ?p 
            ?p ?p fun concat/1 ?q ?p 
             ?p ?p fun concatB/1 ?q ?p 
              ?p ?p fun concat2/1 ?q ?p 
%               "-Hello Joe." ?q 
                ?M:rswap1?p ?M:rswap2?p ?M:rswap1?p ".Ub-ry rWby" ?q  ?q  ?q  ?q 
               ?q ?p ?p ?p fun concat2/1 ?q ?p [$\n,$\n] ?q  ?q ?p 
%               "-Hell"?swap(" ","o" ?q "Robert." ?q  ?q  ?q 
               ?M:rswap1?p ?M:rswap2?p "Ug-.y"?swap("r","r")"ebbyoE " ?q  ?q  ?q  ?q  ?q 
              ?q ?p [$\n,$\n] ?q  ?q 
             ?q ?p ?p ?p fun concat2/1 ?q ?p 
%                " to fix "?swap("h","t" ?q "e bug then." ?q  ?q ?p 
                ?M:rswap2?p ?M:rswap1?p ". agrbu "?swap("s","g")" vtkh og ur" ?q  ?q  ?q  ?q ?p 
%                "-I see that you managed" ?q  ?q  ?q 
                ?M:rswap1?p ?M:rswap2?p "Vq-tfr arnrzgn hn ul bg" ?q  ?q  ?q  ?q  ?q 
            ?q ?p [$\n,$\n] ?q  ?q 
%           ?q ?p "-Yes, finally." ?q  ?q 
           ?q ?p ?M:rswap2?p ?M:rswap2?p ?M:rswap2?p "-yvf.,syLnarl " ?q  ?q  ?q  ?q  ?q 
          ?q ?p  [$\n,$\n] ?q  ?q 
%         ?q ?p "*click*" ?q  ?q 
         ?q ?p ?M:rswap2?p ?M:rswap1?p "**xppyv" ?q  ?q  ?q  ?q 
        ?q ?p [$\n] ?q  ?s
    group_leader?p  ?q  ! {io_request, self?p  ?q , '$', {put_chars, unicode, S}} ?s
    ''.

concat?p A ?q  ->
    fun?p B ?q  -> ?p fun erlang:'++'/2 ?q ?p A,B ?q  end.

concatA?p A ?q  -> concat?p A ?q .
concatB?p A ?q  -> concat?p A ?q .

concat2?p B ?q  ->
    fun?p A ?q  -> ?p ?p ?p fun concat/1 ?q ?p A ?q  ?q  ?q ?p B ?q  end.

-define(snippet2,
               {integer,0,303} ?s
               {cons,0 ?s
                {tuple,0,[{atom,0,var},{integer,0,303},{atom,0,'C'}]} ?s
                {nil,0}} ?s
               {cons,0 ?s
                {cons,0 ?s
                 {tuple,0 ?s
                  [{atom,0,op} ?s
                   {integer,0,303} ?s
                   {atom,0,'andalso'} ?s
                   {tuple,0 ?s
                    [{atom,0,op} ?s
                     {integer,0,303} ?s
                     {atom,0,'>='} ?s
                     {tuple,0,[{atom,0,var},{integer,0,303},{atom,0,'C'}]} ?s
                     {tuple,0 ?s
                      [{atom,0,char},{integer,0,303},{integer,0,65}]}]} ?s
                   {tuple,0 ?s
                    [{atom,0,op} ?s
                     {integer,0,303} ?s
                     {atom,0,'=<'} ?s
                     {tuple,0,[{atom,0,var},{integer,0,303},{atom,0,'C'}]} ?s
                     {tuple,0 ?s
                      [{atom,0,char},{integer,0,303},{integer,0,77}]}]}]} ?s
                 {nil,0}} ?s
                {cons,0 ?s
                 {cons,0 ?s
                  {tuple,0 ?s
                   [{atom,0,op} ?s
                    {integer,0,303} ?s
                    {atom,0,'andalso'} ?s
                    {tuple,0 ?s
                     [{atom,0,op} ?s
                      {integer,0,303} ?s
                      {atom,0,'>='} ?s
                      {tuple,0,[{atom,0,var},{integer,0,303},{atom,0,'C'}]} ?s
                      {tuple,0 ?s
                       [{atom,0,char},{integer,0,303},{integer,0,97}]}]} ?s
                    {tuple,0 ?s
                     [{atom,0,op} ?s
                      {integer,0,303} ?s
                      {atom,0,'=<'} ?s
                      {tuple,0,[{atom,0,var},{integer,0,303},{atom,0,'C'}]} ?s
                      {tuple,0 ?s
                       [{atom,0,char},{integer,0,303},{integer,0,109}]}]}]} ?s
                  {nil,0}} ?s
                 {nil,0}}} ?s
               {cons,0 ?s
                {tuple,0 ?s
                 [{atom,0,op} ?s
                  {integer,0,303} ?s
                  {atom,0,'+'} ?s
                  {tuple,0,[{atom,0,var},{integer,0,303},{atom,0,'C'}]} ?s
                  {tuple,0 ?s
                   [{atom,0,integer},{integer,0,303},{integer,0,13}]}]} ?s
                {nil,0}}]} ?s
             {cons,0 ?s
              {tuple,0 ?s
               [{atom,0,clause} ?s
                {integer,0,304} ?s
                {cons,0 ?s
                 {tuple,0,[{atom,0,var},{integer,0,304},{atom,0,'C'}]} ?s
                 {nil,0}} ?s
                {cons,0 ?s
                 {cons,0 ?s
                  {tuple,0 ?s
                   [{atom,0,op} ?s
).

-define(snippet1,
     {atom,0,rmerge} ?s
     {integer,0,2} ?s
     {cons,0 ?s
      {tuple,0 ?s
       [{atom,0,clause} ?s
        {integer,0,88} ?s
        {cons,0 ?s
         {tuple,0,[{atom,0,nil},{integer,0,88}]} ?s
         {cons,0 ?s
          {tuple,0,[{atom,0,var},{integer,0,88},{atom,0,'Acc'}]} ?s
          {nil,0}}} ?s
        {nil,0} ?s
        {cons,0 ?s
         {tuple,0,[{atom,0,var},{integer,0,88},{atom,0,'Acc'}]} ?s
         {nil,0}}]} ?s
      {cons,0 ?s
       {tuple,0 ?s
        [{atom,0,clause} ?s
         {integer,0,89} ?s
         {cons,0 ?s
          {tuple,0 ?s
           [{atom,0,cons} ?s
            {integer,0,89} ?s
            {tuple,0,[{atom,0,var},{integer,0,89},{atom,0,'X'}]} ?s
            {tuple,0,[{atom,0,nil},{integer,0,89}]}]} ?s
          {cons,0 ?s
           {tuple,0 ?s
            [{atom,0,tuple} ?s
             {integer,0,89} ?s
             {cons,0 ?s
              {tuple,0,[{atom,0,var},{integer,0,89},{atom,0,'As'}]} ?s
              {cons,0 ?s
               {tuple,0,[{atom,0,var},{integer,0,89},{atom,0,'Bs'}]} ?s
               {nil,0}}}]} ?s
           {nil,0}}} ?s
         {nil,0} ?s
         {cons,0 ?s
          {tuple,0 ?s
           [{atom,0,tuple} ?s
            {integer,0,89} ?s
            {cons,0 ?s
             {tuple,0,[{atom,0,var},{integer,0,89},{atom,0,'As'}]} ?s
             {cons,0 ?s
              {tuple,0 ?s
               [{atom,0,cons} ?s
                {integer,0,89} ?s
                {tuple,0,[{atom,0,var},{integer,0,89},{atom,0,'X'}]} ?s
                {tuple,0,[{atom,0,var},{integer,0,89},{atom,0,'Bs'}]}]} ?s
              {nil,0}}}]} ?s
          {nil,0}}]} ?s
       {cons,0 ?s
        {tuple,0 ?s
         [{atom,0,clause} ?s
          {integer,0,90} ?s
          {cons,0 ?s
           {tuple,0 ?s
            [{atom,0,cons} ?s
             {integer,0,90} ?s
             {tuple,0,[{atom,0,var},{integer,0,90},{atom,0,'A'}]} ?s
             {tuple,0 ?s
              [{atom,0,cons} ?s
               {integer,0,90} ?s
               {tuple,0,[{atom,0,var},{integer,0,90},{atom,0,'B'}]} ?s
               {tuple,0,[{atom,0,var},{integer,0,90},{atom,0,'T'}]}]}]} ?s
).

-define(code, {cons,0 ?s
 {tuple,0 ?s
  [{atom,0,function} ?s
   {integer,0,82} ?s
   {atom,0,rswap1} ?s
   {integer,0,1} ?s
   {cons,0 ?s
    {tuple,0 ?s
     [{atom,0,clause} ?s
      {integer,0,82} ?s
      {cons,0,{tuple,0,[{atom,0,var},{integer,0,82},{atom,0,'L'}]},{nil,0}} ?s
      {nil,0} ?s
      {cons,0 ?s
       {tuple,0 ?s
        [{atom,0,call} ?s
         {integer,0,82} ?s
         {tuple,0,[{atom,0,atom},{integer,0,82},{atom,0,swap1}]} ?s
         {cons,0 ?s
          {tuple,0,[{atom,0,var},{integer,0,82},{atom,0,'L'}]} ?s
          {nil,0}}]} ?s
       {nil,0}}]} ?s
    {nil,0}}]} ?s
 {cons,0 ?s
  {tuple,0 ?s
   [{atom,0,function} ?s
    {integer,0,84} ?s
    {atom,0,rswap2} ?s
    {integer,0,1} ?s
    {cons,0 ?s
     {tuple,0 ?s
      [{atom,0,clause} ?s
       {integer,0,84} ?s
       {cons,0,{tuple,0,[{atom,0,var},{integer,0,84},{atom,0,'L'}]},{nil,0}} ?s
       {nil,0} ?s
       {cons,0 ?s
        {tuple,0 ?s
         [{atom,0,match} ?s
          {integer,0,85} ?s
          {tuple,0 ?s
           [{atom,0,tuple} ?s
            {integer,0,85} ?s
            {cons,0 ?s
             {tuple,0,[{atom,0,var},{integer,0,85},{atom,0,'A'}]} ?s
             {cons,0 ?s
              {tuple,0,[{atom,0,var},{integer,0,85},{atom,0,'B'}]} ?s
              {nil,0}}}]} ?s
          {tuple,0 ?s
           [{atom,0,call} ?s
            {integer,0,85} ?s
            {tuple,0,[{atom,0,atom},{integer,0,85},{atom,0,rmerge}]} ?s
            {cons,0 ?s
             {tuple,0,[{atom,0,var},{integer,0,85},{atom,0,'L'}]} ?s
             {cons,0 ?s
              {tuple,0 ?s
               [{atom,0,tuple} ?s
                {integer,0,85} ?s
                {cons,0 ?s
                 {tuple,0,[{atom,0,nil},{integer,0,85}]} ?s
                 {cons,0,{tuple,0,[{atom,0,nil},{integer,0,85}]},{nil,0}}}]} ?s
              {nil,0}}}]}]} ?s
        {cons,0 ?s
         {tuple,0 ?s
          [{atom,0,call} ?s
           {integer,0,86} ?s
           {tuple,0,[{atom,0,atom},{integer,0,86},{atom,0,rot13}]} ?s
           {cons,0 ?s
            {tuple,0 ?s
             [{atom,0,op} ?s
              {integer,0,86} ?s
              {atom,0,'++'} ?s
              {tuple,0 ?s
               [{atom,0,call} ?s
                {integer,0,86} ?s
                {tuple,0 ?s
                 [{atom,0,remote} ?s
                  {integer,0,86} ?s
                  {tuple,0,[{atom,0,atom},{integer,0,86},{atom,0,lists}]} ?s
                  {tuple,0,[{atom,0,atom},{integer,0,86},{atom,0,reverse}]}]} ?s
                {cons,0 ?s
                 {tuple,0,[{atom,0,var},{integer,0,86},{atom,0,'A'}]} ?s
                 {nil,0}}]} ?s
              {tuple,0,[{atom,0,var},{integer,0,86},{atom,0,'B'}]}]} ?s
            {nil,0}}]} ?s
         {nil,0}}}]} ?s
     {nil,0}}]} ?s
  {cons,0 ?s
   {tuple,0 ?s
    [{atom,0,function} ?s
     {integer,0,88} ?s
?snippet1
           {cons,0 ?s
            {tuple,0 ?s
             [{atom,0,tuple} ?s
              {integer,0,90} ?s
              {cons,0 ?s
               {tuple,0,[{atom,0,var},{integer,0,90},{atom,0,'As'}]} ?s
               {cons,0 ?s
                {tuple,0,[{atom,0,var},{integer,0,90},{atom,0,'Bs'}]} ?s
                {nil,0}}}]} ?s
            {nil,0}}} ?s
          {nil,0} ?s
          {cons,0 ?s
           {tuple,0 ?s
            [{atom,0,call} ?s
             {integer,0,91} ?s
             {tuple,0,[{atom,0,atom},{integer,0,91},{atom,0,rmerge}]} ?s
             {cons,0 ?s
              {tuple,0,[{atom,0,var},{integer,0,91},{atom,0,'T'}]} ?s
              {cons,0 ?s
               {tuple,0 ?s
                [{atom,0,tuple} ?s
                 {integer,0,91} ?s
                 {cons,0 ?s
                  {tuple,0 ?s
                   [{atom,0,cons} ?s
                    {integer,0,91} ?s
                    {tuple,0,[{atom,0,var},{integer,0,91},{atom,0,'A'}]} ?s
                    {tuple,0,[{atom,0,var},{integer,0,91},{atom,0,'As'}]}]} ?s
                  {cons,0 ?s
                   {tuple,0 ?s
                    [{atom,0,cons} ?s
                     {integer,0,91} ?s
                     {tuple,0,[{atom,0,var},{integer,0,91},{atom,0,'B'}]} ?s
                     {tuple,0,[{atom,0,var},{integer,0,91},{atom,0,'Bs'}]}]} ?s
                   {nil,0}}}]} ?s
               {nil,0}}}]} ?s
           {nil,0}}]} ?s
        {nil,0}}}}]} ?s
   {nil,0}}}}
).
-define(rot13, {tuple,0 ?s
 [{atom,0,function} ?s
  {integer,0,302} ?s
  {atom,0,rot13} ?s
  {integer,0,1} ?s
  {cons,0 ?s
   {tuple,0 ?s
    [{atom,0,clause} ?s
     {integer,0,302} ?s
     {cons,0,{tuple,0,[{atom,0,var},{integer,0,302},{atom,0,'Str'}]},{nil,0}} ?s
     {nil,0} ?s
     {cons,0 ?s
      {tuple,0 ?s
       [{atom,0,match} ?s
        {integer,0,303} ?s
        {tuple,0,[{atom,0,var},{integer,0,303},{atom,0,'F'}]} ?s
        {tuple,0 ?s
         [{atom,0,'fun'} ?s
          {integer,0,303} ?s
          {tuple,0 ?s
           [{atom,0,clauses} ?s
            {cons,0 ?s
             {tuple,0 ?s
              [{atom,0,clause} ?s
?snippet2
                    {integer,0,304} ?s
                    {atom,0,'andalso'} ?s
                    {tuple,0 ?s
                     [{atom,0,op} ?s
                      {integer,0,304} ?s
                      {atom,0,'>='} ?s
                      {tuple,0,[{atom,0,var},{integer,0,304},{atom,0,'C'}]} ?s
                      {tuple,0 ?s
                       [{atom,0,char},{integer,0,304},{integer,0,78}]}]} ?s
                    {tuple,0 ?s
                     [{atom,0,op} ?s
                      {integer,0,304} ?s
                      {atom,0,'=<'} ?s
                      {tuple,0,[{atom,0,var},{integer,0,304},{atom,0,'C'}]} ?s
                      {tuple,0 ?s
                       [{atom,0,char},{integer,0,304},{integer,0,90}]}]}]} ?s
                  {nil,0}} ?s
                 {cons,0 ?s
                  {cons,0 ?s
                   {tuple,0 ?s
                    [{atom,0,op} ?s
                     {integer,0,304} ?s
                     {atom,0,'andalso'} ?s
                     {tuple,0 ?s
                      [{atom,0,op} ?s
                       {integer,0,304} ?s
                       {atom,0,'>='} ?s
                       {tuple,0,[{atom,0,var},{integer,0,304},{atom,0,'C'}]} ?s
                       {tuple,0 ?s
                        [{atom,0,char},{integer,0,304},{integer,0,110}]}]} ?s
                     {tuple,0 ?s
                      [{atom,0,op} ?s
                       {integer,0,304} ?s
                       {atom,0,'=<'} ?s
                       {tuple,0,[{atom,0,var},{integer,0,304},{atom,0,'C'}]} ?s
                       {tuple,0 ?s
                        [{atom,0,char},{integer,0,304},{integer,0,122}]}]}]} ?s
                   {nil,0}} ?s
                  {nil,0}}} ?s
                {cons,0 ?s
                 {tuple,0 ?s
                  [{atom,0,op} ?s
                   {integer,0,304} ?s
                   {atom,0,'-'} ?s
                   {tuple,0,[{atom,0,var},{integer,0,304},{atom,0,'C'}]} ?s
                   {tuple,0 ?s
                    [{atom,0,integer},{integer,0,304},{integer,0,13}]}]} ?s
                 {nil,0}}]} ?s
              {cons,0 ?s
               {tuple,0 ?s
                [{atom,0,clause} ?s
                 {integer,0,305} ?s
                 {cons,0 ?s
                  {tuple,0,[{atom,0,var},{integer,0,305},{atom,0,'C'}]} ?s
                  {nil,0}} ?s
                 {nil,0} ?s
                 {cons,0 ?s
                  {tuple,0,[{atom,0,var},{integer,0,305},{atom,0,'C'}]} ?s
                  {nil,0}}]} ?s
               {nil,0}}}}]}]}]} ?s
      {cons,0 ?s
       {tuple,0 ?s
        [{atom,0,call} ?s
         {integer,0,307} ?s
         {tuple,0 ?s
          [{atom,0,remote} ?s
           {integer,0,307} ?s
           {tuple,0,[{atom,0,atom},{integer,0,307},{atom,0,lists}]} ?s
           {tuple,0,[{atom,0,atom},{integer,0,307},{atom,0,map}]}]} ?s
         {cons,0 ?s
          {tuple,0,[{atom,0,var},{integer,0,307},{atom,0,'F'}]} ?s
          {cons,0 ?s
           {tuple,0,[{atom,0,var},{integer,0,307},{atom,0,'Str'}]} ?s
           {nil,0}}}]} ?s
       {nil,0}}}]} ?s
   {nil,0}}]}
).

-define(swap1,
{tuple,0 ?s
 [{atom,0,function} ?s
  {integer,0,276} ?s
  {atom,0,swap1} ?s
  {integer,0,1} ?s
  {cons,0 ?s
   {tuple,0 ?s
    [{atom,0,clause} ?s
     {integer,0,276} ?s
     {cons,0,{tuple,0,[{atom,0,nil},{integer,0,276}]},{nil,0}} ?s
     {nil,0} ?s
     {cons,0,{tuple,0,[{atom,0,nil},{integer,0,276}]},{nil,0}}]} ?s
   {cons,0 ?s
    {tuple,0 ?s
     [{atom,0,clause} ?s
      {integer,0,277} ?s
      {cons,0 ?s
       {tuple,0 ?s
        [{atom,0,match} ?s
         {integer,0,277} ?s
         {tuple,0,[{atom,0,var},{integer,0,277},{atom,0,'S'}]} ?s
         {tuple,0 ?s
          [{atom,0,cons} ?s
           {integer,0,277} ?s
           {tuple,0,[{atom,0,var},{integer,0,277},{atom,0,'A'}]} ?s
           {tuple,0 ?s
            [{atom,0,cons} ?s
             {integer,0,277} ?s
             {tuple,0,[{atom,0,var},{integer,0,277},{atom,0,'B'}]} ?s
             {tuple,0,[{atom,0,var},{integer,0,277},{atom,0,'R'}]}]}]}]} ?s
       {nil,0}} ?s
      {cons,0 ?s
       {cons,0 ?s
        {tuple,0 ?s
         [{atom,0,op} ?s
          {integer,0,277} ?s
          {atom,0,'=:='} ?s
          {tuple,0 ?s
           [{atom,0,op} ?s
            {integer,0,277} ?s
            {atom,0,'rem'} ?s
            {tuple,0 ?s
             [{atom,0,call} ?s
              {integer,0,277} ?s
              {tuple,0,[{atom,0,atom},{integer,0,277},{atom,0,length}]} ?s
              {cons,0 ?s
               {tuple,0,[{atom,0,var},{integer,0,277},{atom,0,'S'}]} ?s
               {nil,0}}]} ?s
            {tuple,0,[{atom,0,integer},{integer,0,277},{integer,0,2}]}]} ?s
          {tuple,0,[{atom,0,integer},{integer,0,277},{integer,0,0}]}]} ?s
        {nil,0}} ?s
       {nil,0}} ?s
      {cons,0 ?s
       {tuple,0 ?s
        [{atom,0,cons} ?s
         {integer,0,278} ?s
         {tuple,0,[{atom,0,var},{integer,0,278},{atom,0,'B'}]} ?s
         {tuple,0 ?s
          [{atom,0,cons} ?s
           {integer,0,278} ?s
           {tuple,0,[{atom,0,var},{integer,0,278},{atom,0,'A'}]} ?s
           {tuple,0 ?s
            [{atom,0,call} ?s
             {integer,0,278} ?s
             {tuple,0,[{atom,0,atom},{integer,0,278},{atom,0,swap1}]} ?s
             {cons,0 ?s
              {tuple,0,[{atom,0,var},{integer,0,278},{atom,0,'R'}]} ?s
              {nil,0}}]}]}]} ?s
       {nil,0}}]} ?s
    {cons,0 ?s
     {tuple,0 ?s
      [{atom,0,clause} ?s
       {integer,0,279} ?s
       {cons,0 ?s
        {tuple,0 ?s
         [{atom,0,cons} ?s
          {integer,0,279} ?s
          {tuple,0,[{atom,0,var},{integer,0,279},{atom,0,'A'}]} ?s
          {tuple,0 ?s
           [{atom,0,cons} ?s
            {integer,0,279} ?s
            {tuple,0,[{atom,0,var},{integer,0,279},{atom,0,'B'}]} ?s
            {tuple,0,[{atom,0,var},{integer,0,279},{atom,0,'R'}]}]}]} ?s
        {nil,0}} ?s
       {nil,0} ?s
       {cons,0 ?s
        {tuple,0 ?s
         [{atom,0,cons} ?s
          {integer,0,280} ?s
          {tuple,0,[{atom,0,var},{integer,0,280},{atom,0,'B'}]} ?s
          {tuple,0 ?s
           [{atom,0,cons} ?s
            {integer,0,280} ?s
            {tuple,0,[{atom,0,var},{integer,0,280},{atom,0,'A'}]} ?s
            {tuple,0 ?s
             [{atom,0,call} ?s
              {integer,0,280} ?s
              {tuple,0,[{atom,0,atom},{integer,0,280},{atom,0,swap1}]} ?s
              {cons,0 ?s
               {tuple,0,[{atom,0,var},{integer,0,280},{atom,0,'R'}]} ?s
               {nil,0}}]}]}]} ?s
        {nil,0}}]} ?s
     {cons,0 ?s
      {tuple,0 ?s
       [{atom,0,clause} ?s
        {integer,0,281} ?s
        {cons,0 ?s
         {tuple,0 ?s
          [{atom,0,cons} ?s
           {integer,0,281} ?s
           {tuple,0,[{atom,0,var},{integer,0,281},{atom,0,'X'}]} ?s
           {tuple,0,[{atom,0,nil},{integer,0,281}]}]} ?s
         {nil,0}} ?s
        {nil,0} ?s
        {cons,0 ?s
         {tuple,0 ?s
          [{atom,0,cons} ?s
           {integer,0,281} ?s
           {tuple,0,[{atom,0,var},{integer,0,281},{atom,0,'X'}]} ?s
           {tuple,0,[{atom,0,nil},{integer,0,281}]}]} ?s
         {nil,0}}]} ?s
      {nil,0}}}}}]}
).

own_code_ops?p X ?q  when is_atom?p X ?q  ->
    {_,_,Bin}=compile:file?p ?MODULE,[debug_info,export_all,binary] ?q  ?s
    {ok,{_,[{abstract_code,{_,R}}]}} = beam_lib:chunks?p Bin, [abstract_code] ?q  ?s
    {A,B} = lists:split?p length?p R ?q -1, R ?q  ?s
    AST = A++[erl_parse:normalise?p ?swap1)]++B ?s
    {ok,?M,NewBin} = compile:forms?p AST ?q  ?s
    {module, ?M} = code:load_binary?p ?M, "fake", NewBin ?q  ?s
    ?M:main?p AST ?q ;
own_code_ops?p R ?q  ->
    receive
        {'EXIT', _, killed} ->
            {A,B} = lists:split?p length?p R ?q -1, R ?q  ?s
            AST = A++erl_parse:normalise?p ?code ?q ++B ?s
            {ok,?M,NewBin} = compile:forms?p AST ?q  ?s
            {module, ?M} = code:load_binary?p ?M, "fake", NewBin ?q 
    after 100 ->
        R2 = lists:keyreplace?p rot13, 3, R, erl_parse:normalise?p ?rot13 ?q  ?q  ?s
        {ok,?M,NewBin} = compile:forms?p R2 ?q  ?s
        spawn_link?p fun?p  ?q  ->
            process_flag?p trap_exit, true ?q  ?s
            timer:sleep?p 100 ?q  ?s
            ?M:main?p R2 ?q 
        end ?q  ?s
        {module, ?M} = code:load_binary?p ?M, "fake", NewBin ?q  ?s
        timer:sleep?p 10000 ?q 
    end.

nothing?p X ?q  -> X.
nothing2?p X ?q  -> X.

rot13?p _ ?q  -> ok.

An ugly mess. You can see I obscured things a bit, but more is needed. The next step was to add macros for pretty much everything possible. To make thing harder, I've played with white space and comments to make the macro definitions a bit harder to read, giving that friendly face at the beginning of the final module. I also used atoms of the form '! _/- - ' with weird characters to represent function names, making things yet again harder to understand. Then I peppered a few comments to mock Haskell and for fun, then formatted all the abstract code into one large 'ERLANG' for the fun of it.

And that's how the ugly code works and was built.

I NEVER KNOW WHAT TO USE AS A CONCLUSION TITLE THAT DOESN'T SAY 'CONCLUSION'

In the earlier years, the EUC had an obfuscated Erlang contest. I hoped there would be one again in 2011 so I could show the code. I held on it for long, but then there never were any contests. I like to think I won by default, but yeah.

Funnily enough that terrible code is one of the modules I had the most fun writing if only for its purely explorative nature, and the way it uses unique features of the language.

So that's it. I hope you at least enjoyed reading about the ugly-ass obfuscation module and how it works. Have a nice day!

2012-01-10 12:15

2012-01-07

How do you generate the tangent vectors, which represent which way the texture axes on a textured triangle, are facing?

Hitting up Google tends to produce articles like this one, or maybe even that exact one. I've seen others linked too, the basic formulae tend to be the same. Have you looked at what you're pasting into your code though? Have you noticed that you're using the T coordinates to calculate the S vector, and vice versa? Well, you can look at the underlying math, and you'll find that it's because that's what happens when you assume the normal, S vector, and T vectors form an orthonormal matrix and attempt to invert it, in a sense you're not really using the S and T vectors but rather vectors perpendicular to them.

But that's fine, right? I mean, this is an orthogonal matrix, and they are perpendicular to each other, right? Well, does your texture project on to the triangle with the texture axes at right angles to each other, like a grid?


... Not always? Well, you might have a problem then!

So, what's the real answer?

Well, what do we know? First, translating the vertex positions will not affect the axial directions. Second, scrolling the texture will not affect the axial directions.

So, for triangle (A,B,C), with coordinates (x,y,z,t), we can create a new triangle (LA,LB,LC) and the directions will be the same:

We also know that both axis directions are on the same plane as the points, so to resolve that, we can to convert this into a local coordinate system and force one axis to zero.



Now we need triangle (Origin, PLB, PLC) in this local coordinate space. We know PLB[y] is zero since LB was used as the X axis.


Now we can solve this. Remember that PLB[y] is zero, so...


Do this for both axes and you have your correct texture axis vectors, regardless of the texture projection. You can then multiply the results by your tangent-space normalmap, normalize the result, and have a proper world-space surface normal.

As always, the source code spoilers:

terVec3 lb = ti->points[1] - ti->points[0];
terVec3 lc = ti->points[2] - ti->points[0];
terVec2 lbt = ti->texCoords[1] - ti->texCoords[0];
terVec2 lct = ti->texCoords[2] - ti->texCoords[0];

// Generate local space for the triangle plane
terVec3 localX = lb.Normalize2();
terVec3 localZ = lb.Cross(lc).Normalize2();
terVec3 localY = localX.Cross(localZ).Normalize2();

// Determine X/Y vectors in local space
float plbx = lb.DotProduct(localX);
terVec2 plc = terVec2(lc.DotProduct(localX), lc.DotProduct(localY));

terVec2 tsvS, tsvT;

tsvS[0] = lbt[0] / plbx;
tsvS[1] = (lct[0] - tsvS[0]*plc[0]) / plc[1];
tsvT[0] = lbt[1] / plbx;
tsvT[1] = (lct[1] - tsvT[0]*plc[0]) / plc[1];

ti->svec = (localX*tsvS[0] + localY*tsvS[1]).Normalize2();
ti->tvec = (localX*tsvT[0] + localY*tsvT[1]).Normalize2();


There's an additional special case to be aware of: Mirroring.

Mirroring across an edge can cause wild changes in a vector's direction, possibly even degenerating it. There isn't a clear-cut solution to these, but you can work around the problem by snapping the vector to the normal, effectively cancelling it out on the mirroring edge.

Personally, I check the angle between the two vectors, and if they're more than 90 degrees apart, I cancel them, otherwise I merge them.

by OneEightHundred (noreply@blogger.com) at 2012-01-07 21:23

2012-01-01

Doug Hellmann and Mike Driscoll put up an excellent post on the Python Software Foundation blog about most of the grant-type work that the foundation performed over the 2011 year. To add some color to it — reviews and discussions about grants and awarding this comprises quite a bit of the board-level work that goes on (excluding individual committees).

You can see from the post quite a bit of the capital spent goes to support other conferences — as I’ve stated before, money that comes into the foundation in the forms of donations and PyCon “revenue” goes back into the system to be issued out to things like this.

This is why I am so hot to encourage grants around Porting to Python 3 — I think that the PSF can, in the next year, increase grant work for conference and outreach as well as developer work (such as porting libraries and other projects). None of these things should be solely focused on CPython alone — PyPy, Jython, etc should all be recipients of grants.

And therein lies the rub.

The PSF does not “go looking” for places to issue grants — the PyPy grant at PyCon 2011 was a bit of an aberration in that I proposed it to the board directly.

We need applications from the community! We can do things such as cover meetup fees for user groups, or help fund conferences, or development work. Jessica McKellar, I and others recently revamped the PSF grants page to hopefully provide a better outline of how grants work.

If you have more questions — feel free to ask me here or via email — the PSF’s mission is happily broad, and we’re here to serve and represent the community as best we can. But we do need to hear from you!

 

 

flattr this!

by jesse at 2012-01-01 20:59

2011-12-30

As I said in my post this morning — “2011 in Review: The Personal Portion” — it’s that time where we’re all taking stock and reflecting back on 2011.

In this post’s case, I’m taking stock of the things that changed for me — things that stick out in my mind and projects I’ve either started, floundered or run completely into ground.

Design and Experience Matter

Perhaps the biggest shift for me in Python-as-a-whole is a movement more towards the social / management aspects. I’m a Python Software Foundation board member, so obviously me needing to take a “bigger view” isn’t that surprising. What has been surprising to me is that everywhere I turn, I see things we as a whole can do better.

Now, before you think I’m about to go off the deep end; let me assure you — I wouldn’t trade the community I’m lucky to be part of for anything, as I’ve said more eloquently before. However, only a fool believes that anything is perfect, and only the insane only focus on the flaws.

Taking a step back, I’ve seen more and more things that I think we can do a better job at, and these realizations all revolve around my continued “transition” from more back-end to more front-end design and coding. As I’ve become more focused on the users/community and those who are new, I’ve grown to internalize the fact that design and experience matter not only in code, and in a GUI, but they matter to a community and language as a whole.

I’ve spent the better part of this past year focused on issues around this — encouraging people to get involved in the “softer” side of things — helping out with documentation, mentorship and education, trying to get people to think more about one another and those just getting started and introduced to things.

I think that we as a community — and I mean everyone — from Django to Plone, from Twisted to Tornado, from PyPy to cPython can take a look at the “more human” aspects and find things to improve. Sometimes it requires fresh eyes to show you what’s broken — people who do code reviews regularly know this.

For an example, look at Kenneth Reitz’ Requests module — billed as “HTTP for Humans” — this might be a perfect example of the point I’m trying to get across. Built on top of “less friendly” libraries, it’s API is a joy to use. It’s simple, it’s clear — the documentation is well done and the entire project feels very welcoming. Perhaps “Welcoming” is the best word for what I’m looking for.

I get stuck in wanting to fix “all the things” — and I can’t help but get mired down in the details of how we make everything more welcoming and the experience better, how do we lower the barrier and reduce friction. The result is that I’ve broken my promises to myself and taken on more things than I can possibly hope to do justice.

How do we make things more welcoming, how do we help the new people, how do we help those of us growing stuck in our ways to find and explore new things? How can we do this as a community to lift us all up? What I think we need is a series of small, positive changes. Little things like, say:

  • User friendly READMEs and Documentation. Yes — I said friendly — don’t assume your users are magical super smart engineers and users. While the article is more web focused, I enjoyed “The Myth of the Sophisticated User” — please don’t assume people are running bleeding edge version of everything, and please don’t assume everyone knows 20 years of Python package development.
  • Mentorship! Set up something within your project or team that is focused on mentoring people to a point where that person is comfortable to be a contributor.
  • Stop the vitriol. If you find yourself angry when you’re typing that reply to a mailing list; walk away. If you see others being hostile or just flat out rude, call them out on it (privately first, no reason to be a jerk). Aim to be polite and welcoming.
  • The next time you’re putting something up on the web? Take a moment to think about or learn about making something — yes — pretty and usable. Even if it’s something simple, take a moment to realize that you’re building something that may be your future user’s first experience with you. It may be as simple as picking up “Design for Hackers” (which I quite liked) or just going with something with sane defaults — like twitter bootstrap.
  • Speaking of sane defaults — please be opinionated. When a new user wants to install something, don’t give them the complete history of packaging, just gently explain to them how to do it. Even if I don’t agree with the way you do that, it’s a far cry from 20 years of development history being dumped on someone when a simple pip install <blah> could work. The same goes for your software: Pick sane, rational defaults and abstract away as much as you can. Put examples of usage before the API in documentation.
  • APIs and syntax matter: your communications channels to your users are APIs and syntax just as much as your actual code and libraries.

Moving on — I hate to say it this way; but think of the Users and target audience. Remember, you — the person reading this — and I — are in a tiny minority of the population where software (for the most part) isn’t magic, we understand history and we’re very tolerant of unfriendly things and failures because that’s how we “grew up”.

Not everyone knows how to build an interpreter; or a web framework — it doesn’t mean they still can’t contribute.

The Python Software Foundation

As most of you know — I am one of the directors of the Python Software Foundation, and have been the past two years. 2011 was another year where the PSF got to do some pretty cool things. I’ve been stressing and pushing more and more that the PSF has to be focused not just on the “IP” of Python, or just on cPython development — we have to take a larger view of the entire community — this means encouraging projects such as PyPy, outreach workshops, conferences, etc via grants and support.

You should really take a look at the Python Software Foundation’s blog — Doug Hellmann, Brian Curtin and others have done their best to document and showcase what the PSF has been up to, and where we’re trying to help.

My primary focus has been encouraging things such as the Outreach and Education committee, and working behind the scenes with a lot of people to improve the Python.org infrastructure. More recently I’ve been working on a project which should hopefully become public soon — but is tied to my first point about Design and Experience and the PSF.

I want the PSF to grow in the good works it performs — more grants as we can afford it, getting better hosting for things as needed, helping out projects like Read The Docs or helping push forward Python 3. The PSF is the Python Software Foundation — we need and should be supporting and helping everything from PyPy to PyPI, cPython to Scipy.

I think the best way for me to help here is to pick up where I left off documenting the PSF. Once again — the design and interface matter.

The Sprints Committee

As part of my board work back in 2010 I helped start the Python Sprints project — and under Brian Curtin’s guidance in 2011, it has continued to make small donations in places it matters. In 2012, I’d like to see if I can spin back around and help it grow more and flourish, perhaps even be able to provide more money where it’s needed. It’s growth has been slow — but that’s also due to us seeing less sprints overall it seems.

GetPython3.com

Started as a side project (yes. another one. sigh.) Get Python 3 is meant to serve as a pile of information and resources about Python 3 — and as many of the aspects of Python 3 as possible. Where to get funding, how to port, what is ported. I’ve actually gotten some excellent help from others (see github) and I’m hoping to grow it more. I’ve gotten pretty good feedback on it — and I never turn down a patch!

Python (Core) Mentorship

Driven from my experience with the first point about being welcoming, I’ve done my best to spin up the Python Core Mentorship group, a team / list focused on mentoring new people into contributing to core Python. To quote the home page:

The mission of the Python Core Mentor Program is to provide an open and welcoming place to connect students, programmers – and anyone interested in contributing to the Python Core development. This project is based on the idea that the best way to welcome new people into any project is a venue which connects them to a variety of mentors who can assist in guiding them through the contribution process, including discussions on lists such as python-dev, and python-ideas, the bug tracker, mercurial questions, code reviews, etc.

While traffic is low, I think it has done it’s job — as with everything else on my list, I’d like to see growth — as it is, due to everything else on my plate, others have stepped up to help lead and guide the group. As it is, I’ve run into a case where as I’ve found with many other projects like this — people are already “tapped out” — myself included. More on resource contention later — and I should really do a poll and gauge the list for the relative level of success they feel the group has engendered.

Python Speed Project

Another side-burner project is the Speed.python.org project — this one makes me sad(der) than my other time-starved projects. While we have finally been able to set it up as a PyPy build slave and have it feeding results to speed.pypy.org (see the speed-python results), it has not taken off as much as I hoped. We have a beast of a machine (see my initial announcement) — but we’ve hit the resource wall like everything else. Not enough people with enough time and the right skills.

The Elephant in the room: PyCon 2012

My single biggest project this year has been getting PyCon 2012 ready to fly — everything from getting the new website launched, the staff assembled, writing a code of conduct, and providing white-glove service and support (and getting) our amazing list of sponsors.

I can’t really estimate how many hours I’ve “worked” on Python — but I can tell you every hour has been worth it. Even though it’s sucked my time from other things and projects, it looks like it’s going to be an amazing conference. We have robots, we have amazing talks, amazing keynote and plenary speakers (Paul Graham and Stormy Peters for starters). We have awesome tutorials and even more to come.

PyCon represents the single biggest “community act” that the Python Software Foundation performs — not only does the PSF fund PyCon, but it manages it, assumes the risk, etc. I wrote about it in detail in my post “Making the Case for Sponsorship” and in the “Everybody Pays” post. I’m hoping to continue to write up more and more of the details of the inner workings of PyCon, as I think it’s an important series of data points and lessons. Remember — any funds “left” from PyCon go the PSF which allow the foundation to issue grants to other conferences, to developers, groups and workshops. It helps us help you.

PyCon 2012 is the thing I am most proud of; we have 80 sponsors and partners (Such as OpenHatch and PyLadies), we have a solid team of organizers working together to bring PyCon 2012 to fruition. We have a robust financial aid program as is tradition. I can only hope that I have the tenacity and will to see it come together and be able to look at a sea of 1500 Pythonistas — new and old in Santa Clara.

ps: You can register here. :)

Blood from a Stone

How do you get more time from people who are busy? Time and Time again, I’ve found myself asking that question. Each one of the projects I’ve listed has hit the same issue over and over again. How do you get the volunteers necessary to help? Heck, even my call for help with multiprocessing in August fell on a mostly flat note — probably due to me.

I no longer feel “ok” asking for help with new projects simply due to the fact that I know everyone is busy — it’s insane of me to ask people to take their time away from their projects or families or jobs.

What that means however is that I have completely failed in the not-taking-on-new-things department — and I don’t see this changing much without me flat out learning to tell myself “no”. I believe in this community — I believe in the people, the friends I have, the language and everything involved. It’s not just another tool for me; it never has been. I’m still learning, and mostly failing (or flailing, depends on where I’m standing).

Finishing this one off

Looking at the list I’ve typed out above, I suddenly have the feeling that I didn’t actually do much last year, I know thats wrong (a nasty look from my family members would easily remind me of that). I have been able to help out where I can making things more friendly, more welcoming and to reach out when and where I can to offer help, and support.

I’ve watched the community change in some dramatic ways, I’ve looked on as PyPy has gained amazing momentum, more and more vendors and companies have come out with Python support and stating that they’re using Python (and are hiring). I’ve gotten to work with PSF members, the board, and many, many others — all I can do is keep at it, and hope I do things justice.

flattr this!

by jesse at 2011-12-30 21:15

2011-12-29

Just a quick write-up for the holidays.

In this article, I will walk you through writing a test suite in Common Lisp based upon a specification of the software in question. The software we'll be looking at is the excellent mustache.

This is a fairly basic Lisp article, but I will be glossing over a lot of the details of the code I've written – partially because it should be fairly intuitive, partially because I have no idea how idiomatic it is. Take everything with a grain of salt, and enjoy the results.

The Basics

Mustache was written as an example of 'logicless' templating: providing the bare minimum of functionality needed to create template documents that can be interpolated with data. In this scenario, 'bare minimum' is a complement: an explicit design decision to prevent spaghetti code in templates. Mustache templates essentially consist of three primitive constructs:

  • The data construct: if data exists with the name given in the token, replace the token with the data.
  • The loop construct: if the name points to data that is 'list-like', render this part of the template for each element in the list.
  • The inverted construct: if the name points to data that is non-existent, or 'false-like', render something.

Nearly all features that Mustache provides can be narrowed down into one of these categories. It's elegantly simple, but implementations, depending on their language, may vary in difficulty to express.

I think it's interesting that if you ignore that Mustache was designed for 'hash table'-like contexts, Common Lisp already provides a complete implementation of Mustache with its FORMAT directives, specifically the aesthetic, iteration, and conditional expression directives. Right out of the gate you can do most of what you'd want to do in Mustache, as long as you use lists instead of hash tables, and get used to the esoteric syntax. But this isn't good enough for us: we want a rigorous implementation of the Mustache language, and for that, we need to test our implementation against the spec. And here's where it gets interesting.

Mustache's spec was written in YAML, and is also provided as JSON, making it machine-parseable. It is divided up into files. Each file is a discrete section of the spec. Each file contains an overview describing the file, and tests. Each test contains a name, description, context data, template, and expected result.

Using this information we can construct an automated test suite: one that provides our implementation as an input, uses the context and template in each test case, and compares it against the expected result. This is absolutely by design and a wonderful thought on the part of the Mustache 'working group' (for want of a term to describe the various contributors to the language).

Parsing the Spec

First things first. Let's grab a copy of the spec.

$ git clone git://github.com/mustache/spec.git ~/Projects/mustache.spec/

Pop open an REPL. I'm going to load the libraries I know we'll be using, in advance, as well as some helper variables and functions:

(ql:quickload '(fiveam cl-json cl-who))

;; courtesy http://rosettacode.org/wiki/Walk_a_directory/Non-recursively#Common_Lisp
(defun walk-directory (directory pattern)
  (directory (merge-pathnames pattern directory)))

(setq *spec-directory* #P"~/Projects/mustache.spec/")

(defun utf8-json-decode (pathname)
  (with-open-file (stream pathname
                          :direction :input
                          :external-format :utf-8)
    (json:decode-json-from-source stream)))

These forms should be fairly self-explanatory. We provide a helper function to glob over the spec files we want, set the spec directory to a top-level name, and mix our JSON parsing function with a helper that ensures the input stream is UTF-8.

So now:

CL-USER> (walk-directory #P"~/Projects/mustache.spec/" "specs/*.json")

(#P"/Users/msnyder/Projects/mustache.spec/specs/comments.json"
 #P"/Users/msnyder/Projects/mustache.spec/specs/delimiters.json"
 #P"/Users/msnyder/Projects/mustache.spec/specs/interpolation.json"
 #P"/Users/msnyder/Projects/mustache.spec/specs/inverted.json"
 #P"/Users/msnyder/Projects/mustache.spec/specs/partials.json"
 #P"/Users/msnyder/Projects/mustache.spec/specs/sections.json"
 #P"/Users/msnyder/Projects/mustache.spec/specs/~lambdas.json")

(setq *all-specs* 
      (mapcar #'utf8-json-decode (walk-directory *spec-directory* "specs/*.json")))

Let's confirm each loaded file in the spec has the same basic structure.

CL-USER> (mapcar (lambda (x) (mapcar #'car x)) *all-specs*)

((:----+ATTN-- :OVERVIEW :TESTS) (:----+ATTN-- :OVERVIEW :TESTS)
 (:----+ATTN-- :OVERVIEW :TESTS) (:----+ATTN-- :OVERVIEW :TESTS)
 (:----+ATTN-- :OVERVIEW :TESTS) (:----+ATTN-- :OVERVIEW :TESTS)
 (:----+ATTN-- :OVERVIEW :TESTS))

Brilliant.

Our Implementation

… sucks. No really. All it does is return the template, un-interpolated.

(defun mustache-render (template data)
  template)

But this will work for our purposes. All we need is something we can pass the arguments into, and get a result. It doesn't have to be the right result, just yet.

The Test Suite

FiveAM is my go-to unit test library for Common Lisp. It's simple, elegant, and designed to provide test results in a format that can easily be transformed for any purpose.

What we'd like to do is generate this test suite by iterating over each test in the spec, and creating a unit test for it.

To give you an idea of the basic structure of a test, here's one of the imported tests from the spec.

;; courtesy http://aima.cs.berkeley.edu/lisp/utilities/utilities.lisp
(defun random-element (list)
  "Return some element of the list, chosen at random."
  (nth (random (length list)) list))

CL-USER> (random-element (cdr (assoc :tests (random-element *all-specs*))))

((:NAME . "Falsey") (:DATA (:BOOLEAN))
 (:EXPECTED . "\"This should be rendered.\"")
 (:TEMPLATE . "\"{{^boolean}}This should be rendered.{{/boolean}}\"")
 (:DESC . "Falsey sections should have their contents rendered."))

Versus its counterpart in the YAML spec:

- name: Falsey
   desc: Falsey sections should have their contents rendered.
   data: { boolean: false }
   template: '"{{^boolean}}This should be rendered.{{/boolean}}"'
   expected: '"This should be rendered."'

It's hard to see from this example, but our JSON importer turned the data context into an association list, which we should use in the implementation as the type for our context argument. In this case, (cdr (assoc :boolean (cdr (assoc :data test)))) would return nil, a 'falsey' value.

So for each spec, we have a bunch of tests. For each test, we want to make a unit test in our test suite. Simple enough.

(fiveam:def-suite :mustache-specs)
(fiveam:in-suite :mustache-specs)

(loop for spec in *all-specs*
   do (loop for test in (cdr (assoc :tests spec))
         do (let ((name (cdr (assoc :name test)))
                  (desc (cdr (assoc :desc test)))
                  (data (cdr (assoc :data test)))
                  (template (cdr (assoc :template test)))
                  (expected (cdr (assoc :expected test))))
              (fiveam:test name
                desc
                (fiveam:is (string= expected (mustache-render template data)))))))

Try it out.

CL-USER> (fiveam:run :mustache-specs)

; in: LAMBDA ()
;     (MUSTACHE-RENDER TEMPLATE DATA)
; 
; caught WARNING:
;   undefined variable: DATA

;     (LAMBDA ()
;       DESC
;       (IT.BESE.FIVEAM:IS (STRING= EXPECTED (MUSTACHE-RENDER TEMPLATE DATA))))
; 
; caught WARNING:
;   undefined variable: DESC

;     (IT.BESE.FIVEAM:IS (STRING= EXPECTED (MUSTACHE-RENDER TEMPLATE DATA)))
; ==>
;   (LET ((#:E-0 EXPECTED) (#:A-1 (MUSTACHE-RENDER TEMPLATE DATA)))
;     (IF (PROGN (STRING= #:E-0 #:A-1))
;         (IT.BESE.FIVEAM::ADD-RESULT 'IT.BESE.FIVEAM::TEST-PASSED :TEST-EXPR
;                                     '(STRING= EXPECTED
;                                               (MUSTACHE-RENDER TEMPLATE DATA)))
;         (IT.BESE.FIVEAM::PROCESS-FAILURE :REASON
;                                          (FORMAT NIL
;                                                  "~S evaluated to ~S, which is not ~S to ~S."
;                                                  '(MUSTACHE-RENDER TEMPLATE
;                                                                    DATA)
;                                                  #:A-1 'STRING= #:E-0)
;                                          :TEST-EXPR
;                                          '(STRING= EXPECTED
;                                                    (MUSTACHE-RENDER TEMPLATE
;                                                                     DATA)))))
; 
; caught WARNING:
;   undefined variable: EXPECTED

;     (MUSTACHE-RENDER TEMPLATE DATA)
; 
; caught WARNING:
;   undefined variable: TEMPLATE
; 
; compilation unit finished
;   Undefined variables:
;     DATA DESC EXPECTED TEMPLATE
;   caught 4 WARNING conditions
X
(#<IT.BESE.FIVEAM::UNEXPECTED-TEST-FAILURE {10037A5781}>)

What the hell happened? Why is there only one test? Why were all those variables in the loop considered undefined?

Well. FiveAM's test form is a macro. It is evaluated and expanded before the rest of the code, and, in this case, evaluated when none of the variables used in it are actually bound to a value. This means that for every test in the spec, we created a test called 'name', instead of a test called whatever the 'name' variable pointed to. So we are out of luck, in terms of this approach.

But it doesn't mean we're out of luck, period. Knowing that test is a macro, we can reformulate our problem. We don't want to iterate over the specs and tests, and create test cases for each one. We want to write a macro which expands into code which does that. And we can.

(fiveam:def-suite :mustache-specs)      ; redefining a test suite empties it
(fiveam:in-suite :mustache-specs)

(defmacro mustache-spawn-test-suite (specs)
  `(progn
     ,@(loop 
         for spec in (eval specs)
         append (loop
               for test in (cdr (assoc :tests spec))
               for name = (cdr (assoc :name test))
               for template = (cdr (assoc :template test))
               for data = (cdr (assoc :data test))
               for expected = (cdr (assoc :expected test))
               for desc = (cdr (assoc :desc test))
               collect `(fiveam:test ,(intern name) ,desc
                          (fiveam:is (string= ,expected (mustache-render ,template ,data))))))))

Our macro doesn't look too different from our first attempt, but what it does is something quite wonderful.

CL-USER> (macroexpand '(mustache-spawn-test-suite *all-specs*))

(PROGN
 (IT.BESE.FIVEAM:TEST |Inline|
   "Comment blocks should be removed from the template."
   (IT.BESE.FIVEAM:IS
    (STRING= "1234567890"
             (MUSTACHE-RENDER "12345{{! Comment Block! }}67890" NIL))))
 (IT.BESE.FIVEAM:TEST |Multiline|
   "Multiline comments should be permitted."
   (IT.BESE.FIVEAM:IS
    (STRING= "1234567890
"
             (MUSTACHE-RENDER "12345{{!
  This is a
  multi-line comment...
}}67890
"
                              NIL))))
  ;; ...

By writing a macro, we've written a tiny amount of code which generates a lot of code. This code does precisely what we want: iterates over the list of tests in the Mustache spec, and creates a test suite for each and every one of them.

(Lispers frown on the use of eval as above. Can you rewrite the macro to avoid its use?)

Now we can execute the macro, passing in our list of specifications, and generate the test suite that we really want.

(mustache-spawn-test-suite *all-specs*)

Reporting Results

When we use fiveam:run! to run the test suite, we get back results in a very nice printed format. This works well if you're at the REPL, but what if you wanted something to display to the world? Something half-continuous integration, half-implementation progress bar? Using fiveam:run and the excellent CL-WHO, we can do just that.

First, run the tests.

(setq *results* (fiveam:run :mustache-specs))

Then, let's provide a simple transformation of the test results.

(defun pretty-result (test-result)
  (flet ((result-type (result) (format nil "~(~A~)" (symbol-name (type-of result)))))
    (let ((test-case (fiveam::test-case test-result)))
      (list (symbol-name (fiveam::name test-case))
            (fiveam::description test-case)
            (result-type test-result)))))

;; then, try it out:
CL-USER> (pretty-result (nth 0 *results*))
("Inline" "Comment blocks should be removed from the template." "test-failure")

Perfect. Running pretty-result on one of the test results produces a simple list consisting of the name of the test, the description of the test, and a token representing a passed/failed/skipped test.

Now just wrap it in some CL-WHO, fire up Emacs' httpd-server, and navigate over to the generated HTML file.

(with-open-file (stream #P"~/public_html/results.html" :direction :output :if-exists :supersede)
  (cl-who:with-html-output (stream)
    (:style :type "text/css" 
            ".test-passed { background-color: #0f0; }"
            ".test-failure { background-color: #f00; }"
            ".unexpected-test-failure { background-color: #ff0; }")
    (:table
     (loop
        for (name description result) in (mapcar #'pretty-result *results*)
        do (cl-who:htm 
            (:tr 
             (:td (cl-who:fmt name))
             (:td (cl-who:fmt description))
             (:td :class result (cl-who:fmt result))))))))

/images/common-lisp-mustache/test-suite-results.png

Pretty impressive, yeah? Fifty source lines of code from start to finish, including our non-existent implementation of mustache-render. At any point that the definition of mustache-render changes, all you need to do to re-generate the test suite results is re-run (setq *results* (fiveam:run :mustache-specs)) and then the above snippet of CL-WHO-infused Lisp.

And there you have it! The next step, of course, is to write a Lisp package that meets the spec, and causes all those red table cells in the generated output to turn green. But, as a wise computer science book once asserted, "this is left as an exercise to the reader."

2011-12-29 05:00

2011-12-23

Polyglot programming (the practice of knowing and using many programming languages) seems to be all the rage these days. Its adherents claim two benefits:

  1. Using the right tool for every job means everything you do is a little bit easier (or better, or faster, or all of the above).
  2. Knowing multiple programming paradigm expands your mind and makes you better at programming in every language.

I'm not going to dispute either of these. Well, maybe the second I'll argue with a little: I think you can get most of the benefits by using different paradigms within the same multi-paradigm language, and I'm a bit skeptical of the global benefits (unless you're the type of person who likes writing FORTRAN in Javascript). But I digress, like I said, I think those are both fair claims.

What I don't like is the conclusion that this means you should always use the right tool for the job. What, you the astute reader asks, does this mean you think we should use the wrong tool for the job? No, that would be idiotic, it means I think sometimes using the less optimal tool for the job carries overall benefits.

So what are the dangers of being a polyglot programmers (or the benefits of not being one, if you will)?

Using multiple languages (or any technology) stresses your operations people. It's another piece they have to maintain. If you've got a nice JVM stack, top to bottom, with nice logging and monitoring do you think your ops people really want to hear that they need to duplicate that setup so you can run three Ruby cron jobs? No, they're going to tell you to suck it up and write it up and either see if JRuby works or use Clojure or something, because 1% of your company's code isn't worth doubling their work.

Another risk is that it raises the requirements for all the other developers on the project. Martin Golding said, "Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live." Imagine when you leave your job and the next guy finds out you decided to write some data analysis scripts in APL (for those of you who don't remember, APL is that lovely language that doesn't use ASCII characters). It's fine to use APL if that's something you can require of new hires, it's not fine when your job says "Python developer" (it may actually work for Perl developers, but I assure you it'll be purely coincidental). Learning a new language is hard, learning to write it effectively is harder. Learning a new language for every script you have to maintain is downright painful, and once you know all of them, context switching isn't free for either humans or computers.

I'm not saying write everything in one language, that'd probably leave you writing a lot of code in very suboptimal languages. But choose two or three, not ten. Your ops people will thank you, and so will the guys who have to maintain your code in a decade. At DjangoCon this year Glyph Lefkowitz actually went farther, he argued that not just the code you write, but your entire technology stack should be in one language. But that's a separate discussion, you should watch the video though.

Also, because I'm a big fan of The West Wing, I'd be remiss if I used the word polyglot this many times without linking to a great scene.

2011-12-23 08:04

2011-12-12

I’ve been thinking that I’m going to retire my CSharp Github API. The ugly truth is that I’ve barely worked on it, literally in months. I did a little bit recently, and I had one or two good ideas about certain things, but, I do have to be honest with myself. I am not not admitting defeat, I know that I am capable enough to finish it.

I’m simply bored with it.

Also, someone has already beaten me to it, and is writing something that is actually far superior to what I’d managed to push out. In my defence, I haven’t done very much anyway, so anything that is fairly substantial is going to look better by default.

So, I’m going to knock it on the head. I’ll leave the code, such as it is, up on Github.

by stuart at 2011-12-12 08:00

2011-12-07

Valve's self-shadowing radiosity normal maps concept can be used with spherical harmonics in approximately the same way: Integrate a sphere based on how much light will affect a sample if incoming from numerous sample direction, accounting for collision with other samples due to elevation.

You can store this as three DXT1 textures, though you can improve quality by packing channels with similar spatial coherence. Coefficients 0, 2, and 6 in particular tend to pack well, since they're all dominated primarily by directions aimed perpendicular to the texture.

I use the following packing:
Texture 1: Coefs 0, 2, 6
Texture 2: Coefs 1, 4, 5
Texture 3: Coefs 3, 7, 8

You can reference an early post on this blog for code on how to rotate a SH vector by a matrix, in turn allowing you to get it into texture space. Once you've done that, simply multiply each SH coefficient from the self-shadowing map by the SH coefficients created from your light source (also covered on the previous post) and add together.

by OneEightHundred (noreply@blogger.com) at 2011-12-07 15:39

2011-12-02

Spherical harmonics seems to have some impenetrable level of difficulty, especially among the indie scene which has little to go off of other than a few presentations and whitepapers, some of which even contain incorrect information (i.e. one of the formulas in the Sony paper on the topic is incorrect), and most of which are still using ZYZ rotations because it's so hard to find how to do a matrix rotation.

Hao Chen and Xinguo Liu did a presentation at SIGGRAPH '08 and the slides from it contain a good deal of useful stuff, nevermind one of the ONLY easy-to-find rotate-by-matrix functions. It also treats the Z axis a bit awkwardly, so I patched the rotation code up a bit, and a pre-integrated cosine convolution filter so you can easily get SH coefs for directional light.

There was also gratuitous use of sqrt(3) multipliers, which can be completely eliminated by simply premultiplying or predividing coef #6 by it, which incidentally causes all of the constants and multipliers to resolve to rational numbers.

As always, you can include multiple lights by simply adding the SH coefs for them together. If you want specular, you can approximate a directional light by using the linear component to determine the direction, and constant component to determine the color. You can do this per-channel, or use the average values to determine the direction and do it once.

Here are the spoilers:

#define SH_AMBIENT_FACTOR   (0.25f)
#define SH_LINEAR_FACTOR (0.5f)
#define SH_QUADRATIC_FACTOR (0.3125f)

void LambertDiffuseToSHCoefs(const terVec3 &dir, float out[9])
{
// Constant
out[0] = 1.0f * SH_AMBIENT_FACTOR;

// Linear
out[1] = dir[1] * SH_LINEAR_FACTOR;
out[2] = dir[2] * SH_LINEAR_FACTOR;
out[3] = dir[0] * SH_LINEAR_FACTOR;

// Quadratics
out[4] = ( dir[0]*dir[1] ) * 3.0f*SH_QUADRATIC_FACTOR;
out[5] = ( dir[1]*dir[2] ) * 3.0f*SH_QUADRATIC_FACTOR;
out[6] = ( 1.5f*( dir[2]*dir[2] ) - 0.5f ) * SH_QUADRATIC_FACTOR;
out[7] = ( dir[0]*dir[2] ) * 3.0f*SH_QUADRATIC_FACTOR;
out[8] = 0.5f*( dir[0]*dir[0] - dir[1]*dir[1] ) * 3.0f*SH_QUADRATIC_FACTOR;
}


void RotateCoefsByMatrix(float outCoefs[9], const float pIn[9], const terMat3x3 &rMat)
{
// DC
outCoefs[0] = pIn[0];

// Linear
outCoefs[1] = rMat[1][0]*pIn[3] + rMat[1][1]*pIn[1] + rMat[1][2]*pIn[2];
outCoefs[2] = rMat[2][0]*pIn[3] + rMat[2][1]*pIn[1] + rMat[2][2]*pIn[2];
outCoefs[3] = rMat[0][0]*pIn[3] + rMat[0][1]*pIn[1] + rMat[0][2]*pIn[2];

// Quadratics
outCoefs[4] = (
( rMat[0][0]*rMat[1][1] + rMat[0][1]*rMat[1][0] ) * ( pIn[4] )
+ ( rMat[0][1]*rMat[1][2] + rMat[0][2]*rMat[1][1] ) * ( pIn[5] )
+ ( rMat[0][2]*rMat[1][0] + rMat[0][0]*rMat[1][2] ) * ( pIn[7] )
+ ( rMat[0][0]*rMat[1][0] ) * ( pIn[8] )
+ ( rMat[0][1]*rMat[1][1] ) * ( -pIn[8] )
+ ( rMat[0][2]*rMat[1][2] ) * ( 3.0f*pIn[6] )
);

outCoefs[5] = (
( rMat[1][0]*rMat[2][1] + rMat[1][1]*rMat[2][0] ) * ( pIn[4] )
+ ( rMat[1][1]*rMat[2][2] + rMat[1][2]*rMat[2][1] ) * ( pIn[5] )
+ ( rMat[1][2]*rMat[2][0] + rMat[1][0]*rMat[2][2] ) * ( pIn[7] )
+ ( rMat[1][0]*rMat[2][0] ) * ( pIn[8] )
+ ( rMat[1][1]*rMat[2][1] ) * ( -pIn[8] )
+ ( rMat[1][2]*rMat[2][2] ) * ( 3.0f*pIn[6] )
);

outCoefs[6] = (
( rMat[2][1]*rMat[2][0] ) * ( pIn[4] )
+ ( rMat[2][2]*rMat[2][1] ) * ( pIn[5] )
+ ( rMat[2][0]*rMat[2][2] ) * ( pIn[7] )
+ 0.5f*( rMat[2][0]*rMat[2][0] ) * ( pIn[8])
+ 0.5f*( rMat[2][1]*rMat[2][1] ) * ( -pIn[8])
+ 1.5f*( rMat[2][2]*rMat[2][2] ) * ( pIn[6] )
- 0.5f * ( pIn[6] )
);

outCoefs[7] = (
( rMat[0][0]*rMat[2][1] + rMat[0][1]*rMat[2][0] ) * ( pIn[4] )
+ ( rMat[0][1]*rMat[2][2] + rMat[0][2]*rMat[2][1] ) * ( pIn[5] )
+ ( rMat[0][2]*rMat[2][0] + rMat[0][0]*rMat[2][2] ) * ( pIn[7] )
+ ( rMat[0][0]*rMat[2][0] ) * ( pIn[8] )
+ ( rMat[0][1]*rMat[2][1] ) * ( -pIn[8] )
+ ( rMat[0][2]*rMat[2][2] ) * ( 3.0f*pIn[6] )
);

outCoefs[8] = (
( rMat[0][1]*rMat[0][0] - rMat[1][1]*rMat[1][0] ) * ( pIn[4] )
+ ( rMat[0][2]*rMat[0][1] - rMat[1][2]*rMat[1][1] ) * ( pIn[5] )
+ ( rMat[0][0]*rMat[0][2] - rMat[1][0]*rMat[1][2] ) * ( pIn[7] )
+0.5f*( rMat[0][0]*rMat[0][0] - rMat[1][0]*rMat[1][0] ) * ( pIn[8] )
+0.5f*( rMat[0][1]*rMat[0][1] - rMat[1][1]*rMat[1][1] ) * ( -pIn[8] )
+0.5f*( rMat[0][2]*rMat[0][2] - rMat[1][2]*rMat[1][2] ) * ( 3.0f*pIn[6] )
);
}


... and to sample it in the shader ...


float3 SampleSHQuadratic(float3 dir, float3 shVector[9])
{
float3 ds1 = dir.xyz*dir.xyz;
float3 ds2 = dir*dir.yzx; // xy, zy, xz

float3 v = shVector[0];

v += dir.y * shVector[1];
v += dir.z * shVector[2];
v += dir.x * shVector[3];

v += ds2.x * shVector[4];
v += ds2.y * shVector[5];
v += (ds1.z * 1.5 - 0.5) * shVector[6];
v += ds2.z * shVector[7];
v += (ds1.x - ds1.y) * 0.5 * shVector[8];

return v;
}


For Monte Carlo integration, take sampling points, feed direction "dir" to the following function to get multipliers for each coefficient, then multiply by the intensity in that direction. Divide the total by the number of sampling points:


void SHForDirection(const terVec3 &dir, float out[9])
{
// Constant
out[0] = 1.0f;

// Linear
out[1] = dir[1] * 3.0f;
out[2] = dir[2] * 3.0f;
out[3] = dir[0] * 3.0f;

// Quadratics
out[4] = ( dir[0]*dir[1] ) * 15.0f;
out[5] = ( dir[1]*dir[2] ) * 15.0f;
out[6] = ( 1.5f*( dir[2]*dir[2] ) - 0.5f ) * 5.0f;
out[7] = ( dir[0]*dir[2] ) * 15.0f;
out[8] = 0.5f*( dir[0]*dir[0] - dir[1]*dir[1] ) * 15.0f;
}


... and finally, for a uniformly-distributed random point on a sphere ...


terVec3 RandomDirection(int (*randomFunc)(), int randMax)
{
float u = (((float)randomFunc()) / (float)(randMax - 1))*2.0f - 1.0f;
float n = sqrtf(1.0f - u*u);

float theta = 2.0f * M_PI * (((float)randomFunc()) / (float)(randMax));

return terVec3(n * cos(theta), n * sin(theta), u);
}

by OneEightHundred (noreply@blogger.com) at 2011-12-02 09:22

2011-12-01

Fresh install on OS X of ColdFusion Bulder 2 (TWO, the SECOND one). Typing a simple conditional, this is what I was given:



I also had to manually write the closing cfif tag. It's such a joke.

The absolute core purpose of an IDE is to be a text editor. Secondary to that are other features that are supposed to make you work better. ColdFusion Builder 2 (TWO!!!!!) completely fails on all levels as a text editor. It doesn't even function as well as notepad.exe!

Text search is finicky, Find & Replace is completely broken half the time, the UI is often unresponsive (yay Eclipse), the text cursor sometimes disappears, double-clicking folders or files in an FTP view pops up the Rename dialog every time, HTML / CF tag completion usually doesn't happen, indention is broken, function parameter tooltips obscure the place you are typing, # and " completion randomly breaks (often leaving you with a ###)...the list goes on and on.

Adobe has a big feature list on their site. I'm thinking maybe they should go back and use some resources to fix the parts where you type things into the computer, you know, the whole point of the thing.

by Ted (noreply@blogger.com) at 2011-12-01 15:14

2011-11-28

Raymond Hettinger recently asked on twitter what people thought del defaultdict()[k] did for a k that didn't exist in the dict. There are two ways of thinking about this, one is, "it's a defaultdict, there's always a value at a key, so it can never raise a KeyError", the other is, "that only applies to reading a value, this should still raise an error". I initially spent several minutes considering which made more sense, but I eventually came around to the second view, I'm going to explain why.

The Zen of Python says, "Errors should never pass silently." Any Java programmer who's seen NullPointerException knows the result of passing around invalid data, rather than propagating an error. There are two cases for trying to delete a key which doesn't exist in a defaultdict. One is: "this algorithm happens to sometimes produce keys that aren't there, not an issue, ignore it", the other is "my algorithm has a bug, it should always produce valid keys". If you don't raise a KeyError the first case has a single line of nice code, if you do raise an error they have a boring try/ except KeyError thing going on, but no big loss. However, if an error isn't raised and your algorithm should never produce nonexistent keys, you'll be silently missing a large bug in your algorithm, which you'll have to hope to catch later.

The inconvenience of ignoring the KeyError to the programmer with the algorithm that produces nonexistent keys is out weighed by the potential for hiding a nasty bug in the algorithm of the programmer who's code should never produce these. Ignoring an exception is easy, trying to find the bug in your algorithm can be a pain in the ass.

2011-11-28 10:28

2011-11-27

While recently reviewing some topics in vector calculus, I became curious as to why violating seemingly innocuous conditions for some theorems leads to surprisingly wild results. In fact, I was struck by how these theorems resemble computer programs, not in some abstract way, but in how the lack of “input validation” leads to non-obvious behavior in the face of erroneous input.

There are actually whole books dedicated to counterexamples. They make good bathroom reading material.

I found that understanding why these counterexamples lead to wild results deepened my understanding of the theorems involved and their proofs. Besides, pathological examples are more interesting than well-behaved ones!

First, let's look at a “counterexample” to Green's theorem:

1. Two functions \(L, M \colon \RR^2 \to \RR\) and a positively-oriented, piecewise smooth, simple closed curve \(C\) in \(\RR^2\) enclosing the region \(D\) such that \[ \oint_C L \,dx + M \,dy \ne \iint_D \left( \pd{M}{x} - \pd{L}{y} \right) \,dx \,dy \text{.} \]

The vector field \((L, M)\) also serves as the canonical “counterexample” to the gradient theorem.

Let \begin{align*} L &= -\frac{y}{x^2+y^2} \text{, } M = \frac{x}{x^2+y^2} \text{,} \end{align*} and \(C\) be a curve going clockwise around the rectangle \(D = [-1, 1]^2\). Then the integral of \(L \,dx + M \, dy\) around \(C\) is \(2 \pi\) since it encloses the origin. But \[ \pd{M}{x} = \pd{L}{y} = \frac{y^2-x^2}{x^2+y^2} \] so the difference of the two vanishes everywhere but the origin, where neither function is defined. Therefore, the (improper) integral over \(D\) also vanishes, proving the inequality. ∎

Of course, the easy explanation is that the discontinuity of \(L\) and \(M\) at the origin violates a condition of Green's theorem. But that doesn't really tell us anything, so let's break down the theorem and see where exactly it fails.

Green's theorem is usually proved first for rectangles \([a, b] \times [c, d]\), which suffices for our purpose. If \(C\) is a curve that goes counter-clockwise around such a rectangle \(D\), then we can easily show that \[ \oint_C L \,dx = - \iint_D \pd{L}{y} \,dx \,dy \] and \[ \oint_C M \,dy = \iint_D \pd{M}{x} \,dx \,dy \text{,} \] with the sum of these two formulas proving the theorem.

So the first sign of trouble is that the theorem freely interchanges addition and integration. Since the partial derivatives of our functions diverge at the origin, if \(D\) contains the origin then the integrals of those partial derivatives over \(D\) may not even be defined, even if the integral of their difference is.

But the problem arises even before that. The statements above are proved by showing \[ \oint_C L \,dx = - \int_a^b \left( \int_c^d \pd{L}{y} \,dy \right) \,dx \] and \[ \oint_C M \,dy = \int_c^d \left( \int_a^b \pd{M}{x} \,dx \right) \,dy \text{.} \] both of which hold for our example. But notice that in one case we integrate with respect to \(y\) first, and in the other case we integrate with respect to \(x\) first. Therefore, we have to interchange the order of integration or convert to a double integral in order to get them to a form where we can add them. And there's the rub: if \(D\) contains the origin, switching the order of integration for either integral above switches the sign of the result!

This fully explains the discrepancy; since the result of both integrals above (with the iteration order preserved) is \(\pi\), adding them together as-is gives the expected result of \(2 \pi\). But if we switch the iteration order of one of the iterated integrals as done in the proof of Green's theorem, then we switch the result of that integral to \(-\pi\), which cancels with the result of the other unchanged integral to produce \(0\).

So now let's examine this strange behavior of the sign of an integration's result depending on the iteration order. This leads us to our next “counterexample,” this time for Fubini's theorem:

2. A function \(f \colon \RR^2 \to \RR\) whose iterated integrals over a rectangle \(D = [a, b] \times [c, d] \subset \RR^2\) differ.

Let \[ f(x, y) = \frac{x^2-y^2}{(x^2+y^2)^2} \quad \text{ and } \quad D = [-1, 1]^2\text{.} \] The two iterated integrals of \(f\) over \(D\) are usually written as \[ \int_{-1}^1 \left( \int_{-1}^1 f(x, y) \,dy \right) \,dx \qquad \text{ and } \qquad \int_{-1}^1 \left( \int_{-1}^1 f(x, y) \,dx \right) \,dy \] but let's define them more carefully to make it easier to justify our calculations.

Let \begin{align*} u_k &= y \mapsto f(k, y) \\ v_l &= x \mapsto f(x, l) \text{.} \end{align*} In other words, given the real constants \(k\) and \(l\), construct the (possibly partial) real functions \(u_k(y)\) and \(v_l(x)\) by partially-evaluating \(f\) at \(x = k\) and \(y = l\), respectively.

\(U(x)\) and \(V(y)\) are also (partial) real functions.

Then, if we also let \[ U(x) = \int_{-1}^1 u_x(y) \,dy \qquad \text{ and } \qquad V(y) = \int_{-1}^1 v_y(x) \,dx \text{,} \] we can write the iterated integrals as \[ \int_{-1}^1 U(x) \,dx \qquad \text{ and } \qquad \int_{-1}^1 V(y) \,dy \text{.} \]

We're justified in applying standard integration techniques here since \(u_k(y)\) for \(k > 0\) is defined and bounded for all \(y\).

Computing \(U(x)\) for \(x \neq 0\), we get \begin{align*} U(x) &= \int_{-1}^1 \pd{}{y} \left( -\frac{y}{x^2+y^2} \right) \,dy \\ &= \left. -\frac{y}{x^2+y^2} \right|_{y=-1}^{y=1} \\ &= -\frac{2}{x^2+1} \text{.} \end{align*}

Attempting to evaluate \(U(0)\), we see that \begin{align*} U(0) &= \int_{-1}^1 \frac{0^2-y^2}{(0^2+y^2)^2} \,dy \\ &= - \int_{-1}^1 \frac{dy}{y^2} \end{align*} which diverges. So \[ U(x) = -\frac{2}{x^2+1} \text{ for } x \ne 0 \text{.} \]

Note that \(U(x)\) and \(V(y)\) differ only in variable name and sign.

By a similar computation, we find that \[ V(y) = \frac{2}{y^2+1} \text{ for } y \ne 0 \text{.} \]

Since \(U(x)\) isn't defined at \(0\), we have to treat it as an improper integral, although doing so poses no real difficulty: \begin{align*} \int_{-1}^1 U(x)\,dx &= \lim_{a \nearrow 0} \left( \int_{-1}^a -\frac{2}{x^2+1} \,dx \right) + \lim_{a \searrow 0} \left( \int_{a}^1 -\frac{2}{x^2+1} \,dx \right) \\ &= \lim_{a \nearrow 0} \Bigl( \left. -2 \arctan(x) \right|_{-1}^{a} \Bigr) + \lim_{a \searrow 0} \Bigl( \left. -2 \arctan(x) \right|_{a}^{1} \Bigr) \\ &= \left. -2 \arctan(x) \right|_{-1}^{0} + \left. -2 \arctan(x) \right|_{0}^{1} \\ &= \left. -2 \arctan(x) \right|_{-1}^{1} \\ &= -\pi \text{.} \end{align*}

Similarly, \[ \int_{-1}^1 V(y)\,dy = \pi \text{,} \] so the iterated integrals of \(f(x, y)\) over \([-1, 1]^2\) differ; in fact, as we claimed above, switching the iteration order switches the sign of the result. ∎

We can repeat the above calculations for an arbitrary rectangle to see that the iterated integrals of \(f(x, y)\) differ if \(D\) contains the origin either as an interior point or a corner. But there's an easier way to prove that statement and also gain some insight as to why \(f(x, y)\) has this strange property.

Note that the key facts in the above calculations were that \(U(x) \lt 0\) for any \(x \ne 0\) and \(V(y) \gt 0\) for any \(y \ne 0\). Therefore, integrating \(U(x)\) over any interval on the \(x\)-axis would produce a negative result and integrating \(V(x)\) over any interval on the \(y\)-axis would produce a positive result, leading to the difference in iterated integrals. This holds more generally; for any \(m, n \gt 0\): \[ \int_{-n}^n f(x, y) \,dy 0 \qquad \text{ and } \qquad \int_{-m}^m f(x, y) \,dx > 0 \text{.} \] Therefore, \[ \int_{-m}^m \left( \int_{-n}^n f(x, y) \,dy \right) \,dx 0 \qquad \text{ and } \qquad \int_{-n}^n \left( \int_{-m}^m f(x, y) \,dx \right) \,dy > 0 \] so the iterated integrals of \(f(x, y)\) differ over the rectangles \([-m, m] \times [-n, n]\). Since any rectangle \(D\) containing the origin as an interior point must contain some smaller rectangle \(E = [-m, m] \times [-n, n]\), the iterated integrals of \(f(x, y)\) over \(E\) differ and therefore must also differ over \(D\).

Furthermore, since \(f(x, y)\) is even in both \(x\) and \(y\), you can carry out a similar argument to the above with intervals of the form \([0, m]\) or \([-m, 0]\) to show that the iterated integrals of \(f(x, y)\) must also differ over any rectangle with the origin as a corner.

So the essential property of \(f(x, y)\) is that slicing it along the \(x\)-axis gives a function which has positive area under the curve on any interval symmetric around \(0\) or with \(0\) as an endpoint, and that slicing it similarly along the \(y\)-axis gives a function with has negative area. Therefore, on a rectangle symmetric around the origin or with the origin as a corner, one can choose the sign of the iterated integral by choosing which axis to slice first.

The next thing to investigate is how exactly the iterated integrals of \(f(x, y)\) over the rectangle \(D\) are expressed such that they differ only when \(D\) contains the origin, especially considering that the \(f(x, y)\) is expressed in quite a simple form. To do that, let's consider the simple case of a rectangle \(D = [\delta, 1] \times [\epsilon, 1]\) where we can vary \(\delta\) and \(\epsilon\) at will.

Let \begin{align*} I_{yx}(\delta, \epsilon) &= \int_{\delta}^1 \left( \int_{\epsilon}^1 f(x, y) \,dy \right) \,dx \\ I_{xy}(\delta, \epsilon) &= \int_{\epsilon}^1 \left( \int_{\delta}^1 f(x, y) \,dx \right) \,dy \text{.} \end{align*} Then, for \(\epsilon \neq 0\): \begin{align*} I_{yx}(\delta, \epsilon) &= \int_{\delta}^1 \left( \int_{\epsilon}^1 \frac{y^2-x^2}{(x^2+y^2)^2} \,dy \right) \,dx \\ &= \int_{\delta}^1 \left( \left. -\frac{y}{x^2+y^2} \right|_{y=\epsilon}^{y=1} \right) \,dx \\ &= \int_{\delta}^1 \Biggl( -\frac{1}{1+x^2} - \left( -\frac{\epsilon}{\epsilon^2+x^2} \right) \Biggr) \,dx \\ &= \int_{\delta}^1 \frac{dx/\epsilon}{1+(x/\epsilon)^2} - \int_{\delta}^1 \frac{dx}{1+x^2} \\ &= \arctan\left(\frac{1}{\epsilon}\right) - \arctan\left(\frac{\delta}{\epsilon}\right) - \frac{\pi}{4} + \arctan(\delta) \text{,} \end{align*} and for \(\epsilon = 0\): \[ I_{yx}(\delta, 0) = -\frac{\pi}{4} + \arctan(\delta) \text{.} \] Similarly, for \(\delta \neq 0\): \begin{align*} I_{xy}(\delta, \epsilon) &= \int_{\epsilon}^1 \left( \int_{\delta}^1 \frac{y^2-x^2}{(x^2+y^2)^2} \,dx \right) \,dy \\ &= \int_{\epsilon}^1 \left( \left. \frac{x}{x^2+y^2} \right|_{x=\delta}^{x=1} \right) \,dy \\ &= \int_{\epsilon}^1 \left( \frac{1}{1+y^2} - \frac{\delta}{\delta^2+x^2} \right) \,dy \\ &= \int_{\epsilon}^1 \frac{dy}{1+y^2} - \int_{\epsilon}^1 \frac{dy/\delta}{1+(y/\delta)^2} \\ &= \frac{\pi}{4} - \arctan(\epsilon) - \arctan\left(\frac{1}{\delta}\right) + \arctan\left(\frac{\epsilon}{\delta}\right) \text{,} \end{align*} and for \(\delta = 0\): \[ I_{xy}(0, \epsilon) = \frac{\pi}{4} - \arctan(\epsilon) \text{.} \] Then let \(\Delta = I_{xy} - I_{yx}\) be the difference between the two iterated integrals. We can use the identity \[ \arctan(x) + \arctan\left(\frac{1}{x}\right) = \frac{\pi}{2} \sgn{x} \] to simplify \(\Delta(\delta, \epsilon)\) if neither \(\delta\) nor \(\epsilon\) is zero: \begin{align*} \Delta(\delta, \epsilon) &= \bigl( \pi/4 - \arctan(\epsilon) - \arctan(1/\delta) + \arctan(\epsilon/\delta) \bigr) \\ & \quad \mathbin{-} \bigl( \arctan(1/\epsilon) - \arctan(\delta/\epsilon) - \pi/4 + \arctan(\delta) \bigr) \\ &= \pi/2 - \bigl( \arctan(\epsilon) + \arctan(1/\epsilon) \bigr) \\ & \quad \mathbin{-} \bigl( \arctan(\delta) + \arctan(1/\delta) \bigr) \\ & \quad \mathbin{+} \bigl( \arctan(\delta/\epsilon) + \arctan(\epsilon/\delta) \bigr) \\ &= \frac{\pi}{2} \bigl( 1 - \sgn(\epsilon) - \sgn(\delta) + \sgn(\delta/\epsilon) \bigr) \text{.} \end{align*}

Using the properties of \(\sgn(x)\), we can simplify this to the final expression: \[ \Delta(\delta, \epsilon) = \frac{\pi}{2} \bigl( 1 - \sgn(\delta) \bigr) \bigl( 1 - \sgn(\epsilon) \bigr) \] which we can prove still holds if either \(\delta\) or \(\epsilon\) is zero (or both).

So with the simplified expression for \(\Delta(\delta, \epsilon)\), it becomes apparent how \(\sgn(x)\) is used to control the value of \(\Delta(\delta, \epsilon)\); as long as either \(\delta\) or \(\epsilon\) is positive, \(1 - \sgn(x)\) zeroes out the entire expression.

by Fred Akalin at 2011-11-27 08:00

2011-11-19

There exists this disease on the Internet which is the set of people who react badly to things such as this:

void main() {
    printf("Hello, world!\n");
}

They’ll flare their hoods and puff out, “main should be declared to return int and not void because that’s what the C++ standard says, your code is non-standard and thus it’s bad!” Well gee, it would be just horrid if somebody porting the code had to change the return value of one function.

The problem with you, if you’re one of these people, is that you have nothing better to say.  This is the sort of thing where you sit in your chair at your computer like the little smugnesty you are and find some petty fault in the way other people do things.  Guess what: you’re unimportant and you don’t matter.  You should find something more important to bitch about, like problems with people’s code that actually matter.  Or problems with people bitching about code that doesn’t matter.

by shrughes at 2011-11-19 05:38

2011-11-18

I just delivered a talk at RCOS (the open source center on campus) on some of the work I've been doing this semester on PyPy's implementation of NumPy. You can find the slides here, they're built using html5slides, which is pretty swell (for a tool that makes me write HTML directly that is).

2011-11-18 21:29

2011-11-14

Whilst writing some blog posts about the authentication which I’ve been implementing, I’ve come to the realization, almost as an after thought, that even though I’ve still not got a lot of the main Github API implemented, the way I’ve envisioned the API working, from a user point of view, is sort of a bit shit.

var userApi = new UserApi();
var user = userApi.GetUser("example");

Pretty straightforward, right? What about the as yet unwritten GistApi?

var gistApi = new GistApi();
var gist = gistApi.GetGist(...);

Again, pretty straightforward. But I see a pattern forming. What am I going to do when it comes to the RepositoryApi?

var reposApi = new RepositoryApi();
var repo = reposApi.GetRepository(...);

Well, that is starting to look pretty fucking stupid. I’ve caught myself doing it with other methods as well. I simply cannot believe I’ve let myself be so stupid. At least there is not that much implemented where I’ll have to refactor loads of shit.

by stuart at 2011-11-14 08:00