Although there are readily available skeletons for purchase, I want to print my own skeleton for “profiling and debugging” purposes :)

:)

The March update of Graphite Comics just went live. It’s one of those updates that is very significant, with major changes under the hood — but which users will almost certainly not notice at all.

This makes for an interesting opportunity to talk about balancing needs in software development. In this case, we are looking at balancing speed and efficiency against operating costs. And even more interestingly, we can look at an example where a technical decision that improved efficiency and user experience evolved, over time, to actually have a detrimental effect on user experience!

In the early versions of Graphite, we didn’t have many of the features we have now — notably playlists, notifications, favorites, and so on. We really just had a big list of titles, and when you selected a title, we went out and fetched the books for that title. In order to keep things quick, when we fetched the books, we also fetched the pages for that book. That way when you selected a book to read, you didn’t have to wait for the app to go out and fetch the pages — you could just dive right in super quick.

The user experience, naturally, was amazing. Load times are a sure way to kill the mood, and we were avoiding load times completely by pre-loading the content.

In terms of economy, there was of course a minor increase in cost. This is a given, because we were loading content that may or may not have been necessary or needed. But that cost was so low as to be negligible.

A typical example in the beginning would look like this: a user opened a title, causing the app to fetch the books for that title. Titles had an average of around 10-20 books back then, each with 20-30 pages. So we were loading somewhere around 200-600 objects ahead of time. That might sound like a lot, but it really isn’t these days (I have a mental limit of 1000 objects that I keep in my head as an ideal maximum to fetch at once, and that’s just a personal preference), and because our system is designed efficiently, the overhead and cost to make queries like that is incredibly low. Fetching all of that data ahead of time added a tiny fraction of a second to the initial query, and an even smaller fraction of a penny to the cost of the query, but saved an entire query and all the time a query takes when the issue was tapped. The result was a great user experience at a low overall cost impact.

Over time, however, a few things changed. First, we started adding many, many more books to our system. Titles could now have hundreds or thousands of books. Even worse, books were starting to come in with hundreds of pages — sometimes up to 500 pages!

Additionally, we started adding new features like playlists and notifications. Those playlists included books, and notifications announced the release of new books. Similar to the query we ran when viewing a title, we were preloading books and pages for notifications (and other objects) that referenced a book — that way you could look at your notification feed, tap a book, and immediately — without an additional query — start reading that book. Neat!

Except for those big books, and the amount of top level objects being queried. In the notifications example, we were pulling down up to 1000 objects at the top level. Let’s look at a worst case scenario: you have 1000 notifications which alert you to a new book, each which has 150 pages. That query is pulling down 1000 top level objects, each with a book (so another 1000 objects), each with 150 pages — that’s 151,000 objects in one query!

Suddenly this low impact query which greatly improved user experience by pre-loading content became our longest and slowest query by far, and completely ruined the experience on a few screens. At that point, it became much more efficient and economical to load pages on demand.

Of course, there are ways to improve on this as well, and those improvements were part of the March update. For starters, we cache those loaded pages so, as long as the cache is still fresh, you don’t have to fetch pages to loaded recently (if, for example, you open a book you recently closed).

Also, as we have done from the beginning, we load the actual page content on demand — that’s what makes Graphite a streaming platform, after all — so the real heavy lifting of page content doesn’t happen until you are very close to viewing the page itself. So all in all, the impact of this change ended up being negligible, especially in light of the improvement to load times on those screens that contained hundreds or thousands of top level objects.

If you want to see this in action, download Graphite Comics for iOS or Graphite Comics for Android and start reading today! Graphite is a free to read, ad and subscription based platform for comic books which is currently in a public beta ahead of our official launch at San Diego Comic Con 2019.

I’ve updated my social networking demo app Scrawl to include a couple often requested features – Universal links, email verification, and password reset.

In addition, I’ve integrated Mailgun into the backend application in order to allow the server to send email messages to users (which is obviously a requirement for two of these features).

If you are interested in a slightly deep dive into these features, read on!

Universal links

The first feature added to Scrawl is universal links. In a nutshell, this feature essentially allows our app to request that it be the first responder for any requests to a given domain.

For example, if your app is called MyApp and you own the myapp.com domain, you can set up your app to handle any links to that domain. Instead of the device opening Safari and loading up a webpage, the URL can be passed to the app, and the app can handle the request.

This is great because it allows you to create a mobile website that works perfectly fine for users who don’t have your app installed, but allows those users who do have your app installed to get the best, native app experience possible.

Scrawl actually now supports two URL handling schemes: it is the handler for links from the domain https://getscrawl.com as well as responding to it’s own custom URL scheme scrawl:// — the latter scheme is often useful for inter-app communication and data passing, whereas the former is convenient for users to pass around links to and into your app.

There’s another great advantage to using universal links: iOS devices will store your user’s credentials as if they were website passwords — which makes things much, much easier for your users when they reinstall your app and want to sign in again quickly!

In Scrawl, I’m using universal links to handle two kinds of links: email validation, and password reset requests.

Email verification

Often, you will want to make sure users verify their identity in some way before they have access to some or all of the features in your app. The simplest and most common validation is email verification.

In order to handle email validation requests, a few things have to happen:

1. The user object needs to be updated to include a column to denote if the email address was validated — BUT this column must only be writable by the server! Users should not be able to overwrite this column themselves (e.g. via a public REST api or any other method).
2. When a user signs up, or when the user requests a new validation email, the server should create an object in the database, generate a token, store that token in the new object, and send an email to the email address with a link back to the app. The object created must only be creatable by the server, and read/writeable by the server.
3. When the user clicks this link, the app should handle the request by validating that the request was issued by the server and, if so, mark that new column as true. This is done by simply checking the database to see if an object exists for the given email address matching the token. Since the server created the token and sent it to the email address, and since the object we stored it in is not readable, we can safely assume that anyone who has the token at least has access to the email address.
4. The validation request should then be deleted from the server.

This is all pretty straightforward. The interesting thing about email validation is how to handle the case when the user hasn’t validated their email yet.

Do we pop up a window and refuse to close it until the validation is complete, essentially blocking the user out until validation is successful?

Or do we simply check that column before we allow the user to perform certain actions — both on the client side and the server side?

In Scrawl, I opted for the latter option — primarily because I don’t like putting walls in front of users and prefer to let users do as much as we can possibly allow without any interruption in use.

Specifically, I want to prevent the user from creating any content — a place, a comment, a photo, etc — until they have validated their email address. Other types of actions — like voting on places or comments, or simply reading comments, are perfectly fine to do even if you haven’t verified your email address.

This approach is consistent with my overall philosophy of not annoying the user! Let them have as much of the experience as possible before you ask them to sign up, pay, verify their identity, etc.

However, that’s not always appropriate, and sometimes “as much as we can allow” is… well, nothing!

To see an example of the former case, you can check out GlobeChat, which requires validation to finish signing up or logging in. In the case of GlobeChat, the validation email is sent immediately upon signup (and login, if the user has not verified their email yet!), and a modal dialogue essentially locks users out until the validation occurs.

The final piece of the pie is gatekeeping on the client and server side. I far too often see projects where there is only one check — either client or server — before allowing access to a feature that demands verification.

The server should always have the final say, and in the worst case, if you really, really can only do that check in one place, please make it the server!

That said, client side checks are super important to make sure that you don’t waste calls to the server for events you have the ability to know ahead of time will be rejected. And even more important, you should alert your users to the fact that what they are trying to do is going to fail, without making them wait for a network request to complete. Even better, make sure the UI reflects this fact so they won’t even be able to try!

Password reset

When a user requests a password reset, the server needs to generate a token, and send an email to the user with a link back into the app.

When the user clicks the link, the client app should verify with the server that the request is valid, and then present the user with a password reset screen.

When the password is reset, the server should update the user, and delete the request.

This almost certainly seems really straightforward, but it is absolutely critical to get features like this right. If email validation or password reset requests are not secure and well built, users could potentially change other user’s passwords, create accounts with fake emails… or worse. Scrawl now reflects what I consider best practices in implementing these features.

In this series of posts, I want to discuss several myths, misconceptions, and misunderstandings that threaten to derail inexperienced or non-technical founders of tech startups.  

Note: this post also appears on Glowdot with permission.

In the beginning stages of planning an app development, there are two ubiquitous questions that get asked, and they are the obvious ones:

• How much will it cost to build this app?
• How long will it take to build this app?

These are not unreasonable questions — indeed, when building anything the first two things you need to wrap your head around are budget and timeline.

However, these questions become much more difficult — if not impossible — to answer when it comes to software. Let’s look at why that is.

Software is not physical

To start, when we set out to build software, we really need to understand that we aren’t building a physical, standalone thing that exists in its own space, on its own terms, independent of anything else.

Unlike, say, a house or a car which may need occasional maintenance or upgrades, software often needs constant maintenance and upgrades. It also often needs to evolve. There are a few common reasons for this.

The platform your software runs on has changed

Apple and Google constantly roll out updates to the software (iOS and Android) that powers our devices. These updates can include improvements to the overall performance of our devices, but they can also include breaking changes that cause our apps to suddenly not work as expected. This can include things like changes to way the OS allows apps to interface with hardware (permissions, drivers, etc) or changes to the SDK that we use to perform OS exposed tasks (deprecated methods, etc).

The APIs your software uses have changed

If your app relies on third party APIs — as many do — then you need to respond to changes in those APIs often.

A very common example is the Facebook SDK, which changes often and results in Facebook often retiring old methods. Sometimes these changes can be handled easily by updating the SDK and rewriting a few lines of code. Sometimes these changes can fundamentally change the way your app works.

A few years ago, Facebook changed the way apps could request a user’s friend data. And by “changed the way” I really mean, took away the option. You may remember a time when mobile games were spamming your Facebook wall with requests for you to join so-and-so in a game. When Facebook took away the ability for apps to request a users friends (without special permission to do so), many, many apps had to completely change the way they handled on-boarding and user acquisition.

Users are not static

Your users change, because people change. In order to stay relevant, your software is going to have to evolve to keep up with user expectations.

UI/UX expectations change

In the many years I have been building software, I have seen tremendous changes in the expectations of users in terms of not only functionality and performance, but even more importantly user interface (UI) and user experience (UX).

Its important to remember that the dominant apps — the Facebooks and Instagrams and so on — have whole teams devoted to improving the UI/UX of their apps, and even publishing papers on new theories regarding human computer interaction. When Instagram changes their interface, there is often a shockwave that affects every other app out in the wild, as the UI paradigm shifts in another direction.

User requirements change

Users’ needs change often! Let’s say you have an app that allows people to upload images, get a link, and post the link to their images in various places.

This works great, until a new social network comes along, and they forbid posting links. They do allow posting a special kind of code however. We have seen this in the past in the form of WordPress shortcodes, BBCode, and so on.

If that site blows up, you will definitely want your app to adapt and allow your users to post their images on this new site. If you don’t, I can assure you someone else will come along and fill that user need!

Your expectations change!

There is another even more common change that happens — you change your mind! It happens much, mucb more often than you might think. As development progresses and you get to hold in your hands this thing that started as just an idea, you start get a different perspective on it. You realize there are things you didn’t consider before. You realize some features you thought were critical are actually useless or at least less important. You simply reconsider and adjust, and therefore change direction. Sometimes, you may pivot and completely change the concept entirely!

Although in the best case we always set out to build the thing we originally wanted to build, its good to keep an open mind and remain flexible.

Bug free software is not a thing!

It pains me to say this, but there really is no such thing as bug free software. This is especially true as your software grows in scope and complexity.

What we strive to achieve, really, is a minimum set of bugs. There is not a single app you use that doesn’t have a giant bug list that is waiting to be tackled. If I could pull back the curtain on any piece of software, and it doesn’t matter what type of software, what platform, or who built it, there is a dedicated bug tracking system behind the scenes in which people are constantly entering bugs and developers are constantly pulling out bug reports and trying to fix them. Its just a reality of software development.

That bing said, however, your users should in any case be blissfully unaware of the number of issues in the queue — and often, they are. Many of the bugs you see in these lists are minor issues or issues that affect only a tiny subset of users doing very specific things. However, they still linger and need to be addressed — and as your software evolves, I can almost guarantee new ones will pop up.

Changes trigger new requirements

This is a big one. A change in one place often requires many changes in many other places. Being able to predict those new requirements comes with experience.

For example, lets say the CEO of a blog platform app hears from users that they want to be able to upload photos with their posts. He asks the CTO how long it will take to do it.

The inexperienced CTO thinks about it, realizes adding the client side functionality to choose and upload a photo is pretty simple, and the backend capability of receiving and storing the photo is also pretty simple, and linking the two together by adding a column to the Posts table in the database is a piece of cake, so he answers: “that’ll take a couple days!”.

The experienced CTO realizes that adding photo uploads creates a whole new set of requirements beyond just uploading a photo. We need to be able to delete photos, and replace photos. We need to be able to moderate user uploads to keep unwanted content off our platform — this alone often requires building a whole new software bundle to allow content moderators to do their work! We need to think about things like validating files on the client side and the server side (making sure users don’t upload files that aren’t images). We probably want to add some logic to resize uploads so that if a popular user uploads a 1GB image we don’t have to serve that giant file to everyone who views their post.

And on and on and on. That one simple feature request cascades out into a whole new set of requirements — and possibly those new requirements create their own requirements!

And the list goes on

There are even more reasons we can discuss, but hopefully I’ve made the point.

As a final example, the first big product I ever built was a social network and media sharing site that grew alongside MySpace in the early 2000s.

It started small — just a form to upload photos and get a sharing code running on a single, humble server. Eventually it evolved into a full-fledged social network, and grew to the point where it ran in a stack of highly powerful servers in a dedicated datacenter.

Let’s use that site to answer the original question here: when was development “finished”? When it was retired 12 years after it was launched! New features were constantly being added or removed according to user demands. New enhancements were constantly being made as traffic increased. It truly was a living, evolving being that needed constant care and maintenance, and that constant care and maintenance is largely why it was so successful.

So what’s the answer?

I’ve discussed in part why the original two questions that started this blog post are difficult if not impossible to answer. But I also noted that those questions are crucial to ask before starting any development! So how do we answer those questions?

The key is getting specific. Instead of asking “how much does it cost to build this app?” we break down the lifespan of the product into chunks we can wrap our head around — workshops, prototypes, alpha/beta development phases, release candidates and, after release, versioned updates. The question then becomes, “how long will it take to prototype an app with the following features….” or “how much will it take to get an MVP of my app, on iOS and Android, with this limited feature set“?

In this way, we can more accurately make estimates about how long and at what cost a project will take. Often, the first quote and proposal I offer is a package of several of those questions — we plan out as far ahead as is reasonable. This usually means a few workshops to set the plan, a prototype, an alpha and beta development where we plan out a few critical “must have” features, and then a final pre-release phase where we make sure everything we need to release the app publicly — aka the MVP or minimum viable product — and then we stop there.

Once we reach release, then we can regroup, discuss where we are, put together a list of feature requests, changes, and bug fixes, and estimate a timeline and budget for that new chunk of development. Not only does this help us answer those two hard questions more accurately, it helps us stay focused and on track by encouraging us all to not bite off more than we can chew.

By keeping the size and scope of each phase manageable, we set attainable goals rather than looking too far into the future and risk getting lost along the way.

There is an extra bit of knowledge here: if you are shopping around for quotes and you ask generally “how much will it cost to make this app?” and you get an explicit answer — run and don’t look back. This is a developer that is telling you what they think you want to hear, not what you need to hear. These developers will do anything to get you to sign on with them, from telling you anything and everything is possible, agreeing to every feature request without any advice as to how it might be approached differently, more efficiently, and more cost effectively, and they will approach your project as a single task instead of the complex procedure that it is.

Graphite is a scalable, efficient and multi-platform graphical content distribution system for mobile devices and the web.

I designed and developed the backend system powering Graphite — a Node.js based system that is powered by several AWS services, in addition to some locally hosted server functionality (mostly to handle maintenance jobs and statistical analysis of the live system) and a media sharing system facilitating the onboarding of new users coming from social media sites like Twitter and Facebook.

I also developed the iOS app for Graphite — one of the biggest and most complex mobile projects I have ever taken on. Although on the surface Graphite seems quite simple, in fact the technology powering it is extremely sophisticated and complex.

In addition to developing the iOS app and the server-side platform, I current manage the development of every other current and future platform — including Android, the web, and a few other platforms on the roadmap.

You can download the public beta of Graphite on iOS and on Android and visit the Graphite website here.

If you need an app developed, reach out and let’s talk! You can contact me using the contact form on this site, via Skype at stromdotcom or by visiting my company website at https://glowdot.com.

Scrawl is a social media app developed over the course of one month, representing a typical social media app project.

Aside from being a pretty neat idea, I actually launched this app to demonstrate a few critical concepts that many first time (and even some fairly experienced!) mobile entrepreneurs might not be aware of.  Let’s take a look at a few of them!

Delayed permission requests

One of the surest ways to lose a new user is to ask for permissions too soon — or worse, immediately upon first launch of the app.  Even if you are lucky and don’t immediately cause that user to delete your app, you will almost certainly end up with that user denying your request.

The reason for this is pretty simple, if not immediately obvious.  A new user really doesn’t know much about you or your app at all.  And you certainly haven’t earned their trust yet.  But here you are, asking for access to their photos, contacts, location…. wouldn’t your first response be, “wait, who are you and what are you going to do with this information?”

And while the push notifications permission might not be invasive in terms of privacy, it is invasive in terms of annoyance factor for many apps, as it has been chronically abused in the past.  Consequently, most users tend to answer “no” unless they have a compelling reason to say “yes”.

The worst part of all this is that once a user says no, its really hard to get them to go into the settings app and change their answer to yes.  Primarily because most users don’t even know how do to that even if they wanted to.

A much better approach is to delay asking for those permissions until you have spent a little time with that user, and given them a good reason to accept the request.  In Scrawl, we do this in a few simple ways:

• Push notifications are not requested until you perform an action for which you might receive a notification.  In Scrawl, you can receive notifications when users reply to you, or post in a place you have favorited.  So we don’t ask for push permission until you post something, or favorite a place.
• For location permission, we start users off as if they were sitting in our office in Santa Monica, CA.  At the top of the “nearby places” list, we let them know they are looking at places near us, and offer them the option of sharing their location so we can show them places near them.  In this way, we allow the user to request that we accept their share, rather than intrusively ask them where they are.
• For camera and photo library permissions, we ask only when the user requests to share an image.  This isn’t that interesting, however, as most apps work like this.  It is, in fact, default behavior in iOS apps.

Aside from waiting to ask, we employ one more strategy here to maximize acceptance of permission requests: we ask the user ourselves before we request that iOS make the request.  This is a little more advanced a concept, but it is based on the fact that once iOS itself asks you for permission, it will never ask again — the user MUST change their answer in the Settings app.  So instead of requesting the push notification permission, for example, we pop up an explanatory dialog explaining why we want it, and asking the user if they are interested in granting it.  If they say no, we don’t ask them again for a while, giving them a chance to get more comfortable in the app, and giving us a chance to earn trust.  If they say yes, then we tell iOS to go ahead and request the permission.  If they said yes to us, there’s a pretty good chance they’ll say yes to iOS.  So when the request is made, even though its the last time it will be made, the odds that it will be accepted is much, much higher.

Delayed log in requests

As a general principle, we like to delay asking a user to log in or sign up as long as we possibly can.  At a bare minimum, we try with all our might to avoid requesting users log in immediately upon launching the app!  For some apps, this is inevitable, for most situations, there is a set of functionality that is perfectly reasonable to offer to anonymous users.

In Scrawl, for example, users can search places and read posts, look at photos, and even report objectionable content without logging in.  But to create a place, post a message, upload a photo or vote on any content, they need to log in.  When a user attempts to do one of those things, that’s when we request they log in or sign up — not before.

Content moderation

One of the most important things to consider when creating a social network, and especially one which centers on user generated content (UGC), is content moderation.  You need to have a way to make sure objectionable content is quickly removed and malicious users are removed from the system.  I won’t go into detail about our moderation system here for obvious reasons, but we have created a model for self-moderation that allows objectionable content to be filtered out of the system until such a time that a human moderator can permanently remove it and moderate the offending user accounts.  It’s a pretty smart system, and one that we believe fixes this major problem which has plagued other similar apps in the past.

If you’d like to see Scrawl in action, we’d love for you to check it out on iOS here:

Scrawl on the App Store.

An Android version is coming soon!

Scrawl – post anything, anywhere, anytime!

If you need an app developed, reach out and let’s talk! You can contact me using the contact form on this site, via Skype at stromdotcom or by visiting my company website at https://glowdot.com.

Introduction

In the last post we were left with some tests that exercised some very basic functionality of the Deck class. In this post, we will continue to add unit tests and write production code to make those tests pass, until we get a class which is able to produce a randomised deck of 52 cards.

Test Refactoring

You can, and should, refactor your tests where appropriate. For instance, on the last test in the last post, we only asserted that we could get all the cards for a particular suit. What about the other three? With most modern test frameworks, that is very easy.

Theory]
[InlineData(Suit.Clubs)]
[InlineData(Suit.Diamonds)]
[InlineData(Suit.Hearts)]
[InlineData(Suit.Spades)]
public void Should_BeAbleToSelectSuitOfCardsFromDeck(Suit suit)
{
    var deck = new Deck();

    var cards = deck.Where(x => x.Suit == suit);

    cards.Should().HaveCount(13);
}

More Cards

We are going to want actual cards with values to work with. And for the next test, we can literally copy and past the previous test to use as a starter.

[Theory]
[InlineData(Suit.Clubs)]
[InlineData(Suit.Diamonds)]
[InlineData(Suit.Hearts)]
[InlineData(Suit.Spades)]
public void Should_BuildAllCardsInDeck(Suit suit)
{
    var deck = new Deck();

    var cards = deck.Where(x => x.Suit == suit);

    cards.Should().Contain(new List<Card> 
    { 
        new Card(suit, "A"), new Card(suit, "2"), new Card(suit, "3"), new Card(suit, "4"),
        new Card(suit, "5"), new Card(suit, "6"), new Card(suit, "7"), new Card(suit, "8"),
        new Card(suit, "9"), new Card(suit, "10"), new Card(suit, "J"), new Card(suit, "Q"),
        new Card(suit, "K")
    });
}

Now that I’ve written this, when I compare it to the previous one, it’s testing the exact same thing, in slightly more detail. So we can delete the previous test, it’s just noise.

The test is currently failing because it can’t compile, due to there not being a constructor which takes a string. Lets fix that.

public struct Card
{
    private Suit _suit;
    private string _value;

    public Card(Suit suit, string value)
    {
        _suit = suit;
        _value = value;
    }

    public Suit Suit { get { return _suit; } }
    public string Value { get { return _value; } }

    public override string ToString()
    {
        return $"{Suit}";
    }
}

There are a couple of changes to this class. Firstly, I added the constructor, and private variables which hold the two defining variables, with properties with only public getters. I changed it from being a class to being a struct, and it’s now an immutable value type, which makes sense. In a deck of cards, there can, for example, only be one Ace of Spades.

These changes mean that are tests don’t work, as the Deck class is now broken, because the code which builds set of thirteen cards for a given suit is broken - it now doesn’t understand the Card constructor, or the fact that the .Suit property is now read-only.

Here is my first attempt at fixing the code, which I don’t currently think is all that bad:

private string _ranks = "A23456789XJQK";

private List<Card> BuildSuit(Suit suit)
{
    var cards = new List<Card>(_suitSize);

    for (var i = 1; i <= _suitSize; i++)
    {
        var rank = _ranks[i-1].ToString();
        var card = new Card(suit, rank);
        cards.Add(card);
    }

    return cards;
}

This now builds us four suites of thirteen cards. I realised as I was writing the production code that handling “10” as a value would be straightforward, so I opted for the simpler (and common) approach of using “X” to represent “10”. The test pass four times, once for each suit. This is probably unnecessary, but it protects us in future from inadvertantly adding any code which may affect the way that cards are generated for a particular suit.

Every day I’m (randomly) shuffling

It’s occured to me as I write this that the Deck class is funtionally complete, as it produces a deck of 52 cards when it is instantiated. You will however recall that we want a randomly shuffled deck of cards. If we consider, and invoke the Single Responsibility Principal, then we should add a Dealer class; we are modeling a real world event and a pack of cards cannot shuffle itself, that’s what the dealer does.

Conclusion

In this post I’ve completed the walk through of developing a class to create a deck of 52 cards using some basic TDD techniques. I realised adding the ability to shuffle the pack to the Deck class would be a violation of SRP, as the Deck class should not be concerned or have any knowledge about how it is shuffled. In the next post I will discuss how we can implement a Dealer class, and illustrate some techniques swapping the randomisation algorithim around.

I once ate 10 mg LSD by accident. It was a dilution error. The peak lasted ~10 hr. At some point, I saw the top of my head. But hey, maybe it was just an hallucination ;)

maybe ;)

I know that last time I said I was going to start soldering, but I really wanted to play with the networking capabilities of the ESP8266 first.

There’s a bunch of example programs to do web requests, such as BasicHttpClient, HTTPSRequest, WifiClient, StreamHttpClient, etc. I had trouble getting these working because there’s no built in certificate store or TLS validation capabilities. That means you can’t do a “normal” HTTPS request to test services like RequestBin (since they’re HTTPS only). The example programs have you type in the server’s SHA1 thumbprint, but that didn’t seem to work for me. The certs I inspected were SHA256, which I assume is the problem.

Anyway, I’m not interested in doing HTTP in any of my project ideas right now, so I moved on to what I actually want to do, which is MQTT. Once again it was Hack-a-day that clued me in to this protocol, which is very popular for small devices & home automation. I started out looking for a “simplest possible MQTT example for ESP8266″ and didn’t find anything simple enough initially. Later I realized that there’s two great places to start looking for libraries & examples. First is the esp8266/Arduino repo on Github, which has a list of miscellaneous third party projects compatible with this specific chip. Second is in the Arduino IDE itself; the Library Manager is searchable:

Arduino Library Manager – Searching for “MQTT”

The problem here is that which (if any) are actually good, useful, or correct for the ESP8266. The first search result in that screenshot is only for the “Arduino Uno Wifi Developer Edition”, for example.

Another challenge here is working through all the company branding. The second library listed, “Adafruit MQTT Library”, is ESP8266 compatible and comes with a “simple” example program to get started. However, it’s oriented around the Adafruit IO IoT web service (which is apparently a thing). I did get it to work with the local MQTT broker I’m running here on my PC, but I had to guess if might and try to peel away their extra stuff just to get to the bones.

The ESP8266 Github linked to lmroy/pubsubclient, which itself is a fork of knolleary/pubsubclient which seems more up to date. I don’t know why they’re linking to an out of date fork, except that it appears to more easily support “large” messages. The original has a default max packet size of 128 bytes, which might be too small for real apps, but I’m looking for a simple example so it should be fine.

Here’s a link to the example program from that repo: https://github.com/knolleary/pubsubclient/blob/master/examples/mqtt_esp8266/mqtt_esp8266.ino

The example is easy to set up; just punch in your Wifi access info & MQTT broker host name. Interestingly it does apparently support DNS… from working with the HTTP examples earlier, some of them used IP addresses rather than host names, so it wasn’t clear if DNS was supported in some of these network libraries. This one does, apparently.

Here’s what the output looks like. I’m running mosquitto 1.4.8 on my PC, with mosquitto_sub running in the lower panel subscribed to # (a wildcard, so all topic messages are shown).

Basic MQTT Example on the ESP8266

Actual footage of me as this program was running

I thought it would be fun to give the messages a little personality, so I found a list of all the voice lines from the turrets in Portal 2, copied them into a giant array, and now instead of “Hello World #37″ it’ll send something like “So what am I, uh, supposed to do here?” or “Well, I tried. Best of Luck!” once every few seconds.

Additionally, I made it so that the power LED blinks as it’s sending a message, as a little visual chirp to let you know it’s alive.

via GIPHY

The code is here, and this version is a modification of the Adafruit MQTT example, rather than the other library linked above, because I wrote it before I discovered that simpler example. (Found the list of voice lines here, and removed a few dupes).

I thought it might be fun to play around with programmable microcontrollers, so I bought some to play with. One of the most popular chips right now is the ESP8266 which I first saw pop up on Hack-a-day in 2014. I had to search backwards through 47 pages of blog posts that have been made in the meantime — that might give you a sense of its popularity.

I played around with PIC16/PIC18s in the early 2000s but never actually made anything, but the interest has been there. It’s also been my long time desire to create an E-ink weather display (once again thanks to an old Hack-a-day post, this one from 2012. That’s how far behind on projects I am). Recently I noticed some inexpensive E-ink development boards on Aliexpress and decided to jump into a less shallow end of the pool. I had also been following the ESP8266 for Arduino repo on Github, so I vaguely knew where to begin.

This evening I received the hardware (specifically, three of these) and decided to see if I could get a basic program deployed, just to get started. I don’t really know what I’m doing but I’m pretty good at reading & following directions (that counts for a lot in life).

The basics are:

1. Follow the “Installing with Boards Manager” directions here: https://github.com/esp8266/Arduino/ (which includes grabbing the latest Arduino IDE software, then pulling in the ESP8266 chip configuration, which includes the WeMos D1 mini board configuration.

2. From WeMos’ website, grab the driver for the on-board USB/programmer chip, which for me was the CH240G (https://wiki.wemos.cc/tutorials:get_started:get_started_in_arduino). Nothing more “exciting” than installing Chinese device driver software!

3. I got a little tripped up here, but later figured it out- when you plug one of the D1 boards into your PC via USB, it’ll show up as a COM Port in Device Manager. You have to pick that same COM Port in the Arduino IDE or it can’t find your board to deploy to. This was a little confusing because it won’t show up until you’re plugged in.

4. The Arduino IDE comes with the ability to load in example programs from the board configuration, loaded in Step 1. I wanted the simplest possible thing to make sure everything was working, so picked the “Hello World” of microcontroller programs: “Blink”, which toggles the power LED in an infinite loop.

So far, so good! All told, this took about an hour from opening the package to getting the light to blink (which includes scrounging around for a good USB cable and trying to get a in-focus pictures).

As you can see, I didn’t even bother to solder on the headers yet. I will do that, but I think next I will look into getting some wifi code up and running.

Introduction

In the previous post in this series, we had finished up with a very basic unit test, which didn’t really test much, which we had ran using dotnet xunit in a console, and saw some lovely output.

We’ll continue to write some more unit tests to try and understand what kind of API we need in a class (or classes) which can help us satisfy the first rule of our Freecell engine implementation. As a reminder, our first rule is: There is one standard deck of cards, shuffled.

I’m trying to write both the code and the blog posts as I go along, so I have no idea what the final code will look like when I’ve finished. This means I’ll probably make mistakes and make some poor design decisions, but the whole point of TDD is that you can get a feel for that as you go along, because the tests will tell you.

Don’t try to TDD without some sort of plan

Whilst we obey the 3 Laws of TDD, that doesn’t mean that we can’t or shouldn’t doodle a design and some notes on a whiteboard or a notebook about the way our API could look. I always find that having some idea of where you want to go and what you want to achieve aids the TDD process, because then the unit tests should kick in and you’ll get a feel for whether things are going well or the conceptual design you had is not working.

With that in mind, we know that we will want to define a Card object, and that there are going to be four suits of cards, so that gives us a hint that we’ll need an enum to define them. Unless we want to play the same game of Freecell over and over again, then we’ll need to randomly generate the cards in the deck. We also know that we will need to iterate over the deck when it comes to building the Cascades, but the Deck should not be concerned with that.

With that in mind, we can start writing some more tests.

To a functioning Deck class

First things first, I think that I really like the idea of having the Deck class enumerable, so I’ll start with testing that.

[Fact]
public void Should_BeAbleToEnumerateCards()
{
    foreach (var card in new Deck())
    {
    }
}

This is enough to make the test fail, because the Deck class doesn’t yet have a public definition for GetEnumerator, but it gives us a feel for how the class is going to be used. To make the test pass, we can do the simplest thing to make the compiler happy, and give the Deck class a GetEnumerator definition.

public IEnumerator<object> GetEnumerator()
{
    return Enumerable.Empty<object>().GetEnumerator();
}

I’m using the generic type of object in the method, because I haven’t yet decided on what that type is going to be, because to do so would violate the three rules of TDD, and it hasn’t yet been necessary.

Now that we can enumerate the Deck class, we can start making things a little more interesting. Given that it is a deck of cards, it should be reasonable to expect that we could expect to be able to select a suit of cards from the deck and get a collection which has 13 cards in it. Remember, we only need to write as much of this next test as is sufficient to get the test to fail.

[Fact]
public void Should_BeAbleToSelectSuitOfCardsFromDeck()
{
    var deck = new Deck();

    var hearts = deck.Where();
}

It turns out we can’t even get to the point in the test of asserting something because we get a compiler failure. The compiler can’t find a method or extension method for Where. But, the previous test where we enumerate the Deck in a foreach passes. Well, we only wrote as much code to make that test pass as we needed to, and that only involved adding the GetEnumerator method to the class. We need to write more code to get this current test to pass, such that we can keep the previous test passing too.

This is easy to do by implementing IEnumerable<> on the Deck class:

public class Deck : IEnumerable<object>
{
    public IEnumerator<object> GetEnumerator()
    {
        foreach (var card in _cards)
        {
            yield return card;
        }
    }

    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

I’ve cut some of the other code out of the class so that you can see just the detail of the implementation. The second explicitly implemented IEnumerable.GetEnumerator is there because IEnumerable<> inherits from it, so it must be implemented, but as you can see, we can just fastward to the genericly implemented method. With that done, we can now add using System.Linq; to the Deck class so that we can use the Where method.

var deck = new Deck();

var hearts = deck.Where(x => x.Suit == Suit.Hearts);

This is where the implementation is going to start getting a little more complicated that the actual tests. Obviously in order to make the test pass, we need to add an actual Card class and give it a property which can use to select the correct suit of cards.

public enum Suit
{
    Clubs,
    Diamonds,
    Hearts,
    Spades
}

public class Card
{
    public Suit Suit { get; set; }
}

After writing this, we can then change the enumerable implementation in the Deck class to public class Deck : IEnumerable<Deck>, and the test will now compile. Now we can actually assert the intent of the test:

[Fact]
public void Should_BeAbleToSelectSuitOfCardsFromDeck()
{
    var deck = new Deck();

    var hearts = deck.Select(x => x.Suit == Suit.Hearts);

    hearts.Should().HaveCount(13);
}

Conclusion

In this post, I talked through several iterations of the TDD loop, based on the 3 Rules of TDD, in some detail. An interesting discussion that always rears its head at this point is: Do you need to follow the 3 rules so excruciatingly religously? I don’t really know the answer to that. Certainly I always had it in my head that I would need a Card class, and that would necessitate a Suit enum, as these are pretty obvious things when thinking about the concept of a class which models a deck of cards. Could I have taken a short cut, written everything and then wrote the tests to test the implementation (as it stands)? Probably, for something so trivial.

In the next post, I will write some more tests to continue building the Deck class.

Introduction

I thought Freecell would make a fine basis for talking about Test Driven Development. It is a game which I enjoy playing. I have an app for it on my phone, and it’s been available on Windows for as long as I can remember, although I’m writing this on a Mac, which does not by default have a Freecell game.

The rules are fairly simple:

• There is one standard deck of cards, shuffled.
• There are four “Free” Cell piles, which may each have any one card stored in it.
• There are four Foundation piles, one for each suit.
• The cards are dealt face-up left-to-right into eight cascades
  • The cards must alternate in colour.
  • The result of the deal is that the first four cascades will have seven cards, the final four will have six cards.
• The top most card of a cascade beings a tableau.
• A tableaux must be built down by alternating colours.
• A card in cell may be moved onto a tableau subject to the previous rule.
• A tableaux may be recursively moved onto another tableaux, or to an empty cascade only if there is enough free space in Cells or empty cascades to use as intermediate locations.
• The game is won when all four Foundation piles are built up in suit, Ace to King.

These rules will form the basis of a Frecell Rules Engine. Note that we’re not interested in a UI at the moment.

This post is a follow on from my previous post of how to setup a dotnet core environment for doing TDD.

red - first test

We know from the rules that we need a standard deck of cards to work with, so our initial test could assert that we can create an array, of some type that is yet to be determined, which has a length of 51.

[Fact]
public void Should_CreateAStandardDeckOfCards()
{
    var sut = new Deck();

}

There! Our first test. It fails (by not compiling). We’ve obeyed The 3 Laws of TDD: We’ve not written any production code and we’ve only written enough of the unit test to make it fail. We can make the test pass by creating a Deck class in the Freecell.Engine project. Time for another commit:

green - it passes

It is trivial to make our first test pass, as all we need to do is create a new class in our Freecell.Engine project, and our test passes as it now compiles. We can prove this by instructing dotnet to run our unit tests for us:

nostromo:Freecell.Engine.Tests stuart$ dotnet watch xunit
watch : Started
Detecting target frameworks in Freecell.Engine.Tests.csproj...
Building for framework netcoreapp2.0...
  Freecell.Engine -> /Users/stuart/dev/freecell/Freecell.Engine/bin/Debug/netstandard2.0/Freecell.Engine.dll
  Freecell.Engine.Tests -> /Users/stuart/dev/freecell/Freecell.Engine.Tests/bin/Debug/netcoreapp2.0/Freecell.Engine.Tests.dll
Running .NET Core 2.0.0 tests for framework netcoreapp2.0...
xUnit.net Console Runner (64-bit .NET Core 4.6.00001.0)
  Discovering: Freecell.Engine.Tests
  Discovered:  Freecell.Engine.Tests
  Starting:    Freecell.Engine.Tests
  Finished:    Freecell.Engine.Tests
=== TEST EXECUTION SUMMARY ===
   Freecell.Engine.Tests  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 0.142s
watch : Exited
watch : Waiting for a file to change before restarting dotnet...

It is important to make sure to run dotnet xunit from within the test project folder, you can’t pass the path to the test project like you can with dotnet test. As you can see, I’ve also started watching xunit, and the runner is now going to wait until I make and save a change before automatically compiling and running the tests.

red, green

This first unit test still doesn’t really test very much, and because we are obeying the 3 TDD rules, it forces us to think a little before we write any test code. When looking at the rules, I think we will probably want the ability to move through our deck of cards and have the ability to remove cards from the deck. So, with this in mind, the most logical thing to do is to make the Deck class enumerable. We could test that by checking a length property. Still in our first test, we can add this:

var sut = new Deck();

var length = sut.Length;

If I switch over to our dotnet watch window, we get the immediate feedback that this has failed:

Detecting target frameworks in Freecell.Engine.Tests.csproj...
Building for framework netcoreapp2.0...
  Freecell.Engine -> /Users/stuart/dev/freecell/Freecell.Engine/bin/Debug/netstandard2.0/Freecell.Engine.dll
DeckTests.cs(13,30): error CS1061: 'Deck' does not contain a definition for 'Length' and no extension method 'Length' accepting a first argument of type 'Deck' could be found (are you missing a using directive or an assembly reference?) [/Users/stuart/dev/freecell/Freecell.Engine.Tests/Freecell.Engine.Tests.csproj]
Build failed!
watch : Exited with error code 1
watch : Waiting for a file to change before restarting dotnet...

We know that we have a pretty good idea that we’re going to make the Deck class enumerable, and probably make it in implement IEnumerable<>, then we could add some sort of internal array to hold another type, probably a Card and then right a bunch more code that will make our test pass.

But that would violate the 3rd rule, so instead, we simply add a Length property to the Deck class:

public class Deck 
{
    public int Length {get;}
}

This makes our test happy, because it compiles again. But it still doesn’t assert anything. Let’s fix that, and assert that the Length property actually has a length that we would expect a deck of cards to have, namely 52:

var sut = new Deck();

var length = sut.Length;

length.Should().Be(51);

The last line of the test asserts through the use of FluentAssertions that the Length property should be 51. I like FluentAssertions, I think it looks a lot cleaner than writing something like Assert.True(sut.Length, 51), and it’s quite easy to read and understand: ‘Length’ should be 51. I love it. We can add it with the command dotnet add package FluentAssertions. Fix the using reference in the test class so that it compiles, and then check our watch window:

Detecting target frameworks in Freecell.Engine.Tests.csproj...
Building for framework netcoreapp2.0...
  Freecell.Engine -> /Users/stuart/dev/freecell/Freecell.Engine/bin/Debug/netstandard2.0/Freecell.Engine.dll
  Freecell.Engine.Tests -> /Users/stuart/dev/freecell/Freecell.Engine.Tests/bin/Debug/netcoreapp2.0/Freecell.Engine.Tests.dll
Running .NET Core 2.0.0 tests for framework netcoreapp2.0...
xUnit.net Console Runner (64-bit .NET Core 4.6.00001.0)
  Discovering: Freecell.Engine.Tests
  Discovered:  Freecell.Engine.Tests
  Starting:    Freecell.Engine.Tests
    Freecell.Engine.Tests.DeckTests.Should_CreateAStandardDeckOfCards [FAIL]
      Expected value to be 51, but found 0.
      Stack Trace:
           at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
           at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args)
           at FluentAssertions.Numeric.NumericAssertions`1.Be(T expected, String because, Object[] becauseArgs)
        /Users/stuart/dev/freecell/Freecell.Engine.Tests/DeckTests.cs(16,0): at Freecell.Engine.Tests.DeckTests.Should_CreateAStandardDeckOfCards()
  Finished:    Freecell.Engine.Tests
=== TEST EXECUTION SUMMARY ===
   Freecell.Engine.Tests  Total: 1, Errors: 0, Failed: 1, Skipped: 0, Time: 0.201s
watch : Exited with error code 1
watch : Waiting for a file to change before restarting dotnet...

Now to make our test past, we could again just start implementing IEnumerable<>, but that’s not TDD, and Uncle Bob might get upset at me. Instead, we will do the simplest thing that will make the test pass:

public class Deck
{
    public int Length { get { return new string[51].Length; }}
}

refactor

Now that we have a full test with an assertion that passes, we can about the refactor stage of the red/gree/refactor TDD cycle. As it stands, our simple classes passes our test but we can see right away that newing up an array in the getter of the Length property is not going to be something that is going to serve our interests well in the long run, so we should do something about that. Making it a member variable seems to be the most logical thing to do at the moment, so we’ll do that. We don’t need to make any changes to our test on the refactor stage. If we do, that’s a design smell that would indicate that something is wrong.

ublic class Deck
{
    private const int _size = 51;
    private string[] _cards = new string[_size];
    public int Length { get { return _cards.Length; }}
}

Conclusion

In this post, we’ve fleshed out our Deck class a little more, and gone through the full red/green/refactor TDD cycle. I also introduced FluentAssertions, and showed the output from the watch window as it showed the test failing

A few months ago I left a busy startup job I’d had for over a year. The work was engrossing: I stopped blogging, but I was programming every day. I learned a completely new language, but got plenty of chances to use my existing knowledge. That is, after all, why they hired me.

I especially liked something that might seem boring: combing through logs of occasional server errors and modifying our code to avoid them. Maybe it was because I had setup the monitoring system. Or because I was manually deleting servers that had broken in new ways. The economist in me especially liked putting a dollar value on bugs of this nature: 20 useless servers cost an extra 500 dollars a week on AWS.

But, there’s only so much waste like this to clean up. I’d automated most of the manual work I was doing and taught a few interns how to do the rest. I spent two weeks openly wondering what I’d do after finishing my current project, even questioning whether I’d still be useful with the company’s new direction.

Career Tip: don’t do this.

That’s when we agreed to part ways. So, there I was, no “official” job but still a ton of things to keep me busy. I’d help run a chain of Hacker Hostels in Silicon Valley, I was still maintaining Wine as an Ubuntu developer, and I was still a “politician” on Ubuntu’s Community Council having weekly meetings with Mark Shuttleworth.

Politiking, business management, and even Ubuntu packaging, however, aren’t programming. I just wasn’t doing it anymore, until last week. I got curious about counting my users on Launchpad. Download counts are exposed by an API, but not viewable on any webpage. No one else had written a proper script to harvest that data. It was time to program.

And man, I went a little nuts. It was utterly engrossing, in the way that writing and video games used to be. I found myself up past 3am before I even noticed the time; I’d spent a whole day just testing and coding before finally putting it on github. I rationalized my need to make it good as a service to others who’d use it. But in truth I just liked doing it.

It didn’t stop there. I started looking around for programming puzzles. I wrote 4 lines of python that I thought were so neat they needed to be posted as a self-answered question on stack overflow. I literally thought they were beautiful, and using the new yield from feature in Python3 was making me inordinately happy.

And now, I’m writing again. And making terrible cartoons on my penboard. I missed this shit. It’s fucking awesome.

Lock’n'Roll, a Pidgin plugin for Windows designed to set an away status message when the PC is locked, has received its first update in three and a half years!

Daniel Laberge has forked the project and released a version 1.2 update which allows you to specify which status should be set when the workstation locks. Get it while it’s awesome (always)!

terVec3 lb = ti->points[1] - ti->points[0];
terVec3 lc = ti->points[2] - ti->points[0];
terVec2 lbt = ti->texCoords[1] - ti->texCoords[0];
terVec2 lct = ti->texCoords[2] - ti->texCoords[0];

// Generate local space for the triangle plane
terVec3 localX = lb.Normalize2();
terVec3 localZ = lb.Cross(lc).Normalize2();
terVec3 localY = localX.Cross(localZ).Normalize2();

// Determine X/Y vectors in local space
float plbx = lb.DotProduct(localX);
terVec2 plc = terVec2(lc.DotProduct(localX), lc.DotProduct(localY));

terVec2 tsvS, tsvT;

tsvS[0] = lbt[0] / plbx;
tsvS[1] = (lct[0] - tsvS[0]*plc[0]) / plc[1];
tsvT[0] = lbt[1] / plbx;
tsvT[1] = (lct[1] - tsvT[0]*plc[0]) / plc[1];

ti->svec = (localX*tsvS[0] + localY*tsvS[1]).Normalize2();
ti->tvec = (localX*tsvT[0] + localY*tsvT[1]).Normalize2();

#define SH_AMBIENT_FACTOR   (0.25f)
#define SH_LINEAR_FACTOR    (0.5f)
#define SH_QUADRATIC_FACTOR (0.3125f)

void LambertDiffuseToSHCoefs(const terVec3 &dir, float out[9])
{
  // Constant
  out[0] = 1.0f * SH_AMBIENT_FACTOR;

  // Linear
  out[1] = dir[1] * SH_LINEAR_FACTOR;
  out[2] = dir[2] * SH_LINEAR_FACTOR;
  out[3] = dir[0] * SH_LINEAR_FACTOR;

  // Quadratics
  out[4] = ( dir[0]*dir[1] ) * 3.0f*SH_QUADRATIC_FACTOR;
  out[5] = ( dir[1]*dir[2] ) * 3.0f*SH_QUADRATIC_FACTOR;
  out[6] = ( 1.5f*( dir[2]*dir[2] ) - 0.5f ) * SH_QUADRATIC_FACTOR;
  out[7] = ( dir[0]*dir[2] ) * 3.0f*SH_QUADRATIC_FACTOR;
  out[8] = 0.5f*( dir[0]*dir[0] - dir[1]*dir[1] ) * 3.0f*SH_QUADRATIC_FACTOR;
}


void RotateCoefsByMatrix(float outCoefs[9], const float pIn[9], const terMat3x3 &rMat)
{
  // DC
  outCoefs[0] = pIn[0];

  // Linear
  outCoefs[1] = rMat[1][0]*pIn[3] + rMat[1][1]*pIn[1] + rMat[1][2]*pIn[2];
  outCoefs[2] = rMat[2][0]*pIn[3] + rMat[2][1]*pIn[1] + rMat[2][2]*pIn[2];
  outCoefs[3] = rMat[0][0]*pIn[3] + rMat[0][1]*pIn[1] + rMat[0][2]*pIn[2];

  // Quadratics
  outCoefs[4] = (
        ( rMat[0][0]*rMat[1][1] + rMat[0][1]*rMat[1][0] ) * ( pIn[4] )
      + ( rMat[0][1]*rMat[1][2] + rMat[0][2]*rMat[1][1] ) * ( pIn[5] )
      + ( rMat[0][2]*rMat[1][0] + rMat[0][0]*rMat[1][2] ) * ( pIn[7] )
      + ( rMat[0][0]*rMat[1][0] ) * ( pIn[8] )
      + ( rMat[0][1]*rMat[1][1] ) * ( -pIn[8] )
      + ( rMat[0][2]*rMat[1][2] ) * ( 3.0f*pIn[6] )
      );

  outCoefs[5] =  (
         ( rMat[1][0]*rMat[2][1] + rMat[1][1]*rMat[2][0] ) * ( pIn[4] )
       + ( rMat[1][1]*rMat[2][2] + rMat[1][2]*rMat[2][1] ) * ( pIn[5] )
       + ( rMat[1][2]*rMat[2][0] + rMat[1][0]*rMat[2][2] ) * ( pIn[7] )
       + ( rMat[1][0]*rMat[2][0] ) * ( pIn[8] )
       + ( rMat[1][1]*rMat[2][1] ) * ( -pIn[8] )
       + ( rMat[1][2]*rMat[2][2] ) * ( 3.0f*pIn[6] )
       );

  outCoefs[6] = (
              ( rMat[2][1]*rMat[2][0] ) * ( pIn[4] )
       +      ( rMat[2][2]*rMat[2][1] ) * ( pIn[5] )
       +      ( rMat[2][0]*rMat[2][2] ) * ( pIn[7] )
       + 0.5f*( rMat[2][0]*rMat[2][0] ) * ( pIn[8])
       + 0.5f*( rMat[2][1]*rMat[2][1] ) * ( -pIn[8])
       + 1.5f*( rMat[2][2]*rMat[2][2] ) * ( pIn[6] )
       - 0.5f * ( pIn[6] )
          );

  outCoefs[7] =  (
         ( rMat[0][0]*rMat[2][1] + rMat[0][1]*rMat[2][0] ) * ( pIn[4] )
       + ( rMat[0][1]*rMat[2][2] + rMat[0][2]*rMat[2][1] ) * ( pIn[5] )
       + ( rMat[0][2]*rMat[2][0] + rMat[0][0]*rMat[2][2] ) * ( pIn[7] )
       + ( rMat[0][0]*rMat[2][0] ) * ( pIn[8] )
       + ( rMat[0][1]*rMat[2][1] ) * ( -pIn[8] )
       + ( rMat[0][2]*rMat[2][2] ) * ( 3.0f*pIn[6] )
       );

  outCoefs[8] = (
              ( rMat[0][1]*rMat[0][0] - rMat[1][1]*rMat[1][0] ) * ( pIn[4] )
        +     ( rMat[0][2]*rMat[0][1] - rMat[1][2]*rMat[1][1] ) * ( pIn[5] )
        +     ( rMat[0][0]*rMat[0][2] - rMat[1][0]*rMat[1][2] ) * ( pIn[7] )
        +0.5f*( rMat[0][0]*rMat[0][0] - rMat[1][0]*rMat[1][0] ) * ( pIn[8] )
        +0.5f*( rMat[0][1]*rMat[0][1] - rMat[1][1]*rMat[1][1] ) * ( -pIn[8] )
        +0.5f*( rMat[0][2]*rMat[0][2] - rMat[1][2]*rMat[1][2] ) * ( 3.0f*pIn[6] )
         );
}

float3 SampleSHQuadratic(float3 dir, float3 shVector[9])
{
  float3 ds1 = dir.xyz*dir.xyz;
  float3 ds2 = dir*dir.yzx;    // xy, zy, xz

  float3 v = shVector[0];

  v += dir.y * shVector[1];
  v += dir.z * shVector[2];
  v += dir.x * shVector[3];

  v += ds2.x * shVector[4];
  v += ds2.y * shVector[5];
  v += (ds1.z * 1.5 - 0.5) * shVector[6];
  v += ds2.z * shVector[7];
  v += (ds1.x - ds1.y) * 0.5 * shVector[8];

  return v;
}

void SHForDirection(const terVec3 &dir, float out[9])
{
  // Constant
  out[0] = 1.0f;

  // Linear
  out[1] = dir[1] * 3.0f;
  out[2] = dir[2] * 3.0f;
  out[3] = dir[0] * 3.0f;

  // Quadratics
  out[4] = ( dir[0]*dir[1] ) * 15.0f;
  out[5] = ( dir[1]*dir[2] ) * 15.0f;
  out[6] = ( 1.5f*( dir[2]*dir[2] ) - 0.5f ) * 5.0f;
  out[7] = ( dir[0]*dir[2] ) * 15.0f;
  out[8] = 0.5f*( dir[0]*dir[0] - dir[1]*dir[1] ) * 15.0f;
}

terVec3 RandomDirection(int (*randomFunc)(), int randMax)
{
   float u = (((float)randomFunc()) / (float)(randMax - 1))*2.0f - 1.0f;
   float n = sqrtf(1.0f - u*u);

   float theta = 2.0f * M_PI * (((float)randomFunc()) / (float)(randMax));

   return terVec3(n * cos(theta), n * sin(theta), u);
}

It was Thursday afternoon, a completely sensible hour, but for me I had been woken up by the call. In my sleepy haze I hadn’t realized that this quickly turned into a surprise job interview. I made the mistake of saying that, while I had worked plenty with Ubuntu packaging and scripted application tests, I hadn’t actually written any of Wine’s C code.

“Oh.”

I began to feel the consequences of the impression I’d given. “Well, we want a real developer.” Without thinking, I’d managed to frame my years of experience in precisely the wrong way. All that integration work, application testing, knowledge of scripting languages and the deep internals of Ubuntu suddenly counted for nothing. It didn’t matter how much work I’d done or how many developer summits I’d been sponsored to: in this moment I was someone who didn’t even write simple Wine conformance tests.

We talked some more, and I went back to bed to wake at midnight, technically Friday. Too late and too early to go out, everything was quiet enough to get some real work done. I thought about the earlier conversation, and while I hadn’t written C code since high school I decided to dive right back in and hack at Wine myself. Within minutes I found a bug, and four hours later I had code not only proving it, but also fixing it for good.

Test driven development

Today’s Wine consists of two equally important parts: a series of implementations pretending to be parts of Windows, and an ever-growing suite of unit tests. The implementations are straightforward once you know the right thing to do: if the waiter function in Windows’ restaurant.dll asks for a tip, then ours needs to as well. Similarly, the tests prove what the right thing actually is, on both Wine and Windows. They help us answer weird questions, like if the Windows waiter still demands a tip with a negative bill. Somewhere out there, there’s a Windows program that depends on this behavior, and it will be broken in Wine if we make the mistake of assuming Windows acts reasonably.

I asked a developer to recommend a DLL that needed better tests, picked a random C file in it, and started looking. I soon found my target, a documented and complete implementation of a function with only partial tests. This code is already written, and believed working, in Wine. I was supposed to write a bunch of tests that should pass in Wine. That’s when I learned the function is already broken.

The Wine Testbot

Wine owes a lot to the late Greg Geldorp, an employee of VMware who originally created a Windows testbot for us. Any developer can submit a test patch to it, and those tests will be run on multiple versions of Windows to make sure they pass. It saves us the trouble of having to reboot into 10 different versions of Windows when hacking on Wine.

When I used the testbot to run my new tests, however, I found that while they passed on Wine they actually failed on Windows. Since we’re supposed to do what Windows does, no matter how stupid, that meant two problems: my new tests were bad, and Wine had a bug. Fixing the tests is simple enough – you just change the definition of “pass” – but this kind of unexpected failure can also inspire even more tests. By the end of it I had 6 patches adding different tests to just one function, 3 of which were marked “todo_wine”.

Fixing Wine

While simply submitting tests would certainly be a useful contribution, I felt like I could do more. “You found the mess, you clean it up” is an annoying cliché, but here it has a ring of truth to it: my recent experience writing these tests meant that I had become the world expert on this function. At least, for the next few days, after which I planned on forgetting it forever. That’s what good tests are for – they let us confidently ignore the internals of done code. In the off chance we break it unintentionally, they tell us exactly what’s wrong.

And so I wrote what was supposed to be my final patch: one that fixed Wine and marked the tests as no longer todo. In true open source fashion, I sent it to a friend for review, where he promptly informed me that, while my new tests were passing, I’d created a place where Wine could crash. The solution is, unsurprisingly, yet more tests to see how Windows handles the situation (keeping in mind that sometimes Windows handles things by crashing). This is typical in Wine development: your first attempt at a patch often results in mere discovery that the problem is harder to solve than you thought.

The real world

None of this actually matters, of course, unless the bug I’d fixed was actually affecting a real application that someone would want to run. Did I actually fix anything useful? I don’t know. It’s not exactly easy to get a list of all Windows applications that rely on edge-case behavior of shlwapi.dll’s StrFromTimeInterval function, but at least Wine is more correct now.

Apparent correctness isn’t the end-all of software development, of course. It’s possible doing something seemingly correct in Wine can make things worse: if my initial version of a fix slipped in to the code, for instance, an application could go from displaying a slightly wrong string to flat-out crashing. That’s why unit tests are just one part of software QA – you still need peer review of code and actual application testing.

Part of something greater

Incidentally, the whole experience reminded me of a blog post I had written over a year ago about modeling Wine development. My model was telling me that what I had just done was a bit inefficient: I made a modest improvement to Wine, but it wasn’t directly inspired by a particular real world application. Perhaps it would have been better had I tackled a more salient bug in a big name application, rather than polishing off some random function in string.c. But it wasn’t random: another developer recommended this particular code section to me because it was missing tests, and he noticed this precisely because in the past some untested behavior in a similar function was breaking a real application for him.

This function is now done. The coding was relatively simple by Wine standards – no need for expertise in COM, Direct3D, OLE, or any number of Windows conventions that O’Reilly writes entire books about. Wine doesn’t need experts: it needs a lot of grunt work from people like me. People willing to tackle one function at a time until by sheer attrition we end up with a test suite so exhaustive that everything can be simply guaranteed to work. That’s how we win in the end. That’s how real developers do it.

The Problem

For those using chef to automate your server infrastructure you probably find managing third-party cookbooks to be a pain. Ideally I want to make custom changes to a cookbook while still being able to track for upstream enhancements.

A few techniques I see being used are:

no tracking: Manually download an archive from github or opscode community and drop it in your cookbooks/ directory. Easy to make custom changes but you have no automated way to check for updates.

git submodules: This tracks upstream well, but unless you own the repo you can’t make changes.

fork it: Since pretty much all cookbooks reside on github, so you can fork a copy. This works, but now you might have dozens of different repos to manage. And checking for updates means going into each repo and manually merging in enhancements from the upstream.

knife vendor: Now we are getting somewhere. Chef’s knife command has functionality for dealing with third-party cookbooks. It looks something like this:

knife cookbook site vendor nginx

This downloads the nginx cookbook from the opscode community site, puts an unmodified copy in a chef-vendor-nginx git branch, and then puts a copy in your cookbooks/nginx dir in your master branch. When you run the install again it will download the updated version into the chef-vendor-nginx branch, and then merge that into master.

This is a good start, but it has a number of problems. First you are restricted to using what is available on the opscode community site. Second, although this seems like a git-centric model, knife is actually downloading a .tar.gz file. In fact if you visited the nginx cookbook page you would see it only offers an archive download, no way to inspect what this cookbook actually provides before installing.

There is a sea of great high-quality cookbooks on github. Since we all know and love git it would be great if we could get the previous functionality but using git repositories as a source instead.

A Solution

Enter knife-github-cookbooks. This gem enhances the knife command and lets us get the above functionality by pulling from github rather than downloading from opscode. To use it just install the gem and run:

knife cookbook github install cookbooks/nginx

By default it assumes a username/repo from github. So for each cookbook you install you will have a chef-vendor-cookbookname branch storing the upstream copy and a cookbooks/cookbook-name directory in your master branch to make local changes to.

If you want to check for upstream changes:

knife cookbook github compare nginx

That will launch a github compare view. You can even pass this command a different user who has forked the repo and see or merge in changes from that fork! Read more about it on the github page.

One thing to keep in mind is this gem doesn’t pull in required dependencies automatically, so you will have to make sure you download any requirements a cookbook might have. You can check what dependencies a cookbook requires by inspecting the metadata files.

Bonus Tip!

Opscode has a github repository full of recipes you probably want to use (opscode/cookbooks). Unfortunately using this repository would mean pulling in *all* of those cookbooks. That just clutters up your chef project with cookbooks you don’t need. Luckily there is https://github.com/cookbooks! This repository get updated daily and separates each opscode/cookbook cookbook into a separate git repository.

Now you can cherry-pick the cookbooks you want, and manage them with knife-github-cookbooks!

Make sure you have the same path to the binaries you want to profile on your target device as your workstation.

On your profiling target as root:

export KREXP=dpkg -L kernel-debug | grep "vmlinux-2.6"
opcontrol --init
opcontrol --vmlinux=$KREXP
opcontrol --separate=kernel
opcontrol --status
opcontrol -c=8

start your application

as root again:

opcontrol --stop;  opcontrol --reset; opcontrol --start; sleep 5; opcontrol --stop

rm -rf /var/lib/oprofile && scp -r root@192.168.2.15:/var/lib/oprofile /var/lib/
this should give you some log info:
opreport -l -r

opreport -c /path/to/binary_to_profile > /tmp/opreport.txt
oprofile-callgraph-to-svg -e 1 -n 1 /tmp/opreport.txt

for(nsIntRegionRectIterator iter(mUpdateRegion); const nsIntRect* R=iter.Next();)
//do something with R

The problem:

Sometimes one version of Wine can run an application, but fail on others.  Maybe there’s a regression in the latest Wine version, or you’ve installed an optional patch that fixes one application but breaks another.

The solution to this is to have more than one version of Wine installed on the system, and have the system determine which version of Wine is best for the application.  This implies two different levels of usability – advanced users may want to configure and tweak which Wine runs which App, but mere humans won’t even want to know such a thing exists.

This is the reason why many people have created Wine front ends: they worry about things like patches and registry hacks and native DLLs so that users won’t have to.  You just click a button for the application you want to install, put in the requisite disc or password, and it does all the shaman dances for you.  Codeweaver’s, the chief sponsor of Wine, stays in business through sales of Crossover, which is basically just a front end to a special version of Wine.

So, these front ends exist to solve some very real and important problems. But now we have a new problem, in that we might have more than one front end — playonlinux, winetricks, and others all need to deal with this too.  In true open source fashion, we need to work together and come up with a standard.

My proposed solution:

Distribution packagers like me make a separate package, say, wine-hotfixes.  This package replaces (on Ubuntu via dpkg-diversions) the /usr/bin/wine binary, moving the existing one to /usr/bin/wine-default.

The new /usr/bin/wine will pass everything to wine-default, unless it detects the environment variable WINEHOTFIXES is set to something other than null. If it is set (to, say WINEHOTFIXES=”6971,12234″), then the script will look in /etc/wine/hotfixes.d/ and /etc/wine/hotfixes.conf for alternative wine versions that might make it happy.  In the case of a partial match it will prioritize bugs in the listed order.

hotfixes.d will contain a series of config files, one for each alternative version of wine.  These could be installed manually, but generally they’ll come from special packages — say, a version of Wine built with a workaround for that annoying mouse escape bug.  Each file will give a path (say, /opt/wine-hotfixes/yokos-tweaked-wine) and which bugs it hotfixes.  hotfixes.conf can specify a list of bugs that the default Wine already fixes, as well as which bugs to ignore (eg that are probably fixed by every hotfix).

Start menu items (.desktop entries) can then work exactly as they do now, except they will have the WINEHOTFIXES environment variable set, generally as created by a front end.  If the user has no alternative wine versions, or no wine-hotfixes package, nothing different will happen and everything will still use the wine-default.  If the user upgrades Wine to a version that fixes all the worked around bugs, the default will be used again (forward-compatible) — all that’s needed is for the newer Wine package to ship an updated hotfixes.conf.

The beauty of this is that a front end can specify a list of bugs to workaround without actually having a ready hotfix – if needed that can be handled by someone else.  Similarly, the hotfixed Wines don’t actually need to know about which app they’ll be running, as wine-hotfixes will do all the matchmaking.  We also keep /usr/bin free, except for one added binary.

Wrapping it all up

The real test of a design, of course, is if it can be any simpler.  With the above, we’ve got it down to a single configuration item depending on who’s using it — hotfixes.d for the packager, hotfixes.conf for a manual tweaker, and the WINEHOTFIXES environment variable for the front end author.

It is of course worth asking if we should be doing this at all.  Zero configurations are better than one, after all, and ideally one magic Wine package would run every application flawlessly.  But that’s what we’ve been trying for years now, and people keep feeling the need to write these front ends anyway — we’re clearly not doing well enough without them, so we might as well manage them and work together.  This way, at least everything is backwards (and forwards) compatible: environment variables mean nothing without wine-hotfixes, and if we ever do invent a perfect version of Wine all the applications installed by front ends will continue to work just fine.

That should wrap it up, unless I’ve missed something.

My last post detailed how to compile the Io language from source and install it in Ubuntu (10.10 Maverick). Io has a growing set of addons such as GUI’s, sound and image manipulation, OpenGL, and database support to name a few. However they will not be enabled if you don’t have the proper development libraries installed.

I’ll go through a couple of addons in this article, but if you just want to make sure you have as many dependencies as possible to run the addons here is a line you can paste:

$ sudo apt-get install build-essential cmake libreadline-dev libssl-dev ncurses-dev libffi-dev zlib1g-dev libpcre3-dev libpng-dev libtiff4-dev libjpeg62-dev python-dev libpng-dev libtiff4-dev libjpeg62-dev libmysqlclient-dev libmemcached-dev libtokyocabinet-dev libsqlite3-dev libdbi0-dev libpq-dev libgmp3-dev libogg-dev libvorbis-dev libtaglib-cil-dev libtag1-dev libtheora-dev libsamplerate0-dev libloudmouth1-dev libsndfile1-dev libflac-dev libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev libxmu-dev libxi-dev libxml2-dev libyajl-dev uuid-dev liblzo2-dev zlib1g-dev

You will need to rebuild Io once these are all installed.

I would encourage you to browse the addons/* directory in the Io source tree. There are many good useful addons and samples, although unfortunately there are few that do not seem to currently work or are missing samples, so dust off that book on C

Sockets

sudo apt-get install libevent-dev

Here is a minimal webserver using sockets:

WebRequest := Object clone do(
    cache := Map clone
    handleSocket := method(socket, server,
        socket streamReadNextChunk
        if(socket isOpen == false, return)
        request := socket readBuffer betweenSeq("GET ", " HTTP")         
 
        data := cache atIfAbsentPut(request,
            writeln("caching ", request)
            f := File clone with(request)
            if(f exists, f contents, nil)
        )                                                                
 
        if(data,
            socket streamWrite("HTTP/1.0 200 OK\n\n")
            socket streamWrite(data)
        ,
            socket streamWrite("Not Found")
        )                                                                
 
        socket close
        server requests append(self)
    )
)                                                                        
 
WebServer := Server clone do(
    setPort(7777)
    socket setHost("127.0.0.1")
    requests := List clone
    handleSocket := method(socket,
        WebRequest handleSocket(socket, self)
    )
) start

Lots of other good socket based examples in addons/Socket/samples.

Regex

sudo apt-get install libpcre3-dev

That will install Perl Compatible Regular Expression support for Io. You can use it like:

regex := Regex with("(?\\d+)([ \t]+)?(?\\w+)")
match := "73noises" matchesOfRegex(regex) next

CFFI

During the configure process you might have noticed a message saying Could NOT find FFI  (missing:  FFI_INCLUDE_DIRS).  FFI (foreign function interface) is basically a system that lets us call functions in different programming languages. First make sure you have the development libraries:

$ sudo apt-get install libffi-dev

How FFI functions is very architecture and compiler dependent, and it seems debian places the includes in a location the cmake scripts aren’t looking. I’m not that familiar with cmake and couldn’t find a very elegant solution, so just place the following line in the modules/FindFFI.cmake script:

$ vim modules/FindFFI.cmake
 
# Add the following line
set(FFI_INCLUDE_DIRS /usr/include/x86_64-linux-gnu)
# Above these two
include(FindPackageHandleStandardArgs)
FIND_PACKAGE_HANDLE_STANDARD_ARGS(FFI DEFAULT_MSG FFI_INCLUDE_DIRS FFI_LIBRARIES)

Here is a small program that gets us direct access to libc’s puts(3) function:

CFFI
 
lib := Library clone setName("libc.so.6")
puts := Function with(Types CString) setLibrary(lib) setName("puts")
 
puts "Hello Io!"

Python

sudo apt-get install python-dev

Want to access Python from Io?

# Import a module
sys := Python import("sys")
 
"Which version of python are we running?" println
sys version println
 
"Split a string" println
str := "Brave brave Sir Robin"
str println
string split(str) println
 
"Load a C module (.so)" println
t := Python import("time")
 
writeln("Current time is: ", t time)

Databases

sudo apt-get install libmysqlclient-dev libmemcache-dev libtokyocabinet-dev libsqlite3-dev libdbi0-dev

Io has addons for MySQL, PostgresQL, memcached, Tokyo Cabinet, SQLite and a few others.

Sound

sudo apt-get install libgmp3-dev libogg-dev libvorbis-dev libtaglib-cil-dev libtag1-dev libtheora-dev libsamplerate0-dev libloudmouth1-dev libsndfile1-dev libflac-dev

Various sound processing libraries.

Images

$ sudo apt-get install libpng-dev libtiff4-dev libjpeg62-dev

Various image loading libraries.

GUI

$ sudo apt-get install x11proto-xf86misc-dev xutils-dev libxpm-dev libpango1.0-dev libcairo2-dev libfreetype6-dev 
 
$ sudo apt-get install libclutter-1.0-dev libatk1.0-dev

There is also a GUI called Flux that requires OpenGL support. I wasn’t able to get it working however.

OpenGL

$ sudo apt-get install libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev libxmu-dev libxi-dev

Lots of great examples in addons/OpenGL/samples.

XML and JSON

$ sudo apt-get install libxml2-dev libyajl-dev

If you need to do any XML or JSON parsing.

UUID

$ sudo apt-get install uuid-dev

Support for UUID generator. Seems to be broken however.

Misc

$ sudo apt-get install libreadline-dev libssl-dev ncurses-dev libffi-dev zlib1g-dev liblzo2-dev zlib1g-dev

SSL, archives, REPL history, curses GUI.

I have recently begun reading through Bruce Tate’s fun Seven Languages In Seven Weeks book. One of the chapters focuses the Io language and it’s installation can be a little bit non-standard to get it to my liking.

Generally on my development machine when I compile from source I like to install locally to my home directory rather than system wide. This way sudo privileges are not needed plus I just like the idea of keeping these items close to home.

First Io requires the cmake build system so make sure that is available.

$ sudo apt-get install cmake

Next download and extract the source code.

$ wget --no-check-certificate http://github.com/stevedekorte/io/zipball/master -O io-lang.zip
$ unzip io-lang.zip
$ cd stevedekorte-io-[hash]

Io provides a build script, however it is setup to install the language to /usr/local. Since I want it to go in $HOME/local you just need to modify that file. Here is a quick one liner:

$ sed -i -e 's/^INSTALL_PREFIX="\/usr\/local/INSTALL_PREFIX="$HOME\/local/' build.sh

Now build and install.

$ ./build.sh
$ ./build.sh install

Since we are installing into a location our OS doesn’t really know about, we need to configure a few paths.

$ vim ~/.bashrc
export PATH="${HOME}/local/bin:${PATH}"
export LD_LIBRARY_PATH="${HOME}/local/lib:${LD_LIBRARY_PATH}"
 
# You might want these too
export LD_RUN_PATH=$LD_LIBRARY_PATH
export CPPFLAGS="-I${HOME}/local/include"
export CXXFLAGS=$CPPFLAGS
export CFLAGS=$CPPFLAGS
export MANPATH="${HOME}/local/share/man:${MANPATH}"

Lastly restart your shell and type ‘io’ and you should be dropped into Io’s REPL!

A side benefit to this method is you can install anything you build into $HOME/local. Usually you just need to pass the –prefix=$HOME/local parameter when you run a ./configure script.

int iY = px[0] + 2*px[1] + px[2];    // 0..1020
int iCo, iCg;

if (iY == 0)
{
    iCo = 0;
    iCg = 0;
}
else
{
    iCo = (px[0] + px[1]) * 255 / iY;
    iCg = (px[1] * 2) * 255 / iY;
}

px[0] = (unsigned char)iCo;
px[1] = (unsigned char)iCg;
px[2] = 0;
px[3] = (unsigned char)((iY + 2) / 4);

float3 DecodeYCoCgRel(float4 inColor)
{
    return (float3(4.0, 0.0, -4.0) * inColor.r
        + float3(-2.0, 2.0, -2.0) * inColor.g
        + float3(0.0, 0.0, 4.0)) * inColor.a;
}