I just had to setup my vimrc and vimfiles on a new laptop for work, and had some fun with Vim, mostly as it’s been years since I had to do it. I keep my vimfiles folder in my github, so I can grab it wherever I need it.

To recap, one of the places that Vim will look for things is $HOME/vimfiles/vimrc, where $HOME is actually the same as %USERPROFILE%. In most corporate environments, the %USERPROFILE% is actually stored in a networked folder location, to enable roaming profile support and help when a user gets a new computer.

So you can put your vimfiles there, but, it’s a network folder - it’s slow to start an instance of Vim. Especially if you have a few plugins.

Instead, what you can do is to edit the _vimrc file in the Vim installation folder (usually in C:\Program Files (x86)\vim), delete the entire contents and replace it with:

set rpt+=C:\path\to\your\vimfiles
set viminfo+=nC:\path\to\your\vimfiles\or\whatever
source C:\path\to\your\vimfiles\vimrc

What this does is:

1. Sets the runtime path to be the path to your vimfiles
2. Tells vim where to store/update the viminfo file (which stores useful history state amongst other things)
3. Source your vimrc file and uses that

This post largely serves as a memory aid for myself when I need to do this again in future I won’t spend longer than I probably needed to googling it to find out how to do it, but I hope it helps someone else.

Recently I was inspired by @buhakmeh’s blog post, Supercharge Blogging With .NET and Ruby Frankenblog to write something similar, both as an exercise and excuse to blog about something, and as a way of tidying up the metadata on my existing blog posts and adding header images to old posts.

High level requirements

The initial high level requirements I want to support are:

1. Cross-platform. This blog is jekyll based, and as such is written in markdown. Any tool I write for automation purposes should be cross-platform.
2. Easily add posts from the command line, and have some default/initial yaml header metadata automatically added.
3. See a high level overview of the current status of my blog. This should include things like the most recent post, how many days I’ve been lazy and not published a post, available drafts etc
4. Publish posts from the command line, which should update the post with published status and add the published date to the yaml header and filename.
5. Create a customised post header for each post on the blog, containing some kind of blog branding template and the post title, and update or add the appropriate yaml header metadata to each post. This idea also comes from another @buhakmeh’s post.
6. The blog has many years of blog posts, spread across several different blogging platforms before settling on Jekyll. As such, some of the yaml metadata for each blog post is… not consistent. Some effort should go into correcting this.
7. Automaticlly notify Twitter of published posts.

Next steps

The next series of posts will cover implementing the above requirements… not necessarily in that order. First I will go over setting up the project and configuring Oakton.

After that I will probably cover implementing fixes to the existing blog metadata, as I think that is going to be something that will be required in order for any sort of Info function to work properly, as all of the yaml metadata will need to be consistent.

Then I think I’ll tackle the image stuff, which should be fairly interesting, and should give a nice look to the existing posts, as having prominent images for posts is part of the theme for the blog, which I’ve not really taken full advantage of.

I’ll try to update this post with links to future posts, or else make it all a big series.

dotnet new console --name BlogHelper9000

At work, we have recently been porting our internal web framework into .net 6. Yes, we are late to the party on this, for reasons. Suffice it to say I currently work in an inherently risk averse industry.

Anyway, one part of the framework is responsible for getting reports from SSRS.

The way it did this is to use a wrapper class around a SOAP client generated from good old ReportService2005.asmx?wsdl, using our faithful friend svcutil.exe. The wrapper class used some TaskCompletionSource magic on the events in the client to make the client.LoadReportAsync and the other *Async methods actually async, as the generated client was not truely async.

Fast forward to the modern times, and we need to upgrade it. How do we do that?

Obviously, Microsoft are a step ahead: svcutil has a dotnet version - dotnet-svcutil. We can install it and get going:

dotnet too install --global dotnet-svcutil

Once installed, we can call it against the endpoint:

dotnet-svcutil http://server/ReportServer/ReportService2005.asmx?wsdl

In our wrapper class, the initialisation of the client has to change slightly, because the generated client is different to the original svcutil implementation. Looking at the diff between the two files, it’s because the newer version of the client users more modern .net functionality.

The wrapper class constructor has to be changed slightly:

public Wrapper(string url, NetworkCredential credentials)
{
    var binding = new BasicHttpBinding(BasicHttpSecurityMode.TransportCredentialOnly);
    binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Ntlm;
    binding.MaxReceivedMessageSize = 10485760; // this is a 10mb limit
    var address = new EndpointAddress(url);

    _client = new ReportExecutionServiceSoapClient(binding, address);
    _client.ClientCredentials.Windows.AllowedInpersonationLevel = TokenImpersonationLevel.Impersonation;
    _client.ClientCredentials.Windows.ClientCredential = credentials;
}

Then, the code which actually generates the report can be updated to remove all of the TaskCompletionSource, which actually simplifies it a great deal:

public async Task<byte[]> RenderReport(string reportPath, string reportFormat, ParameterValue[] parameterValues)
{
    await _client.LoadReportAsync(null, reportPath, null);
    await _client.SetExecutionParametersAsync(null, null, parameterValues, "en-gb");
    var deviceInfo = @"<DeviceInfo><Toolbar>False</ToolBar></DeviceInfo>";
    var request = new RenderRequest(null, null, reportFormat, deviceInfo);
    var response = await _client.RenderAsync(request);
    return response.Result;
}

You can then do whatever you like with the byte[], like return it in an IActionResult or load it into a MemoryStream and write it to disk as the file.

who is eating cereal anymore? Literally don’t think I’ve seen someone eat a bowl of cereal in twenty years

Recently we realised that we had quite a few applications being deployed through Octopus Deploy, and that we had a number of Environments, and a number of Channels, and that managing the ports being used in Dev/QA/UAT across different servers/channels was becoming… problematic.

When looking at this problem, it’s immediately clear that you need some way of dynamically allocating a port number on each deployment. This blog post from Paul Stovell shows the way, using a custom Powershell build step.

As we’d lost track of what sites were using what ports, and that we also have ad-hoc websites in IIS that aren’t managed by Octopus Deploy, we thought that asking IIS “Hey, what ports are the sites you know about using?” might be a way forward. We also had the additional requirement that on some of our servers, we also might have some arbitary services also using a port and that we might bump into a situation where a port was chosen that was already being used by a non-IIS application/website.

Researching the first situation, it’s quickly apparent that you can do this in Powershell, using the Webadministration module. Based on the answers to this question on Stackoverflow, we came up with this:

Import-Module Webadministration

function Get-IIS-Used-Ports()
{
    $Websites = Get-ChildItem IIS:\Sites

    $ports = foreach($Site in $Websites)
    {
        $Binding = $Site.bindings
        [string]$BindingInfo = $Binding.Collection
        [string]$Port = $BindingInfo.SubString($BindingInfo.IndexOf(":")+1,$BindingInfo.LastIndexOf(":")-$BindingInfo.IndexOf(":")-1)

        $Port -as [int]
    }

    return $ports
}

To get the list of ports on a machine that are not being used is also fairly straightforward in Powershell:

function Get-Free-Ports()
{
    $availablePorts = @(49000-65000)
    $usedPorts = @(Get-NetTCPConnection | Select -ExpandProperty LocalPort | Sort -Descending | Where { $_ -ge 49000})

    $unusedPorts = foreach($possiblePort in $usedPorts)
    {
        $unused = $possiblePort -notin $usedPorts
        if($unused)
        {
            $possiblePort
        }
    }

    return $unusedPorts
}

With those two functions in hand, you can work out what free ports are available to be used as the ‘next port’ on a server. It’s worth pointing out that if a site in IIS is stopped, then IIS won’t allow that port to be used in another website (in IIS), but the port also doesn’t show up as a used port in netstat -a, which is kind of what Get-NetTCPConnection does.

function Get-Next-Port()
{
    $iisUsedPorts = Get-IIS-Used-Ports
    $freePorts = Get-Free-Ports

    $port = $freePorts | Where-Object { $iisUsedPorts -notcontains $_} | Sort-Object | Select-Object First 1

    Set-OctopusVariable -Name "Port" -Value "$port"
}

Then you just have to call it at the end of the script:

Get-Next-Port

You’d also want to have various Write-Host or other logging messages so that you get some useful output in the build step when you’re running it.

If you found this because you have a build server which is ‘offline’, without any external internet access because of reasons, and you can’t get your build to work because dotnet fails to restore the tool you require for your build process because of said lack of external internet access, then this is for you.

In hindsight, this may be obvious for most people, but it wasn’t for me, so here it is.

In this situation, you just need to shy away from local tools completely, because as of yet, I’ve been unable to find anyway of telling dotnet not to try to restore them, and they fail every build.

Instead, I’ve installed the tool(s) as a global tool, in a specific folder, e.g. C:\dotnet-tools, which I’ve then added to the system path on the server. You may need to restart the build server for it to pick up the changes to the environment variable.

One challenge that remains is how to ensure the dotnet tools are consistent on both the developer machine, and the build server. I leave that as an exercise for the reader.

I’m leaving this here so I can find it again easily.

We had a problem updating the Visual Studio 2019 Build Tools on a server, after updating an already existing offline layout.

I won’t go into that here, because it’s covered extensively on Microsoft’s Documentation website.

The installation kept failing, even when using --noweb. It turns out that when your server is completely cut off from the internet, as was the case here, you also need to pass --noUpdateInstaller.

This is because (so it would seem) that even though --noweb correctly tells the installer to use the offline cache, it doesn’t prevent the installer from trying to update itself, which will obviously fail in a totally disconnected environment.

Since a technical breakdown of how Betsy does texture compression was posted, I wanted to lay out how the compressors in Convection Texture Tools (CVTT) work, as well as provide some context of what CVTT's objectives are in the first place to explain some of the technical decisions.

First off, while I am very happy with how CVTT has turned out, and while it's definitely a production-quality texture compressor, providing the best compressor possible for a production environment has not been its primary goal. Its primary goal is to experiment with compression techniques to improve the state of the art, particularly finding inexpensive ways to hit high quality targets.

A common theme that wound up manifesting in most of CVTT's design is that encoding decisions are either guided by informed decisions, i.e. models that relate to the problem being solved, or are exhaustive.  Very little of it is done by random or random-like searching. Much of what CVTT exists to experiment with is figuring out techniques which amount to making those informed decisions.

CVTT's ParallelMath module, and choice of C++

 

The plan to post a play-by-play for dev kind of fell apart as I preferred to focus on just doing the work, but the Windows port was a success.

If you want some highlights:

• I replaced the internal resource format with ZIP archives to make it easier to create custom resource archives.
• PICT support was dropped in favor of BMP, which is way easier to load.  The gpr2gpa tool handles importing.
• Ditto with dropping "snd " resource support in favor of WAV.
• Some resources were refactored to JSON so they could be patched, mostly dialogs.
• Massive internal API refactoring, especially refactoring the QuickDraw routines to use the new DrawSurface API, which doesn't have an active "port" but instead uses method calls directly to the draw surface.
• A bunch of work to allow resolution changes while in-game.  The game will load visible dynamic objects from neighboring rooms in a resolution-dependent way, so a lot of work went in to unloading and reloading those objects.
  

The SDL variant ("AerofoilSDL") is also basically done, with a new OpenGL ES 2 rendering backend and SDL sound backend for improved portability.  The lead version on Windows still uses D3D11 and XAudio2 though.

Unfortunately, I'm still looking for someone to assist with the macOS port, which is made more difficult by the fact that Apple discontinued OpenGL, so I can't really provide a working renderer for it any more.  (Aerofoil's renderer is actually slightly complicated, mostly due to postprocessing.)

Goin' mobile

In the meantime, the Android port is under way!  The game is fully playable so far, most of the work has to do with redoing the UI for touchscreens.  The in-game controls use corner taps for rubber bands and battery/helium, but it's a bit awkward if you're trying to use the battery while moving left due to the taps being on the same side of the screen.

Most of the cases where you NEED to use the battery, you're facing right, so this was kind of a tactical decision, but there are some screens (like "Grease is on TV") where it'd be really nice if it was more usable facing left.

I'm also adding a "source export" feature: The source code package will be bundled with the app, and you can just use the source export feature to save the source code to your documents directory.  That is, once I figure out how to save to the documents directory, which is apparently very complicated...

Anyway, I'm working on getting this into the Google Play Store too.  There might be some APKs posted to GitHub as pre-releases, but there may (if I can figure out how it works) be some Internal Testing releases via GPS.  If you want to opt in to the GPS tests, shoot an e-mail to codedeposit.gps@gmail.com

Will there be an iOS port?

Maybe, but there are two obstacles:

The game is GPL-licensed and there have reportedly been problems with Apple removing GPL-licensed apps from the App Store, and it may not be possible to comply with it.  I've heard there is now a way to push apps to your personal device via Xcode with only an Apple ID, which might make satisfying some of the requirements easier, but I don't know.

Second, as with the macOS version, someone would need to do the port.  I don't have a Mac, so I don't have Xcode, so I can't do it.

A conservative estimate has me shooting hogs in 45 seconds

As part of modernisng, updating and generally overhauling my blog, I thought it would be nice to add some consistancy to the Yaml front matter used by Jekyll. For those who do not know, Jekyll uses Yaml front matter blocks to process any file which contains one as a special file. The front matter can contain variables in the form foo: value. Jekyll itself defines some predefined globabl variables and variables for posts, but anything else is valid and can be use in Liquid tags.

I wondered if I could write some F# to:

1. Load all the markdown files.
2. Parse all the front matter.
3. Modify the front matter to drop variables no longer required by a theme.
4. Update the front matter with new variables which are understand by the current theme.
5. Randomly assign a path to a header image file for each post which doesn’t already have one.
6. Write the front matter back to its post.

Fairly straightforward requirements.

Loading and parsing the front matter

I’m using YamlDotNet to do most of the heavy lifting. I think could also have used the FSharp.Configuration Type Provider, but I’m not sure that it would have done exaclty what I wanted.

I’m just writing this in an F# script, hosted in a project. After adding the YamlDotNet NuGet package, we can reference it and get to work:

#r "../../.nuget/packages/YamlDotNet/8.1.2/lib/netstandard2.1/YamlDotNet.dll"

open System.IO
open System.Text.RegularExpressions
open YamlDotNet.Serialization
open YamlDotNet.Serialization.NamingConventions

let path = "../sgrassie.github.io/_posts"

Here, we reference the package, and then open various namespaces for use later on. The code for my blog is kept in a separate folder, relative to the project which has got the fsharp scripts I’m writing abot in it. This is nice and easy.

type FrontMatter() =
    member val Title = "" with get, set
    member val Description = "" with get, set
    member val Layout = "" with get, set
    member val Tags = [|""|] with get, set
    member val Published = "" with get, set
    member val Category = "" with get, set
    member val Categories = "" with get, set
    member val Metadescription = "" with get, set
    member val Series = "" with get, set
    member val Featured = false with get, set
    member val Hidden = false with get, set
    member val Image = "" with get, set
    [<YamlMember(Alias = "featured_image", ApplyNamingConventions = false)>]
    member val FeaturedImage = "" with get, set
    [<YamlMember(Alias = "featured_image_thumbnail", ApplyNamingConventions = false)>]
    member val FeaturedImageThumbnail = "" with get, set
    [<YamlIgnore>]
    member val MarkdownFilePath = "" with get, set

This is a class with auto-implemented properties. You can see three attributes in use. The YamlMember attribute allows us to alias a property in Yaml which doesn’t follow the CamelCase convention we configured the deserialiser with. I think that a C# version of this would look pretty much the same.

let deserializer = DeserializerBuilder()
                     .WithNamingConvention(CamelCaseNamingConvention.Instance)
                     .Build()

This initialises the YamlDotNet deserialiser, and is pretty much almost exactly how you would do this in C#. To deserialise something, we need some Yaml. When I was testing this, I got an error in YamlDotNet that was pretty weird and essentially means that it can’t parse the file, and it turns out it’s because all the other stuff outside the Yaml front matter that is upsetting it.

let expression = "(?:---)(?<yaml>[\\s\\S]*?)(?:---)"

Oh regex, I do love thee.

Very simply, this regex will parse everything in a file between two --- blocks, into a named Yaml group. We now have actual front matter, we still need to parse into an object.

let extractFrontmatter filePath =
    let file = File.ReadAllText(filePath)
    let result = Regex.Match(file, expression).Groups.["yaml"].Value
    let frontMatter =
        let frontMatter = deserializer.Deserialize<FrontMatter>(result)
        frontMatter.MarkdownFilePath <- filePath
        frontMatter
    frontMatter

This is a bit more complex so lets unpack it:

1. Pass in the filePath.
2. Read all of the text from it.
3. Strip only the front matter from the text.
4. Parse the front matter test with an inner function, which uses the deserializer, and return it. Here, we also keep track of the file path (we will need this later).

We also need to load all of the markdown files:

let loadMarkdownFiles path = Directory.EnumerateFiles(path, "*.md", SearchOption.AllDirectories) 

Notice how those last couple of functions are using ‘currying’. It lets us do all of the work in one pipeline:

path |> loadMarkdownFiles |> Seq.map extractFrontmatter |> Seq.iter (fun x -> printfn "%s - %s" x.MarkdownFilePath x.Title)

This gives us a dataset to work with. Next time we’ll continue with the rest of the requirements.

Many years ago, after working in my first programming job for a couple of years the company was taken over, and coding tests for new hires were introduced. The incumbent developers all decided to take the test, and it was seen as a fun diversion for a couple of hours.

I don’t have access to the actual wording of the requirements given to candidates, but the test required a text file containing around 100k words to be loaded and sorted into the largest set of the longest anagram. For example in the words file I’m using in this blog post, there are 466544 words in the file, 406627 of which are anagrams. The largest set is for a 7 letter anagram, of whih there are 15 words. There are smaller sets of longer anagrams, we’re not interested in those. And, it had to run in in less a second. They had three hours to write it, on a computer not connected to the internet. They had access to Java, through Eclipse, C/C++/C# through Visual Studio and Delphi through Embarcadero Studio.

I don’t know where the test originally came from - I think it originated in a different company which had been acquired by the same company I now worked for, but I’m not sure. I think the intent of the test was to in part gauge how the candiate reacted to the deadline pressure, part how they could understand the requirements given to them, and lastly what sort of code they wrote.

As it has been a long time and the company no longer recruits after moving most development overseas, so, I’m going to present my solution.

The Solution

First we have to load the file, and figure out to generate the anagram and keep track of how many instances of that anagram there are. It turned out for the candidates taking the test that this was the bit that most got stuck on, specifically the short mental leap it took to working out you needed to sort the letters of the word alphabetically to create the key.

private static string CreateKey(string word)
{
    var lowerCharArray = word.ToLowerInvariant().ToCharArray();
    Array.Sort(lowerCharArray);
    return new string(lowerCharArray);
}

private static void LoadWords(string filePath, Dictionary<string, List<string>> words)
{
    using (var streamReader = File.OpenText(filePath))
    {
        string s;

        while ((s = streamReader.ReadLine()) != null)
        {
            var key = CreateKey(s);

            if (words.TryGetValue(key, out var set))
            {
                set.Add(s);
            }
            else
        
                var newSet = new List<string> {s};
                words.Add(key, newSet);
            }
        }
    }
}

words is a Dictionary<string, List<string>>, which we use to track the count of anagrams. The rest of the file loading is a fairly standard while loop over the reader ReadLine method, checking the dictionary to see if the anagram has already been found, and if so add the new word to the set, otherwise, add the anagram and create a new list to hold the word(s).

Once we have all the words loaded and matched into sets of anagrams, we can process them to work out which is the largest set with the longest word.

private static KeyValuePair<string, List<string>> ProcessAnagrams(Dictionary<string, List<string>> words)
{
    var largestSet = 0;
    var longestWord = 0;
    var foundSet = new KeyValuePair<string, List<string>>();

    foreach (var set in words)
    {
        if (set.Value.Count >= largestSet)
        {
            largestSet = set.Value.Count;

            if (set.Key.Length > longestWord)
            {
                longestWord = set.Key.Length;
                foundSet = set;
            }
            else
            {
                longestWord = 0;
            }
        }
    }

    return foundSet;
}

Here we simply bruteforce check all of the entries in the dictionary to find the answer. It’s not elegant, but it gets the job done. Running it on my Macbook Pro gives:

406627 anagrams processed from 466544 in 00:02:850
File read and key generation in 00:02:829
Anagrams searched in: 00:00:021
Found: 
Key: AEINRST (7), Count: 15
aeinrst
antsier
asterin
eranist
nastier
ratines
resiant
restain
retains
retinas
retsina
stainer
starnie
stearin
Tersina

There are lots of blog comment systems, and this blog has used Disqus as the comment system for a long time. I’m not going to go into all the reasons to move away from Disqus, but page load times and wanting more control over your data and being able to respect your readers privacy figure highly.

Also, this blog is a technical blog focused on software development and associated topics, and this means that anyone who wants to comment on my blog is almost certain to be familar with Github and have an account, and also be as uncomfortable using Disqus as I have been.

I did investigate rolling my own code based on examples from other blogs, who have used some jekyll liquid templates and javascript to pull from the Github API and use it to post comments back to the repo hosting the blog. This has some attraction, but also has a big drawback, which is the authorisation situation to the Github API, as you don’t really want your client id and client secret exposed in the repo.

Enter utteranc.es

You can get around this by hosting an app in heroku to use as the postback url so that you can hide the client id and client secret, and there is also staticman, but none of these seemed as simple as just using utteranc.es

To configure utteranc.es, head over to the website and follow the instructions, and fill out the form to suit you. For the blog post to issue mapping, I chose ‘Issue title contains page title’, and I also chose to have utteranc.es add a ‘Comment’ label to the issue it creates in the blog repository. After you do that, you’ll get a code snippet generated for you that looks somewhat like this:

<script src="https://utteranc.es/client.js"
        repo="sgrassie/sgrassie.github.io"
        issue-term="title"
        label="Comment"
        theme="github-light"
        crossorigin="anonymous"
        async>
</script>

Add this to a jekyll include, for example utterances.html and then include it in your post.html layout at the position you want the blog comments to appear. Most jekyll blog templates have Disqus support, so it will probably just be a simple case of finding where in the layout that Disqus is included, and replacing it.

Exporting existing comments

If your existing comments are not important to you, then at this point you can stop and enjoy your new Github powered comment system. Personally for me, it’s the principle of the thing, and the fact that the comments on my blog belong to me, and the author of the comment. So, we can do something about it.

Disqus allows you to export your comments, and once you do so, you will get your comments emailed to the email registered with your Disqus account. I’ve done a lot of work with XML in a previous role, and I think that the Disqus XML export looks… odd. The reason I say that is that each post on your blog appears to be mapped to a <thread> element, which contains a bunch of expected metadata about the blog post. I would expect each individual comment to be a nested in a <comments> element, but this is not the case. Instead, each individual comment has an entry as a <post> element at the same level as the <thread>, and they are mapped to each other using and attribute id. I don’t think that makes any sense, I’m sure there must be good reasons. I just can’t think what might be.

A comment then, looks like this:

<thread dsq:id="1467739952">
    <id>218 http://temporalcohesion.co.uk/?p=218</id>
    <forum>temporalcohesion</forum>
    <category dsq:id="2467491" />
    <link>http://temporalcohesion.co.uk/2010/10/25/lets-write-an-api-library-for-github/</link>
    <title>Let&amp;#8217;s write an API library for Github</title>
    <message />
    <createdAt>2010-10-25T12:00:24Z</createdAt>
    <author>
        <name>Stuart Grassie</name>
        <isAnonymous>false</isAnonymous>
        <username>stuartgrassie</username>
    </author>
    <isClosed>false</isClosed>
    <isDeleted>false</isDeleted>
</thread>

An actual comment on this post looks like:

<post dsq:id="952258229">
    <id>wp_id=25</id>
    <message><![CDATA[<p>Great post Stu!</p>]]></message>
    <createdAt>2010-10-25T22:47:44Z</createdAt>
    <isDeleted>false</isDeleted>
    <isSpam>false</isSpam>
    <author>
        <name>John Sheehan</name>
        <isAnonymous>true</isAnonymous>
    </author>
    <thread dsq:id="1467739952" />

You can see the way that the post element is mapped back to the containing thread using the dsq:id attribute.

Parsing the XML

The strange structure of the XML makes it less straightforward to parse the XML, as it means we’ll have to do a little bit of work in matching up blog posts and the comments on them. Also very annoying is the fact that a thread element doesn’t know if it actually has any associated post comments.

We can acomplish this fairly easily with a little bit of F# and the FSharp.Data XmlProvider. Setting the provider up is straightforward, here I’m just using a direct reference to the assembly which I’d previously added via NuGet.

#r "../../.nuget/packages/fsharp.data/3.3.3/lib/netstandard2.0/FSharp.Data.dll
open FSharp.Data

type Disqus = XmlProvider<"/Users/stuart/Downloads/temporalcohesion-2020-07-13T20 27 09.014136-all.xml">

type Comment = { Author: string; Message: string; Created: System.DateTimeOffset; ParentThreadId: int64; }
type BlogPost = { Title: string; Url: string; Author: string; ThreadId: int64; Comments : Comment list }

let data = Disqus.Load("/Users/stuart/Downloads/temporalcohesion-2020-07-13T20 27 09.014136-all.xml")

If you are new to F# (and I’m still fairly new) this might look scary, but it really isn’t. After referencing the assembly in the script, we open the FSharp.Data namespace, and then initialise an XmlProvider by passing it the XML the file we’re going to parse.

That enables the XmlProvider to infer a lot of things about the XML in the file, and then the XmlProvider loads the actual data from the file. Two records are also defined to hold the details about the Threads/Posts that are going to imported, and how multiple comments map refer to a single blog post. These records are analagous to simple C# POCO classes with getters and setters.

With these types ready, we can define a couple of functions to convert the XML into them, and thus do a way with a lot of the extraneous noise from the XML, that we don’t really care about.

let toComments posts =
    posts
    |> Seq.filter (fun (post : Disqus.Post) -> not post.IsSpam || not post.IsDeleted)
    |> Seq.map (fun (post : Disqus.Post) -> {Author = post.Author.Name; Message = post.Message; Created = post.CreatedAt; ParentThreadId = post.Thread.Id})
    |> Seq.toArray

let toBlogPosts posts =
    posts
    |> Seq.filter (fun (thread : Disqus.Thread) -> not thread.IsDeleted)
    |> Seq.map (fun (thread : Disqus.Thread) -> {Title = thread.Title; Url = thread.Link.Substring(0, thread.Link.Length - 1); Author = thread.Author.Name; ThreadId = thread.Id; Comments = [] })

These functions use currying, which as a longtime C# developer I’m still getting the hang of, and that will come in handy shortly. They map the Disqus types generated by the XmlProvider into the custom types I defined, taking care to filter out comments we don’t want to import and not importing any blog posts which Disqus says have been deleted.

With those functions defined, we need a way of mapping the comments to the correct blog post.

let mapBlogToComments(post, comments) =
    let commentsOnPost = comments 
                         |> Array.filter (fun comment -> comment.ParentThreadId = post.ThreadId) 
                         |> Array.toList
    {post with Comments = commentsOnPost}

Here we take a single post, and all of the comments, and then use a nested function to grab the set of comments associated to that post, by way of the ThreadId. With that written, we can use some more currying to create another function that will do a lot of hard work for us:

let addCommentsToTheirPosts comments = data.Threads |> toBlogPosts |> Seq.map (fun post -> mapBlogToComments(post, comments))

This function will take the threads, use the toBlogPosts method to turn them into BlogPost and then map each blog post to the correct comments using the method we’ve just defined to do that. But where do the comments come from? Well, it turns out this currying thing is really quite useful, as it enables all this magic looking |>, or ‘piping’ to happen.

let toImport = data.Posts
               |> toComments
               |> addCommentsToTheirPosts 
               |> Seq.filter (fun x -> x.Comments.Length > 0)

Take all the posts data, turn them all into comments, and then pipe that to the addCommentsToTheirPosts function, and then filter out blog posts which don’t have any comments, as importing those is pointless. All for around 24 lines of code. I know full well the C# it would take do all that, and whilst with C# 8 you could probably get close, I doubt you’d equal 24 lines.

Just to be on the safe side, it’s probably a good idea to look through each of the posts and comments that we’ve now got to just to see if things are matching up correctly.

toImport |> Seq.iter (fun post -> printfn "%s - %s - comments: %d" post.Title post.Url post.Comments.Length)

Running that will give you an idea of what blog posts are going to be imported, and the number of comments. The first time I ran this, I found some of the blog posts in the Disqus XML import did not have the posts title set, so I was getting duplicated post titles. As there were only three instances of this error, I just manually corrected the XML and re-reran the script to check I had everything correct.

Uploading to GitHub

So far, so good. Now comes the fun part and something I’ve yet to do in F#, which is interop with a C# library. It turns out that it’s not so hard, but that makes perfect sense when you understand that F# is a .net language, just like C#. A long time ago I started to write an API library for GitHub, but I gave it up in favour of Octokit.net.

We can easily reference Octokit and open the namespace as before:

#r "../../.nuget/packages/octokit/0.48.0/lib/netstandard2.0/Octokit.dll"
open Octokit

Then we just need to setup a few variables:

let repo = "sgrassie.github.io"
let githubApp = "foo"
let token = "<your-personal-access-token-here>"
let credentials = Credentials(token)
let header = ProductHeaderValue(githubApp)
let client = GitHubClient(header, Credentials = credentials) 

These just get us a client to work with, and all I did was just register a new Personal Access Token on my account to use as the password. Notice how with F# you don’t need to new anything, even though they are classes from a C# assembly. These can then be used in the following function, which I’m gonna prefix with this warning:

It does work though, so just… use at your own caution.

let exportToGithub posts =
    for post in posts do
        System.Threading.Thread.Sleep(2000)
        let issuebody = sprintf "Comment thread for the post [%s](%s)" post.Title post.Url
        printfn "%s" issuebody
        let newIssue = NewIssue(post.Title, Body = issuebody)
        let issue = client.Issue.Create("sgrassie", repo, newIssue) |> Async.AwaitTask |> Async.RunSynchronously
        printfn "New issue created for %s" post.Title
        for comment in post.Comments do
           System.Threading.Thread.Sleep(2000)
           let message = sprintf "Comment by **%s** on **%s** (imported from Disqus):\r\n\r\n%s" comment.Author (comment.Created.ToString("f")) comment.Message
           let newComment = client.Issue.Comment.Create("sgrassie", repo, issue.Number, message) |> Async.AwaitTask |> Async.RunSynchronously
           printfn "    New comment created for %s" comment.Author

toImport |> exportToGithub 

I’m sure that a more experienced F# person is going to look at that and be like “WTF”, but as I said, it does work. I left the printfn log messages in, but essentially it loops over each post, waits a couple of seconds and then creates the new issue, and then loops over all of the comments for that post and adds then as comments to the issue. I put the Thread.Sleep’s in the there just so I didn’t hammer the Github API, but honestly there was that few to import I doubt it would have trigged the rate limit, but I imagine a more popular blog with more comments on the posts woould.

I’ve been upgrading part of our build infrastructure to handle the ongoing upgrade to .net core, and as part of that, I’ve had to update the Cake build script to handle doing the restore in an offline environment, on the build server.

There is a great post on the Octopus blog about writing a Cake build script for .net core, I encourage you to check that out, I’m not going to repeat too much of that.

My specific requirement is that the `DotNetCoreRestore’ needs to succeed on an ‘offline’ build server, that is, a build server that has no access to the internet.

In order for this to succeed, you are going to need provide a way for NuGet to get the packages, usually this is done by maintaining an offline NuGet cache which you can point NuGet at, or even checking the packages into the repository. I’d always recommend going with the first option, although there are scenarios were the second option might be required.

However you do it, you need to tell NuGet where they are. The easiest thing to do is to use a NuGet.Config local to the .sln, but it is possible to code a location into the script.

Here is the restore task:

Task("Restore")
    .IsDependentOn("Clean")
    .Does(() =>
    {
        var settings = new DotNetCoreRestoreSettings();

        if(BuildSystem.IsRunningOnTeamCity)
        {
            settings.PackagesDirectory = "./packages";
            settings.IgnoreFailedSources = true
            //optionally
            //settings.Sources = new[] { "http://someinternalfeed/nuget" }
        }

        foreach(var project in projects)
        {
            DotNetCoreRestore(project.FullPath, settings);
        }
    });

This project uses a NuGet.config to add the paths of internal package sources, and sets the location of the packages folder to be local to the .sln - we’ve found this cuts down on conflicts on developer machines.

In the previous post, I displayed my fledgling understanding of F# by writing a script which can parse the CSV set of results of the English Premier League to generate the league table. The script does this primarily by using a mutable BCL Dictionary. F# is immutable by default, and whilst that is itself not immutable, you have to go out of your way to enable it. I’ll try to save repeating Scott Wlashin.

There are some improvements that can be made to the script. I’ll highlight them here and then link to the full script as a gist.

First a note on pattern matching. In the previous post I mentioned that I thought I could use pattern matching in a particular place, and obviously I can:

let fullTimeResult =
        match row.FTR with
        | "H" -> Home
        | "A" -> Away
        | _ -> Draw

Rather than if/then/else. Here, the “” is equal to the default in a C# switch statement, if it’s not a Home or Away (win), then it _must be a draw.

Making things immutable

To start making things immutable, we can update the updateTeam function from the previous post, and pass in a Map<string, LeagueRow>:

let updateTeam (league : Map<string, LeagueRow>, team : string, points : int, forGoals : int, againstGoals: int, won : int, drawn, lost: int) =
    if league.ContainsKey team then
        let existing = league.[team]
        let updated = {existing with Played = existing.Played + 1; Won = existing.Won + won; Drawn = existing.Drawn + drawn; Lost = existing.Lost + lost; For = existing.For + forGoals; Against = existing.Against + againstGoals; Points = existing.Points + points}
        league.Add(team, updated)
    else
        let leagueRow = {Team = team; Played = 1; Won = won; Drawn = drawn; Lost = lost; GD = 0; For = forGoals; Against = againstGoals; Points = points}
        league.Add(team, leagueRow)

The code is almost the same as the previous version, except that we no longer use the <- operator to update the mutable dictionary. What’s going on instead is that F# creates a new instance of the LeagueRow, with updated values, and adds that to the Map, by key, which has the side-effect of creating a new instance of the whole Map, with the league row identified by the key replaced with the updated version.

The updateHomeWin function becomes:

let updateHomeWin (league : Map<string, LeagueRow>, result : MatchResult) =
    let league = updateTeam(league, result.HomeTeam, 3, result.HomeGoals, result.AwayGoals, 1, 0, 0)
    let league = updateTeam(league, result.AwayTeam, 0, result.AwayGoals, result.HomeGoals, 0, 0, 1)
    league

This again replaces the BCL Dictionary with the Map, and simply passes the league map through each updateTeam call, and then returns the updated league object.

processMatchResult is also updated to pass in a Map, and calling the fold with it and a default map is straightforward:

|> Seq.fold processMatchResult (Map<string, LeagueRow> [])

This makes the script much more ‘the way of things’ in F#, which is to say it’s using an immutable data structure.

During lockdown, I’ve made another effort at learning F#. This time I think I’ve had a bit more success. Processing data is something that we as developers do on a weekly or even daily basis, so it seems quite natural to practice that in F#. As a big football fan, I’ve decided to use the English Premier League results for season 2019/2020, as it’s a dataset I implicitly understand.

The EPL results set is available in CSV format from football-data.co.uk, and rather than having to parse it all by hand or hitting up CsvHelper and still having to write some C# code to actually use it, in F# we can use a Type Provider, specifically the CsvProvider from FSharp.Data.

Loading and parsing the data

FSharp.Data is easily added via NuGet, and using an .fsx script, we can easily reference the assembly and open the namespace:

#r "../../.nuget/packages/fsharp.data/3.3.3/lib/netstandard2.0/FSharp.Data.dll"
open FSharp.Data
open System.Collections.Generic

I didn’t have any luck in the script with referencing a more local copy of the assembly, such as one in the /bin folder, due to it complaining about not being able to find the FSharp.Data.DesignTime.dll, but going directly to the assembly in the NuGet packages folder seems to work just fine. It is also worth noting that I’m writing this on a Mac (in VS Code), so your path syntax might vary. Also note that we also open the BCL System.Collections.Generic namespace. We’ll need that later.

Next, comes the part that blows my mind. Here is how we generate a type which knows how to load and parse a CSV file of a given structure:

type Results = CsvProvider<"../../Downloads/epl1920.csv">

That’s it. It’s pretty amazing. The Results type is now also type safe, and it’s had a guess at infering what the types are for each column of the data. We could probably do something similar to this in C# using CsvHelper and either Castle.DynamicProxy or some magic with the new Roslyn compiler, but I think it would take quite a bit of code to create something that came close to what this can do.

Skipping over some important stuff that we’ll get to in just short a while, we can now easily load the full results set:

Results.Load("../../Downloads/epl1920.csv")

This is fairly straightforward, and does exactly what it looks like. The data loaded from the file is available in a .Rows property, that we’ll use shortly.

Parsing the data

All good so far, but now things get a little more complicated. Now we need to think somewhat about the data, and if you look in the file… it’s got a LOT of information. Mostly related to betting information for the match, but there is also quite a lot of information about the match itself. For the purposes of calculating the league, most of thie information in the file is redundant. In order to just get the information we need, we can define a Record to hold to that information. A Record in F# is somewhat analagous to a C# POCO class, but with automatic type safety and full equality comparisons out of the box.

type FullTimeResult = | Home | Away | Draw
type MatchResult = {HomeTeam : string; AwayTeam: string; HomeGoals: int; AwayGoals: int; Result: FullTimeResult}

The FullTimeResult type is just like a C# enum, and is easier to read than the ‘A’, ‘H’ or ‘D’ we get from the CSV file for the FTR (Full Time Result) column. I think it also looks nicer to read when it comes to the pattern matching, but we’ll get to that. With those types defined, we can get to the real meat of this and actually parse the data:

let league = Results.Load("../../Downloads/epl1920.csv")
                .Rows
                |> Seq.map toMatchResult
                |> Seq.fold processMatchResult (Dictionary<string, LeagueRow>())
                |> Seq.sortByDescending (fun (KeyValue(_, v)) -> v.Points)

Here, we load the file as we discussed earlier, but now, we forward pipe the data returned from the .Rows property to Seq.map through the toMatchResult function, which takes a Row and extracts the data we’re interested in and returns a new MatchResult. In C# this is the same as doing .Rows.Select(new MatchResult {...}). Then, the resulting sequence of MatchResults is piped forward through the processMatchResult function, using the scary sounding Seq.fold, and it also passed a new instance of a BCL Dictionary, with a string key and a LeagueRow type as the value. I’ve not yet mentioned the LeagueRow type… it’s not super important to proceedings, it just a type which holds all the data you would expect to see in a football league table. For reference it’s included below in the full script.

Amazingly, those five lines load the file, process all the data, and provide an object which contains a fairly accurate version of the English Premier League table. Obviously things are a little more involved than that.

Examing the parsing in more detail

As you’ll recall, the there is a lot of data in the CSV file that is irrelevant when it comes to generating the league table. We can map all the data we need into the MatchResult type, which we do by forward piping the data through Seq.map and the toMatchResult function:

let toMatchResult (row: Results.Row) =
    let fullTimeResult = 
        if row.FTR = "H" then FullTimeResult.Home
        elif row.FTR = "A" then FullTimeResult.Away
        else FullTimeResult.Draw
    {
        HomeTeam = row.HomeTeam
        AwayTeam = row.AwayTeam
        HomeGoals = row.FTHG
        AwayGoals = row.FTAG
        Result = fullTimeResult
    }

This is mostly just a simple mapping from the results row into the new MatchResult type. You’ll notice we don’t need to explicitly ‘new’ anything up, don’t forget, we’re in a functional world now, so the MatchResult is returned as a side affect of what we’re doing. We also define a nested method which processes the full time result using a simple if/else construct. I think I could also have used pattern matching, but it’s simple enough that I’m not going to worry about it.

Next, comes the scarying sounding fold. The method looks like this:

let processMatchResult (league : Dictionary<string, LeagueRow>) result  =
    match result.Result with
    | Home -> updateHomeWin(league, result)
    | Away -> updateAwayWin(league, result)
    | Draw -> updateDraw(league, result)
    league

What happens is that we tell Seq.fold to use this method to do the folding, and we give it an initial state of a new and empty Dictionary<string, LeagueRow>(). Seq.fold carries the state over to each subseqent ‘fold’ over the sequence of MatchResults it was piped. You’ll note that the final thing returned as a side effect of the method is the same dictionary which was passed in. This essentially forms the core of the algorithm to produce the league. The pattern matching of match <thing> with is equivalent to a C# switch statement on steroids. I am barely scratching the surface of what can be done with pattern matching in F#.

The patten patch decides what kind of result we are dealing with, and delegates further processing to the relevant method. Here is the definition for updateHomeWin. The other two methods are exactly the same, except they distribute the points/goals/wins/losses/draws accordingly, so I won’t go into those in detail.

let updateHomeWin (league : Dictionary<string, LeagueRow>, result : MatchResult) =
    updateTeam(league, result.HomeTeam, 3, result.HomeGoals, result.AwayGoals, 1, 0, 0)
    updateTeam(league, result.AwayTeam, 0, result.AwayGoals, result.HomeGoals, 0, 0, 1)

Each MatchResult consists of two teams, and we have to update each entry in the league for both of these teams, with the correct number of points, goals for, goals against, win, draw and loss. The real part of this is in the updateTeam function:

let updateTeam (league : Dictionary<string, LeagueRow>, team : string, points : int, forGoals : int, againstGoals: int, won : int, drawn, lost: int) =
    if league.ContainsKey team then
        let existing = league.[team]
        let updated = {existing with Played = existing.Played + 1; Won = existing.Won + won; Drawn = existing.Drawn + drawn; Lost = existing.Lost + lost; For = existing.For + forGoals; Against = existing.Against + againstGoals; Points = existing.Points + points}
        league.[team] <- updated
    else
        let leagueRow = {Team = team; Played = 1; Won = won; Drawn = drawn; Lost = lost; GD = 0; For = forGoals; Against = againstGoals; Points = points}
        league.Add(team, leagueRow)

This is just a simple dictionary update where we check if a team already has an entry, and if so, update it, otherwise we create it. Things of note here are that whilst F# is mostly immutable, types from System.Collections.Generic are mutable, which is how this whole thing works. I’m sure that someone much better at F# can come along and tell me how to do this with immutable F# collections. Also of note is the collection access of league.[team], which is different than in C#. We also update the value in the dictionary by using <-.

After that, we can define a simple method to print out a row from the league for us, and then iterate through the entries in the dictionary, to get a league table:

let print league =
    printfn "Team: %s | Played: %d | Won: %d | Lost: %d | Drawn: %d | For: %d | Against: %d | GD: %d | Points: %d" league.Team league.Played league.Won league.Lost league.Drawn league.For league.Against (league.For - league.Against) league.Points

league
|> Seq.iter (fun (KeyValue(_, v)) -> print v)

The KeyValue is an active pattern, which matches values of KeyValuePair objects from the BCL Dictionary, and this produces (with data correct as at the publication of this post):

Team: Liverpool | Played: 33 | Won: 29 | Lost: 2 | Drawn: 2 | For: 72 | Against: 25 | GD: 47 | Points: 89
Team: Man City | Played: 33 | Won: 21 | Lost: 9 | Drawn: 3 | For: 81 | Against: 34 | GD: 47 | Points: 66
Team: Leicester | Played: 33 | Won: 17 | Lost: 9 | Drawn: 7 | For: 63 | Against: 31 | GD: 32 | Points: 58
Team: Chelsea | Played: 33 | Won: 17 | Lost: 10 | Drawn: 6 | For: 60 | Against: 44 | GD: 16 | Points: 57
Team: Man United | Played: 33 | Won: 15 | Lost: 8 | Drawn: 10 | For: 56 | Against: 33 | GD: 23 | Points: 55

For completeness here is a gist of the full script:

Suppose you’re on a rocket ship, and you’re given the choice of three buttons: one button starts up your advanced MHE thrusters, but the other buttons explode the ship. You choose some button (say, number 1), and the rocket ship itself (which is a sentient AI), who knows what the buttons are connected to, chooses another button (say, number 3). It then says to you, “Do you want to choose button number 2?”

Is it to your advantage to switch your choice? Also, who designed this control panel?

I second this. I’ve been taping my mouth closed at night for the past year…

¯\_(ツ)_/¯

In every introduction to a potential client, partner, or other associate, the first thing I do is give a brief overview of my history. I know this is common in just about any business or social interaction, but its especially important in my line of work, since communicating my curriculum vitae is so critical to demonstrating my technical competence. And in truth, a technical lead can be as charming as you’d like but unless they are extremely technically competent, nothing else matters. But its hard to compress a lot of data into a short introduction!

I’ve worked on projects for some of the biggest companies in the world — not just Fortune 500 companies, but Fortune 50 companies! — as well as launched dozens of startups, some that didn’t work, some that did, and some that were extremely successful. Those larger companies include industry giants like Disney (three divisions — Disney Parks, Disney Channel, and Disney film!), Dreamworks, Warner Bros., Accenture, CBC, Sega, Sonos, and many more, as well as successful, venture backed startups like Graphite Comics, for which I currently serve as CTO.

I’ve also been written about, and my projects (both professional and personal) have been written about, by publications like the New York Times (which covered the launch of Graphite Comics, and the AI recommendation system I built for it, on the front page of the Business section), The Guardian (twice!), Fortune, TechCrunch, USA Today, AdWeek and many other internationally known outlets, in addition to multiple smaller but no less influential blogs and online journals like PocketGamer, TouchArcade, VentureBeat, and many, many more.

Just typing up that last paragraph and linking to all of that old press was a proud moment. In addition to mainstream press, I’ve also published papers on game related AI, including writeup of my graduate thesis on Hebbian learning in artificial neural networks — a topic I want to get into in more depth later.

That being said, I get asked a lot to dive deeper into my background — not only what I’ve worked on in the past, but why and how, which, to me, are much more interesting questions!

(Note: I will be adding to this as time permits until I’m finished!)

When I first became interested in computer science and programming specifically, there really wasn’t much of an industry per se — at least not anything that looks anything like the industry today. What did exist then was a collection of business-centric hardware and software providers for the most part, building tools to help businesses do the kinds of things businesses did in the 80s — spreadsheets, simple printing software, things like that.

If I try to remember exactly why I became fascinated with computers back then, I genuinely couldn’t tell you! There was very little for me to sink my teeth into — and keep in mind I was a very young kid at this time — certainly not interested in spreadsheets and early databases!

I guess I can liken it to those early video game experiences. Why was it fun to make the white dot chase the grey dot? Why were any of those Atari 2600 games I played as a kid fun in the least? I also can’t answer this question, except to say that in both cases, I think I, and all the other kids intrigued by tech back then, just sensed that something cool was in there. We knew that someday the ultra simple games we played like Pong and Pac Man would mature into nearly photo-realistic, open world masterpieces like GTA V. Maybe we sensed those big, boxy, monochrome machines that just displayed green text would someday fit in our hand, and allow us to do amazing things like fly a drone, video chat with someone around the world, or find our way home.

Whatever the case may be, I found myself drawn to technology in a major way around the age of 10. My neighbor friend’s dad worked for IBM, and he had an early IBM PC that didn’t do much, and I had a few friends from early on who had early Apple computers. But my very first computer was a Commodore 128.

The Commodore 128 was an upgrade from the immensely popular Commodore 64. I believe the Commodore 64 is still the biggest selling single computer of all time. At any rate it was a very popular machine, primarily because you could run games on it — and the games it ran were pretty amazing, especially for the time. Even better, you could plug an Atari 2600 controller right into it.

The C128 came with BASIC, which was my first programming language. I’m actually surprised BASIC, or something like it, isn’t more prevalent these days. BASIC is, after all, extremely basic, and it was a great way for 10 year old me to start learning the essentials of programming. The C128 came with a few big thick manuals, and one of them was a complete guide to programming the machine in BASIC.

This is one of the most interesting and stark differences between the early days of PC use and today. Computers shipped with compilers and manuals explaining how to use them. Imagine if every iMac you bought came with a big book on how to code in Swift and XCode not only preinstalled, but tightly integrated into the operating system! That’s what things were like then — if you shelled out the money for one of these things, it was more likely than not that you planned to build software for it.

But while BASIC was fun, it didn’t take me very far. You really couldn’t do much with BASIC, so I found myself focusing on generating sounds with it. One of the demo programs in the manual (printed out, so you had to copy it line by line off of paper to get it into your computer!) was a program to play a piece of music by generating tones with the C128s pretty amazing audio chip. But other than that — well, without a lot more skill than I had, there wasn’t much farther for me to go.

So I switched my focus to networking. That little C128 had a slot you could plug a modem into, and you could use it to dial into a few online services. Some of these were early Internet-like things that let you play games or talk with people, and as I reflect back on them, they were really pretty amazing pieces of software for the time.

So when I finally upgraded to an IBM compatible PC, the first thing I did is install a modem, and then start to tinker around with Wildcat BBS. I quickly met up with a group of local (Orange County, California) tinkerers who were using Wildcat to connect with other enthusiasts, and eventually became one of the first members of my school’s computer club — where we mostly just messed around with Wildcat. There still wasn’t much out there that was fascinating on the consumer level, at least not, to me, moreso than BBS software. The idea that I could link up with other people through my computer was amazing.

But my time with Wildcat didn’t involve any coding, and programming was what I really wanted to learn.

Eventually, I went to high school, just in time for the school to offer an AP Computer Science course, which I took my junior year. At that time, the course was taught in Pascal, which was a much more complex language than BASIC and I quickly started to imagine the possibilities with this new language. Pascal lead me to C, which led me to C++, and then to Java in its early days. Since my high school only offered the one CS class, I took some college level classes in high school at Fullerton College and California State University, Fullerton in Unix, C, C++ and a data structures course taught in Pascal. I was actually starting to become a polyglot at an early age, which I’m very grateful for, as I know a few very talented developers who struggle learning new languages and platforms. Being thrown into so many so young (mostly because the industry was all over the place back then!) really helped me later on, I think.

I started school at the University of California, Santa Barbara as a double major — Mathematics and Computer Science. This was so early on that very few students even had an email account, as everything was done through dial up into a Unix system using text-based Unix utilities like mail, finger, talk… You could look things up in a really ridiculous way using gopher, archie and eventually on the web using lynx. But then, I was lucky to get one of a very few SLIP accounts, and eventually a PPP account. Don’t even bother looking those up, it was a short lived thing but it was how you could jump on the graphical internet over dialup. Using those technologies, I was the first person I knew in the entire world who could look things up on the web, in an actual browser, with images and everything. And imagine how life changing that was.

I feel like people in their 20s just really can’t imagine how Earth shattering this all was back then.

Anyway, I ended up only taking the Mathematics degree, as I planned on graduate studies in Computer Science and wanted to get to that as soon as possible rather than to an extra year as an undergrad. Specifically, I had planned to chase a PhD in Computer Science and then figure out where to go from there. But then something strange happened…

I went to film school.

I’m not sure when it happened, but it occurred to me that one of the things I loved the most about technology was the creative side of it. I loved games, and music, and film, and all of these wonderful things that you could do with computers suddenly in addition to writing interesting software on them. And I wanted to hone my creative skills. So I started an MFA program in film production at Chapman University.

After school I moved to Los Angeles and ventured briefly into the creative arts — first as a struggling filmmaker, then as a punch up writer (I would be hired to take scripts that didn’t quite work and add jokes or interesting scenes to them). Eventually I got into some support work with a few local film festivals. But I really didn’t enjoy the film industry at all so I started dabbling in something I did enjoy: music.

Now this is going to sound like I was all over the place, and I was. But I started writing and producing electronic music, and pretty quickly started getting hired to mix and produce CDs, remix songs for some well known artists, and eventually even landed a record deal. But the whole time I was more interested in the idea of pushing creative boundaries than anything, and since I was working in the electronic music arena, the way you did that was with software.

So suddenly I found myself back in the software world — this time writing music software. Specifically, I was building virtual effects and instruments using a technology called VST and VSTi. This technology allowed you (and still does to this day) to build instruments and audio processing plugins for any DAW that supports the VST format.

I quickly found myself spending more time writing software than music, and thus I was thrust back into the world of technology.

Around this time I took a job as a network engineer at a company in Orange County, and later in downtown Los Angeles. The life of a network tech is dull, but I did it because it gave me access to two things: tons of computer hardware, and immensely huge Internet bandwidth — the latter of which was tremendously expensive and tremendously hard to get back then.

I used those resources to do two things: first, to run a series of Internet radio stations (yes, “Radio on Internet”) called the Glowdot network, and to build a photo sharing site called Glowfoto.

Glowfoto eventually took over my life, and I developed it into a full-scale social network. I was lucky to ride the MySpace wave early on, and Glowfoto became an early companion site to MySpace. Glowfoto allowed MySpace users to upload more than 10 photos to their profile, back when MySpace had a 10 photo limit (if you can believe that).

Eventually MySpace went away, and about 5 years later I finally retired Glowfoto as well. But during that time I started getting many, many requests for me to build similar services for other companies. And thus my career as a contract software developer began.

(more coming soon!)

I usually like to keep my posts more positive-focused — here’s what you should do vs. here’s what you shouldn’t do. But this week alone I had three potential clients relay to me a very common experience:

I thought I finally found a good developer, but then they suddenly just disappeared!

I’ll admit its hard to explain to entrepreneurs sometimes why that developer ghosted them — usually the explanation feels personal, or accusatory. But in my experience its just a matter of communication! Often its something that comes up in the initial conversations that seems innocuous to the person looking for a developer, but to that developer, its a huge red flag.

In this post, I’ll try to give a few of the reasons that developer may have disappeared into thin air.

I think possibly the best way to explain what’s going on here is to break down a few common statements, and look at what the entrepreneur probably meant to express, and how the candidate developer interpreted it.

However, its important to note that sometimes what was said was exactly what was intended, and what was heard was exactly what was intended! In those cases, I think typically the problem is that the entrepreneur just doesn’t quite understand the market they are entering. You’ll see what I mean when we get to one of those cases.

I also think its important to state a few very important facts right out of the gate.

Most important among these is that contract software developers are still hugely in demand, and hugely in short supply. This may not seem to be evident when you put out an RFP and get snowed with replies. Its important to keep in mind that the vast, vast majority of those replies are from incompetent or just junior developers, non-technical middleman project managers who are going to outsource your project, or direct offshore dev shops staffed by ludicrously unqualified programmers.

When you strip away the worthless proposals and just look at the qualified, experienced developers who are actually capable of building your app, you quickly realize there is a tiny handful of qualified, experienced freelancers out there compared to the massive number of RFPs put up daily.

It’s important, then, to understand that when you do finally zero in on that great developer you should convey an understanding and respect for his or her talent, time and attention. Far too often, good developers get spoken down to by potential clients as if they are in that same pool of dreck that I mentioned above. And in fairness, from the client’s perspective, that might as well be true for the initial portion of the conversation. But it’s still not an encouraging attitude to take with someone you are about to entrust with the technology powering your business!

Another reason for the apparent discrepancy is that to date there still hasn’t been anyone to come along and crack the problem of pairing quality projects with quality developers. Some companies like Upwork have taken great strides in improving the way they match candidates to jobs, but it is still pretty far from perfect, and I’ve found that far too many decent jobs are getting 50+ responses from horrifying developers. That’s not to mention the insane number of very low quality jobs from completely unfunded clients that get posted daily.

Why is this important? Because some potential clients are initially confused why a developer would aggressively pursue them, only to vanish into thin air. After all, if you needed the work bad enough to contact me, the thinking may go, then you must need me more than I need you.

However, that’s seldom true. Instead, because that matching has not been solved, developers often need to cast a wide net to find quality clients, just as clients need to cast a wide net to find a quality developer. So in the end, both parties have been sifting through enormous piles of potential candidates and clients to find the one good one in the bunch.

And for that reason, incidentally, I plan to do a follow up post to this one about why that client ghosted the developer! Believe me, this situation goes both ways.

So anyway, there is still a lot of work to do. But the most important takeaway here, and something I must absolutely stress in the extreme is: there are very few good developers out there, and very many jobs. Once a developer connects with a potential client, it becomes an immediate game of filtering the good from the bad, and trying to determine which employers will be a pleasure to work with, and which will be a nightmare.

Remember: its not just you interviewing a developer. They are also interviewing you!

Ok let’s run through a few of the things you might say that will scare that developer away.

This project is easy, it should only take you a couple hours

What you probably meant to say: I’m not trying to build a nuclear submarine here.

Oof. Gotta get the most common and possibly the worst one out of the way first.

Here’s the deal. If you aren’t a developer, then you most likely don’t really know how long anything takes. That’s one of your developer’s jobs, in fact — to help you estimate time and cost.

But the big problem with a client who doesn’t know how long things take but thinks they do is that those clients will never be happy. Everything will feel like it is taking too long and costing too much money if they think it’s “easy” and “should only take a couple hours” but is in fact hard and takes a long time. And the developer can expect a constant stream of frustrated emails to that effect.

Secondly: almost nothing takes “a couple hours”. Maybe a quick little bug fix — assuming you know where and what the bug is. But otherwise, no.

So this is a bad assumption to make right out of the gate. And although you probably mean this to say “hey, I’m not trying to make your life tough with this!” what the developer hears is “I’m going to make your life a living hell”.

Seriously.

I only need a developer for a few hours to work on something small.

What you probably meant to say: I only need a developer for a few hours to work on something small.

Here is one of those examples of the client saying exactly what they mean. Absolutely nothing wrong with that! And yes, this looks a lot like the last one. It is! Just without the accidental insults.

But the fact is, all you need is a couple hours of that developer’s time! However there are a few considerations to keep in mind

First, that gig that takes just a few hours also requires a few hours to set up and tear down. In development, there is a time cost to jumping into something new that is often hard to explain to a client.

It can take an hour or two just to discuss the requirements of the job. It can take a couple hours to download the current project, open it up, poke around, get everything set up in the dev environment, etc. Then, finally, there is a cost to getting out of the project — delivery, invoicing, debrief meeting, etc.

Are you ok with being billed for those hours? A 2 hour quick in-and-out project could actually take 4-5 hours given the above. Some developers I know who would otherwise take on small quick gigs like this actually end up passing because they find it really hard to explain those extra hours to clients, and it ends up not being worth it.

Second, most freelancers are looking for something long term, not very short term. They’re looking for the weeks-long or months-long project that will sustain them for a while. It would be incredibly brutal to put in all the legwork involved in pairing up with a client a hundred times a month to gather 80 or so 2-hour gigs a month just to earn a sustainable living. In fact, its not even humanly possible — this is a subject for another post, but you might be very surprised to learn the actual cost in time and money involved in securing a client. Suffice to say it is much too high to allow for too many extremely short term clients in the “couple hours of work” range.

And so, unfortunately, these projects often get dropped — even though they very well may turn into more work in the future!

I don’t have any good suggestions for this one. Unfortunately this just is what is is. I can say that eventually you will find someone willing to help you out for that short term. It just might take longer than you expect!

I need someone to come in and take over for the last developer do some bug fixes.

What you probably meant to say: Help!

Ok this one is interesting. From your perspective, “taking over” a project probably seems like a pretty normal thing. However, from a developer’s perspective, a whole lot of alarm signals go up right away.

Here are the first questions that immediately pop into my head:

What happened to the last developer?

Did they quit? Why? Did they get frustrated? Did the project go to hell? Did the client start making unreasonable demands or developer unrealistic expectations? Did the client start demanding a lot of free work?

It is important for the developer to have an understanding of what working with a new client might be like, and you are providing a piece of potentially valuable information here: someone was working for you, and now they aren’t. Something, surely, went wrong.

In order to mitigate against this, be upfront about what happened! Get ahead of this question, and let the prospective developer know what happened to the last one.

How bad is the situation?

No one wants to walk into a project that already has problems. In the best case, software doesn’t have bugs. It isn’t riddled with problems and unknown issues that need to be tracked down.

Now, in the real world, of course there are bugs to be squashed, almost always. But in my experience, the kind of project that starts with “I want to hire you to fix some bugs” is utterly rife with bugs and faults. And walking into that scenario is simply not appealing to any developer.

Mitigate this question by being transparent about the situation. Is the app functional, but there are just some small usability issues? Or is the app crashing constantly and losing data like crazy?

What sort of headspace is this client in?

Are you completely frustrated? Are you now convinced all freelance developers are as incompetent as the offshore developers you just hired for $20 an hour?

If you just had your money lit on fire by a worthless team of offshore developers, and you are now looking for a local developer to fix the situation, its important that you understand the vast difference between the two.

A good local developer is college educated, possibly at the graduate level, and has real experience, has shipped highly rated apps and earns a real living doing this.

The offshore developers you just worked with barely have a highschool education in some instances, and took a 3 week coding crash course before being shoved in a room and told to write code for projects they barely understand. I’m not kidding about this.

So make sure you don’t treat the developer who is potentially going to save your product the same way you want to treat the people who just burned it to the ground.

I have had potential clients straight up tell me they don’t value developers at all and see us all as incompetent wastes of space. And while I really, truly do understand their frustration, having lost it all by going with the hilariously bad cheap option — do you think I had any interest in taking on that job?

Yikes.

No thanks! I’m here to help, not to be insulted. I have a MS in Computer Science and left the Ph.D. program at UCLA to do the work I do now. I’ve been doing this for over 20 years. I am not on the same planet as the $20/hour developer you just hired with 3 weeks of bootcamp training.

Please keep this in mind when you are interviewing prospective onshore developers!

I need a HIPAA compliant app for the public education sector funded by the federal government leveraging military grade encryption for export to 66 countries

What you probably meant to say: this is going to be tough, so I’m willing to pay top dollar.

This one comes up a lot. There are certain types of software development that are just hard work out of the gate. Anything that requires compliance with government regulations, opens up clients or companies to liability of any kind, apps dealing with copyrighted material, etc.

Some of the absolutely best ideas I’ve heard over the years involve the healthcare industry or the public education sector. However, in my experience these projects are brutal and come with huge obstacles right out of the gate!

Sometimes the client just doesn’t know what they are getting themselves into. That happens. But more often than not, they do.

The bigger issue here is that there are other jobs out there that simply don’t come with the headaches of compliance and liability. In order to make these jobs appealing, they just have to come at a higher price, unfortunately.

Sadly, I have known some good clients in the past that did not bring their budgets up to the level required for the quality work required for a project like this, and eventually went offshort for low-quality, low-cost work. In literally every single case, they either ended up with no app at all, or an app that utterly and completely failed to comply with the regulations.

Let me just stress again: these apps are tough. Not only are they stressful for the developer, they require a lot of experience, and that experience, unfortunately, is expensive.

I’m looking for a developer with 8-10 years of experience building apps for the vegan hula hoop industry

What you probably meant to say: my industry is important to me, and I want it to be important to you too.

This one is more humorous to me than anything. But I know a few developers that use this to screen out potential clients.

Here’s the thing: you may be an expert in the sneaker flipping industry, but I’m an expert at software development. You are massively unlikely to find a good software developer who has focused on your particular industry. It just doesn’t work like that.

We jump from project to project by the very nature of what we do. Sometimes we make social apps, sometimes B2B apps, sometimes fintech, sometimes e-commerce. I literally don’t know a single freelance software developer who has built a career focusing on only one industry.

Oh and that sneaker example? That’s one I just saw recently. They wanted a developer who was a sneaker fanatic. And you know what? They may find one, because at least that exists and there may be some overlap. But generally speaking, you really shouldn’t limit the pool of candidates you are interviewing by insisting that they have particular expertise in your field.

And furthermore, it almost never actually matters! Unless you are building a complex AI driven system, in which case experience with AI would be mandatory, or if you are building an image processing app, in which case image manipulation experience would be important.

But the key is: unless you are specifically seeking someone with experience in the particular technology you are developing, then the experience really is unlikely to matter much.

If you are building an e-commerce platform for buying and selling medical equipment, then experience building storefronts, taking payments, managing invoices, etc would be a huge plus. Experience in the medical industry would not.

Please put (some random text) in the subject line. Don’t bother responding unless you are (some random quality)

What you probably meant to say: last time I posted this I got a bunch of weird responses from Indian developers and wasted a whole bunch of time.

I am very sympathetic about this one. However, it is generally not a good idea to treat potential hires like peons just because you had to deal with a bunch of jokers the last time you tried this.

I completely get it though — you will get flooded with responses from low quality developers. However, it is a very, very bad idea to let the one good developer think you have been soured and now consider all developers to be annoying garbage.

I think this one sort of speaks for itself. If I’m deciding whether or not I want to take this project, the last thing I want to risk is working with someone who has a chip on their shoulder!

I want to add a dozen AR filters like Instagram and Facebook have to my app, and I need it done at a fixed price in the next couple weeks!

What you probably meant to say: is this feasable?

The short take on this is simple: the apps you are trying to replicate didn’t start out as complex as you see them now. And unless you want to spend the time and money they did to get there, you should probably rethink this plan.

Instagram, for example, didn’t have video and stories and IGTV, or even gallery posts. It was a dead simple feed of square images bundled with a couple very simple filters.

I replicated the core functionality for two apps i built for Warner Bros. to promote two films several years ago (Tim Burton’s Dark Shadows, and Christopher Nolan’s Dark Knight Rises). And I built the first of those apps in 3 weeks on iOS — and this was before Apple released the CoreImage library.

How? Well because Instagram was pretty simple back then! It took me a couple hours to reverse engineer how the filters were assembled, build a little framework for configuring these filters and applying them to an image, and voila!

However, if you asked me to replicate Instagram now, I’d probably run for the hills. Instagram is complex now! But they got there after many years, many millions of dollars, and an acquisition by Facebook.

See its not just the time requirement. Some of these features are just not feasible when you are still trying to break your development into bite-sized fixed price chunks and having a freelancer build them piecemeal. At a certain point, you need to actually start to structure your company like a company — and that means raising capital to build these functions, hiring full-time engineers instead of bouncing from freelancer to freelancer, and organizing and coalescing your team — engineers, project managers, designers, et al — under one roof, so that they can effectively work as a unit to build the project according to the bigger vision and roadmap you have.

This is a much longer post as well, but to keep it brief: it’s just not realistic to work the way you did at the beginning all the way through this complexity, as your requirements become more and more complex. The first version of Instagram can be built by a freelancer. The modern day version of Instagram requires Facebook.

Now, why is this a problem for a developer? Because by simply asking for this feature in this timeline, they can tell right away that you have unrealistic expectations — not only in terms of the time required to develop these features, but the company structure and funding required to do so. There is a good chance that you don’t have a real solid grasp of what exactly you are asking for, and just how big a task it is, and that can make your project tough to work on!

A lot of developers have come to learn that clients with unrealistic expectations often can’t be talked out of those expectations. And, so, they typically just move on.

I need a scalable backend, an iOS app, and Android app, and a custom algorithm, and I need them all at very high quality in 6 months. And my budget is $3000.

What you probably meant to say: I have no money

It’s ok to take a shot at bringing your vision to life, even if you don’t have enough money to do it exactly the way you’d like. It really is! I am in this business because I love working with visionary people with big, outrageous dreams. The tech industry is a thrilling, vibrant and lucrative place to work because those people exist.

What’s a little far out though is the client who is not doing the math at all.

I had a meeting this morning that went exactly like this. The client needed an iOS app, Android app, web app, backend REST API and custom web-based admin panel. The client understood it would take anywhere from 3-6 months to do this in the best case, and even seemed to understand I would need to bring in help to do it in that timeline. Then told me with a straight face that the budget was $3000 firm.

So I did the math. Two developers working around the clock to get this done in an extremely optimistic 3 months works out to about $3 an hour each. Considering the 3 months was, as I said, very optimistic in this case, I would project closer to $1-2 an hour.

And they were specifically looking for a developer based in Southern California. Well, I’m in Southern California, and I can tell you if I made $3000 a week I’d be living a dumpster next week. I don’t even see how I could make it past week three if I was earning $1000 or less a month. Its expensive here. And even if it wasn’t, that is a ludicrous amount of money for anyone, anywhere.

But not only that, you are hiring highly skilled, highly trained, in demand people to do this work. Or, at least you should be if you care at all about what you’re building! But you are expecting them to leap at a job that pays about 5% of the local minimum wage.

I don’t even know what else to say.

Looking for a technical co-founder!

What you probably meant to say: I have no money

Let’s finish with the big deal breaker. This, too is a post all on its own, but its a big one and it comes up all the time.

There are a few other common and related statements I can throw in with this one:

• I want someone who is in this for the long haul
• I’m looking for someone who really believes in our idea, and isn’t just looking for a paycheck
• This is the next billion dollar idea!

Really quickly, let me break down a few things.

First, this is our job. We aren’t a charity giving up our time to help you strike it big. We have bills to pay and loved ones to take care of, and reality just doesn’t permit us to take on projects pro-bono. It just doesn’t, and I don’t know anyone who knows what they are doing who ever jumps on board on an equity basis at the early stage.

Again, there are a lot of reasons for this, and hopefully I’ll get to post about them one day. But for now, the simple version is: you just aren’t there yet.

The time to ask a developer to come on for equity is after you have a product, and you have some traction in the market. NOT when all you have is an idea, or some awful prototype you had build overseas that barely works.

While I’m absolutely sure that you are super excited about your idea, you’re asking a developer to be just as excited as you are, when in reality all you have is an idea. And to someone who listens to ideas all day long, an idea alone just isn’t that exciting. What is exciting is when that idea starts to work. When users start to react positively to the released product. When the non-technical team starts to put together those critical deals and build those incredibly important relationships that take that humble idea and turn it into a viable company. That’s when you should ask someone to come on for equity, not before!

And typically, equity offers come along with pay, not in lieu of pay. In fact, its your primary job as an entrepreneur at the earliest stages to raise money for your company! I know its hard — believe me, I’ve been through it many times. But it’s a crucial part of launching a company, and it’s critical to building a competent team that will be able to build the technical foundation for your company.

But until I write that big post explaining this in more depth, let’s just keep it short and simple: you get what you pay for.

One of the advantages (if you can call it that) of being in this industry as long as I have is that I’ve been through multiple economic disasters, technological paradigm shifts, and — it’s not all bad! — economic and technological boom periods.

So I figured I might as well throw out a few predictions as to how I feel technology, and more specifically the app world, is going to change after this is over. What kinds of apps will consumers want, what kinds of apps will entrepreneurs build?

Keep in mind this is just my opinion based on my own experience, and should not be taken as anything but that. That being said, lets start with the big question: what sorts of apps are going to be big after the coronovirus scare of 2020?

Remote working apps

One of the initial download spikes I saw after people here in California were asked to stay home was Zoom. Not just Zoom though — Skype, Discord, and other communication apps suddenly became much more useful as we all retreated home.

What’s interesting to me in times like this is how people start bending existing technologies to their needs. For example, you may have an app that was used primarily for virtual face-to-face conferencing for businesses now being used to host virtual get togethers for people under lockdown.

That’s an interesting opportunity for entrepreneurs to start to see gaps in the technological landcape. The market will tell you really quickly that there is a need for a specific app tailored for that particular use case. Its possible that platforms like Zoom will step up and fill the gap themselves. But sometimes those apps can’t or simply won’t, and that’s a huge opportunity to get in and fill an existing need.

One of the biggest struggles I see with many app startups is that they created something very cool, very interesting, but not very in demand. That means when they launch, they not only need to find users and let them know they exist, they also have to explain to them why they should care. It’s always an easier road when the market is already waiting with baited breath for your product.

Social networks

I know the social networking landscape is already pretty crowded, but I predict we’ll see a few more innovative apps in this space in the next year or two.

I have been watching a few giant gaps of my own for the last few years (and hopefully one day I’ll have time to set about filling them!) but more are sure to pop up as the battle against Coronavirus goes on.

Many of us, for example, are cut off from our parents right now. I am, and I could be for months. Are our existing social networks adequate for facilitating communication with our older family members? Are they easy to use, do they offer the tools we need to help our parents and grandparents out in a time of crisis?

One immediate issue that came to my attention when all of this happened was that my parents need someone to bring supplies to them — if it is unsafe for them to venture outside, they will need someone to make sure they have food and toilet paper and soap and every other necessity. And for the very old among us there is the added concern of whether or not they are able to stay aware of their own needs.

At home entertainment

I don’t personally feel like gaming or streaming entertainment is ripe for any newcomers as a result of Coronavirus. After all just about every type of media is available to stream these days — even comics! But I do think their importance in our lives is going to become much more evident now.

I have, for a while, started to think about ways that entrepreneurs can help consumers manage the growing number of subscriptions that are required now — either by way of bundles, or technologies to manage payment options, sharing, etc.

Information broadcasting

And finally, the big one: we need better technology to get information to the community.

And we don’t just need better channels — we need better noise reduction. The amount of utter nonsense I’ve seen on Twitter, Facebook and Instagram this week is depressing. I have had several friends and family members ask me if I had heard some tidbit working its way through the grapevine which was either comically false or, worse, potentially dangerous.

We need better ways to spread valid information than our current UGC platforms, which we have seen over the past few years are highly susceptible to misinformation and confusion.

I would personally love to see someone finally come in and solve not only the UGC news and information dissemination problem, but also make sure we have adequate tools in our pockets to receive and process updates from agencies like the CDC or WHO to keep us informed in the event of health crises or natural disasters.

That’s all for now. I’m sure over the next couple weeks or possibly months we’ll see more gaps in the market as a result of this.

One thing I would like to remind everyone is that the iPhone boom happened essentially right after the financial crisis of 2007-2008. And the just came out a ridiculously long period of growth since then. So stay positive, stay creative, and most importantly stay safe!

Most of the images in Glider PRO's resources are in PICT format.

The PICT format is basically a bunch of serialized QuickDraw opcodes and can contain a combination of both image and vector data.

The first goal is to get all of the known resources to parse.  The good news is that none of the resources in the Glider PRO application resources or any of the houses contain vector data, so it's 100% bitmaps.  The bad news is that the bitmaps have quite a bit of variation in their internal structure, and sometimes they don't match the display format.

Several images contain multiple images spliced together within the image data, and at least one image is 16-bit color even though the rest of the images are indexed color.  One is 4-bit indexed color instead of 8-bit.  Many of them are 1-bit, and the bit scheme for 1-bit images is also inverted compared to the usual expectations (i.e. 1 is black, 0 is white).

Adding to these complications, while it looks like all of the images are using the standard system palette, there's no guarantee that they will - It's actually even possible to make a PICT image that combines multiple images with different color palettes, because the palette is defined per picture op, not per image file.

There's also a fun quirk where the PICT image frame doesn't necessarily have 0,0 as the top-left corner.

I think the best solution to this will simply be to change the display type to 32-bit and unpack PICT images to a single raster bitmap on load.  The game appears to use QuickDraw abstractions for all of its draw operations, so while it presumes that the color depth should be 8-bit, I don't think there's anything that will prevent GlidePort from using 32-bit instead.

In the meantime, I've been able to convert all of the resources in the open source release to PNG format as a test, so it should be possible to now adapt that to a runtime PICT loader.

Recently found out that Classic Mac game Glider PRO's source code was released, so I'm starting a project called GlidePort to bring it to Windows, ideally in as faithful of a reproduction as possible and using the original data files.  Some additions like gamepad support may come at a later time if this stays on track.

When adding ETC support to Convection Texture Tools, I decided to try adapting the cluster fit algorithm used for desktop formats to ETC.

Cluster fit works by sorting the pixels into an order based on a color axis, and then repeatedly evaluating each possible combination of counts of the number of pixels assigned to each index.  It does so by taking the pixels and applying a least-squares fit to produce the endpoint line.

For ETC, this is is simplified in a few ways: The axis is always 1,1,1, so the step of picking a good axis is unnecessary.  There is only one base color and the offsets are determined by the table index, so the clustering step would only solve the base color.

Assuming that you know what the offsets for each pixel are, the least squares fit amounts to simply subtracting the offset from each of the input pixels and averaging the result.

For a 4x2 block, there are 165 possible cluster configurations, but it turns out that some of those are redundant, given certain assumptions.  The base color is derived from the formula ((color1-offset1)+(color2-offset2)+...)/8, but since the adds are commutative, that's identical to ((color1+color2+...)-(offset1+offset2+...))/8

The first half of that is the total of the colors, which is constant.  The second is the total of the offsets.

Fortunately, not all of the possible combinations produce unique offsets.  Some of them cancel out, since adding 1 to or subtracting 1 from the count of the offsets that are negatives of each other produces no change.  In an example case, the count tuples (5,0,1,2) and (3,2,3,0) are the same, since 5*-L + 0*-S + 1*S + 2*L = 3*-L + 2*-S + 3*S + 0*L.

For most of the tables, this results in only 81 possible offset combinations.  For the first table, the large value is divisible by the small value, causing even more cancellations, and only 57 possible offset combinations.

Finally, most of the base colors produced by the offset combinations are not unique after quantization: Differential mode only has 5-bit color resolution, and differential mode only has 4-bit resolution, so after quantization, many of the results get mapped to the same color.  Deduplicating them is also inexpensive: If the offsets are checked in ascending order, then once the candidate color progresses past the threshold where the result could map to a specific quantized color, it will never cross back below that threshold, so deduplication only needs to inspect the last appended quantized color.

Together, these reduce the candidate set of base colors to a fairly small number, creating a very optimal search space at low cost.

There are a few circumstances where these assumptions don't hold:

One is when the clamping behavior comes into effect, particularly when a pixel channel's value is near 0 or 255.  In that case, this algorithm can't account for the fact that changing the value of the base color would have no effect on some of the offset colors.

One is when the pixels are not of equal importance, such as when using weight-by-alpha, which makes the offset additions non-commutative, but that only invalidates the cancellation part of the algorithm.  The color total can be pre-weighted, and the rest of the algorithm would have to rely on working more like cluster fit: Sort the colors along the 1,1,1 line and determine the weights for the pixels in that order, generate all 165 cluster combinations, and compute the weight totals for each one.  Sort them into ascending order, and then the rest of the algorithm should work.

One is when dealing with differential mode constraints, since not all base color pairs are legal.  There are some cases where a base color pair that is just barely illegal could be made legal by nudging the colors closer together, but in practice, this is rare: Usually, there is already a very similar individual mode color pair, or another differential mode pair that is only slightly worse.

In CVTT, I deal with differential mode by evaluating all of the possibilities and picking the best legal pair.  There's a shortcut case when the best base color for both blocks produces a legal differential mode pair, but this is admittedly a bit less than optimal: It picks the first evaluation in the case of a tie when searching for the best, but since blocks are evaluated starting with the largest combined negative offset, it's a bit more likely to pick colors far away from the base than colors close to the base, even though colors closer to the average tend to produce smaller offsets and are more likely to be legal, so this could be improved by making the tie-breaking function prefer smaller offsets.

In practice though, the differential mode search is not where most of the computation time is spent: Evaluating the actual base colors is.

As with the rest of CVTT's codecs, brute force is still key: The codec is designed to use 8-wide SSE2 16-bit math ops wherever possible to processing 8 blocks at once, but this creates a number of challenges since sorting and list creation are not amenable to vectorization.  I solve this by careful insertion of scalar ops, and the entire differential mode part is scalar as well.  Fortunately, as stated, the parts that have to be scalar are not major contributors to the encoding time.


You can grab the stand-alone CVTT encoding kernels here: https://github.com/elasota/ConvectionKernels

Introduction

In the last post we were left with some tests that exercised some very basic functionality of the Deck class. In this post, we will continue to add unit tests and write production code to make those tests pass, until we get a class which is able to produce a randomised deck of 52 cards.

Test Refactoring

You can, and should, refactor your tests where appropriate. For instance, on the last test in the last post, we only asserted that we could get all the cards for a particular suit. What about the other three? With most modern test frameworks, that is very easy.

[InlineData(Suit.Clubs)]
[InlineData(Suit.Diamonds)]
[InlineData(Suit.Hearts)]
[InlineData(Suit.Spades)]
public void Should_BeAbleToSelectSuitOfCardsFromDeck(Suit suit)
{
    var deck = new Deck();

    var cards = deck.Where(x => x.Suit == suit);

    cards.Should().HaveCount(13);
}

More Cards

We are going to want actual cards with values to work with. And for the next test, we can literally copy and past the previous test to use as a starter.

[Theory]
[InlineData(Suit.Clubs)]
[InlineData(Suit.Diamonds)]
[InlineData(Suit.Hearts)]
[InlineData(Suit.Spades)]
public void Should_BuildAllCardsInDeck(Suit suit)
{
    var deck = new Deck();

    var cards = deck.Where(x => x.Suit == suit);

    cards.Should().Contain(new List<Card> 
    { 
        new Card(suit, "A"), new Card(suit, "2"), new Card(suit, "3"), new Card(suit, "4"),
        new Card(suit, "5"), new Card(suit, "6"), new Card(suit, "7"), new Card(suit, "8"),
        new Card(suit, "9"), new Card(suit, "10"), new Card(suit, "J"), new Card(suit, "Q"),
        new Card(suit, "K")
    });
}

Now that I’ve written this, when I compare it to the previous one, it’s testing the exact same thing, in slightly more detail. So we can delete the previous test, it’s just noise.

The test is currently failing because it can’t compile, due to there not being a constructor which takes a string. Lets fix that.

public struct Card
{
    private Suit _suit;
    private string _value;

    public Card(Suit suit, string value)
    {
        _suit = suit;
        _value = value;
    }

    public Suit Suit { get { return _suit; } }
    public string Value { get { return _value; } }

    public override string ToString()
    {
        return $"{Suit}";
    }
}

There are a couple of changes to this class. Firstly, I added the constructor, and private variables which hold the two defining variables, with properties with only public getters. I changed it from being a class to being a struct, and it’s now an immutable value type, which makes sense. In a deck of cards, there can, for example, only be one Ace of Spades.

These changes mean that are tests don’t work, as the Deck class is now broken, because the code which builds set of thirteen cards for a given suit is broken - it now doesn’t understand the Card constructor, or the fact that the .Suit property is now read-only.

Here is my first attempt at fixing the code, which I don’t currently think is all that bad:

private string _ranks = "A23456789XJQK";

private List<Card> BuildSuit(Suit suit)
{
    var cards = new List<Card>(_suitSize);

    for (var i = 1; i <= _suitSize; i++)
    {
        var rank = _ranks[i-1].ToString();
        var card = new Card(suit, rank);
        cards.Add(card);
    }

    return cards;
}

This now builds us four suites of thirteen cards. I realised as I was writing the production code that handling “10” as a value would be straightforward, so I opted for the simpler (and common) approach of using “X” to represent “10”. The test pass four times, once for each suit. This is probably unnecessary, but it protects us in future from inadvertantly adding any code which may affect the way that cards are generated for a particular suit.

Every day I’m (randomly) shuffling

It’s occured to me as I write this that the Deck class is funtionally complete, as it produces a deck of 52 cards when it is instantiated. You will however recall that we want a randomly shuffled deck of cards. If we consider, and invoke the Single Responsibility Principal, then we should add a Dealer class; we are modeling a real world event and a pack of cards cannot shuffle itself, that’s what the dealer does.

Conclusion

In this post I’ve completed the walk through of developing a class to create a deck of 52 cards using some basic TDD techniques. I realised adding the ability to shuffle the pack to the Deck class would be a violation of SRP, as the Deck class should not be concerned or have any knowledge about how it is shuffled. In the next post I will discuss how we can implement a Dealer class, and illustrate some techniques swapping the randomisation algorithim around.

Convection Texture Tools is now roughly equal quality-wise with NVTT at compressing BC7 textures despite being about 140 times faster, making it one of the fastest and highest-quality BC7 compressors.

How this was accomplished turned out to be simpler than expected.  Recall that Squish became the gold standard of S3TC compressors by implementing a "cluster fit" algorithm that ordered all of the input colors on a line and tried every possible grouping of them to least-squares fit them.

Unfortunately, using this technique isn't practical in BC7 because the number of orderings has rather extreme scaling characteristics.  While 2-bit indices have a few hundred possible orderings, 4-bit indices have millions, most BC7 mode indices are 3 bits, and some have 4.

With that option gone, most BC7 compressors until now have tried to solve endpoints using various types of endpoint perturbation, which tends to require a lot of iterations.

Convection just uses 2 rounds of K-means clustering and a much simpler technique based on a guess about why Squish's cluster fit algorithm is actually useful: It can create endpoint mappings that don't use some of the terminal ends of the endpoint line, causing the endpoint to be extrapolated out, possibly to a point that loses less accuracy to quantization.

Convection just tries cutting off 1 index at each end, then 1 index at both ends.  That turned out to be enough to place it near the top of the quality benchmarks.

Now I just need to add color weighting and alpha weighting and it'll be time to move on to other formats.

Introduction

In the previous post in this series, we had finished up with a very basic unit test, which didn’t really test much, which we had ran using dotnet xunit in a console, and saw some lovely output.

We’ll continue to write some more unit tests to try and understand what kind of API we need in a class (or classes) which can help us satisfy the first rule of our Freecell engine implementation. As a reminder, our first rule is: There is one standard deck of cards, shuffled.

I’m trying to write both the code and the blog posts as I go along, so I have no idea what the final code will look like when I’ve finished. This means I’ll probably make mistakes and make some poor design decisions, but the whole point of TDD is that you can get a feel for that as you go along, because the tests will tell you.

Don’t try to TDD without some sort of plan

Whilst we obey the 3 Laws of TDD, that doesn’t mean that we can’t or shouldn’t doodle a design and some notes on a whiteboard or a notebook about the way our API could look. I always find that having some idea of where you want to go and what you want to achieve aids the TDD process, because then the unit tests should kick in and you’ll get a feel for whether things are going well or the conceptual design you had is not working.

With that in mind, we know that we will want to define a Card object, and that there are going to be four suits of cards, so that gives us a hint that we’ll need an enum to define them. Unless we want to play the same game of Freecell over and over again, then we’ll need to randomly generate the cards in the deck. We also know that we will need to iterate over the deck when it comes to building the Cascades, but the Deck should not be concerned with that.

With that in mind, we can start writing some more tests.

To a functioning Deck class

First things first, I think that I really like the idea of having the Deck class enumerable, so I’ll start with testing that.

[Fact]
public void Should_BeAbleToEnumerateCards()
{
    foreach (var card in new Deck())
    {
    }
}

This is enough to make the test fail, because the Deck class doesn’t yet have a public definition for GetEnumerator, but it gives us a feel for how the class is going to be used. To make the test pass, we can do the simplest thing to make the compiler happy, and give the Deck class a GetEnumerator definition.

public IEnumerator<object> GetEnumerator()
{
    return Enumerable.Empty<object>().GetEnumerator();
}

I’m using the generic type of object in the method, because I haven’t yet decided on what that type is going to be, because to do so would violate the three rules of TDD, and it hasn’t yet been necessary.

Now that we can enumerate the Deck class, we can start making things a little more interesting. Given that it is a deck of cards, it should be reasonable to expect that we could expect to be able to select a suit of cards from the deck and get a collection which has 13 cards in it. Remember, we only need to write as much of this next test as is sufficient to get the test to fail.

[Fact]
public void Should_BeAbleToSelectSuitOfCardsFromDeck()
{
    var deck = new Deck();

    var hearts = deck.Where();
}

It turns out we can’t even get to the point in the test of asserting something because we get a compiler failure. The compiler can’t find a method or extension method for Where. But, the previous test where we enumerate the Deck in a foreach passes. Well, we only wrote as much code to make that test pass as we needed to, and that only involved adding the GetEnumerator method to the class. We need to write more code to get this current test to pass, such that we can keep the previous test passing too.

This is easy to do by implementing IEnumerable<> on the Deck class:

public class Deck : IEnumerable<object>
{
    public IEnumerator<object> GetEnumerator()
    {
        foreach (var card in _cards)
        {
            yield return card;
        }
    }

    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

I’ve cut some of the other code out of the class so that you can see just the detail of the implementation. The second explicitly implemented IEnumerable.GetEnumerator is there because IEnumerable<> inherits from it, so it must be implemented, but as you can see, we can just fastward to the genericly implemented method. With that done, we can now add using System.Linq; to the Deck class so that we can use the Where method.

var deck = new Deck();

var hearts = deck.Where(x => x.Suit == Suit.Hearts);

This is where the implementation is going to start getting a little more complicated that the actual tests. Obviously in order to make the test pass, we need to add an actual Card class and give it a property which can use to select the correct suit of cards.

public enum Suit
{
    Clubs,
    Diamonds,
    Hearts,
    Spades
}

public class Card
{
    public Suit Suit { get; set; }
}

After writing this, we can then change the enumerable implementation in the Deck class to public class Deck : IEnumerable<Deck>, and the test will now compile. Now we can actually assert the intent of the test:

[Fact]
public void Should_BeAbleToSelectSuitOfCardsFromDeck()
{
    var deck = new Deck();

    var hearts = deck.Select(x => x.Suit == Suit.Hearts);

    hearts.Should().HaveCount(13);
}

Conclusion

In this post, I talked through several iterations of the TDD loop, based on the 3 Rules of TDD, in some detail. An interesting discussion that always rears its head at this point is: Do you need to follow the 3 rules so excruciatingly religously? I don’t really know the answer to that. Certainly I always had it in my head that I would need a Card class, and that would necessitate a Suit enum, as these are pretty obvious things when thinking about the concept of a class which models a deck of cards. Could I have taken a short cut, written everything and then wrote the tests to test the implementation (as it stands)? Probably, for something so trivial.

In the next post, I will write some more tests to continue building the Deck class.

Lock’n’Roll, a Pidgin plugin for Windows designed to set an away status message when the PC is locked, has received its first update in three and a half years!

Daniel Laberge has forked the project and released a version 1.2 update which allows you to specify which status should be set when the workstation locks. Get it while it’s awesome (always)!

How do you generate the tangent vectors, which represent which way the texture axes on a textured triangle, are facing?

Hitting up Google tends to produce articles like this one, or maybe even that exact one. I've seen others linked too, the basic formulae tend to be the same. Have you looked at what you're pasting into your code though? Have you noticed that you're using the T coordinates to calculate the S vector, and vice versa? Well, you can look at the underlying math, and you'll find that it's because that's what happens when you assume the normal, S vector, and T vectors form an orthonormal matrix and attempt to invert it, in a sense you're not really using the S and T vectors but rather vectors perpendicular to them.

But that's fine, right? I mean, this is an orthogonal matrix, and they are perpendicular to each other, right? Well, does your texture project on to the triangle with the texture axes at right angles to each other, like a grid?


... Not always? Well, you might have a problem then!

So, what's the real answer?

Well, what do we know? First, translating the vertex positions will not affect the axial directions. Second, scrolling the texture will not affect the axial directions.

So, for triangle (A,B,C), with coordinates (x,y,z,t), we can create a new triangle (LA,LB,LC) and the directions will be the same:

We also know that both axis directions are on the same plane as the points, so to resolve that, we can to convert this into a local coordinate system and force one axis to zero.



Now we need triangle (Origin, PLB, PLC) in this local coordinate space. We know PLB[y] is zero since LB was used as the X axis.


Now we can solve this. Remember that PLB[y] is zero, so...


Do this for both axes and you have your correct texture axis vectors, regardless of the texture projection. You can then multiply the results by your tangent-space normalmap, normalize the result, and have a proper world-space surface normal.

As always, the source code spoilers:

terVec3 lb = ti->points[1] - ti->points[0];
terVec3 lc = ti->points[2] - ti->points[0];
terVec2 lbt = ti->texCoords[1] - ti->texCoords[0];
terVec2 lct = ti->texCoords[2] - ti->texCoords[0];

// Generate local space for the triangle plane
terVec3 localX = lb.Normalize2();
terVec3 localZ = lb.Cross(lc).Normalize2();
terVec3 localY = localX.Cross(localZ).Normalize2();

// Determine X/Y vectors in local space
float plbx = lb.DotProduct(localX);
terVec2 plc = terVec2(lc.DotProduct(localX), lc.DotProduct(localY));

terVec2 tsvS, tsvT;

tsvS[0] = lbt[0] / plbx;
tsvS[1] = (lct[0] - tsvS[0]*plc[0]) / plc[1];
tsvT[0] = lbt[1] / plbx;
tsvT[1] = (lct[1] - tsvT[0]*plc[0]) / plc[1];

ti->svec = (localX*tsvS[0] + localY*tsvS[1]).Normalize2();
ti->tvec = (localX*tsvT[0] + localY*tsvT[1]).Normalize2();

There's an additional special case to be aware of: Mirroring.

Mirroring across an edge can cause wild changes in a vector's direction, possibly even degenerating it. There isn't a clear-cut solution to these, but you can work around the problem by snapping the vector to the normal, effectively cancelling it out on the mirroring edge.

Personally, I check the angle between the two vectors, and if they're more than 90 degrees apart, I cancel them, otherwise I merge them.

Valve's self-shadowing radiosity normal maps concept can be used with spherical harmonics in approximately the same way: Integrate a sphere based on how much light will affect a sample if incoming from numerous sample direction, accounting for collision with other samples due to elevation.

You can store this as three DXT1 textures, though you can improve quality by packing channels with similar spatial coherence. Coefficients 0, 2, and 6 in particular tend to pack well, since they're all dominated primarily by directions aimed perpendicular to the texture.

I use the following packing:
Texture 1: Coefs 0, 2, 6
Texture 2: Coefs 1, 4, 5
Texture 3: Coefs 3, 7, 8

You can reference an early post on this blog for code on how to rotate a SH vector by a matrix, in turn allowing you to get it into texture space. Once you've done that, simply multiply each SH coefficient from the self-shadowing map by the SH coefficients created from your light source (also covered on the previous post) and add together.