Project: Brixton Design Trail responsive website

I was asked to make a basic responsive website to support the Brixton Design Trail, part of the London Design Festival this year, after I made their site last year.

I used Google Sheets for a very simple back end (based on my previous work with Google Scripts as a free CMS), and the rest of the site is largely a Google Maps implementation with a fair bit of customisation, using the JS API v3.


Brixton Pounds: even the paper money is amazing!

Project: Brixton Pound technology audit

After having helped the Brixton Design Week team with a project, my details were passed along to the Brixton Pound when they were looking for some help.

The Brixton Pound, if you didn’t know, is a community currency started as part of Transition Town Brixton in order to investigate ways to make local towns more self-sufficient, in this case by encouraging local funds to stay local.

They provide a wealth of services, including pay-by-text as well as cash, and are soon to be offering a lottery and fund for local causes. Their technological setup is also wide-ranging and impressive, given the budgets they work with; I know this because I dove into the details of each piece and talked to their providers in order to create a report on the state of it all.

I delivered the report and talked about my recommendations; they seemed very happy!

My first PyPI package

I guest authored another post over at the YunoJuno Tech blog, about my recent integration of our Django codebase witht he BaseCRM API, and the creation of my first PyPI package. Here’s the syndicated post…

Base CRM is a Customer Relationship Management system that integrates with Google Apps. They recently released the second version of their API and we’ve started integrating our codebase with it, in order to automate the creation of Contacts and Deals.

We created a PyPI package, django_basecrm that anyone can use to integrate their Django projects with Base. You will need to add ‘django_basecrm’ to your INSTALLED_APPS, and add the following to your settings.py:


It’s mainly a lightweight wrapper around the requests library, that adds the required headers and generates API endpoints, but it also provides Serializers (inspired by the Django REST Framework), that provide a powerful and simple way to maintain a mapping between your ORM objects and the Base CRM objects.

It also provides some helper methods to strip out metadata, retrieve particular types of data, etc.

An example call to retrieve all notes related to a particular contact would be as simple as:

import django_basecrm
# 545 is the base_id for your contact</span>
django_basecrm.get_notes('contact', 545)
# [{'id': 655423, 'content': 'A note against the contact', ...}, ...]

We run all our integration methods (mainly one-way updates to Base) from RQ; this app helps make atomic, pickled calls easy.

It is meant as a lightweight, easy-to-use drop-in connector for your Django project, but it’s also not complete – please feel free to fork the GitHub repo and contribute to the project.

dotfiles and configuration

I’ve long used config scripts to customise my terminal environment, providing aliases to commands I use often. Over the last couple of years I’ve included git completion, git flow tools and a customised PS1. A while ago I decided to version my setup on GitHub and subsequently found out about GitHub dotfiles: a portal introducing a lot of people and approaches to doing that very thing.

Having cribbed from a lot of them I formalised my own setup over the Easter weekend and here’s my setup. Feel free to fork it and use it yourself if you’re interested: https://github.com/marcelkornblum/dotfiles

It includes:

  • Git completion for bash (from here)
  • Customised command prompt including git status (from here)
  • Git flow toolset (from here)
  • Completion for ssh-config host aliases
  • Many aliases for navigating the CLI and git with git flow
  • Extra utility aliases for controlling the OS, mainly for OS X (from here)
  • All aliases implementing autocompletion of arguments (from here)
  • A script for symlinking the repo files into your home directory (from here)
  • My personal Sublime Text settings and solarized colour theme


Getting the most out of ElasticSearch and Django

This post was written for the YunoJuno tech blog, where I’m currently freelancing, and was first posted there.

ElasticSearch is a powerful, easy-to-install, easy-to-use, low config, clustered search engine based on Lucene, and is a popular choice for all sorts of applications.

In the Django world, Haystack seems to be the de-facto way to implement search outside the ORM – and it certainly seems the fastest to implement.

There are a lot of good reasons to use Haystack: it’s very easy to get up and running, it’s super-simple to configure based on your existing models, it comes with a great SearchQuerySet that lets you chain filters and use Django ORM-like syntax to create queries that lazy load, and it’s portable between ElasticSearch, Solr, Xapian and Whoosh.

Although I’d implemented Lucene some years ago and knew a little about search in general, I’d never used either ES or Haystack before I was asked to improve the existing YunoJuno search functionality. One of the ideas I wanted to implement was a tag search, where users can type into a text box and matching tags will appear as search autocomplete, and can then be used to filter the results.

To do this I realised I’d need an autocomplete-type nGram search that’s superfast for the input typeahead, and also some fancy tokenising to handle multiple-word tags and a synonym feature to keep the total number of tags low.

It turns out that, in exchange for making a powerful search very easy to add to your Django project, Haystack exacts a price, by putting a fairly low ceiling on what you can achieve with the search. In order to preserve portability, some advanced ES features such as aliases and warming aren’t supported in Haystack.

We can live with this, but I was a little disappointed by a couple of the other design decisions Haystack makes (presumably for the same reasons). It can only run queries against a single field on the document, which you implement as a kind of munged data catchall using Django’s template language to combine all the model fields you want indexed. It also puts all your models into a single ‘type’ under the index of your choice, so if you want to split them out you have to split them into new indices. Both of which mean you have an artificially large number of artificially large indices.

Worse still, Haystack comes with no out-of-the-box way to change any of the analysers or tokenisers you use, just allowing you to pick between snowball (a great English-focused text analyser) and nGrams, which weren’t even working for me as expected.

I then found this excellent blog post (which resulted in the elastickstack Django app), and followed it to extend Haystack and put my own analyser together for my special tag-search requirements.

That was when the real fun began. Nothing seemed to work. Sure it was my inexperience, I played around with analysers and tokenisers, validating them all through the ElasticSearch analyse and mapping APIs over the course of a couple of days.

The truth was slowly dawning on me: although Haystack was accepting my custom analysers, and ES was being told about them when I indexed my files, the analysers simply weren’t being applied to the fields as I was indexing them; only on the queries I was running. Which means that, at best, my custom analysis was being ignored and, at worst that it was being applied only to one side of the equation, giving me bogus – or no – results.

Which isn’t to say that elasticstack doesn’t work – if all you want is synonyms it should do the job really well, but don’t expect anything really complex from Haystack.

So now we’re starting the process of communicating directly with elasticsearch-py, the official ES Python library. We’re writing a lightweight wrapper that emphasises ES configuration and style over Django, in order to allow full access to all ES functionality in exchange for portability.

Luckily our codebase only touches ES in a couple of places…


Projects for OstModern: a healthcare startup and a huge Football video site

I’ve just finished freelancing at Ostmodern, a digital agency specialising in digital products and servicing a range of clients including the Beeb, BFI and News Corp.

I was ostensibly* brought in to work on a prototype healthcare application for use in hospitals, coded in Django and AngularJS — my favourite combination at the moment. I worked on the Django side and created a REST API instead of an HTML templated interface.

Some of the more interesting functionality I implemented was a full audit history (every change tracked and pre-change objects restorable), online/offline login with the same password (via the API, and including JWT access tokens), and a secured socket connection for realtime multi-user updates. For the latter two parts I implemented both the Django back end and the Angular front end.

I hope to write about some of these in more detail if I get permission from Ost to do so.

In addition to that project, I helped work on BallBall, a multilingual, minute-by-minuted updated website making football video highlights form the European leagues available to fans, mainly based in Southeast Asia. I implemented a little functionality in Django for the site, but the main focus of my work was Sysadmin.

BallBall front page in Vietnamese

BallBall front page in Vietnamese

The project has 5 online environments, and each environment comprises 6 systems running distinct codebases; not to mention static file hosting, managed database services and caching — both using Memcached and Akamai. All of this was hosted on Amazon, and each functional unit comprised at least two servers, managed by a Scaling Group, inside an individual Security Group, and accessed via its own Elastic Load Balancer. All of this was managed via a series of CloudFormation templates.

My sysadmin work on this project was mainly in optimisation (including implementing NewRelic tracking to analyse what could best be optimised) and migrating their environments to new AWS accounts — no trivial task.

Assuming I haven’t burned my bridges with my Christmas party performance (not to be published online!) I hope to work there again some time…

* see what I did there? ;-)

Project: A secret Agile startup project

In late 2013 I helped a well known investor collaborate with a retail service company in creating a really innovative new digital product. It’s not launched so I can’t mention any names, but it was codenamed ReTech and initially branded as Rapt :)

I assembled a small but brilliant freelance team: myself as part back end developer, part EP and part product owner; an iOS developer (Nick Wood); a brand/interface/UX designer (Paul Macgregor) and a scrum master (Smita Das). Never having worked in a fully agile project, it was a good opportunity to learn and our scrum master was excellent in steering us into really owning the process.

We spent a total of about 6 weeks working in 2-week sprints, and by the end we had an end-to-end working iOS application that demonstrated the core functionality of the startup, albeit with a limited data set. It looked great and worked pretty solidly — and perhaps most impressively we hadn’t worked past 6pm once…

Unfortunately the investor and the company with the domain-specific knowledge had a fallout, and so our 10-week project never culminated in the popup retail experience we’d been aiming for, and I’m still not really sure of the status of the project.

It’s a huge shame as the potential was enormous and I think we were all really proud of what we’d achieved and learned; we still send each other sad emails with links to startups who are implementing parts of the functionality we’d brainstormed together (although obviously their implementation aren’t as good as our would have been!)

If you get the chance to work in a pure agile way, where there can be full transparency between the product team and the people paying, I’d really recommend it — it’s amazingly productive and once you learn what you can and can’t commit to, you find yourself able to make promises you can keep to every time.