David Burke

Divergent thoughts on AI

Fri, 23 Jan 2026 12:00:00 +0000

GlitchTip, the open source error monitoring tool I started, features no capital-fueled AI language on it’s home page. Our LLM wrapper isn’t changing your world, it doesn’t exist. Compare that to a competitor like Sentry’s seer, where the marketing features some fleshy, pained “Allied Mastercomputer” or AM watching your errors.

Was that intentional to look so pained that it might get attention? I suppose it worked.

Use the chat widget on the site, and I’ll respond slowly when I get around it it. You might think I’m a Luddite. I do have a Minimal Phone now, that I like. It’s less distracting. I can’t watch video on my phone. Reading slow is fine, but scrolling media is pained. It’s much less enticing for my kids to want to grab. It turns internet ads with motion into eink flashing nausea. Which I perversely enjoy hating. It’s ugly, but I’m glad the ad is rendered incomprehensible.

In my day job (Not GlitchTip) I actually do a good bit of ai work. I mean both using ai and implementing it (Building wrappers). Not the fancy building of GPT systems, I understand only at a elementary level, maybe as much as I understand quantum physics.

In spite of my technology reservations, I actually do like the idea of more tooling to automate coding. I’ve always felt drawn more to programming as a way to build something, while not really enjoying writing algorithms very much. I find some coding frustrating. Stop upgrading your packages, I hate your breaking changes.

Last summer I built this unfinished media visualizer https://gitlab.com/bufke/disk-player using only ai tools. I just wanted to see if I could. I lack the attention to finish it, but it is kind of cool. I found the experiment successful, I could use my existing knowledge to let a LLM build something for me, only reviewing and guiding it.

In the fall, I built django-vcache and django-vtasks to replace Celery for me with more asyncio friendly and lighter code. I used a combination of Gemini CLI and Claude+Zed. Working on open source, I can’t say I built some amazing $80k project with $200 of credits. All I could say is that I built a $0 money pit with $200 of credits and now I cannot afford coffee. Gemini pro $20/month pricing is reasonable and I paid for Zed pro mainly to support an open source project that I like. You should do this too, maybe for Zed or GlitchTip. My total budget is then capped $50/month, that’s less coffee but not no coffee for me.

I use a Gemini Pro web conversation as a architect/planner and CLI/Zed as the doer. The web conversation is a playground for ideas and longer historical context. LLM context is kept lower as it’s not writing much code. I might have a good idea while on the subway and quickly update the conversation. The Minimal phone does has a nice physical keyboard for quick writing. To execute, I can have the architect LLM write a prompt for CLI/Zed. I can let CLI run while cooking. Then give a deep dive when merging, when I’m not distracted. I ended up writing many unit tests myself to ensure correctness, the models tend to write tests for the sake of testing and not proving anything. They are too full of mocks and misunderstanding. The better tests ground changes, avoiding one feature breaking another. Or the totally bonkers things LLM’s do sometimes. Early results for these projects are very promising. They perform well and are good for my half-distracted lifestyle. I don’t suspect I would consider a “make my own Celery” type project without them.

I could imagine using voice controls for these tools. Maybe I talk to gemini while holding my baby. I get work done at all hours of the day, supervising it. Let it work alone at night. During dedicated work time, I review deeply and commit. Sounds like a hell AM might dream up. Total distraction. Total focus on a feeling of work. Unsure if it’s useful, effective, dangerous.

I think I’ll avoid voice controls for now.

Here’s a better use case. I maintain all of these test projects. https://gitlab.com/glitchtip/error-factories. They ensure the sentry-sdk works with GlitchTip. They auto update themselves with Renovate. They send events to a staging GlitchTip server every day. A LLM manages their maintenance when tests fail. This is a highly effective use of ai in my opinion. As a large language model, I—just kidding. Nothing in error factories is production code. And it’s boring code. In this one case, I think the ai usage is firmly more time with family and coffee.

openapi-typescript with Angular Resource

Wed, 01 Jan 2025 12:00:00 +0000

Angular 19 introduces Resource, an async signal, typically used for web requests. I find it reminiscent of TanStack Query but more minimalist and signal based. While RxResource works great with Angular HTTPClient, the regular “Resource” works with good old fetch. Which means we can combine it with openapi-typescript and openapi-fetch. The result will let us quickly grab typed API resources and process them with Angular Signals, no rxjs needed.

Follow openapi-fetch’s documentation to install and generate your openapi types. I called mine src/app/api/api-schema.d.ts. Then create a file such as src/app/api/api.ts

import createClient from "openapi-fetch";
import type { paths } from "./api-schema";

export const client = createClient<paths>();

Let’s imagine fetching a Foo api in a Service.

export class FooService {
  foosResource = resource({
    loader: async () => {
      const { data, error } = await client.GET("/api/foo/");
      // Consider error handling here
      return data
  })
  foos = computed(() => this.foosResource.value() || [])
}

Our foos will be typed according to the openapi spec. We can also refer to the types directly:

import type { components } from "src/app/api/api-schema";
type Foo = components["schemas"]["FooSchema"];

With this approach, we can easily get typed API data into Angular and it looks more like any other JS framework code.

Monitor network endpoints with Python asyncio and aiohttp

Fri, 06 Aug 2021 12:00:00 +0000

My motivation - I wanted to make a network monitoring service in Python. Python isn’t known for it’s async ability, but with asyncio it’s possible. I wanted to include it in a larger Django app, GlitchTip. Keeping everything as a monolithic code base makes it easier to maintain and deploy. Go and Node handle concurrent IO a little more naturally but don’t have any web framework even close to as feature complete as Django.

How asyncio works compared to JavaScript

I’m used to synchronous Python and asynchronous JavaScript. asyncio is a little strange at first. It’s far more verbose than just stringing along a few JS promises. Let’s compare this example of JS and Python.

fetch('http://example.com/example.json')
  .then(response => response.json())
  .then(data => console.log(data));

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://example.com/example.json') as response:
            html = await response.json()
            print(html)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

There’s more boilerplate in Python. aiohttp has three chained async calls while fetching JSON in JS requires just two chained promises. Let’s break these differences down a bit

An async call to GET/POST/etc the resource. At this time, we don’t have the body of the request. fetch vs sessions.get are about the same here.
An async call to get the body contents (and perhaps process them in some manner such as converting a JSON payload to a object or dictionary). If we only need say the status code, there is no need to spend time doing this. Both have async text() and json() functions that also work similarly.
aiohttp has a ClientSession context manager that closes the connection. The only async IO occurs when closing the connection. It’s possible to reuse a session for some performance benefit. This is often useful in Python as our async code block will often live nested in synchronous code. Fetch does not have this (as far as I’m aware at the time of this writing).
get_event_loop and run_until_complete allow us to run async functions from a synchronous code function. Python is synchronous by default, so this is necessary. When running Django or Celery or a python script, everything is blocking until explicitly run async. JavaScript on the lets you run async code with 0 boilerplate.

One other thing to note is that both Python and JavaScript are single threaded. While you can “do multiple things” while waiting for IO, you cannot use multiple CPU cores without starting multiple processes, for example by running uwsgi workers. Thus in Python it’s called asyncio.

Source: docs.aiohttp.org

Network Monitoring with aiohttp

Network monitoring can easily start as a couple line script or be a very complex, massive service depending on scale. I won’t claim that my method is the best, mega-scale method ever, but I think it’s quite sufficient for small to medium scale projects. Let’s start with requirements

Must handle 1 million network checks per minute
Must run at least every 30 seconds (smaller scale this could probably go much shorter)
Must only run Python and get embedded into a Django code base
Must not require anything other than a Celery compatible service broker and Django compatible database

And a few non-functional requirements that I believe will help scale

Must scale to run from many servers (Celery workers)
Must batch database writes as efficiently as possible to avoid bottlenecks

Overview of architecture

A Celery beat scheduler will run a “dispatch_checks” task every 30 seconds. Dispatch checks will determine which “monitors” need checked based on their set interval frequency and last check. It will then batch these in groups and dispatch further parallel celery tasks called “perform_checks” to actually perform the network check. The perform_checks task will then fetch additional monitor data in one query and asynchronously check each network asset. Once done, it will save to the database using standard Django ORM. By batching inserts, we should be able to improve scalability. It also means we don’t need a massive number of celery tasks, which would be unnecessary overhead. In real life, we may only have a few celery works for the “small or medium scale” so it would waste resources to dispatch 1 million celery tasks. If we batch inserts by 1000 and really have our max target of 1 million monitors, then we would want 1000 celery workers. Another variable is the timeout for each check. Making it lower, means our workers get done faster instead of waiting for the slowest request.

See the full code on GlitchTip’s GitLab.

Celery Tasks

@shared_task
def dispatch_checks():
    now = timezone.now()
    latest_check = Subquery(
        MonitorCheck.objects.filter(monitor_id=OuterRef("id"))
        .order_by("-start_check")
        .values("start_check")[:1]
    )
    monitor_ids = (
        Monitor.objects.filter(organization__is_accepting_events=True)
        .annotate(
            last_min_check=ExpressionWrapper(
                now - F("interval"), output_field=DateTimeField()
            ),
            latest_check=latest_check,
        )
        .filter(latest_check__lte=F("last_min_check"))
        .values_list("id", flat=True)
    )
    batch_size = 1000
    batch_ids = []
    for i, monitor_id in enumerate(monitor_ids.iterator(), 1):
        batch_ids.append(monitor_id)
        if i % batch_size == 0:
            perform_checks.delay(batch_ids, now)
            batch_ids = []
    if len(batch_ids) > 0:
        perform_checks.delay(batch_ids, now)

@shared_task
def perform_checks(monitor_ids: List[int], now=None):
    if now is None:
        now = timezone.now()
    # Convert queryset to raw list[dict] for asyncio operations
    monitors = list(Monitor.objects.filter(pk__in=monitor_ids).values())
    loop = asyncio.get_event_loop()
    results = loop.run_until_complete(fetch_all(monitors, loop))
    MonitorCheck.objects.bulk_create(
        [
            MonitorCheck(
                monitor_id=result["id"],
                is_up=result["is_up"],
                start_check=now,
                reason=result.get("reason", None),
                response_time=result.get("response_time", None),
            )
            for result in results
        ]
    )

The fancy Django ORM subquery is to ensure we are able to determine which monitors need checked while being as performant as possible. While some may prefer complex queries in raw SQL, for some reason I prefer ORM and I’m impressed to see how many use cases Django can cover these days. Anything to avoid writing lots of join table SQL 🤣️

aiohttp code

async def process_response(monitor, response):
    if response.status == monitor["expected_status"]:
        if monitor["expected_body"]:
            if monitor["expected_body"] in await response.text():
                monitor["is_up"] = True
            else:
                monitor["reason"] = MonitorCheckReason.BODY
        else:
            monitor["is_up"] = True
    else:
        monitor["reason"] = MonitorCheckReason.STATUS

async def fetch(session, monitor):
    url = monitor["url"]
    monitor["is_up"] = False
    start = time.monotonic()
    try:
        if monitor["monitor_type"] == MonitorType.PING:
            async with session.head(url, timeout=PING_AIOHTTP_TIMEOUT):
                monitor["is_up"] = True
        elif monitor["monitor_type"] == MonitorType.GET:
            async with session.get(url, timeout=DEFAULT_AIOHTTP_TIMEOUT) as response:
                await process_response(monitor, response)
        elif monitor["monitor_type"] == MonitorType.POST:
            async with session.post(url, timeout=DEFAULT_AIOHTTP_TIMEOUT) as response:
                await process_response(monitor, response)
        monitor["response_time"] = timedelta(seconds=time.monotonic() - start)
    except SSLError:
        monitor["reason"] = MonitorCheckReason.SSL
    except asyncio.TimeoutError:
        monitor["reason"] = MonitorCheckReason.TIMEOUT
    except OSError:
        monitor["reason"] = MonitorCheckReason.UNKNOWN
    return monitor

async def fetch_all(monitors, loop):
    async with aiohttp.ClientSession(loop=loop) as session:
        results = await asyncio.gather(
            *[fetch(session, monitor) for monitor in monitors], return_exceptions=True
        )
        return results

That’s it. Ignoring my models and plenty of Django boilerplate, we have the core of a reasonably performant uptime monitoring system in about 120 lines of code. GlitchTip is MIT licensed so feel free to use as you see fit. I also run a small SaaS service at app.glitchtip.com which helps fund development.

On testing

I greatly prefer testing in Python over JavaScript. I’m pretty sure this 15 line integration test would require a pretty complex Jasmine boilerplate and run about infinite times slower in CI. I will gladly put up with some asyncio boilerplate to avoid testing anything in JavaScript. In my experience, there are Python test driven development fans and there are JS developers who intended to write tests.

    @aioresponses()
    def test_monitor_checks_integration(self, mocked):
        test_url = "https://example.com"
        mocked.get(test_url, status=200)
        with freeze_time("2020-01-01"):
            mon = baker.make(Monitor, url=test_url, monitor_type=MonitorType.GET)
        self.assertEqual(mon.checks.count(), 1)

        mocked.get(test_url, status=200)
        with freeze_time("2020-01-01"):
            dispatch_checks()
        self.assertEqual(mon.checks.count(), 1)

        with freeze_time("2020-01-02"):
            dispatch_checks()
        self.assertEqual(mon.checks.count(), 2)

There’s a lot going on in little code. I use aioresponses to mock network requests. Django baker to quickly generate DB test data. freezegun to simulate time changes. assertEqual from Django’s TestClient. And not seen, CELERY_ALWAYS_EAGER in settings.py to force celery to run synchronously for convenience. I didn’t write any async tests code yet I have a pretty decent test covering the core functionality from having monitors in the DB to ensuring they were checked properly.

JS equivalent

describe("test uptime", function() {
  it("should work", function() {
    // TODO
  });
});

Joking aside, I find it quite hard to find a good Node based task runner like Celery, ORM, and test framework that really work well together. There are many little niceties like running Celery in always eager mode that make testing a joy in Python. Let me know in a comment if you disagree and have any JavaScript based solutions you like.

Deploy Saleor E-commerce with Kubernetes and Helm

Fri, 04 Jun 2021 12:00:00 +0000

Saleor is a headless, Django based e-commerce framework. This post will show how to deploy Saleor using Django Helm Chart. It will focus on deploying the Django Backend. The dashboard is a static HTML site and is left out. The front-end is something you should build yourself.

First you need a Docker image. You can build this yourself or use https://hub.docker.com/r/mirumee/saleor/. I suggest building it yourself so that it’s possible to add plugins and set the specific version you’d like. Here’s a snippet for Gitlab CI as a starting point. This can be run on a fork of Saleor which already contains a Dockerfile. I like to tag my image with both the git ref name and short sha for reference later on.

build:
  stage: build 
  image: docker:20
  services:
    - docker:20-dind
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

Next, create a values.yml file to power Django Helm Chart. Here’s an example:

image:
  repository: your-docker-image
  tag: latest

env:
  normal:
    ALLOWED_CLIENT_HOSTS: localhost,127.0.0.1,your-frontend-url
    ALLOWED_HOSTS: "*"
    ENABLE_SSL: "True"
    DEFAULT_EMAIL_FROM: noreply@example.com
    AWS_ACCESS_KEY_ID: XXXXXXXXXX
    AWS_MEDIA_BUCKET_NAME: bucket-name
    AWS_DEFAULT_ACL: public-read
    SENTRY_DSN: https://something@app.glitchtip.com/project-id
  secret:
    DATABASE_URL: your-postgres-connection-string
    SECRET_KEY: your-secret-key
    AWS_SECRET_ACCESS_KEY: XXXXXXXXX
    EMAIL_URL: email-connection-string
    CELERY_BROKER_URL: your-redis-connection-string

web:
  replicaCount: 2
  port: 8000
  args: ["gunicorn", "--bind", ":8000", "--workers", "4", "--worker-class", "uvicorn.workers.UvicornWorker", "saleor.asgi:application"]
  autoscaling:
    enabled: false
  livenessProbe:
    failureThreshold: 5
    initialDelaySeconds: 5
    timeoutSeconds: 2
    path: "/graphql/"
  readinessProbe:
    failureThreshold: 10
    initialDelaySeconds: 5
    timeoutSeconds: 2
    path: "/graphql/"
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts:
      - host: your-host
        paths:
          - path: /
            pathType: ImplementationSpecific

worker:
  enabled: true
  args:
    - celery
    - -A
    - saleor
    - --app=saleor.celeryconf:app
    - worker
    - --loglevel=info

redis:
  architecture: standalone
  auth:
    password: redis-password
  master:
    persistence:
      enabled: false

Let’s break this apart, as there are many options here.

I choose to enable Kubernetes managed Redis but not Postgres. I don’t trust running stateful services like Postgres in Kubernetes but you could set postgres.enabled=true.
Many Saleor settings are managed via environment variables. Documentation for them exists here. In the example, I configure AWS S3 and set my DATABASE_URL to a managed Postgres instance like RDS.
The web and worker args (which maps to Docker’s command) are set to the specific run commands for Saleor which uses celery and gunicorn. This is also the place to edit configuration options, such as number of gunicorn workers.
The example contains a ingress. Don’t forget to add a ingress controller like ingress-nginx to your cluster. If you don’t need a internet accessible URL, remove it.

Next add the chart repo (or fork it).

helm repo add django https://burke-software.gitlab.io/django-helm-chart/
helm install your-app django/django -f values.yml

The chart should output instructions on accessing the new site. Make sure to review logs to ensure Celery is running as well. Now you have a Saleor backend running on Kubernetes. If running in production, make sure to review affinity values, Saleor configuration, resource limits, and add tls to the ingress.

Deploy Django with Helm

Mon, 15 Feb 2021 12:00:00 +0000

This is a follow up post to Deploy Django with helm to Kubernetes which focused on the CI pipeline. This post will highlight a generic Django Helm Chart I made for GlitchTip, an open source alternative to Sentry issue tracking.

Note that Kubernetes and Helm are very complex systems that a simple Django app does not need. GlitchTip, for example, can run on DigitalOcean App Platform.

Django + Celery + Redis Helm Chart

Django Helm Chart can act as a starter Helm Chart for any Django project. While you could use the chart directly, I recommend forking it instead so that you can include your own customizations.

Let’s start with some language definitions and assumptions

A django app is made of several “components” - often the web server, Celery worker queue, Celery Beat, DB migration job, and databases including Postgres and Redis.
In addition, we need Helm specific infrastructure such as a Service, Load Balancer, and/or Ingress.
Environment variables will be stored as Helm values. This is a method that can easily be done with Helm alone. You may want a more complex solution for secrets such as Hashicorp Vault.
I highly recommend installing Helm Diff. I wouldn’t suggest using Helm without it. One small typo in Helm can delete your entire app.
This chart supports Helm based Postgres and Redis, however I strongly recommend using a managed service for any persistent data.

Forking the Helm Chart

Start by forking the Django Helm Chart. If you’d like to keep your own secrets in a values.yaml file - ensure this repo is private.

Edit Chart.yaml and set the name to your project. You may want to edit appVersion, but I find it a little burdensome to increment this. Unless you plan to host a helm chart for many users, it’s not necessary.
Carefully review values.yaml
Notice the tree structure reflects our web and worker components.
For new kubernetes users, leave autoscaling off and set replicas manually.
Defaults for resources are a guess. Adjust them based on your app’s real world usage.
The default web affinity attempts to implement the business rule “Do not run all app pods on the same node”
- This will only work if you have at least one more node than you need. If one node becomes unhealthy, Kubernetes may be forced to do “bad” things like schedule all pods on one Node. I recommend at least three Nodes for even a smaller production deployment.
If your app needs to live on the internet, enable the Ingress. You’ll also need one Load Balancer for your Kubernetes cluster. Technically you can make just a Load Balancer and ignore the Ingress, but I don’t recommend this. Most service providers charge for Load Balancers and it may get expensive running them for staging/testing instances of your app.
Consider if you need to enable Postgres and Redis. Personally, I always use a managed, non-kubernetes, Postgres server like RDS or DigitalOcean’s managed databases. If my Redis instance doesn’t need to persist, for example it’s used only for cache or a task queue, I enable it in Kubernetes.
Any key-value under environmentVariables will map to environment variables that can be read manually or by django-environ.

Health checks assume a endpoint at /_health/. I suggest adding this to your Django admin, although you could change it in templates/web/deployment.yaml. It isn’t desired to run, for example, your homepage if it involves database lookups. Don’t test your entire app infrastructure, only test that Django is responding.

def health(request):
    return HttpResponse("ok", content_type="text/plain")

path("_health/", health),

Now install your app. Something like

helm install your-chart -f your-values.yaml

What’s next?

Consider contributing any improvements. Here’s a few ideas:

Better budget, affinity, autoscaling, and resource defaults.
Ideas for secrets management.
~~Enable/Disable features like Celery and more.~~ Done
Is it worth publishing this chart instead of only forking it?

If you found this chart useful, consider donating time or money to GlitchTip. Donations let me spend more time on various open source projects. GlitchTip is an open source error tracking platform that is compatible with the open source Sentry SDKs. It’s a free software alternative to proprietary platforms like DataDog and Sentry. We also offer a paid hosted service.

Wagtail Single Page App Integration news

Sun, 28 Jun 2020 12:00:00 +0000

wagtail-spa-integration is a Python package I started with coworkers at thelab.

Version 2.0

Wagtail SPA Integration 2.0 is released! The release is actually maintenance only, but now requires Wagtail 1.8+ thus making it a potentially breaking change.

Coming soon version 3.0 with wagtail-headless-preview

A major feature of Wagtail SPA Integration is preview support. Torchbox (the creators of Wagtail) developed their own solution called wagtail-headless-preview. We’ll be migrating to this and it will be a significant breaking change. Aligning with Torchbox’s implementation will reduce our maintenance burden and allow for a generally more “normal” experience. It will also remove a feature/quirk in Wagtail SPA that generated links that could be used for up to one day without authentication.

NextJS Support coming soon!

We have a proof of concept for NextJS support. Preview/contribute it here. This package utilizes NextJS’s dynamic routing and dynamic components making it possible base NextJS pages on Wagtail page types. One difference from the Angular Wagtail implementation is that all communication is handled via the NextJS Node server instead of direct Wagtail REST API calls. This results in simpler code, but slightly worse performance.

Angular Wagtail will also receive a minor update to work with 3.0.

Shameless advertising

Looking for an open source error monitoring solution for your Django, Angular, or NextJS apps? Try out glitchtip.com. GlitchTip is compatible with Sentry’s open source SDK but unlike Sentry is 100% open source. We offer paid SaaS hosting as an option. Your support gives us time to continue work on these various open source projects!

Deploy Django with helm to Kubernetes

Fri, 24 Jan 2020 12:00:00 +0000

This guide attempts to document how to deploy a Django application with Kubernetes while using continuous integration It assumes basic knowledge of Docker and running Kubernetes and will instead focus on using helm with CI. Goals:

Must be entirely automated and deploy on git pushes
Must run database migrations once and only once per deploy
Must revert deployment if migrations fail
Must allow easy management of secrets via environment variables

My need for this is to deploy GlitchTip staging builds automatically. GlitchTip is an open source error tracking platform that is compatible with Sentry. You can find the finished helm chart and gitlab CI script here. I’m using DigitalOcean and Gitlab CI but this guide will generally work for any Kubernetes provider or Docker based CI tool.

Building Docker

This guide assumes you have basic familiarity with running Django in Docker. If not, consider a local build first using docker compose. I prefer using compose for local development because it’s very simple and easy to install. Build a Docker image and tag it with the git short hash. This will allow us to specify an exact image build later on and will ensure code builds are tied to specific helm deployments. If we used “latest” instead, we may end up accidentally upgrading the Docker image. Using Gitlab CI the script may look like this:

docker build -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME} -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}

This uses -t to tag the new build with the Gitlab CI environment variables to specify the docker registry and tags. It uses “ref name” which is the tag or branch name. This will result in a tag such as “1.3” or branch such as “dev”. This tagging is intended for users who may just want a specific named version or branch. The second -t tags it with the git short hash. This tag will be referenced later on by helm. Before moving on - make sure you can now docker pull your CI built image and run it. Make sure to set the Dockerfile CMD to use gunicorn, uwsgi, or another production ready server. We’ll deal with Django migrations later using Helm.

Setting up Kubernetes

This guide assumes you know how to set up Kubernetes. I chose DigitalOcean because they provide managed Kubernetes, it’s reasonably priced, and I like supporting smaller companies. DigitalOcean limits choice which makes it easier to use for average looking projects. It doesn’t offer the level of customization and services AWS does. If you decide to use DigitalOcean and want to help offset the cost of my open source projects, considering using this affiliate link. My goals for a hosting platform are:

Easy to use
Able to be managed via terraform
Managed Postgres
Managed Kubernetes
Able to restrict network access for internal services such as the database

Whichever platform you are using, make sure you have a database and it’s connection string and can authenticate to Kubernetes. If you are new to Kubernetes, I suggest deploying any docker image manually (without tooling like helm) to get a little more familiar. Technically, you could also run your database in Kubernetes and Helm. However I prefer managed stateful services and will not cover running the database in Kubernetes in this guide.

Deploy to Kubernetes with Helm in Gitlab CI

Update Feb 2021 The GlitchTip Helm Chart is now a generic Django + Celery Helm chart. Read more here.

Now that you have a Docker image and Kubernetes infrastructure, it’s time to write a Helm chart and deploy your image automatically from CI. A Helm chart allows you to write Kubernetes yaml configuration templates using variables. The chart I use for GlitchTip should be a good starting point for most Django apps. At a minimum, read the getting started section for Helm’s documentation. The GlitchTip chart includes one web server deployment and a Django migration job with helm lifecycle hook. You may need to set up an additional deployment if you use a worker such as Celery. The steps are the same, just override the Docker RUN command to start celery instead of your web server. Run the initial helm install locally. This is necessary to set initial variables such as the database connection that don’t need to be set in CI each deploy. Reference each value to override in your chart’s values.yaml. If following my GlitchTip example, that will be databaseURL and secretKey. databaseURL is the Database connection string. I use django-environ to set this. You could also define a separate databaseUser, databasePassword, etc if you like making more work for yourself. The key to make this work is to ensure one way or another the database credentials and other configuration get passed in as environment variables that are read by your settings.py file. Ensure your CI server has built at least one docker image. Place your chart files in the same git repo as your Django project in a directory “chart” Run helm install your-app-name ./chart --set databaseURL=string --set secretKey=random_string --set image.tag=git_short_hash If you use GlitchTip’s chart - it will not set up a load balancer but it will show output that explains how to connect locally just to test that everything is working. The Django migration job should also run and migrate your database. This guide will not include the many options you have for load balancing. I choose to use DigitalOcean’s load balancer and having it directly select the deployment’s pods. Note that in Kubernetes, a service of type Load Balancer may run a service providers load balancer and allow you to configure it through kubernetes config yaml. This will vary between providers. Here’s a sample load balancer that can be applied with kubectl –namespace your-namespace apply -f load-balancer.yaml note that it uses selector to directly send traffic from the load balancer to pods. It also contains DigitalOcean specific annotations, which is why I can’t document a universal way to do this.

apiVersion: v1
kind: Service
metadata:
  name: your-app-staging
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: long-id
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: /
    service.beta.kubernetes.io/do-loadbalancer-protocol: http
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: your-app-staging
    app.kubernetes.io/name: your-app

At this point you should have a fully working Django application.

Updating in CI using Helm

Now set up CI to upgrade your app on git pushes (or other criteria). While technically optional, I suggest making separate namespaces and service accounts for each environment. Unfortunately this process can feel obtuse at first and I felt was the hardest part of this project. For each environment, we need the following:

Service Account
Role Binding
Secret with CA Cert and token

For a rough analogy the service account is a “user” but for a bot instead of a human. A role binding defines the permissions that something (say a service account) has. The role binding should have the “edit” permission for the namespace. The secret is like the “password” but is actually a certificate and token. Read more from Kubernetes documentation. Once this is set up locally, test it out. For example, use the new service account auth in your ~/.kube/config and run kubectrl get pods –namespace=your-namespace. The CA cert and token from your recently created secret should be what is in your kube’s config file. I found no sane manner of editing multiple kubernetes configurations and resorted to manually editing the config file.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: big-long-base64 
    server: https://stuff.k8s.ondigitalocean.com
  name: some-name

...

users:
- name: default
  user:
    token big-long-token-from-secret

Notice I used certifate-authority-data so I could reference the cert inline as base64. Next save the entire config file in Gitlab CI under settings, CI, Variables. There’s actually a lot happening in this little bit of configuration. File type in Gitlab CI will cause the value to save into a random tmp file. The key “KUBECONFIG” will be set to the file location. KUBECONFIG is also the environment variable helm will use to locate the kube config file. Protected will allow this only to be available to protected git branches/tags. If we didn’t set protected, someone with only limited git access could make their own branch that runs echo $KUBECONFIG and view the very confidential data! If set up right, you should now be able to run helm with the authentication that just works. Finally add the deploy step to Gitlab CI’s yaml file.

deploy-staging:
  stage: deploy
  image: lwolf/helm-kubectl-docker
  script:
    - helm upgrade your-app-staging ./chart --set image.tag=${CI_COMMIT_SHORT_SHA} --reuse-values
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - master

stage ensures it runs after the docker build. For image, use lwolf/helm-kubectl-docker which has helm already installed. The script is amazingly just one line thanks to the previous authentication and Gitlab CI variable tricks done. It runs helm upgrade with –set image.tag to the new git short hash and –reuse-values allows it to set this new value without overriding previous values. Using helm this way allows you to keep database secrets outside of Gitlab. Do note however that anyone with helm access can read these values. If you need a more robust system then you’ll need something like Vault. But even without Vault, we can isolate basic git users who can create branches and admin users who have access to helm and the master branch. The environment section is optional and let’s Gitlab track deploys. “only” causes the script to only run on the master branch. Alternatively it could be set for other branches or tags. If you need to change an environment variable, run the same upgrade command locally and –set as many variables as needed. Keep the –reuse-values. Because the databaseURL value is marked as required, helm will error instead of erase previous values should you forget the important –reuse-values.

Conclusion

I like Kubernetes for it’s reliability but I find it creates a large amount of decision fatigue. I hope this guide provides one way to do things that I find works. If you have a better way - let me know by commenting here or even open an issue on GlitchTip. I’m sure there’s room for improvement. For example, I’d rather generate the django secret key automatically but helm’s random function doesn’t let you store it persistently. I don’t like Kube’s, maddening at times, complexity. Kubernetes is almost never a solution by itself and requires additional tools to make it work for even very basic use cases. I found Openshift to handle a lot of common use cases like deploy hooks and user/service management much easier. Openshift “routes” are also defined in standard yaml config rather than forcing the user to deal with propreitary annotations on a Load Balancer. However, I’m leery of using Openshift Online considering it hasn’t been updated to version 4 and no roadmap seems to exist. It’s also quite a bit more expensive (not that it’s bad to pay more for good open source software). Finally if you need error tracking for your Django app and prefer open source solutions - give GlitchTip a try. Contributors are preferred, but you can also support the project by using the DigitalOcean affiliate link or donating. Burke Software also offers paid consulting services for open source software hosting and software development.

Django Rest Framework ModelViewSets with natural key lookup

Wed, 04 Dec 2019 12:00:00 +0000

DRF ModelViewSet can easily support detail views by slug via the lookup_value attribute. But what if you had compound keys (aka natural keys)? For example a url structure like

/api/computers/<organization-slug>/<computer-slug>/

A computer slug may only be unique per organization. That means different organizations may have computers with the same slug. But no computer may have the same slug in one organization. By using both slugs, we can look up a specific computer. We can use the lookup_value_regex attribute for this.

class ComputerViewSet(viewsets.ModelViewSet):
    queryset = Computer.objects.all()
    serializer_class = ComputerSerializer
    lookup_value_regex = r"(?P<org_slug>[^/.]+)/(?P<slug>[-\w]+)"

    def get_object(self):
        queryset = self.filter_queryset(self.get_queryset())
        obj = get_object_or_404(
            queryset,
            slug=self.kwargs["slug"],
            organization__slug=self.kwargs["org_slug"],
        )

        # May raise a permission denied
        self.check_object_permissions(self.request, obj)

        return obj

This works with drf-nested-routers. For example, we could add a nested /hard_drives viewset. The url values are in self.kwargs.

class HardDriveViewSet(viewsets.ModelViewSet):
    queryset = HardDrive.objects.all()
    serializer_class = HardDriveSerializer

    def get_queryset(self):
        return (
            super()
            .get_queryset()
            .filter(
                computer__slug=self.kwargs["slug"],
                computer__organization__slug=self.kwargs["org_slug"],
            )
        )

Angular Wagtail 1.0 and getting started

Sun, 20 Oct 2019 12:00:00 +0000

Angular Wagtail and Wagtail Single Page App Integration are officially 1.0 and stable. It’s time for a more complete getting started guide. Let’s build a new app together. Our goal will be to make a multi-site enabled Wagtail CMS with a separate Angular front-end. When done, we’ll be set up for features such as

Map Angular components to Wagtail page types to build any website tree we want from the CMS
All the typical wagtail features we expect, drafts, redirects, etc. No compromises.
SEO best practices including server side rendering with Angular Universal, canonical urls, and meta tags.
Correct status codes for redirects and 404 not found
Lazy loaded modules
High performance, cache friendly, small JS bundle size (In my experience 100kb - 270kb gzipped for large scale apps)
Absolutely no jank. None. When a page loads we get the full page. Nothing “pops in” unless we want it to. No needless dom redraws that you may see with some single page apps.
Scalable - add more sites, add translations, keep just one “headless” Wagtail instance to manage it all.

Start with a Wagtail project that has wagtail-spa-integration added. For demonstration purposes, I will use the sandbox project in wagtail-spa-integration with Docker. Feel free to use your own Wagtail app instead.

git clone https://gitlab.com/thelabnyc/wagtail-spa-integration.git
Install docker and docker-compose
docker-compose up
docker-compose run –rm web ./manage.py migrate
docker-compose run –rm web ./manage.py createsuperuser
Go to http://localhost:8000/admin/ and log in.

Set up Wagtail Sites. We will make 1 root page and multiple homepages representing each site. You may want to rename the “Welcome to Wagtail” default page to “API Root” just for clarity. Then create two child pages of any type to act as homepages. If you don’t need multi-site support, just add one instead. Wagtail requires the Sites app to be enabled even if only one site is present. The API Root will still be important later on for distinguishing the Django API server from the front-end Node server. Next head over to Settings, Sites. Keep the default Site attached to the API Root page. Add another Site for each homepage. If you intend to have two websites, you should have three Wagtail Sites (API Root, Site A, Site B). Each hostname + port combination must be unique. For local development, it doesn’t matter much. For production you may have something like api.example.com, www.example.com, and intranet.example.com. Next let’s set up the Wagtail API. This is already done for you in the sandbox project but when integrating your own app, you may follow the docs here. Then follow Wagtail SPA Integration docs to set up the extended Pages API. Make sure to set WAGTAILAPI_BASE_URL to localhost:8000 if you want to run the site locally on port 8000. Here’s an example of setting up routes. api.py

from wagtail.api.v2.router import WagtailAPIRouter
from wagtail_spa_integration.views import SPAExtendedPagesAPIEndpoint

api_router = WagtailAPIRouter('wagtailapi')
api_router.register_endpoint('pages', SPAExtendedPagesAPIEndpoint)

urls.py

from django.conf.urls import include, url
from wagtail.core import urls as wagtail_urls
from wagtail_spa_integration.views import RedirectViewSet
from rest_framework.routers import DefaultRouter
from .api import api_router

router = DefaultRouter()
router.register(r'redirects', RedirectViewSet, basename='redirects')

urlpatterns = [
    url(r'^api/v2/', api_router.urls),

Test this out by going to localhost:8000/api/ and localhost:8000/api/v2/pages/ If you’d like to enable the Wagtail draft feature - set PREVIEW_DRAFT_CODE in settings.py to any random string. Note this feature will generate special one time, expiring links that do not require authentication to view drafts. This is great for sharing and the codes expire in one day. However if your drafts contain more sensitive data, you may want to add authentication to the Pages API. This is out of scope for Wagtail SPA Integration, but consider using any standard Django Rest Framework authentication such as tokens or JWT. You may want to check if a draft code is present and only check authentication then, so that the normal pages API is public.

Angular Front-end

Now let’s add a new Angular app (or modify an existing one).

ng new angular-wagtail-demo
cd angular-wagtail-demo
npm i angular-wagtail –save

In app.module.ts add

import { WagtailModule } from 'angular-wagtail';
WagtailModule.forRoot({
  pageTypes: [],
  wagtailSiteDomain: 'http://localhost:8000',
  wagtailSiteId: 2,
}),

In app-routing.module.ts add

import { CMSLoaderGuard, CMSLoaderComponent } from 'angular-wagtail';
const routes: Routes = [{ path: '**', component: CMSLoaderComponent, canActivate: [CMSLoaderGuard] }];

This is the minimal configuration. Notice the domain and site ID are set explicitly. This is not required as Wagtail can determine the appropriate site based on domain. However, it’s much easier to set it explicitly so that we don’t have to set up multiple hostnames for local development. Next let’s add a lazy loaded homepage module. Making even the homepage lazy loaded will get us in the habit of making everything a lazy loaded module which improves performance for users who might not visit the homepage first (Such as an ad or search result to a specific page).

ng generate module home --routing
ng generate component home

In app.module.ts add a “page type”. An Angular Wagtail page type is a link between Wagtail Page Types and Angular components. If we make a Wagtail page type “cms_django_app.HomePage” we can link it to an Angular component “HomeComponent”. Page types closely follow the Angular Router, so any router features like resolvers will just work with exactly the same syntax. In fact, angular-wagtail uses the Angular router behind the scenes.

pageTypes: [
  {
    type: 'sandbox.BarPage',
    loadChildren: () => import('./home/home.module').then(m => m.HomeModule)
  },
]

This maps sandbox.BarPage from the wagtail-spa-integration sandbox to the HomeModule. “sandbox” is the django app name while BarPage is the model name. This is the same syntax as seen in the Wagtail Pages API and many other places in django to refer to a model (app_label.model). “loadChildren” is the same syntax as the Angular Router. I could set the component instead of loadChildren if I didn’t want lazy loading. Next edit home/home-routing.module.ts. Since our homepage has only one component, set it to always load that component home-routing.module.ts with WagtailModule.forFeature

const routes: Routes = [{
  path: '',
  component: HomeComponent
}];

To test everything is working run ”npm start” and go to localhost:4200. We now have a home page! However, it doesn’t contain any actual CMS data. Let’s start by adding the page’s title. We could get this data on ngOnInit however this would load the data asynchronously after the route is loaded. This can lead to jank because any static content would load immediately on route completion but async data would pop in later. To fix this, we’ll use a resolver. Resolvers can get async data before the route completes. Edit home-routing.module.ts

import { GetPageDataResolverService } from 'angular-wagtail';
const routes: Routes = [{
  path: '',
  component: HomeComponent,
  resolve: { cmsData: GetPageDataResolverService }
}];

This resolver service will assign an Observable with the CMS data for use in the component. We can use it in our component: home.component.ts

import { ActivatedRoute } from '@angular/router';
import { Observable } from 'rxjs';
import { map } from 'rxjs/operators';
import { IWagtailPageDetail } from 'angular-wagtail';

interface IHomeDetails extends IWagtailPageDetail {
  extra_field: string;
}

@Component({
  selector: 'app-home',
  template: `
    <p>Home Works!</p>
    <p>{{ (cmsData$ | async).title }}</p>
  `,
})
export class HomeComponent implements OnInit {
  public cmsData$: Observable<IHomeDetails>;

  constructor(private route: ActivatedRoute) { }

  ngOnInit() {
    this.cmsData$ = this.route.data.pipe(map(dat => dat.cmsData));
  }
}

Going top to bottom, notice how IHomeDetails extends IWagtailPageDetail and adds page specific fields. This should mimic the fields you added when defining the Wagtail Page model. Default Wagtail fields like “title” are included in IWagtailPageDetail. The template references the variable cmsData$ which is an Observable with all page data as given by the Wagtail Pages API detail view. ngOnInit is where we set this variable, using route.data. Notice how cmsData is available from the resolver service. When you load the page, you should notice “Home Works!” and the title you set in the CMS load at the same time. Nothing “pops in” which can look bad. At this point, you have learned the basics of using Angular Wagtail!

Adding a lazy loaded module with multiple routes

Sometimes it’s preferable to have one module with multiple components. For example, there may be 5 components and two of them represent route-able pages. Keeping them grouped in a module increases code readability and makes sense to lazy load the components together. To enable this, make use of WagtailModule.forFeature. Let’s try making a “FooModule” example to demonstrate.

ng generate module foo
ng generate component foo

Edit foo.module.ts

import { NgModule, ComponentFactoryResolver } from '@angular/core';
import { CommonModule } from '@angular/common';
import { WagtailModule, CoalescingComponentFactoryResolver } from 'angular-wagtail';
import { FooComponent } from './foo.component';

@NgModule({
  declarations: [FooComponent],
  entryComponents: [FooComponent],
  imports: [
    CommonModule,
    WagtailModule.forFeature([
      {
        type: 'sandbox.FooPage',
        component: FooComponent
      }
    ])
  ]
})

export class FooModule {
  constructor(
    coalescingResolver: CoalescingComponentFactoryResolver,
    localResolver: ComponentFactoryResolver
  ) {
    coalescingResolver.registerResolver(localResolver);
  }
}

FooComponent is added to both declarations and entryComponents as it’s not directly added to the router. WagtailModule.forFeature will link the wagtail page type with a component. You can also add a resolver here if needed. Lastly, the constructor adds coalescingResolver. This enabled dynamic component routing between modules and likely won’t be needed in Angular 9 with Ivy and future versions of Angular Wagtail. Add as many types of page types as desired.

Angular Universal

Angular Universal can generate pages in Node (or prerender them). This is nice for SEO and general performance. The effect is to generate a minimalist static view of the page that runs without JS enabled. Later the JS bundle is loaded and any dynamic content (shopping carts, user account info) is loaded in. Because the server side rendered static page is always the same for all users, it works great with a CDN. I’ve found even complex pages will be around 50kb of data for the first dom paint. Installation is easy.

ng add @nguniversal/express-engine --clientProject angular.io-example

Compile with npm run build:ssrand serve with npm run serve:ssr. Angular Wagtail supports a few environment variables we can set in node. Setting the API server domain and site per deployment is possible:

export WAGTAIL_SITE_ID=2
export CMS_DOMAIN=http://localhost:8000

Confirm it’s working by disabling JavaScript in your browser. Angular Wagtail provides a few extras for Angular Universal when run in Node (serve:ssr). You can return 404, 302, and 301 status codes by editing server.ts as documented. You can also add the wagtail generated sitemap. Not directly related to Wagtail, but I found helmet and adding a robots.txt pretty helpful too. Angular Univeral just runs express, so anything possible in express is possible in Angular Universal.

Bells and whistles - not found and more SEO

For a real site, consider adding a 404 not found component, setting page meta tags and canonical url. Edit the WagtailModule.forRoot configuration to modify this however you wish. If you followed the server set up from above then Wagtail redirects and drafts should “just work”. Any time Angular Wagtail can’t match a url path to component, it will query the Wagtail SPA Integration redirects API and will redirect if it finds one. If not, Angular Wagtail will show the 404 not found component to the user. You can find the full angular wagtail demo source on gitlab.

Talk: Integrating Angular with "Headless" Wagtail CMS

Wed, 28 Aug 2019 12:00:00 +0000

I gave a talk at the Django NYC meetup. Here is a link. I spoke about Wagtail SPA Integration and Angular Wagtail. I am hoping to call both projects 1.0 in September. Please give them and test and report bugs!

Controlling a ceiling fan with Simple Fan Control

Sun, 30 Jun 2019 12:00:00 +0000

I released Simple Fan Control today on Google Play, web, and source on Gitlab. This project’s genesis was the purchase of a Hunter Advocate fan with Internet connectivity. It’s app doesn’t work, which I wrote about recently. Simple Fan Control’s web version The app is build with NativeScript and works by interacting with Alya Network’s Internet of Things (IoT) service. If there was interest, I would explore communicating with the fan directly instead of through Alya Networks. The IoT world scares me a bit because users transmit personal data to a third party service they may not be aware even exists. Alya’s service collects name, address, email, and GPS coordinates. It’s scary to think what this data could be used for or it being leaked. There’s also the concern that the fan control app becomes useless if the internet is down or should the company shut it down. If you are using a Alya Networks based device or want to collaborate on using the code for other IoT projects please let me know by opening a Gitlab issue. I’m charging $4 for the app, but you can of course build it yourself for source. By purchasing the app, you’d support further development. Alya Networks dev boards aren’t free and would let me test out other configurations and device wifi connectivity. I do consulting work if you are a IoT company looking to improve your software. Get in touch with info at burkesoftware.com if you’d like to know more.

How to set up a Hunter fan Wifi control by decompiling the app

Sat, 01 Jun 2019 12:00:00 +0000

Update I made an open source web/android app that can do this for you. Note you’ll still need the SimpleConnect app to connect the fan to wifi. I got a hunter fan recently that was supposed to be controllable via an app called SimpleConnect. Looking at the reviews, it doesn’t work. It gets stuck on the email verification step. You get an email link that opens in the app and does nothing. I decided to inspect the apk file with dex2jar and JD-GUI. All the confirm account step actually has to do is send a PUT request to a url with a token from the email. No need for an app at all really.

Confirming the account

To confirm the email sign up and get the email link. It should look something like https://app-launcher.aylanetworks.com/launch?custom_url=aylacontrol://user_sign_up_token?token=XXXXXXXX All we need is the token. The app is supposed to then make a PUT request to https://user.aylanetworks.com/users/confirmation.json with a payload of

{
    "confirmation_token": "XXXXXXXX"
}

You can do this yourself in postman. Just enter the payload in the Body tab as raw JSON. You should get a response with the same personal data you entered before. It should include approved: true.

Adding the fan

The next issue you’ll face is that the QR code is printed too small with very poor quality. The app also seems to set the camera in some sort of poor quality mode. I tried a couple devices and eventual got the code to scan with a Pixel C tablet. Pretty amazing Ayla Networks made such a worthless app. No testing at all. But what do you expect with internet of things devices.

Google Assistant Integration

It’s not that hard to set up, but it’s not well documented. You just go here to set it up after setting up the simple connect wifi app. It’s a bit clunky saying “tell simple connect to do something” but it works.

Using Angular with a "headless" Wagtail CMS

Sat, 11 May 2019 12:00:00 +0000

Wagtail is a great Django-based content management system. Angular is a full-featured JavaScript framework. I wanted to use them together, so I made some helper libraries. Below, I explain how I did it. Goals:

Enable Wagtail features like preview and redirects.
Allow routing to be defined (mostly) in Wagtail
Maintain great performance through
Lazy loading JS modules
Compatible with Angular Universal for server side rendering
Ensure Wagtail Multi-site functionality works

wagtail and angular

Setting up Wagtail

Install from pypi wagtail_spa_integration using the instructions here. Since there is nothing angular specific about it - you could also use this with other front-end solutions. This package provides an extended Wagtail V2 Pages Endpoint.

Setting up Angular

Install angular-wagtail. Follow the instructions. At a high level, instead of setting routes to components, you will set Wagtail page types to either components or modules. For example the Wagtail Page “foo.MyPage” might map to MyPageComponent in Angular. I will call this dynamic routing, as opposed to Angular router’s fixed routes. This is all that’s needed for simple websites. However, angular-wagtail works by having the Angular project request page data for every page. This is a problem if your site has thousands of blog pages. Your Angular app may not need to know every blog URL up front. It just needs to know that they follow a schema like “blog/blog-post-slug”. You can make a lazy loaded module for blog and set the route like

{
  type:"cool_blog.BlogIndexPage",
  loadChildren:"./blog/blog.module#BlogModule"
}

There are some limitations. loadChildren won’t work with nested routes. If you have two components in BlogModule, then you can’t both lazy load the module and use the dynamic Wagtail driven routes. There are two workarounds. Ensure the lazy loaded modules only have one route or keep them in Angular’s routing instead of WagtailModule’s page type mapping. In the blog example, you may have a blog index page and a blog post page nested under. As long as you assume the route is always /blog and /blog/post-slug you don’t really need the dynamic routing that WagtailModule provides. Finally you need to gather page detail data in our components. In the ngOnInit function for add something like

constructor(private wagtail: WagtailService) {}
...
this.cmsData$=this.wagtail.getPageForCurrentUrl<IMyCoolPage>();

IMyCoolPage is the interface for the data you expect to receive from wagtail for this page. This works with both fixed routes in Angular router and dynamic routes in WagtailModule. These functions will also automatically check for redirects if a page is not found.

Closing thoughts

I really enjoy having all view logic in Angular instead of attempting to mix Django templates with a JS framework’s view layer (JSX, Angular’s templates, etc). Previously, this meant giving up a lot of features that “just work” in Wagtail. With using these packages, I can quickly bootstrap a headless Wagtail server with a separate Node/JavaScript front-end and keep all the features I’m used to. Please consider contributing some code, unit tests, or make a JS integration with your favorite framework.

Review: Pixel Slate for Linux and Web Development

Mon, 24 Dec 2018 12:00:00 +0000

The Pixel Slate (i7 model) can be a decent computer for web development, including Docker, Node, and Android development. My workplace recently got me one so I decided to review it for anyone curious about using it for Linux-based development.

Performance

I’m reviewing the highest-end version with an i7-8500Y CPU. Let’s break that down:

Y series is the 5 watt low power offering (not to be confused with the 15 watt U series which is for “Ultrabooks”). This allows the Slate to not have any vents or fans, making it perfectly quiet.
The 8 stands for 8th generation which is the newest generation for the Y series.
The “i7” means it’s both more expensive and faster than the same class i5. But that doesn’t mean an i7 Y series is going to be faster than a very old i3 desktop K series CPU. It’s essentially the same chip with more cores enabled and a higher clock frequency.

The i7-8500Y is considerably slower than a roughly equivalent i7-8550U as seen in the XPS 13 9370. (See my review of the XPS 13 developer edition here). Since I have both, I’ll do a few comparisons. All tests on the XPS 13 are run on Ubuntu 18.10. Basemark Web 3.0 Pixel Slate - 500.4 XPS 13 - 365.8 Wow - the Slate beats the XPS here - this is surprising! Both are running Chrome. My guess is that Chrome on the Slate has far better optimized drivers than stock Ubuntu on the XPS. This probably allowed the GPU to do more of the work, resulting in a higher score. Webpack I tested building Passit, the open source password manager I’m working on. Passit is built with angular-cli and uses webpack to build bundles. See the repo here if you want to compare. I ran a development build with “npm run build” Pixel Slate - 16 seconds XPS 13 - 11 seconds CPU benchmark I ran “sysbench –test=cpu –cpu-max-prime=20000 –num-threads=8 run” Pixel Slate - 5.8 seconds XPS 13 - 10.0 seconds Lower is better - and the Slate wins. I don’t understand this. It should be a simple CPU test, and the XPS 13 has a faster CPU with more cores. Since this test had odd results, I ran “stress-ng –cpu 6 –cpu-method matrixprod –metrics-brief –perf -t 60” too. 6 cores: Pixel Slate: 22839 ops XPS 13: 46106 ops 2 cores: Pixel Slate: 25464 ops XPS 13: 33069 ops This time the XPS got more than twice as many operations done in the 6 core test - presumably due to its extra cores. Even with just 2 cores, the XPS is still faster.

Docker and Django

As an example of back-end development, I’ll run the passit-backend (Django) tests in Docker. This shows the time required for creating a PostgreSQL database and running the Python tests. I ran: - docker-compose up db - time docker-compose run –rm web ./manage.py test Pixel Slate - 38 seconds XPS 13 - 26 seconds This test involves a mix of CPU and I/O bound operations. It’s not surprising that the XPS wins.

Linux Apps

Just a typical day in Chrome OS running Firefox, VS Code, and Docker I installed Firefox within five minutes of opening the Pixel Slate - because why not? Linux apps run mostly well on the Slate. Setting them up is easy - just enable that option in settings. Installing apps is easy for someone experienced with the Linux command line, but harder for someone new to Linux. For example, on most Linux OS’s, you can double click a package file (such as a .deb file) and it installs. Not so on Chrome OS - you’ll need to use apt and dpkg to install programs like VS Code and Firefox. Linux in Chrome OS (called Crostini) runs Debian Stretch in a container-based environment. That means it’s more efficient than a virtual machine and more secure that just executing Linux programs directly. It does add some inconveniences, such as having a separate file storage area (similar to Android). Most things work just fine, but an exception was Docker. I followed the comments here to get it working. I ran into another minor kink when installing gnome-terminal because no shortcut was created (every other app I installed did so and “just worked”). Crostini doesn’t support GPU acceleration at this time, so Steam gaming with the Slate isn’t going to be a great experience. Actual virtualization doesn’t work at all, although Wine does. One perk of using the Slate as a developer is that you can develop Android apps and run them right on the device without an emulator. This does require enabling developer mode, which leads you to a rather annoying startup screen that must be bypassed by pressing CTRL-D or waiting 10 seconds. It’s actually really handy running Android apps directly in Chrome OS and not taking the typical performance hit from full virtualization.

Mobility, Battery Life, and Other Features

The Slate weighs 1.6 lbs by itself; with the keyboard it’s 2.9 lbs. For comparison, the XPS 13 weighs 2.67 lbs - so the Slate as a laptop substitute is not a lighter option. I get 4-6 hours of battery life on the XPS 13 when actually working. The Pixel Slate does better - more like 6-12 hours. (It’s hard to estimate because I’m typically not continuously coding/compiling things for more than 6 hours at a stretch.) This is no surprise given the lower power requirements of the Slate’s CPU. The Slate easily goes into a suspend mode when inactive, just like an Android phone or tablet would. Ubuntu on the XPS is more finicky - it mostly works, but consumes more power when suspended and occasionally has glitches when waking. I would feel comfortable simply suspending the Slate when I step away from my work, whereas I often shut down my XPS 13 to avoid the issues just mentioned. The Slate doesn’t have a headphone jack, and only has two USB-C ports. If I want to charge it, listen to music (through an adapter), and plug in a second monitor at the same time, then I need a USB-C dock. Google doesn’t provide much guidance on what adapters or docks are supported. I found USB-C to DisplayPort to work fine with a 4k monitor at 60hz, while a USB-HDMI adapter I use for my XPS didn’t work at all with the Slate. USB-C docks don’t support 4k at 60hz, and the ports appear not to be Thunderbolt-compatible. I found this whole connection process confusing and annoying - but in the end I got what I wanted using a USB-C dock (for power and audio) and a separate cable for DisplayPort. The official Slate keyboard works as well as any device in this tablet-to-computer product class. It’s usable on your lap, but not good. It’s perfectly fine on a table. The round keys are a little odd, but I got used to them. It’s almost a full keyboard, including escape and F row keys - meaning I can use vim with it. This may be a matter of personal taste, but I find the Slate too large and burdensome for reading an e-book. One advantage of the size, though, is that I can read full-size magazine articles without having to zoom or use the lite version. The Slate’s magnets seem to be weaker than the Pixel-C’s, or maybe they’re the same but not strong enough for the increased weight. The Slate wouldn’t stay up when I tried sticking it to the fridge like I do with my Pixel-C. At the Slate’s vastly elevated price point, however, I probably wouldn’t trust it in the kitchen anyway!

Conclusion

As a developer, I’d feel confident using the Pixel Slate as a replacement for my tablet and laptop. I’d still want a faster desktop with this set up and as a backup just in case Docker or something didn’t work right. As something I got from work and didn’t pay for myself - it’s great! Pros

Great battery life
Fast web performance
A good way to run Linux with a solid, stable base OS that runs without glitches
Running Android apps next to Linux apps all inside Chrome OS is really cool

Cons

Expensive - I could buy both an XPS 13 and a small tablet for less money
CPU performance is slower than an “ultrabook”
No headphone jack and not enough USB-C ports

rxjs check as your type validation

Wed, 11 Jul 2018 12:00:00 +0000

rxjs has a steep learning curve, but can do some really cool things. Let’s say you want an input form to do “as you type” async validation. Perhaps it’s checking if the username is taken or not. Another use case could be checking if some url is valid. I implemented this with ngrx-effects (after failing a lot!) and thought I would share.

  @Effect()
  asyncServerUrlCheck$ = this.store.select(fromAccount.getLoginForm).pipe(
    filter(form => form.value.showUrl),
    distinctUntilChanged(
      (first, second) => first.value.url === second.value.url
    ),
    switchMap(form =>
      concat(
        timer(300).pipe(
          map(
            () => new StartAsyncValidationAction(form.controls.url.id, "exists")
          )
        ),
        this.userService.checkUrl(form.value.url).pipe(
          map(() => new ClearAsyncErrorAction(form.controls.url.id, "exists")),
          catchError(() => [
            new SetAsyncErrorAction(
              form.controls.url.id,
              "exists",
              form.value.url
            )
          ])
        )
      )
    )

Lots going on here and it sums up both my love and hate of rxjs. It’s unreadable garbage code until you understand it - then it’s fantastic. Let’s try to break this mess down. First off - notice the asyncServerUrlCheck observable (All ngrx effects are just observables) is watching state instead of actions$. This is done because I’m watching the form field’s state instead of waiting for an action. Then I filter out changes that are not to the form field in question and I make sure it’s a real change. Now the magic starts - next in my pipe is switchMap which is important because I want to cancel any previous observable. If the user types google.co and then google.com I probably don’t want to check if google.co exists. switchMap throws out any previous work and start over. Of note - if I didn’t use switchMap I would see a LOT more network requests. Next up is concat. concat is what is going to allow me to return multiple actions. Without it, I would just get the first start async validation and nothing else. concat is a static function and not a operator (Oh but it was an operator in rxjs 5 - which actually makes me hate rxjs a little because it’s so much mental overhead!). We’ll pass observables as parameters to concat. Our first concat observable is a timer. Timer is what implements the logic to wait until the user stops typing. Because we use it with switchMap earlier - it will get canceled if the user types something else! We could stop right now and have a start async validation action dispatch when the user stops typing. Cool. I do suggest trying this out piece by piece if you want to understand it rather than coping the entire snippet. Now I need to add success and failure actions after the async call is made. I’ll add a second parameter to concat which is my service function call. The function will return an Observable (Once again remember that concat accepts a list of observables). I pipe this into a map and catchError. That logic should look familiar if you used effects before so I won’t go into detail. This is how it looks in redux devtools. I get lots of set value’s from each user character input. But I don’t get a start async validation for each one (meaning I don’t excessively check the server!). Then I get either set async error or clear async error (success) actions depending on if the server url is valid. I’ve found this pattern hard to grok initially - but now is easy to apply elsewhere. Making async validation easy means I’ll be more likely to use it and give users a more interactive experience. Try it yourself at https://passit.io and download the chrome or firefox extension (The web version ask for server url). And if you aren’t a regular visitor to my blog - Passit is an open source, online password manager that my company built so please give it a try.

Forms with ngrx, NativeScript, and Angular

Mon, 09 Apr 2018 12:00:00 +0000

There are many ways to make forms in Angular. There’s template driven, reactive, and the question of syncing with ngrx state or keeping the it local to the component. When making a NativeScript app it’s not always obvious how to reuse these forms. For example, template driven forms in Angular might use the dom’s “required” attribute. NativeScript doesn’t have a dom or input component at all, so the required logic would have to be remade, perhaps using a required directive. Redux/ngrx driven forms offer a significant advantage when we have multiple platforms as ngrx is platform agnostic and we can perform the validation logic in the reducer instead of the component or a directive. As a case study, I recently rewrote Passit’s login form with the fantastic ngrx-forms package. ngrx-forms takes care of common use cases while providing a blueprint and examples of how to make the state driven forms. The validation logic is in the reducer, here two different platforms show the errors that get pulled from state Using ngrx-forms on the web is straight-forward, just follow the docs. For NativeScript you’ll have to make a few changes:

There is no “form” component in NativeScript, so you’ll have to manage isSubmitted yourself. You could modify the submit state in the reducer itself or in the component with MarkAsSubmittedAction.
ngrx-forms comes with directives to keep the form and ngrx data in sync. But these won’t work out of the box with NativeScript components. Here is a NgrxTextFieldViewAdapter for a TextInput. Just add the directive like [ngrxFormControlState]=”form.controls.yourField” and the TextInput state will sync with the form state, just like on the web.

Now I can reuse all of my form validation logic in both platforms. The only difference is the presentational components for app and web. Overall I think ngrx-forms offers a straight forward, redux friendly, and platform agnostic solution to forms. Please feel free to take a look at my Android preview release of Passit, the open source password manager. As I create more forms on both platforms I’m looking forward to having a single strategy for building them. Be sure to report bugs on gitlab.

Does your password manager really need permissions to do anything ever?

Sat, 17 Mar 2018 12:00:00 +0000

Almost all password manager’s browser extensions have permission to “Read and change all your data on the websites you visit”. If that sounds scary, it’s because it is. That’s the “” permission. It means the extension is allowed to execute arbitrary JavaScript at any time on any website without warning. Here’s some examples of what I could do if you installed an extension I made with .

Run a keylogger on every webpage you visit
Inject extra ads into every website
Run a password manager that autofills every login form even when you do not ask it to - this is in fact common. This includes malicious forms that might have been injected into a victims website to steal your password.
Run a password manager that checks each and every domain you ever visit and forget to sanitize the domain url making it vulnerable to code injection attacks that could lead to a rouge website capturing all of your passwords.
If I’m feeling only slightly evil, I simply record each domain you ever visit and sell that data to advertisers.

Passit does not require the permission. This doesn’t make us invulnerable to all extension-based attacks, but it greatly mitigates them. Let’s consider some:

“I sell you out (perhaps to another company) and start making my extension serve ads or other garbage” - you’d get a notification about the increased permissions and hopefully you’d check our blog to see why we want scary permissions.
“My sloppy code is vulnerable to JS injection attacks” - but because Passit doesn’t run until you invoke it, that probably only happens on websites you already know and trust at least somewhat.

Forget security vs convenience

(Sort of) Passit has easy to use shortcuts for autofill, so you don’t give up much convenience. I personally don’t find pressing a shortcut key to log in to be a big burden, especially with such nice security gains. Our strategy also means Passit will never have Clippy-esque “Would you like to save this password??” forms because Passit will never bother you until you activate it. That said, Passit is positioned to be a web-based, easy-to-sync, and share/organization-friendly password manager. I think password managers like pass still offer a benefit for personal use when you don’t care about sharing or autofill. The most secure password manager would be a piece of paper in a safe in a fort. At some point, we have to pick where we are comfortable between security and privacy vs convenience. I hope Passit makes an appealing choice that is nicer to use than programs like pass or KeePass while providing better security and privacy than LastPass or 1Password. Try Passit out today. Use our free hosted service or run it yourself. If you like it, please star us on Gitlab and report some feature requests or issues.

Review: Dell XPS 13 9370 Developer Edition

Fri, 09 Feb 2018 12:00:00 +0000

As the owner of Burke Software and Consulting I get to play with a few more Linux laptops than I would as an individual. I recently picked up Dell’s latest XPS 13 (9370) Developer Edition. Here’s my review as a developer.

Comparing Laptops

I compared the current 9370 model with the 2016 9350 model. I also compared a couple benchmarks with the Galago Pro 2 from System76, which is another laptop with Linux preloaded. Both XPS computers are top of the line with Intel i7 branded CPUs. The 9370 is an 8th generation Intel i7-8550U CPU while the 9350 is a 6th generation i7-6560U. The Galago Pro was configured with a i5 processor - so while it’s interesting to throw in it’s not a totally fair comparison.

Hardware and Appearance

The 9370 model is noticeably slimmer as seen in these photos. The other dimensions are the same. It’s noticeably a little lighter too. I tested it with a friend, having them close their eyes and pick the lighter laptop to make sure it wasn’t a placebo effect. The Dell USB-C charger is also a little bit smaller which is nice. The camera is still unusable due to its placement. Dell moved it to the center in the 9370 model which is maybe a slight improvement, but all you will see is your fingers on the keyboard and up your nose with this camera. I can’t think of any circumstance in which I would use this horribly placed camera.

Benchmarks

I benchmarked some typical developer tasks. I tested by building the open source password manager Passit. This has been my passion project for the past couple of years and if you aren’t using a password manager yet and like open source I encourage you to try it out (and give me feedback). I tested a Webpack bundle from the passit-frontend Gitlab repo. This runs angular-cli’s serve command. yarn start (webpack)9370: 11173ms 9350: 12322ms Galago: 13945ms It appears the 9370 is only slightly faster, saving about 1 full second. That is not impressive for a upgrade of two CPU generations. Ubuntu Disk Utility BenchmarkAverage Read Rate 9370: 2.7 GB/s 9350: 1.6 GB/s Average Access Time (lower is better) 9370: 0.04 msec 9350: 0.22 msec Looks like the 9370 has a faster SSD. That’s always a good thing. We can see how benchmarks look impressive but real world results don’t reflect it. time tns run android 9370: 1m 49s 9350: 2 m9s This test uses the passit-mobile repo. There’s a lot going on. It needs to compile the typescript into JavaScript, build the apk file, start an android emulator, and load the apk file on the device. I did my test using the Unix “time” command. I manually stopped it when I saw the Passit app fully loaded in the emulator. I like this test because it does SO much. Some of the Android compilers take advantage of multiple cores while the node based tools can use only one core. A big change on the i7-8550U CPU is having 4 cores and 8 threads (previous comparable U series CPUs had 2 cores). We can see a decent speed boost on the new XPS here. Nothing ground shattering. If your workload is more multi core utilizing you might see bigger jumps in performance. All tests were done on battery.

Battery

I was pleased to see a good boost in battery life on the new XPS 9370 despite it having smaller battery and a higher resolution display of 3840 x 2160 (9350 topped out at 3200 x 1800). Battery testing is hard and your results will vary. In the type of work I do (web development, vim, browsing) I got about 4-5 hours on the 9350. I seem to get around 7-8 on the 9370. I’ve only been using it a couple days, so I’ll update this post if I find it varies. On both set ups I do not have powertop or TLP installed - I find they can make Linux too unstable for me. If you opt for the lower screen resolution on any XPS model, you will get vastly better battery life. Unfortunately, Dell does not offer the lower resolution with 16GB of RAM, which made it not an option for me.

Gaming

The XPS 13 line is not for gaming so I’ll keep this short. The 9350 had an Intel Iris integrated GPU option while the 9370 offers only the less powerful UHD 620 integrated GPU. At first I was worried this would mean no more Cities Skylines for me - but that is not the case. Cities Skylines runs just fine on very low settings on the 9370. Dell says the heat dissipation is improved on their new laptop so it may be one reason for decent performance even without the better Iris GPU. If you want some light gaming, the XPS 13 dev edition is a solid choice.

Conclusion

The new Dell XPS developer edition is a modest improvement. I think the better battery life even on the higher end model is the biggest improvement. It’s probably not worth upgrading from the 9350 model but it might be from anything earlier. Performance gains are minimal, possibly due to Intel CPU’s having only minor performance upgrades the past couple years.

Building Nativescript apps on Gitlab CI

Sat, 27 Jan 2018 12:00:00 +0000

I’m building a Passit Android app through Gitlab CI. My goal is to produce a signed apk file in passit-mobile’s public Gitlab repo without exposing my signing key. This post will outline what I did and act as a starting point to anyone wishing to do the same. It should also be relevant for anyone using React Native or another Docker based CI solution. At a high level: In order to generate the APK file automatically in Gitlab CI, I need to build my project in Docker. That means I need the Android SDK and the Node environment for Nativescript in Docker. I also need to get my key file and password to the CI runner in a secure manner. Finally, I need to place the file somewhere where it can be downloaded. Here’s my complete .gitlab-ci.yml file.

Local Docker

Local docker isn’t CI! OK, but we need to run everything locally inside Docker first to make sure it works. It’s no fun waiting on slow CI when debugging. I’ll create a Dockerfile for my project. I found runmymind/docker-android-sdk:ubuntu-standalone to be the most popular and still-updated Android SDK Docker image at the time of this writing. However, the image doesn’t have node installed, since Android is Java-based. So my Dockerfile is going to install node, Nativescript, other packages, and the Android platform version I want. Here are the relevant lines:

# Installs Node.js
ENV NODE_VERSION 8.9.4
RUN cd && \
 wget -q http://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.gz && \
 tar -xzf node-v${NODE_VERSION}-linux-x64.tar.gz && \
 mv node-v${NODE_VERSION}-linux-x64 /opt/node && \
 rm node-v${NODE_VERSION}-linux-x64.tar.gz
ENV PATH ${PATH}:/opt/node/bin

# Installs nativescript
RUN npm install -g nativescript --unsafe-perm
RUN npm install nativescript --unsafe-perm
# installs android platform
RUN $ANDROID_HOME/tools/bin/sdkmanager "tools" "platform-tools" "platforms;android-25" "build-tools;25.0.2" "extras;android;m2repository" "extras;google;m2repository"

Neat. Notice the –unsafe-perm in the Nativescript install. That’s right from the Nativescript advanced install steps. At this time platform version (25) works with Nativescript, but you may need to change it. I also made a tiny docker compose file. I probably don’t need to but I like it’s easy to remember cli syntax. To build and run the docker image:

Install Docker if you haven’t already
docker-compose build
docker-compose run –rm android bash

The last step is to get a bash terminal inside the container. I suggest trying a tns command to see if it works - it may not connect to any emulators nor devices, but it should be able to build.

Building

Before continuing, I suggest you read https://docs.nativescript.org/publishing/publishing-android-apps if you haven’t already. Generally I build with webpack, aot, and snapshot but not uglify (can’t get it to work yet). This looks like: tns build android --release --bundle --env.snapshot --env.aot --key-store-path your-key.jks --key-store-password <your-password> --key-store-alias-password <your-password> --key-store-alias your-alias I’d suggest ensuring this command works outside of Docker first, then within your Docker container, and finally on CI. The reason to use all of this and not just tns build Android is to make app performance and startup better. Nativescript + Angular is quite slow otherwise. If you aren’t using Angular, it will be a little different but you can still use snapshot.

Gitlab CI

At this point you have verified that you can build an APK file inside of Docker. Next up is the .gitlab-ci.yml file. I basically just mimic my local Docker file. You could actually use the same image if you wanted and use Docker in Docker. It seems a little simpler to me to keep them separate. There are a few extra commands I’ll call out. echo $ANDROID_KEY_BASE64 | base64 -d > key.jks I need to get my key file into gitlab CI without committing to the public repo. I can use Gitlab CI’s protected secret variables for this. Unfortunately, Gitlab doesn’t support private files at this time. But I can base64 encode bytes - and what is a file if not a series of bytes? cat your-key.jks | base64 -w 0 This will convert your key file to base64 without newlines (which will be more gitlab CI variable friendly). Now paste this base64 into a secret and protected variable. I called mine ANDROID_KEY_BASE64. I also set my password for the key to ANDROID_KEY_PASSWORD (doesn’t need to be base64). Now I can regenerate my key file on the fly in the CI runner. Protected secret variables will only show up in a protected branch - and I can limit who has access to protected branches. So no one can get my info by, say, submitting a merge request with echo $ANDROID_KEY_PASSWORD. Even if your repo is private, it’s not a bad idea to keep secrets hidden so that others can contribute without seeing them. The last thing I’ll note is my artifacts section, which will store my APK file after each build. This is great for daily build users or if you want to just manually upload the APK to Google Play and other stores. Ensuring the app builds with production settings is also a fantastic test in itself.

Final Thoughts

That’s it - fully automated APK builds. This method will not work for iOS, as Apple does not allow you to build iOS apps in non-Apple operating systems such as Linux. And Docker runs in Linux. (Side rant: Screw you, Apple! I sure hope the U.S. gets an administration that cares about anti-trust again - then maybe your development process wouldn’t be two decades in the past. Why would you make a human build something when it could just be automated?) Improvements and/or the next levels of this process:

Running unit and integration tests. Unit testing comes with nativescript and integration testing could be done with Appium. You can actually run an Android emulator in Docker.
Automating the deploy based on branch or tags to publish to any stores.
It would be great if someone made a Nativescript app-building Docker image - that would let me simplify many steps here. I could imagine even tagging each supported Android platform version so the only steps to do per project would be sending over the key file, npm install, and tns build.
You could try iOS builds by hooking up a physical Apple computer to gitlab CI - you can keep it in your garage right next to your horse and buggy!

Feel free to comment if any of these steps aren’t working right!

Server side tracking with piwik and Django

Sun, 26 Nov 2017 12:00:00 +0000

Business owners want to track usage to gain insights on how users actually use their sites and apps. However tracking can raise privacy concerns, lead to poor site performance, and raises security concerns by inviting third party javascript to run. For Passit, an open source password manager, we wanted to track how people use our app and view our passit.io marketing site. However we serve a privacy sensitive market. Letting a company like Google snoop on your password manager feels very wrong. Our solution is to use the open source and self hosted piwik analytics application with server side tracking.

Traditional client side tracking for our marketing site

passit.io uses the piwik javascript tracker. It runs on the same domain (piwik.passit.io) and doesn’t get flagged by Privacy Badger as a tracking tool. It won’t track your entire web history like Google Analytics or Facebook like buttons do. Nice green 0 from privacy badger! To respect privacy we can keep on the default piwik settings to anonomize ip addresses and respect the do not track header.

Server side tracking for app.passit.io

We’d like to have some idea of how people use our app as well. Sign ups, log ins, groups usage, ect. However injecting client side code feels wrong here. It would be a waste of your computer’s resources to track your movements to our piwik server and provides an attack vector. What if someone hijacked our piwik server and tried to inject random js into the passit app? We can track usage of the app.passit.io api on the server side instead. We can simply track how many people use different api endpoints to get a good indication of user activity.

Django and piwik

Presenting django-server-side-piwik- a drop in Django app that uses middleware and Celery to record server side analytics. Let’s talk about how it’s built. server_side_piwik uses the python piwikapi package to track server side usage. Their quickstart section shows how. We can implement it as Django middleware. Every request will have some data serialized and sent to a celery task for further processing. This means our main request thread isn’t blocked and we don’t slow down the app just to run analytics.

class PiwikMiddleware(object):
  """ Record every request to piwik """
  def __init__(self, get_response):
  self.get_response = get_response

def __call__(self, request):
  response = self.get_response(request)

  SITE_ID = getattr(settings, 'PIWIK_SITE_ID', None)
  if SITE_ID:
    ip = get_ip(request)
    keys_to_serialize = [
      'HTTP_USER_AGENT',
      'REMOTE_ADDR',
      'HTTP_REFERER',
      'HTTP_ACCEPT_LANGUAGE',
      'SERVER_NAME',
      'PATH_INFO',
      'QUERY_STRING',
    ]
    data = {
      'HTTPS': request.is_secure() 
    }
    for key in keys_to_serialize:
      if key in request.META:
        data[key] = request.META[key]
    record_analytic.delay(data, ip)
  return response

Now you can track usage from the backend which better respects user privacy. No javascript and no Google Analytics involved! Feel free to check out the project on gitlab and let me know any comments or issues. Passit’s source is also on gitlab.

Using libsodium in Android and NativeScript.

Tue, 20 Jun 2017 12:00:00 +0000

Libsodium is a fantastic crypto library, however it’s documentation can be lacking. Libsodium supports many languages via native wrappers and javascript via asm.js. I’ll document here how I got it working in Android both in Java and NativeScript. The target audience is someone who knows web development but not Android development.

Java + Android

We’ll use libsodium-jni. The install instructions didn’t make sense to me at first because I’m not an experienced Android developer. There is no need to manually compile it - they publish builds (AAR files) to the Sonatype OSS repository. This is analogous to publishing packages to the NPM repository if you’re more familiar with javascript. Just like how there are a million ways to install javascript dependencies, there are a million and one ways to install java dependencies. This can make documentation appear confusing or contradictory. Here’s how I did it. Make a new Android Studio project (I suggest starting simple instead of using your existing project). Edit app/build.gradle and in the dependencies section add:

compile 'com.github.joshjdevl.libsodiumjni:libsodium-jni-aar:1.0.6

Note the latest version might change. The format of that string is ‘groupId:artifactId:version’ which you can find in libsodium-jni’s readme. That’s actually all you need to do to install it. gradle will find the package for you. Now try to run the app, you may get errors about android:allowBackup and the minimum android version supported. I don’t understand why allowBackup could have any connection to libsodium - Native dev kind of sucks if you are used to something nicer like Python or Web development. We can fix it by editing app/src/main/AndroidManifest.xml and adding an attribute to the application tag.

tools:replace="android:allowBackup"

If you run again, it will complain about the “tools” attribute. Android XML configuration is fucking awful - go add this attribute to the manifest tag.

xmlns:tools="http://schemas.android.com/tools"

Next we may need to fix the version number - this issue actually makes sense. Both your application and libsodium-jni declare the minimum android version supported. Just edit app/build.gradle and change minSdkVersion to whatever libsodium is asking for - at the time of this writing it’s minSdkVersion: 16. It should now compile. Next let’s add some test code. Open MainActivity.java and we’ll generate a key pair to prove libsodium works. Add imports:

import org.libsodium.jni.SodiumConstants;
import org.libsodium.jni.crypto.Random;
import org.libsodium.jni.keys.KeyPair;

Modify the onCreate function to generate and print out the key pair’s public key:

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    byte[] seed = new Random().randomBytes(SodiumConstants.SECRETKEY_BYTES);
    KeyPair encryptionKeyPair = new KeyPair(seed);
    System.out.println(encryptionKeyPair.getPublicKey());
    setContentView(R.layout.activity_main);
}

Now the app should run and the key should print out in logging (logcat). Success!

Nativescript

Screw Java and Android XML and gradle configuration - let’s use javascript. A few years ago this might have been a sarcastic remark… Amazingly, you can generate typescript definitions using this. It works great - but I’d actually suggest getting a little familiar with the Java library in Java first. Java and javascript use very different conventions and the generated type definitions might not be perfect. Here’s a generated libsodium-jni type def. https://gitlab.com/snippets/1665527 To install make the same changes you made in the pure java app. The gradle and xml files are in platforms/android/. Drop in the libsodium type def file into the root of your project. You should now have code completion and type checking. Now let’s add that keypair generation test again to make sure it’s working.

let rand = new org.libsodium.jni.crypto.Random().randomBytes(org.libsodium.jni.SodiumConstants.SECRETKEY_BYTES);
let key = new org.libsodium.jni.keys.KeyPair(rand);
console.log(key.getPublicKey());

Looks weird right? But it should work. You probably want to make a wrapper around this in either javascript or Java. Mixing Java calls into your javascript will look strange and be hard to read and debug. Hopefully you can run the project as usual with tns run android. You may want to build the project or run gradew clean first to avoid any cached builds. If you are successful you should again get the public key printed out in the log. Excellent! Now you can make a web/native hybrid app that uses libsodium’s cryptography and can run in the web and as an app!

One more gotcha

If you have used libsodium in another language like Python or Javascript you may be spoiled with higher level function wrappers. For example in javascript libsodium you can run “let keypair = sodium.crypto_box_keypair();” to generate a keypair. libsodium-jni has very few high level wrappers so you’ll need to write C-like code in Java to mimic the low level functions. To make matters worse libsodium-jni isn’t documented. I found searching the code for unit tests to be the only way to find the right syntax. If you just start calling functions at will you’ll probably get a missing implementation error. If this happens you may want to see how it’s done in a unit test. Here is a link to the earlier key pair example but in a low level way. Now you are writing C like code in Java in Javascript. This is your life now.

Building a continuous integration workflow with gitlab and openshift 3

Fri, 01 Apr 2016 12:00:00 +0000

In this post I’ll go over building and testing a Docker image with gitlab CI and then pushing that image to Openshift 3. It should be somewhat helpful for people using other Docker solutions like Kubernetes too or CI solutions like Jenkins. I’m using Django for the project with some front end assets built in node. Our goal is to have one docker environment used in development, CI, staging, and production. We’ll avoid repeating ourselves with image building. At a high level my workflow looks like

Local development

All local development happens with docker compose. There is plenty of info on the matter so I’ll skip most of this. I will point out that I want to use the same python based docker image for development and later in production.

Continuous Integration

I’m using gitlab and gitlab CI runner to do testing and build a docker image. Gitlab has some docs on how to build a docker image. The choices are shell and docker-in-docker. I found docker-in-docker to be slow, complex, and error prone. In theory it would be better since the environments are more isolated.

Gitlab CI building images with shell executor

Here is my full .gitlab-ci.yml file for reference. The goal here is to build the image, run unit tests, deploy on success, and always clean up. I had to run gitlab ci runner as root otherwise I would get permission errors. :( In a non trivial CI system, shell can get messy too. We need to be concerned about building too many images and filling all disk space, exhausting the number of docker network subnet pools, and ensuring concurrency works (if you need that). Disk space - I suggest using a service that lets you attach a large volume that is formatted with an lot of inodes and using Docker’s overlayfs storage engine. I used AWS’s EC2 with a 120gb mount for /var/docker. See this blog post for details. Pay attention to the part where you define inodes. I went with 16568256. Docker clean up - Gitlab has a docker image that can help clean up docker images and containers for you here. I’d also consider restarting the server at night and running your own clean up scripts too. I also place a CI cleanup stage like

stages:
  - build
  - test
  - deploy
  - clean

...
variables:
  PROJECT_NAME: myproject$CI_BUILD_REF
  COMPOSE: docker-compose -p myproject$CI_BUILD_REF

...
clean_docker:
  stage: clean
  when: always
  script:
    - $COMPOSE stop
    - $COMPOSE down
    - $COMPOSE rm -f

I’ve been using docker since 1.0 and I’m still always amazed by how it finds new ways of breaking itself. You may need to add your own hacks to seek and destroy docker images and containers that will want to build up forever. The $CI_BUILD_REF is to ensure each docker image is unique - this allows us to run multiple builds and have some certainty the image being tested is the one being pushed to docker hub. The test stages are rather django/node specific. Just place whatever code needs to execute to run tests here. If it gets a success exit code gitlab CI will know it passed. Pushing to docker hub - I’m tagging my tested image, pushing it to docker hub, and running a webhook to notify openshift to automatically pull the image and deploy it to staging.

deploy_staging:
  stage: deploy
  only:
   - qa
  script:
   - echo &quot;Tag and push ${PROJECT_NAME}_web&quot;
    - docker tag ${PROJECT_NAME}_web ${IMAGE_NAME}qa
    - docker push ${IMAGE_NAME}:qa
    - &quot;./bin/send-deploy-webhook.sh qa $DEPLOY_WEBHOOK_STAGING&quot;

Notice how I’m only running this on the qa branch and that I’m tagging the image as “qa”. I’m using docker tags so that I can have one image that has different development stages - dev, staging, and production.

Openshift with docker build strategy

Openshift lets you build using a source to image strategy or docker. Source to image would mean rebuilding a docker image - which we already did in CI. So let’s not use that. The docker strategy was a bit confusing to me however. I ended up having a very minimal build stage using Openshift’s docker build strategy. Here is a snippet from my build yaml.

  source:
    type: Dockerfile
    dockerfile: &quot;FROM thelab/tsi-cocoon:devnRUN ./manage.py collectstatic --noinput&quot;
  strategy:
    type: Docker
    dockerStrategy:
      from:
        kind: DockerImage
        name: 'docker.io/user/image:dev'
      pullSecret:
        name: dockerhub
      forcePull: true

Notice the source type Dockerfile with a VERY minimal inline dockerfile that just gets the right image and collects static (Django specific - this could really just be FROM image-name) The strategy is set to type: Docker and includes my docker image and the “secret” needed to pull the image from my private repo. Note that if you must specify the full docker registry (docker.io/ect) or else it will not work with a private registry. You need to add the secret using oc secrets new dockerhub .dockercfg=dockercfg where dockercfg is the file that might be under ~/.dockercfg. forcePull is set to true so that openshift does a docker pull each time. You’ll need to define deployment, services, ect in openshift - but I include that in the scope of this post. I switched a source to image build to docker based without having to touch anything else. That’s it - the same docker image you used with compose locally should be on openshift. I set up a workflow where git commits on specific branches automatically deploy on openshift staging environments. Then I manually trigger the production deploy using the same image as staging.

Extending Openshift's source to image

Wed, 02 Mar 2016 12:00:00 +0000

Openshift 3 introduces a new concept “source to image” or s2i. It’s a way to create a Docker image out of some source code and a Docker image - for example a python s2i image. It makes Openshift 3’s docker based workflow feel more like Openshift 2 or Heroku. Image from http://blog.octo.com/en/openshift-3-private-paas-with-docker/ One problem I ran into using s2i-python was a lack of binary packages installed that are needed to build things like Pillow. For this we need to extend the base image to add what we want. You can see the end result on github. Let’s review the Dockerfile I build. We start by extending centos’s s2i image. Note I’m using Python 3.4. You can review the upstream project here to see what I’m extending. FROM centos/python-34-centos7 Next I’m mimicking how centos installs packages from the upstream file. Upstream and my version You can see I installed postgresql client tools and node too which I need. I also install a couple pip packages I know I’ll need on all my python projects - this is done just to speed up the build time. Do as much or as little as you want. Next I’ll post it on Docker hub with automated builds. See here. I added centos/python-34-centos7 as a linked repository. This way my docker image builds any time the upstream image builds too - ensuring I get security updates. Finally I just use the image as I would the original s2i python in openshift. I can add it as I would any image with oc import-image s2i-python --from="thelab/s2i-python:latest" --confirm Next post I’ll describe how to reuse a customized s2i python image on local development with Docker compose.

Finding near locations with GeoDjango and Postgis Part II

Fri, 02 Oct 2015 12:00:00 +0000

This is a continuation of part I.

Converting phrases into coordinates

Let’s test out Nominatim - the Open Street Maps tool to search their map data. We basically want whatever the user types in to turn into map coordinates.

from geopy.geocoders import Nominatim
from django.contrib.gis.geos import Point

def words_to_point(q):
    geolocator = Nominatim()
    location = geolocator.geocode(q)
    point = Point(location.longitude, location.latitude)
    return point

We can take vague statements like New York and get the coordinates of New York City. So easy!

Putting things together

I’ll use Django Rest Framework to make an api where we can query this data and return json. I’m leaving out some scaffolding code - so make sure you are familiar with Django Rest Framework first - or just make your own view without it.

class LocationViewSet(mixins.ListModelMixin, viewsets.GenericViewSet):
    """ Searchable list of locations

    GET params:

    - q - Location search paramater (New York or 10001)
    - distance - Distance away. Defaults to 50
    """
    serializer_class = LocationSerializer
    queryset = Location.objects.all()

    def get_queryset(self, *args, **kwargs):
        queryset = self.queryset
        q = self.request.GET.get('q')
        distance = self.request.GET.get('distance', 50)
        if q:
            point = words_to_point(q)
            distance = D(mi=distance)
            queryset = queryset.filter(point__distance_lte=(point, distance))
        return queryset

My central park location shows when I search for New York, but not London. Cool. At this point we have a usable backend. Next steps would be to pick a client side solution to present this data. I might post more on that later.

Finding near locations with GeoDjango and Postgis Part I

Thu, 01 Oct 2015 12:00:00 +0000

With GeoDjango we can find places in proximity to other places - this is very useful for things like a store locator. Let’s use a store locater as an example. Our store locator needs to be able to read in messy user input (zip, address, city, some combination). Then, locate any stores we have nearby.

General concept and theory

We have two problems to solve. One is to turn messy address input into a point on the globe. Then we need a way to query this point against other known points and determine which locations are close.

Set up known locations

Before we can really begin we need to set up GeoDjango. You can read the docs or use docker-compose. It’s still a good idea to read the tutorial even if you use docker. Let’s add a location. Something like:

class Location(models.Model):
    name = models.CharField(max_length=70)
    point = models.PointField()

    objects = models.GeoManager()

A PointField stores a point on the map. Because Earth is not flat we can’t use simple X, Y coordinates. Luckily you can almost think of Latitude and Longitude as X, Y. GeoDjango defaults to this. It’s also easy to get Latitude and Longitude from places like Google Maps. So if we want - we can ignore the complexities of mapping coordinates on Earth. Or you can read up on SRID if you want to learn more. At this point we can start creating locations with points - but for ease of use add GeoModelAdmin to Django Admin to use Open Street Maps to set points.

from django.contrib import admin
from django.contrib.gis.admin import GeoModelAdmin
from .models import Location

@admin.register(Location)
class LocationAdmin(GeoModelAdmin):
    pass

Wow! We’re doing GIS! Add a few locations. If you want to get their coordinates just type location.point.x (or y).

Querying for distance.

Django has some docs for this. Basically make a new point. Then query distance. Like this:

from django.contrib.gis.geos import fromstr
from django.contrib.gis.measure import D
from .models import Location

geom = fromstr('POINT(-73 40)')
Location.objects.filter(point__distance_lte=(geom, D(m=10000)))

m is meters - you can pass all sorts of things though. The result should be a queryset of Locations that are near our “geom” location. Already we can find locations near other locations or arbitrary points! In Part II I’ll explain how to use Open Street Maps to turn a fuzzy query like “New York” into a point. And from there we can make a store locator!

Review of Tutanota

Sun, 27 Sep 2015 12:00:00 +0000

Tutanota is an open source email provider. It features easy to use end to end encryption. It’s notable as a modern, libre, and cheap hosted email provider.

Why not gmail

Gmail is easily the best email provider. It’s light years ahead of any open source system. It’s gratis for individual, education, nonprofit, and small business use. That makes it hard to compete with. However it’s also a giant proprietary service directing a huge portion of your communication to the world. As a free software advocate it’s hard to feel good about using Google services - especially ones as critical as communication. Google also took concerning steps to remove interoperability in it’s chat service by removing XMPP in the Google Hangouts. In looking at alternatives I want something hosted (I do not want to deal with tech problems at home for something as important as email). I’d like something cheap - because even cheap is more expensive that Gmail. I also want something modern looking and easy to use.

The Good

Tutanota’s big feature is security. It’s not hosted in the United States. You can encrypt emails by clicking a button. It doesn’t sell your information to ad networks. It doesn’t leak your personal information when such ad networks or government agencies get hacked. Tutanota has a modern web interface with native mobile apps (web wrapper, but acceptable). It’s minimalist but that isn’t necessarily bad. Tutanota is easy to use. You can use their hosted version if you trust them and don’t want to host yourself. Setting up a custom domain was pretty easy. Aliases are no problem. It supports multiple users too (but I haven’t tested this out yet). It’s cheap at 1€ per month per user for the premium account. It also has a free tier if you can go without a custom domain.

The Bad

Tutanota doesn’t support IMAP. That’s pretty annoying. I’d be highly annoyed if there was any decent open source email client - but as there is not I can forgive it. IMAP wouldn’t work with their security model. I wish they had it as an option with a big warning about security. It lacks a lot of features you expect in email. As I said Gmail is way ahead here. No magic category filtering. No filtering rules. No contact book integration. Not even keyboard shortcuts. Tutanota only recently open sourced it’s code. It needs to do more to promote a developer community. A public issue tracker would be nice. So would a contribution guide. Right now they only offer a user voice page that is more consumer focused. I left my feedback.

Conclusion

Tutanota is a great option if you want an open source email provider - but only if your feature requirements are minimal. It has great potential as they add more features. The world needs an easy way to send private messages and right now such privacy is a luxury for those very few who understand PGP encryption and can set up services themselves. I applaud anyone trying to make this easier.

Mistakes to make when starting a business (with Python)

Sat, 18 Jul 2015 12:00:00 +0000

I gave this talk today. Here are the slides. Feel free to reach out to me david at burkesoftware . com

Building an api for django activity stream with Generic Foreign Keys

Wed, 08 Jul 2015 12:00:00 +0000

I wanted to build a django-rest-framework api for interacting with django-activity-stream. Activity stream uses Generic Foreign Keys heavily which aren’t naturally supported. We can however reuse existing serializers and nest the data conditionally. Here is a ModelSerializer for activity steam’s Action model.

from rest_framework import serializers
from actstream.models import Action
from myapp.models import ThingA, ThingB
from myapp.serializers import ThingASerializer, ThingBSerializer

class GenericRelatedField(serializers.Field):
    def to_representation(self, value):
        if isinstance(value, ThingA):
            return ThingASerializer(value).data
        if isinstance(value, ThingB):
            return ThingBSerializer(value).data
        # Not found - return string.
        return str(value)

class ActionSerializer(serializers.ModelSerializer):
    actor = GenericRelatedField(read_only=True)
    target = GenericRelatedField(read_only=True)
    action_object = GenericRelatedField(read_only=True)

    class Meta:
        model = Action

GenericRelatedField will check if the value is an instance of a known Model and assign it the appropriate serializer. Next we can use a viewset for displaying Actions. Since activity stream uses querysets it’s pretty simple to integrate with a ModelViewSet. In my case I’m checking for a get parameter to determine whether we want all actions, actions of people the logged in user follows, or actions of the user. I added some filters on action and target content type too.

from rest_framework import viewsets
from actstream.models import user_stream, Action
from .serializers import ActionSerializer

class ActivityViewSet(viewsets.ReadOnlyModelViewSet):
    serializer_class = ActionSerializer

    def get_queryset(self):
        following = self.request.GET.get('following')
        if following and following != 'false' and following != '0':
            if following == 'myself':
                qs = user_stream(self.request.user, with_user_activity=True)
                return qs.filter(actor_object_id=self.request.user.id)
            else:  # Everyone else but me
                return user_stream(self.request.user)
        return Action.objects.all()

    filter_fields = (
        'actor_content_type', 'actor_content_type__model',
        'target_content_type', 'target_content_type__model',
    )

Here’s the end result, lots of nested data.

Open source chat

Thu, 04 Jun 2015 12:00:00 +0000

Web based chatting services like Hipchat and Slack are catching on these days. Such services are proprietary which severely limits your freedom to extend them and gives control over communication to a for profit company and probably the NSA. What free alternatives are out there? We have good old IRC and XMPP servers. But these are typically a no go for non technical folks and time consuming to set up. Let’s review web based offerings starting with my favorite.

Let’s Chat

Let’s chat is based on node and mongodb. It features a nice web interface and some very basic xmpp support. It is not federated sadly so you can’t talk with xmpp users outside your server. It has no hosted offering. Installing it was the easiest of the options I looked at yet very buggy and annoying. This is why Slack is winning folks. Anyway here’s how I ran it.

Install from git. Do not use docker due to this bug.
Enable xmpp in the settings. You must set the host setting due to this bug. Here is my settings. I also had to enable TLS to get xmpp to work.
I use the hubot connector for fun stuff like animated gifs and scripting - which makes it more comparable to slack which offers a lot of integrations.
For Android I use Conversations which is an amazing xmpp client.

While I really hated installing Let’s Chat due to all the bugs - I’m happy with it now that it’s running. While I speak a bit badly about all the bugs I really appreciate the work put into it.

Kaiwa

Kaiwa is a web front end for an XMPP server. The huge benefit of this is you get a full XMPP server like Prosody which supports many of the more modern XMPP conventions (XEP’s). Federation is REALLY amazing and awesome except you will never use it because no one uses XMPP. In theory it would let you talk with people on other servers. Kaiwa has some heavy requirements of a web server, ldap server, database, and xmpp server. ldap is a deal breaker for me. I would prefer storing user data by marking rusty nails that I must arrange myself in a garden of broken glass - because that would be far more pleasant than writing ldif files. The default kaiwa would lose all data when restarting docker even after mounting (looks to be an issue with the ldap container). I just gave up. Kaiwa also has no way to search history which is pretty important to me.

RocketChat

RocketChat is worth keeping an eye on, but is not finished or near feature complete as of this writing.

On just xmpp

XMPP is a standard for interoperable chat. That’s a great goal but also a big challenge. There is no good xmpp desktop client. None support images (Animated gifs I think are a big draw on why people like slack). The default chat client on Ubuntu, Empathy, hasn’t seen development in years. Google used to champion XMPP but hangouts dropped most support except very basic one to one non federated chat. You suck Google. Thanks to websockets it’s really easy to make chat apps. You can learn in a weekend. I made a simple app here with Django and Swampdragon. Learning XMPP on the other hand is hard. There are few resources and existing resources are out of date. XMPP tackles much more like federation so it’s unfair to compare it directly with websockets. Still, people are going to use what’s easy and that’s hosted proprietary services built on websockets. Hopefully open source can catch up. It would be great to see some crowd funded efforts in this space - I’d contribute.

The only time I will spam you

Thu, 02 Oct 2014 12:00:00 +0000

I’m applying to this $150,000 grant. We need 250 votes to be considered. Voting for us is an easy way to support projects like django-report-builder and django-simple-import. We spin off these third party apps whenever possible. Burke Software’s main focus is an open source school information system. If anything on this blog has helped you please consider giving us a vote. Thank you! Be warned the website requires a Facebook login. I apologize for that, I don’t even have a Facebook account myself and had to use my good old Testy McTest account to vote.