Integrating GitHub Issues as a Blogging Platform

Building a simple blog is my go-to project for trying out new backend or frontend technologies. While learning Erlang/Elixir at university, I created a prototype using an ad-hoc escript that generated things via a cronjob.

It was quick, dirty, and fun. I started with concatenating handwritten .html files and building an index page. Eventually, it evolved to compiling .md files by shelling out to pandoc.

However, as I began drafting posts, I realized that some features were missing. Post tagging was limited to the directory structure, and syntax highlighting wasn't working well. I wanted a platform that allowed me to make edits from anywhere, including devices without a command-line or git.

I needed to fix typos on the spot. After careful consideration, I decided to implement my blog abusing GitHub as a backend.

GitHub was a free service with a nice web UI that provided excellent syntax highlighting support, customizable tagging, pinning, and even supported comments for future extensibility.

Implementation

GitHub offers an open API with several options available. While I haven't really used it much beyond tinkering with language tutorials, I've heard great things about it.

Out of all the APIs provided, I opted for their GraphQL API since I'm a big fan of GraphQL and have prior experience with it, having worked on consulting projects for companies that use it on a large scale.

Initially, I stored my posts as files in my repo, and my backend server would crawl through directories based on the URL slug (e.g., localhost:4000/some/post would look for $MY_REPO/some/post.*). This approach was solid, but when I tried to query files through the repository { object(...) } API, it got a little gnarly 😅. I assume they designed it that way to avoid extensive post-processing on the underlying git protocol.

Honestly, I'd much rather query objects through repository { files("somePath") { ... }} instead, since it would make my server logic more manageable. The object field required me to pass instructions that git rev-parse could understand, and that's no fun 🤮.

After digging around, I found out that querying GitHub Issues could be done from the repository node and gave me the freedom to do whatever I wanted. So, I scrapped everything I had done so far and started fresh with this second iteration.

If you're interested, GitHub's GraphiQL explorer is awesome for playing around and getting a feel for GraphQL (if you're not already familiar with it) and understanding what's happening behind the scenes on my blog.

For instance, the basic query I'm doing is:

query($owner: String!, $name: String!) { 
  repository(owner: $owner, name: $name) {
    nameWithOwner,
    issues(first: 10, orderBy: {field: CREATED_AT, direction: DESC}) {
      nodes {
        number
        bodyHTML
        createdAt
        updatedAt
        title
        labels(first: 10) {
          nodes {
            name
          }
        }     
      }
      totalCount
    }
  }
}

This snippet gets the first ten issues ordered by CREATED_AT descending from a given repository. If one were concerned that anyone would be able to make issues in a given repo, you could further constrain the query with something like:

issues(filterBy: {createdBy: $owner}, first: 10, orderBy: {field: CREATED_AT, direction: DESC})

Absinthe is an absolutely amazing GraphQL server implementation for the backend, but since my site is rendered by the backend, this wasn't what I needed. Whenever I've worked on frontend code interacting with GraphQL, I've always used a popular library called Apollo but (at least at the time of implementation) there was no Apollo library for Elixir.

I ended up rolling my own simple GraphQL client library because I didn't think it would be too complex. I'm sure as time goes on, simple libraries will be built, but honestly, it was easy enough to work with by just POST-ing to GitHub's API endpoint. For future reference, I ended up with something similar to the following:

def fetch_issues_for_repo(owner, repo) do
  %HTTPoison.Response{body: body} =
    HTTPoison.post(
      "https://api.github.com/graphql",
      Jason.encode!(%{
        query: """
        query {
          repository(owner: \"#{owner}\", name: \"#{repo}\") {
            issues(first: 10, orderBy: {field: CREATED_AT, direction: DESC}) {
              nodes {
                title
                createdAt
                updatedAt
                body
                author {
                  login
                }
              }
            }
          }
        }
        """
      })
    )

  body
  |> Jason.decode!()
  |> Map.get("data")
  |> Map.get("repository")
  |> Map.get("issues")
  |> Map.get("nodes")
end

This gives me some HTML (and other metadata like tags, etc.) which I essentially insert ad-hoc into my own Phoenix templates, and it just works. If you're reading this post, you can see this approach works pretty well (I'm unlikely to change it going forwards 😜)

One downside I can see is that you're limited to a few thousand queries as rate-limiting* so as time goes forwards, I'll probably cache posts in an ETS table or something so that I'm not wasting my quota whenever someone clicks on a post.

Just for a bit of fun, you can see the source of this post. Features I'll definitely add going forwards are reactions to the main post, which I can display nicely on the index page, and comments!

I'll continue playing around with my blog and the technology driving it, as is tradition for software engineers, but all-in-all, this is a pretty comfy blogging setup 😈

Edit: I'm still using this approach after a few months, but I definitely needed to implement caching. I woke up one night to my Graph API rate limits exceeded and couldn't do anything about it but wait! That's a good problem to have though 😉

Edit: One fancy thing I've seen is adding estimated reading times to blog posts. This proved a little hard because when I request blog contents from GitHub, I get them all as one monolithic block, which I inject into my templates...

To get it working, I end up processing the HTML returned by GitHub to calculate a reading time, injecting the reading time in the template and maximally abusing CSS Grid to re-order the flow of the page as follows:

sub + h1 {
  grid-area: afterHeaderBeforeContent
}

h1 + p:first-of-type {
  grid-area: afterContent
}

Edit: Updated writing since it was full of typos. This draft should be much clearer — my writing has really improved thanks to this blog... I highly recommend anyone start their own blog just for the fun and flexibility it enables you and be a great place to improve your writing skills!

Setting up a development environment on Windows 10

I find myself writing a lot of code, both professionally and as a hobby. When I write code, 99% of the time, I'm targeting a Unix-like environment.

I also find myself doing a lot of other things, such as playing games or streaming some TV. It is my sincere opinion that for these things, Windows is the operating system to beat: monopolistic malpractices or not, 4K Netflix only works on Windows 10, as do the majority of games I play which I use to keep in contact with old friends.

Even when I'm not using my powerful workstation with its beefy GPU to play games, I frequently use various models of the Microsoft Surface lineup because their form factor is really appealing to me—especially for a machine for the go. Linux, without surprise, does not run very well at all on these devices 😅

None of this is Linux's fault, of course. If I could use Linux on everything, I would, but the year of the Linux desktop is certainly not 2019 for me. It's not for lack of trying either, I've been a longtime Linux user, and I've tried many approaches to try and get the best of both worlds. I hope that this post can share what I've learnt along the way 💪

Popular (or not) approaches I've tried

As I've said, I've tried a lot of different ways to combine the leisure of Windows and the development potential (and customisability) or Linux. Some of the ones I actually stuck to for a while are outlined in the seconds below:

Windows–native development

For a long time, primarily during the first half of my University experience, I used Windows as a host OS and developed exclusively on/for Windows. If I were writing Python, or Erlang, or Ocaml, I'd use the Windows-native builds of those programming languages/compilers.

This felt natural to me because I started my developer journey when I was nine and playing around with Game Maker. It also worked relatively well for interpreted languages, as well as for C/C++ so long as I was developing against a Windows API. Everything kind of works, but a few weird things also don't work how you'd expect; maybe because most developers writing in languages such as Python or Erlang aren't using the Windows builds of their toolchains? 🤔 I assume if I wanted to stick to C# or .NET development, say, things would go a lot smoother, but alas.

When I couldn't manage to compile some NIFs for the BEAM (which I needed for a particular project at University), I ended up looking for another solution.

Linux VM on a Windows host

After trying to go all-in with native Windows development and being bitten by a couple of small and weird issues, I decided to do what anyone in my situation might and download a virtual machine to run Linux on.

The first thing I tried was VirtualBox, which ran pretty slow and had awful input latency when doing anything. I think this might have been a hardware issue.

VMware Workstation worked a lot better but still suffered from weird quirks like the virtual sound device not working correctly.

If not for the sound device issue, I probably could have lived with this setup. VMware could be maximised, and all of its Window decorations/toolbars could be hidden. It had some rudimentary 3D acceleration support as well, and things generally ran very smoothly. Unfortunately, if I were to do this, I'd want the sound to work as expected; it was also expensive both monetarily and battery-wise for my non-workstation machines.

The Windows ricing community: Cygwin & bblean

In 2015, I really fell into the Windows ricing community. To eke out the customisation that Linux gave me on Windows, I hacked around with hundreds of custom AutoHotkey scripts for system automation and custom keybindings, custom theming, custom shells to replace Window's default explorer.exe and more.

I discovered a custom shell for Windows called bblean which was based on Blackbox, a Linux window manager, perhaps best known nowadays for being a precursor of the much more popular Openbox and Fluxbox window managers. This scratched my customisation itch like never before and was genuinely perfect—until I switched to Windows 10 for modern DirectX support. Unfortunately, bblean was built targeting Windows 2000, and it was a miracle it worked as well as it did on every OS until Windows 10, where it still kind of works to this day.

There weren't very many people using bblean, and other alternative shells on Windows, however. The community was very tight-knit, and we shared a lot. Through being a part of this community, I also discovered Cygwin, which is essentially a collection of GNU (and other) tools, and a DLL providing POSIX-like functionality that you can install on Windows. Cygwin allows you to both install applications you might expect to find as part of a Linux distribution (such as zsh, vim, git etc.), and also compile POSIX-compliant applications to run on Windows natively (provided you have the DLL Cygwin provides).

The community was very focused on sharing setups and ideas. I'm unfortunately no longer an active participant in the community because I simply don't use Cygwin anymore (for my use case, a tool we'll explore in a moment is a much better fit). Still, I do have a few old screenshots of the lengths I'd go using these tools—something always fun to reminisce about.

The following images are Windows 7 configurations I had in 2015 and beyond. I taught myself CSS and basic JavaScript to be able to customise my Firefox 😁

Retrospectively, these tools taught me a lot and even contributed a lot to my first paying software development experience (which was a small company I interned at during my University degree. We used Cygwin on Windows extensively, and I attribute my outside experience with having used Cygwin as a strong selling point!)

These tools shall remain forever in my heart ❤️

Windows VM on a Linux host with GPU passthrough

After a few years of the bblean/Cygwin setup, I decided that I would try and learn to use Linux properly and full-time: my internship had just ended, and I had a lot of time on my hands before my final year at University, I had used Linux as a server OS and was familiar with the GNU userland because of Cygwin, but I was not 100% familiar with using it as a host OS for day to day computing.

It worked relatively well, all things being equal. Games, in general, don't work (though Valve's Proton is making genuinely amazing leaps to this end), especially online multiplayer games with anti-cheat. There were other small annoyances, such as high-dpi support not quite working properly (Wayland was not anywhere near being ready yet 😏), but it worked fine in general.

To get around the gaming issue, I set up a virtual machine using QEMU + GPU passthrough, which allowed me to run a Windows 10 in a virtual machine with a dedicated GPU. This worked amazingly, and game performance was effectively native—I couldn't even tell I was in a virtual machine!

It did require 2 monitors, however, and a software/hardware KVM to be used so that I could move my input devices from my host machine to the VM... and every time my GPU's driver updated, I'd end up getting bluescreens 😅

So close, yet so far...

Windows Subsystem for Linux

I heard about Windows Subsystem for Linux on Hacker News while I was still using my GPU passthrough solution and decided I'd try it out. If it worked well enough, I'd switch to it, given the grief my driver updates were causing me.

In short: it was similar to Cygwin in that it gives you a Linux-like CLI on Windows, but it worked by intercepting and translating Linux syscalls to NT syscalls in real-time, providing a real Linux userspace! No more Cygwin bundled DLLs and compiling any software I wanted for myself. I was effectively running a reverse Wine, and things worked pretty well!

This was pretty frictionless, but the performance wasn't the best (primarily because of Windows Defender and NTFS filesystem performance, apparently), but it was good enough for most things. Since WSL worked via syscall translation, however, there were some things that would not work: the lack of a real Linux kernel meant no Docker, no rootfs, no app images, but you could typically work around these issues.

Microsoft then decided to up their game by providing an alternative implementation of WSL: instead of translating syscalls, they would do the following:

1. Run a real Linux kernel alongside Window's NT Kernel using Hyper-V
2. Wrap said Linux kernel into a super lightweight VM (kind of like a Docker container)
3. Perform some black magic kernel extensions to enable Linux–Windows interoperability
4. Name it WSL 2 and profit 💰💰💰

I have been using WSL 2 in production since it was first available to try out in a preview branch of Windows. It's been nothing but a pleasure to use, and when using it, I genuinely feel like I have the best of both worlds.

The rest of this post will be about how to build a development environment atop WSL 2; however, before carrying on, there are two main things to note:

1. WSL 2 requires Hyper-V, which means your Windows 10 version will need to support it, and at the time of writing, this makes a machine using WSL 2 incompatible with Virtualbox or VMware software.

2. WSL 2 is currently only available in preview builds of Windows 10 as it's currently in development. WSL 2 is due for general release in Q1 2020, which isn't too far off, at least 🙏

Setting up WSL 2

Right now, WSL 2 is only available on Insider builds of Windows 10. You need to be on build number 18917 or higher to install it. You can follow these instructions to opt into these Insider builds, but it is important to note that you'll need to re-install Windows if you ever want to roll back to the standard distribution channel.

Next, we can enable the pre-requisite services for WSL 2 by executing the following lines in an elevated Powershell terminal (win+x should help bring it up):

Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

You'll be asked to reboot after running these commands, and after you do, you can go to the Microsoft Store and install any WSL distribution you'd like. I usually go for Ubuntu since it's the most supported one at the time of writing, though others are available.

To speed things up, after installing the distribution of your choice, but before executing it, in a Powershell or CMD prompt, execute the following:

wsl.exe --set-default-version 2 

This will cause your installed distribution to initialise itself in WSL 2 mode. You can convert versions freely between WSL 1 mode and WSL 2 mode, but it takes quite a long time, even for a base installation of Ubuntu, so running wsl.exe --set-default-version is preferable in my mind.

Congratulations, you now have access to a pretty complete Linux computing environment inside Windows 10! 🎉 Assuming you're using Ubuntu, you'll be able to apt-get install and run most commands you usually would.

After getting this far, one glaringly obvious issue is that your WSL 2 installation is being hosted in an old-school CMD prompt. This isn't ideal because it doesn't do very much: no true colour support, copy/paste is broken, and it generally isn't very customisable. Thankfully we can work around that in the next section 😉

Choosing a Terminal Emulator

There isn't too much choice for Terminal Emulators on Windows; unfortunately, every single one has some tradeoffs to consider.

For example, the default CMD prompt is actually blazingly fast with regards to input latency. It lacks, however, pretty much everything else I'd want from a terminal outside of decent mouse support (surprisingly). At this point, both my customised Vim configuration and Tmux completely break text rendering because of weird Unicode rendering issues however 😅

Cygwin's default terminal is pretty good. It's called minTTY, and it basically feels like the standard CMD prompt but supercharged. It has mouse support, 24-bit true colour support and more. Unfortunately, it also doesn't support rendering Unicode, so it's a no-go for my use case. If you're okay with Unicode not being supported, you can download wslTTY, which is essentially minTTY with WSL-integration—neat!

ConEmu is another super popular terminal emulator for Windows. It (and its several derivatives) are extremely customisable, approaching the level of terminal emulators you'd find on Linux. There are two main downsides with ConEmu, however: it doesn't easily support WSL by default (it's possible to get a working configuration, but of all the ones I tried, there are always tradeoffs with respect to losing some functionality, be it mouse support or weird text wrapping issues), and it gets very, very slow when running WSL for some reason 🤔 This might be due to the extreme newness of WSL 2 (or the fact what I'm running an insider build of Windows). Because of these issues, I cannot recommend ConEmu at this moment in time.

Then there's Windows Terminal, a super-strong new contender from Microsoft. They really have been doing amazing stuff in the last few years! This terminal emulator supports almost everything I want (lightning-fast, customisation features like the ability to set window padding, ligature support, true colour support), but unfortunately, it doesn't support any kind of mouse integration at the moment, which is a crutch I use a lot in my workflow 😭 Support is planned for the future, and right now the project is super early access. This is one to keep an eye on I'm sure.

Foregoing any of the imperfect choices already talked about (but really, feel free to try them all yourself. It's pretty workflow dependant after all), you can actually run an X Server on Windows and simply use X11 forwarding to run your favourite Linux terminal emulator as though it was running natively. I personally use this approach with the popular terminal emulator called Terminator, but as Windows Terminal matures, I may switch to that.

X11-forwarding between Windows and WSL

A few popular X Servers exist for Windows that work, and most of them are pretty similar. The most popular seem to be VcXsrv, MobaXTerm and X410.

I personally choose to use VcXsrv because it's the one I was already using from my Cygwin days (I believe Cygwin has its own X Server, but I've never tried it). VcXsrv is pretty fast, and it is free.

I have tried X410, and while it worked really seamlessly (it was built with WSL integration in mind, seemingly), my display is 1440p, and running applications would stutter and lag, unlike on VcXsrv. This is likely an issue that'll be fixed soon, but for what it's worth, you also need to buy a license to run X410 from the Microsoft Store.

Both VcXsrv and X410 (and I believe others) will enable you to start them in "multiwindow" mode. This essentially behaves as though Window's explorer.exe is the window manager for your X11 applications, and everything feels super native. Here you can see an image of me running a popular database management GUI, dbeaver, on WSL with X11 forwarding to Windows:

As you can see, it looks and behaves just like any other application!

Before you can run any applications via X11 forwarding, you'll need to add the following line to your shell's configuration:

WSL_HOST=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}')
export WSL_HOST=$WSL_HOST
export DISPLAY=$WSL_HOST:0.0

This sets your WSL instance's X11 display to the server running on Windows. For maximum performance, it's also often recommended to add the following line as well:

export LIBGL_ALWAYS_INDIRECT=1

This forces rendering to happen on Windows' side, rather than in software via WSL itself. Having done this, source your changes, and you should be able to just launch any GUI program and have it just work. Even chromium and vscode work if you really want to run them inside WSL for some reason; performance is decent too!

You can make X410 or VcXsrv automatically start by adding shortcuts to them in the following directory: %appdata%/Roaming/Microsoft/Start\ Menu/Programs/Startup/. I really recommend doing this since there isn't much reason not to avoid running an X Server on Windows. It helps improve the system interoperability story even more than already exists!

Other minor notes

How I use Docker

When I was using the original WSL, I was forced to run Docker on Windows and connect to it from my WSL instance. Occasionally things just broke, and this took time to debug and troubleshoot.

Now that WSL 2 is a thing, Docker for Windows has been updated to have its own WSL 2 backend. This works a lot better than standard WSL + Docker on Windows, but it's still not perfect. Occasionally, not shutting down or restarting my workstation for a few days would cause strange networking issues between my containers. I could get around this by running wsl.exe --shutdown but this, of course, also forced me to stop my work.

WSL 2 does, unlike WSL 1, include a real Linux kernel, however, so I opt for just running the native Linux implementation of Docker inside my WSL 2 distribution, and that just works. At this point, I'm not sure why Docker for Windows is even a thing 🤔

After switching to doing this, everything has been super stable! WSL doesn't support any kind of system services, however, so at least once per boot, you'll need to remember to run sudo service docker start to get everything in working order.

Creating shortcuts to Linux tools

Right now, any GUI application you want to run from WSL needs a command prompt open. This is because WSL shuts itself down if no terminals are open.

You can use (probably with some minor tweaking neccessary) the following Visual Basic script to spawn an invisible terminal as a host to launch your applications:

Set oShell = CreateObject ("Wscript.Shell") 
Dim strArgs
strArgs = "wsl bash -c 'DISPLAY=WSLHOST:0 <APPLICATION>'"
oShell.Run strArgs, 0, false

Simply change <APPLICATION> to your application of choice (i.e. vscode) and save this script as (in this example) vscode.vbs. Running this script will cause Visual Studio code to open as though it were a normal Windows shortcut 💪

Miscellaneous tips

You can get Linux window management capabilities such as easy dragging/resizing with AltDrag

Microsoft Powertools is a collection of utilities such as a pseudo tiling window manager.

You can set up hotkeys easily with AutoHotkey. I personally utilise these hotkeys in conjunction with the shortcut visual basic script I detailed above to launch Linux applications via keyboard shortcuts.

WSL allows you to import and export your distribution by simply running wsl.exe --export <YOUR_DISTRO> <PATH_TO_EXPORT_TO> and wsl.exe --import <NAME_FOR_IMPORTED_DISTRO> <PATH_TO_IMPORT_TO> <PATH_OF_EXPORTED_DISTRO>

Conclusion

I hope that this post helps you set up and configure a WSL 2 based development environment on Windows 10! There were definitely a few bumps along the way, but I think it works pretty well when you consider how new the technology is and how many small moving parts there are!

This technology is super excited for me, and it's only going to get better going forward. I'm just happy I get all the benefits of running Windows while still being able to develop with Unix-like semantics/idioms. 👀 on the future for even more improvements and interoperability! 🤞

Elixir pet peeves—Ecto virtual fields

Ecto is an amazing ORM that is super easy to use, and with good practices and conventions, it becomes extremely composable and flexible while at the same time being super light weight with minimal magic.

At times though, the lack of "magic" when working with Ecto can prove to be a little limiting: one such time this is an issue in my opinion is when using virtual fields.

What is a virtual field?

Virtual fields are just schema fields that aren't persisted/fetched from the database. For example, if we define the following Ecto.Schema:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string
  end
end

We could define a virtual field full_name which is defined as: field :full_name, :string, virtual: true. This effectively tells Elixir that there can exist a field called full_name but won't do anything beyond that.

This is useful because even though we're forced to write a function for resolving this virtual field (with or without the Ecto schema field definition added), it makes working with the struct itself in Elixir a little more annoying without defining it:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string
  end

  def resolve_full_name(%User{first_name: first_name, last_name: last_name} = user) do
    Map.put(user, :full_name, first_name <> " " <> last_name)
  end
end

iex(1)> MyApp.resolve_full_name(%MyApp.User{first_name: "Chris", last_name: "Bailey"})
%{__struct__: MyApp.User, first_name: "Chris", last_name: "Bailey", full_name: "Chris Bailey"}

Notice how that breaks the rendering of the MyApp.User struct? It also messes with autocomplete/dialyzer at times as well as locks you out from being able to reason about what fields a struct might have after one resolves all the custom fields one might want to attach to said struct.

If we define the schema with the virtual field, we can do the following instead:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string

    field :full_name, :string, virtual: true
  end

  def resolve_full_name(%User{first_name: first_name, last_name: last_name} = user) do
    %User{full_name: first_name <> " " <> last_name}
  end
end

iex(1)> MyApp.resolve_full_name(%MyApp.User{first_name: "Chris", last_name: "Bailey"})
%MyApp.User{first_name: "Chris", last_name: "Bailey", full_name: "Chris Bailey"}

Note that the inspection of the struct appears unbroken. Doing things this way also allows us to have default field values set up as well as peace of mind when we consider that we essentially enumerate every field we will want to ever work with (plus type definitions) on the schema itself.

Using virtual fields

Certain virtual fields can be resolved during query time, but more often than not (at least on projects I've seen or worked on) virtual fields are resolved after query time (i.e. in your contexts).

Virtual fields that are resolved at query time can be done via using Ecto.Query.select_merge/4: I could probably resolve MyApp.User.full_name at query time via the following Ecto query:

# This is certainly ugly and unneccessary to do like this, but more realistically if your
# virtual fields are just an average over some aggregate or join, this makes more sense to do.
# This is just to follow through with the example we've used thus far
from u in MyApp.User,
  where: u.id = 1,
  select_merge: %{full_name: fragment("? || ' ' || ?", u.first_name, u.last_name)}
  # better example would be:
  preload: [:friends],
  select_merge: %{friend_count: count(u.friends)}

But for simplicity and readability sake having it in your contexts is much nicer for simple virtual fields:

defmodule MyApp.Users do
  def get_user_by_id(id) do
    User
    |> User.where_id(id)
    |> Repo.one()
    |> case do
      %User{} = user ->
        {:ok, User.resolve_full_name(user)}
      nil ->
        {:error, :not_found}
    end
  end
end

The only annoyance with the latter approach is composability and repetition: wherever we want to resolve this virtual field (likely everywhere), we need to make sure to call User.resolve_full_name/1, and if we have multiple virtual fields to resolve this can become unwieldy (though one can have a User.resolve_virtual_fields/1 function instead, but the former point still stands. This is a longstanding and common pain point to using virtual fields in my opinion.

The pre-query approach to resolving virtual fields thus has the advantage that you don't necessarily need to worry about it, since select_merge calls can be composed and you build your query multiple times for your context functions in any case. Whether or not all virtual fields can be resolved this way, or whether or not it's worth the reduction of readability should be done on a case by case basis however.

Introducing Ecto.Hooks

In the past, Ecto actually provided functionality to execute callbacks whenever Ecto.Models (the old way of defining schemas) were created/read/updated/deleted from the database. This is exactly the kind of functionality one would like when trying to reduce duplication and points of modification when it comes to resolving virtual fields.

I wrote a pretty small library called EctoHooks which re-implements this callback functionality ontop of modern Ecto.

You can simply add use EctoHooks.Repo instead of use Ecto.Repo and callbacks will be automatically executed in your schema modules if they're defined.

Going back to the example given above, we can centralise virtual field handling as follows, using EctoHooks:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string

    field :full_name, :string, virtual: true
  end

  def after_get(%__MODULE__{first_name: first_name, last_name: last_name} = user) do
    %__MODULE__{user | full_name: first_name <> " " <> last_name}
  end
end

Simply add the following line to your application's corresponding MyApp.Repo
module:

use Ecto.Repo.Hooks

Any time an Ecto.Repo callback successfully returns a struct defined in a module that use-es Ecto.Model, any corresponding defined hooks are executed.

All hooks are of arity one, and take only the struct defined in the module as an argument. Hooks are expected to return an updated struct on success, any other value is treated as an error.

A list of valid hooks is listed below:

• after_get/1 which is executed following Ecto.Repo.all/2, Ecto.Repo.get/3, Ecto.Repo.get!/3, Ecto.Repo.get_by/3, Ecto.Repo.get_by!/3, Ecto.Repo.one/2, Ecto.Repo.one!/2.
• after_insert/1 which is executed following Ecto.Repo.insert/2, Ecto.Repo.insert!/2, Ecto.Repo.insert_or_update/2, Ecto.Repo.insert_or_update!/2
• after_update/1 which is executed following Ecto.Repo.update/2, Ecto.Repo.update!/2, Ecto.Repo.insert_or_update/2, Ecto.Repo.insert_or_update!/2
• after_delete/1 which is executed following Ecto.Repo.delete/2, Ecto.Repo.delete!/2

The EctoHooks repo repo has links to useful places like the Hex.pm entry for the library as well as documentation!

I hope this proves useful!

My usage of Nix, and Lorri + Direnv

I recently wrote about me transitioning my main working environment to run on NixOS. I've used NixOS on my main workstation and Nix atop Ubuntu on WSL2 on my laptop for ~5 months now and I feel I've really gotten into the flow with it on a handful of client projects.

The coolest bit of the setup is Nix/NixOS agnostic: lorri and direnv. At face value, they're usecase is pretty basic and we will dive into how you're meant to use them on projects, but using them actually enables some pretty cool functionality I've been trying to implement for awhile now.

What is Lorri and Direnv?

Nix provides a really cool utility built in called nix-shell. Basically, invoking nix-shell starts a new interactive shell after evaluating a given Nix expression (which by default is assumed to be ./shell.nix, relative to the CWD). This means that you can write a Nix expression detailing what packages you want to install, what environment variables to set, and automatically set/build/enable them within the context of the interactive shell.

This is already extremely powerful as you can declarative manage software versions etc, but Lorri + Direnv make this even nicer to use!

direnv is a super neat project which provides a shell hook that allows your shell to automatically set environment variables defined within a .envrc file (again, relative the the CWD) whenever you change directories. A basic .envrc is very simple and can look like the following:

export SOME_ENV_VARIABLE="test"

A major win here is that it's super fast, supports most popular shells (including some more esoteric ones), is language agnostic, and is pretty extensible.

lorri is essentially an extension of nix-shell built atop direnv. As soon as you change directory, if the new CWD has a shell.nix and an appropriate .envrc (autogenerated by running lorri init), the shell.nix is immediately evaluated. Lorri improves upon nix-shell in a bunch of ways as well, w.r.t caching of built derivations so it is also super quick.

General usage

Once you have lorri and direnv set up, the most basic use case in my opinion is replacing something like asdf which does much of the same stuff:

• it enables you to document what packages are needed for a certain project (within a certain limited, but extensible list)
• it enables you to specify certain versions of said packages
• it will automatically build said packages by running asdf install
• those packages will only be available if you're in a directory (or a child directory) with a .tool-versions file

Since lorri is built with Nix however, we get the following instead:

• it enables you to document what packages are needed for a certain project (any package available for Nix, which is huge)
• it enables you to specify certain versions of said packages, override versions, build certain custom versions
• it will automatically build said packages by entering the directory if needed
• those packages will only be available if you're in a directory (or a child directory) with a shell.nix file
• you can do anything Nix enables you to do, including file manipulation and describing environment variables

For example, here is an example shell.nix I needed to bootstrap an Elixir project which set some private credentials and needed Docker, GCS utilities, a DB client, and Minikube installed to run:

let
  pkgs = import <nixpkgs> {};
in
  g_sec_user = <REDACTED>;
  g_sec_password = <REDACTED>;

  pkgs.mkShell {
    buildInputs = [
      pkgs.elixir_1_10
      pkgs.nodejs-10_x
      pkgs.yarn
      pkgs.inotify-tools
      pkgs.openssl
      pkgs.kubectl
      pkgs.jq
      pkgs.google-cloud-sdk
      pkgs.minikube
      pkgs.kubernetes-helm
    ];
  }

Once this file is created, everything automagically works when entering that directory. Changes to the shell.nix are also automatically tracked so you don't need to do anything fancy: your shell.nix is your single source of truth.

Another advantage is that once your project has a shell.nix checked into it's repository, anyone running Nix/NixOS can start hacking away by just running shell.nix, but if you they Direnv + Lorri, they just have to enter the directory and stuff just works 🎉

Usage for Nix unfriendly projects

I'm an Erlang/Elixir consultant, and as a result of this, I have the opportunity to work on a lot of projects for a lot of different clients. I also know that a relatively esoteric package manager (or worse, an entirely different Linux distribution) is a hard sell.

Thankfully though, even in environments where you can't get away with commiting any of the lorri/direnv artefacts into the core repository of a project, you can get away with using lorri + direnv!

Because this entire workflow is dependent on directory structure, my workstation typically has a structure like this:

/home/chris/git
├── esl
│   ├── shell.nix
│   ├── internal_project_1
│   └── internal_project_2
│       └── shell.nix
├── client_1
│   ├── shell.nix
│   ├── client_project_1
│   ├── client_project_2
│   ├── client_project_3
│   └── client_project_4
├── client_2
│   ├── shell.nix
│   └── client_project_1
└── vereis
    ├── shell.nix
    ├── blog
    │   └── shell.nix
    ├── build_you_a_telemetry
    │   └── shell.nix
    ├── horde
    ├── httpoison
    └── nixos

If I'm not allowed to add direnv/lorri artefacts, I can simply put them a directory above the repository and I still get all of the benefits 😄 You need to add the source_up to any child .envrc files in order to also execute their parent but otherwise it works as expected.

This means that for I can install packages / handle dependencies on a project by project basis, or a client by client bases. Because the effects of executing these Nix expressions is cumulative, you can even do it on a client by client basis, except for a given project for that client.

As Lorri/Direnv lets me install dependencies and set environment variables, this setup is also perfect for a seamless dev credential manager solution. For example, assuming a client has it's own npm registry for JavaScript dependencies, you can simple do the following:

let
  pkgs = import <nixpkgs> {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.nodejs-10_x
  ];

  npm_config_user_config = toString ./. + "/.npmrc";
}

Where /home/chris/client_1/.npmrc contains:

registry=<REDACTED>
email=<REDACTED>
init.author.name="Chris Bailey"
init.author.email=<REDACTED>
always-auth=true
_auth=<REDACTED>

Notice that esl, client_1, client_2, and vereis all have a top level shell.nix. This is also perhaps a slight abuse of direnv/lorri but the end effect is that I'm able to treat these different directories as completely separate development domains. Each of these top level shell.nix files also set environment variables to set my git username and email. This means that I no longer need to faff around with setting up each and every git repo I clone: if I'm in /home/chris/git/vereis, I'm going to be working as @vereis, if I'm in /home/chris/git/esl/ I'll be @cbaileyesl etc.

Conclusion

I hope this small write-up has been helpful 😄 I genuinely think that the Nix ecosystem offers a lot of advantages, especially for developers. Tools like nix-shell, lorri, and direnv are super helpful in streamlining the development process.

Of course, tools like asdf, dotenv and others exist which allow you to emulate all of what I've described, but the true advantage of Nix/NixOS is that the configuration is very declarative and built into the core experience. Any nix user can do exactly what I've described, even if they're not running lorri or direnv simply by virtue that it's all built upon the first-class nix-shell.

I hope if you're not a nix user, this has piqued your interests a little bit, and if you are a nix user, this will help you jump a few of the small roadbumps when trying to approach development leveraging all the niceties nix provides you.

Drinking the NixOS kool aid

The last time I blogged about my development environment I had become disillusioned with the numerous problems that affected me when using Linux as my daily driver—scrolling and microstutters in Firefox, mouse acceleration and sensitivity setup was archaic and obscure, GPU issues to be worked around.

Ultimately, the times I did get a comfortable setup going on, after a few months something would inadvertantly break; or if I were trying to configure another machine, it'd be a lot of time spent emulating my exact setup.

I've heard about Nix and NixOS passingly over the last few years and with some down time in between projects at work, I decided to take a look at it in depth and I've really started to drink the kool aid.

What is NixOS?

At it's simplest, NixOS is a Linux distribution that aims to let you do system management / package management in a reliable, reproducible and declarative way built atop the Nix package manager (which can be installed independantly in a plethora of operating systems).

It enables you to do all your OS management, package management and configuration in the Nix programming language (which again, you can read more about here—Whats more: every time you make a change to your NixOS configuration and tell NixOS to rebuild your system, the exact specifications of your current system aren't just overwritten but instead a new 'generation' is created and you can switch between these versions of your operating system setup as you please. This means that if you somehow happen to break your environment you can just roll back to how it was before.

When jumping to NixOS, I made the mistake of setting up my configuration such that I could do what I usually do on UNIX-like operating systems: install a bunch of packages to enable me to compile whatever tools I need for work, and while this worked for awhile, for a few things, it wasn't the optimal way of doing things on NixOS. I spend some time looking the the Nix language and some NixOS configurations from other people around the internet and I slowly pieced together a configuration that worked for me. Besides just working for me: I can take my configuration and put it on any machine I own and run nixos-rebuild switch and boom, my entire operating system is set up exactly how I like it.

My NixOS configuration has come a long, long way since the first commit (documentation for Nix is pretty sparse) and I've learned a bunch of things that I feel would be useful to share, so take a look at my NixOS configuration if you'd like a decent, modular starting point and I'll outline a few things below.

My configuration

Modular configuration

A lot of NixOS configurations I've looked at essentially just look like one absolutely huge configuration.nix file; this is an ok way to do things but it can definitely get a bit unwieldy very easily. One of the really cool things about the Nix language is that it is a programming language. I don't leverage anything particularly complicated in my setup, but one nice thing is that you can define modules and import them in a top level configuration.nix module.

My NixOS configuration has the top level configuration.nix and hardware-configuration.nix files gitignored because installing NixOS will generate these for you anyway—because these files contain auto-generated nix expressions and are install/machine dependant (UEFI installation, GPU drivers, boot device etc) I don't think it makes sense to version these. Instead, my NixOS config is set up with the following three directories:

1. profiles/, which contains nix modules such as desktop.nix, laptop.nix, server.nix. These nix modules, when imported, set up things that I'd expect from that particular class of machine. If I import server.nix, I won't bother setting up any X11 related functionality because I won't need a GUI. If I import laptop.nix, I might automatically include packages such as powertop or enable bluetooth. In fact, my profiles/ directory also contains a base.nix in which I define a standard set of packages to install which for me I use on all machines, as well as configure my preferred timezone, keyboard layout and other random things.
2. modules/ which contains nix modules such as neovim.nix, dwm.nix, amd_gpu.nix. These are smaller nix modules which are set up to be enabled or disabled explicitly. My neovim.nix module not only downloads Neovim, but also downloads my dotfiles and configures Neovim exactly how I like it, plugins and all; my dwm.nix grabs my personal build of DWM, compiles it and installs it. Modules defined in modules/ can literally be anything that I might want to install but want to abstract away—installing GPU drivers on my workstation requires three different lines of configuration, so I'd rather just add modules.amd_gpu.enable = true; to my configuration instead. The idea there is that these modules can be easily enabled/disabled on a machine-machine basis.
3. machines/ which contains nix modules which acts as configuration for particular machines of mine. These modules will do things such as set the hostname of a machine, as well as include any profiles/*.nix and modules/*nix files neccessary for that particular machine. The VPS running my site has a HTTP server, SSL certificate generation and related things configured in machines/cbailey.co.uk.nix for example 😄

The main upside to this kind of configuration is that I can share configuration between multiple machines I want to manage in a DRY fashion. My girlfriend also recently installed NixOS and since she's not too familiar with Linux, it's extremely convinient that she can use my NixOS configuration; swapping out say, modules/dwm.nix for modules/plasma5.nix and I can contribute changes that solve technical difficulties she might be having.

Writing a Nix module

I won't go too deeply into the details of the Nix language (I wouldn't call myself an authority on this, I picked up what I know from other configs and playing around), but there are a few nice features/quirks of Nix which seemed you might not expect, which can be leveraged in your modules.

Configuration merging

Nix modules are essentially just attribute sets (basically a map/JSON object like thing) containing keys and values. There are a few different sections to a nix module, but the main bit that is important for NixOS configuration is that an attribute set has a value set to the key config, like this:

{ config, lib, pkgs, ... }:
{
  config = {
   something = true;
  }
}

When this module is imported by your top level configuration.nix, the key config gets merged with your configuration.nix's configuration.

We can utilise this to define a modules/vmware_guest.nix module for instance:

{ config, lib, pkgs, ... }:
{
  virtualisation.vmware.guest.enable = true;
}

When this module gets imported to your top level configuration.nix, virtualisation.vmware.guest.enable = true gets added to your configuration. This is pretty cool because you can utilise this to break your configuration up into smaller chunks.

The merging that nix does is a deep merge as well, so if you have services.xserver.enable = true; in your top level configuration and want to import multiple window managers or desktop environments, these window managers and desktop environments not only can set values nested inside services.xserver, but these values can be used in the top level configuration.nix as well.

If your top level configuration.nix has a list of packages to install; but you also include another nix module that defines a seperate list of packages to install:

# configuration.nix
environment.systemPackages = with pkgs; [
  firefox
];

# some_module.nix
environment.systemPackages = with pkgs; [
  wget
  unzip
  git
];

Then these lists get concatenated and merged together as well, installing all the packages define across all modules imported. This seemed pretty unintuitive but it's pretty cool!

Enabling/disabling modules

Alongside the config key, you can also create options of varying types in your module's options key. Actually, all nix packages you install are defined with modules, and they use this options key to define what configuration options can be set at all!

In my modules/ directory, I have a convention where all of my modules define a modules.<name>.enable boolean option for instance, so I can explicitly enable or disable these modules regardless of whether or not they're imported.

You get access to a bunch of types such as paths, strings, integers etc; pretty much what you'd expect. I'm not sure where exactly this is documented since I've only really needed to have simple boolean type options but reading other nixpkgs was a decent and quick way to learn.

You can define a modules.<name>.enable options like I do with the following boilerplate:

{  config, lib, pkgs, ...}:
with lib;
  let
    cfg = config.modules.NAME;
  in
  {
    options.modules.NAME = {
      enable = mkOption {
        type = types.bool;
        default = false;
        description = ''
          Some documentation here
        '';
      };
    };
  
    config = mkIf cfg.enable (mkMerge [{
      # my optional configuration here
    }]);
  }

Overriding packages

NixOS and Nix (the language) can be pretty intimidating at times. The documentation around Nix/NixOS is pretty sparse, potentially outdated and dense. I got a lot of help from the related IRC channels though and definitely suggest you seek help there since it's one of the friendliest and most helpful communities I've had the pleasure of interacting with!

One of the coolest features hidden behind the lack of documentation is the fact that you can override packages. This is especially useful if you want to install a standard package with a patch or small tweak; the standard library for Nix contains functions to fetch remote files from Git or just fetch a tarball from somewhere. I try not to overuse this feature though because it kind of clashes with the declarative-ness of the system otherwise; but for things like my DWM build, it's super helpful.

DWM is a window manager which is configured by recompiling it. When setting up NixOS, I didn't want to learn how to package my personal build of DWM before even getting to grips with the OS, so being able to override the existing DWM package but point at a different set of source code was super helpful. You can override a package with the following pattern:

nixpkgs.overlays = [
  (self: super: {
    dwm = super.dwm.overrideAttrs(_: {
      src = builtins.fetchGit {
        url = "https://github.com/vereis/dwm";
        rev = "b247aeb8e713ac5c644c404fa1384e05e0b8bc6f";
        ref = "master";
      };
    });
  })
];

Home-Manager

Home-Manager is an application which lets you configure your applications (basically a declarative nix-ish implementation of dotfiles). It's super useful and I've transitioned to using it for a bunch of the applications I use.

My terminal emulator's colors, default Firefox plugins and other things are configured via Home-Manager and thus are versioned in my NixOS generations and upgraded whenever I do a nixos-rebuild switch.

There isn't too much else to say about Home-Manager; it integrates well with Nix and it just works. Check it out if you're using NixOS.

Developer Environments

NixOS by default comes with a feature which lets you set up environments for different projects (similar to how ASDF might work for your Erlang/Elixir projects).

Essentially, you can create a shell.nix file in a directory that contains a list of applications you want to be available in that directory and activate it by invoking nix-shell. You can see an example shell.nix set up for a Phoenix application below:

let
  pkgs = import <nixpkgs> {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.elixir_1_10
    pkgs.nodejs-10_x
    pkgs.yarn
  ];
}

If you're using direnv (a really nice application which lets you automatically set environment variables when you enter a directory), you can use a cool project called lorri which sets up a daemon and automatically updates your local dev environment when you enter a directory (avoiding the need to run nix-shell manually) as well as automatically watching for changes in your shell.nix.

This emulates the flow of using asdf and defining a .tool-versions file pretty well, and I've actually found it even more useful because if someone has trouble compiling one of my applications, even if they don't use Nix, you can look at the shell.nix and tell them exactly what packages you need and at what versions.

Wrapping up

The more I play with Nix and NixOS, the more in love I fall with it. I've not had this much fun (and drunk this much kool-aid) since falling for OTP or discovering Arch Linux or Gentoo when I was just starting to get into Linux.

Hopefully the contents of this post will be helpful for other people looking to get into NixOS and I'm sure I'll have more to contribute w.r.t this as time goes on!

Thanks for reading 😄

edit: man configuration.nix gives you an exhaustive list of configuration options! happy tinkering!

Testing with tracing on the BEAM

When I was writing some tests for a event broadcasting system (the one I outlined in this blog post I found that, as expected, the unit tests worked very easily but when it came to testing that the entire system worked as intended end to end, it was a little more difficult because I'd be trying to test completely asynchronous and pretty decoupled behaviour.

Testing with Mock/:meck

At this point in time, the test suites inside our code base use mocks very extensively which might be a topic for another day, because I don't think I believe that mocking code in tests in the best—nor most idiomatic—approach. One nice bit about the de-facto mocking library for Elixir is that it also comes with a few assertions you can import into your test suites; namely assert_called/1.

Using this library, we could write our tests, mocking the event handling functions in the event listeners we're interested in for the particular test and then use assert_called/n to determine whether or not that function was called as expected and optionally you can provide what looks roughly like an Erlang match spec to assert that the function was called with specific arguments.

Our tests end up looking roughly like the following:

defmodule E2ETests.EctoBroadcasterTest do
  use ExUnit.Case
  alias EventListener.EctoHandler
  import Mock

  setup_with_mocks(
    [
      {EctoHandler, [:passthrough],
       handle_event: fn {_ecto_action, _entity_type, _data} ->
         {:ok, :mocked}
       end}
    ],
    _tags
  ) do
    :ok
  end

  test "Creating entity foo is handled by EctoHandler" do
    assert {:ok, %MyApp.Foo{}} = MyApp.Foos.create_foo(...)
    assert_called EctoHandler.handle_event({:insert, MyApp.Foo, :_})
  end
end

This approach basically met all of our requirements except for the fact that mocking via :meck (the de-facto Erlang mocking library and used internally by Mock) does things in a pretty heavy handed way.

At a high level, all :meck does is replace the module you're trying to mock with a slightly edited one where the functions you want to mock are inserted in place of the original implementations. This comes with three disadvantages for our particular use case:

1. Because we're essentially loading code into the BEAM VM, we can no longer run our tests asynchronously because we've patched code at the VM level and these patches are global
2. Because we're replacing the function we want to call; we can't actually test that that particular function behaves as expected; only that it was called
3. If we want to mock our functions purely for the assert_called/1 assertion (though this has annoyed us somewhat elsewhere too) we need to be careful and essentially re-implement any functionality that you actually expect to happen for that to be testable; which itself might lead itself to brittle mocks.

My main qualm was with the second problem; we ran into it pretty hard for a few edge cases (though we resolved the second case too!): say EctoHandler.handle_event({:delete, _, _}) needed to be implemented in a special way where it persists these actions into an ETS table and you want to assert that that actually happened, you'd be a little stuffed... image your source code looks like:

defmodule EventListener.EctoHandler do
  ...
  def handle_event(:delete, entity_type, data) do
    Logger.log("Got delete event for #{entity_type} at #{System.monotonic_time()}")
    data
    |> EventListener.DeletePool.write()
    |> process_delete_pool_response()
    |> case do
      {:ok, _} = response -> 
        response
      error ->
        add_to_retry_queue({:delete, entity_type, data})
        Logger.error("Failed to add to delete pool, adding to retry queue")
        error
    end
  end
  ...
end

To mock this function properly, you'd probably just have to implement your mock such that we essentially duplicate the implementation, but this of course will blow up when the mocks come out of sync with the implementation.

A less heavy handed approach

One of the coolest but possibly least used features of the BEAM is the built in ability to do tracing.

Tracing essentially lets you capture things such as messages being passed to particular processes, functions being called among other things. It does come with some disadvantages (i.e. if you're not careful you'll blow up your application because of the performance impact (be careful if you trace on a production deployment 😉)) and it's very low level but it works well and there exist several high level wrappers.

I wanted to experiment with tracing because I've never used it very much so I chose a pretty low level abstraction to do the work for me: the :dbg application which comes built into the normal Erlang OTP distribution.

I'm in the process of cleaning up the actual implementation, but if you wanted to build your own replacement for the assert_called/1 function (with a nice added feature) feel free to follow along!

You can start the :dbg application via the following:

def init() do
  # Flush :dbg in case it was already started prior to this
  :ok = :dbg.stop_clear()
  {:ok, _pid} = :dbg.start()

  # This is the important bit; by default :dbg will just log what gets traced on stdout
  # but we actually need to handle these traces because we want to capture that the
  # function was called programmatically rather than reading it off the shell/log file
  # This tells `:dbg` to call the `process_trace/2` function in the current running process
  # whenever we get a trace message
  {:ok, _pid} = :dbg.tracer(:process, {&process_trace/2, []})

  # And this sets up `:dbg` to handle function calls
  :dbg.p(:all, [:call])
end

Now that the tracer is started up; we need to tell it to start tracing certain functions. You can do this quite easily by passing an mfa into :dbg.tpl/4 as follows:

def trace(m, f, a) do
  # :dbg.tpl traces all function calls including private functions (which we needed for our use-case)
  # but I think it's a good default
  # 
  # The 4th parameter basically says to trace all invocations of the given mfa (`:_` means we pattern
  # match on everything), and we also pass in `{:return_trace}` to tell the tracer we want to receive
  # not just function invocation messages but also capture the return of those functions 
  :dbg.tpl(m, f, a, [{:_, [], [{:return_trace}]}])
end

Now provided a super simple stub function for process_trace/2, you should be able to see your traces come back looking all good:

# Using the following process_trace/2 implementation:
def process_trace({:trace, _pid, :call, {module, function, args}} = _message, _state) do
  IO.inspect(inspect({module, function, args}) <>" - call start")
end

def process_trace({:trace, _pid, :return_from, {module, function, arity}, return_value}, _state)
    when length(args) == arity do
  IO.inspect(inspect({module, function, arity}) <>" - returns: " <> inspect(return_value))
end
-------------------------------------------------------------------------------------------------
iex(1)> Assertions.init()
{:ok, [{:matched, :nonode@nohost, 64}]}
iex(2)> Assertions.trace(Enum, :map, 2)
{:ok, [{:matched, :nonode@nohost, 1}, {:saved, 1}]}
iex(3)> Enum.map([1, 2, 3], & &1 + 2)
"{Enum, :map, [[1, 2, 3], #Function<44.33894411/1 in :erl_eval.expr/5>]} - call start"
"{Enum, :map, 2} - returns: [3, 4, 5]"
[3, 4, 5]

This gets us most of the way to what we want! We can see what functions were called and capture the variables passed into it; as well as capturing what the functions returned.

But if you look closely, the message for :call and :return_from have a few issues:

1. Ideally, I'd have preferred to have a single message rather than two, because we have some additional complexity here now...
2. Especially because the :return_from message doesn't get the args passed into it but instead only the function's arity... which means we need to try our best to marry them up as part of the process_trace function...
3. Just as a note; I don't think we can pattern match on lambdas easily on the BEAM (nor do things like send them over :rpc.calls so at least its consistent 🙂

We can solve the first two problems by amending the implementation of process_trace/2 quite easily though (it might not be a foolproof implementation but it worked well enough for our usecase that I'll just add the code here verbatim):

def process_trace({:trace, _pid, :call, {module, function, args}} = _message, state) do
  [{:call, {module, function, args}, :pending} | state]
end

def process_trace({:trace, _pid, :return_from, {module, function, arity}, return_value}, [
      {:call, {module, function, args}, :pending} | remaining_stack
    ])
    when length(args) == arity do
  IO.inspect(inspect({module, function, args}) <> " - returns: " <> inspect(return_value))

  remaining_stack
end
-------------------------------------------------------------------------------------------------
iex(1)> Assertions.init()
{:ok, [{:matched, :nonode@nohost, 64}]}
iex(2)> Assertions.trace(Enum, :map, 2)
{:ok, [{:matched, :nonode@nohost, 1}, {:saved, 1}]}
iex(3)> Enum.map([1, 2, 3], & &1 + 2)
"{Enum, :map, [[1, 2, 3], #Function<44.33894411/1 in :erl_eval.expr/5>]} - returns: [3, 4, 5]"
[3, 4, 5]
iex(4)> Enum.map([[1, 2, 3], [:a, :b, :c], []], fn list -> Enum.map(list, &IO.inspect/1) end)
1
2
3
:a
"{Enum, :map, [[1, 2, 3], &IO.inspect/1]} - returns: [1, 2, 3]"
:b
:c
"{Enum, :map, [[:a, :b, :c], &IO.inspect/1]} - returns: [:a, :b, :c]"
"{Enum, :map, [[], &IO.inspect/1]} - returns: []"
[[1, 2, 3], [:a, :b, :c], []]

Because whatever gets returned from process_trace/2 accumulates into future calls of process_state/2 in the second parameter, we can essentially do the above and treat it like a stack of function calls. We only emit the actual IO.inspect we care about if we end up popping the function call (because it's resolved) off the stack.

All I do from here is persist these functions into an :ets table of type :bag that we create when init/0 is called, and have process_trace({_, _, :return_from, _}, _) write to it whenever a function call succeeds.

For convenience I really hackily threw together the following macros (please don't judge me 🤣 -- this is going to be cleaned up a lot before I throw it up on hex.pm):

defp apply_clause(
       {{:., _, [{:__aliases__, _, mod}, fun]}, _, args},
       match \\ {:_, [], Elixir}
     ) do
  module = ["Elixir" | mod] |> Enum.map(&to_string/1) |> Enum.join(".") |> String.to_atom()

  arg_pattern =
    args
    |> Enum.map(fn
      {:_, _loc, _meta} -> :_
      other -> other
    end)

  quote do
    case :ets.select(unquote(@mod), [
           {{:_,
             {unquote(module), unquote(fun), unquote(length(arg_pattern)), unquote(arg_pattern),
              :_}}, [], [:"$_"]}
         ]) do
      [] ->
        false

      matches ->
        Enum.any?(matches, fn
          {_timestamp, {_mod, _fun, _arity, _args, unquote(match)}} -> true
          _ -> false
        end)
    end
  end
end

defp capture_clause({{:., _, [{:__aliases__, _, mod}, fun]}, _, args}) do
  module = ["Elixir" | mod] |> Enum.map(&to_string/1) |> Enum.join(".") |> String.to_atom()

  arg_pattern =
    args
    |> Enum.map(fn
      {:_, _loc, _meta} -> :_
      other -> other
    end)

  quote do
    case :ets.select(unquote(@mod), [
           {{:_,
             {unquote(module), unquote(fun), unquote(length(arg_pattern)), unquote(arg_pattern),
              :_}}, [], [:"$_"]}
         ]) do
      [] ->
        false

      [{_timestamp, {_mod, _fun, _arity, _args, val}} | _] ->
        val
    end
  end
end

defmacro called?({{:., _, _}, _, _} = ast) do
  apply_clause(ast)
end

defmacro called?({op, _, [lhs, {{:., _, _}, _, _} = rhs]}) when op in [:=, :"::"] do
  apply_clause(rhs, lhs)
end

defmacro called?({op, _, [{{:., _, _}, _, _} = lhs, rhs]}) when op in [:=, :"::"] do
  apply_clause(lhs, rhs)
end

defmacro capture({op, _, [lhs, {{:., _, _}, _, _} = rhs]}) when op in [:=, :<-] do
  captured_function = capture_clause(rhs)

  quote do
    var!(unquote(lhs)) = unquote(captured_function)
  end
end

Which lets us finally put it all together and do the following 🎉

iex(1)> require Assertions
Assertions
iex(2)> import Assertions
Assertions
iex(3)> Assertions.init()
{:ok, [{:matched, :nonode@nohost, 64}]}
iex(4)> Assertions.trace(Enum, :map, 2)
{:ok, [{:matched, :nonode@nohost, 1}, {:saved, 1}]}
iex(5)> called? Enum.map([:hi, "world!"], _) = [:atom, :string]
false
iex(6)> Enum.map([:hi, "world!"], & (if is_atom(&1), do: :atom, else: :string))
[:atom, :string]
iex(7)> called? Enum.map([:hi, "world!"], _) = [:atom, :string]
true
iex(8)> called? Enum.map([:hi, "world!"], _) = [_, :atom]
false
iex(9)> called? Enum.map([:hi, "world!"], _) = [_, _]
true
iex(10)> called? Enum.map([:hi, "world!"], _)
true
iex(11)> capture some_variable <- Enum.map([:hi, "world!"], _)
[:atom, :string]
iex(12)> [:integer | some_variable]
[:integer, :atom, :string]

So as you can see; these macros give us some nice ways to solve two of the three issues we had with Mock:

1. We can assert that functions were called (passing in a real pattern match v.s. needing to pass in a match spec like DSL)
2. And we can assert that the functions were called and returned a particular value which again can be any Elixir pattern match
3. And as a bonus, you can bind whatever some function returned to a variable with my capture x <- function macro! 🤯

The remaining steps for this is to add a before_compile step to automatically parse invocations of called? and capture so that I can automate the Assertions.trace/3 function to make it completely transparent (it'll automatically trace the functions you care about) as well as extensively test some edge cases because we've run into a few already.

I'll also be looking into massively cleaning up these macros since they were thrown together literally in a couple of hours playing around with the ASTs I thought to test but I hope this blog post helped as an intro into tracing on the BEAM but also just a fun notepad of my iterative problem solving process 🙂

Taming my Vim configuration

As a long-time Vim user, I've put a lot of time into my Vim configuration over the years. It is something that has organically grown as my editing habits and needs change over time.

Recently, when trying to figure out why certain things weren't working the way I expected, I realised that there was much in my config I either didn't need or just blatantly did not understand. I took this as an opportunity to remove all the old cruft and start my configuration file again from scratch, taking only what I absolutely needed.

My configuration file was also around 2000 lines in length. Organisation was a huge mess: related commands strewn around all over the place due to years of ad-hoc editing. During my refactor of my configuration, one key goal was also to clean this up.

When ranting to a colleague of mine about my configuration woes, he showed me his, and I was struck by inspiration 👼 A means of micromanaging—architecting—Vim configuration to make it saner, compositional, and more optimal too.

A few key ideas I've used in my configuration are outlined below: hopefully, they'll help anyone else also looking to tame their Vim configuration.

Initial refactoring

The first thing I did when breaking apart my 2000 line configuration file was to untangle the organisational mess I've made.

I started this by moving all the configuration related to specific ideas (i.e. tabbing: tabstop, softtabstop, expandtab) into contiguous blocks like the following:

" Cache undos in a file, disable other backup settings
set noswapfile
set nobackup
set undofile

" Better searching functionality
set showmatch
set incsearch
set hlsearch
set ignorecase
set smartcase

" Search related settings
set showmatch
set incsearch
set hlsearch
set ignorecase
set smartcase

This let me tell at a glance what essential "modules" I was dealing with. These "modules" would be blocks of configuration concerning both related settings and configuration for certain plugins. After doing this, my Vim configuration was actually larger, but at least it was easier to read 😅

Dealing with Plugins

I personally use vim-plug for managing my Vim plugins. Some of the reasons I use vim-plug include features such as the minimal amount of boilerplate needed to get plugins working, as well as hooks for lazy loading: we'll get to this later.

The only requirement to use vim-plug is to install it as per its installation instructions, followed by adding the following block to the top of your Vim configuration:

call plug#begin('~/.vim/my_plugin_directory_of_choice')

Plug 'https://github.com/some_repo/example.git'
Plug 'scrooloose/nerdtree'

call plug#end()

Most Vim plugins will have installation instructions for if you're using vim-plug (and if you're not, I'm sure you know how to configure this bit already anyway!), but that's pretty much it. Running :plug-install will fetch all of the configured plugins and download them to the path you give into the call plug#begin(...) option. Once this is done, you're basically good to go.

Because vim-plug is just standard viml, I use the same "module" idea and segregate groups of similar/related plugins together, as follows.

call plug#begin('MY_CONFIG_PATH/bundle')

" General stuff
Plug 'scrooloose/nerdtree'
Plug 'jistr/vim-nerdtree-tabs'
Plug 'neoclide/coc.nvim', {'branch': 'release'}
Plug 'easymotion/vim-easymotion'
Plug 'majutsushi/tagbar'
Plug 'ludovicchabant/vim-gutentags'
Plug 'tpope/vim-surround'
Plug 'tpope/vim-repeat'
Plug 'tpope/vim-eunuch'
Plug 'machakann/vim-swap'

" Setup fzf
Plug '/usr/local/opt/fzf'
Plug '~/.fzf'
Plug 'junegunn/fzf.vim'

" Git plugins
Plug 'Xuyuanp/nerdtree-git-plugin'
Plug 'airblade/vim-gitgutter'
Plug 'tpope/vim-fugitive'

" Erlang plugins
Plug 'vim-erlang/vim-erlang-tags'         , { 'for': 'erlang' }
Plug 'vim-erlang/vim-erlang-omnicomplete' , { 'for': 'erlang' }
Plug 'vim-erlang/vim-erlang-compiler'     , { 'for': 'erlang' }
Plug 'vim-erlang/vim-erlang-runtime'      , { 'for': 'erlang' }

" Elixir plugins
Plug 'elixir-editors/vim-elixir' , { 'for': 'elixir' }
Plug 'mhinz/vim-mix-format'      , { 'for': 'elixir' }

" Handlebar Templates
Plug 'mustache/vim-mustache-handlebars' , { 'for': 'html' }

" Bunch of nice themes
Plug 'flazz/vim-colorschemes'

call plug#end()

Here we can see another killing feature of vim-plug: by specifying options such as { 'for': 'erlang' }, I'm instructing that that particular plugin should only be loaded if the filetype of the currently opened file is erlang. A 2000 line config file isn't necessarily huge by hardcore Vim standards, but even at 2000 lines, my Vim configuration took a second to load, which was distracting. You can use many different hooks to optimise your vim-plug loading time, so I suggest you consult their README for more information; for what it's worth, I get by simply with this { 'for': $filetype } directive.

Turning comments into real "modules"

Now that everything is super well organised, I did a second refactoring pass over my configuration. Since all my related commands were already in logical blocks, I further broke them out into their own files so that I wouldn't need to have the informational overhead in my brain whenever I wanted to change a trivial setting.

The cool thing is that viml is a Turing complete programming language, able to do pretty much anything a normal programming language can do. One thing I'm abusing is the idea of automatically sourcing files from elsewhere, essentially breaking out my logically separated comments into real, logical modules.

I do this via a very naive for ... in ... loop as follows:

for config in split(glob('MY_CONFIG_PATH/config/*.vim'), '\n')
  exe 'source' config
endfor

This basically tells vim to look for all .vim files inside my configured config/ directory and source them. Therefore, I end up organising my configuration modules as follows:

MY_CONFIG_PATH/config
├── coc_config.vim
├── ctag_config.vim
├── easymotion_config.vim
├── editor_config.vim
├── elixir_config.vim
├── erlang_config.vim
├── fzf_config.vim
├── jsonc_config.vim
└── nerdtree_config.vim

In the future as this grows, I might even split this up to house modules into different domains (i.e. MY_CONFIG_PATH/plugins/*.vim versus MY_CONFIG_PATH/languages/*.vim versus plain old MY_CONFIG_PATH/tab_settings.vim). In the meantime, however, this level of encapsulation is more than enough: editor_config.vim is where I keep all my editor commands such as tab stop settings; everything else is either plugin configuration (all my settings related to, say, fzf). Otherwise, files such as elixit_config.vim are configuration files that get loaded when I'm editing elixir files specifically.

Plugin Configuration

My nerdtree.vim plugin configuration file is listed below, but essentially all of these files (except language configuration files) look the same:

" NERDTREE toggle (normal mode)
nnoremap <C-n> :NERDTreeToggle<CR>

" Close VIM if NERDTREE is only thing open
autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTree") && b:NERDTree.isTabTree()) | q | endif

" Show hidden files by default
let NERDTreeShowHidden=1

" If more than one window and previous buffer was NERDTree, go back to it.
autocmd BufEnter * if bufname('#') =~# "^NERD_tree_" && winnr('$') > 1 | b# | endif

" Selecting a file closes NERDTREE
let g:NERDTreeQuitOnOpen = 1

" Style
let NERDTreeMinimalUI = 1
let NERDTreeDirArrows = 1

Since the plugins can be set to be lazily loaded, I don't bother with any further tinkering with these files. They're literally just plain viml that is sourced by my top-level init.vim or .vimrc.

Language configuration

Language configuration is likewise simple but a little different. I actually end up writing custom viml functions for overriding configuration, which might be set elsewhere.

As an example, I've included a minimal copy of my elixir.vim file below:

function! LoadElixirSettings()
  set tabstop=2
  set softtabstop=2
  set shiftwidth=2
  set textwidth=80
  set expandtab
  set autoindent
  set fileformat=unix
endfunction

au BufNewFile,BufRead *.ex,*.exs,*.eex call LoadElixirSettings()

" Mix format on save
let g:mix_format_on_save = 1

As you can see, I basically do two different things:

1. I set a few options specific to plugins that might load for filetypes marked elixir, i.e. let g:mix_formation_on_save is a configuration option for a plugin I load in my top-level init.vim or .vimrc via vim-plug

2. Settings that are sourced in other files (such as tabstop or autoindent) can't be guaranteed to be sourced in any particular order, because again, that's done via a naive for ... in ... loop. To ensure that these settings override the default settings I've defined, I set up an autogroup: something that automatically executes based on some conditions. I'm telling vim that when the buffer is editing a file ending in .ex, .exs, .eex, then run the function I've called—this, in turn, overrides the existing settings with no issues 💪

Conclusion

I find that structuring my Vim configuration this way is super helpful in cutting down the amount of mental overhead I have when trying to edit specific settings. All of my configuration is separated and logically grouped, and when I decide I no longer use a given plugin, I can delete its associated configuration file.

I hope this helps, and if you're interested, you can find my personal vim configuration here. This repo changes often and grows over time, so what you find there might not be 100% the same approach as I've outlined here, but that means I've improved and streamlined it even more! 💪

As an aside, I personally opt to use Neovim over Vim because of asynchronous plugin support.

I originally had a section covering how to convert an existing Neovim configuration to be loadable by Vim (such as to ignore any differences and to enable Vim users to make meaningful use of this guide), but this was removed simply because even if the raw configuration can be made to work for both editors, plugins cannot.

Following this realisation, I've edited this article to be completely editor agnostic. I apologise for any confusion early drafts of this article might have caused!

Refining Ecto Query Composition

This post is a follow-up to the somewhat popular Ecto Query Patterns post I wrote over a year ago, so if you haven't read that, I definitely recommend skimming over it just to set the groundwork for this post.

In that post, I detailed my preferred way of breaking down Ecto Queries into small, reusable functions for three major reasons: minimising code duplication, isolating where Ecto queries live and are built in your codebase, and increasing the consistency of query patterns in your modules.

Despite these goals, when implementing this idea more fully in Vetspire's codebase, we still ran into a few pain points and oversights which we've attempted to address since the publishing of the original post—something I hope to share with you all now.

Pain Points & Oversights

One of the main issues with the composable Ecto Query pattern is that while we've reduced code duplication by preventing you from writing adhoc Ecto Queries outside of your Ecto Schemas, but it definitely doesn't go as far as it could. For example, if we have multiple functions dealing with Users in your context, your functions probably end up duplicating a lot of logic in your pipelines like so:

defmodule MyApp.Accounts do
  alias MyApp.Accounts.User
  alias MyApp.Repo

  def count_users(org_id) do
    User
    |> User.where_not_deleted()
    |> User.where_org_id(org_id)
    |> Repo.aggregate(:count)
  end

  def list_users(org_id) do
    User
    |> User.where_not_deleted()
    |> User.where_org_id(org_id)
    |> Repo.all()
  end
  
  def get_user(org_id, user_id) do
    User
    |> User.where_not_deleted()
    |> User.where_org_id(org_id)
    |> Repo.get_by(id: user_id)
  end
end

While this evidently has a lot of advantages over duplicating the raw Ecto Queries in all three functions (if only for IDE completion, being able to write multiple clauses per function, etc), we've still got a lot of rampant duplication. We still have to remember to add the User.where_not_delete/1 call to each pipeline, not doing so could easily lead to bugs; We potentially have to add nil pattern clauses to each composable function if we wanted to allow users to opt-out of passing parameters; We either have to duplicate built-in Ecto business logic (implementing a User.where_id/2 versus using Repo.get_by/2) or accept having multiple methods of querying data in our functions; etc.

Additionally, while I touched briefly on being able to reuse any defined composable query functions as part of your application's Dataloader sources, or leveraging named bindings to more easily allow cross-schema query composition for joins/preloads, the solution still forces you to build queries in places outside of your schema and contexts.

This might be fine in the preload/join case if your schemas are related and coupled under a single context (i.e. MyApp.Accounts.User might build a preload query using functions defined in MyApp.Accounts.PhoneNumer of vice versa), doing so in the case where you're reaching between contexts (MyApp.Billing.Invoice building queries reaching into MyApp.Accounts.User) or via Dataloader feels a lot less clean.

While I don't think we necessarily want to avoid reaching into other schemas, especially since we want to leverage pattern matching on structs and type definitions across our codebase, we do want to avoid exposing and using the low-level details too much. I'd definitely put the scale of query fragment we usually compose as low-level details... so being able to avoid doing this as much as possible is a nice incentive to revisit how we do things!

Lastly, we end up creating an ad-hoc, informally specified DSL that barely wraps the underlying functionality. It was questioned in our team whether or not the compositional query pattern was quite the right level of abstraction for what we were trying to do.

As we were evaluating how our compositional query rollout was going, we decided as a team to double down and try to maximise how encapsulated and reusable our queries would be. The direction we ended up going very much embraces the DSL/pattern we were trying to enforce, so maybe this won't be as universally applicable in other applications, but it's definitely something I think applies to the vast majority of apps which utilise Ecto/Phoenix/Absinthe. For other specific use cases (i.e. building ETL pipelines handling very polymorphic data) it might still be beneficial to write very composable Ecto Queries, but as said, for most things, this approach simplifies and ties everything together a lot neater.

Metaprogramming Elixir

What's the solution you ask? The answer with Elixir is almost always (regrettably) metaprogramming!

The high-level idea is that we didn't want to build a pattern or abstraction for query composition; we wanted to build an abstraction to allow us to query data. This small shift in thinking led us down the metaprogramming rabbit hole: in short, entities in our codebase needed to be queryable. This queryability needed to be implemented in a generic but consistent way and needed to be trivially extendable on a schema-by-schema basis.

We took this thinking and started hacking something together quite literally: we decided that all schemas which wanted to be queryable needed to implement a new behaviour and some callbacks for consistency while retaining customisability, and that schemas implementing said behaviour should expose a very simple API to hide query details from the caller.

When looking at the bulk of our code which was already written with compositional query patterns, we saw that the most common pattern for allowing very customisable filtering was the Enum.reduce/3 pattern outlined in the original blog post. We decided that this would be a very flexible base for our behaviour callback; we intended to do something as follows:

def MyApp.Queryable do
  @callback query(opts :: Keyword.t()) :: Ecto.Queryable.t()
end

def MyApp.Accounts.User do
  use Ecto.Schema
  import Ecto.Query

  alias MyApp.Entities.Org
  alias __MODULE__

  schema do
    field :id, :integer
    field :name, :string
    field :password, :string
    field :deleted, :boolean

    belongs_to :org, Org
  end

  def changeset(%User{} = user, attrs) do
    cast(user, attrs, [:name, :password, :boolean, :org_id])
  end

  @impl MyApp.Queryable
  def query(opts) do
    Enum.reduce(opts, (from x in User, as: :user), fn
      {:id, id}, query when is_integer(id) ->
        from [user: user] in query, where user.id == ^id

      {:org_id, org_id}, query when is_integer(org_id) ->
        from [user: user] in query, where user.org_id == ^org_id
        
      {:name, name}, query when is_binary(name) ->
        from [user: user] in query, where user.name == ^name

      {:deleted, true}, query ->
        from [user: user] in query, where not user.deleted
    end)
  end
end

At a super basic level, this minimal implementation allows us to replace all of the examples in the context snippet above like so:

defmodule MyApp.Accounts do
  alias MyApp.Accounts.User
  alias MyApp.Repo

  def count_users(org_id), do:
    Repo.aggregate(User.query(org_id: org_id, deleted: true), :count)

  def list_users(org_id), do:
    Repo.all(User.query(org_id: org_id, deleted: true))
  
  def get_user(org_id, user_id), do:
    Repo.one(User.query(id: user_id, org_id: org_id, deleted: true))
end

Which is definitely a little terser and reduces the number of lines of code duplication we had. One of the big tradeoffs this makes is losing IDE autocompletion as a perk; which for Vetspire's use case, was acceptable. In lieu of having IDE auto-completion, we only have one function we need to remember to use (User.query/1, Org.query/1, etc), and we can still catch bad key errors by using Dialyzer.

A Little Extensibility

Additionally, a lot of schemas in Vetspire happen to utilise soft deletion rather than hard deletion. In 99% of cases, we never want to query the deleted entities from the database. For cases like this, we can extend out MyApp.Queryable behaviour by adding a new @callback base_query :: Ecto.Queryable.t() which could be implemented thusly:

@impl MyApp.Queryable
def base_query, do:
  from x in User, as: :user, where: not x.deleted

@impl MyApp.Queryable
def query(base_query \\ base_query(), opts) do
  Enum.reduce(opts, base_query, fn
    {:id, id}, query when is_integer(id) ->
      from [user: user] in query, where user.id == ^id

    {:org_id, org_id}, query when is_integer(org_id) ->
      from [user: user] in query, where user.org_id == ^org_id
      
    {:name, name}, query when is_binary(name) ->
      from [user: user] in query, where user.name == ^name
  end)
end

Which lets us easily enforce base queries always contain filters that we want in most cases, while still allowing callers to opt out of it by passing in a different base query as a first parameter to the query/2 callback.

Consistency is Key

Do note that the API for MySchema.query/2 is very similar to the API provided by Ecto.Repo.get_by/2. This, of course, is not unintentional.

One of the issues with the compositional query function pattern was that each schema would end up implementing and defining very low-level wrappers over simple Ecto Query clauses; increasing the function surface area of your app's API, and introducing a lot of room for inconsistency in function names/parameter handling semantics to sneak in.

Arguably, the Ecto built-in callback avoids this by tightly coupling supported parameters/filters to your schema's defined fields. In fact, this is something we can trivially do too, to avoid people from needing to redefine benign filters over and over again across your codebase. Instead, we should just have to call out filters which are tightly coupled to your schema; anything generic should be provided out of the box so as to minimise the amount of thinking engineers need to do.

Since our Queryable behaviour relies on implementing a query/2 callback and enumerating filters in an Enum.reduce/3 callback, we need to write a function which takes some pattern {key :: atom(), value :: any()}, Ecto.Queryable.t() and determines if the key is known in the given schema, and if it is known, automatically apply filtering.

We ended up implementing something like the following to achieve this:

def MyApp.Queryable do
  import Ecto.Query

  @callback base_query :: Ecto.Queryable.t()
  @callback query(base_query :: Ecto.Queryable.t(), opts :: Keyword.t()) :: Ecto.Queryable.t()

  @spec apply_filter(Ecto.Queryable.t(), field :: atom(), value :: any()) :: Ecto.Queryable.t()
  def apply_filter(query, :preload, value) do
    from(x in query, preload: ^value)
  end

  def apply_filter(query, :limit, value) do
    from(x in query, limit: ^value)
  end

  def apply_filter(query, :offset, value) do
    from(x in query, offset: ^value)
  end

  def apply_filter(query, :order_by, value) when is_list(value) do
    from(x in query, order_by: ^value)
  end

  def apply_filter(query, :order_by, {direction, value}) do
    from(x in query, order_by: [{^direction, ^value}])
  end

  def apply_filter(query, :order_by, value) do
    from(x in query, order_by: [{:desc, ^value}])
  end

  def apply_filter(query, field, value) when is_list(value) do
    from(x in query, where: field(x, ^field) in ^value)
  end

  def apply_filter(query, field, value) do
    from(x in query, where: field(x, ^field) == ^value)
  end
end

@impl MyApp.Queryable
def query(base_query \\ base_query(), opts) do
  Enum.reduce(opts, base_query, fn {key, value}, query ->
    Queryable.apply_filter(query, key, value)
  end)
end

This not only minimises the number of clauses we need to write, this means that we can support generic Ecto-like semantics such as preloading, naive-paginating, sorting, etc. Engineers are still able to specify more complex filters, or override filters entirely by implementing their own reduce clauses above the apply_filter/3 call.

Solving Reusability

Now that we've extended our Queryable behaviour to be way more generic, we've come to a pretty foundational crossroads. Not only have we written a relatively small amount of code, we've got a very small API surface area. Any function that wants to query, join, or further compose other queries simply needs to know two things: one - the schema we want to join/query/filter/compose on, and two - the parameters which need to be passed into said schema's query/2 callback.

This lends itself well for, well, the intended purpose. To reiterate; one of the problems we didn't quite manage to solve with the previous compositional query pattern was preventing low-level query details from leaking out into adjacent contexts/domains. While we could somewhat mitigate this for our core contexts, Dataloader in particular proved a genuine code-duplication challenge.

With the old composition pattern, we would be forced to implement some Dataloader source roughly as follows:

defmodule MyAppWeb.Loader do
  def query(MyApp.Billing.Payment = query, args) do
    limit = Map.get(args, :limit)
    offset = Map.get(args, :offset)
    sort = Map.get(args, :sort, :asc)
    ...

    MyApp.Billing.Payment
    |> MyApp.Billing.Payment.paginate(limit, offset)
    |> MyApp.Billing.Payment.sort(sort)
    |> ...
  end
end

This effectively means that we're duplicating all our context functions between the core contexts defined within MyApp and MyAppWeb. Not just that; either you split MyAppWeb.Loader into various submodules with delegated functions or dispatched function calls (thus even duplicating the layout of your core contexts...), or you build one massive god module which is a problem in and of itself.

Without even really trying; we've solved this code duplication problem using our Queryable behaviour; if a given schema is defined as queryable, then we know that it implements the query/2 callback. If we know this, then we can dynamically dispatch to it without needing ever to write additional code.

The only thing holding us back from easily doing this is programmatically determining whether or not a module implements our behaviour. There are two easy ways of doing this; behaviour reflection and function reflection.

In the BEAM, behaviour metadata is compiled into the bytecode which is loaded by the BEAM when modules are loaded. This metadata can be reflected upon like so:

@spec implements_behaviour?(module :: atom()) :: boolean()
def implements_behaviour?(module) do
  behaviours =
    module.module_info(:attributes)
    |> Enum.filter(&match?({:behaviour, _behaviours}, &1))
    |> Enum.map(&elem(&1, 1))
    |> List.flatten()

  __MODULE__ in behaviours
end

Alternatively, you can check if the given schema exposes a function of the correct name and arity via function_exported?(module, function, arity). The former is probably more semantically correct and won't return any outliers, but I'm not really sure what's preferable. In fact, the Vetspire codebase does both things at times... we definitely have some more cleaning up to do!

Regardless, for the sake of this example, it doesn't particularly matter; so I'll go with the latter simply because the result is slightly terser:

defmodule MyApp.Queryable do
  import Ecto.Query

  @callback base_query :: Ecto.Queryable.t()
  @callback query(base_query :: Ecto.Queryable.t(), opts :: Keyword.t()) :: Ecto.Queryable.t()

  ...

  @spec implemented_by(module :: atom()) :: boolean()
  def implemented_by(module), do: function_exported?(module, :query, 2)
end

defmodule MyAppWeb.Loader do
  def query(module, args) do
    if MyApp.Queryable.implemented_by(module) do
      module.query(args)
    else
      # Alternatively, fall back to some `do_query/2` function to support both
      # queryable schemas and the compositional query pattern where it is advantageous.
      raise "Dataloader relations must implement the `MyApp.Queryable` behaviour."
    end
  end
end

Of course, this is much easier to do if your Absinthe schema's filtering options are relatively similar to your Ecto schema's fields. However, of course, if this is not the case you can always still supplement and extend the query/2 callback yourself; though you literally have to write no code if things align in your favor!

Finishing Touches

Looking over where we've gotten to; we've effectively reduced our API surface area with respect to code duplication to pretty much a single function call; we've taken a stab at minimising room for error and inconsistency in one's implementation, and we've got something that can be minimally used across contexts without leaking too many implementation details. I think that's pretty impressive for how little code we've written here.

The main win has been taking a step back from the original level of abstraction and building something that abstracts over "queryability" rather tha "query composition". Is there anything we can do to further improve the UX and make things even nicer or more automatic? Perhaps.

One of the main learnings of the Elixir community is to not reach for macros unneccessarily, and this is advice I generally agree with. However, for building developer tools, frameworks, and for increasing DX, I do think that macros are an awfully tempting tool to use, especially if one doesn't overly hide the complexities of what you're building.

At Vetspire, we've coupled the above approach to making our schemas queryable with a few other conveniences via a very minor use macro as follows:

defmodule MyApp.Queryable do
  defmacro __using__(_opts \\ []) do
    quote do
      use Ecto.Schema

      import Ecto.Changeset
      import Ecto.Query

      import unquote(__MODULE__)

      @behaviour unquote(__MODULE__)
      
      @impl unquote(__MODULE__)
      def base_query, do: from x in __MODULE__, as: :self

      @impl unquote(__MODULE__)
      def query(base_query \\ base_query(), filters) do
        Enum.reduce(filters, base_query, fn {field, value}, query ->
          apply_filter(query, field, value)
        end)
      end

      defoverridable query: 1, query: 2, base_query: 0
    end
  end

  @optional_callbacks base_query: 0
  @callback query(base_query :: Ecto.Queryable.t(), Keyword.t()) :: Ecto.Queryable.t()

  @spec apply_filter(Ecto.Queryable.t(), field :: atom(), value :: any()) :: Ecto.Queryable.t()
  def apply_filter(query, :preload, value) do
    from(x in query, preload: ^value)
  end

  def apply_filter(query, :limit, value) do
    from(x in query, limit: ^value)
  end

  def apply_filter(query, :offset, value) do
    from(x in query, offset: ^value)
  end

  def apply_filter(query, :order_by, value) when is_list(value) do
    from(x in query, order_by: ^value)
  end

  def apply_filter(query, :order_by, {direction, value}) do
    from(x in query, order_by: [{^direction, ^value}])
  end

  def apply_filter(query, :order_by, value) do
    from(x in query, order_by: [{:desc, ^value}])
  end

  def apply_filter(query, field, value) when is_list(value) do
    from(x in query, where: field(x, ^field) in ^value)
  end

  def apply_filter(query, field, value) do
    from(x in query, where: field(x, ^field) == ^value)
  end
end

This means that we can swap any use Ecto.Schema with use MyApp.Queryable and we'll end up importing every we need to define a schema, define our changeset, and define our queryable behaviour. Since we inject default implementations for base_query/0 and query/2, the majority of our simple schemas don't need to define any other code to make use of everything we've built. Any schemas that do require extension only need you to implement whatever callbacks you explicitly need to extend; everything else should work out of the box!

Conclusion

This leaves us at an interesting place: I feel we're doing well with respect to our Ecto Queries; we've largely eliminated duplication and solved a few problems (though admittedly, making some tradeoffs along the way).

The next step we could take is extending our MyApp.Queryable module to not only give us better query semantics but also to improve a bunch of schema/changeset stuff which always grinds my gears with vanilla Ecto. We've taken some steps to solve at Vetspire, and as that matures, I'm sure I'll write up a related follow-up to this post!

Another interesting direction to expand would be to take this idea one further: if we're going as far as to couple our queryable behaviour with Absinthe schema definitions, how easily could we extend Ecto schemas to automatically generate Absinthe schemas/queries/mutations? I'm sure something like this would both be super fun to do and valuable too!

Until then, however, I'd love to hear what the Elixir community thinks about this abstraction—I'm aware this isn't the most mindblowing thing in the world, but its a much-appreciated small step in the right direction for DX and discoverability in my own code, and where we've embraced it in Vetspire, we've definitely eased a lot of low level, repetitive code writing.

Happy hacking, and thanks for reading!

Compositional Elixir—Ecto Query Patterns

When I was a consultant, one of the opportunities I was most grateful for was the chance to get myself knee's deep in a dizzying variety of Elixir codebases. A direct result of this led me to hone in on a set of Elixir patterns and conventions that I end up pulling into just about every single new project I get my hands on.

I've written high-level posts about patterns I particularly like, such as railway–oriented programming and ways to make your Elixir code more composable, but today I felt like going on a little bit of a deep-dive into getting the most out of one of the most common Elixir libraries in common usage: Ecto!

ELI5—Ecto

Obviously, for a much more fully-featured introduction to Ecto, I recommend jumping right in and reading the official docs/guide as well as Programming Ecto, but a brief and abridged overview I can do.

Ecto is effectively the equivalent of an ORM that you'll find used in, or bundled with, popular web frameworks. While Ecto isn't strictly and ORM, you can think about it doing the same basic job: you define schemas and queries which allow you to read/write records stored in your database of choice (most commonly Postgres) using standard Elixir structs.

On a high level, Ecto is a collection of three different bits and pieces: Ecto.Schema, Ecto.Query, and Ecto.Changeset; providing you with the means of mapping between Elixir structs and your database schema, an extremely robust query builder, and data validation library respectively. Hopefully, that clears up the title of this post—today we'll be focusing on ways to make the most out of Ecto.Query specifically 💪

A Common Problem

One of the things I optimize for as a developer is separating my concerns whenever possible and creating boundaries/domains to isolate different "ideas" and "concepts" in whatever codebase I'm working on.

Unfortunately, on most projects I've worked on, Ecto code tends to be strewn all across the codebase with little regard for how/why it's being strewn about. When projects use libraries such as Absinthe (especially with Dataloader), problems usually end up compounding even more.

For (a surprisingly realistic) example's sake, imagine we're working on a standard Elixir project which uses Phoenix and Absinthe; it'll usually end up having a directory structure not very different from this:

├── my_project/
│   ├── my_context/my_entity.ex
│   └── my_context.ex
└── my_project_web/
    ├── controllers/my_entity_controller.ex
    ├── resolvers/my_entity_resolver.ex
    ├── types/my_entity_types.ex
    └── my_loader.ex

And while organization wise this is what we've come to expect when following standard Elixir conventions, the shocking thing is that every file I've listed here usually contains Ecto code. To make things even worse, these files don't just contain any old Ecto code, but they go as far as to define new Ecto.Querys! In my mind, this is a huge no–no.

Before I get ahead of myself, I'll briefly explain how I interpret the different roles of the files shown in the directory structure example:

• Everything within my_project/ is what I'd define as "application logic". This is the real meat and potatoes and our app; this is where we would read/write stuff to/from the database, where we'd write code to actually do stuff (as nebulous as that sounds).
  • my_context.ex is a standard Phoenix style context, which in other words, is just a simple Elixir module that we treat as the public API for doing "things" concerning context. A better example to name my_context.ex would be something like accounts.ex which exposes functions such as Accounts.register_new_user/N, Accounts.create_login_token_for_user/N, Accounts.send_forgot_password_email/N.
  • my_entity.ex is a standard Ecto.Schema, something that maps records in your database into standard Elixir structs. A more realistic schema name would be user.ex, or admin.ex.
• Everything within my_project_web/ is what I'd define as a "consumer application". Typically, I don't expect very much real business logic to live in this directory. Modules defined in these files will simply wrap one or more functions defined by contexts in my_project/, maybe doing some mapping/rendering/converting of different data formats.
  • controllers/my_entity_controller.ex is a standard Phoenix controller which lets us trigger code in response to certain REST-like actions.
  • resolvers/my_entity_resolver.ex and types/my_entity_types.ex are relatively normal Absinthe files which are effectively the equivalent of our Phoenix controller, but for GraphQL.
  • my_loader.ex is a Dataloader specific concept; think of it as a way of batching GraphQL requests to avoid the N+1 problem.

Hopefully, then, you can see why having quite literally every single file here create Ecto.Querys is a code smell? To re-iterate, in case it isn't clear, Ecto.Querys are basically a way for you to write SQL queries to read/write from your database. Now, in reality, you'll likely have a bunch of different resolvers, controllers, types, schemas, contexts; so your problems can only get worse from here... 😅

The issues described here are all effectively different flavors of DRY: I've seen many maintenance nightmares where there is no source of truth for how something is meant to work; instead, you get hundreds of duplicated, one-off Ecto.Querys built all over the place, doing similar but slightly different things. Not only is this the antithesis of comprehensibility, but it's also absolutely unmaintainable: how can anyone sleep at night knowing the next day they'll need to find and update every single query dealing with SomeEntity?!

Much like how I describe error handling and propagation, Ecto.Querys are something I want to push deep into one layer of my codebase. In my mind, Ecto.Querys are an extremely low-level implementation detail that is tightly coupled to your Ecto.Schema and database, and we should treat them as such.

Thankfully, with some elbow grease and hard work (and hopefully, a comprehensive test suite 🤞), we can gradually refactor codebases from the state I've just described into something a little more "onion-like" in shape: having a dedicated database-layer (containing all your Ecto code in one centralized place), a business logic layer (Pretty much just contexts and loader.ex, consumers of your database layer), and one or more consumer layers (Phoenix controllers, Oban workers, Absinthe resolvers, etc.).

One Small Step...

Continuing along with the example directory structure above, let's imagine that our context currently looks like this:

defmodule MyProject.MyContext do
  alias MyProject.MyContext.MyEntity

  alias MyProject.Repo

  import Ecto.Query

  def does_entity_exist?(entity_id) do
    from(
      e in MyEntity,
      where: e.id == ^entity_id,
      where: coalesce(e.deleted, false) != true,
      limit: 1
    )
    |> Repo.exists?()
  end

  def count_entities do
    from(e in MyEntity, where: coalesce(e.deleted, false) != true)
    |> Repo.aggregate(:count)
  end

  def count_entities(org_id) do
    from(
      e in MyEntity,
      where: e.org_id == ^org_id,
      where: coalesce(e.deleted, false) != true,
    )
    |> Repo.aggregate(:count)
  end
end

This looks okay so far, right? I'd go as far as to call this pretty comfortable since this is the state a lot of Elixir projects end up looking like. It works well and is easy to grok in isolation, but we can already foresee some issues that might come up (even with this one module):

1. Note the repeated query expressions (where: coalesce(e.deleted, false)), which are present in every query. This makes sense since we don't want deleted entities to "exist" or be "counted", but it definitely makes future refactorings in this file a little daunting.

   What if we were asked to amend this where: coalesce(e.deleted, false) clause to also consider something like e.updated_at so that we can effectively ignore the existence of entities that haven't been updated in over some time period? We'd need to go and carefully update every function and manually implement this check ourselves. Not exactly safe or not error-prone.

2. Since we're building queries on the fly, how is anyone supposed to know what fields are available in our schema? Someone who needs to maintain this file will need to know, say, that MyEntity has an org_id field or a deleted flag. IDE autocompletion gets us part of the way there, but it's flakey at best (at least in Vim).

   This might be fine right now, as we're working with a single schema, but imagine we need to start joining on different entities and remembering what each entity's respective fields are. This requires a great amount of additional mental overhead and exponentially increases the surface area for things to go wrong.

For me, both of these issues highlight the fact that we aren't working at the right level of abstraction for our contexts. The first issue shows us that we need to centralize our queries into a single source of truth. The second issue shows that we're leaking database-level implementation details all over the place (extrapolate what this means when you have queries being built this way across your entire codebase...)

The first thing I usually do when confronted with a codebase that looks like this is to look at all the individual query expressions being used and refactor those into functions. I like putting these functions directly in whatever schema the query's primary entity is (in our case, it'd be my_entity.ex). That way, we can compose our Ecto.Query the same way as before, but leveraging functions instead:

defmodule MyProject.MyContext do
  alias MyProject.MyContext.MyEntity

  alias MyProject.Repo

  def does_entity_exist?(entity_id) do
    MyEntity
    |> MyEntity.where_not_deleted()
    |> MyEntity.where_id(entity_id)
    |> Ecto.Query.limit(1) 
    |> Repo.exists?()
  end

  def count_entities do
    MyEntity
    |> MyEntity.where_not_deleted()
    |> Repo.aggregate(:count)
  end

  def count_entities(org_id) do
    MyEntity
    |> MyEntity.where_not_deleted()
    |> MyEntity.where_org_id(org_id)
    |> Repo.aggregate(:count)
  end
end

defmodule MyProject.MyContext.MyEntity do
  ...

  def where_not_deleted(query) do
    from e in query,
      where: coalesce(e.deleted, false) != true
  end

  def where_id(query, id) do
    from e in query,
      where: e.id == ^id
  end

  def where_org_id(query, org_id) do
    from e in query,
      where: e.org_id == ^id
  end

  ...
end

Hopefully, it's clear that we're already starting to address the issues I described. Thankfully, Ecto makes it dead easy to make queries composable out-of-the-box, so this refactor ends up not being very much work.

The first issue ends up being resolved quite directly, whereas the second one is resolved by... you got me: IDE autocompletion! Why is this an acceptable fix now when it wasn't before? Because we're now working at a higher level of abstraction—it doesn't matter, say, if where_org_id needs to join some other entities to do its job, those are just database level implementation details that are hidden from you. You're composing natural-language statements in far more of a declarative way than if you were to just write the Ecto.Querys by hand.

Going back to the bits of added complexity I mentioned when describing the problems above, imagine we really did want to implement additional filters such as this where_not_outdated check (perhaps where_not_archived would be a better name?): all we would need to do is implement that as a function in my_entity.ex and call it as part of whichever pipeline needs the addition!

Not only do we minimize code duplication in your contexts, because these new composable query functions can be used elsewhere, you can also deduplicate the queries strewn all over your codebase (loader.ex, I'm looking at you 👀). The ultimate goal is to have the source of truth for anything database-related (insofar as MyEntity is concerned) be contained within my_entity.ex itself.

Down the Rabbit Hole

We're not done yet: iterative software development throws another spanner in your works! While the code we've been left with looks to be in a much better place, imagine we need to revisit how we've implemented the MySchema.where_org_id/2 check above: instead of us being able to store an org_id field directly on entity's schema, we need to join the owner of our entity, and join the org of our owner.

Jumping right in, if we try to solve this problem naturally and iteratively using the pattern we've just described, we should end some with something quite similar to the following:

defmodule MyProject.MyContext.MyEntity do
  ...

  def where_org_id(query, org_id) do
    from e in query,
      join: c in assoc(e, :owner),
      join: o in assoc(c, :org),
      where: o.id == ^org_id
  end

  ...
end

Surprisingly, this teeny three-line change does the trick! Before we celebrate, however, I'll point out the Pandora's box we just unleashed into our codebase 😉

The new issue we've introduced is that MyEntity.where_org_id/2 implements a check on data it doesn't own! This might sound fine on such a small scale. Still, I've seen this kind of thing escalate to ridiculous levels—its easy to see how this regresses into the same problem we were trying to solve at the beginning of all of this: if multiple entities all need to check their org_id, at this rate, we'd be reimplementing this same check in literally every schema in our project!

Ideally, we'd just be able to join on our owner struct and call Owner.where_org_id(owner, ^org_id) on it; however, this isn't something that can be expressed in Ecto.Query's DSL of course—the owner struct isn't realized here yet and just represents an expression in a yet-to-be-executed SQL query.

We can express something roughly analagous however! If we keep going down the rabbit hole of trying to break the query example we arrived at into smaller and smaller pieces, let's see if anything clicks and becomes self-evident:

def where_org_id(query, org_id) do
  query
  |> join_owner()
  |> where_owner_org_id(org_id)
end

defp join_owner(query) do
  from e in query,
    join: c in assoc(e, :owner)
end

defp where_owner_org_id(query, org_id) do
  from [..., c] in query,
    where: c.org_id == ^org_id
end

This is a little closer to where we want to be: it encapsulates all of the concerns we have into their own functions, which is good, but we still have yet to solve our issue. Let's take a look at what exactly is insufficient with these three functions.

1. We still can't move this where_owner_org_id/2 check into the Owner schema. We've decomposed our query function into atomic pieces that we literally can't split anymore, but we still have a big problem.

   Queries thus far have been composable because we're always querying against the main entity (from here on out, we'll call it the first binding) of a query. After we do a join_owner/1, we need to keep in mind that we have an owner binding in our query as well. In addition to this, if we want to move our where_owner_org_id/2 to Owner.where_org_id/2, we need to be somehow able to handle the fact that owner isn't the first binding of this query.

2. Even if we could somehow determine that owner is the second binding of a query in some cases, how do we get this to work in general? What if we expose multiple join_blah/1 functions? We would need to micromanage the order entities are joined all over the place to try and be consistent. This isn't maintainable or anywhere near the idea of a fun time...

3. If multiple queries end up calling join_owner/1 internally, Ecto.Query will end up generating SQL with duplicated joins. This might cause issues when running your final SQL query—it isn't optimal, that's for sure! This is likely(?) optimized away by some databases, but maybe not. Either way, it's something to avoid if we can help it.

Thankfully, Ecto provides us with the tool we need to solve all of these issues in one fell swoop by unlocking a new composition superpower: named bindings!

Named bindings work as they sound: they let you name bindings such that you can refer to them later in the query by a given name rather than position. This is in contrast to Ecto's default behavior, which is positional binding: the order in which you bind entities into your query determines how you use them later.

You can create a named binding as simply as using the as keyword following a from or join keyword like so:

defp join_owner(query) do
  from e in query,
    join: c in assoc(e, :owner),
    as: :owner
end

Since naming is hard, I follow the simple convention whereby if I'm joining on a module named SomeSchema, I tend to want to be consistent and name my binding something like :some_schema. This convention is nice because I can reuse any queries that work for :some_schema, no matter the source of the query.

Once a named binding has been created, you can query against said name binding with the syntax below:

defp where_owner_org_id(query, org_id) do
  from [owner: owner] in query,
    where: owner.org_id == ^org_id
end

This, if you noticed, is pretty much exactly what we want. This query is not coupled to anything but the fact that owner is a named binding in our query, which naturally leads us to my next big point (and quite likely my most contrarian one): I opt for never using positional bindings; instead, I tend to use named bindings for quite literally everything.

If we use named bindings (especially if they're consistently named), that means we can freely compose literally any queries between any schemas we've defined in our application! Talk about reaching the end of the rabbit hold 💪

(Consistently named) Named bindings also let us avoid Ecto generated duplicate joins where none are needed, Ecto.Query provides the predicate function Ecto.Query.has_named_binding?/2 which will return true if the given Ecto.Query has a named binding with the given name. This means we can update our join_owner/1 function to be a no-op if we already have the required join in place:

defmodule MyProject.MyContext.MySchema do
  alias MyProject.MyContext.Owner
  ...

  def where_org_id(query, org_id) do
    query
    |> join_owner()
    |> Owner.where_org_id(org_id)
  end

  defp join_owner(query) do
    if has_named_binding?(query, :owner) do
      query
    else
      from [my_context: my_context] in query,
        join: _owner in assoc(my_context, :owner),
        as: :owner
    end
  end

  ...
end

defmodule MyProject.MyContext.Owner do
  ...

  defp where_org_id(query, org_id) do
    from [owner: owner] in query,
      where: owner.org_id == ^org_id
  end

  ...
end

As a side note, I raised an issue to the Ecto mailing list about implementing a guard for this check (since all the data is available by inspecting the Ecto.Query struct).

It won't be available any time soon since it requires the latest version of Elixir. Still, hopefully, one day, we can re-implement the if check by purely utilizing pattern matching a function head when when has_named_binding(query, binding) instead! How neat is that?!

Going the extra mile (and to make automating our named binding generation easier since we're going to want to do it every time we query something), instead of having query pipelines in our context start with a schema module (i.e. MyEntity), I typically define base_query/0 functions in all of my schemas.

These base_query/0 functions serve two purposes: the first being a place to hide the implementation detail of making a named binding, and the second being a convenient place to group queries we always want to do in one place (i.e., the where_not_deleted/1 and where_not_archived/1 checks we talked about above). This equates to the following changes:

# Instead of this...
defmodule MyProject.MyContext do
  ...

  def count_entities do
    MyEntity
    |> MyEntity.where_not_deleted()
    |> Repo.aggregate(:count)
  end

  ...
end

# We do this...
defmodule MyProject.MyContext do
  ...

  def count_entities do
    MyEntity.base_query()
    |> Repo.aggregate(:count)
  end

  ...
end

defmodule MyProject.MyContext.MyEntity do
  ...

  def base_query() do
    query = from e in query, as: :my_entity

    query
    |> MyEntity.where_not_deleted()
  end

  ...
end

With this pattern in hand, you should find yourself empowered to weave, stitch, and compose even the most complex of Ecto queries together! Even so, there is much further we can go—Ecto.Query provides other ways to extending/composing queries beyond the simple expression chaining we've discussed.

If you're interested in this, I definitely suggest reading into fragments, dynamic queries, and subqueries. You can truly build some amazing things with these tools, but dedicated deep-dives for all of these will be saved for another time as they'll prove to be quite the detour.

Compose all the things!

There's still a little bit more we can do to eke out the most composability in our application layer. Now that our Ecto.Querys are written with composability in mind, we can go ahead and refactor our application layer to be dynamic rather than static: what I mean by this is until now, we've written our queries as standard and static Elixir pipelines.

There isn't a reason to limit ourselves to this since Ecto.Query composition is ultimately just accumulating expressions in a struct; it's something we can accumulate over! Since our example talked about being a Phoenix/Absinthe app in the beginning (though we don't really care too much about those details), naturally, our app exposes some public API that provides our users with the ability to provide us arbitrary query parameters that we need to handle.

Tangentially: while I recognize the necessity of a project like Dataloader, it's annoying to me that Dataloader effectively ends up forcing us to duplicate functions we've already defined in our contexts.

For the sake of simplicity, I'm going to ignore Dataloader's existence for the following example.

If your project does use Dataloader, at least with compositional queries, you can try to re-use them between your loader.ex and contexts.

If we assume our app exposes a GraphQL endpoint that ends up looking like this:

object :entity_queries do
  field :entities, non_null(list_of(non_null(:entity))) do
    description("Retrieve a list of all patients at your org")

    arg(:owner_id, description: "Include only entities belonging to the given owner")

    arg(:include_deleted, description: "Include deleted entities in results")
    arg(:include_archived, description: "Include archived entities in results")

    arg(:limit, :integer, description: "Limit the number of results")
    arg(:offset, :integer, description: "For pagination, offset the return values by a number")

    resolve(fn args, _context = %{current_user: %{org: %Org{} = org}} ->
      # somehow implements this endpoint
    end)
  end
end

We now have the problem of needing to query data from our database based on whatever combination of args gets passed in by our API user.

One way of dealing with this need is to build the query in our resolver function itself dynamically (this is typically what gets done by Dataloader or Relay), but I don't want to conflate application layer concerns with consumer layer concerns; I don't want to go this route.

We already have a perfectly operational function in our contexts, so more often than not, my resolver functions end up being very lightweight wrappers around context functions (again, everything in my_project_web/ lives in our consumer layer, so this is to be expected). Usually I end up end up writing something analagous to:

# My resolver function in my Absinthe schema
resolve(fn args, _context = %{current_user: %{org: %Org{} = org}} ->
  MyContext.list_entities(org, args)
end)

# Corresponding in context
defmodule MyProject.MyContext do
  ...
  
  def list_entities(%Org{} = org, args) when is_map(args) do
    args
    |> MyEntity.base_query()
    |> MyEntity.where_org_id(org.id)
    |> MyEntity.where_owner_id(args.owner)
    |> Ecto.Query.limit(args.limit)
    |> Ecto.Query.offset(args.offset)
    |> Repo.all()
  end

  ...
end

# Corresponding in schema
defmodule MyProject.MyContext.MyEntity do
  ...
  
  def where_owner_id(query, nil), do: query

  def where_owner_id(query, owner_id) do
    from [entity: entity] in query,
      where: entity.owner_id == ^owner_id
  end

  ...
end

As you can see, simply leveraging pattern matching in our composable query expression functions let us define quite clearly what possible filters a given function can do, while not explicitly requiring any particular pieces of data.

For particular gnarly or complex outside-world dependant queries (i.e., if a single arg ends up chaining multiple query expressions), I tend to prefer to write in the following pattern instead:

defmodule MyProject.MyContext do
  ...
  
  def list_entities(%Org{} = org, args) when is_map(args) do
    base_query =
      MyEntity.base_query()
      |> MyEntity.where_org_id(org.id)

    query =
      Enum.reduce(args, base_query, fn
        {:limit, limit}, query ->
          Ecto.Query.limit(query, limit)

        {:offset, offset}, query ->
          Ecto.Query.offset(query, offset)

        {:owner_id, owner_id}, query ->
          MyEntity.where_owner_id(query, owner_id)

        {:something_complex, complex}, query ->
          query
          |> MyEntity.do_something()
          |> MyEntity.something_else()
          |> MyEntity.where_complex_thing(complex)

        _unsupported_option, query ->
          query
      end)

    Repo.all(query)
  end

  ...
end

Alternatively, you can define MyEntity.where_complex_thing/2 to be a function that internally combines multiple other, small queries.

It can always be a delicate balance between creating hundreds of these one-off queries which map one-to-one with API options versus having super low-level query functions which don't do anything on their own. Still, much like the difficulty of naming things, this is where we just need to make judgment calls and refactor as we start seeing problems.

Thankfully, once we've reached this stage of API design and query composition, refactoring and reorganizing things tends to be much less daunting a task, so it's not too big of an ordeal ✊

Conclusion

Hopefully, I've said and showed enough to convince you that many projects use Ecto sub-optimally (or at very least, there are ways to make Ecto work harder for you!).

Ecto really does prove to be one of the nicest ORM-like pieces of technology I've ever used, and I'd love to see more people embrace the power that Ecto has to offer.

One of the huge benefits of composing queries like this outside of centralizing the source of truth for querying the database is the increase in confidence it gives us. I'm not personally a TDD Zealot (or hardcore believer in any development dogma), but I do aim to test the majority of the code I write, at the very least, with unit tests.

The nice thing about unit testing context functions made up of these composable query functions is that every single test compounds the confidence of my underlying query expressions more and more. Since I'm extensively testing the bare building blocks of all the business logic in my app, I'm really compounding what I get out of my test suite—definitely by a few orders of magnitude compared to projects which don't centralize their queries in one place.

There is still much, much more that could be said about Ecto, we've only really scratched the surface—I want to eventually talk about how you can make good use of Ecto.Query.fragment/N (has anyone else noticed it's a variadic macro?! How?! 🤯), subqueries, dynamic queries, etc. Until then, if you enjoyed this (admittedly opinionated) post, I've written another Ecto related post on the shortcomings of, and solutions to Ecto's virtual fields.

I hope this has helped! Until next time ☕

Compositional Elixir—Contexts, Error Handling, and, Middleware

This post serves as a follow-up to a post I recently wrote about Railway Oriented Programming, and abusing Elixir's with statement to great effect. As part of that post, I briefly talked about one major advantage of the with pattern being the deference of error handling to the site at which one needs to consume the errors, rather than sprinkling error handling code all over the place.

The main idea is that all of the business logic you write should be made of small, composable units which you can string together in such a way that you shouldn't have to handle errors in each individual unit. The way I build programs is that each small unit essentially has a hard dependency on other units that it itself calls: if any of these sub-units fail, then that failure should propagate itself up to the client or call site—de-coupling high-level logic/thinking such as "what do I do if this subsystem is down from the expectation that the happy path should just work.

TLDR; the high-level pattern

For those uninterested or just wanting to get the gist of what was discussed in the linked post above: the with statement gives two major wins over Elixir's simpler case construct:

1. It stops you from having to break every possible piece of conditional logic in either another function (in many cases, adding much noise to a module) or in a further nested case (which only serves to make things much harder to follow, especially when combined with the second point).with`.

2. It allows you to be very terse. If you don't explicitly want to handle errors, you don't have to. Errors automatically bubble up for you to conditionally handle, or not, as you please. This is especially useful if you otherwise would have many nested case statements—something common in the Erlang world, and perhaps only isn't so in the Elixir world because of constructs such as with.

When I'm writing logic in Elixir, I almost always write it in the following form:

def some_function() do
  with {:ok, step_1} <- do_step_one(),
       {:ok, step_2} <- do_step_two(),
       {:ok, step_3} <- do_step_three() do
    {:ok, step_3}
  end
end

If I call such a function, any of do_step_one/0, do_step_two/0, or do_step_three/0 may fail. If either one of these functions doesn't return something in the shape {:ok, _something}, I will receive it. This means that everything is dandy so long as those three functions each return a domain-specific and helpful error.

Assuming function do_step_one/0 fails, and assuming it does somethign super important as follows:

defp do_step_one() do
  case check_database_connection() do
    {:ok, %Db.Connection{} = connection} ->
      {:ok, connection}

    _otherwise ->
      {:error, :database_down}
  end
end

As a caller of some_function/0, I will very nicely get a reason as to _why my invocation of some_function/0 failed. No additional code needed!

Alas, not all systems are so conveniently self-contained. The majority of the systems I've worked on in the last two years as a consultant on a few large Elixir projects for multiple clients have been services exposed on the internet and utilised by numerous different devices. The kind of "letting errors propagate to the client" sounds great, but what does that actually mean in the context of a large Elixir application that's serving users over some API or website versus someone playing around in a REPL?

High-level architecture for a Phoenix application

Phoenix is probably one of the most common frameworks we see as Elixir developers. Phoenix has a very loosely defined convention of splitting up your business logic into the following sections:

• Controllers: these are modules responsible for mapping HTTP requests to business logic in Contexts. Controllers typically just encapsulate CRUD actions, but ultimately, are just small wrappers transforming external requests into one (or more) Context function invocations in series.

• Contexts: these are small modules that encapsulate "units" of business logic. In never-so-certain terms, I might have a high-level context called Auth, which provides functions that work on entities vaguely related to the concept of "authorization", such as: create_user, validate_user_password, user_belongs_to_organization? etc.

• Schemas: these are modules that define structs for your contexts to work against. More often than not, these structs are backed by Ecto (regardless of whether or not you're using a database). These modules might provide minimal utility functions for performing work on these structs, validating them, etc.

Even if you're not using Phoenix in your Elixir app, I've definitely found much merit in this way of splitting up a system. In the majority of projects or libraries I work on, I adopt something similar. You can get further decoupling using umbrella projects instead of standard Elixir projects, but this is not an important detail for now.

One nice thing about this controller/context/schema split in particular is that not only is the concern split (web-layer/business-logic-layer/database-layer) evident, the error handling concern is appropriately split too:

# This is our controller:
defmodule MyAppWeb.SomethingController do
  use MyAppWeb, :controller

  alias MyApp.Somethings
  alias MyApp.Somethings.Something

  def show(conn, %{"id" => id}) do
    case Somethings.get_something_by_id(id) do
      {:ok, %Something{} = something} ->
        conn
        |> put_status(:ok)
        |> render("show.json", %{something: something})

      {:error, reason} when reason in [:not_found, ...] ->
        conn
        |> put_status(reason)

      _error ->
        conn
        |> put_status(:internal_server_error)
  end
end

# This is our context:
defmodule MyApp.Somethings do
  alias MyApp.Something
  alias MyApp.Repo

  def get_something_by_id(id) do
    Something
    |> Something.where_id(id)
    |> Something.not_deleted()
    |> Repo.one()
    |> case do
      %Something{} = something ->
        {:ok, something}

      nil ->
        {:error, :not_found}
    end
  end
end

# This is our schema:
defmodule MyApp.Somethings.Something do
  use Ecto.Schema
  import Ecto.Query

  schema "somethings" do
    field :something, :string
    field :deleted, :boolean, default: false
  end

  def changeset(%__MODULE__{} = entity, attrs) do
    entity
    |> cast(attrs, [:something, :deleted])
  end

  def where_id(query, id) do
    from s in query,
      where: s.id == ^id
  end

  def where_not_deleted(query) do
    from s in query,
      where: s.deleted == false
  end
end

Note that each layer of our business logic example here handles its own concerns:

• Our schema only cares about providing composable, minimal functions which detail operating with the database. It validates stuff going into the database and provides composable queries which can be chained together arbitrarily.

• Our context uses the small functions available to it from schemas (and other contexts, such as MyApp.Repo) to provide atomic chunks of logic: fetch X from the database, create a new Y, etc.

• Our controller maps both successful and unsuccessful invocations of our context functions into domain-specific responses, which we return to the client making the web request.

An example of composability and extensibility

This is essentially the entire idea at work already, just a straightforward version of it. Imagine, instead of the simple example above, we're working to authenticate a user trying to log into a service. I'll leave out the schema implementation details for brevity, but an example would end up looking like this:

# This is our controller:
defmodule MyAppWeb.AuthTokenController do
  use MyAppWeb, :controller

  alias MyApp.Auth
  alias MyApp.Auth.User
  alias MyApp.Auth.Token
  alias MyApp.Auth.Organization

  @encodable_errors [:unauthorized, :not_found, :invalid_password]

  def create(conn, %{"username" => username, "password" => password, "org_code" => org_code}) do
    with {:ok, %Organization{} = organization} <- Auth.get_organization_by_code(org_code),
         {:ok, %User{} = user} <- Auth.get_user_by_username_and_organization(username, organization),
         {:ok, %User{} = user} <- Auth.validate_user_password(user, password),
         {:ok, %Token{} = token} <- Auth.create_auth_token(user, organization) do
       conn
       |> put_status(:ok)
       |> render("show.json", %{token: token})
    else
      {:error, reason} in @encodable_errors ->
        conn
        |> put_status(reason)

      {:error, %Ecto.Changeset{} = error} ->
        conn
        |> put_status(:bad_request, build_error(error))
    end
  end

  # Some extensive pattern matching
  defp build_error(error), do: ...
  defp build_error(error), do: ...
  defp build_error(error), do: ...
  defp build_error(error), do: ...
end

# This is our context:
defmodule MyApp.Auth do
  alias MyApp.Auth.User
  alias MyApp.Auth.Token
  alias MyApp.Auth.Organization
  alias MyApp.Repo

  @ttl_minutes 60

  def get_organization_by_code(org_code) do
    Organization
    |> Organization.where_org_code(org_code)
    |> Repo.one()
    |> case do
      %Organization{} = organization ->
        {:ok, organization}

      nil ->
        {:error, :not_found}
    end
  end

  def get_user_by_username_and_organization(username, %Organization{id: organization_id}) do
    User
    |> User.where_username(username)
    |> User.where_is_authenticated()
    |> User.where_organization_id(organization_id)
    |> User.is_not_deleted()
    |> Repo.one()
    |> case do
      %User{} = user ->
        {:ok, user}

      nil ->
        {:error, :unauthorized}
    end
  end

  def validate_user_password(%User{} = user, password) do
    if User.encrypt_password(password) == user.hashed_password do
      {:ok, user}
    else
      {:error, :invalid_password}
    end
  end

  def create_auth_token(%User{} = user, %Organization{} = organization, ttl_minutes \\ @ttl_minutes) do
    %Token{}
    |> Token.changeset(%{user: user, organization: organization, ttl_minutes: ttl_minutes})
    |> Repo.insert()
  end
end

Ignoring whether or not we want to expose such fine-grained errors for an auth flow (we never do!), we can see the strength of being able to compose everything. Literally every single piece of business logic here is built to be trivially composable. If we wanted to make it such that a user could request a higher token ttl, we could extend the controller function:

def create(conn, %{"username" => username, "password" => password, "org_code" => org_code} = params) do
  custom_ttl = Map.get(params, "ttl", 1200)

  with {:ok, %Organization{} = organization} <- Auth.get_organization_by_code(org_code),
       {:ok, %User{} = user} <- Auth.get_user_by_username_and_organization(username, organization),
       {:ok, %User{} = user} <- Auth.validate_user_password(user, password),
       {:ok, %Token{} = token} <- Auth.create_auth_token(user, organization, custom_ttl) do
     conn
     |> put_status(:ok)
     |> render("show.json", %{token: token})
  else
    {:error, reason} in @encodable_errors ->
      conn
      |> put_status(reason)

    {:error, %Ecto.Changeset{} = error} ->
      conn
      |> put_status(:bad_request, build_error(error))
  end

  # Some extensive pattern matching
  defp build_error(error), do: ...
  defp build_error(error), do: ...
  defp build_error(error), do: ...
  defp build_error(error), do: ...
end

And suppose if Auth.Token.changeset/2 had some validation for the minimum or maximum ttl that can be set? Well, those errors would propagate up through the context and the controller, be serialized into some format the client can understand, and be returned to the user!

This is the power of writing code with respect primarily towards the happy path.

Committing to happy-path-programming: Middleware!

Now that we've outlined the gist, we can dive even deeper into this cult of composability: let's remove error handling from view entirely!

If we take the auth flow example we've been working on, this is all well and good, but we've ended up with a bunch of boilerplate. We still need to extensively pattern match and handle errors from our contexts to map said errors into something client-specific for every action we want to expose through our controllers. Isn't this orthogonal to the point of Railway-Oriented programming? Perhaps.

Unfortunately, for any case where we expect an error to occur, we really do want to map it for the user's benefit. Unexpected errors will always arise, and unexpected errors warrant exceptional behaviour (i.e. crashing the process responsible for handling the user's request a-la OTP), but errors that are entirely within the domain of errors we expect? We should handle those.

Thankfully, instead of writing macros we inject everywhere (a common approach on a few projects I've seen), or just decided to say "screw it" and manually implementing the boilerplate, we can be a little smarter about it.

I'm not too familiar with the internals of Phoenix, but essentially, all requests that come in go through a pipeline of functions (plugs). I'm not sure if semantically your controller function is a plug, but it's essentially just another function that gets executed as part of this pipeline. When we call the render function on the happy path, we're essentially just saying: "return this to the client who requested this", but Phoenix actually has a built-in mechanism for handling the case where we aren't explicitly calling render: a fallback plug!

defmodule MyAppWeb.AuthTokenController do
  use MyAppWeb, :controller
  action_fallback MyAppWeb.ErrorHandler

  alias MyApp.Auth
  alias MyApp.Auth.User
  alias MyApp.Auth.Token
  alias MyApp.Auth.Organization

  @encodable_errors [:unauthorized, :not_found, :invalid_password]

  def create(conn, %{"username" => username, "password" => password, "org_code" => org_code}) do
    with {:ok, %Organization{} = organization} <- Auth.get_organization_by_code(org_code),
         {:ok, %User{} = user} <- Auth.get_user_by_username_and_organization(username, organization),
         {:ok, %User{} = user} <- Auth.validate_user_password(user, password),
         {:ok, %Token{} = token} <- Auth.create_auth_token(user, organization) do
       conn
       |> put_status(:ok)
       |> render("show.json", %{token: token})
    end
  end
end

defmodule MyAppWeb.ErrorHandler do
  @http_errors ["forbidden", "not_found", "conflict", ...]

  def call(conn, {:error, "unauthorized"}) do
    conn
    |> put_resp_header("www-authenticate", "Bearer")
    |> put_status(:unauthorized)
    |> halt()
  end

  def call(conn, {:error, http_error}) when http_error in @http_errors do
    conn
    |> put_status(http_error)
    |> json(%{errors: [%{code: status}]})
    |> halt()
  end

  ...
end

This is much nicer, in my opinion. We massively clean up the core business logic of our main controller in-so-far that it is essentially just describing a bunch of steps that have to happen to create an auth token. If anything unexpected happens, those errors are handled in a module whose entire purpose is to map context errors into client errors. We have the added benefit of centralising how we handle errors and thus returning errors consistently no matter where they happen, so long as our error handler is being used!

Anything unexpected will also either propagate to the client untouched (which is one valid option) or will just cause a crash (another valid option). That implementation detail can be left solely to you 😊

Addendum: Error handling for Absinthe and GraphQL

As stated previously, Phoenix is ubiquitous on Elixir projects. The majority of Elixir projects I've been brought to help out on have had Phoenix as part of their stack.

This is also true of Absinthe, a really nice framework for serving GraphQL.

At the time of writing, error handling honestly isn't even nearly standardized for GraphQL. I've worked on approaches that use union types to represent typed errors for the client as well as just settling for the more standard top-level errors list.

While there is a bunch of development here that will change whether or not this is the best thing to do going forwards, at the time of writing, we've ended up extending our error-handling Middleware idea to Absinthe as well.

Much like Phoenix, Absinthe is essentially designed as a pipeline of functions that end up with your resolvers being executed. GraphQL Schemas and their corresponding resolvers are kind of analogous to Phoenix controllers in that they essentially just call functions in our standard contexts.

Resolvers in Absinthe are expected to return either {:ok, _some_happy_path_result} or {:error, some_supported_error_format}, but much like our Phoenix controller functions, we can implement a middleware to take care of anything that isn't expected transparently.

A similar middleware to the Phoenix one can be implemented as follows:

defmodule MyAppGraphQL.ErrorHandler do
  @behaviour Absinthe.Middleware

  def call(%Absinthe.Resolution{errors: errors} = resolution, _config) do
    %Absinthe.Resolution{resolution | errors: Enum.flat_map(errors, &process_error/1)}
  end

  defp process_error(:unauthorized) do
    [{:error, %{status: 401, error: %{code: "unauthorized"}}}]
  end

  ...
end

And once written, it either needs to be added to Absinthe's top-level middleware execution pipeline, which is executed for all resolvers, or to individual resolvers:

# Top level middleware registration
defmodule MyAppGraphQL.Schema do
  use Absinthe.Schema

  def middleware(middleware, _field, _context) do
    middleware ++ [MyAppGraphQL.ErrorHandler]
  end
end

# Resolver specific middleware registration
field :auth_token, :auth_token do
  middleware MyAppGraphQL.AuthenticationErrorHandler
  ...
end

Addendum: More complex Ecto.Schema/Ecto.Query composition

I'm often asked how we can write composable functions to dynamically build queries involving all sorts of things such as joins, sub-queries etc.

This is a good question since the naive implementation gets verbose and extremely brittle very quickly.

Typically, if I need to define functions that do complex joins, instead of doing it as follows:

defmodule MyApp.Auth.User do
  use Ecto.Schema
  import Ecto.Query

  alias MyApp.Auth.Organization

  schema "user" do
    field :username, :string
    belongs_to :organization, Organization
  end

  def changeset(%__MODULE__{} = entity, attrs) do
    entity
    |> cast(attrs, [:username, :organization_id])
  end

  def where_id(query, id) do
    from u in query,
      where: u.id == ^id
  end

  def join_organization(query) do
    from u in query,
      join: o in assoc(u, :organization)
  end

  def where_organization_region(query, organization_region) do
    from u, o in query,
      where: o.region == ^organization_region
  end
end

# Used as follows:
def user_has_region?(user_id, region) do
  User
  |> User.where_id(user_id)
  |> User.join_organization()
  |> User.where_organization_region(region)
  |> Repo.exists?
end

I tend to use what Ecto calls named bindings, which instead of composing queries and joined entities based on position a-la from a, b, c, d in query, ..., you can do the following instead: