Programming Elixir ≥ 1.6

Built-in Protocols

Elixir comes with the following protocols:

Enumerable and Collectable
Inspect
List.Chars
String.Chars

To play with these, let’s work with MIDI files.

A MIDI file consists of a sequence of variable-length frames. Each frame contains a four-character type, a 32-bit length, and then length bytes of data.^[38]

We’ll define a module that represents the MIDI file content as a struct, because the struct lets us use it with protocols. The file also defines a submodule for the individual frame structure.

protocols/midi.exs

	defmodule Midi do

	defstruct(content: <<>>)

	defmodule Frame do
	defstruct(
	type: "xxxx",
	length: 0,
	data: <<>>
	)

	def to_binary(%Midi.Frame{type: type, length: length, data: data}) do
	<<
	type::binary-4,
	length::integer-32,
	data::binary
	>>
	end
	end

	def from_file(name) do
	%Midi{content: File.read!(name)}
	end
	end

Built-in Protocols: Enumerable and Collectable

The Enumerable protocol is the basis of all the functions in the Enum module—any type implementing it can be used as a collection argument to Enum functions.

We’re going to implement Enumerable for our Midi structure, so we’ll need to wrap the implementation in something like this:

	defimpl Enumerable, for: Midi do
	# ...
	end

The protocol is defined in terms of four functions:

	defprotocol Enumerable do
	def count(collection)
	def member?(collection, value)
	def reduce(collection, acc, fun)
	def slice(collection)
	end

count returns the number of elements in the collection, member? is truthy if the collection contains value, and reduce applies the given function to successive values in the collection and the accumulator; the value it reduces becomes the next accumulator. Finally, slice is used to create a subset of a collection. Perhaps surprisingly, all the Enum functions can be defined in terms of these four.

However, life isn’t that simple. Maybe you’re using Enum.find to find a value in a large collection. Once you’ve found it, you want to halt the iteration—continuing is pointless. Similarly, you may want to suspend an iteration and resume it sometime later. These two features become particularly important when we talk about streams, which let you enumerate a collection lazily.

We’ll start with the most difficult function to implement, Enumerable.reduce/3. It is worth reading the documentation for it before we start:

	iex> h Enumerable.reduce

	def reduce(enumerable, acc, fun)

	@spec reduce(t(), acc(), reducer()) :: result()

	Reduces the enumerable into an element.

	Most of the operations in Enum are implemented in terms of reduce. This
	function should apply the given t:reducer/0 function to each item in the
	enumerable and proceed as expected by the returned accumulator.

	See the documentation of the types t:result/0 and t:acc/0 for more information.

	## Examples

	As an example, here is the implementation of reduce for lists:

	def reduce(_, {:halt, acc}, _fun),
	do: {:halted, acc}

	def reduce(list, {:suspend, acc}, fun),
	do: {:suspended, acc, &reduce(list, &1, fun)}

	def reduce([], {:cont, acc}, _fun),
	do: {:done, acc}

	def reduce([h \| t], {:cont, acc}, fun),
	do: reduce(t, fun.(h, acc), fun)

The first two function heads do housekeeping: they handle the cases where the enumeration has been halted or suspended. Here are the versions for our MIDI enumerator:

protocols/midi.exs

	def _reduce(_content, {:halt, acc}, _fun) do
	{:halted, acc}
	end

	def _reduce(content, {:suspend, acc}, fun) do
	{:suspended, acc, &_reduce(content, &1, fun)}
	end

The next two function heads do the actual iteration. In the list example in the documentation, you’ll see the typical pattern: check for the end condition ([ ]) and the recursive step [h|t].

We’ll do the same with our MIDI file, but we’ll use binaries instead of doing list pattern matching:

protocols/midi.exs

	def _reduce(_content = "", {:cont, acc}, _fun) do
	{:done, acc}
	end

	def _reduce(<<
	type::binary-4,
	length::integer-32,
	data::binary-size(length),
	rest::binary
	>>,
	{:cont, acc},
	fun
	) do
	frame = %Midi.Frame{type: type, length: length, data: data}
	_reduce(rest, fun.(frame, acc), fun)
	end

See how we split out the binary content of a frame, then wrap it into a Midi.Frame struct before passing it back. This means that folks who use our MIDI module will only see these frame structures, and not the raw data.

Before we try this, there’s one little tweak we have to make. You may have noticed that all my functions were named _reduce, with a leading underscore. That’s because they need to work on the content of the MIDI file, and not on the structure that wraps that content. We have a single function head that implements the actual reduce function, and that then forwards the call on to _reduce:

protocols/midi.exs

	def reduce(%Midi{content: content}, state, fun) do
	_reduce(content, state, fun)
	end

At this point we have enough code to try it out. I’ve included a MIDI file in the code/protocols directory for you to play with (courtesy of midiworld.com).^[39]

	iex midi.exs
	warning: function count/1 required by protocol Enumerable is not
	implemented (in module Enumerable.Midi) midi.exs:21

	warning: function member?/2 required by protocol Enumerable is not
	implemented (in module Enumerable.Midi) midi.exs:21

	warning: function slice/1 required by protocol Enumerable is not
	implemented (in module Enumerable.Midi) midi.exs:21

	Interactive Elixir (1.6.0-rc.0) - press Ctrl+C to exit (type h() ENTER for help)
	iex> midi = Midi.from_file("dueling-banjos.mid")
	%Midi{
	content: <<77, 84, 104, 100, 0, 0, 0, 6, 0, 1, 0, 8, 0, 120, 77, 84, 114, 107,
	0, 0, 0, 66, 0, 255, 3, 14, 68, 117, 101, 108, 105, 110, 103, 32, 66, 97,
	110, 106, 111, 115, 0, 255, 3, 11, 68, 101, 108, 105, 118, ...>>
	}
	iex> Enum.take(midi, 2)
	[
	%Midi.Frame{data: <<0, 1, 0, 8, 0, 120>>, length: 6, type: "MThd"},
	%Midi.Frame{
	data: <<0, 255, 3, 14, 68, 117, 101, 108, 105, 110, 103, 32, 66, 97, 110,
	106, 111, 115, 0, 255, 3, 11, 68, 101, 108, 105, 118, 101, 114, 97, 110,
	99, 101, 0, 255, 88, 4, 4, 2, 24, 8, 0, 255, 89, 2, 0, 0, ...>>,
	length: 66,
	type: "MTrk"
	}
	]

First, see how we get warnings because we haven’t yet implemented the full Enumerable protocol. Normally this would worry me, but I happen to know that we won’t be using those functions just yet.

Next, look at how we called Enum.take/2 and got back two Midi.Frame structures. That’s because take/2 is defined in terms of reduce/3, and we supplied an implementation of it.

So far, so good. Let’s implement count/1 next.

If the collection is countable, the count function returns a tuple containing {:ok, count}. If it isn’t countable (perhaps it is being read one piece at a time from an external source), count should return {:ok, __MODULE__}.

In our case, we have the whole MIDI file available in memory, and we have a way to traverse it using reduce, so counting it easy:

protocols/midi.exs

	def count(midi = %Midi{}) do
	frame_count = Enum.reduce(midi, 0, fn (_, count) -> count+1 end)
	{ :ok, frame_count }
	end

Let’s try it:

	iex> r Enumerable.Midi
	warning: redefining module Midi (current version defined in memory)
	midi.exs:2

	warning: redefining module Midi.Frame (current version defined in memory)
	midi.exs:6

	warning: redefining module Enumerable.Midi (current version defined in memory)
	midi.exs:21

	warning: function member?/2 required by protocol Enumerable is not
	implemented (in module Enumerable.Midi) midi.exs:21

	warning: function slice/1 required by protocol Enumerable is not
	implemented (in module Enumerable.Midi) midi.exs:21

	{:reloaded, Enumerable.Midi, [Midi.Frame, Midi, Enumerable.Midi]}
	iex> Enum.count midi
	9

On to member? and slice.

Technically, both of these can be implemented using reduce. But the Elixir team recognized that some types of collection have more direct ways to test membership and partition elements. For example, if you implement a set using a map, then testing to see if a key is present can be done in constant time. If you have an array-like structure with fixed-length elements, you can split it into two in (almost) constant time.

So the implementations of both member? and slice depend on the characteristics of your collection. In our case, we don’t currently have a fast way of testing for membership, nor do we have a fast way to slice a MIDI file into two. In both cases, we return an error tuple. This doesn’t actually cause an error for the user; it just tells Enumerable to fall back to a naive algorithm.

protocols/midi.exs

	def member?(%Midi{}, %Midi.Frame{}) do
	{ :error, __MODULE__ }
	end

	def slice(%Midi{}) do
	{ :error, __MODULE__ }
	end

And with that, we’re done. Our Midi type is enumerable, and you can use every function in Enum on it.

This gives us the ability to treat a MIDI stream as a collection of MIDI frames. But how do we assemble frames back into a MIDI stream? That’s what we’ll address next.

Collectable

We’ve already seen Enum.into/2. It takes something that’s enumerable and creates a new collection from it:

	iex> 1..4 \|> Enum.into([])
	[1, 2, 3, 4]
	iex> [ {1, 2}, {"a", "b"}] \|> Enum.into(%{})
	%{1 => 2, "a" => "b"}

The target of Enum.into must implement the Collectable protocol. This defines a single function, somewhat confusingly also called into. This function returns a two-element tuple. The first element is the initial value of the target collection. The second is a function to be called to add each item to the collection. (If this reminds you of the second and third parameters passed to Enum.reduce, that’s because in a way into is the opposite of reduce.)

Let’s look at the code first:

protocols/midi.exs

	defimpl Collectable, for: Midi do
	use Bitwise

	def into(%Midi{content: content}) do
	{
	content,
	fn
	acc, {:cont, frame = %Midi.Frame{}} ->
	acc <> Midi.Frame.to_binary(frame)

	acc, :done ->
	%Midi{content: acc}

	_, :halt ->
	:ok
	end
	}
	end
	end

It works like this:

Enum.into calls the into function for Midi, passing it the target value—Midi{content: content} in this case.
Midi.into returns a tuple. The first element is the current content of the target. This acts as the initial value for an accumulator. The second element of the tuple is a function.
Enum.into then calls this function, passing it the accumulator and a command. If the command is :done, the iteration over the collection being injected into the MIDI stream has finished, so we return a new Midi structure using the accumulator as a value. If the command is :halt, the iteration has terminated early and nothing needs to be done.
The real work is done when the function is passed the {:cont, frame} command. Here is where the Collectable appends the binary representation of the next frame to the accumulator.

We can call it in IEx:

	iex> list = Enum.to_list(midi)
	[
	%Midi.Frame{data: <<0, 1, 0, 8, 0, 120>>, length: 6, type: "MThd"},
	%Midi.Frame{
	data: <<0, 255, 3, 14, 68, 117, 101, 108, 105, 110, 103, 32, 66, 97, 110,
	106, 111, 115, 0, 255, 3, 11, 68, 101, 108, 105, 118, 101, 114, 97, 110,
	99, 101, 0, 255, 88, 4, 4, 2, 24, 8, 0, 255, 89, 2, 0, 0, ...>>,
	length: 66,
	type: "MTrk"
	},
	. . .
	]

	iex> new_midi = Enum.into(list, %Midi{})
	%Midi{
	content: <<77, 84, 104, 100, 0, 0, 0, 6, 0, 1, 0, 8, 0, 120, 77, 84, 114, 107,
	0, 0, 0, 66, 0, 255, 3, 14, 68, 117, 101, 108, 105, 110, 103, 32, 66, 97,
	110, 106, 111, 115, 0, 255, 3, 11, 68, 101, 108, 105, 118, ...>>
	}
	iex> new_midi == midi
	true
	iex> Enum.take(new_midi, 1)
	[%Midi.Frame{data: <<0, 1, 0, 8, 0, 120>>, length: 6, type: "MThd"}]

Because the into function uses the initial value of the target collection, we can use it to append to a MIDI stream:

	iex> midi2 = %Midi{}
	%Midi{content: ""}
	iex> midi2 = Enum.take(midi, 1) \|> Enum.into(midi2)
	%Midi{content: <<77, 84, 104, 100, 0, 0, 0, 6, 0, 1, 0, 8, 0, 120>>}
	iex> midi2 = [Enum.at(midi, 3)] \|> Enum.into(midi2)
	%Midi{
	content: <<77, 84, 104, 100, 0, 0, 0, 6, 0, 1, 0, 8, 0, 120, 77, 84, 114, 107,
	0, 0, 8, 34, 0, 255, 33, 1, 0, 0, 193, 25, 0, 177, 7, 127, 0, 10, 100, 0,
	64, 0, 134, 24, 145, 43, 99, 22, 43, 0, 15, ...>>
	}
	iex> Enum.count(midi2)
	2

Remember the Big Picture

If you think all this enumerable/collectable stuff is complicated—well, you’re correct. It is. In part that’s because these conventions allow all enumerable values to be used both eagerly and lazily. And when you’re dealing with big (or even infinite) collections, this is a big deal.

Built-in Protocols: Inspect

This is the protocol that is used to inspect a value. The rule is simple—if you can return a representation that is a valid Elixir literal, do so. Otherwise, prefix the representation with #Typename.

We could just delegate the inspect function to the Elixir default. (That’s what we’ve been doing so far.) But we can do better. Not surprisingly, we do that by implementing the Inspect protocol. We’ll do it for both the overall Midi type and for the individual Midi.Frames.

protocols/midi_inspect.exs

	defimpl Inspect, for: Midi do
	def inspect(%Midi{content: <<>>}, _opts) do
	"#Midi[«empty»]"
	end

	def inspect(midi = %Midi{}, _opts) do
	content =
	Enum.map(midi, fn frame-> Kernel.inspect(frame) end)
	\|> Enum.join("\n")
	"#Midi[\n#{content}\n]"
	end
	end

	defimpl Inspect, for: Midi.Frame do
	def inspect(%Midi.Frame{type: "MThd",
	length: 6,
	data: <<
	format::integer-16,
	tracks::integer-16,
	division::bits-16
	>>},
	_opts) do
	beats = decode(division)
	"#Midi.Header{Midi format: #{format}, tracks: #{tracks}, timing: #{beats}}"
	end

	def inspect(%Midi.Frame{type: "MTrk", length: length, data: data}, _opts) do
	"#Midi.Track{length: #{length}, data: #{Kernel.inspect(data)}"
	end

	defp decode(<< 0::1, beats::15>>) do
	"♩ = #{beats}"
	end

	defp decode(<< 1::1, fps::7, beats::8>>) do
	"#{-fps} fps, #{beats}/frame"
	end
	end

Run the code in IEx:

	iex> midi = Midi.from_file "dueling-banjos.mid"
	#Midi[
	#Midi.Header{Midi format: 1, tracks: 8, timing: ♩ = 120}
	#Midi.Track{length: 66, data: <<0, 255, 3, 14, 68, 117, 101, ...>>
	. . .
	#Midi.Track{length: 6291, data: <<0, 255, 33, 1, 0, 0, 185, ... >>
	#Midi.Track{length: 9, data: <<0, 255, 33, 1, 0, 0, 255, 47, 0>>
	]

I added a little bit of decoding for the header frame, but just treated the track frames as binary. You could extend this to do a full decode of each track frame too.

There’s a wrinkle here. If you pass structs: false to IO.inspect (or Kernel.inspect), it never calls our inspect function. Instead, it formats it as a struct.

Better Formatting with Algebra Documents

The formatting of our MIDI stream leaves a little to be desired: there’s no indentation or reasonable line wrapping.

To fix this, we use a feature called algebra documents. An algebra document is a tree structure that represents some data you’d like to pretty-print.^[40] Your job is to create the structure based on the data you want to inspect, and Elixir will then find a nice way to display it.

I’d like the inspect string to show the nesting of data, and to wrap long lines honoring that nesting.

We do this by having our inspect function return an algebra document rather than a string. In that document, we indicate places where breaks are allowed (but not required) and we show how the nesting works:

protocols/midi_algebra.exs

	defimpl Inspect, for: Midi do
	import Inspect.Algebra

	def inspect(%Midi{content: <<>>}, _opts) do
	"#Midi[«empty»]"
	end

	def inspect(midi = %Midi{}, opts) do
	open = color("#Midi[", :map, opts)
	close = color("]", :map, opts)
	separator = color(",", :map, opts)

	container_doc(
	open,
	Enum.to_list(midi),
	close,
	%Inspect.Opts{limit: 4},
	fn frame, _opts -> Inspect.Midi.Frame.inspect(frame, opts) end,
	separator: separator,
	break: :strict
	)
	end
	end

	defimpl Inspect, for: Midi.Frame do
	import Inspect.Algebra

	def inspect(
	%Midi.Frame{type: "MThd",
	length: 6,
	data: <<

	format::integer-16,
	tracks::integer-16,
	division::bits-16
	>>
	},
	opts)
	do
	concat(
	[
	nest(
	concat(
	[
	color("#Midi.Header{", :map, opts),
	break(""),
	"Midi format: #{format},",
	break(" "),
	"tracks: #{tracks},",
	break(" "),
	"timing: #{decode(division)}",
	]
	),
	2
	),
	break(""),
	color("}", :map, opts)
	]
	)
	end


	def inspect(%Midi.Frame{type: "MTrk", length: length, data: data}, opts) do
	open = color("#Midi.Track{", :map, opts)
	close = color("}", :map, opts)
	separator = color(",", :map, opts)
	content = [
	length: length,
	data: data
	]

	container_doc(
	open,
	content,
	close,
	%Inspect.Opts{limit: 15},
	fn {key, value}, opts ->
	key = color("#{key}:", :atom, opts)
	concat(key, concat(" ", to_doc(value, opts)))
	end,
	separator: separator,
	break: :strict
	)
	end

	defp decode(<< 0::1, beats::15 >>) do
	"♩ = #{beats}"
	end

	defp decode(<< 1::1, fps::7, beats::8 >>) do
	"#{-fps} fps, #{beats}/frame"
	end

	defp decode(x) do
	raise inspect x
	end
	end

On a narrow terminal window, we get this output:

	iex> Midi.from_file "dueling-banjos.mid"
	#Midi[
	#Midi.Header{
	Midi format: 1,
	tracks: 8,
	timing: ♩ = 120
	},
	#Midi.Track{
	length: 66,
	data: <<0, 255, 3, 14, 68, 117, 101, 108, 105,
	110, 103, 32, 66, ...>>
	},
	#Midi.Track{
	length: 1319,
	data: <<0, 255, 33, 1, 0, 0, 192, 105, 0, 176, 7,
	127, 0, ...>>
	},
	#Midi.Track{
	length: 2082,
	data: <<0, 255, 33, 1, 0, 0, 193, 25, 0, 177, 7,
	127, 0, ...>>
	},
	...
	]

For more information, see the documentation for Inspect.Algebra.

Built-in Protocols: List.Chars and String.Chars

The List.Chars protocol is used by Kernel.to_charlist to convert a value into a list of characters (think single-quoted string).

The String.Chars protocol is used to convert a value to a string (binary, or double-quoted string). This is the protocol used for string interpolation.

The protocols are implemented identically, except List.Chars requires that you write a to_charlist function, and String.Chars requires you write to_string.

Although we could implement a String.Chars.to_string for our Midi struct, it probably wouldn’t make much sense. What would you interpolate into the string?