Distillery is an Elixir package that makes most release tasks easy. In particular, it can take the complexity that is the source of your project, along with its dependencies, and reduce it down to a single deployable file.
Imagine you were managing the deployment of hundreds of thousands of lines of code into running telephone switches, while maintaining all the ongoing connections, providing a full audit trail, and maintaining contractual uptime guarantees. This is clearly complex. Very complex. And this is the task the Erlang folks faced, so they created tools that help.
Distillery is a layer of abstraction on top of this complexity. Normally it manages to hide it, but sometimes the lower levels leak out and you get to see how the sausage is made.
This book isn’t going to get that deep. Instead, I just want to give you a feel for the process.
In Elixir, we version both the application code and the data it operates on. The two are independent—we might go for a dozen code releases without changing any data structures.
The code version is stored in the project dictionary in mix.exs. But how do we version the data? Come to think of it, where do we even define the data?
In an OTP application, all state is maintained by servers, and each server’s state is independent. So it makes sense to version the app data within each server module. Perhaps a server initially holds its state in a two-element tuple. That could be version 0. Later, it is changed to hold state in a three-element tuple. That could be version 1.
We’ll see the significance of this later. For now, let’s just set the version of the state data in our server. We use the @vsn (version) directive:
| | defmodule Sequence.Server do |
| | use GenServer |
| | |
| | @vsn "0" |
Now let’s generate a release.
First, we have to add distillery as a project dependency. Open the sequence project’s mix.exs file and update the deps function.
| | defp deps do |
| | [ |
| | {:distillery, "~> 1.5", runtime: false}, |
| | ] |
| | end |
(The runtime: false option tells mix that Distillery is not to be started with the running application.)
Remember to install the dependency:
| | $ mix do deps.get, deps.compile |
Distillery makes sensible choices for the various configuration options, so for a basic app like this we’re now ready to create our first release. To start off we create the release configuration:
| | $ mix release.init |
| | |
| | An example config file has been placed in rel/config.exs, review it, |
| | make edits as needed/desired, and then run `mix release` to build |
| | the release |
If you’re following along at home, have a look at rel.config.exs (it’s too large to show here). The defaults look good, so let’s build the actual release. We’ll tell it we want a production version of the release. If we don’t, the release it generates will be for development, and won’t be a self-contained package.
| | $ mix release --env=prod |
| | ==> Assembling release.. |
| | ==> Building release sequence:0.1.0 using environment prod |
| | ==> Including ERTS 9.1 from /usr/local/Cellar/erlang/20.1/lib/erlang/erts-9.1 |
| | ==> Packaging release.. |
| | ==> Release successfully built! |
| | You can run it in one of the following ways: |
| | Interactive: _build/dev/rel/sequence/bin/sequence console |
| | Foreground: _build/dev/rel/sequence/bin/sequence foreground |
| | Daemon: _build/dev/rel/sequence/bin/sequence start |
Distillery got the application name and version number from your mix.exs file, and packaged your app into the _build/dev/rel/ directory:
| | _build/dev/rel |
| | └── sequence |
| | ├── bin « global scripts |
| | │ ├── nodetool |
| | │ ├── release_utils.escript |
| | │ ├── sequence |
| | │ ├── sequence.bat |
| | │ ├── sequence_loader.sh |
| | │ └── start_clean.boot |
| | ├── erts-9.1 « the runtime (Erlang + Elixir) |
| | │ ├── . . . |
| | │ ├── elixir-1.5.2 |
| | │ ├── sequence-0.1.0 « our compiled application |
| | │ │ ├── consolidated |
| | │ │ │ ├── Elixir.Collectable.beam |
| | │ │ │ ├── Elixir.Enumerable.beam |
| | │ │ │ ├── Elixir.IEx.Info.beam |
| | │ │ │ ├── Elixir.Inspect.beam |
| | │ │ │ ├── Elixir.List.Chars.beam |
| | │ │ │ └── Elixir.String.Chars.beam |
| | │ │ └── ebin |
| | │ │ ├── Elixir.Sequence.Application.beam |
| | │ │ ├── Elixir.Sequence.Server.beam |
| | │ │ ├── Elixir.Sequence.Stash.beam |
| | │ │ ├── Elixir.Sequence.beam |
| | │ │ └── sequence.app |
| | │ └── stdlib-3.4.2 |
| | └── releases « release specific stuff |
| | ├── 0.1.0 « our initial release |
| | │ ├── commands |
| | │ ├── hooks |
| | │ ├── . . . |
| | │ ├── libexec |
| | │ ├── . . . |
| | │ ├── sequence.bat |
| | │ ├── sequence.boot |
| | │ ├── sequence.rel |
| | │ ├── sequence.script |
| | │ ├── sequence.sh |
| | │ ├── sequence.tar.gz « the packaged release |
| | │ ├── start_clean.boot |
| | │ ├── sys.config |
| | │ └── vm.args |
| | ├── RELEASES |
| | └── start_erl.data |
The most important file is rel/sequence/releases/0.0.1/sequence.tar.gz. It contains everything needed to run this release. This is the file we deploy to our servers.
I don’t want to slow things down by having you provision a server in the cloud, so I’m going to deploy to my local machine. However, to make it a little more realistic, I’ll pretend this machine is remote, and use ssh to do all the deploying. I’ll also be creating directories and copying files manually. In practice, you’d want to automate all of this with something like Capistrano or Ansible.
We’ll store the releases in a deploy directory. I’ll put this inside my home directory—feel free to put it anywhere (writable) you want.
| | $ ssh localhost mkdir ~/deploy |
Now we need to set up the initial release and its directory structure. Copy the sequence.tar.gz file into the deploy directory, and then extract its contents.
| | $ scp _build/dev/rel/sequence/releases/0.1.0/sequence.tar.gz localhost:deploy |
| | $ ssh localhost tar -x -f ~/deploy/sequence.tar.gz -C ~/deploy |
The app is now ready to run. The scripts in deploy/bin control it. These, in turn, delegate to scripts in the current release directory (all on the server).
Let’s start an IEx console. (The ssh -t option lets us control the remote IEx with ^C and ^G.)
| | $ ssh -t localhost ~/deploy/bin/sequence console |
| | Using /Users/dave/deploy/releases/0.0.1/sequence.sh |
| | Interactive Elixir (1.x) - press Ctrl+C to exit (type h() ENTER for help) |
| | |
| | iex(sequence@127.0.0.1)2> Sequence.Server.next_number |
| | 456 |
| | iex(sequence@127.0.0.1)3> Sequence.Server.next_number |
| | 457 |
(Leave this session running—we’ll use it to demonstrate hot code reloading.)
Our marketing team ran a focus group. It seems our customers want the next_number function to return a message like “the next number is 458.”
First we’ll change server.ex:
| | def next_number do |
| | with number = GenServer.call(__MODULE__, :next_number), |
| | do: "The next number is #{number}" |
| | end |
Then we’ll bump the application’s version number in mix.exs.
| | def project do |
| | [ |
| | app: :sequence, |
| » | version: "0.2.0", |
| » | elixir: "~> 1.6-dev", |
| » | start_permanent: Mix.env() == :prod, |
| » | deps: deps() |
| » | ] |
| » | end |
(We don’t have to change the @vsn value—the representation of the server’s state is not affected by this change.)
After exhaustive testing, we decide we’re ready to create a new release. Here we have a choice. If we just run mix release we’ll create a whole new releasable application. To deploy it, we’d basically copy it just as we did before, then stop the old app and start the new one.
The alternative is to deploy an upgrade release (sometimes called a hot upgrade). This is code that will upgrade the application while it is still running—there should be no downtime. This is one reason why Elixir apps can achieve such high availability numbers. Let’s take the upgrade path:
| | $ mix release --env=prod --upgrade |
| | ==> Assembling release.. |
| | ==> Building release sequence:0.2.0 using environment prod |
| | ==> Including ERTS 9.1 from /usr/local/Cellar/erlang/20.1/lib/erlang/erts-9.1 |
| » | ==> Generated .appup for sequence 0.1.0 -> 0.2.0 |
| | ==> Relup successfully created |
| | ==> Packaging release.. |
| | ==> Release successfully built! |
| | You can run it in one of the following ways: |
| | Interactive: _build/dev/rel/sequence/bin/sequence console |
| | Foreground: _build/dev/rel/sequence/bin/sequence foreground |
| | Daemon: _build/dev/rel/sequence/bin/sequence start |
The key thing to note is the creation of the .appup file. This is what tells the Erlang runtime how to upgrade our running app.
The deployment of the first release of an app is special: it has to create an environment for that app. With that in place, this release (and all subsequent releases) will be slightly different. We have to create a release directory on the server and copy the tarball into it. The directory will be under deploy/releases, and will be named the same as the release’s version number.
| | $ ssh localhost mkdir deploy/releases/0.2.0 |
| | $ scp _build/dev/rel/sequence/releases/0.2.0/sequence.tar.gz \ |
| | localhost:deploy/releases/0.2.0 |
Now let’s upgrade the running code:
| | $ ssh localhost ~/deploy/bin/sequence upgrade 0.2.0 |
| | Release 0.2.0 not found, attempting to unpack releases/0.2.0/sequence.tar.gz |
| | Unpacked successfully: "0.2.0" |
| | Release 0.2.0 is already unpacked, now installing. |
| | Installed Release: 0.2.0 |
| | Made release permanent: "0.2.0" |
Head back over to the terminal session that’s talking to the app. Don’t restart it—just make another request:
| | iex(sequence@127.0.0.1)4> Sequence.Server.next_number |
| | "The next number is 458" |
| | iex(sequence@127.0.0.1)5> Sequence.Server.next_number |
| | "The next number is 459" |
Erlang can actually run two versions of a module at the same time. Currently executing code will continue to use the old version until that code explicitly cites the name of the module that has changed. At that point, and for that particular process, execution will swap to the new version.
This is a critical part of hot loading of code. We want to let code that is currently running continue without interruption, but the new release may not be compatible with it. So Erlang lets it run on the old release. But the next request will reference the module explicitly, and the new code will be loaded.
Here, when we say Sequence.Server.next_number, the reference to Sequence.Server triggers the reload, so the 0.2.0 release handles the next request.
What if our new release was a disaster? That’s not a problem—we can always downgrade to a previous version.
| | $ ssh localhost ~/deploy/bin/sequence downgrade 0.1.0 |
| | Release 0.1.0 is already unpacked |
| | Release 0.1.0 is marked old, switching to it. |
| | Installed Release: 0.1.0 |
| | Made release permanent: "0.1.0" |
| | Warning: "/Users/dave/deploy/releases/0.0.1/relup" missing (optional) |
| | |
| | iex(sequence@127.0.0.1)6> Sequence.Server.next_number |
| | 460 |
| | iex(sequence@127.0.0.1)7> Sequence.Server.next_number |
| | 461 |
Cool. Let’s go back to the current version before continuing.
| | $ ssh localhost ~/deploy/bin/sequence upgrade 0.2.0 |
Our boss calls. We’re about to go for a second round of funding for our wildly successful sequence-server business, but customers have noticed a bug. We implemented increment_number to add a delta to the current number—a one-time change. But apparently it was instead supposed to set the difference between successive numbers we served.
Let’s try the existing code in our already-running console:
| | iex(sequence@127.0.0.1)8> Sequence.Server.next_number |
| | The next number is 462 |
| | iex(sequence@127.0.0.1)9> Sequence.Server.increment_number 10 |
| | :ok |
| | iex(sequence@127.0.0.1)10> Sequence.Server.next_number |
| | The next number is 472 |
| | iex(sequence@127.0.0.1)10> Sequence.Server.next_number |
| | The next number is 473 |
Yup, we’re applying the delta only once.
Well, that’s an easy change to the code. We simply have to keep one extra thing in the state—a delta value. Here’s the updated server code:
| | defmodule Sequence.Server do |
| | use GenServer |
| | require Logger |
| | |
| | @vsn "1" |
| | |
| | defmodule State do |
| | defstruct(current_number: 0, delta: 1) |
| | end |
| | |
| | ##### |
| | # External API |
| | |
| | def start_link(_) do |
| | GenServer.start_link(__MODULE__, nil, name: __MODULE__) |
| | end |
| | |
| | def next_number do |
| | with number = GenServer.call(__MODULE__, :next_number), |
| | do: "The next number is #{number}" |
| | end |
| | |
| | def increment_number(delta) do |
| | GenServer.cast __MODULE__, {:increment_number, delta} |
| | end |
| | |
| | ##### |
| | # GenServer implementation |
| | |
| | def init(_) do |
| | state = %State{ current_number: Sequence.Stash.get() } |
| | { :ok, state } |
| | end |
| | |
| | def handle_call(:next_number, _from, state = %{current_number: n}) do |
| | { :reply, n, %{state | current_number: n + state.delta} } |
| | end |
| | |
| | def handle_cast({:increment_number, delta}, state) do |
| | { :noreply, %{state | delta: delta }} |
| | end |
| | |
| | def terminate(_reason, current_number) do |
| | Sequence.Stash.update(current_number) |
| | end |
| | |
| | end |
The big change is that we made the state a struct rather than a tuple and added the delta value. We updated the increment handler to change the value of delta, and the next number handler now adds in the delta each time.
The format of the state changed, so we updated the version number (@vsn) to 1.
If we simply stop the old server and start the new one, we’ll lose the state stored in the old one. But we can’t just copy the state across—the old server had a single integer and the new one has a struct.
Fortunately, OTP has a callback for this. In the new server, implement the code_change function.
| | def code_change("0", old_state = current_number, _extra) do |
| | new_state = %State{ |
| | current_number: current_number, |
| | delta: 1 |
| | } |
| | Logger.info "Changing code from 0 to 1" |
| | Logger.info inspect(old_state) |
| | Logger.info inspect(new_state) |
| | { :ok, new_state } |
| | end |
The callback takes three arguments—the old version number, the old state, and an additional parameter we don’t use. The callback’s job is to return {:ok, new_state}. In our case, the new state is a struct containing the stash PID and the old current number, along with the new delta value, initialized to 1. We’ll need to bump the version number in mix.exs.
| | def project do |
| | [ |
| | app: :sequence, |
| » | version: "0.3.0", |
| | elixir: "~> 1.6-dev", |
| | start_permanent: Mix.env() == :prod, |
| | deps: deps() |
| | ] |
| | end |
Time to create the new release:
| | mix release --env=prod --upgrade |
| | ==> Assembling release.. |
| | ==> Building release sequence:0.3.0 using environment prod |
| | ==> Including ERTS 9.1 from /usr/local/Cellar/erlang/20.1/lib/erlang/erts-9.1 |
| | ==> Generated .appup for sequence 0.2.0 -> 0.3.0 |
| | ==> Relup successfully created |
| | ==> Packaging release.. |
| | ==> Release successfully built! |
| | You can run it in one of the following ways: |
| | Interactive: _build/dev/rel/sequence/bin/sequence console |
| | Foreground: _build/dev/rel/sequence/bin/sequence foreground |
| | Daemon: _build/dev/rel/sequence/bin/sequence start |
Copy it into the deployment location:
| | $ ssh localhost mkdir ~/deploy/releases/0.3.0/ |
| | $ scp _build/dev/rel/sequence/releases/0.3.0/sequence.tar.gz \ |
| | localhost:deploy/releases/0.3.0/ |
Cross your fingers, and upgrade the app:
| | $ ssh localhost ~/deploy/bin/sequence upgrade 0.3.0 |
| | Release 0.3.0 not found, attempting to unpack releases/0.3.0/sequence.tar.gz |
| | Unpacked successfully: "0.3.0" |
| | Release 0.3.0 is already unpacked, now installing. |
| | Installed Release: 0.3.0 |
| | Made release permanent: "0.3.0" |
But the real magic happened over in the console window:
| | 16:03:12.096 [info] Changing code from 0 to 1 |
| | 16:03:12.096 [info] 459 |
| | 16:03:12.096 [info] %Sequence.Server.State{current_number: 459, delta: 1} |
That’s the logging we added to our code_change function. We seem to have migrated the server’s state into our new structure. Let’s try it out:
| | iex(sequence@127.0.0.1)10> Sequence.Server.next_number |
| | "The next number is 459" |
| | iex(sequence@127.0.0.1)11> Sequence.Server.increment_number 10 |
| | :ok |
| | iex(sequence@127.0.0.1)13> Sequence.Server.next_number |
| | "The next number is 460" |
| | iex(sequence@127.0.0.1)14> Sequence.Server.next_number |
| | "The next number is 470" |
That’s the new behavior, running with our new state structure. We updated the code twice and migrated data once, all while the application continued to run. There was no service disruption, and no loss of data.
Plutarch records the story of a ship called Theseus. Over the course of many years most of the ship’s structure was replaced, piece by piece. While this was happening, the ship was in continuous use. Plutarch raises the question, “Is the renovated Theseus the same as the original?”
Using Elixir release management, our applications can work the same way the Theseus did, running continuously but being updated all the time.
Is the latest application the same as the original? Who cares, as long as it’s still running?