At this point, it might seem like you have all you need to create process-oriented projects with Erlang. You know how to create useful functions, can work with recursion, know the data structures Erlang offers, and probably most important, know how to create and manage processes. What more could you need?
Process-oriented programming is great, but the details matter. The basic Erlang tools are powerful, but can also lead you into frustrating mazes debugging race conditions that happen only once in a while. Mixing different programming styles can lead to incompatible expectations, and code that worked well in one environment may prove harder to integrate in another.
Ericsson encountered these problems early, and created a library of code that eases them. OTP, originally the Open Telecom Platform, is useful for pretty much any large-scale project you want to do with Erlang, not just telecom work. It’s included with Erlang, and though it isn’t precisely part of the language, it is definitely part of the Erlang environment and helps to define Erlang programming culture. The boundaries of where Erlang ends and OTP begins aren’t always clear, but the entry point is definitely behaviors. Your applications will combine processes built with behaviors and managed by supervisors in an OTP application.
So far, the lifecycle of the processes shown in the previous chapters has been pretty simple. If needed, they set up other resources or processes to get started. Once running, they listen for messages and process them, collapsing if they fail. Some of them might restart a failed process if needed.
OTP formalizes those activities, and a few more, into a set of behaviors (or behaviours—OTP was originally created with the British spelling). The most common behaviors are gen_server (generic server) and supervisor. gen_statem (state machine), gen_fsm (finite state machine), and gen_event are also available. The application behavior lets you package your OTP code into a single runnable (and updatable) system.
The behaviors predefine the mechanisms your code will use to create and interact with processes, and the compiler will warn you if you’re missing some of them. Your code needs to handle the callbacks, specifying how to respond to particular kinds of events. OTP offers you some choices about how to structure your application as well.
If you’d like a free one-hour video introduction to OTP, see Steve Vinoski’s “Erlang’s Open Telecom Platform (OTP) Framework”. You probably already know the first half hour or so of it, but the review is excellent. In a very different style, if you’d like an explanation of why it’s worth learning OTP and process-oriented development in general, Francesco Cesarini’s slides work even without narration, especially the second half.
Much of the work you think of as the core of a program—calculating results, storing information, and preparing replies—will fit neatly into the gen_server behavior. It provides a core set of methods that let you set up a process, respond to requests, end the process gracefully, and even pass state to a new process if this one needs to be upgraded in place.
Table 11-1 shows the methods you need to implement in a service that uses the gen_server behavior. For a simple service, the first two or three are the most important, and you may just use placeholder code for the rest.
| Method | Triggered by | Does |
|---|---|---|
|
|
Sets up the process |
|
|
Handles synchronous calls |
|
|
Handles asynchronous calls |
|
random messages |
Deals with non-OTP messages |
|
failure or shutdown signal from supervisor |
Cleans up the process |
|
system libraries for code upgrades |
Lets you switch out code without losing state |
Appendix B shows a complete gen_server template from the Erlang emacs-mode, which is worth exploring in particular for the models it offers for the return value. (In this context, a template is just a file full of code you can use as a base for creating your own code.) However, it’s pretty big. Example 11-1, which you can find in ch11/ex1-drop, shows a less verbose example (based on the template) that you can use to get started. It mixes a simple calculation from way back in Example 2-1 with a counter like that in Example 8-4.
-module(drop).-behaviour(gen_server).-export([start_link/0]).% convenience call for startup-export([init/1,handle_call/3,handle_cast/2,handle_info/2,terminate/2,code_change/3]).% gen_server callbacks-define(SERVER,?MODULE).% macro that just defines this module as server-record(state,{count}).% simple counter state%%% convenience method for startupstart_link()->gen_server:start_link({local,?SERVER},?MODULE,[],[]).%%% gen_server callbacksinit([])->{ok,#state{count=0}}.handle_call(_Request,_From,State)->Distance=_Request,Reply={ok,fall_velocity(Distance)},NewState=#state{count=State#state.count+1},{reply,Reply,NewState}.handle_cast(_Msg,State)->io:format("So far, calculated~wvelocities.~n",[State#state.count]),{noreply,State}.handle_info(_Info,State)->{noreply,State}.terminate(_Reason,_State)->ok.code_change(_OldVsn,State,_Extra)->{ok,State}.%%% Internal functionsfall_velocity(Distance)->math:sqrt(2*9.8*Distance).
The module name (drop) should be familiar from past examples. The second line is a -behaviour declaration specifying that this is going to be using the gen_server behavior. That declaration tells Erlang that it can expect this code to support the core callback functions of that behavior.
You can spell the -behaviour declaration -behavior if you prefer the American version. Erlang doesn’t mind.
The -export declarations are pretty standard, though they break out the start_link/0 method into a separate declaration from the core gen_server methods. This isn’t necessary, but it’s a nice reminder that start_link isn’t required for the gen_server behavior to work. (It calls gen_server code, but isn’t a callback itself.)
The -define declaration is probably unfamiliar. Erlang lets you declare macros using -define. Macros are simple text replacements. This declaration tells the compiler that any time it encounters ?SERVER, it should replace it with ?MODULE. What is ?MODULE? That’s a built-in macro that always refers to the name of the module it appears in. In this case, that means it will be processed into drop. (You may find cases where you want to register the server under a name other than the module name, but this is a workable default.)
The -record declaration should be familiar, though it contains only one field, to keep a count of the number of calls made. Many services will have more fields, including things like database connections, references to other processes, perhaps network information, and metadata specific to this particular service. It is also possible to have services with no state, which would be represented by an empty tuple here. As you’ll see further down, every single gen_server function will reference the state.
The state record declaration is a good example of a record declaration you should make inside of a module and not declare through an included file. It is possible that you’ll want to share state models across different gen_server processes, but it’s easier to see what State should contain if the information is right there.
The first function in the sample, start_link/0, is not one of the required gen_server functions. Instead, it calls the gen_server’s start_link function to start up the process. When you’re just getting started, this is useful for testing. As you move toward production code, you may find it easier to leave these out and use other mechanisms.
The start_link/0 function uses the ?SERVER macro defined in the -define declaration as well as the built-in ?MODULE declaration.
%%% convenience method for startupstart_link()->gen_server:start_link({local,?SERVER},?MODULE,[],[]).
The first argument, a tuple, opens with an atom that must be local or global, depending on whether you want the name of the process registered just with the local Erlang instance or with all associated nodes. The ?SERVER macro will be expanded to ?MODULE, which will itself be expanded to the name of the current module, and that will be used as the name for this process. The second argument is the name of the module, here identified with the ?MODULE macro, and then lists for arguments and options follow. In this case, they’re both empty. Options can specify things like debugging, timeouts, and options for spawning the process.
You may also see a form of gen_server:start_link with via as the atom in the first tuple. This lets you set up custom process registries, of which gproc is the best known. For more on that, see https://github.com/uwiger/gproc.
All of the remaining functions are part of the gen_server behavior. init/1 creates a new state record instance and sets its count field to zero—no velocities have yet been calculated. The two functions that do most of the work here are handle_call/3 and handle_cast/2. For this demonstration, handle_call/3 expects to receive a distance in meters and returns a velocity for a fall from that height on Earth, while handle_cast/2 is a trigger to report the number of velocities calculated.
handle_call/3 makes synchronous communications between Erlang processes simple.
handle_call(_Request,_From,State)->Distance=_Request,Reply={ok,fall_velocity(Distance)},NewState=#state{count=State#state.count+1},{reply,Reply,NewState}.
This extracts the Distance from the _Request, which isn’t necessary except that I wanted to leave the variable names for the function the same as they were in the template. (handle_call(Distance, _From, State) would have been fine.) Your _Request is more likely to be a tuple or a list rather than a bare value, but this works for simple calls.
It then creates a reply based on sending that Distance to the simple fall_velocity/1 function at the end of the module. It then creates a NewState containing an incremented count. Then the atom reply, the Reply tuple containing the velocity, and the NewState containing the count get passed back.
Because the calculation is really simple, treating the drop as a simple synchronous call is perfectly acceptable. For more complex situations where you can’t predict how long a response might take, you may want to consider responding with a noreply response and using the _From argument to send a response later. (There is also a stop response available that will trigger the terminate/2 method and halt the process.)
By default, OTP will time out any synchronous calls that take longer than 5 seconds to calculate. You can override this by making your call using gen_server:call/3 to specify a timeout (in milliseconds) explicitly, or by using the atom infinity.
The handle_cast/2 function supports asynchronous communications. It isn’t supposed to return anything directly, though it does report noreply (or stop) and updated state. In the following example, it takes a very weak approach, but one that does well for a demonstration, calling io:format/2 to report on the number of calls:
handle_cast(_Msg,State)->io:format("So far, calculated~wvelocities.~n",[State#state.count]),{noreply,State}.
The state doesn’t change, because asking for the number of times the process has calculated a fall velocity is not the same thing as actually calculating a fall velocity.
Until you have good reason to change them, you can leave handle_info/2, terminate/2, and code_change/3 alone.
Making a gen_server process run and calling it looks a little different than starting the processes you saw in Chapter 8. Be very careful as you type this in: mistakes, as you’ll see soon, can have unexpected effects:
1>c(drop).{ok,drop}2>drop:start_link().{ok,<0.33.0>}3>gen_server:call(drop,20).{ok,19.79898987322333}4>gen_server:call(drop,40).{ok,28.0}5>gen_server:call(drop,60).{ok,34.292856398964496}6>gen_server:cast(drop,{}).So far, calculated 3 velocities.ok
The call to drop:start_link() sets up the process and makes it available. Then, you’re free to use gen_server:call or gen_server:cast to send it messages and get responses.
While you can capture the pid, you don’t have to keep it around to use the process. Because start_link returns a tuple, if you want to capture the pid you can do something like {ok, Pid} = drop:start_link().
Because of the way OTP calls gen_server functions, there’s an additional bonus—or perhaps a hazard—in that you can update code on the fly. For example, I tweaked the fall_velocity/1 function to lighten Earth’s gravity a little, using 9.1 as a constant instead of 9.8. Recompiling the code and asking for a velocity returns a different answer:
7>c(drop).{ok,drop}8>gen_server:call(drop,60).{ok,33.04542328371661}
This can be very convenient during the development phase, but be careful doing anything like this on a production machine. OTP has other mechanisms for updating code on the fly. There is also a built-in limitation to this approach: init gets called only when start_link sets up the service. It does not get called if you recompiled the code. If your new code requires any changes to the structure of its state, your code will break the next time it’s called.
When you started the drop module from the shell, you effectively made the shell the supervisor for the module (though the shell doesn’t really do any supervision). You can break the module easily:
9>gen_server:call(drop,-60).=ERROR REPORT==== 2-Dec-2012::21:14:51 ===** Generic server drop terminating** Last message in was -60** When Server state == {state,0}** Reason for termination ==** {badarith,[{math,sqrt,[-1176.0],[]},{drop,fall_velocity,1,[{file,"drop.erl"},{line,42}]},{drop,handle_call,3,[{file,"drop.erl"},{line,23}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}** exception exit: badarithin function math:sqrt/1called as math:sqrt(-1176.0)in call from drop:fall_velocity/1 (drop.erl, line 42)in call from drop:handle_call/3 (drop.erl, line 23)in call from gen_server:handle_msg/5 (gen_server.erl, line 588)in call from proc_lib:init_p_do_apply/3 (proc_lib.erl, line 227)10>gen_server:call(drop,60).** exception exit: {noproc,{gen_server,call,[drop,60]}}in function gen_server:call/2 (gen_server.erl, line 180)
The error message is nicely complete, even telling you the last message and the state, but when you go to call the service again on line 10, it isn’t there. You can restart it with drop:start_link/0 again, but you’re not always going to be watching your processes personally.
Instead, you want something that can watch over your processes and make sure they restart (or not) as appropriate. OTP formalizes the process management you saw in Example 8-10 with its supervisor behavior. Example B-2 in Appendix B shows a full template (again, from the Erlang mode for Emacs), but you can create a less verbose supervisor.
A basic supervisor needs to support only one callback function, init/1, and can also have a start_link function to fire it up. The return value of that init/1 function tells OTP which child processes your supervisor manages and how you want to handle their failures. A supervisor for the drop module might look like Example 11-2, which is in ch11/ex2-drop-sup.
-module(drop_sup).-behaviour(supervisor).-export([start_link/0]).% convenience call for startup-export([init/1]).% supervisor calls-define(SERVER,?MODULE).%%% convenience method for startupstart_link()->supervisor:start_link({local,?SERVER},?MODULE,[]).%%% supervisor callbackinit([])->SupFlags=#{strategy=>one_for_one,intensity=>1,period=>5},Drop=#{id=>'drop',start=>{'drop',start_link,[]},restart=>permanent,shutdown=>5000,type=>worker,modules=>['drop']},{ok,{SupFlags,[Drop]}}.%%% Internal functions (none here)
The init/1 function’s job is to assemble a fairly complex data structure, held in two maps.
The first map defined in the template, SupFlags, defines how the supervisor should handle failure. The strategy of one_for_one tells OTP that it should create a new child process every time a process that is supposed to be permanent fails. You can also go with one_for_all, which terminates and restarts all of the processes the supervisor oversees when one fails, or rest_for_one, which restarts the process and any processes that began after the failed process started. There’s also a simple_one_for_one optimized for the case where all child processes run identical code.
When you’re ready to take more direct control of how your processes respond to their environment, you might explore working with the dynamic functions supervisor:start/2, supervisor:terminate_child/2, supervisor:restart_child/2, and supervisor:delete_child/2, as well as the restart strategy simple_one_for_one.
The next two values define how often the worker processes can crash before terminating the supervisor itself. In this case, the default is one (intensity) restart every 5 (period) seconds. Customizing these values lets you handle a variety of conditions, but probably won’t affect you much initially.
Those values, which here get combined into the map contained in SupFlags, apply to all of the workers managed by this supervisor. The next few lines define properties that apply to only one worker process, in this case the gen_server specified by Drop. It is designed to be a permanent service, so the supervisor should restart it when it fails. The supervisor can wait 2 seconds before shutting it off completely, and this worker is only a worker, not itself a supervisor. More complex OTP applications can contain trees of supervisors managing other supervisors, which themselves manage other supervisors or workers.
The Drop map might seem a bit repetitive, but it creates a complete set of information to get the Drop process started. First, it specifies an id, and then a tuple containing the name of the module containing the code, the function to use to start the process and a list of arguments. (Here there aren’t any arguments.) Then the restart, shutdown, and type are specified, and the final modules list identifies all the modules on which this process will depend. In this case, it all fits into a single module, so the list contains only the name of that module.
OTP wants to know the dependencies so that it can help you upgrade software in place. It’s all part of the magic of keeping systems running without ever bringing them to a full stop.
Now that you have a supervisor process, you can set up the drop function by just calling the supervisor. However, running a supervisor from the shell using the start_link/0 function call creates its own set of problems; the shell is itself a supervisor, and will terminate processes that report errors. After a long error report, you’ll find that both your worker and the supervisor have vanished.
In practice, this means that there are two ways to test supervised OTP processes (that aren’t yet part of an application) directly from the shell. The first explicitly breaks the bond between the shell and the supervisor process by catching the pid of the supervisor (line 2) and then using the unlink/1 function to remove the link (line 3). Then you can call the process as usual with gen_server:call/2 and get answers. If you get an error (line 6), it’ll be okay. The supervisor will restart the worker, and you can make new calls (line 7) successfully. The calls to whereis(drop) on lines 5 and 8 demonstrate that the supervisor has restarted drop with a new pid.
1>c(drop_sup).{ok,drop_sup}2>{ok, Pid} = drop_sup:start_link().{ok,<0.80.0>}3>unlink(Pid).true4>gen_server:call(drop,60).{ok,34.292856398964496}5>whereis(drop).<0.81.0>6>gen_server:call(drop,-60).** exception exit: {{badarith,[{math,sqrt,[-1176.0],[]},{drop,fall_velocity,1,[{file,"drop.erl"},{line,42}]},{drop,handle_call,3,[{file,"drop.erl"},{line,23}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server,call,[drop,-60]}}in function gen_server:call/2 (gen_server.erl, line 204)7>=ERROR REPORT==== 16-Jan-2017::13:42:32 ===** Generic server drop terminating** Last message in was -60** When Server state == {state,1}** Reason for termination ==** {badarith,[{math,sqrt,[-1176.0],[]},{drop,fall_velocity,1,[{file,"drop.erl"},{line,42}]},{drop,handle_call,3,[{file,"drop.erl"},{line,23}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}7>gen_server:call(drop,60).{ok,34.292856398964496}8>whereis(drop).<0.86.0>
The other approach leaves the link in place, but wraps the calls to gen_server/2 in a catch statement. In this case, using catch just keeps the shell from ever receiving the exception, so the supervisor remains untouched. You don’t have to use catch to make a call, as line 8 shows, but if the call fails, you’ll have to restart the supervisor process yourself. (Line 6 is also a bit split by the error message. Sometimes the timing will make it look like the prompt disappeared. Don’t worry, it hasn’t.)
1>c(drop_sup).{ok,drop_sup}2>drop_sup:start_link().{ok,<0.38.0>}3>whereis(drop).<0.39.0>4>catchgen_server:call(drop,60).{ok,34.292856398964496}5>5>catchgen_server:call(drop,-60).{'EXIT',{{badarith,[{math,sqrt,[-1176.0],[]},{drop,fall_velocity,1,[{file,"drop.erl"},{line,42}]},{drop,handle_call,3,[{file,"drop.erl"},{line,23}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},{gen_server,call,[drop,-60]}}}6>=ERROR REPORT==== 2-Dec-2012::21:21:10 ===** Generic server drop terminating** Last message in was -60** When Server state == {state,1}** Reason for termination ==** {badarith,[{math,sqrt,[-1176.0],[]},{drop,fall_velocity,1,[{file,"drop.erl"},{line,42}]},{drop,handle_call,3,[{file,"drop.erl"},{line,23}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}catch gen_server:call(drop, 60).{ok,34.292856398964496}7>whereis(drop).<0.43.0>8>gen_server:call(drop,60).{ok,34.292856398964496}
You can also tell the shell to stop worrying about such exceptions by issuing the shell command catch_exception(true). However, that turns off the behavior for the entire shell, which may not be what you want. (It will return false, the previous setting for that property. Don’t worry, it did set it to true.)
You can also open Process Manager or Observer and whack away at worker processes through the Kill option on the Trace menu and watch them reappear.
This works, but is only the tiniest taste of what supervisors can do. They can create child processes dynamically, and manage their lifecycles in greater detail. Experimenting with supervisors is the best way to learn about them.
OTP also lets you package sets of components into an application. While stopping and starting OTP workers and supervisors may be easier than dealing with processes directly, OTP’s facilities for describing applications will lead you down a path to much easier starting, updating, administering, and (if you must) stopping your projects.
Erlang applications include two extra components beyond the workers, supervisors, and related files they need. The application resource file, also known as an app file, provides a lot of metadata about your application. You’ll also need a module with the behavior application to define starting and stopping.
If you’re on a Mac, the file extension for the .app file will disappear and the operating system will think it’s some kind of broken Mac application. Don’t worry. It’ll still work in Erlang, though the Mac won’t know what to do if you double-click on it.
The app file is a large tuple but is easier to read than the one returned by a supervisor’s init/1 functions. Example 11-3, in ch11/ex3-drop-app, shows a minimal app file, placed in an ebin subdirectory, that sets up this simple drop application.
{application,drop,[{description,"Dropping objects from towers"},{vsn,"0.0.1"},{modules,[drop,drop_sup,drop_app]},{registered,[drop,drop_sup]},{applications,[kernel,stdlib]},{mod,{drop_app,[]}}]}.
The first line identifies this as an application named drop, and then a list of arguments provides more information:
The description is a (sometimes) human-friendly description of what’s here. vsn is a version number, which in this case is tiny.
modules lists the modules that make up the application, in this case drop, drop_sup, and drop_app.
registered lists modules that are publicly visible, again drop and drop_sup.
applications lists the required applications on which this application depends, and the kernel and stdlib seem to be the minimal standard set.
mod has a tuple that points to the module with the application behavior. It can take a list of arguments that will go to the start/2 function of the module, though there aren’t any here.
That module is trivial even compared to the other OTP code you’ve seen, as shown in Example 11-4, which is also in ch11/ex3-drop-app. (Example B-3 shows a fuller template.)
-module(drop_app).-behaviour(application).-export([start/2,stop/1]).start(_Type,_StartArgs)->drop_sup:start_link().stop(_State)->ok.
The only thing you really have to do is start up the supervisors for your application in the start/2 function. In this case there’s only one, and the _Type and _StartArgs don’t matter.
Running this application from the shell will require one bit of extra effort on your
part. You’ll need to compile drop_app, of course, but you’ll also need to tell Erlang about the ebin directory containing the drop.app file, as shown on line 2. (OTP expects it to be there, but will give you "no such file or directory" errors if you don’t tell Erlang about the directory.)
1>c(drop_app).{ok,drop_app}2>code:add_path("ebin/").true3>application:load(drop).ok4>application:loaded_applications().[{kernel,"ERTS CXC 138 10","2.15.2"},{drop,"Dropping objects from towers","0.0.1"},{stdlib,"ERTS CXC 138 10","1.18.2"}]5>application:start(drop).ok6>gen_server:call(drop,60).{ok,34.292856398964496}
Once Erlang knows where to look, you can use the application module’s functions to load the application and check that Erlang found it. Once you start the application, you can go ahead and make calls to it with gen_server:call. Because the supervisor is bound to an application, you don’t need to worry about the shell shutting you down. You can go ahead and break the drop calculation process with a negative value, and the supervisor will just fire it back up.
7>whereis(drop).<0.45.0>8>gen_server:call(drop,-60).=ERROR REPORT==== 2-Dec-2012::21:25:38 ===** Generic server drop terminating** Last message in was -60** When Server state == {state,1}** Reason for termination ==** {badarith,[{math,sqrt,[-1176.0],[]},{drop,fall_velocity,1,[{file,"drop.erl"},{line,42}]},{drop,handle_call,3,[{file,"drop.erl"},{line,23}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}** exception exit: {{badarith,[{math,sqrt,[-1176.0],[]},{drop,fall_velocity,1,[{file,"drop.erl"},{line,42}]},{drop,handle_call,3,[{file,"drop.erl"},{line,23}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},{gen_server,call,[drop,-60]}}in function gen_server:call/2 (gen_server.erl, line 180)9>gen_server:call(drop,60).{ok,34.292856398964496}10>whereis(drop).<0.49.0>
There is much, much more to learn. OTP deserves a book or several all on its own. Hopefully this chapter provides you with enough information to try some things out and understand those books. However, the gap between what this chapter can reasonably present and what you need to know to write solid OTP-based programs is… vast.
You can learn more about working with OTP basics in Chapters 11 and 12 of Erlang Programming (O’Reilly); Chapters 22 and 23 of Programming Erlang, 2nd Edition (Pragmatic); Chapter 4 of Erlang and OTP in Action (Manning); and Chapters 14 through 20 of Learn You Some Erlang For Great Good! (No Starch Press). You can move much deeper into OTP with Designing for Scalability with Erlang/OTP (O’Reilly).