When to spawn a process?

Process spawning model

One of the most popular mistakes in Erlang development is a wrong choice of process spawning model. Basically people spawn too many or too few processes. On internet there are many recommendations on when to spawn a process. One of them is "process per message", which encourages to spawn a process for each concurrent entity. It could be not completely clear or could even be misunderstood. An example below illustrates different variants of spawning model.

Public key encryption example

It's needed to build an Erlang application, which provides functionality of public key encryption. Encryption could take a significant time, and algorithm is implemented in pure Erlang. Clients of the application should call a function encrypt/1, providing a data to be encrypted as an argument. The application should manage keys without exposing that complexity to end users. Encryption key is stored in file on disk.

0 processes

The first naive implementation could be the following:

encrypt(Data) ->
  {ok, Key} = file:read_file(?KEY_FILE_PATH),
  encrypt(Key, Data).

On each encryption request key file is read and it's content is passed as argument to the algorithm together with data. Slow disk operation is probably something, we would like to minimize in our system.

1 process

The more advanced implementation is a gen_server, which caches the key in it's state in init/1 callback and does an encryption in handle_call/3.

encrypt(Data) ->
  gen_server:call(?SERVER, {data, Data}).

start_link() ->
  gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).

init([]) ->
  {ok, Key} = file:read_file(?KEY_FILE_PATH),
  {ok, #state{key = Key}}.

handle_call({data, Data}, _From, State) ->
  Enc = encrypt(State#state.key, Data),
  {reply, Enc, State}.

This approach eliminates the problem with constant disk reading, but might lead to the process's queue being overwhelmed by requests, because new encryption process can not be started until previous one is finished.

1+N processes

Some people improve performance of the previous example with spawning a new process, which does the actual encryption, in handle_call.

handle_call({data, Data}, From, State) ->
  erlang:spawn(fun() ->
    Enc  = encrypt(State#state.key, Data),
    gen_server:reply(From, Enc)
  end),
  {noreply, State}.

< That speeds things up, but leads to complicated and error-prone logic in the main "dispatcher" process, which should now monitor all other process it spawns as well as take care of reporting errors back to clients.

Pool of processes

Another alternative could be using one of pooling libraries, which organizes pre-allocated workers and takes care of task dispatching. Basically worker's code is the same as in example with single process besides that it should not be registered with name neither locally nor globally.

start_link() ->
  gen_server:start_link(?MODULE, [], []).

Again 1 process

But if we analyze the original task, we see, that we cache in state only the key. So the only thing, that needs to be done in handle_call is obtaining the key, and heavy encryption algorithm call could be moved to the context of client's process.

encrypt(Data) ->
  Key = gen_server:call(?SERVER, get_key),
  encrypt(Key, Data).

handle_call(get_key, _From, State) ->
  {reply, State#state.key, State}.

Getting a key from process's state is a relatively fast operation, thereby gen_server is not a bottle neck anymore.

Conclusion

Examples illustrate how good process spawning model can lead to efficient, simple and elegant code.
The rule of thumb for spawning could be the following:
A new process should be started to serialize an access to a shared resource. The resource could be cached memory, file descriptor(socket) and so on. People from C/C++ world can think about a process as about mutex. This rule might not fit all scenarios, but could be considered as a good starting point for design. It leads to elegant and highly concurrent code.
There are some exceptions of course. First of all, "let it crash" principle, which encourages to spawn a process for a code, which likely crashes.
Also people should not mix cached memory with memory for objects (in terms of OOP). For example, in online real-time strategy game there is a number of units, which belong to a specific user. Each unit stores in memory it's state, but developer should not spawn a process per unit. In fact shared resource in that case is user's session, which is represented by tcp socket. By analogy with OS mutex it's unlikely that synchronization primitive is needed for each unit in game.
Another exception could be a situation, when it's easier to describe some algorithm as Finite State Machine (FSM). It might be reasonable to spawn a process, which implements gen_fsm behaviour. 

Erlang enum

Rationale

One day each Erlang developer with C/C++ background tries to find mechanism similar to enum. Enum it's light-weight datatype with all possible values enumerated. There are multiple options of Erlang implementation.

Macros

Many developers just define macros with some meaningful name and unique (within enumeration) value.

-define(AUTH_ERROR, 404).
-define(FORMAT_ERROR, 400).
-define(INTERNAL_ERROR, 500).

Usage of such enums requires including header file with definition. Typo in macro's name would lead to compilation error.

Advanced developers even define a macro, which checks whether a value belongs to enumeration type.

-define(IS_ERROR(V), (
    V =:= ?AUTH_ERROR orelse
    V =:= ?FORMAT_ERROR orelse
    V =:= ?INTERNAL_ERROR
)).
handle_error(E) when ?IS_ERROR(E) ->
  case E of
    ?AUTH_ERROR -> abort();
    ?FORMAT_ERROR -> abort();
    ?INTERNAL_ERROR -> retry()
  end.

The problem with this approach is that uniqueness of values is not checked by compiler. One can easily add a new error into enumeration with value assigned to a different element of enumeration.

-define(NOT_FOUND, 404).

Obviously it can break logic which relies on the enum.

Atoms

A set of atoms can also be used as enumeration. Atoms are unique identifiers within entire system. Header file is not required for atom-enum.

 handle_error(E) ->
  case E of
    auth_error -> abort();
    format_error -> abort();
    internal_error -> retry()
end.

The only problem is to check whether a given atom is an element of enumeration. This issue arises if somebody makes a typo error in atom name, for example auth_eror instead of auth_error. One can try to resolve it by defining macro function, which checks if an atom belongs to enum. The macro could be used as a guard experession.

-define(IS_ERROR(V), (
  V =:= auth_error orelse
  V =:= format_error orelse
  V =:= internal_error
)).
handle_error(E) when ?IS_ERROR(E) ->
  ...,
  ok.
...
handle_error(auth_eror).

It again requires including header file to use IS_ERROR macro in different modules, and the issue could be only found on run-time. Error in atom would not lead to compilation error.
In addition to that IDE would not be able to do automatic refactoring such as renaming, because it's not safe to rename an atom in entire project, and scope can not be narrowed down due to global nature of atoms. Even "find usages" might return irrelevant results.

Records

There is one more non-obvious way to implement enum in Erlang. Just define a record, where each field is an element of enum.

-record(http_errors,{
  auth_error,
  format_error,
  internal_error
}).

Now each element of enumeration can be referenced as a record field's position.

handle_error(E) ->
  case E of
    #http_errors.auth_error -> abort();
    #http_errors.format_error -> abort();
    #http_errors.internal_error -> retry()
  end.
handle_error(#http_errors.auth_error).

Obviously all enumeration elements are unique since record field's position is unique, and it's guaranteed by compiler (compare with macros approach).
Any typo in field name will cause a compilation error (compare with atoms approach).
Automatic refactoring becomes easy for IDE, for example, renaming record's field is trivial. "Find usages" function has a well defined scope.
Of course, usage of record requires including header file with it's definition, but that is a fair price for compile-time checks it provides.
Since record field's position is just an integer, it can be used for defining external API. If record definition string ("-record(http_errors,{") is on the first line of a file, and each field is on new line, each enumeration element would have an integer value, which equals to line number in source code, where it's defined.

1. -record(http_errors,{
2.   auth_error,        %%authentication error
3.   format_error,      %%wrong json
4.   internal_error     %%connection to db server is lost
5. }).

For example, in http_errors enum auth_error element is on line 2, so it's integer values is 2. This value could be used as error code in RESTful API. It turns out that documentation is "generated" automatically.
Enum implementation using records is very close to C equivalent. It matches some verbose id with integer value. It automatically increments an interger value for each new element in enumeration. The only constraint is that integer value for the first element is always 2 and can not be changed.

Conclusion

To sum up, a comparison table is provided.

	Macro	Atom	Record
No include	-	+	-
Uniqueness	-	+	+
Compile time membership check	+	-	+
Automatic rename refactoring	+	-	+
Force value	+	-	-

Erlang record approach seems to be the best option for it compile-time uniqueness guarantees and membership check.

poolboy pitfall 2

The poolboy issue described in this article was successfully fixed, thereby in order to avoid it just make sure that you use at least version 1.5.1 of the library. Unfortunately another problem popped up.

Retries with poolboy

For some resources it is important to do retries within given time before reporting failure to the caller. From the first sight poolboy library provides all necessary features.

1. Internally it uses supervisor for restarting terminated processes (for retries);
2. Client can specify a timeout to wait for worker checkout.

So in worker's code one just needs to terminate a process if resource is not available and retries will be organised by poolboy.

handle_call(die, _From, State) ->
    {stop, {error, died}, dead, State}.

The issue

In fact the situation is a bit more complicated. Whenever gen_server callback returns a stop-tuple, it just instructs the underlying code in OTP to terminate the process, it is not stated in documentation, if caller gets response first and then the process terminates or vice versa. In addition to that poolboy's supervisor is notified about worker's termination, which is not reported to the caller without additional link/monitor.

So processes of worker's termination and checking out from the pool are not synchronized, as a result poolboy:checkout function might return a Pid of worker, which is already terminated. Further usage of the Pid will lead to exception exit: {noproc, ...}. Obviously the same could happen using poolboy:transaction function.

Client can handle this error by reporting failure, but that does not fulfill the requirement of retries for given time period.

The issue is reported in best traditions of TDD as a pull request with failing tests.

Workarounds

Since it was not fixed quickly the issue seems to be quite fundamental. In order to continue using poolboy some workaround is required.

The issue popped up, when I tried to organise retry logic by means of poolboy. An obvious workaround for this would be moving this logic to either worker or client, but my perfectionism did not allow me to expose such a complexity to that level.

Fortunately, commiters of the project advised me a simple technique to overcome the problem. Worker should check-in itself back to the pool in case of success, so that termination and check-in are synchronised.

handle_call(die, _From, State) ->
    {stop, {error, died}, dead, State};
handle_call(ok, _From, State) ->
    poolboy:checkin(pool_name, self()),
    {reply, ok, State}.

This trick implies, that poolboy:transaction can not be used anymore. It also breaks separation of concerns and abstraction, because worker starts "knowing" about the pool. But overall I find it as a "good deal" comparing to other workarounds for it's simplicity.

OOP in Erlang. Part 2. Polymorphism

Encapsulation was covered in previous article of my OOP in Erlang series.

Polymorphism

Polymorphism is ability to present the same interface for different instances (types).

Processes

First of all, process is a unit of encapsulation in Erlang, polymorphism could be implemented on that level.
Client can send the same message to different processes, which will handle the message differently.
For example, we can implement gen_server behaviour in two modules.
serv1.erl

handle_call(message, _From, State) ->
  {reply, 1, State}.

serv2.erl

handle_call(message, _From, State) ->
  {reply, 2, State}.

Then client chooses one of gen_servers and starts it, saving Pid of the process. And then gen_server:call(Pid, message) can be called and caller experiences different behaviour based on module chosen in the beginning.

Dynamic types

Erlang is dynamically typed language, as a result module and even function name could be taken from variable during function call. For example, interfaces of dict and orddict from OTP are unified, and following code shows polymorphism implementation.

Module =
       case Ordering of
           unordered ->
               dict;
           ordered ->
               ordict
       end,
Container = Module:from_list(Values),
Module:find(Key, Container).

Pattern matching

Pattern matching is a powerful feature of the language, which can be used for polymorphism implementation. The idea is that function changes it's behaviour depending on arguments it gets. The most obvious data type to match on is record.

move(#duck{}) -> move_duck();
move(#fish{}) -> move_fish().

Extensive usage of pattern matching for that purpose leads to decoupling of "object" from it's behaviour (business logic), as a result for adding of a new "type" changes in multiple modules are required. This is very similar to Anemic domain model, which has some advantages though.

Conclusion

Polymorphism in Erlang could me implemented in different ways. Even though processes provide the most clear interface for that, developer should not create processes only for modeling business logic, because:

1. Logic changes quite often and changing all boilerplate code of processes is an overhead.
2. System becomes more complicated with each new type of process spawned.

Resource management idioms in Erlang

Resource management problem

Resource management is an important question in any programming language. It stays important even if runtime provides garbage collector, because there are other resource besides memory. Resource is something, what client has to initialise and later clean up, for example, socket, file descriptor or connection to RDBMS.

Language specific solutions

Resource management problem is solved in different languages differently.

C

Every C developer has to take care of resource deallocation after some usage manually. For example, each memory allocation with malloc must be followed with free. From the first sight such approach might look not that difficult, because there are no exceptions in C. But in fact it makes code much more complicated. Situation is slightly improved in frameworks such as Glib.

C++

C++ is successor of C, it introduces classes with constructors and destructors and guarantees calling of object's destructor when it goes out of scope. As a result Resource Acquisition Is Initialisation (RAII) idiom emerged together with smart pointers, which are an implementation of the idiom. In C++ exceptions are also introduced and RAII helps in exception safety. So resource could be represented as an object, where it's initialised in constructor and deallocated in destructor.

Java

Even though objects in Java have finalize method, which is called before destruction, it can not be safely used for resource deallocation as C++ destructor. It's because Java runtime uses garbage collector and the moment of clean up is not guaranteed.
Instead try-finally block can be used

Type res = new Type();
try
{
    res.use();
}
finally
{
    res.deallocate();
}

The same approach is used in other languages with garbage collector (C#, Python, etc.) sometimes with additional syntax sugar.

Erlang

In Erlang developer can use all there techniques of resource management listed above.

Manual management

I assume that everybody is aware of disadvantages of C-style resource management and tries to limit it's usage, I would not go into details.

RAII

Some of OTP behaviours (gen_serv, gen_fsm) have init and terminate callbacks, which could be used for resource allocation and deallocation correspondingly. terminate is called right before process termination and fits for most usages of resource deallocation. The only potential issue could be a strict synchronisation of clean up with another process, which requires additional message sending.
Famous examples of RAII idiom from OTP are ETS and gen_tcp. ETS table is cleaned up by default, if process, which created it, terminates. Socket opened in context of process is closed on process termination as well.

try-catch

Resource deallocation function is guaranteed to be called in after clause of try-catch block. For example, this is how poolboy:transaction function is implemented.

transaction(Pool, Fun, Timeout) ->
    Worker = poolboy:checkout(Pool, true, Timeout),
    try
        Fun(Worker)
    after
        ok = poolboy:checkin(Pool, Worker)
    end.

Erlang-specific recommendations

The rule of thumb for choosing between try-catch block and RAII(separate process) is quite simple. Try-catch is easier and shorter, but is not applicable if allocated resource should exist during processing of next message/callback in caller process. It also can not be used if you need to share a resource between multiple processes.

Do not try to save few lines of boilerplate code for a new gen_server/gen_fsm module for resource management, when it's needed. It will eventually pay off with cleaner code.

General recommendations

Try to avoid manual (C-style) resource management.

Do not handle multiple resources neither in single try-catch block, nor in single object, which implements RAII idiom, because it's difficult to implement correctly.

poolboy pitfall

Description of library

Poolboy is a popular Erlang library for organisation of workers' pools. For example, it's often used for RDBMS connections. It's API is extremely simple, after start client uses just one function poolboy:transaction, which calls poolboy:checkout and poolboy:checkin wrapped into try/catch block.

transaction(Pool, Fun, Timeout) ->

    Worker = poolboy:checkout(Pool, true, Timeout),

    try

        Fun(Worker)

    after

        ok = poolboy:checkin(Pool, Worker)

    end.

Restart of terminated workers, queueing and other complicated things are completely hidden from user.

Hidden restrictions

But there is one pitfall, which developers should be aware of. By default checkout is a blocking operation( and it's used with default settings in transaction function), it means that client code will not return until worker is allocated. But nothing lasts forever, poolboy:checkout is implemented as gen_server:call/2 and has timeout argument (default is 5 seconds).

-define(TIMEOUT, 5000).

checkout(Pool, Block, Timeout) ->

    try

        gen_server:call(Pool, {checkout, Block}, Timeout)

    catch

        Class:Reason ->

            gen_server:cast(Pool, {cancel_waiting, self()}),

            erlang:raise(Class, Reason, erlang:get_stacktrace())

    end.

If timeout occurs client is "exited" with timeout reason. Attempt to handle such situation has even worse consequences.
Poolboy correctly recovers from termination of process, which checked out a worker (this test case passes). But if poolboy:checkout exits with error and client tries to handle it without termination of process, the worker might stay blocked (this test case fails).

transaction_timeout() ->

    {ok, Pid} = new_pool(1, 0),

    ?assertEqual({ready,1,0,0}, pool_call(Pid, status)),

    WorkerList = pool_call(Pid, get_all_workers),

    ?assertMatch([_], WorkerList),

    ?assertExit(

        {timeout, _},

        poolboy:transaction(Pid,

            fun(Worker) ->

                ok = pool_call(Worker, work)

            end,

            0)),

    ?assertEqual(WorkerList, pool_call(Pid, get_all_workers)),

    ?assertEqual({ready,1,0,0}, pool_call(Pid, status)). 

One can say that this issue could be experienced with enormous timeout value for checkout, but that could happen also in case of slow worker start, which is called in the same gen_server:handle_call, if overflow is allowed. Message from call might be queued for a long time, if worker is being restarted due to termination, as a result the same exit occurs on poolboy:checkout.

new_worker(Sup) ->

    {ok, Pid} = supervisor:start_child(Sup, []),

    true = link(Pid),

    Pid.

Found issue is reported to the author of poolboy together with PR, which reproduces the problem via unit test. I do not think this could be easily fixed with current architecture of poolboy, but following simple rule in client code can prevent problems.

Recommendation to avoid issues

Loosing of worker in pool could be avoided simply by not handling exits in process which call checkout. If handling of timeout on worker checkout is necessary just spawn a special process, which calls poolboy:transaction or poolboy:checkout, and "let it crash" handling it's exit as you want. 

OOP in Erlang. Part 1. Encapsulation

Introduction

There is a popular statement, that principles from object oriented programming (OOP) are not be applicable in functional languages and particularly in Erlang. Let's try to analyse it.
The main concepts of object oriented design (OOD) are

1. Encapsulation.
2. Polymorphism.
3. Inheritance

Encapsulation

Encapsulation concept in Erlang could be found at least in process model.

Processes

Each process has it's own state, which can only be modified by handling of message sent to the process. Process is free to ignore any kind of message, handling particular ones.
One of Erlang's "design pattern" is process per message, which encourages to spawn a process for each concurrent instance in a system. In case of HTTP service requests' handling is concurrent, but having encapsulation only on that level is not enough for a good Web framework. Also good people never force client to call gen_server:call/2 for a gen_server they've implemented, they wrap such calls with functions in client API modules.

Modules

Erlang module has export attribute,  which allows to call some functions implemented in it from outside. What else is it, if not splitting to public and private functions in OOP language?
Modules can represent objects even in easier way than processes. For example, queue and proplists. One can say that there is no protection from module's function being called with wrong argument, but the same situation is in Python, where private functions are distinguished from public only by naming conventions.

Conclusion

Erlang has two levels on encapsulation, which could be combined in order to create a good design for system.


To be continued.

default value for application environment variable

Current Erlang release contains a function application:get_env/3. Until it was included into OTP, many projects had developed similar functionality of specifying a default value for application environment variable.
This function in some cases makes system more complicated in terms of understanding and configuration. Basically it transfers value of variable from node config and .app file to the code. As a result if default value is used, user of node/application has no chance to notice such variable besides analysing of documentation, which is not always relevant or event exists, or source code.
In general it's a good practice to put some reasonable default value for each variable into the .app file of application.

unimplemented callback

Erlang behaviour

Erlang provides mechanism of behaviours, which allows to guarantee API of certain module. If one of behaviour's callback is not implemented, compiler generates warning.

gen_server

Let's examine the importance of these warnings on example of gen_server. Of course, process can not be started and stopped correctly without implementing init/2 and terminate/2 callbacks. Then gen_server implements some payload: handle_info, handle_call or handle_cast.

Neglect of unexpected messages

If one of these callbacks is not needed, some people do some default implementation, for example, logging of unexpected message and proceeding of program execution. This approach allows to avoid compilation warnings, but may hide serious problem in system if careful log monitoring is not organised. Also function clause with the same logic, which matches on everything, is often appended to each callback.

Crash on unexpected message

Another option could be termination of the process via returning stop-tuple, which is sometimes much better. The only argument against is that it happens by default in OTP but via crashing.

Hot code upgrade

The last callback not covered is code_change. It does not require a dummy implementation. During hot code upgrade system uses the newest callback to update process's state.

gen_fsm

If problem of unused code in gen_server's callbacks does not look that serious comparing to possible advantages of handling all possible messages, let's analyse another standard OTP behaviour - gen_fsm. Besides handle_info, init, terminate and code_change, which have logic similar to gen_server, gen_fsm has a number of callbacks for handling different types of events.

All state events

All state events could be synchronous and asynchronous and are handled in handle_sync_event/4 and handle_event/3 correspondently. For handling of unexpected messages of that type both strategies from gen_server (ignoring and termination) are applicable with the same advantages and disadvantages.

Named events

"Named" events could be synchronous and asynchronous as well and are handled by StateName/3 and StateName/2 callbacks of gen_fsm behaviour, where StateName is the name of current state of FSM. Since names of these callbacks are dynamic, not implementing them does not lead to compilation warnings, but somebody could understand fault-tolerance as not-crashing in any circumstances including receiving of unexpected events.
One, who decides to achieve this, must be really tolerant, since he needs to accumulate state names from entire FSM, and implement StateName/3 and StateName/2 dummy function clauses for each of them. Also he needs to maintain such FSM with growing number of states.

Conclusion

It is recommended not to implement any behaviour's callback just for the sake of suppressing of compilation warnings or being pseudo-fault-tolerant to unexpected messages/situations. Each callback implementation needs to have business logic. It's better to spend time on system's test automation instead of trying to make system, which can survive (not crash) if something unexpected happens.

lower bound

Lower bound

Sometimes lower bound algorithm, which returns an ordered subset of container's elements which are greater or equal to given value, is required in development. Surprisingly there is no implementation of it in OTP.

OTP containers

In Erlang there is a number of key-value containers with efficient search operation:

1. ETS;
2. mnesia;
3. gb_trees;
4. dict;
5. orddict.

Several implementations of sets: gb_sets, sets, ordsets, which are more or less the same as key-value container besides that only keys are present.

DETS - is a specialised file storage. Maps are not covered in this article since the appeared in Erlang 17.

Basically there are some alternatives to base lower bound algorithm implementation on.

dict, sets

In documentation it's stated that the representation of a dictionary is not defined. Thereby implementation of lower bound algorithm requires reverse engineering and would be based on data structure which might be changed in next version.

orddict, ordsets

Sorted lists from the first sight is one of the best data for lower bound implementation, but in fact lists in Erlang do not provide random access to it's elements with constant complexity (singly linked list). In ordict:find/2 source code one can see that it just traverses the entire list starting from the head. So binary search is not used, as a result not efficient lower bound implementation is possible.

ETS, mnesia

ETS and mnesia are based on native code of Erlang VM and should have the best performance. Both containers have iterators so that resulting subset of lower bound algorithm could be traversed without copying. Unfortunately in ets:match_object as well as in mnesia:match_object key must be bound for efficient search, otherwise the entire container is searched.
In order to find out if optimisation exists for range matching in ETS, test is created. Two functions are implemented: first just finds element in ETS and returns it's key (key can be used as iterator), second - returns key of first element of lower bound . Both functions are implemented based on ets:select/3 with limit 1. ETS is initialised with 20000000 elements. Each element is a tuple {Key, Value}, where Key = Value. Integers incremented from 1 to 20000000  are used as keys. In performance test both functions are called with different element to search for in the same ETS. Here are function execution times.

Key	Bound key time, s	Range key time,s
10000	0.000	0.002
100000	0.000	0.015
1000000	0.000	0.163
10000000	0.000	1.332
20000000	0.000	1.537

From provided values it's obvious that ets:select/3 makes full table scan if the key is not bound. It's important to notice that select function stops after first matching element is found because of limit argument set to one.
Although optimisation of key range matching may appear in next version of Erlang at least for some types (ordered_set), ETS and mnesia are not suitable for lower bound algorithm implementation at the moment.

gb_sets, gb_trees

gb_trees has documented representation, it's ordinary binary search tree. Iterators are also available for this container. Implementation of lower bound algorithm is very elegant with recursion, function returns iterator to the first element of subset.

Conclusion

Surprisingly none of OTP containers has an implementation lower bound algorithm, which is very common for set theory. Implementation based on ETS of type ordered_set turned out to be inefficient in terms of complexity without any obvious reasons. The most suitable container for lower bound implementation is gb_trees, where binary tree search can be used. 

empty supervisor

Problem

Sometimes Erlang application consists only of utility functions without any processes spawning. In such cases usually supervisor child specification in supervisor:init/1 is an empty list. Supervisor's process, which does nothing, is likely something developers should not worry about, but this situation can be resolved in more elegant way.

Solution

If application's boilerplate code is generated from standard template (for example,
rebar create-app), it usually contains:

1. <application>.app (<application>.app.src) with application specification;
2. <application>_sup.erl, which implements supervisor behaviour;
3. <application>_app.erl, which implements application behaviour.

If supervisor's module is removed, application crashes on start since application:start/2 callback must return a pid of supervisor. So <application>_app.erl should be removed as well. After that application can not be started again, because module with application behaviour implementation is specified in <application>.app (<application>.app.src) in mod section.

{mod, { <application>_app, []}}

So that should be deleted too. As a result two unnecessary files were deleted from the project. Example of described application could be found here.

Exceptions

In very rare applications "state" could be stored not in process' state but in public ETS or other "not pure functional" techniques might be used. For that cases application:start/2 callback should be implemented even with empty supervisor. For instance, public ETS could be created on start of application.

Sometimes processes are started not under top-level supervisor, illustration of this is Cowboy web server. In all examples empty supervisor is used and Cowboy is started in application:start/2 callback.

Misuse of environment variables

Application environment variables

Environment variables are the main configuration mechanism for Erlang applications. Configuration of Erlang node basically is a list of application names together with list of application's environment variables.

Usually variables are set once on start of application and can be read any time during application's execution and in any place in the code. It's also possible to set or update it in runtime.

Application usually uses only it's environment variables, but accessing of other application's environment is also possible.

With such a "freedom" this mechanism is often misused. Let's see an example.

Testing

In gen_server:handle_call callback environment variable is read:

handle_call(use, _From, State) ->
 {ok, Var2} = application:get_env(var2),
 {reply, Var2, State}.

It works fine until we start writing of unit tests and application:gen_env/1 returns undefined leading to process crash. Next attempt could be setting variable in fixture of test case.

basic_test_() ->
 {
 setup,
 fun() ->
   application:set_env(app_env_var, var2, 3),
   start_link()
 end,

If it's launched in gen_server's test suite without starting application it crashes again. It happens because code is executed in context of other application (application:get_application/0 returns undefined until application is started). The only reasonable fix in this case is specifying of application name on getting environment variable value.

{ok, Var2} = application:get_env(app_env_var, var2)

To conclude, getting application environment variables in low-level functions makes testing more difficult.

Another approach

In order to overcome difficulties in testing, variables could be used in a different way. All environment variables should be read in application:start/2 callback and passed further to supervisor and rest processes in the chain.

start(_StartType, _StartArgs) ->
 {ok, Var1} = application:get_env(var1),
 app_env_var_sup:start_link(Var1).

In testing variables setup is changed with passing appropriate value to start_link function of gen_server and saving in state.

basic_test_() ->
 {
 setup,
 fun() -> start_link(3) end,

This kind of environment variable usage implies additional code for passing values to the place where it is actually used. One more disadvantage is necessary restart of the node for changing variable's value.

Environment variable can be compared with global variable in languages like C++ or Java. It is accessible everywhere, what could be convenient for some task, but has all the disadvantages, which are well know. 

Performance

One more argument in favour of not using environment variables is performance. Two gen_servers, which access data from application environment variable and from it's state correspondingly , were compared. Here are results.

gs1 (value from state accessed)  : 291802

gs2 (environment variable)       : 325638

Time is specified in microseconds, results are provided for 100000 runs of the same code.

Conclusion

Code of all unit tests and benchmarking is provided in github repository.

Passing environment variables' values as arguments from application:start/2  callback results into some additional boilerplate code, requires application's restart on configuration change, but makes code much cleaner and easier to test comparing to direct reading of env variables. It is important to find balance between both approaches. 

For global functionality such as logging passing configuration to each call using additional arguments is a big overhead in terms of code readability. If configurations of some application changes often and/or it's restart is undesirable accessing env variables is preferred. In most of other cases suggested approach fits better.

tuples vs records vs proplists

Data structures for business logic

When function is being designed, one of the most important questions is data structure to use. In Erlang there are several compound "types": lists, records, maps and tuples. ETS is separate type which has a specific usage, in most cases there is no confusion with other data structures. Maps appeared in Erlang 17 and not covered in this article even though they combine properties of both lists and tuples.
List contains variable number of elements. One of the popular "subtype" of it is proplist, which is a list of {Key, Value} tuples. It does not allow pattern matching on elements except on head.
Tuple's arity is fixed and client code in most cases should be aware of exact size. It provides intensive usage of pattern matching. With growing size usage of tuples becomes error-prone.
Records solve complexity of big tuples' usage by adding syntax sugar on compile time. In fact records are translated to tuples. Usage of records might require some additional compile dependencies.

Another alternative to tuples, records and proplists is complex function signature.

save_user(Name, Surname, Age)

where  each property is a separate argument. It's mostly equivalent to

save_user({Name, Surname, Age}).

except that if signature of function changes ofter it's much more work to adapt code comparing to single tuple argument.

In API design performance of operations on types of arguments and return values is not as crucial as simplicity of usage, protection from misuse and other criteria of "good code/design". That's why no benchmarks are provided.
Recipe of good design in this scope is quite straightforward: "Data structure shall be chosen based on it's characteristics and logic of code". It is easier to illustrate this rule with an example.

Example

Service with RESTful interface is being developed. User information is received in JSON format in POST request.

{
 "name": "Eddy",
 "surname": "Snow",
 "age": 28
}

There is a general purpose JSON object parsing function (arrays are not covered here for simplicity), it accepts binary as an argument. Data structure for return value is not that obvious.
First of all, client code needs to detect errors in parsing. Assuming that exceptions are not used, function should return tuple {ok, Result} on success or {error, Reason} on failure. Tuple for error handling here fits perfect, there is no need to introduce record since tuple size is two.
Next is data structure for Result. Since function parse_json_object is generic, it can parse any object and result  is dynamic. Proplist is suitable type for it.

-spec parse_json_object(JSON::binary()) ->
  {ok, Result::list()} | {error, Reason::term()}.

Once user information is correctly parsed, it might be needed to validate it and store in the database. So kind of internal user object is required. It could be tempting to continue using proplist, but record fits much better for it, because it applies number of compile-time checks.

-record(
  user,
  {
     name,
     surname,
     age
  }
).

Advantages and disadvantages of records

Elements of record are accessed by name and Erlang compiler verifies that only existing properties are "got/set". If tuple is used instead data could be accessed only by index of element, which is error-prone with big tuple's size.

Sometimes it's declared that records are not suitable for storing in riak in erlang binary format, because if record declaration is changed data in storage becomes invalid in terms of matching it to the new version. In fact it's worth to implement some serialisation layer for this task as versioning might also be required for proplist or tuple.

Another argument against records could be hot code upgrade because of the same problem with changing of record declaration. In that case law of "not using records as tuples" could be broken in code_change callback.

Conclusion

To sum up, general recommendation for choosing data structure are:

1. Do not use tuples of size more than 3.
2. Use proplists only if number of "object's" properties varies.
3. Consider records as a main alternative to tuple of big size.

Monitoring of Erlang systems

Monitoring

One of advantages of Erlang over other languages is scalability, which implies executing tasks on multiple machines. With growing number of servers used, monitoring of system becomes more and more complicated.

Monitoring in Erlang

What is monitoring for functional language such as Erlang? It is collecting of some attributes of function. In my opinion two most important of them are time of execution and return value.
Return value may indicate success or failure of certain operation, for example knowing if http server returned 200 or 404 is nice to track, another example is status of payment transaction: successful or declined.
Monitoring of time is important for diagnosing of performance issues, it is very useful see how long certain SQL query takes.

Possible implementations

Logging

In most projects people start adding code for monitoring directly to functions.
Fist naive implementation could be just logging of metrics. It works more or less fine except fact extracting data from logs is a tedious task. Also in most cases we are interested in relative value rather than absolute one. For example, fact that some SQL query takes 2 seconds does not tell much about system without context.

Reporting

Context in that case is history of metric, the most convenient way of representation is plot. Once people realise it, they setup up plot building software like graphite, zabbix or opsview,  and implement Erlang clients for reporting. One project which is worth mentioning here is folsom, which is Erlang application for collecting metrics of a system.
On stage of integration of code, which reports data to plot building service, developers realise that almost all tests are broken, because this functionality needs to be mocked. It could be done by ordinary mocking or implementing of "empty" reporter, which does nothing.
It also turns out that monitoring code is global, and it's hard to track which metrics are monitored.

Usage of tracing

Erlang provides a convenient way of tracing software - erlang:trace function, which is reused by higher-level applications such as dbg, ttb and redbug. It could be also used for monitoring of system. The idea behind that is very simple. In demo project elmon tracing is enabled for all new processes spawned using this line of code:

    erlang:trace(new, true, [call, timestamp])

call option enables tracing of function calls, timestamp is just added to trace information.
Next step is in specifying functions to trace.

    MatchSpec = dbg:fun2ms(fun(_)->exception_trace()     end),
    erlang:trace_pattern(MFA, MatchSpec, [global]).

MFA here is tuple {Module, Function, Arity}, global option is specified to trace only exported functions, match specification sets up tracing of exceptional returns from function as well as normal ones.
Tracing could also be enabled for local functions, but likely it's time to refactor code when you need it. Match specification could potentially be more advanced with unbind variables ('_') and etc, but it seems to be a refactoring required instead. trace_pattern returns a number of functions matched, it is considered to be one for the same reason.
Combination of these two functions enables tracing of specified function calls in all processes spawned after. Trace information is sent to process, which called erlang:trace, in messages. Each message contains timestamp of occurred event. Events are function call and returning from it.
The only thing left to do is to store timestamp of call and subtract it from appropriate timestamp of return message. Key {Pid, MFA} is unique in scope of one call even for recursive functions, since finish message is sent after a chain of tail recursive calls is ended.
In elmon calculated trace information is reported gen_event subscribers registered for it. Monitoring may be disabled simply by not starting the application.
Usually sysops are interested in some metrics of Erlang VM such as memory usage, number of processes, etc. A separate application is a nice place to put them in.

Conclusions

Of course using of tracing has a significant drawback comparing to injecting of monitoring code directly to functions. It is performance. I would not provide any measurements, because it can vary depending on process model of software and many other factors.
Clear separation of concerns, easy enabling/disabling make Erlang tracing one of variants to consider when choosing monitoring strategy. 

rebar and repository organisation

Introduction

In Erlang ecosystem the most popular build tool seems to be rebar. It is quite easy to start using. If you have an Erlang application with directory structure organised according to OTP requirements, you just launch rebar compile in the root directory of app and it compiles.
Whenever you need a dependency from another application which is not in OTP, you create a rebar.config file with following content
{deps, [
  {dep_name, dep_version, path_to_dep}
]}.
Most likely path_to_dep would be VCS url, for example,
  {git, "git://github.com/rebar/rebar.git"}.
But what shall we do when we need to create one more own Erlang application? Let's overview possible code organisation strategies.

Repository per application

The most popular repository organisation used with rebar is a separate repository for each application. It provides a nice separation of concerns, great support from rebar and it is quite easy.
A new repository is created, source code is placed. Then algorithms for 3rd-party dependencies is repeated - reference to the new application is set up in rebar.config of original application.
Not everything is so bright with this approach. Even if you have only 3rd-party dependencies you need to take care of versioning, because depending just only on branch (especially on one under active development) could bring instability into the system you are working on. To solve this problem people specify a release branch as dependency (in case of git it is 'tag'). Rebar supports it from the box, even erlang application's version from .app file is supported (see dep_version in rebar.config), but it is not extensively used as it is the second versioning mechanism to take care of.
During development of system which consists of multiple applications it is very common that changes are required in several components for one feature. So feature branch is created in each repository, and in order to keep consistency during development this branch is specified in rebar.config.
{deps, [
    {app1, ".*",
        {git, "git_url/app1.git",
            {branch, "feature12"}}},
    {app2, ".*",
        {git, "git_url/app2.git",
            {branch, "feature12"}}}

]}.

You need to take care of rebar.config before or after merging to the development branch. With growing number of repositories code management could start to be a pain, which could be reduced with automation of routine tasks.

Single repository

An alternative to writing scripts for VCS handling is keeping everything in a single repository. The idea is to place each Erlang application into separate directory in the repository. As a result directory structure looks like this:
repo_dir
|-app1
|-app2
|-app3
|-rebar.config

In each directory there is a standard OTP application structure. The only change required is putting
{deps_dir, ".."}.
into each app's rebar.config and
{deps_dir, "."}.
into rebar.config in root directory. Dependencies from the same repository are specified only by name:
{deps, [app2]}.
The entire project could be found here.
All rebar commands such as compile, eunit and clean could be launched in each application's directory separately or in root. It is convenient, for example, to launch unit tests in all apps once change is made with rebar eunit in top level directory.
Keeping things in a single repository allows to do an "atomic commit", when single merge adds feature into development or integration branch. You can create a release (tag in git) with a single command as well. In over words it allows to have a snapshot of entire system easily.
It is quite easy to move modules from one application to another.
Of course there is a drawback with single repository. It does not allow to manage access for developers in flexible way. For example, company decides to opensource one of applications, so that rest of them are still private. It could only be achieved by moving the code into a separate repository. The same problem occurs if access must be granted only to the part of system (not for all applications).

To SumUp

I would recommend to start with single repository for all Erlang applications and move some of them to separate repositories only for a good reason.