Chapter 13. Writing a Distribution

In Chapter 12, you created a fictional Island::Plotting::Maps module, and built the right support for Exporter so that you could include use Island::Plotting::Maps in a program.

While the resulting .pm file is useful, it’s not very practical. There’s a lot more to building a real module than just creating the .pm file. You’ll also need to consider and implement the following questions:

Installation location

How and where is the .pm file installed so a program can find the module in its @INC path?

Documentation

Where is the documentation for the module? How is the documentation installed so the user can read it?

Archive completeness

If there are any accompanying files, where are they? How can the end user know if any missing files are missing?

Testing

What test harnesses can the developer run to verify correct operation, including testing against older known bugs? Can these same tests be run by the installer to ensure proper operation in the target environment?

C-language interfaces

If the module contains C or C++ code (not covered here), how can the developer describe how to compile and link the code in the developer’s environment, or the end user environment?

As Roy Scheider uttered in the movie Jaws: “You’re gonna need a bigger boat.” That “bigger boat” is the difference between a module and a distribution.

A distribution contains the module (or collection of related modules), plus all the support files required to document, test, ship, and install the module. While you could potentially construct all these files by hand, it’s much simpler to use a tool that comes with Perl, awkwardly called h2xs [76]

The h2xs tool creates a series of template files that serve as a starting point for the distribution files. You simply need to say h2xs -XAn, followed by the name of the module—in this case, Island::Plotting::Maps.[77] Here’s what the output looks like:[78]

$ h2xs -XAn Island::Plotting::Maps
Defaulting to backwards compatibility with perl 5.8.0
If you intend this module to be compatible with earlier perl versions, please
specify a minimum perl version with the -b option.

Writing Island/Plotting/Maps/Maps.pm
Writing Island/Plotting/Maps/Makefile.PL
Writing Island/Plotting/Maps/README
Writing Island/Plotting/Maps/t/1.t
Writing Island/Plotting/Maps/Changes
Writing Island/Plotting/Maps/MANIFEST

You’ll create some subdirectories below the current directory and a bunch of files in the lowest of those directories. Let’s look at them to see what they are. First, cd into the directory:

$ cd Island/Plotting/Maps

Now, let’s examine the MANIFEST file:

$ cat MANIFEST
Changes
Makefile.PL
MANIFEST
Maps.pm
README
t/1.t

The MANIFEST file resembles a table of contents for the distribution. When you eventually bundle up everything and ship it off to the CPAN or the ultimate recipient, the bundling tool looks at MANIFEST to know what to include, and the unpacking tool verifies that everything in MANIFEST is present at the destination. Of course, MANIFEST lists MANIFEST, but it also lists everything else created by h2xs automatically.

While maintaining a MANIFEST sounds like it might be painful, you can be assured that you won’t accidentally include your “notes to self” in the distribution just because the file happened to be in the wrong directory. Specific steps discussed later keep the MANIFEST up to date.

The next, mostly uninteresting, file is README, which is self-describing:

$ cat README
Island/Plotting/Maps version 0.01
=  ==  ==  ==  ==  ==  ==  ==  ==  ==  ==  ==  ==  ==  ==  ==  ==

The README is used to introduce the module and provide instructions on
how to install the module, any machine dependencies it may have (for
example C compilers and installed libraries) and any other information
that should be provided before the module is installed.

A README file is required for CPAN modules since CPAN extracts the
README file from a module distribution so that people browsing the
archive can use it get an idea of the modules uses. It is usually a
good idea to provide version information here so that people can
decide whether fixes for the module are worth downloading.

INSTALLATION

To install this module type the following:

   perl Makefile.PL
   make
   make test
   make install

DEPENDENCIES

This module requires these other modules and libraries:

  blah blah blah

COPYRIGHT AND LICENCE

Put the correct copyright and licence information here.

Copyright (C) 2002 Ginger Grant

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself.

Obviously, you will want to edit this file to be whatever you want it to be. The phrase “blah blah blah” is often used in the templates to indicate things that must be changed.[79] If you leave unchanged the blah blah blah and other notes from h2xs to you, potential users will suspect that bugs in the code have also escaped your scrutiny, so proofread this stuff (and your code) before you distribute your module.

Pay special attention to the copyright and license section. (It should have your name in place of Ginger’s name, unless your machine is very confused about who is sitting at the keyboard.) Your employer may require you to change the copyright notice to include your company name rather than your name, for example. Or, if you’re including someone else’s code in your module, you may need to mention their copyright (or lack thereof) as well.

The README file also has a special responsibility: the master CPAN archiving tools pull out the README file as a separate entry automatically, permitting the file to be indexed by search engines on the various worldwide archives, and to be downloaded and read trivially by the CPAN installation tools. In the CPAN.pm shell, for example, you can say:[80]

$ perl -MCPAN -eshell
cpan> readme Island::Plotting::Maps

and the contents of the README file will be shown without having to download and unpack the entire distribution.

Another file created as a template is Changes:

$ cat Changes
Revision history for Perl extension Island::Plotting::Maps.

0.01  Wed Oct 16 15:53:23 2002
        - original version; created by h2xs 1.22 with options
                -XAn Island::Plotting::Maps

You’ll need to maintain this file manually, unless your interactive development environment has automated tools for such maintenance. Most people will expect to be able to look here to see what has been updated in new releases of your module. Try not to disappoint them. One of the main purposes of the Changes file is debugging: if you realize that a certain bug turned up three releases back, you can look here to remind yourself of new features or bug fixes that were introduced in that release.

Now you come to the most important part of the distribution: the module itself. Finally, actual code:

$ cat Maps.pm
package Island::Plotting::Maps;

It looks good so far. The module automatically starts with an appropriate package directive. Following that, you see:

use 5.008;
use strict;
use warnings;

Now you’re declaring that the module is compatible with Perl 5.8.0 or later and that the compiler restrictions and warnings are enabled automatically. Good practices are encouraged here. Obviously, you’re free to delete or modify anything inappropriate.

Then comes Exporter:

require Exporter;

our @ISA = qw(Exporter);

These lines support the import method call needed to make use work. This is fine for a nonobject-oriented module, but for an object-oriented module inheriting from a parent (base) class, you’ll want to replace them with:

use base qw(Our::Parent::Class);

Resulting in more Exporter control:

our %EXPORT_TAGS = ( 'all' => [ qw(

) ] );

our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );

our @EXPORT = qw(

);

Every symbol you wish to possibly import as a result of the use operation should be placed into the first qw( ) list. This lets you say:

 use OurModule qw(:all)

to get all possible symbols.[81]

Every symbol you wish to export by default on a missing import list should go into the last qw( ). (Any symbol in the @EXPORT list is automatically a member of the @EXPORT_OK list, but you’ll want to list it again it the first qw( ) list so it comes in via the :all method.)

Recall from Chapter 12 that a typical object-oriented module exports nothing because the interface is entirely through class method calls and instance method calls, neither of which require subroutines in the invoker’s package.

Next, you’ll see the version number:

our $VERSION = '0.01';

This version number is important for many reasons:

Generally, you can start with the 0.01 given by the template and increase it consistently with each new test release. Often, you can coordinate this version number with some source code control system.[82]

Now you’re past the header information and down to the core of your module. In the template, it is indicated by a simple comment:

# Preloaded methods go here.

What? You didn’t think the h2xs tool would even write the module for you, now did you? Anyway, this is where the code goes, usually as a series of subroutines, possibly preceded by some shared module data (using my declarations), and perhaps a few package variables (using our declarations in recent Perl versions). Following the code, you’ll find your necessary true value:

1;

so the require (inside the use) doesn’t abort.

Immediately following the mandatory true value in the file, you’ll find the _ _END_ _ marker:

_ _END_ _

This marker tells Perl that there’s no Perl code anywhere later in the file and that the Perl parser can stop now.[83]

Following the _ _END_ _ marker, you’ll find the embedded documentation for the module:

# Below is stub documentation for your module. You'd better edit it!

=head1 NAME

Island::Plotting::Maps - Perl extension for blah blah blah

=head1 SYNOPSIS

  use Island::Plotting::Maps;
  blah blah blah

=head1 ABSTRACT

  This should be the abstract for Island::Plotting::Maps.
  The abstract is used when making PPD (Perl Package Description) files.
  If you don't want an ABSTRACT you should also edit Makefile.PL to
  remove the ABSTRACT_FROM option.

=head1 DESCRIPTION

Stub documentation for Island::Plotting::Maps, created by h2xs. It looks like the
author of the extension was negligent enough to leave the stub
unedited.

Blah blah blah.

=head1 EXPORT

None by default.



=head1 SEE ALSO

Mention other useful documentation such as the documentation of
related modules or operating system documentation (such as man pages
in UNIX), or any relevant external documentation such as RFCs or
standards.

If you have a mailing list set up for your module, mention it here.

If you have a web site set up for your module, mention it here.

=head1 AUTHOR

Ginger Grant, <ginger@island.cocoanet>

=head1 COPYRIGHT AND LICENSE

Copyright 2002 by Ginger Grant

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself. 

=cut

This documentation is in POD format. The perlpod manpage describes the POD format in detail. Like Perl itself, POD is said to mean various things, such as Perl Online Documentation or Plain Old Documentation, and so on. To most of us, it’s just POD.

As the template describes, you’re expected to edit portions of this text to fit your particular module. In particular, leaving the blah blah blah is considered bad form.

The POD information is extracted automatically by the installation process to create the native documentation format, such as Unix manpages or HTML. Also, the perldoc command can locate POD in the installed scripts and modules and format it for the screen.

One nice thing about POD is that it can be interspersed with the Perl implementation code it describes. For example, from Chapter 12, you needed three subroutines for the Island::Plotting::Maps module. You can commingle the code and documentation. Each POD directive (a line beginning with an equal sign) switches from Perl mode (lines interpreted as Perl code) to POD mode (lines interpreted as documentation), and each line beginning with =cut switches back. Thus, the resulting file looks something like:

package Island::Plotting::Maps;
[... stuff down to the $VERSION setting above ...]

=head1 NAME

Island::Plotting::Maps - Plot maps on the island

=head1 SYNOPSIS

  use Island::Plotting::Maps;
  load_map("/usr/share/map/hawaii.map");
  scale_map(20, 20);
  draw_map(*STDOUT);

=head1 DESCRIPTION

This module draws maps.  [ more here ]

=over

=item load_map($filename)

This function [ more here ].

=cut

sub load_map {
  my $filename = shift;
  [ rest of subroutine ]
}

=item scale_map($x, $y)

This function [ more here ].

=cut

sub scale_map {
  my ($x, $y) = (shift, shift);
  [ rest of subroutine ]
}

=item draw_map($filehandle)

This function [ more here ].

=cut

sub draw_map {
  my $filehandle = shift;
  [ rest of subroutine ]
}

=back

=head1 SEE ALSO

"Map Reading for Dummies", "First Mates: why they're not the captain",
and be sure to consult with the Professor.

=head1 AUTHOR

Ginger Grant, <ginger@island.cocoanet>

=head1 COPYRIGHT AND LICENSE

Copyright 2002 by Ginger Grant

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself. 

=cut

1;

As you can see, the documentation for the subroutines is now very near the subroutine definition, in hope that as one gets updated, the other will be similarly changed. (Out-of-date documentation is often worse than no documentation at all because at least with no documentation at all, the user is forced to look at the source code.) Many modules in the CPAN do this. The penalty is a very slight increase in compile-time activity because the Perl parser has to skip over the embedded POD directives.

Whether you place your POD at the end of the file (as the template suggests) or intertwined with your code (as presented in the preceding paragraphs) the important thing to remember is always document your code. Even if it’s just for you, a few months later, when your brain has been 42,000 other places before you look at the code again, you’ll be glad to have those notes. Documentation is important.

The Perl developers have chosen to rely on the standard Unix make utility to build and install Perl itself, and that same mechanism is used for additional modules.[84] If you have a non-Unix system, a make-like utility should also be available. On Windows, for example, you may have dmake or another program. The command perl -V:make will tell you the name of your make utility program; if it says make='nmake', simply use nmake wherever you use make. In any case, you should call the controlling file a Makefile, even though its name may vary as well.

However, crafting a Makefile is tricky but repetitive. And what better way to accomplish a tricky but repetitive task than with a program? Since you’re talking about Perl add-on modules, you know that Perl is available, so how about a Perl program to generate the Makefile?

That’s exactly what happens. The distribution is required to contain a Makefile.PL, which is a Perl program to build a Makefile. Then from there, you use make (or something like it) to control all of the remaining tasks.

The h2xs tool generates a template Makefile.PL that you probably won’t even need to touch for single-module distributions:

$ cat Makefile.PL
use 5.008;
use ExtUtils::MakeMaker;
# See lib/ExtUtils/MakeMaker.pm for details of how to influence
# the contents of the Makefile that is written.
WriteMakefile(
    'NAME'                => 'Island::Plotting::Maps',
    'VERSION_FROM'        => 'Maps.pm', # finds $VERSION
    'PREREQ_PM'                => {  }, # e.g., Module::Name => 1.1
    ($] >= 5.005 ?    ## Add these new keywords supported since 5.005
      (ABSTRACT_FROM => 'Maps.pm', # retrieve abstract from module
       AUTHOR     => 'Ginger Grant <ginger@island.cocoanet>') : (  )),
);

Yes, this is a Perl program. The WriteMakefile routine is defined by the ExtUtils::MakeMaker module (included with Perl) to generate a Makefile. As the developer of the module, use this makefile to build and test your module and prepare a distribution file:

$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Island::Plotting::Maps

The ultimate user of your distribution will execute the identical command at their site. However, the Makefile will most likely be different, reflecting the differences in installation locations, local policies, and even the C compiler and linking instructions appropriate for their architecture. It’s a nice system that has worked quite well over the years.

The creation of the Makefile.PL (and resulting Makefile) is quite flexible. For example, you can run code to ask the person installing your module about the locations of other installed libraries or tools, or get options for variations in activity.[85]

The PREREQ_PM setting is important if your module depends on non-core Perl modules, especially if you plan to upload your code to the CPAN. Proper use of the prerequisites list can make installing your module nearly painless, and your user community will thank you.

Speaking of installation locations, the Makefile built by the default invocation of Makefile.PL presumes that the module will be installed in the system-wide Perl directory that all Perl programs can access directly with the built-in @INC path.

However, if you are testing a module, you certainly don’t want to install it into the system directories, possibly corrupting a previous version of your module and breaking production programs.

Also, if you’re not the system administrator, it’s unlikely that you can change those central Perl directories because that would be a great way to insert a trojan horse for privileged users to stumble across.[86]

Luckily, the Makefile contains provisions for considering an alternate installation location for scripts, manpages, and libraries. The easiest way to specify an alternate location is with a PREFIX value as a parameter on the command line:

$ perl Makefile.PL PREFIX=~/Testing
Checking if your kit is complete...
Looks good
Writing Makefile for Island::Plotting::Maps

Although the messages don’t indicate anything different, the Makefile will now install scripts to $PREFIX/bin, manpages below $PREFIX/man, and libraries below $PREFIX/lib/site_perl. In this case, you’re selected a subdirectory of your home directory called Testing as the value of $PREFIX.

If you were a project librarian, managing code for a team of developers, you might instead say something like:

$ perl Makefile.PL PREFIX=/path/to/shared/area

which then builds the files into a shared area. Of course, you’d need write privileges to such a directory, and the rest of the team would have to add the bin subdirectory to their PATH, the man subdirectory to their MANPATH, and the lib/site_perl directory to their @INC path, as you’ll see shortly.

Testing. Testing. Testing.

Testing is important. First, you should at least ensure that the code you’ve written even compiles before you install it and start playing with it. That test is free. You can invoke it directly from the newly created Makefile by simply typing make test, as in:

$ make test
cp Maps.pm blib/lib/Island/Plotting/Maps.pm
PERL_DL_NONLAZY=1 /usr/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 
'blib/lib', 'blib/arch')" t/*.t
t/1....ok
All tests successful.
Files=1, Tests=1,  1 wallclock secs ( 0.08 cusr +  0.04 csys =  0.12 CPU)

But what happened there?

First, the .pm file was copied to the testing staging area: the area headed by blib (build library) below the current directory.[87]

Next, the perl that invoked the Makefile.PL is called upon to run the test harness—a program that manages all test invocations and reports the results at the end.[88]

The test harness runs all files in the t subdirectory that end in .t in their natural order. You have only one file (created by h2xs), which looks like this:

$ cat t/1.t
# Before 'make install' is performed this script should be runnable with
# 'make test'. After 'make install' it should work as 'perl 1.t'

#########################

# change 'tests => 1' to 'tests => last_test_to_print';

use Test::More tests => 1;
BEGIN { use_ok('Island::Plotting::Maps') };

#########################

# Insert your test code below, the Test::More module is use(  )ed here so read
# its man page ( perldoc Test::More ) for help writing this test script.

It’s a simple test program. The test pulls in the Test::More module, described further in Chapter 14. The import list for the module is treated specially; you’re declaring that this test file consists of only one “test.”

The test is given in the following line and attempts to use the module. If this succeeds, you get an “OK” sign, and the overall test file succeeds. This would fail with bad syntax, or perhaps if you forgot to have that true value at the end of the file.

In this example, the test succeeds, so you get a message for it and a summary of the CPU time used for the test.

Since you know the module can at least compile, let’s be daring and install it. Of course, you’re installing it only into the path specified by the PREFIX in the earlier step, but that’s enough to show how it would have worked for the ultimate user’s installation.[89]

The installation is triggered with make install:

$ make install
Manifying blib/man3/Island::Plotting::Maps.3
Installing /home/ginger/Testing/lib/site_perl/5.8.0/Island/Plotting/Maps.pm
Installing /home/ginger/Testing/man/man3/Island::Plotting::Maps.3
Writing /home/ginger/Testing/lib/site_perl/5.8.0/darwin/auto/Island/Plotting/Maps/.
packlist
Appending installation info to /home/ginger/Testing/lib/site_perl/5.8.0/darwin/
perllocal.pod

Note that you’re installing the module below the $PREFIX/lib/site_lib directory (presuming a PREFIX of /home/ginger/Testing from earlier) and a manpage below $PREFIX/man (on Unix machines, in the Section 3 area for subroutines, for example). The manpage comes automatically when you extract the module’s POD data and convert it to troff -man code, making it compatible with the Unix man command.[90]

After some testing, you may decide it’s time to share your work with friends and associates. To do this, make a single distribution file. Many mechanisms are available to do this, but the most common one on most modern Unix platforms is the GNU gzip compressed tar archive, commonly named with a .tar.gz or .tgz extension.

Again, with a simple make invocation (make dist), you end up with the required file:

$ make dist
rm -rf Island-Plotting-Maps-0.01
/usr/local/bin/perl "-MExtUtils::Manifest=manicopy,maniread" \
        -e "manicopy(maniread(  ),'Island-Plotting-Maps-0.01', 'best');"
mkdir Island-Plotting-Maps-0.01
mkdir Island-Plotting-Maps-0.01/t
tar cvf Island-Plotting-Maps-0.01.tar Island-Plotting-Maps-0.01
Island-Plotting-Maps-0.01/
Island-Plotting-Maps-0.01/Changes
Island-Plotting-Maps-0.01/Makefile.PL
Island-Plotting-Maps-0.01/MANIFEST
Island-Plotting-Maps-0.01/Maps.pm
Island-Plotting-Maps-0.01/README
Island-Plotting-Maps-0.01/t/
Island-Plotting-Maps-0.01/t/1.t
rm -rf Island-Plotting-Maps-0.01
gzip --best Island-Plotting-Maps-0.01.tar

Now there’s a file named Island-Plotting-Maps-0.01.tar.gz in the directory. The version number in the name comes from the module’s $VERSION variable.[91]

The libraries are installed relative to the PREFIX specified earlier. If Ginger used a PREFIX of /home/ginger/Testing, you need to add the appropriate directory below it to the search path. The use lib directive of:

use lib "/home/ginger/Testing/lib/site_perl";

does the right thing to find the version-specific directory below it, as well as the architecture-specific directory below it, if needed (usually for architecture-specific files, such as compiled binaries).

You can also specify the include directory on the command line with a -M option:

$ perl -Mlib=/home/ginger/Testing/lib/site_perl myproggy

or a -I option:

$ perl -I /home/ginger/Testing/lib/site_perl myproggy

or even by setting the PERL5LIB environment variable (using sh-like syntax here):

$ PERL5LIB=/home/ginger/Testing/lib/site_perl; export PERL5LIB
$ ./myproggy

However, the downside of any of these methods (other than the use lib method) is that they require you to do something more than just execute the file. If someone (or something) else (such as a coworker or a web server) executes your program, it’s unlikely that the proper environment variable or command-line option will be present. Your program will fail because it can’t find your locally installed module.

Use use lib, when you can. The other ways are useful mainly for trying out a new version of an old module before replacing the old module (and possibly breaking the programs that use it).

The answers for all exercises can be found in Section A.12.

Package up the module from Chapter 12 as a distribution. Be sure to add the proper POD documentation for the subroutine. Test the module, install it locally, and then build a distribution file. If you have time, unpack the distribution into a different directory, pick a new prefix, and install it again to verify that the distribution archive contains everything necessary.



[76] The name h2xs has an interesting pedigree. Back in the early days of Perl 5, Larry invented the XS language to describe the glue code that Perl needs to talk to C-language functions and libraries. Originally, this code was written entirely by hand, but the h2xs tool was written to scan simple C-include header files (ending in .h) and generate most of the XS directly. Hence, h “2” (to) XS. Over time, more functions were added, including generating template files for the rest of the distribution. Now here we are, about to describe how to use h2xs for things that aren’t either h or xs. Amazing.

[77] If there’s more than one module in the distribution, it should be the name of the most important module. Others can be added later.

[78] The exact behavior and output of h2xs may vary depending upon your version of Perl.

[79] When you’re bored, you might find it amusing to do a search of the current CPAN for all places in which blah blah blah occurs.

[80] Well, you would be able to do this, if there were actually a module on CPAN named Island::Plotting::Maps.

[81] Technically, you don’t need :all because /^/ (include all symbols that match at least a beginning of string) does the same. Many people are familiar with typing :all, and it’s far more self-documenting than /^/ is, so include it if you can.

[82] The perlmod page includes an example that extracts the CVS/RCS version number for the module’s version number.

[83] The data immediately following the _ _END_ _ marker is available by reading from the DATA filehandle, which is a great way to include a small amount of constant data with your program. However, that’s not why we’re doing that here.

[84] The Module::Build module currently in development may replace all of that some day but is only in a pre-release form at the time of this writing.

[85] Please keep the number of questions to a minimum, however. Most people are irritated when asked a series of questions, especially when they are just upgrading your module. If possible, store the answers in a configuration module that you install so that a later invocation of your installer can pull the previous answers as defaults.

[86] Even if you weren’t the system administrator, you’d soon have all the powers of the system administrator.

[87] Had there been XS files or other more complex build steps, these also would have happened here.

[88] The perl that invoked the Makefile.PL is used for all configuration decisions. If you have more than one version of Perl installed on your system, be sure to execute the Makefile.PL with the correct one. From there, full paths are always used, so there’s no chance of mixing anything else up.

[89] If you’re playing along at home, be sure not to install this pretend module anywhere but a temporary, testing, directory. Although removing an installed module is generally difficult, you’ll be able to simply delete the testing directory, along with its contents.

[90] On a non-Unix system, or even a few odd Unix systems, you’ll see different behavior, but roughly the same overall result.

[91] If there’s more than one module, you need to designate the primary module in the Makefile.PL.