Chapter 12. Java Platform Modules

With the release of Java 9, the platform finally gained the long-awaited modules system. This feature had originally been intended to ship as part of Sun’s Java 7 release, before the acquisition by Oracle. However, the task proved to be far more complex and subtle than anticipated.

When Oracle acquired Java (as part of the technology they received from Sun Microsystems) Mark Reinhold, Java’s Chief Architect, proposed “Plan B”, which reduced the scope of Java 7 to allow for a quicker release.

Java platform modules (“Project Jigsaw”) was pushed back, along with lambdas, to Java 8. However, during the development of Java 8, the size and complexity of the feature led to a decision that, rather than delay Java 8 (and availability of lambdas and other highly desired features), it would be better to defer modules to Java 9.

The end result was that the modules capability was delayed first to Java 8 and then to Java 9. Even then, the scope of the work led to substantial delays in the release of Java 9, and so modules did not actually ship until September 2017.

In this chapter we will provide a basic introduction to the Java Platform Modules System (JPMS). However, this is a large and complex subject—interested readers may well require a more in-depth reference, such as Java 9 Modularity by Sander Mak and Paul Bakker (O’Reilly).

Warning

Modules are a relatively advanced feature that are primarily about packaging and deploying entire applications and their dependencies. It is not necessary for a new Java programmer to fully understand this topic while still learning how to write simple Java programs.

Due to the advanced nature of modules, this chapter assumes you are familiar with a modern Java build tool, such as Gradle or Maven. If you are new to Java, you can safely ignore references to those tools and just read the chapter to get a first, high-level overview of JPMS.

Why Modules?

There were several major motivating reasons for wanting to add modules to the Java platform. These included a desire for:

  • Strong encapsulation

  • Well-defined interfaces

  • Explicit dependencies

These are all language (and application design) level, and they were combined with the promise of new platform-level capabilities as well:

  • Scalable development

  • Improved performance (especially startup time) and reduced footprint

  • Reduced attack surface and better security

  • Evolvable internals

On the encapsulation point, this was driven by the fact that the original language specification only supports private, public, protected, and package-private visibility levels. There is no way to control access in a more fine-grained way to express concepts such as:

  • Only specified packages are available as an API—others are internal and may not be accessed

  • Certain packages can be accessed by this list of packages but no others

  • Defining a strict exporting mechanism

The lack of these and related capabilities has been a significant shortcoming when architecting larger Java systems. Not only that, but without a suitable protection mechanism, it would be very difficult to evolve the internals of the JDK—as nothing prevents user applications from directly accessing implementation classes.

The modules system attempts to address all of these concerns at once and to provide a solution that works both for the JDK and for user applications.

Modularizing the JDK

The monolithic JDK that shipped with Java 8 was the first target for the modules system, and the familiar rt.jar was broken up into modules. This built upon work done in Java 8 before modules were pushed back into Java 9—the Compact Profiles feature.

java.base is the module that represents the minimum that’s actually needed for a Java application to start up. It contains core packages, such as:

java.io
java.lang
java.math
java.net
java.nio
java.security
java.text
java.time
java.util
javax.crypto
javax.net
javax.security

along with some subpackages and non-exported implementation packages such as sun.text.resources. Some of the differences in compilation behavior between Java 8 and modular Java can be seen in this simple program, which extends an internal public class contained in java.base:

import java.util.Arrays;
import sun.text.resources.FormatData;

public final class FormatStealer extends FormatData {
    public static void main(String[] args) {
        FormatStealer fs = new FormatStealer();
        fs.run();
    }

    private void run() {
        String[] s = (String[]) handleGetObject("japanese.Eras");
        System.out.println(Arrays.toString(s));

        Object[][] contents = getContents();
        Object[] eraData = contents[14];
        Object[] eras = (Object[])eraData[1];
        System.out.println(Arrays.toString(eras));
    }
}

When compiled and run under Java 8, this produces a list of Japanese eras:

[, Meiji, Taisho, Showa, Heisei]
[, Meiji, Taisho, Showa, Heisei]

However, attempting to compile the code on Java 11 produces this error message:

$ javac javanut7/ch12/FormatStealer.java
javanut7/ch12/FormatStealer.java:4:
        error: package sun.text.resources is not visible
import sun.text.resources.FormatData;
               ^
  (package sun.text.resources is declared in module
        java.base, which does not export it to the unnamed module)
javanut7/ch12/FormatStealer.java:14: error: cannot find symbol
        String[] s = (String[]) handleGetObject("japanese.Eras");
                                ^
  symbol:   method handleGetObject(String)
  location: class FormatStealer
javanut7/ch12/FormatStealer.java:17: error: cannot find symbol
        Object[][] contents = getContents();
                              ^
  symbol:   method getContents()
  location: class FormatStealer
3 errors

With a modular Java, even classes that are public cannot be accessed unless they are explicitly exported by the module they are defined in. We can temporarily force the compiler to use the internal package (basically reasserting the old access rules) with the --add-exports switch, like this:

$ javac --add-exports java.base/sun.text.resources=ALL-UNNAMED \
        javanut7/ch12/FormatStealer.java
javanut7/ch12/FormatStealer.java:5:
        warning: FormatData is internal proprietary API and may be
        removed in a future release
import sun.text.resources.FormatData;
                         ^
javanut7/ch12/FormatStealer.java:7:
        warning: FormatData is internal proprietary API and may be
        removed in a future release
public final class FormatStealer extends FormatData {
                                         ^
2 warnings

We need to specify that the export is being granted to the unnamed module, as we are compiling our class standalone and not as part of a module. The compiler warns us that we’re using an internal API and that this might break with a future release of Java.

Interestingly, if our code is run on Java 11, then the output produced is slightly different:

[, Meiji, Taisho, Showa, Heisei, NewEra]
[, Meiji, Taisho, Showa, Heisei, NewEra]

This is because the Japanese era will change from Heisei (the current era) to a new one on May 1, 2019. By tradition, the name of the new era is not known ahead of time, so “NewEra” is a placeholder for the name that will be replaced by the official name in a future release. Unicode code point U+32FF has been reserved for the character that will represent the new era name.

Although java.base is the absolute minimum that an application needs to start up, at compile time we want the visible platform to be as close to the expected (Java 8) experience as possible.

This means that we use a much larger set of modules, contained under an umbrella module, java.se. This module has a dependency graph, shown in Figure 12-1.

JN7 1201
Figure 12-1. Module dependency graph of java.se

This brings in almost all of the classes and packages that most Java developers expect and use. However, the modules defining the CORBA and Java EE APIs are not required by java.se, but they are required by the java.se.ee module.

Warning

This means that any project that depends on the Java EE APIs (or CORBA) will not compile by default on Java 9 onward and a special build config must be used.

This includes APIs like JAXB—to make such projects compile, java.se.ee must be explicitly included in the build.

As well as these changes to compilation visibility, due to the modularization of the JDK, the modules system is also intended to allow developers to modularize their own code.

Writing Your Own Modules

In this section, we will discuss the basic concepts needed to start writing modular Java applications.

Basic Modules Syntax

The key to modularizing is the new file module-info.java, which contains a description of a module. This is referred to as a module descriptor.

A module is laid out for compilation correctly on the filesystem in the following way:

  • Below the source root of the project (src), there needs to be a directory named the same as the module (the moduledir).

  • Inside the moduledir is the module-info.java, at the same level as where the packages start from.

The module info is compiled to a binary format, module-info.class, which contains the metadata that will be used when a modular runtime attempts to link and run our application. Let’s look at a simple example of a module-info.java:

module kathik {
    requires java.net.http;

    exports kathik.main;
}

This introduces some new syntax: module, exports, and requires—but these are not really full keywords in the accepted sense. As stated in the Java Language Specification SE 9:

A further ten character sequences are restricted keywords: open, module, requires, transitive, exports, opens, to, uses, provides, and with. These character sequences are tokenized as keywords solely where they appear as terminals in the ModuleDeclaration and ModuleDirective productions.

This means that these keywords can only appear in the module metadata and are compiled into the binary format by javac. The meaning of the major restricted keywords is:

module

Starts the module’s metadata declaration

requires

Lists a module on which this module depends

exports

Declares which packages are exported as an API

The remaining keywords will be introduced throughout the rest of the chapter.

In our example, this means that we’re declaring a module kathik that depends upon the module java.net.http that was standardized in Java 11 (as well as an implicit dependency on java.base). The module exports a single package, kathik.main, which is the only package in this module that will be accessible from other modules at compile time.

Building a Simple Modular Application

As an example, let’s build a simple tool that checks whether websites are using HTTP/2 yet, using the API that we met in Chapter 10:

import static java.net.http.HttpResponse.BodyHandlers.ofString;

public final class HTTP2Checker {
    public static void main(String[] args) throws Exception {
        if (args.length == 0) {
            System.err.println("Provide URLS to check");
        }
        for (final var location : args) {
            var client = HttpClient.newBuilder().build();
            var uri = new URI(location);
            var req = HttpRequest.newBuilder(uri).build();

            var response = client.send(req,
                    ofString(Charset.defaultCharset()));
            System.out.println(location +": "+ response.version());
        }
    }
}

This relies on two modules—java.net.http and the ubiquitous java.base. The module file for the app is very simple:

module http2checker {
    requires java.net.http;
}

Assuming a simple, standard module layout, this can be compiled like this:

$ javac -d out/http2checker\
    src/http2checker/javanut7/ch12/HTTP2Checker.java\
    src/http2checker/module-info.java

This creates a compiled module in the out/ directory. For use, it needs to be packaged as a JAR file:

$ jar -cfe httpchecker.jar javanut7.ch12.HTTP2Checker\
    -C out/http2checker/ .

We used the -e switch to set an entry point for the module—that is, a class to be executed when we use the module as an application. Let’s see it in action:

$ java -jar httpchecker.jar http://www.google.com
http://www.google.com: HTTP_1_1
$ java -jar httpchecker.jar https://www.google.com
https://www.google.com: HTTP_2

This shows that, at the time of writing, Google’s website was serving its main page over HTTPS using HTTP/2, but still over HTTP/1.1 for legacy HTTP service.

Now that we have seen how to compile and run a simple modular application, let’s meet some more of the core features of modularity that are needed to build and run full-size applications.

Open Modules

As noted, simply marking a method public no longer guarantees that the element will be accessible everywhere. Instead, accessibility now depends also upon whether the package containing that element is exported by its defining module. Another major issue in the design of modules is the use of reflection to access classes.

Reflection is such a wide-ranging, general-purpose mechanism that it is difficult to see, at first glance, how it can be reconciled with the strong encapsulation goals of JPMS. Worse yet, so many of the Java ecosystem’s most important libraries and frameworks rely on reflection (e.g., unit testing, dependency injection, and many more) that not having a solution for reflection would make modules impossible to adopt for any real application.

The solution provided is twofold. First, a module can declare itself an open module, like this:

open module kathik {
    exports kathik.api;
}

This declaration has the effect that:

  • All packages in the module can be accessed via reflection

  • Compile-time access is not provided for non-exported packages

This means that the configuration behaves like a standard module at compile time. The overall intent is to provide simple compatibility with existing code and frameworks and ease migration pain. With an open module, the previous expectation of being able to reflectively access code is restored. In addition, the setAccessible() hack that allows access to private and other methods that would not normally permit access is preserved for open modules.

Finer-grained control over reflective access is also provided via the opens restricted keyword. This selectively opens specific packages for reflective access by explicitly declaring packages to be accessible via reflection:

module kathik {
    exports kathik.api;
    opens kathik.domain;
}

This type of usage is likely to be useful when, for example, you are providing a domain model to be used by a module-aware object-relational mapping (ORM) system that needs full reflective access to the core domain types of a module.

It is possible to go further and restrict reflective access to specific client packages, using the to restricted keyword. Where possible, this can be a good design principle, but of course such a technique will not work well with a general-purpose framework such as an ORM.

Note

In a similar way, it is possible to restrict the export of a package to only specific external packages. However, this feature was largely added to help with the modularization of the JDK itself, and it has limited applicability to user modules.

Not only that, but it is also possible to both export and open a package, but this is not recommended—during migration, access to a package should ideally be either compile-time or reflective but not both.

In the case where reflective access is required to a package now contained in a module, the platform provides some switches to act as band-aids for the transitional period. In particular, the java option --add-opens module/package=ALL-UNNAMED can be used to open a specific package of module for reflective access to all code from the classpath, overriding the behavior of the modules system.

Tip

For code that is already modular, it can also be used to allow reflective access to a specific module.

When you are migrating to modular Java, any code that reflectively accesses internal code of another module should be run with that switch at first, until the situation can be remediated.

Related to this issue of reflective access (and a special case of it) is the issue of widespread use of internal platform APIs by frameworks. This is usually characterized as the “Unsafe problem” and we will encounter it toward the end of the chapter.

Multi-Release JARs

To explain the problem that is solved by multi-release JARs, let’s consider a simple example: finding the process ID (PID) of the currently executing process (i.e., the JVM that’s executing our code).

Note

We didn’t use the HTTP/2 example from earlier on, as Java 8 doesn’t have an HTTP/2 API—so we would have had to do a huge amount of work (essentially a full backport!) to provide the equivalent functionality for 8.

This may seem like a simple task, but on Java 8 this requires a surprising amount of boilerplate code:

public class GetPID {
    public static long getPid() {
        // This rather clunky call uses JMX to return the name that
        // represents the currently running JVM. This name is in the
        // format <pid>@<hostname>—on OpenJDK and Oracle VMs only—there
        // is no guaranteed portable solution for this on Java 8
        final String jvmName =
            ManagementFactory.getRuntimeMXBean().getName();
        final int index = jvmName.indexOf('@');
        if (index < 1)
            return -1;

        try {
            return Long.parseLong(jvmName.substring(0, index));
        } catch (NumberFormatException nfe) {
            return -1;
        }
    }
}

As we can see, this is nowhere near as straightforward as we might like. Worse still, it is not supported in a standard way across all Java 8 implementations. Fortunately, from Java 9 onward we can use the new ProcessHandle API, like this:

public class GetPID {
    public static long getPid() {
        // Use new Java 9 Process API...
        ProcessHandle processHandle = ProcessHandle.current();
        return processHandle.getPid();
    }
}

This now utilizes a standard API, but it leads to an essential problem: how can the developer write code that is guaranteed to run on all current Java versions?

What we want is to build and run a project correctly in multiple Java versions. We want to depend on library classes that are only available in later versions, but still run on an earlier version by using some code “shims.” The end result must be a single JAR and we do not require the project to switch to a multi-module format—in fact, the JAR must work as an automatic module.

Let’s look at an example project that has to run correctly in both Java 8 and Java 11. The main codebase is built with Java 8 and the Java 11 portion must be built with Java 11. This part of the build must be isolated from the main codebase to prevent compilation failures, although it can depend on the build artifacts of the Java 8 build.

To keep the build configuration simple, this feature is controlled using an entry in MANIFEST.MF within the JAR file:

Multi-Release: True

The variant code (i.e., that for a later version) is then stored in a special directory in META-INF. In our case, this is META-INF/versions/11.

For a Java runtime that implements this feature, any classes in the version-specific directory override the versions in the content root. On the other hand, for Java 8 and earlier versions, both the manifest entry and the versions/ directory are ignored and only the classes in the content root are found.

Migrating to Modules

Many Java developers are facing the question of when they should migrate their applications to use modules.

Tip

Modules should be the default for all greenfield apps, especially those that are architected in a microservices style.

When considering a migration of an existing app (especially a monolithic design), you can use the following roadmap:

  1. First upgrade the application runtime to Java 11 (running from the classpath initially)

  2. Identify any application dependencies that have been modularized and migrate those dependencies to modules

  3. Retain any non-modularized dependencies as automatic modules

  4. Introduce a single monolithic module of all application code

At this point, a minimally modularized application should now be ready for production deployment. This module will usually be an open module at this stage of the process. The next step is architectural refactoring, and it is at this point that applications can be broken out into individual modules as needed.

Once the application code runs in modules, it can make sense to limit reflective access to your code via opens. This access can be restricted to specific modules (such as ORM or dependency injection modules) as a first step toward removing any unnecessary access.

For Maven users, it’s worth remembering that Maven is not a modules system, but it does have dependencies—and (unlike JPMS dependencies) they are versioned. The Maven tooling is still evolving to fully integrate with JPMS (and many plug-ins have not caught up yet at the time of this writing). However, some general guidelines for modular Maven projects are emerging, specifically:

  • Aim to produce one module per Maven POM

  • Don’t modularize a Maven project until you are ready (or have an immediate need to)

  • Remember that running on a Java 11 runtime does not require building on a Java 11 toolchain

The last point indicates that one path for migration of Maven projects is to start by building as a Java 8 project and ensuring that those Maven artifacts can deploy cleanly (as automatic modules) on a Java 11 runtime. Only once that first step is working properly should a full modularization be undertaken.

There is some good tooling support available to help with the modularization process. Java 8 and up ships with jdeps (see Chapter 13), a tool for determining what packages and modules your code depends upon. This is very helpful for migrations from Java 8 to 11, and the use of jdeps when rearchitecting is recommended.

Custom Runtime Images

One of the key goals of JPMS is the possibility that applications may need not every class present in the traditional monolithic runtime of Java 8, and instead can manage with a smaller subset of modules. Such applications can have a much smaller footprint in terms of startup time and memory overhead. This can be taken further: if not all classes are needed, then why not ship an application together with a reduced, custom runtime image that only includes what’s necessary?

To demonstrate the idea, let’s package the HTTP/2 checker into a standalone tool with a custom runtime. We can use the jlink tool (which has been part of the platform since Java 9) to achieve this as follows:

$ jlink --module-path httpchecker.jar:$JAVA_HOME/jmods \
--add-modules http2checker \
--launcher http2chk=http2checker \
--output http2chk-image

Note that this assumes that the JAR file httpchecker.jar was created with a main class (aka entry point). The result is an output directory, http2chk-image, which is about 39M in size, much less than the full image, especially taking into account that because the tool uses the new HTTP module it requires the libraries for security, crypto, and so on.

From within the custom image directory we can run the http2chk tool directly, and see that it works even when the machine does not have the required version of java:

$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
$ ./bin/http2chk https://www.google.com
https://www.google.com: HTTP_2

The deployment of custom runtime images is still a very new tool, but it has great potential to reduce your code footprint and help Java remain competitive in the age of microservices. In the future, jlink could even be combined with new approaches to compilation, including the Graal compiler, which can be used as an ahead-of-time (AOT) compiler as well as a JIT compiler (see the jaotc ). As of Java 11, however, combining jlink and jaotc does not seem to offer any conclusive performance gains.

Issues with Modules

The modules system, despite being the flagship feature of Java 9 and having had a large amount of engineering time devoted to it, is not without its problems. This was, perhaps, inevitable—the feature fundamentally changes the nature of how Java applications are architected and delivered. It would have been almost impossible for modules to avoid running up against some problems when trying to retrofit over the large, mature ecosystem that is Java.

Unsafe and Related Problems

The class sun.misc.Unsafe is a class that is both widely used and popular with framework writers and other implementors within the Java world. However, it is an internal implementation class and is not part of the standard API of the Java platform (as the package name clearly indicates). The class name also provides a fairly strong clue that this is not really intended for use by Java applications.

Unsafe is an unsupported, internal API and so could be withdrawn or modified by any new Java version, without regard to the effect on user applications. Any code which does use it is technically directly coupled to the HotSpot VM, and is also potentially nonstandard and may not run on other implementations.

Although not an official part of Java SE in any way, Unsafe has become a de facto standard and key part of the implementation of basically every major framework in one way or another. Over subsequent versions it has evolved into a kind of dumping ground for nonstandard but necessary features. This admixture of features is a real mixed bag, with varying degrees of safety provided by each capability. Example uses of Unsafe include:

  • Fast de-/serialization

  • Threadsafe 64-bit sized native memory access (e.g., offheap)

  • Atomic memory operations (e.g., Compare-and-Swap)

  • Fast field/memory access

  • Multi-operating system replacement for JNI

  • Access to array items with volatile semantics

The essential problem is that many frameworks/libraries were unable to move to Java 9 without replacement for some Unsafe features. This impacts everyone using any libraries from a wide range of frameworks—basically every application in the Java ecosystem.

To fix this problem, Oracle created new supported APIs for some of the needed functionality, and segregated APIs that could not be encapsulated in time into a module, jdk.unsupported, which makes it clear that this is not a supported API and that developers use it at their own risk.

This gives Unsafe a temporary pass (which is strictly limited time) while encouraging library and framework developers to move to the new APIs. An example of a replacement API is VarHandles. These extend the Method Handles concept (from Chapter 11), and add new functionality, such as concurrency barrier modes for Java 9. These, along with some modest updates to JMM, are intended to produce a standard API for accessing new low-level processor features without allowing developers full access to dangerous capabilities, as were found in Unsafe.

More details about Unsafe and related low-level platform techniques can be found in Optimizing Java (O’Reilly).

Slow Adoption Rates

With the release of Java 9, the Java release model fundamentally changed. Java 8 and 9 used the “keystone release” model—where one star feature (such as lambdas or modules) essentially defines the release and so the ship date is determined by when the feature is done. The problem with this model is that it can cause inefficiencies due to uncertainty about when versions will ship. In particular, a small feature that just misses a release will have to wait a long time for the next major release.

As a result, from Java 10 onward, a new release model was adopted, which introduces strict time-based versioning. This involves:

  • Java releases are now classified as “feature” releases, which occur at a regular cadence of once every six months.

  • Features are not merged into the platform until they are essentially complete.

  • The mainline repo is in a releasable state at all times.

These releases are only good for six months, after which time they are no longer supported. Every three years, a special release is designated as a long-term support (LTS) release, which has extended support available.

Although the Java community is generally positive on the new faster release cycle, adoption rates of Java 9 and above have been much smaller than for previous releases. This may be due to the desire of larger enterprises to have longer support cycles, rather than upgrading to each feature release after only six months.

It is also the case that the upgrade from Java 8 to 9 is not a drop-in replacement (unlike 7 to 8, and to a lesser extent 6 to 7). The modules subsystem fundamentally changes many aspects of the Java platform, even if end-user applications do not take advantage of modules. This makes teams reluctant to upgrade from 8 unless they can see clear benefits in doing so.

This leads to “chicken-and-egg” problems, where teams don’t move, because they perceive that the libraries and other components that they depend upon don’t yet support modular Java. On the other side of the equation, the companies and open source communities who maintain libraries and other tools may well feel that because the user population for modular Java is still so small, it’s a low priority to support versions beyond 8.

The arrival of Java 11, the first post-8 LTS release, may help this situation, as it provides a supported environment that enterprise teams may find more comfortable as a migration target.