Clojure in a World of Containers

This is a write-up of the ideas and techniques I showed at re:Clojure 2019 in London. I talked about how the JVM interacts with Docker and demoed a few ideas for Fast JVM Startup, Efficient Images, and Respecting the Container Environment.

Slides Link - Video Link: It’s coming!

TL;DR, I recommend:

  • Using JDK11 or newer
  • Using an Alpine Linux base image
  • Separating your dependencies into a different image layer from your app
  • Being aware of JVM ergonomics

More complex, but also useful is Application CDS. Read on for details:

Fast JVM Startup

The technique I showed is called Class Data Sharing. At launch, a JVM needs to load classes. This usually means that for each class bytecode is read from disk, parsed, verified and eventually held in a datastructure in memory. So long as the classes (and JVM) are the same, this process always has the same result.

CDS is a way of caching that result in an “archive” file which can be read much more quickly at startup. As a bonus, this archive is memory-mapped read-only and can be shared between multiple running JVMs. By using Docker image layers appropriately we can even share the same memory between different containers.

There are two related JVM features:

  • Class Data Sharing - caching of classes distributed with the JDK (ie core Java classes)
  • Application Class Data Sharing - caching of application classes (ie your app and its dependencies)

Class Data Sharing

Since Java 5 CDS has been available in OpenJDK. You can generate the archive file by running:

  java -Xshare:dump

This creates the file $JAVA_HOME/lib/server/classes.jsa which is usually around 18Mb. You can then use that archive when you launch your app by running:

  java -Xshare:on <the rest of your startup command>

In a dockerfile, this will work:

FROM adoptopenjdk/openjdk11:alpine

RUN java -Xshare:dump

ADD target/reclojure-0.1.0-SNAPSHOT.jar .

CMD ["java", "-Xshare:on", "-jar", "reclojure-0.1.0-SNAPSHOT.jar"]

I recommend a tool called dive for exploring docker images. You can see what is added in each layer and more:

Screenshot of dive

Application CDS

Application CDS was originally an OracleJDK commercial feature, but was contributed to OpenJDK in JDK10. I recommend using the LTS version JDK11 or newer. There are a few steps to Application CDS:

  1. You will need to provide a list of which classes to include in the archive, which can be generated by running your app with -XX:DumpLoadedClassList
  2. Then create the archive with -Xshare:dump and -XX:SharedArchiveFile
  3. Now you can use that archive with -Xshare:on and -XX:SharedArchiveFile. You don’t need the list of classes at this point.

Here’s a worked example:

java -Xshare:off \   
     -XX:DumpLoadedClassList=appcds.classlist \
     -jar target/hello-reclojure-0.1.0-SNAPSHOT-standalone.jar

This generates the file appcds.classlist which you can inspect if you like. The first line is probably java/lang/Object. For a quick summary of where the classes come from, I used:

$ cat appcds.classlist|cut -d/ -f1|sort|uniq -c
   1788 clojure
      5 hello_reclojure
    747 java
    146 jdk
     91 sun

That’s 2777 classes for a “Hello World”. A web app with ring and compojure etc will likely be thousands more again.

Generate the archive with:

java -Xshare:dump \
     -XX:SharedClassListFile=appcds.classlist \
     -XX:SharedArchiveFile=appcds.cache \
     -jar target/hello-reclojure-0.1.0-SNAPSHOT-standalone.jar

The file appcds.cache can be added to your docker image and used at runtime:

FROM adoptopenjdk/openjdk11:alpine

# Regular CDS as before
RUN ["java", "-Xshare:dump"]

ADD target/hello-reclojure-0.1.0-SNAPSHOT-standalone.jar .

# Application CDS
RUN java -Xshare:off \   
	-XX:DumpLoadedClassList=appcds.classlist \
	-jar hello-reclojure-0.1.0-SNAPSHOT-standalone.jar \
    && \
    java -Xshare:dump \
	-XX:SharedClassListFile=appcds.classlist \
	-XX:SharedArchiveFile=appcds.cache \
	-jar hello-reclojure-0.1.0-SNAPSHOT-standalone.jar \
    && \
    rm appcds.classlist

CMD ["java", "-Xshare:on", "-XX:SharedArchiveFile=appcds.cache", "-jar", "hello-reclojure-0.1.0-SNAPSHOT-standalone.jar"]

Note that I’ve done steps 1 & 2 and deleted appcds.classlist in the same RUN command. This avoids having space taken up in the final image by the classlist as each dockerfile command creates a layer, and files added in one layer then removed later on still count toward the final size. This can be investigated using dive and looking at the “wasted space” stat.

Efficient Images

Total image size is important as images are copied over networks all the time. The first tip is to use the Alpine Linux base images produced by AdoptOpenJDK. I’ve done that above - using adoptopenjdk/openjdk11:alpine as your base image is enough. It saves just over 100mb compared with adoptopenjdk/openjdk11.

As well as reducing total image size, it’s a good idea to be thoughtful about how you use image layers. As I mentioned above, each command in a dockerfile produces a new layer. Layers are cached by docker and can be shared at runtime too. All layers are read-only and docker creates a Copy-On-Write (COW) layer so that processes which write to the filesystem are isolated from each other and can still share layers.

It makes sense to have a layer for your dependencies and a separate layer for your application. This means that every time you make an application code change that doesn’t change the dependencies you can reuse the “dependencies” layer. This implies not using uberjars!

Instead of this:

Image layers with uberjars

Aim for this:

Image layers with this jars

To get this effect, I recommend adding the lein-metajar plugin to your build. Once this is added, build your application with:

lein metajar

It’s a good idea to add AOT and direct linking to your metajar profile too with:

:profiles {:metajar {:aot :all
                     :direct-linking true}}

lein metajar builds your app into a thin jar in target/ and all the dependencies are copied into target/lib. The thin jar has metadata listing the relative links to the dependencies so you don’t need to set the classpath manually, but you do need to preserve the relative layout of the jar and dependencies in your Docker image. This dockerfile does the trick, assuming that you ran lein metajar already:

FROM adoptopenjdk/openjdk11:alpine

ADD target/lib /lib/
ADD target/reclojure-0.1.0-SNAPSHOT.jar .

CMD ["java", "-jar", "reclojure-0.1.0-SNAPSHOT.jar"]

Respecting the Container Environment

The JVM has a feature called “Ergonomics” which inspects the available hardware at startup and sets some tuning parameters appropriately. The best-known behaviour of ergonomics is that the heap size (ie memory available to your app) is set according to how much physical memory is available, but there is more to it than that. Thread-pool sizes are set depending on how many CPUs are available, and it’s reflected to your application by java.lang.Runtime::getAvailableCPUs. This is often used by libraries to size threadpools.

Docker allows us to restrict what hardware a running container has access to, and the JVM has been looking for container restrictions since JDK10. Some changed were backported to JDK8, so use 8u212 or newer if you’re on JDK8. Better still use JDK11 or even newer if you can, as improvements to container-aware ergonmics are still being added.

We can check the effect of ergonomics by running using the -XX:+PrintFlagsFinal flag to java and grepping the output for ergonomic. Without the grep we will see all the JVM flags, of which there are 666 at the current build of adoptopenjdk/openjdk11. Here’s how:

$ docker run adoptopenjdk/openjdk11:alpine java -XX:+PrintFlagsFinal -version  | grep ergonomic
     intx CICompilerCount                          = 4                                         {product} {ergonomic}
     uint ConcGCThreads                            = 2                                         {product} {ergonomic}
     uint G1ConcRefinementThreads                  = 8                                         {product} {ergonomic}
   size_t G1HeapRegionSize                         = 1048576                                   {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
   size_t InitialHeapSize                          = 262144000                                 {product} {ergonomic}
   size_t MarkStackSize                            = 4194304                                   {product} {ergonomic}
   size_t MaxHeapSize                              = 4173332480                                {product} {ergonomic}
   size_t MaxNewSize                               = 2503999488                                {product} {ergonomic}
   size_t MinHeapDeltaBytes                        = 1048576                                   {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 5836300                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122910970                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 122910970                              {pd product} {ergonomic}
    uintx ReservedCodeCacheSize                    = 251658240                              {pd product} {ergonomic}
     bool SegmentedCodeCache                       = true                                      {product} {ergonomic}
     bool UseCompressedClassPointers               = true                                 {lp64_product} {ergonomic}
     bool UseCompressedOops                        = true                                 {lp64_product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}
openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.5+10, mixed mode)

The last value is particularly important - if we restrict the memory available to the JVM to 1Gb by using docker run -m 1024m ... then UseG1GC = true is replaced by UseSerialGC = true. SerialGC and G1GC are different implementations of the Java Garbage Collector which have very different behaviours at runtime. Baeldung’s article on Java GC goes into a lot of detail and is recommended. Here’s a key quote, though:

[Serial GC] freezes all application threads when it runs. Hence, it is not a good idea to use it in multi-threaded applications like server environments.

Summary

Many many folks are using containers to build, test and deploy their JVM-based apps. I hope that my talk and this post have been useful to your understanding of how to take advantage of containers better. If you have any comments or questions, please get in touch: