Project Lombok: Creating Custom Transformations

Project Lombok aims to reduce Java boiler-plate via annotations that perform class transformations at compile time. Project Lombok comes with a decent set of transformations, but you may also want to create your own custom Lombok tranformations. In this blog, I will walk you through the process of extending Project Lombok to do a simple Hello World transformer. What I present here is an approach that worked for me. At the moment, there are scarce few resources out there on this subject. I started by reading Nicolas Frankel's blog and a post in the Project Lombok discussion group but mostly it came down to groking the Project Lombok source code. With that disclaimer, let's dive in. Overview Project Lombok runs as an annotation processor. The annotation processor acts as a dispatcher that delegates to Lombok annotation handlers (this is what we're going to create). Handlers are discovered via a framework called SPI. Lombok annotation handlers declare the specific annotation that they handle. When delegating to a handler, the Lombok annotation processor provides an object representing the Abstract Syntax Tree (AST) of the annotated node (e.g. class, method, field, etc). The Lombok annotation handler is free to modify the AST by injecting new nodes such as methods, fields and expressions. After the annotation processing stage, the compiler will generate byte code from the modified AST. Here's an overview of the compilation process and how Project Lombok fits in: The basic classes you'll need to write are: - Annotation class - Eclipse handler - Javac handler In this example, we will create a very simple Lombok transformation that adds a helloWorld method to any class annotated as @HelloWorld. This is a trivial and useless transformation but serves as a simple starting point. For example, given the following source code:

@HelloWorld
public class MyClass {

}

Our custom handler will transform the class to the following equivalent source:

public class MyClass {
  public void helloWorld() {
    System.out.println("Hello World");
  }
}

Annotation class This is the easy part. We just need to create an annotation called HelloWorld that can be applied to classes.

package lombok;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.SOURCE)
public @interface HelloWorld {}

This simple annotation can only be applied to Types (interface, class, enum). Since we plan on adding a concrete method, this annotation should only be used on classes but @Target doesn't give us that granularity. Our annotation handler will be responsible for generating an error if @HelloWorld is used on an interface. The annotation only needs to be retained in the source because we're only using the annotation to generate a method during compilation. The annotation is not needed at runtime. Handler Overview The handler class will be responsible for creating the AST that represents the helloWorld method and then injecting it into the AST of the Class declaration. A simplified AST for a Class (type) declaration looks something like this: Our handler will be adding a new Method Declaration to the Type Declaration. A Method Declaration is composed of several components. The AST for our Method Declration will have this form: Filling in the above template with our implementation specifics, the AST starts to resemble something that looks more like Java: Writing the Handler class Since AST modifications are compiler specific, we'll need to provide implementations for both Javac and Eclipse. If Intellij supported Lombok, we'd likely have to provide a 3rd implementation. Since NetBeans uses Javac, when we're done we'll be able to compile using commandline javac (ant and maven included), NetBeans and Eclipse. Both the Javac and Eclipse handlers must use "lombok" as the top-level package. In order for our handler to be discovered by the Lombok annotation processor, the Eclipse handler must be annotated as @ProviderFor(EclipseAnnotationHandler.class) and implement the EclipseAnnotationHandler interface. Likewise the Javac handler must be annotated as @ProviderFor(JavacAnnotationHandler.class) and implement the JavacAnnotationHandler interface. Here is the starting point for our Eclipse handler:

package lombok.eclipse.handlers;
import lombok.HelloWorld;
import lombok.core.AnnotationValues;
import lombok.eclipse.EclipseAnnotationHandler;
import lombok.eclipse.EclipseNode;
import org.mangosdk.spi.ProviderFor

@ProviderFor(EclipseAnnotationHandler.class)
public class HandleHelloWorld implements EclipseAnnotationHandler<HelloWorld> {

 @Override
 public boolean handle(AnnotationValues<HelloWorld> annotation, Annotation ast,
   EclipseNode annotationNode) {
  // our logic here
 }
}

Here is the starting point for our Javac handler:

package lombok.javac.handler;

import lombok.HelloWorld;
import lombok.core.AnnotationValues;
import lombok.javac.JavacAnnotationHandler;
import lombok.javac.JavacNode;
import org.mangosdk.spi.ProviderFor

@ProviderFor(JavacAnnotationHandler.class)
@SuppressWarnings("restriction") 
public class HandleHelloWorld implements JavacAnnotationHandler<HelloWorld>{

 public boolean handle(AnnotationValues<HelloWorld> annotation, JCAnnotation ast,
   JavacNode annotationNode) {
          // logic here
        }
}

Handle Logic Our handle method will need to do the following:

1. Mark annotation as processed (Javac only)
2. Create the helloWorld method
3. Inject the helloWorld method into the AST of the annotated class

Our handle method looks very similar for both Eclipse and Javac. Eclipse version:

@Override
 public boolean handle(AnnotationValues<HelloWorld> annotation, Annotation ast,
   EclipseNode annotationNode) {
  EclipseNode typeNode = annotationNode.up();

  MethodDeclaration helloWorldMethod = 
   createHelloWorld(typeNode, annotationNode, annotationNode.get(), ast);
  
  EclipseHandlerUtil.injectMethod(typeNode, helloWorldMethod);
  
  return true;
 }

Javac version:

@Override public boolean handle(AnnotationValues<HelloWorld> annotation, JCAnnotation ast,
   JavacNode annotationNode) {
  JavacHandlerUtil.markAnnotationAsProcessed(annotationNode, HelloWorld.class);
  JavacNode typeNode = annotationNode.up();
  
  JCMethodDecl helloWorldMethod = createHelloWorld(typeNode);
  
  JavacHandlerUtil.injectMethod(typeNode, helloWorldMethod);
  return true;
 }

Lombok provides our handler with the AST of the annotation node. We know that the annotation can only be applied to a Type. To get the AST for the Type (Class), we call annotationNode.up(). The annotation is a child of the Type AST, so by calling up() we get the parent AST which is the Type AST we need to modify. For simplicity, I've omitted the logic to check that the Type is actually a Class. Next we create the node representing createHelloWorld method node. We still need to write this method which we'll look at in the next section. Once we've created the method method, we inject it the AST of our Type. This is accomplished by Lombok utility classes JavacHandlerUtil and EclipseHandlerUtil. We'll also be using these utility classes to help implement the createHelloWorld method. Creating the helloWorld Method Now we get the crux of the problem: creating the helloWorld method. The basic recipe will be the same for both Javac and Eclipse:

• Start with a method node.
• Add the return type, parameters, access level, throw clause, etc to the method node.
• Create an expression statement to represent System.out.println("Hello World")
• Add the expression to the method node.

Implementing this logic is by far the hardest part. We'll need to figure out how to programatically create the various AST objects. To really grok this code, you'll need to look at the Java source for the various classes in the Javac and Eclipse syntax tree packages. You'll see that Eclipse and Javac implementations differ drastically. Here is the Javac implementation

 private JCMethodDecl createHelloWorld(JavacNode type) {
  TreeMaker treeMaker = type.getTreeMaker();
  
  JCModifiers           modifiers          = treeMaker.Modifiers(Modifier.PUBLIC);
  List<JCTypeParameter> methodGenericTypes = List.<JCTypeParameter>nil();
  JCExpression          methodType         = treeMaker.TypeIdent(TypeTags.VOID);
  Name                  methodName         = type.toName("helloWorld");
  List<JCVariableDecl>  methodParameters   = List.<JCVariableDecl>nil();
  List<JCExpression>    methodThrows       = List.<JCExpression>nil();

  JCExpression printlnMethod = 
   JavacHandlerUtil.chainDots(treeMaker, type, "System", "out", "println"); 
  List<JCExpression> printlnArgs = List.<JCExpression>of(treeMaker.Literal("hello world"));
  JCMethodInvocation printlnInvocation = 
   treeMaker.Apply(List.<JCExpression>nil(), printlnMethod, printlnArgs);
  JCBlock methodBody = 
   treeMaker.Block(0, List.<JCStatement>of(treeMaker.Exec(printlnInvocation)));

  JCExpression defaultValue = null;

  return treeMaker.MethodDef(
    modifiers, 
    methodName, 
    methodType,
    methodGenericTypes, 
    methodParameters, 
    methodThrows, 
    methodBody, 
    defaultValue 
   );
 }

With Javac, we need to generate an object for all parts of the method: modifiers, generic types, return type, method name, parameters, throw clause, and method body. Creating these object is via a TreeMaker class that is part of Javac. TreeMaker is a factory class for creating all the different types of nodes. The method body is comprised of various nodes as well: method reference to System.out.println, arguments to println which includes the String literal "hello world". It's all tied together as a method invocation of the println method reference. Finally we use treeMaker to combine all the pieces into a method defintion. Now let's look at the Eclipse implementation:

private MethodDeclaration createHelloWorld(EclipseNode typeNode, EclipseNode errorNode, ASTNode astNode, Annotation source) {
  TypeDeclaration typeDecl = (TypeDeclaration) typeNode.get();

  MethodDeclaration method = new MethodDeclaration(typeDecl.compilationResult);
  Eclipse.setGeneratedBy(method, astNode);
  method.annotations = null;
  method.modifiers = Modifier.PUBLIC;
  method.typeParameters = null;
  method.returnType = new SingleTypeReference(TypeBinding.VOID.simpleName, 0);
  method.selector = "helloWorld".toCharArray();
  method.arguments = null;
  method.binding = null;
  method.thrownExceptions = null;
  method.bits |= ECLIPSE_DO_NOT_TOUCH_FLAG;
  
  NameReference systemOutReference = createNameReference("System.out", source);
  Expression [] printlnArguments = new Expression[] { 
   new StringLiteral("Hello World".toCharArray(), astNode.sourceStart, astNode.sourceEnd, 0)
  };
  
  MessageSend printlnInvocation = new MessageSend();
  printlnInvocation.arguments = printlnArguments;
  printlnInvocation.receiver = systemOutReference;
  printlnInvocation.selector = "println".toCharArray();
  Eclipse.setGeneratedBy(printlnInvocation, source);
  
  method.bodyStart = method.declarationSourceStart = method.sourceStart = astNode.sourceStart;
  method.bodyEnd = method.declarationSourceEnd = method.sourceEnd = astNode.sourceEnd;
  method.statements = new Statement[] { printlnInvocation };
  return method;
 }

With the Eclipse implementation, instead of using a TreeMaker to create the various components, we just create the components via their constructors and attach child nodes to parents via property assignments. Eclipse developers apparently aren't big on encapsulation. They also seem to be fixated on using char arrays instead of Strings. Another thing we need to deal with is telling Eclipse that the HelloWorld annotation was responsible for generating the method. That way if there is an error with our generated method, Eclipse will associate the error with @HelloWorld annotation. The complete source along with a Maven project setup can be found here Going beyond HelloWorld Hopefully by now the architecture of Lombok is a little clearer. Once you've mastered Hello World you're ready to start creating more useful transformations. As you would expect, more interesting behavior requires more complex AST transformations. You'll probably spend a lot of time looking at the source code for the handlers that the Project Lombok maintainers have already created. This is the downside of using private APIs to generate code. You're on your own to figure out how these APIs work. Appendix - Project Setup Before you can build any Lombok transformations, you'll need a way to build your code. Where to put your code The first thing you need to decide is where your code will live. There are 2 options: 1. Fork Project Lombok source 2. Create a new source project Let's look at both options in more detail: Forking Lombok By far the easiest way to get started is just to clone the Project Lombok git repo and place your custom classes alongside the core Lombok classes. This is going to be the fastest and easiest way to get started. You won't have to deal with configuring library dependencies or patching Eclipse. Down the road you can easily move to a standalone project for your custom classes. The downside is that you are forking the project is that you'll have to adopt Lombok's project structure and build system (Ant+Ivy). Using ant, you can generate an Eclipse project which makes things easy. Cloning Project Lombok is simple. You'll need to have git installed. Then run the command:

git clone https://github.com/rzwitserloot/lombok.git

Navigating source project The top-level lombok directory contains a build.xml. To compile and build the lombok jar run: ant To generate an eclipse project run: ant eclipse The lombok jar is generated under the dist subdirectory. Once you've added your custom handlers, you'll need to patch Eclipse with your updated jar by running: java -jar <new lombok jar>: Source code is organized into the following directories: src/core/lombok - Annotations src/core/lombok/eclipse/handlers - Eclipse annotation handlers src/core/lombok/javac/handlers - Javac annotation handlers Creating a stand-alone project Skip this section if you chose to fork Project Lombok. Instead of forking the Project Lombok repo, another more complicated approach is to place your custom Lombok extensions in a separate standalone project. Setting up custom project comes with some pain though. You'll need to handle dependency management and Eclipse patching yourself. At a minimum your project will need to provide the following libraries: - Project Lombok - SPI - Eclipse Core JDT - Sun's tools.jar (this contains javac classes) The example I provide here uses Maven2. This presents a challenge because SPI library does not have a public maven repo (that I could find), nor does Eclipse Core JDT or tools.jar. So these have to be installed manually into your local repo. The easiest way to obtain theses jars is by checking out Project Lombok and doing a build (see above). The build will download dependent libraries under lib/build. The lombok dist jar can be found under the dist/. With Maven, you'll need to install these jars these jars in your local repo. Download the Maven project for the HelloWorld project here Once you've built the jar, you'll need to patch it manually into Eclipse. Copy the jar to the same folder as your eclipse.ini. Edit eclipse.ini, and add your jar to the bootclasspath. Here is an example for adding the hello-lombok.jar from the example project to eclipse.ini.

...
-javaagent:lombok.jar
-Xbootclasspath/a:lombok.jar
-Xbootclasspath/a:hello-lombok.jar
...