Friday, November 26, 2010

Project Lombok - Trick Explained

In my previous blog post, I introduced Project Lombok, a library that can inject code into a class at compile time. When you see it in action, it almost seems magical. I will attempt to explain the trick behind the magic.

Java Compilation

To understand how Project Lombok works, one must first understand how Java compilation works. OpenJDK provides an excellent overview of the compilation process. To paraphrase, compilation has 3 stages:
1. Parse and Enter
2. Annotation Processing
3. Analyse and Generate



In the Parse and Enter phase, the compiler parses source files into an Abstract Syntax Tree (AST). Think of the AST as the DOM-equivalent for Java code. Parsing will only throw errors if the syntax is invalid. Compilation errors such as invalid class or method usage are checked in phase 3.

In the Annotation Processing phase, custom annotation processors are invoked. This is considered a pre-compilation phase. Annotation processors can do things like validate classes or generate new resources, including source files. Annotation processors can generate errors that will cause the compilation process to fail. If new source files are generated as a result of annotation processing, then compilation loops back to the Parse and Enter phase and the process is repeated until no new source files are generated.

In the last phase, Analyse and Generate, the compiler generates class files (byte code) from the Abstract Syntax Trees generated in phase 1. As part of this process, the AST is analyzed for broken references (e.g. class not found, method not found), valid flow is checked (e.g. no unreachable statements), type erasure is performed, syntactic sugar is desugared (e.g. enhanced for loops become iterator loops) and finally, if everything is successful, class files are written out.

Project Lombok and Compilation

Project Lombok hooks itself into the compilation process as an annotation processor. But Lombok is not your normal annotation processor. Normally, annotation processors only generate new source files whereas Lombok modifies existing classes.

The trick is that Lombok modifies the AST. It turns out that changes made to the AST in the Annotation Processing phase will be visible to the Analyse and Generate phase. Thus, changing the AST will change the generated class file. For example, if a method node is added to the AST, then the class file will contain that new method. By modifying the AST, Lombok can do things like generate new methods (getter, setter, equals, etc) or inject code into an existing method (e.g. cleaning up resources).

Trick or Hack?

Some people call Lombok's trick a hack, and I'd agree. But don't pass judgement yet. Like any hack, you should examine the risk/reward and alternatives before determining if you are comfortable with it.

The "hack" in Lombok is that, strictly speaking, the annotation processing spec doesn't allow you to modify existing classes. The annotation processing API doesn't provide a mechanism for changing the AST of a class. The clever people at Project Lombok got around this through some unpublished APIs of javac. Since Eclipse uses an internal compiler, Lombok also needs access to internal APIs of the Eclipse compiler.

If Java officially supported compile-time AST transformations then Lombok wouldn't need to rely on backdoor APIs. This makes Project Lombok vulnerable to future changes in the JDK. There is no guarantee the private APIs won't change in a later JDK and break Project Lombok. If that happens, then you're left hoping that the guys at Lombok will be responsive about patching their library to work with the new JDK. Same thing goes for the new Eclipse compilers. Given how often we get a new version of Java, this may not be that big of an issue.


Alternatives in Java

There are other alternatives for modifying the behavior of classes. One approach is to use byte-code manipulation at runtime via a library like CGLib or ASM. This is how Hibernate is able to do things like lazily initialize a persistent Collection the first time it is accessed. In general, this can be used to enhance the behavior of existing methods. This trick could possibly be used to implement the @Cleanup behavior in Lombok, so that a resource is closed when it goes out of scope. Runtime byte-code manipulation is no help for generating getters and setters which you intend to reference in source code.

Another approach is to use byte-code manipulation on the class files. For example, Kohsuke Kawaguchi of Hudson fame created a library called Bridge Method Injector, that helps perserve binary compatibility when changing a method's return type in a way that is source compatible but not binary compatible. Kohsuke implements this by using ASM to modify the byte-code in a class file after compilation. This trick could be used to mimic the behavior of the Getter/Setter/ToString/EqualsHashCode annotations of Lombok with one caveat: generated methods would only be visible to classes external to your library but not to classes within your library. In other words, projects that depended on classes in your library as a jar would see your getters and setters, but classes within your library would not see these getters and setters at compile time.

The trick that makes Lombok special is that the code it generates is weaved in before Analyze and Generate phase of compilation. This allows classes within the same compilation unit to have visibility to the generated methods. It appears another library called Juast may be using a similar trick (modifying the AST) to do things like operator overloading. For some developers, the immediate benefits of Lombok's approach may outweigh the potential risks.


Alternatives outside Java

If you're willing to switch to Scala, Lombok becomes a moot point. Scala has Case classes that eliminate the getter/setter/toString/hashCode/equals boiler-plate. Scala also has Automatic Resource Management that covers Lombok's @Cleanup behavior.

Another option is Groovy if you don't care about static typing. Groovy has similar support for Scala-like Case classes. Groovy also officially supports compile-time, AST transformations.

Final thoughts

Project Lombok can do tricks that are impossible via other dynamic code generation methods in Java but you should be aware the it uses some back-door APIs to accomplish it.

53 comments:

  1. Great article. Just one question: can you paste example of Groovy Scala-like case classes?

    ReplyDelete
    Replies
    1. Thanks for the post, I am techno savvy. I believe you hit the nail right on the head. I am highly impressed with your blog. It is very nicely explained. Your article adds best knowledge to our Java EE Training in Chennai. or learn thru Java EE Training in Chennai Students.

      Delete
  2. Thank you for great explanation. I've used your post as a source reference in my internal Lombok workshop.

    ReplyDelete
  3. lombok could be used as source code preprocessor by delombok (this method is actually used for maven builds).
    so closing compilers api could make developing in IDE's with lombok less pleasant but bearable (thanks to manual code generation ability)

    ReplyDelete
  4. For the record, Groovy also supports static type checking and static compilation, and they also work great with our AST transformations and extension methods, etc.

    ReplyDelete
  5. You really made my day to provide such a clean explanation about "lombok". That article should be on the top of google search. I wasted many hours to get a explanation like this.
    Thanks.

    ReplyDelete
  6. Interesting and (still) relevant article. It'd be great if few more details can be made up-to-date.

    ReplyDelete
  7. Nowadays we have both http://immutables.github.io/ and https://github.com/google/auto/tree/master/value and both works on source code so the need for lombok trick is not justified anymore since we have better alternatives.

    ReplyDelete
  8. Useful information.I am actual blessed to read this article.thanks for giving us this advantageous information.I acknowledge this post.and I would like bookmark this post.Thanks
    Useful information.I am actual blessed to read this article.thanks for giving us this advantageous information.I acknowledge this post.and I would like bookmark this post.Thanks

    ReplyDelete
  9. Thank you so much for a well written, easy to understand article on this. It can get really confusing when trying to explain it – but you did a great job. Thank you!
    Data Science training in Chennai
    Data science training in Bangalore
    Data science training in pune
    Data science online training
    Data Science Interview questions and answers
    Data Science Tutorial

    ReplyDelete
  10. Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.
    Best Devops online Training
    Online DevOps Certification Course - Gangboard

    ReplyDelete
  11. It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command
    Data Science training in Chennai
    Data science training in Bangalore
    Data science training in pune
    Data science online training
    Data Science Interview questions and answers
    Data Science Tutorial
    Data science training in bangalore

    ReplyDelete
  12. Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
    Thanks & Regards,
    VRIT Professionals,
    No.1 Leading Web Designing Training Institute In Chennai.

    And also those who are looking for
    Web Designing Training Institute in Chennai
    SEO Training Institute in Chennai
    Photoshop Training Institute in Chennai
    PHP & Mysql Training Institute in Chennai
    Android Training Institute in Chennai

    ReplyDelete
  13. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Java training in Chennai

    Java training in Bangalore

    ReplyDelete
  14. Amazing info about Java blog to know more about java visit below

    java training in chennai

    ReplyDelete
  15. Nice post!Everything about the future(học toán cho trẻ mẫu giáo) is uncertain, but one thing is certain: God has set tomorrow for all of us(toán mẫu giáo 5 tuổi). We must now trust him and in this regard, you must be(cách dạy bé học số) very patient.

    ReplyDelete
  16. Time is free but it's priceless(khóa học toán tư duy) . You cannot own it, but you can use it(cách dạy bé học số) . You can use it, but you can't keep it(toán tư duy logic là gì). Once you lose it, you will not be able to get it back.

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete

  18. Thanks for sharing valuable information. Your blogs were helpful to Azure learners. I request to update the blog through step-by-step. Also, find the Azure news at
    Such an ideal piece of blog. It’s quite interesting to read content like this. I appreciate your blog Data Science training in Bangalore

    ReplyDelete
  19. Nice post. I learned some new information. Thanks for sharing.

    karnatakapucresult
    Guest posting sites

    ReplyDelete
  20. Well with your permission let me grab your RSS feed to keep updated with forthcoming post. Thanks a million, and please keep up the gratifying work.
    nebosh course in chennai
    offshore safety course in chennai

    ReplyDelete
  21. I and my friends were going through the nice, helpful tips from the blog then the sudden came up with an awful suspicion I never expressed respect to the website owner for those secrets.
    safety course in chennai
    nebosh course in chennai

    ReplyDelete
  22. Appericated the efforts you put in the content of Data Science .The Content provided by you for Data Science is up to date and its explained in very detailed for Data Science like even beginers can able to catch.Requesting you to please keep updating the content on regular basis so the peoples who follwing this content for Data Science can easily gets the updated data.
    Thanks and regards,
    Data Science training in Chennai
    Data Science course in chennai with placement
    Data Science certification in chennai
    Data Science course in Omr

    ReplyDelete
  23. This is really an amazing article. Your article is really good and your article has always good thank you for information

    โปรโมชั่นGclub ของทางทีมงานตอนนี้แจกฟรีโบนัส 50%
    เพียงแค่คุณสมัคร Gclub กับทางทีมงานของเราเพียงเท่านั้น
    ร่วมมาเป็นส่วนหนึ่งกับเว็บไซต์คาสิโนออนไลน์ของเราได้เลยค่ะ
    สมัครสมาชิกที่นี่ >>> Gclub online

    ReplyDelete
  24. Hey there ! i come here for the fist time ! and i impressed with your writing and your blog

    เว็บไซต์คาสิโนออนไลน์ที่ได้คุณภาพอับดับ 1 ของประเทศ
    เป็นเว็บไซต์การพนันออนไลน์ที่มีคนมา สมัคร Gclub Royal1688
    และยังมีเกมส์สล็อตออนไลน์ 1688 slot อีกมากมายให้คุณได้ลอง
    สมัครสมาชิกที่นี่ >>> Gclub Royal1688

    ReplyDelete
  25. One of the best content i have found on internet for Data Science training in Chennai .Every point for Data Science training in Chennai is explained in so detail,So its very easy to catch the content for Data Science training in Chennai .keep sharing more contents for Trending Technologies and also updating this content for Data Science and keep helping others.
    Cheers !
    Thanks and regards ,
    DevOps course in Velachery
    DevOps course in chennai
    Best DevOps course in chennai
    Top DevOps institute in chennai

    ReplyDelete
  26. Do you think you're encountering issues in running of QuickBooks Premier? We urge you to definitely not ever suffer from losses due to longer QuickBooks 247 Customer Support
    USA
    downtime of one's respective QB Premier. Simply make contact with us at our website . and from now on we'll pro-actively resolve every one of the errors and issues faced by you in your kind of QuickBooks Premier.

    ReplyDelete
  27. QuickBooks Pro Support contact number is ready beforehand to give you customer-friendly assistance in the event that you talk to a problem using
    QuickBooks Support Number Pro. All of us is skilled, talented, knowledgeable and spontaneous. Without taking most of your time, our team gets you rid of all of the unavoidable errors with this software.

    ReplyDelete
  28. Prior to Payroll integration, if customers require any expert suggestion or recommendation, they could connect to our QuickBooks Payroll expert on our toll-free helpline. QuickBooks Payroll Support Number

    ReplyDelete
  29. Run the payroll software and enter the employee details on the foundation of hours or days. You can enter up to 50 employees and also the software automatically calculates the wages per employee therefore the tax deduction QuickBooks Payroll Support Number It does the tax estimation and provides the outcome almost instantly.

    ReplyDelete
  30. QuickBooks Support Phone Number, a credit card applicatoin solution which will be developed this kind of a means that you can manage payroll, inventory, sales and each other need of small enterprises.

    ReplyDelete
  31. You can expect the greatest technical assistance. The dedication of your QuickBooks Support team makes us unique. Regardless of whenever or wherever you desire us, our experts will always be there. We have been prepared to help you over the phone, although the live chat, via e-mail, or online with your QuickBooks Customer Support Number team. Currently have a review of the services we provide.

    ReplyDelete
  32. Payroll management is truly an important part these days. QuickBooks Enterprise Support Help Phone Number USA has many employees. Employers need to manage their pay.

    ReplyDelete
  33. If you're trying to download QuickBooks Payroll update, the next error message error is encountered “QuickBooks Desktop Error 15270: Payroll update did not complete successfully. The update is missing a file.”
    VISIT : https://www.247supportphonenumber.com/how-to-fix-quickbooks-error-15270/

    ReplyDelete
  34. This system is wonderful for a medium-sized company. You may get the most wonderful financial tool. QuickBooks Desktop Payroll Support Phone Number is present 24/7. You'll be able to call them anytime. The experts are thrilled to aid.

    ReplyDelete
  35. QuickBooks error code 15223 is as normal as almost every other error code that develops in this software. However it is extremely important to eliminate it to ensure no hindrance is caused although the user is preparing to do his job in this software.
    visit : [url=https://www.247supportphonenumber.com/quickbooks-error-15223/]https://www.247supportphonenumber.com/quickbooks-error-15223/[/url]

    ReplyDelete