Tangram Release 0.9

Over the last year, Tangram has changed very much and does not look the same in all of its modules and options.

Persistence Options

The Tangram Dynamic Web Application Framework has been extended to support the Java Persistence API (JPA) and EBean as persistence layers in addition to the already available Java Data Objects solution. This greatly extends the number of options for platforms to use with Tangram.
The Tangram examples reflect this with the JDO/DataNucleus, JDO/Google App Engine, EBean, and JPA example where the later is switchable with tested API implementations for OpenJPA, EclipseLink, DataNucleus, and Hibernate.
The simple and more or less generic JDO editor now is a generic editor for all the storage solutions and built as a separate module. Except for Google App Engine this module can be left out if there is another solution available to get the content into the repository.

Dynamic Model Extenions

In addition to the Groovy implemented classes stored in the repository, which are used to extend the view and controller part of the application, you can now create JDO annotated classes which are immetiately usable as model classes in the repository. This option can only be presented for the JDO implementation.
To make the building of Tangram itself, any Tangram application, or even any application using byte-code transformation for JDO, JPA, and Ebean easier, a gradle plugin has been introduced. The examples and over blog entries illustrate the usage of this plugin.

IDE Integration

To be able to synchronize codes stored in the repository a separate FTP module has be introduced. Most IDE due to their strong PHP past support FTP better than any other protocol. CSS, JavaScript, Groovy codes can be imported and exported.

Generic Import and Export

Additionally and apart from the FTP module Tangram now contains a generic importer and exporter for the whole content stored in the repository. The XML representation can be used to transfer content between application using JPA, JDO, or EBean. IDs cannot be preserved.

Framework Independency

For the whole Tangram code the Springframework now is only an option to do the setup of the application by Spring's Dependency Injection IoC-Container and for the controller and view parts of the application.
Along this way it was easier to present a custom DI component, Dinistiq, than to use one of the available solutions. Tests have been done with Google Guice, TinyDI, and JSR-330. As of Tangram 0.9 none of these are more than experiments. Work on a Tangram JavaEE integration is also in an experimental stage and would require a major refactoring of some parts of Tangram, see other entries in this blog on the JavaEE topic.
It should now be fairly easy to do more experiments with other IoC/DI frameworks and containers or to integrate other web frameworks starting from the new plain servlet based solution.

Options

Tangram 0.9 has been tested, verified, or is even in production use on the following platforms: Google App Engine, run@CloudBees, OpenShift, and Standalone with Apache Tomcat using RDBMS or MongoDB.
The option matrix now looks like this:
Persistence: JDO, JPA, EBean
IoC: Springframework, Dinistiq
Security: Spring Security, Apache Shiro
Hosting: On Premise, OpenShift, Cloudbees, Google App Engine
Storage: RDBMS, No-SQL e.g. MongoDB, Files

Code Reduction

Tangram started as an idea to plug existing components together to form a web application with as little glue code as possible. Some of the ideas like object oriented templating though had to be coded explicitly. After this starting point the Tangram codes started growing and the 0.9 release got an additional design goal to stop this. Still some of the codes in Tangram duplicate functionality which would otherwise externally be available but avoids a depedency just to call one method. And there are still codes duplicates within Tangram where the different options e.g. for persistence might get a stronger common code base. In these areas it is still not clear if they will differentiate or grow together.

Outlook

The experiences with CapeDwarf showed the way into a Java EE module as an alternative to dinistiq, the Springframework, and the many tests with e.g. Google Guice. The not that encouraging experiences with the EBean ORM might lead to discontinued support for that module. Dinistiq will become more Java EE / CDI compatible and CDI without the full Java EE stack also seems to be an interesting option. Work on the generic import and export functions using XStream needs to be extended and easier integration of OpenID or OAuth outside of the Google App Engine is needed for many applications. Google App Engine with JPA is still on the list but not desperately needed.

150 Lines just for one Collection

Some 15 years ago I left the Java Enterprise Edition train and am now pushed by things like CapeDwarf on JEE Servers to the evaluation of running Tangram somehow in a JEE environment. I read the promise, that things have become a lot easier and that JEE has adopted the annotation and convention based style I now feel familiar.
I came across the JSR330 annotations from the javax.injection package which were introduced in the JEE world. JSR330 Annotations are recognized by several Dependency Injection frameworks like Guice, Springframework, my homegrown dinistiq, and obviously the JEE Containers.

Step 1: Get rid of Spring

This is were I started to make Tangram less depending on the Springframework directly or provide interfaces to be implemented in a spring and a non-spring way. For the dependency injection part it was mostly as easy as to migrate from

    @Autowired
    private ClassRepository classRepository;

to

    @Inject
    private ClassRepository classRepository;
I lost the required = false option at that time but was able to live with that given the advantages of portability.

Step 2: Find Alternatives

The spring parts of the application where re-implemented using a plain servlet based solution, and springsecurity was replaced by Apache Shiro for those scenarios.
After having migrated to the new annotations most of my Components where portable across containers. So I don't have to sell my soul to a special DI framework for very many cases, which helps choosing the right infrastructure for every project. But there are several details which still remain problematic, most notably the missing required = false option and the missing collection of instances to be able to automatically let the whole set of instances be injected.

Step 3: Annoying Details

The annoying collection problem looks like this:

@Autowired(required = false)
private Collection<ControllerHook> controllerHooks = new HashSet<ControllerHook>();

Later I migrated this to the JSR-330 Annotations - which also works with spring.

@Inject
private Collection<ControllerHook> controllerHooks = new HashSet<ControllerHook>();

I made this work with dinistiq as well.
With Guice I learned, that it didn't like to collect the instances for me and provide me with the collection out of the box.
I'm expecting here, that any instance implementing ControllerHook from the application context gets added to a collection which then is injected into the consuming component. Spring does it, dinistiq does it, Guice can be convinced to do it by the additional Multibinder:
 
Multibinder<ControllerHook> multibinder = 
                            Multibinder.newSetBinder(binder(), ControllerHook.class);
multibinder.addBinding().to(UniqueUrlHook.class);
multibinder.addBinding().to(ProtectionHook.class);

Step 4: Show Stopper

Inspired by the CapeDwarf project, promising to be able to deploy applications developed for the Google App Engines and their APIs on a JEE infrastructure, I tried to deploy GAE based Application to such an Infrastructure based on a JBoss AS7 or Wildfly 8 Server using CapeDwarf Versions 1 or 2 respectively.
This approach doesn't work for Tangram since the used DI solution within the application - be it dinistiq or the Springframework - interferes with the JEE Container's CDI implementation. The Application Server starts to interpret the annotations intended for the other DI framework. This would instantiate the application components twice and right at the moment it also fails on some of those components.

Step 5: JEE Wordiness

So quite naturally I now went over to directly migrate Tangram to the JEE world. But also JEE has the same problem as plain Guice has.

@Inject
private Collection<ControllerHook> controllerHooks = new HashSet<ControllerHook>();
This time the solution really is ugly. It seems I have to implement a different consuming component and thus instead of the two lines, a separate ControllerHookProvider is needed

public interface ControllerHookProvider {

    Collection<ControllerHook> getControllerHooks();

} // ControllerHookProvider
Of course the generic implementation looks like the good old two lines which did the whole job for me so far.
 
public class GenericControllerHookProvider implements ControllerHookProvider {
    @Inject
    private Collection<ControllerHook> controllerHooks;


    public Collection<ControllerHook> getControllerHooks() {
        return controllerHooks;
    } // getControllerHooks()

} // GenericControllerHookProvider
For JEE an alternative implementation has to be chosen, which made the interface necessary in the first place, and it is surprisingly wordy for that common and simple scenario.

@Named("controllerHooksProvider")
@Singleton
public class JeeControllerHooksProvider implements ControllerHookProvider {

    private Collection<ControllerHook> controllerHooks = new HashSet<>();

    @Inject
    public void setControllerHooks(@Any Instance<ControllerHook> hooks) {
        for (ControllerHook hook : hooks) {
            controllerHooks.add(hook);
        } // for
    } // setControllerHooks()

    public Collection<ControllerHook> getControllerHooks() {
        return controllerHooks;
    } // getControllerHooks()

} // JeeControllerHooksProvider
This really didn't invite me to go deep into JEE again. Additionally it is missing any of the configuration file based bean definitions avoiding much of the interfacing of classes where in fact just two injected values make the difference. With the Springframework configuration files and auto scanning of beans go hand in hand. JEE intentionally left out this part.

Byte-Code Transformation is no big deal

To avoid another language in my web projects I'm using Java as the design language for the Objects to be persisted as well. It seemed easy to use with the different ORM Standards and implementations available.

Take me from Java to the Database

Many ORM implementors tend to recommend using a byte-code transformation process to make the classes usable in their respective persisting contextes (e.g. http://www.avaje.org/doc/ebean-userguide.pdf - Chapter 15). This in fact means, after you did your job of coding and compiling the classes, some other component takes this code and transforms it into some other code additionally dealing with the ORM/Database related stuff.
The idea is, to avoid runtime penalties or the generation of subclasses dealing with the additional database related issues which would show up at runtime potentially screwing up your idea of the class hierarchy. (Which it did for me. See below.)

Why class weaving or enhancing is a big deal

Of course this still means that you are running code, you don't now in detail.
The assumption of any of the ORM framework authors is, that the byte-code transformation process can be easily automated and as far as possible be hidden from the application developer. JPA based JEE applications are expected to do the transformation at deployment time to the container - so this doesn't even happen within your development tool-set.
My experience is a different story. And the hiding of things during development once again was no good idea for me. 

The easy Start

I started with the Eclipse IDE some years ago and the Google App Engine Plugin. It does the byte-code transformation for the JDO implementation from DataNucleus automatically at compile time. This worked fine as long as I was coding in "play-around" mode. When the code started to grow into modules, from time to time the classes were propagated to the client module unenhanced. This is were I learned what the use of DataNucleus feels like, when in fact just the transformation is missing (of course it doesn't tell "you missed to transform classe this-and-that"). I got around these issues with the dumb "clean nearly everything in your work environment" pattern.

Build System Integration

Things got even more complicated when I started to write build scripts, since the project grew and was supposed to be published. You don't want to give anyone a 20 page description on how to setup the IDE just as you did. You simply give friends a script which describes the necessary parts in human and machine readable form. So the project gets cloned from the source and a simple build tool call - hopefully in default mode with no to few options -  will create a usable result.
But the promise to support me as a developer from the ORM provider still holds true for these situations. I was just expected to change the way I was using the transformation tool. DataNucleus comes with an ant task, a compiler plugin and so on. Since I didn't want to use the obsolete legacy tools Maven or ANT (hey, why not use make or punch cards?) I "simply" plugged in the compiler plugin since there is no direct support for Gradle and the integration of the ANT task was not that easy at the initial try.
First of all this compiler plugin was not able deal with all of the versions of the Oracle Java Compiler and all language levels beyond "Java 6" so I had to prepare the source code carefully.
This gave me enhanced classes and sure the enhancer was running, but...
When packaging the classes to JARs as build systems tend to do after compilation nearly automatically, those classes where unusable again with the error messages I was already familiar with.

Unit-Testing the Byte-Code Transformation instead of my Code

At this point in time I started writing JUnit tests to test, if my build environment was working and not to test if my code was correct. This gave me the impression that some things are going wrong.
I learned, that the compiler plugin took some time after compilation before it started to "enhance" (byte-code transform) the class files. It used some sort of threading for this so that Gradle already had packaged the jar files, before the process was completed. I started to add some 10s waiting to my build scripts. Argh...

Refactoring - Get the same Thing you already had

I took a second look at the DataNucleus Enhancer's ANT task to integrate this into the build process as a Gradle task without those eratic 10s of waiting. I also needed this step since I was updating DataNucleus from the old version used in the Google App Engine at that time to a newer one also meant for stand-alone use.

Use other APIs as well like they were simple Libraries

After all these pieces were working, I started playing around with the Java Persistence API JPA. Also the implementations of JPA I came accross - OpenJPA, EclipseLink, and again DataNucleus - recommended the use of byte-code transformations called Enhancement (OpenJPA and DataNucleus) or Weaving (EclipseLink).
The integration of that many APIs and byte-code transformers made things more complicated again, while the code I wrote still is not that complicated. It's just the byte-code transformation which adds to the complexity. I needed to present OpenJPA, EclipseLink, and DataNuceus Versions of my single JAR archive with only very few classes and only two of them needed to be byte-code transformed. Additionally with JPA I have the option to use the original classes without byte-code transformation in some scenarios with certain limitations (Only DataNucleus is really capable of a automatic discovery of available classes for database access, the others need detailed lists passed over to the implementation in different ways. This is anything but portable!)

Stop pretending it is easy and write a decent Tool to do the Job

Since not all of the implementations can be on the compile time classpath of the JPA relying portions of my project, it was now time - just because of the necessary byte-code transformations - to write a Gradle Plugin dealing with this.
Very easily this plugin was generic enough to be used in any project using JPA, JDO, or Ebean as the ORM Solution for Java and the Gradle build tool.
Two third of the work on the build-scripts of the Tangram dynamic webapplication framework were related to the byte-code transformations over the last five years.

Conclusion after some Years

So my best friend now is OpenJPA which can relatively easy be used without transformation. Yes, it presented me the nice subclassing issue where I am at runtime dealing with subclasses of the classes I designed myself, but this was solvable with half a dozen lines of code.
My second best friend is DataNucleus where I am now able to integrate the byte-code transformer into the runtime environment of my framework and write JDO annotated classes in groovy, put the code into the JDO based database layer itself and thus be able to extend the object oriented storage at runtime. This is what adds very nicely to Stylesheets, JavaScript Codes, URL-Formats and Business Logic in Groovy in the Database layer resembling the dynamic part of Tangram. I tried this with the OpenJPA Enhancer and EclipseLink Weaver as well but with no success.
Also I now got a code base which was easily extended with another ORM Solution called EBean.  It was meant as an option with a smaller footprint but does not present any advantaged over the other options already implemented and proven in real projects live on the web using the Tangram dynamic web application framework.
So, anyone still thinks that byte-code transformation is a non-issue as you may read on introductory web pages on ORM? Give me some 30s to make your build process break - at least every once in a while when you don't expect it and won't easily discover the source of your pain.
But in the end with my Gradle based plugin, things are definitely a lot easier and reliable - again after a lot of work with things that were supposed to be easy, automatic, or hidden from me.