Tangram Release 1.1

The 1.1 release is a consolidation release removing outdated stuff leaving
room for latest versions of the used libraries and modules and providing some
polishing in the existing feature set.

Beyond the App Engine

The sources and release notes can be found on github.com, the binaries again reside on JCenter. This release includes some notable overdue dependency updates like

• Servlet 3.1 API now included
• JSP 2.1 now included
• DataNucleus updated to 4.1
• dinistiq updated to 0.5

To allow this, it leaves out the Google App Engine support.

Gretty Plugin

Since not only the Google App Engine has a defect with old releases of Java
Web APIs but also Gradle has a obsolete Jetty plugin with an old version of
Jetty, all examples have been migrated to Gretty which is now the preferred container integration and even application packaging option.

Morphia Support

Latest Tangram applications always used Mongo DB as the storage backend with either JPA or JDO as the API. If you don't need the API abstraction or want an even leaner storage integration (before that DataNucleus JDO and EBean had the smallest footprint) you now can start using the "Mongo DB only Mapping" Morphia.

Markdown Support

Up to now every property using char[] for getters and setters was handled as structured text. It was edited as an HTML fragment (p or div) using the CKEditor.
Starting with this release, every property using org.tangram.content.Markdown for the getters and setters is handled through the CodeMirror editor as Markdown syntax. It is transformed to HTML in rendering.
Also templates can now be in Markdown syntax whenever appropriate while also here the output is transformed to HTML before passing the templates to the other layers of Tangram.

Taking Care of the Modification Date and Time

If you need to track the modification time of an object through the Tangram editor module or the ftp service, you can now do so and the standard storage backed classes of Tangram now do so.

File Restart Cache

The restart caches used for faster restarting of the application with data which should not be changing between sessions of the application now only have to be cleared manually on error conditions but not on changes of the application any more.

Exporting and Importing

The export and import features of Tangram are now strictly symmetrical: Full content export and import  - more or less independent of the storage flavour in use - and a code exporter and importer for the sources stored in the repository.

Fat JARs

Additionally it is now possible to leave out the "war" dependencies since
resources like templates can be placed in the respective JARs.

Extended testing

Tangram now for the first time has a decent test coverage. While there still
is room for more in depth testing of many functions, we now can put more trust
into the single snapshot builds. This was quite important since most Tangram
applications not residing on the Google App Engine already migrated to the 1.1
level.

Tangram Release 1.0

Since Tangram reached more than the set of features originally intended back in 2009, it was time to call the current state a 1.0 release.

Why?

The name Tangram points out that the main objective was to create web applications from a fixed set of modules and options which can form nearly any shape you would need. You can start quickly with a limited set of functionality and let the application grow. Fixed set in this case only means that you will need e.g. a model implementation but still have a choice, which one you would want to use. Also the glueing together with a Dependency Injections solution provides several options.
Another intention was to provide reasonable defaults for nearly every aspect where this is possible. You don't need to copy tons of files which do things technically necessary, which you didn't think of at that time. You only set up the things you know that you need them, and all the other features stay quiet. At least they should do something reasonable without getting in your way.

Users

Tangram is in production use since 2011 by Provocon, Ponton, Naturinspiriert.org and others.

Dynamic Extendibility

For many extensions you will not even need a deployment but a change in the data repository forming small and stable cores where the application as whole can be modified to changing requirements quickly, easily and securely.
Apart from the obvious presentation stuff this includes Groovy codes to create and parse URL, extend the business logic, and even the model layer with new model classes.

Core Features

We provide object oriented templating for objects from JPA, JDO, EBean, and CoreMedia repositories. The core system emphasises dynamic web programming through CSS, JavaScript, Velocity Templates, and Groovy Codes in the repository together with basic CDN support, code minification, and caching support.

URLs can be formatted in nice, SEO friendly schemes and be easily mapped to actions to provide the result views for users.
The implementation of authentication and authorization now uses the pac4j set of libraries to provide a seamless and easy integration of plain user lists in files or the repository, OAuth providers, OpenID providers, Google App Engine user service and others for the web application and the base system itself.
This authentication solution is used grant access to the system itself but also to provide support for protected content areas in your applications. You only have to focus on the question, which content needs to be protected by a access granting scheme. Logins can be a generic login page or integrated in you application design.

Generic Editor

Except for CoreMedia we provide a generic web editor which is now responsive (there is now separate mobile editor anymore) which can be loaded from the respective cloud locations of the used components (CKEditor, CodeMirror). Also it can be extended with our dynamic web programming facilities.

Glueing Stuff Together

The mentioned default set of configurations and the option to customize this to your needs is achieved by Dependency Injection libraries. Tangram supports the usage of the Spring Framework, Google Guice, and dinistiq. It should be possible to add other solutions as well provided that they offer a decent set of functionality (which is not always the case for smaller DI libs.) including e.g. optional values, overriding of configurations, and deep generics introspection.

Readware

To illustrate the usage and provide a nice starting point, a set of example applications is provided in sync with the releases.
A documentation wiki is now starting.

Where the 1.0 Release Ends

One real limitation is the older set of JSP/Servlet APIs which need to stay in place to support the Google App Engine. So this is the final release to support the App Engine to be able to move ahead to newer APIs for several areas. This will not be achieved in the 1.0 branch.

Dinistiq Version 0.5

The minimalistic Dependency Injection library for Java - Dinistiq - has been published in version 0.5.
This version still doesn't change the main outline of the project: It deals with a single - singleton - scope with a few extensions in a very small footprint library. The library does not introduce any custom tags but fully relies on a subset of the JSR330 definitions.
Despite its limited scope it still successfully executes the TCK and comes with a reasonable unit test coverage. These unit tests have been migrated to TestNG to be in sync with my other projects reflecting the license which better fits into the set of licenses of the used dependencies in this project.

Fixes

The once more extended test coverage after the migration to TestNG showed two bugs relating to class discovery. These have been fixed in this release.

Leaner Web Integration

Due to the use of older versions of the Servlet API, dinistiq had to use a central dispatcher servlet dispatching to other so called registerable servlets. This level of indirection has been complete removed. With dinistiq 0.5 the web integration is mostly rewritten now depending on the Servlet API version 3.1.
Sadly this also removes its URL pattern matching capabilities.
RegisterableServlet instances from the dinistiq scope are now registered with the Servlet container directly. They now present a number of Servlet specification compatible URL patterns (no regular expressions any more - sorry) and and the same ordering indicator as with previous releases.

Availability

Starting from release 0.4 dinistiq is available from the JCenter repository at

https://jcenter.bintray.com/dinistiq/

Latest snapshots are provided at

https://oss.jfrog.org/oss-snapshot-local/ 

There are already some 0.6-SNAPSHOTs available, but they don't present changes worth mentioning right now.

Migration

Take a look at your web.xml. If you see something like

  <servlet>
    <servlet-name>dinistiq</servlet-name>
    <servlet-class>dinistiq.web.DinistiqServlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>dinistiq</servlet-name>
    <url-pattern>/s/*</url-pattern>
  </servlet-mapping>

simply remove it!
The RegisterableServlet interface has been change as mentioned. You will have to update you regular expressions to Servlet specification compatible URL patterns. Of your the respective method signature was changed from

    @Override
    public Set<String> getUriRegex() {
        Set<String> result = new HashSet<>();
        result.add("");
        result.add("/.*");
        return result;
    } // getUriRegex()
to

    @Override
    public Set<String> getUrlPatterns() {
        Set<String> result = new HashSet<>();
        result.add(source.getDispatcherPath()+"/");
        result.add(source.getDispatcherPath()+"/*");
        return result;
    } // getUrlPatterns()
Mind the missing regular expression syntax and that the full URI path has to be given.

Dinistiq Version 0.4

The minimalistic Dependency Injection library for Java Dinistiq has been published in version 0.4.
This version doesn't present anything specifically new. It still deals with a single - singleton - scope with a few extensions in a very small footprint library. The library does not introduce any custom tags but fully relies on an subset of the JSR330 definitions.
Despite its limited scope it still successfully executes the TCK and comes with a once more extended unit test coverage.

Availability Release

One of the main changes of this release is the availability of the release artifacts through the JCenter service of bintray. This renders the integration of Dinistiq even easier since you will most likely not need to add another repository or mirror to your infrastructure.
Dinistiq is in daily production use for some time now and will remain supported as long as there is not always a need for more feature complete solutions like Google Guice or the Springframework.

Next Steps

The versioneye service shows that Dinistiq is up to date with its dependencies with only one intentional exception. This release will be the last to support the Google App Engine through the use of outdated Servlet API versions.

Dinistiq Version 0.3

Dinistiq ist entstanden, weil es schneller zu implemtieren war, als die Integration anderer DI-Framework und Container in Tangram dauerte.
Da es dann eine Weile einfach alles tat, was notwendig war, wurde nur geringe Code-Optimierungen eingeführt.

Qualitätssicherung

Zu seinem ersten Geburtstag hat Dinistiq aber ein echte Upgrade verdient.
Seit August befindet es sich im produktiven Einsatz in einem Testframework zur Validierung der Einhaltung der Anforderungen für ein großes deutsches Medienportal, das auf einer neuen technischen Basis komplett reimplementiert wird.
Dabei sollte ein einfacher, effizienter, wartbarer und schlanker Technologiestack zu Einsatz kommen.
Die alternativen Stapelten Selenium, PHP und Shellskripte und waren nicht in der Lage, die angeforderten Varianten an Plattformen und Browsern mit Testfällen zusammenzuführen. Außerdem paßten sie natürlich nicht in die reine Java-Welt des Restes des Projektes.
Gleichzeitig sollten veraltete Techniken wie Maven natürlich ebenso außen vor bleiben.
Nun muß sich Dinistiq jeden Tag darum kümmern, hunderte von Testfällen mit jeweils unterschiedlichen Browsern und Zielplattformen durch Dependency Injection zusammenzuführen.

Qualitätssicherung

Dinistiq befindet sich nun täglich im Projekteinsatz und es ist nicht mehr hinnehmbar, wenn dabei genutzte Funktionen nicht zuverlässig nutzbar sind. Da Dinistiq mit dem Einsatz reift, sind dennoch häufige Snapshots nötig gewesen, sodaß ist einer verstärkten UnitTest Abdeckung und dem Einsatz von PMD zur Source-Code Kontrolle zwar nur ein Fehler befunden wurde, das Neu-Einführen von Fehlern aber von vorneherein úmgangen wurde.
Zur Generierung der Übersicht wird Jacoco eingesetzt.
Der Impuls war aber hier auch, daß der Einsatz in der Qualitätssicherung für ein ungleich größeres Projekt erfolgen sollte und so mußte sich Dinistiq als Testfeld für die angeforderten Qualitätsmaße an den Source-Code zur Verfügung stellen.
Bisher war dieser Schritt ein voller Erfolg und in jedem Falle viel einfacher umzusetzen als alle mir bekannten Alternativen wie Springframework, Google Guice, Weld, OpenWebBeans, die den Job sicherlich auch erledigen können.
Dafür gibt es heute den Stempel "0.3".

Details

Mit Version 0.3 ist auch das Standard Repository für die Artefakte gewandert und der CI Server ebenfalls. Dies ist nur der Tatsache geschuldet, daß Cloudbees sich aus dem Geschäft zum Jahresende zurückzieht.

Dinistiq 0.2 Release

The minimalistic Dependency Injection solution for the setup of component based Java applications using Singletons and the JSR330 annotations has stabilized to a 0.2 release.
With some minor bug fixes it also better deals with misconfigured setups.
All system properties from the JVM are now part of the scope. The logging framework has been changed from Apache Commons Logging to Simple Logging Facade for Java (slf4j).
Decent handling of maps and booleans has been added and circular dependencies are now detected and reported.
Along all these minor changes the documentation has been extended and a javadoc archive is added to the deliverables.

150 Lines just for one Collection

Some 15 years ago I left the Java Enterprise Edition train and am now pushed by things like CapeDwarf on JEE Servers to the evaluation of running Tangram somehow in a JEE environment. I read the promise, that things have become a lot easier and that JEE has adopted the annotation and convention based style I now feel familiar.
I came across the JSR330 annotations from the javax.injection package which were introduced in the JEE world. JSR330 Annotations are recognized by several Dependency Injection frameworks like Guice, Springframework, my homegrown dinistiq, and obviously the JEE Containers.

Step 1: Get rid of Spring

This is were I started to make Tangram less depending on the Springframework directly or provide interfaces to be implemented in a spring and a non-spring way. For the dependency injection part it was mostly as easy as to migrate from

    @Autowired
    private ClassRepository classRepository;

to

    @Inject
    private ClassRepository classRepository;
I lost the required = false option at that time but was able to live with that given the advantages of portability.

Step 2: Find Alternatives

The spring parts of the application where re-implemented using a plain servlet based solution, and springsecurity was replaced by Apache Shiro for those scenarios.
After having migrated to the new annotations most of my Components where portable across containers. So I don't have to sell my soul to a special DI framework for very many cases, which helps choosing the right infrastructure for every project. But there are several details which still remain problematic, most notably the missing required = false option and the missing collection of instances to be able to automatically let the whole set of instances be injected.

Step 3: Annoying Details

The annoying collection problem looks like this:

@Autowired(required = false)
private Collection<ControllerHook> controllerHooks = new HashSet<ControllerHook>();

Later I migrated this to the JSR-330 Annotations - which also works with spring.

@Inject
private Collection<ControllerHook> controllerHooks = new HashSet<ControllerHook>();

I made this work with dinistiq as well.
With Guice I learned, that it didn't like to collect the instances for me and provide me with the collection out of the box.
I'm expecting here, that any instance implementing ControllerHook from the application context gets added to a collection which then is injected into the consuming component. Spring does it, dinistiq does it, Guice can be convinced to do it by the additional Multibinder:
 
Multibinder<ControllerHook> multibinder = 
                            Multibinder.newSetBinder(binder(), ControllerHook.class);
multibinder.addBinding().to(UniqueUrlHook.class);
multibinder.addBinding().to(ProtectionHook.class);

Step 4: Show Stopper

Inspired by the CapeDwarf project, promising to be able to deploy applications developed for the Google App Engines and their APIs on a JEE infrastructure, I tried to deploy GAE based Application to such an Infrastructure based on a JBoss AS7 or Wildfly 8 Server using CapeDwarf Versions 1 or 2 respectively.
This approach doesn't work for Tangram since the used DI solution within the application - be it dinistiq or the Springframework - interferes with the JEE Container's CDI implementation. The Application Server starts to interpret the annotations intended for the other DI framework. This would instantiate the application components twice and right at the moment it also fails on some of those components.

Step 5: JEE Wordiness

So quite naturally I now went over to directly migrate Tangram to the JEE world. But also JEE has the same problem as plain Guice has.

@Inject
private Collection<ControllerHook> controllerHooks = new HashSet<ControllerHook>();
This time the solution really is ugly. It seems I have to implement a different consuming component and thus instead of the two lines, a separate ControllerHookProvider is needed

public interface ControllerHookProvider {

    Collection<ControllerHook> getControllerHooks();

} // ControllerHookProvider
Of course the generic implementation looks like the good old two lines which did the whole job for me so far.
 
public class GenericControllerHookProvider implements ControllerHookProvider {
    @Inject
    private Collection<ControllerHook> controllerHooks;


    public Collection<ControllerHook> getControllerHooks() {
        return controllerHooks;
    } // getControllerHooks()

} // GenericControllerHookProvider
For JEE an alternative implementation has to be chosen, which made the interface necessary in the first place, and it is surprisingly wordy for that common and simple scenario.

@Named("controllerHooksProvider")
@Singleton
public class JeeControllerHooksProvider implements ControllerHookProvider {

    private Collection<ControllerHook> controllerHooks = new HashSet<>();

    @Inject
    public void setControllerHooks(@Any Instance<ControllerHook> hooks) {
        for (ControllerHook hook : hooks) {
            controllerHooks.add(hook);
        } // for
    } // setControllerHooks()

    public Collection<ControllerHook> getControllerHooks() {
        return controllerHooks;
    } // getControllerHooks()

} // JeeControllerHooksProvider
This really didn't invite me to go deep into JEE again. Additionally it is missing any of the configuration file based bean definitions avoiding much of the interfacing of classes where in fact just two injected values make the difference. With the Springframework configuration files and auto scanning of beans go hand in hand. JEE intentionally left out this part.

Fast on the first Request

It seems to be common sense with so many developers, that "expensive" things that only happen once are not that bad. And when those things happen at the wrong point in time that it is a good idea to move them to the "application startup".
So most of us are not very surprised and consider it no problem, if this startup takes some time.

Welcome to the Cloud

At a first sight, deploying my applications in the cloud is exactly the same as it used to be. I follow accepted standards, use common frameworks and respect many best practices. So any runtime platform supporting this should do the job. Cloud in that case means, that many more aspects of deployment and operations get automated - including that starting and stopping of additional instances, load balancing and so on.
But wait... When does the platform get the idea to start new instances?

Let the Customer wait?

It's just load. And load means, that Customers issued HTTP Requests and are awaiting responses in time. From research we know that in time means something around 3s. But when a user request leads to the start of a new instance to handle the load it's quite too late to do all the expensive stuff. Application startup is not the right point in time anymore.
Additionally you can learn, that all those nice precomputations simply re-calculate values the instances from yesterday or the other instance on the other machine already learned. And of course it still is a good idea to have all those values at hand - meaning to "cache" them. But very many of the values which are not coded or configured into the application deployment stay the same as long as the deployment environment doesn't change. Or they stay the same unless the application itself changes them and thus is able to tell when really to re-calculate derived values in the caches.

Examples for all this are:

• Database derived value caches like query results with calculations based on the result, where just this application writes to the database. Those values might be needed at startup but stay constant until the application itself changes the database contents.
• Classpath scanning to auto-discover software components. The developers and deployers wont want to collect the lists manually or at build time but the components don't change after the deployment. And definitely not on every startup.
• Webtemplates and other codes which need to be fetched and prepared for use (e.g. get compiled). Those codes get prepared on every startup of the application while they change not that frequently and especially not on application startup. Obviously they add a big amount to the the response time since those codes need to be executed for the generation of the response.

Just use a Cache

What? Oh, well not that Cache. This Cache. This cache doesn't need to be as fast as the in memory caches already available and I didn't want to re-invent the wheel. They just need to have some values at hand some other instance already calculated and they have to be changed when these values change - which is - as already pointed out - not as frequent as other runtime values.
The values in the cache still are volatile and can be re-calculated by the application at any time. But they should be persisted for some time to be available at startup and reduce the time for the first request in web applications.
The jsr107 cache implementation in the Google App Engine is an example of exactly this approach. I wrapped this cache as a PersistentRestartCache in Tangram and cut reponse times for the first request to a third (or half as the worst case scenario) from around 30s to 11s. For all other platforms available for the Tangram dynamic webapplication framework I presented a simple (maybe too simple) file based implementation which does most of the job as well.
Still this leaves out the classpath scanning of the Springframework or dinistiq. In my environment the Springframework uses between 3s and 7s and dinistiq around 4s to do the basic application setup based on classpath scanning and additional configuration files. So half of the time the startup deals with the framework code and not the applicaiton startup itself. And this value only was achieved by nearly brutal reduction of the portion of the classpath to be scanned creating a "components" package where all autoscanned components reside for the Springframework and dinistiq.

Resume

Caches are a good Idea. Applications with a dynamic deployment are a good idea. Thinking of the application startup as a point in time where long calculations might take place is not that much of a good idea (at least anymore) where instances should be automatically braught up depending on load and not human decisions.
So simply bring together caches with persistence and knowledge when those caches can be invalidated. As an example I did this for the tangram web application framework and had quite some success on the Google App Engine, run@cloudbees, and OpenShift cloud platforms for the Java world.
The last but also important point: Optimizing the application startup in general is worth the time nowadays.

Google App Annoyance - aka Engine

Together with the missing support for the Servlet 3.0 specification in Google App Engine (and, yes, there are still too many situations were this specification level ist not available) we are reading from Google for their App Engine that classpath scanning is an issue on that platform and that this is one of the reasons that kept them from supporting this version of the servlet specification.
What I didn't read from Google is, that they noticed that the Servlet 3.0 version is (arguably) one of the most important steps since Java came to the web. There is no major framework in the Java world anymore not doing any classpath scanning right now. The mentioned Springframework I'm using is just one example.
After some five years of work with the Google App Engine and Java as the only language in use, I changed my mind and don't consider the App Engine as one of the first choices in cloud platforms for the Java world anymore.
And this is really just because of these two small issues.
Like very many others I'm using the Springframework - with component scan and thus with classpath scanning. Any first request to an instance take ages. But "first requests" are common in the cloud, where instances need to be shut down and brought up depending on load. And the cloud is what Google App Engine is about, isn't it?
I invested quite some effort to learn how to be fast on the first request while still using the Springframework and even developed my own stripped down, minimalistic Dependency Injection environment for the application setup (dinistiq) to only have the features I'm using at hand. But this all didn't help to make the end user experience satisfying. Things feel slow.
So in the end this all gave the push for Tangram to support that many new platforms, use the CI Features and Repositories at cloudbees, enjoy the command line access of OpenShift. This brought options of different frameworks and I learned much about cloud deployment and operation scenarios. So in that respect we should be thankful for the Google App Engine weakness.
But it still is the source of some level of complexity and number of artifacts flying around in what I call my dynamic web application framework Tangram. Also this currently renders Google App Engine the second best cloud platform for Java while still having great web based monitoring tools.

Why use JSR330 Dependency Injection annotations

I rather apologise to introduce another Dependency Injection Container for the Java world - dinistiq - a very minimalistic approach to the topic. It turned out to be easier to implement another one, than to use others listed here. Limited in features, easy to use, and still more configurable than other options I could think of. After some months of use, I now can invite other users to take a look at it and try it in their own projects.

Also this text gives you a "why" on the use of the JSR 330 annotations for Dependency Injection. It simply makes your code even more reusable in case your development or deployment environment changes.

Since tangram is much more about glueing together proven existing software components and frameworks than writing code, I felt the need to check if the existing code base was really fully dependent on the Spring Framework.

Despite the fact that spring more or less in many ways does what I need, it sometimes feels a bit bloated and does too much magic I don't understand in detail (which I still had to learn when debugging things). So I tried to isolate the spring code during the tangram 0.9 work and present at least a second solution for all the things I did with spring so far.

For tangram spring does three things

• Dependency Injection to plug the whole application together
• support a decent view layer with JSP and Apache Velocity views
• A concise way to map http requests to code - controller classes or methods

So I took a look at other view frameworks like Vaadin, GWT, Apache Wicket, Play, Struts, JSF/JEE, Stripes. Right at the moment I think Vaading, GWT, Wicket, and Play are no really good fit for tangram, Struts in my eyes is a fading technology, and only JSF/JEE is an obvious option. With Java Server Faces I only had unsatisfying project experiences and the rest of JEE goes for plain Servlet. So tangram had to be provided with a plain Servlet way of doing the view layer.

Since the modularity of tangram was achieved by the Spring way of plugging components together with Dependency Injection, the first thing to do was, to mark the generic components in a spring independent way and to look at the other options for the Dependency Injection part. Only then it would be possible to replace the spring view layer with a Servlet view layer during the startup and wire-up of the application.

So the list of relevant DI frameworks gets shortened to those supporting the generic Dependency Injection annotations from JSR330 which are intended for JEE and can e.g. also be used with Google Guice and the Spring Framework alike.

From the reading Google Guice seemed to be a good alternative for the proof of concept phase, but it took me that much work to get something to run with it (not everything can be plugged together programmatically in my case), that I came out faster with my own Dependency Injection Container. Rather minimalistic and only suited for the setup of components.

Its advantage over Guice is that it's smaller and easier configurable with properties files. Weeks later I discovered TinyDI as another option. While this container seems to be a lot cleverer about the search of annotated classes it seems to lack the needed option of extending the configuration aspects from the annotations with properties files - defaults and overridden values and references.

So right at the moment I still don't have a running tangram application but all of the tangram framework now can be used with dinistiq. This example shows that now over 90% of the classes of tangram are free of direct dependencies to the Spring Framework while still taking advantage of its features and runtime environment. The code definitely got cleaner and more reusable.