Nick Johnson

Tonight I created the first localized, arbitrary HTML page in Tally-Ho. I don't have all of the corner cases handled yet, but I was able to go to the creation page, choose a Locale, give an arbitrary "path", filename, title and content, save the page and then view the page with a nice URL that makes it look like a static page.

The whole mess integrates directly into the Tally-Ho BinaryResourceService locator/localizer system, just as the existing Wicket integration via the BinaryResourceStreamLocator does now, so it automatically takes advantage of localization and caching.

One of the more cumbersome challenges in this effort was getting a nice URL. Wicket has many URL coding strategies for Bookmarkable pages; I use IndexedParamUrlCodingStrategy quite a bit. But IndexedParamUrlCodingStrategy wasn't going to work in this case. It takes each element of a path and associates it with an index number. What I needed was the path itself as a parameter, not chopped up and indexed.

It turns out that writing one of these coding strategies from scratch for Wicket is difficult if you don't know what you're doing (like me), and the Javadoc for the various classes involved is sadly a bit lacking in direction. So I took a different approach: I stole a bunch of code from IndexedParamUrlCodingStrategy and modified it to fit my needs. Behold, UriPathUrlCodingStrategy (with comments snipped for space... they're in CVS, though, and rest assured they credit Igor for writing IndexedParamUrlCodingStrategy):

package net.spatula.tally_ho.wicket;

import java.io.UnsupportedEncodingException;
import java.util.Map;

import wicket.Application;
import wicket.PageMap;
import wicket.PageParameters;
import wicket.WicketRuntimeException;
import wicket.protocol.http.request.WebRequestCodingStrategy;
import wicket.request.target.coding.BookmarkablePageRequestTargetUrlCodingStrategy;
import wicket.settings.IRequestCycleSettings;
import wicket.util.string.AppendingStringBuffer;
import wicket.util.value.ValueMap;

public class UriPathUrlCodingStrategy extends BookmarkablePageRequestTargetUrlCodingStrategy {


    public UriPathUrlCodingStrategy(String mountPath, Class bookmarkablePageClass) {
        super(mountPath, bookmarkablePageClass, PageMap.DEFAULT_NAME);
    }

    public UriPathUrlCodingStrategy(String mountPath, Class bookmarkablePageClass, String pageMapName) {
        super(mountPath, bookmarkablePageClass, pageMapName);
    }

    protected void appendParameters(AppendingStringBuffer url, Map parameters) {
        if (parameters.containsKey("uri")) {
            String[] pathParts = ((String) parameters.get("uri")).split("/+");
            for (String string : pathParts) {
                if (string == null || "".equals(string)) {
                    continue;
                }
                try {
                    Application app = Application.get();
                    IRequestCycleSettings settings = app.getRequestCycleSettings();
                    url.append("/").append(java.net.URLEncoder.encode(string, settings.getResponseRequestEncoding()));
                } catch (UnsupportedEncodingException e) {
                    throw new WicketRuntimeException(e);
                }
            }
        }

        String pageMap = (String) parameters.get(WebRequestCodingStrategy.PAGEMAP);
        if (pageMap != null) {
            url.append("/").append(WebRequestCodingStrategy.PAGEMAP).append("/").append(urlEncode(pageMap));
        }

    }

    protected ValueMap decodeParameters(String urlFragment, Map urlParameters) {
        PageParameters params = new PageParameters();
        if (urlFragment == null) {
            return params;
        }
        if (urlFragment.startsWith("/")) {
            urlFragment = urlFragment.substring(1);
        }

        String[] parts = urlFragment.split("/");
        StringBuilder builder = new StringBuilder();
        for (int i = 0; i < parts.length; i++) {
            if (WebRequestCodingStrategy.PAGEMAP.equals(parts[i])) {
                i++;
                params.put(WebRequestCodingStrategy.PAGEMAP, parts[i]);
            } else {
                builder.append("/").append(parts[i]);
            }
        }
        params.put("uri", builder.toString());
        return params;
    }

}

The next steps will be update capability for the HTML pages and then an attachment selector/uploader tool to handle the association of other resources to the HTML page.

This will be a giant leap forward for Tally-Ho and allow for the conversion of dozens of old morons.org pages to the new system.

Labels: software, tally-ho, wicket

And just where is that 1.0 release, anyway?

Well, I made good on my threat to rip out Toplink Essentials and replace it with OpenJPA. OpenJPA is a bit more pedantic about some things. For example, this code would run fine in Toplink but would throw an IllegalStateException in OpenJPA:

    entityManager.getTransaction.begin();
    entityManager.close();

While I was working on dropping in OpenJPA, I decided that I really wanted my tests to pass from within Maven, so I could be sure that run-time enhanced (woven) classes were all going to work nicely. I also wanted to make sure that none of the tests depended on any data to be in the database that wasn't put there by the SQL scripts to initialize the DB. So I modified my base test class to perform one-time database wiping/initialization prior to running any tests. This exposed a great many flaws in tests that I wrote in a fairly lazy fashion to assume that certain objects were already present.

After fixing all of that, I decided to let Eclipse clean up a lot of other code for me. Eclipse's Source-->Cleanup feature is very powerful, allowing you to "final" gobs of things and implement the default serial ID for Serializable classes in one giant swoop.

Then I got to work on what the next major project / feature is for Tally-Ho: arbitrary HTML pages. For quite a while I agonized over how to manage associations between pages. In most of the world, if a page changes its name or location, links to it break. I also needed the capability to attach images, PDFs or other documents to arbitrary HTML. It turns out that the solution to both of these problems is the same. In a massive refactoring of BinaryResource, any BinaryResourceReference can now be attached to any other BinaryResourceReference. BinaryResource is gone, and instead the relationship is now 1 BinaryResourceReference has many BinaryResourceReferenceLocales, each of which has one BinaryResourceContent. A BinaryResourceReference may also have many Attachments, which have an sequence number and a reference to the attached BinaryResourceReference. An HtmlPage is just a subclass of BinaryResourceReference with some bits added for the title, keywords, whether to include a message board, etc.

Attachments are numbered in sequence (1, 2, 3). Inside the HtmlPageService, references to attachments are converted by an AttachmentUrlProvider (an interface) and Velocity to their URLs. So if you want to refer to the URL for attachment #1, you use ${1} in the HTML. (Roughly... this bit isn't done yet.) It is up to the AttachmentUrlProvider to decide how to make the URL, given the scope, path and extension.

This refactoring is probably 75% complete. I'm too burnt out on code and too tired to work on it any more this weekend.

So to summarize what's changed:

1. OpenJPA replaced Toplink Essentials
2. Everything builds and tests in Maven (including compile time bytecode instrumentation)
3. Refactoring of binary resources
4. Initial HtmlPage work
5. Binary resource attachment support.

A couple other things I learned today about JPA:

1. If you JOIN multiple things, at least with OpenJPA you need to alias each thing you join. Ie, JOIN x.foo foo JOIN foo.bar bar. The parser will complain if you leave off that last "bar" in that example.
2. Your ability to lazy load ends once the EntityManager you used to load your object is closed. I knew this before and subsequently forgot, and then learned it again the hard way. Merging the entity with a new EntityManager doesn't work either. You need to keep the original one open until you're done navigating your object graph.

Labels: openjpa, software, tally-ho, toplink essentials, velocity

I've been using Toplink Essentials as the JPA provider for Tally Ho almost exclusively since the project began, except for a quick look at OpenJPA. This is in part because of my experience with the commercial Toplink product- I know that Oracle's Toplink is a mature product (having started out in the early 90's as a Smalltalk persistence provider), and I am comfortable working with it. Unfortunately, the open source Toplink Essentials product does not live up to the promise of Toplink. I've reached the point where I'm tired of coding around its bugs, and now that there are other, healthier projects out there, I shouldn't have to.

That got me to thinking: a lot of us developers use a lot of open source software. How do we choose which packages we want to use? Obviously whatever we choose has to be a good technical fit for our needs... what's the point if it doesn't do the job we're after? But now it occurs to me that open source software has to meet a particular social need as well. One way we can gauge a project's health is by how strong it is socially. How interested are people? Is the project active? Are people excited about the project? Excited enough to fix bugs?

It's a little hard to compare apples to apples in this case, but let's look at a couple things and try to relate them as best we can. Toplink Essentials is maintained as part of the Glassfish project.

In the last 30 days at the time of this writing, the folks working on it have fixed 10 bugs. In that same amount of time, 15 bugs were opened (or changed and left in an opened state). About 53 messages have been posted on the discussion forum. The oldest unresolved bug has been open for about a year and 10 months.

In that same amount of time, the OpenJPA folks have resolved 19 bugs while 20 have been opened. The mailing list has had about 215 posts. The oldest unresolved bug is a year and 4 months old (though it was touched 3 months ago). OpenJPA is using Jira which makes it a bit easier to produce meaningful metrics such that we can find that the average unresolved age of a bug in the last month is about 3 months, which has been fairly consistent.

(I gave up trying to compute the average unresolved age of bugs for Toplink Essentials. It's just too annoying to figure out if the bug tracking tool doesn't do it for you.)

It is probably the case that most open source projects (and probably closed ones too) have a few ancient bugs gathering dust. I think that it's more interesting to look at what a project has been doing recently, like in the last 30-180 days. Are they keeping up with their bug backlog? Is there an active community? Are you likely to get help if you ask for it? Of the bugs that come in, what percentage get fixed and what percentage get dumped in the attic?

And perhaps the most important criteria of all: are they fixing MY bug?

While I wasn't watching, OpenJPA reached a 1.0.0 release. It's available under the Apache 2 license from a Maven 2 repository. They fixed the bug I opened earlier this year (within a day even). It is full-featured and even has an extensive manual. Though, like Toplink, their ant task doesn't work very well.

I used to be concerned about the large number of dependencies that OpenJPA has, but now that the project is building with Maven 2, it's much less of a concern for me. It isn't necessary to go manually fetch anything to build the project, since Maven 2 takes care of all the direct and transitive dependencies. One thing I did have to manually tweak was to force inclusion of commons-collections 3.2 in my pom.xml, because something else in my project depends on an earlier version of commons-collections, and OpenJPA needs a later version.

So it's time to give Toplink one final heave-ho. My reasons for sticking with it have now been outweighed by my need of having compile-time weaving that works and a project where problems are likely to be fixed within my lifetime. It's time for Toplink and I to start seeing other people.

New releases of Tally-Ho will be using OpenJPA as the persistence provider... just as soon as I get all the unit tests passing.

Labels: JPA, open JPA, open source, openjpa, software, tally-ho, toplink, toplink essentials, weaving

I've always wanted to make sure that the build process for Tally Ho is as seamless as possible, requiring minimal configuration by someone who downloads the monster. Currently, I do have one small step that people have to do- they must install the Toplink Essentials jar files in their local repository. This is necessary because of a licensing issue from Oracle apparently... you have to agree to a license before their jar will unpack. Fortunately, this is a fairly simple procedure, which I cover in README-MAVEN thusly:

toplink-essentials-V2_build_58: Get this from
https://glassfish.dev.java.net/downloads/persistence/JavaPersistence.html.
Download the jar, then run java -jar glassfish-persistence-installer*.jar,
accept the license agreement, and the installer will create a
glassfish-persistence directory. Change into that directory and run
mvn install:install-file -Dfile=toplink-essentials.jar \
-DgroupId=toplink-essentials -DartifactId=toplink-essentials \
-Dversion=V2_build_58 -Dpackaging=jar

So far in the project I haven't used static weaving, because it hasn't been all that vital. But an upcoming change in the way binary resources work requires it. (I need to be able to refer to a binary resource without loading all of that resource's data, since that could conceivably be a tremendous amount of data and a huge memory hog... this requires a lazy-loaded 1:1 relationship (Toplink Essentials does not support lazily loaded compositions)).

The Toplink folks do not provide a Maven 2 plugin for doing static weaving (and you must use static weaving if you're deploying to a J2SE servlet container due to classloader constraints on javaagent). They do, however, provide an ant task for static weaving, and Maven 2 can run ant tasks.

It took some time to figure out how to get the Maven dependency classpath into the Ant task so that the Ant task could find the class for static weaving, but after some digging I found the answer.

First, we define a build.xml for the ant task, to keep from severely uglifying pom.xml:

<project name="Weaver" default="weaving" basedir=".">

 <description>
  Run the ant task for performing static weaving on model classes.  This
  is meant to be run from m2 with the compile_classpath variable set.
 </description>
 
 <target name="define.task" description="New task definition for toplink static weaving">  
  <taskdef name="weave" classname="oracle.toplink.essentials.weaving.StaticWeaveAntTask">
   <classpath>
    <path path="${compile_classpath}" />
   </classpath>
  </taskdef>  
 </target>
 
 <target name="weaving" description="perform weaving" depends="define.task">
  <echo>Performing static weaving on model classes
  
  <weave source="target/classes" target="target/classes" persistenceinfo="src/main/resources">
   <classpath>   
             <path path="${compile_classpath}"/>
         </classpath>
  </weave>
 </target>
 
</project>

The Eclipse ant task editor will of course complain that the taskdef class cannot be found, but that's okay because we don't intend to run this with ant. We're going to run it from Maven2, using the ant task runner.

One nice thing about Maven 2 is that they've included a phase just for post-processing of classes, which is the ideal place to hook into the compilation process. We add this to our pom.xml inside the <build><plugins>:

      <plugin>
       <groupId>org.apache.maven.plugins
        <artifactId>maven-antrun-plugin
        <executions>
          <execution>
            <id>process-classes
            <phase>process-classes
            <configuration>          
              
                <echo>Beginning process-classes phase...
                <property name="compile_classpath" refid="maven.compile.classpath"/>    
                <ant antfile="${basedir}/build.xml">                 
                  <target name="weaving"/>
                </ant>
              </tasks>
            </configuration>
            <goals>
              <goal>run
            </goals>
          </execution>
        </executions>
      </plugin>

Maven creates an ant property called compile_classpath which dereferences to maven.compile.classpath property, which includes all of the compile-time dependencies declared in the pom.xml. In this case, since the pom already contains the Toplink Essentials jar file, the compile classpath will contain the class needed to run the ant taskdef.

There are, of course, still bugs with static weaving. The weaver still breaks if there is a space in the path to your classes and it still incorrectly weaves classes with lazy 1:1 relationships, failing to add some required methods for 1:1 fields that aren't lazy loaded. As these bugs haven't been touched since March of this year, I don't hold out a lot of hope for seeing them fixed any time soon.

The workaround for the first problem is to move your Eclipse workspace (or other working directory) to a path with no spaces in the name. The second can be worked around by using property access on lazy-loaded fields, although that is lame and stupid.

What kills me about that latter bug is that in the comments, the person the bug is assigned to describes exactly what needs to be done to fix the bug and where in the code to fix it. This means he had to have been looking around in the code to find it. And once he found it, rather than just fixing the damn problem, then saving and committing, he talked about how to fix it on the bug report instead. And indeed, one can still go look at the code and see how it's still broken to this day, when it could have been resolved 8 months ago for less effort than it took to write about it. It's literally a one-line fix. Maybe even half-a-line, if you want to get technical. Select some text, hit backspace, ctrl-S, run unit tests, commit.

Incidentally, it also turns out that the Toplink ant task does absolutely nothing at present. If you find that troubling, you can swap out the <weave> task for a kludge like this:

  <java classname="oracle.toplink.essentials.weaving.StaticWeave" >
   <classpath>
    <path path="${compile_classpath}"/>
   </classpath>
   <arg value="-persistenceinfo" />
   <arg value="src/main/resources" />
   <arg value="-loglevel" />
   <arg value="finest" />
   <arg value="${target_directory}" />
   <arg value="${target_directory}" />
  </java>
 

At least this way the ugliness is encapsulated in an ant build.xml file, and some day when the ant task gets fixed, you can return to using it easily.

Labels: ant, maven, software, toplink, weaving