Nick Johnson

Tonight I created the first localized, arbitrary HTML page in Tally-Ho. I don't have all of the corner cases handled yet, but I was able to go to the creation page, choose a Locale, give an arbitrary "path", filename, title and content, save the page and then view the page with a nice URL that makes it look like a static page.

The whole mess integrates directly into the Tally-Ho BinaryResourceService locator/localizer system, just as the existing Wicket integration via the BinaryResourceStreamLocator does now, so it automatically takes advantage of localization and caching.

One of the more cumbersome challenges in this effort was getting a nice URL. Wicket has many URL coding strategies for Bookmarkable pages; I use IndexedParamUrlCodingStrategy quite a bit. But IndexedParamUrlCodingStrategy wasn't going to work in this case. It takes each element of a path and associates it with an index number. What I needed was the path itself as a parameter, not chopped up and indexed.

It turns out that writing one of these coding strategies from scratch for Wicket is difficult if you don't know what you're doing (like me), and the Javadoc for the various classes involved is sadly a bit lacking in direction. So I took a different approach: I stole a bunch of code from IndexedParamUrlCodingStrategy and modified it to fit my needs. Behold, UriPathUrlCodingStrategy (with comments snipped for space... they're in CVS, though, and rest assured they credit Igor for writing IndexedParamUrlCodingStrategy):

package net.spatula.tally_ho.wicket;

import java.io.UnsupportedEncodingException;
import java.util.Map;

import wicket.Application;
import wicket.PageMap;
import wicket.PageParameters;
import wicket.WicketRuntimeException;
import wicket.protocol.http.request.WebRequestCodingStrategy;
import wicket.request.target.coding.BookmarkablePageRequestTargetUrlCodingStrategy;
import wicket.settings.IRequestCycleSettings;
import wicket.util.string.AppendingStringBuffer;
import wicket.util.value.ValueMap;

public class UriPathUrlCodingStrategy extends BookmarkablePageRequestTargetUrlCodingStrategy {


    public UriPathUrlCodingStrategy(String mountPath, Class bookmarkablePageClass) {
        super(mountPath, bookmarkablePageClass, PageMap.DEFAULT_NAME);
    }

    public UriPathUrlCodingStrategy(String mountPath, Class bookmarkablePageClass, String pageMapName) {
        super(mountPath, bookmarkablePageClass, pageMapName);
    }

    protected void appendParameters(AppendingStringBuffer url, Map parameters) {
        if (parameters.containsKey("uri")) {
            String[] pathParts = ((String) parameters.get("uri")).split("/+");
            for (String string : pathParts) {
                if (string == null || "".equals(string)) {
                    continue;
                }
                try {
                    Application app = Application.get();
                    IRequestCycleSettings settings = app.getRequestCycleSettings();
                    url.append("/").append(java.net.URLEncoder.encode(string, settings.getResponseRequestEncoding()));
                } catch (UnsupportedEncodingException e) {
                    throw new WicketRuntimeException(e);
                }
            }
        }

        String pageMap = (String) parameters.get(WebRequestCodingStrategy.PAGEMAP);
        if (pageMap != null) {
            url.append("/").append(WebRequestCodingStrategy.PAGEMAP).append("/").append(urlEncode(pageMap));
        }

    }

    protected ValueMap decodeParameters(String urlFragment, Map urlParameters) {
        PageParameters params = new PageParameters();
        if (urlFragment == null) {
            return params;
        }
        if (urlFragment.startsWith("/")) {
            urlFragment = urlFragment.substring(1);
        }

        String[] parts = urlFragment.split("/");
        StringBuilder builder = new StringBuilder();
        for (int i = 0; i < parts.length; i++) {
            if (WebRequestCodingStrategy.PAGEMAP.equals(parts[i])) {
                i++;
                params.put(WebRequestCodingStrategy.PAGEMAP, parts[i]);
            } else {
                builder.append("/").append(parts[i]);
            }
        }
        params.put("uri", builder.toString());
        return params;
    }

}

The next steps will be update capability for the HTML pages and then an attachment selector/uploader tool to handle the association of other resources to the HTML page.

This will be a giant leap forward for Tally-Ho and allow for the conversion of dozens of old morons.org pages to the new system.

Labels: software, tally-ho, wicket

And just where is that 1.0 release, anyway?

Well, I made good on my threat to rip out Toplink Essentials and replace it with OpenJPA. OpenJPA is a bit more pedantic about some things. For example, this code would run fine in Toplink but would throw an IllegalStateException in OpenJPA:

    entityManager.getTransaction.begin();
    entityManager.close();

While I was working on dropping in OpenJPA, I decided that I really wanted my tests to pass from within Maven, so I could be sure that run-time enhanced (woven) classes were all going to work nicely. I also wanted to make sure that none of the tests depended on any data to be in the database that wasn't put there by the SQL scripts to initialize the DB. So I modified my base test class to perform one-time database wiping/initialization prior to running any tests. This exposed a great many flaws in tests that I wrote in a fairly lazy fashion to assume that certain objects were already present.

After fixing all of that, I decided to let Eclipse clean up a lot of other code for me. Eclipse's Source-->Cleanup feature is very powerful, allowing you to "final" gobs of things and implement the default serial ID for Serializable classes in one giant swoop.

Then I got to work on what the next major project / feature is for Tally-Ho: arbitrary HTML pages. For quite a while I agonized over how to manage associations between pages. In most of the world, if a page changes its name or location, links to it break. I also needed the capability to attach images, PDFs or other documents to arbitrary HTML. It turns out that the solution to both of these problems is the same. In a massive refactoring of BinaryResource, any BinaryResourceReference can now be attached to any other BinaryResourceReference. BinaryResource is gone, and instead the relationship is now 1 BinaryResourceReference has many BinaryResourceReferenceLocales, each of which has one BinaryResourceContent. A BinaryResourceReference may also have many Attachments, which have an sequence number and a reference to the attached BinaryResourceReference. An HtmlPage is just a subclass of BinaryResourceReference with some bits added for the title, keywords, whether to include a message board, etc.

Attachments are numbered in sequence (1, 2, 3). Inside the HtmlPageService, references to attachments are converted by an AttachmentUrlProvider (an interface) and Velocity to their URLs. So if you want to refer to the URL for attachment #1, you use ${1} in the HTML. (Roughly... this bit isn't done yet.) It is up to the AttachmentUrlProvider to decide how to make the URL, given the scope, path and extension.

This refactoring is probably 75% complete. I'm too burnt out on code and too tired to work on it any more this weekend.

So to summarize what's changed:

1. OpenJPA replaced Toplink Essentials
2. Everything builds and tests in Maven (including compile time bytecode instrumentation)
3. Refactoring of binary resources
4. Initial HtmlPage work
5. Binary resource attachment support.

A couple other things I learned today about JPA:

1. If you JOIN multiple things, at least with OpenJPA you need to alias each thing you join. Ie, JOIN x.foo foo JOIN foo.bar bar. The parser will complain if you leave off that last "bar" in that example.
2. Your ability to lazy load ends once the EntityManager you used to load your object is closed. I knew this before and subsequently forgot, and then learned it again the hard way. Merging the entity with a new EntityManager doesn't work either. You need to keep the original one open until you're done navigating your object graph.

Labels: openjpa, software, tally-ho, toplink essentials, velocity

I've been using Toplink Essentials as the JPA provider for Tally Ho almost exclusively since the project began, except for a quick look at OpenJPA. This is in part because of my experience with the commercial Toplink product- I know that Oracle's Toplink is a mature product (having started out in the early 90's as a Smalltalk persistence provider), and I am comfortable working with it. Unfortunately, the open source Toplink Essentials product does not live up to the promise of Toplink. I've reached the point where I'm tired of coding around its bugs, and now that there are other, healthier projects out there, I shouldn't have to.

That got me to thinking: a lot of us developers use a lot of open source software. How do we choose which packages we want to use? Obviously whatever we choose has to be a good technical fit for our needs... what's the point if it doesn't do the job we're after? But now it occurs to me that open source software has to meet a particular social need as well. One way we can gauge a project's health is by how strong it is socially. How interested are people? Is the project active? Are people excited about the project? Excited enough to fix bugs?

It's a little hard to compare apples to apples in this case, but let's look at a couple things and try to relate them as best we can. Toplink Essentials is maintained as part of the Glassfish project.

In the last 30 days at the time of this writing, the folks working on it have fixed 10 bugs. In that same amount of time, 15 bugs were opened (or changed and left in an opened state). About 53 messages have been posted on the discussion forum. The oldest unresolved bug has been open for about a year and 10 months.

In that same amount of time, the OpenJPA folks have resolved 19 bugs while 20 have been opened. The mailing list has had about 215 posts. The oldest unresolved bug is a year and 4 months old (though it was touched 3 months ago). OpenJPA is using Jira which makes it a bit easier to produce meaningful metrics such that we can find that the average unresolved age of a bug in the last month is about 3 months, which has been fairly consistent.

(I gave up trying to compute the average unresolved age of bugs for Toplink Essentials. It's just too annoying to figure out if the bug tracking tool doesn't do it for you.)

It is probably the case that most open source projects (and probably closed ones too) have a few ancient bugs gathering dust. I think that it's more interesting to look at what a project has been doing recently, like in the last 30-180 days. Are they keeping up with their bug backlog? Is there an active community? Are you likely to get help if you ask for it? Of the bugs that come in, what percentage get fixed and what percentage get dumped in the attic?

And perhaps the most important criteria of all: are they fixing MY bug?

While I wasn't watching, OpenJPA reached a 1.0.0 release. It's available under the Apache 2 license from a Maven 2 repository. They fixed the bug I opened earlier this year (within a day even). It is full-featured and even has an extensive manual. Though, like Toplink, their ant task doesn't work very well.

I used to be concerned about the large number of dependencies that OpenJPA has, but now that the project is building with Maven 2, it's much less of a concern for me. It isn't necessary to go manually fetch anything to build the project, since Maven 2 takes care of all the direct and transitive dependencies. One thing I did have to manually tweak was to force inclusion of commons-collections 3.2 in my pom.xml, because something else in my project depends on an earlier version of commons-collections, and OpenJPA needs a later version.

So it's time to give Toplink one final heave-ho. My reasons for sticking with it have now been outweighed by my need of having compile-time weaving that works and a project where problems are likely to be fixed within my lifetime. It's time for Toplink and I to start seeing other people.

New releases of Tally-Ho will be using OpenJPA as the persistence provider... just as soon as I get all the unit tests passing.

Labels: JPA, open JPA, open source, openjpa, software, tally-ho, toplink, toplink essentials, weaving

I've always wanted to make sure that the build process for Tally Ho is as seamless as possible, requiring minimal configuration by someone who downloads the monster. Currently, I do have one small step that people have to do- they must install the Toplink Essentials jar files in their local repository. This is necessary because of a licensing issue from Oracle apparently... you have to agree to a license before their jar will unpack. Fortunately, this is a fairly simple procedure, which I cover in README-MAVEN thusly:

toplink-essentials-V2_build_58: Get this from
https://glassfish.dev.java.net/downloads/persistence/JavaPersistence.html.
Download the jar, then run java -jar glassfish-persistence-installer*.jar,
accept the license agreement, and the installer will create a
glassfish-persistence directory. Change into that directory and run
mvn install:install-file -Dfile=toplink-essentials.jar \
-DgroupId=toplink-essentials -DartifactId=toplink-essentials \
-Dversion=V2_build_58 -Dpackaging=jar

So far in the project I haven't used static weaving, because it hasn't been all that vital. But an upcoming change in the way binary resources work requires it. (I need to be able to refer to a binary resource without loading all of that resource's data, since that could conceivably be a tremendous amount of data and a huge memory hog... this requires a lazy-loaded 1:1 relationship (Toplink Essentials does not support lazily loaded compositions)).

The Toplink folks do not provide a Maven 2 plugin for doing static weaving (and you must use static weaving if you're deploying to a J2SE servlet container due to classloader constraints on javaagent). They do, however, provide an ant task for static weaving, and Maven 2 can run ant tasks.

It took some time to figure out how to get the Maven dependency classpath into the Ant task so that the Ant task could find the class for static weaving, but after some digging I found the answer.

First, we define a build.xml for the ant task, to keep from severely uglifying pom.xml:

<project name="Weaver" default="weaving" basedir=".">

 <description>
  Run the ant task for performing static weaving on model classes.  This
  is meant to be run from m2 with the compile_classpath variable set.
 </description>
 
 <target name="define.task" description="New task definition for toplink static weaving">  
  <taskdef name="weave" classname="oracle.toplink.essentials.weaving.StaticWeaveAntTask">
   <classpath>
    <path path="${compile_classpath}" />
   </classpath>
  </taskdef>  
 </target>
 
 <target name="weaving" description="perform weaving" depends="define.task">
  <echo>Performing static weaving on model classes
  
  <weave source="target/classes" target="target/classes" persistenceinfo="src/main/resources">
   <classpath>   
             <path path="${compile_classpath}"/>
         </classpath>
  </weave>
 </target>
 
</project>

The Eclipse ant task editor will of course complain that the taskdef class cannot be found, but that's okay because we don't intend to run this with ant. We're going to run it from Maven2, using the ant task runner.

One nice thing about Maven 2 is that they've included a phase just for post-processing of classes, which is the ideal place to hook into the compilation process. We add this to our pom.xml inside the <build><plugins>:

      <plugin>
       <groupId>org.apache.maven.plugins
        <artifactId>maven-antrun-plugin
        <executions>
          <execution>
            <id>process-classes
            <phase>process-classes
            <configuration>          
              
                <echo>Beginning process-classes phase...
                <property name="compile_classpath" refid="maven.compile.classpath"/>    
                <ant antfile="${basedir}/build.xml">                 
                  <target name="weaving"/>
                </ant>
              </tasks>
            </configuration>
            <goals>
              <goal>run
            </goals>
          </execution>
        </executions>
      </plugin>

Maven creates an ant property called compile_classpath which dereferences to maven.compile.classpath property, which includes all of the compile-time dependencies declared in the pom.xml. In this case, since the pom already contains the Toplink Essentials jar file, the compile classpath will contain the class needed to run the ant taskdef.

There are, of course, still bugs with static weaving. The weaver still breaks if there is a space in the path to your classes and it still incorrectly weaves classes with lazy 1:1 relationships, failing to add some required methods for 1:1 fields that aren't lazy loaded. As these bugs haven't been touched since March of this year, I don't hold out a lot of hope for seeing them fixed any time soon.

The workaround for the first problem is to move your Eclipse workspace (or other working directory) to a path with no spaces in the name. The second can be worked around by using property access on lazy-loaded fields, although that is lame and stupid.

What kills me about that latter bug is that in the comments, the person the bug is assigned to describes exactly what needs to be done to fix the bug and where in the code to fix it. This means he had to have been looking around in the code to find it. And once he found it, rather than just fixing the damn problem, then saving and committing, he talked about how to fix it on the bug report instead. And indeed, one can still go look at the code and see how it's still broken to this day, when it could have been resolved 8 months ago for less effort than it took to write about it. It's literally a one-line fix. Maybe even half-a-line, if you want to get technical. Select some text, hit backspace, ctrl-S, run unit tests, commit.

Incidentally, it also turns out that the Toplink ant task does absolutely nothing at present. If you find that troubling, you can swap out the <weave> task for a kludge like this:

  <java classname="oracle.toplink.essentials.weaving.StaticWeave" >
   <classpath>
    <path path="${compile_classpath}"/>
   </classpath>
   <arg value="-persistenceinfo" />
   <arg value="src/main/resources" />
   <arg value="-loglevel" />
   <arg value="finest" />
   <arg value="${target_directory}" />
   <arg value="${target_directory}" />
  </java>
 

At least this way the ugliness is encapsulated in an ant build.xml file, and some day when the ant task gets fixed, you can return to using it easily.

Labels: ant, maven, software, toplink, weaving

In the 17 or so years that I've been actively writing and debugging software, I have never come across a problem as convoluted as the one I finally resolved this evening. I literally fought with this problem for months before finally cracking it.

The background: I work on a project called Tally-Ho, a community management system that powers morons.org. It's a massive Java application using the Wicket framework which I run inside Tomcat 6.0.14.

One of the features of Tally Ho is a "story lead" queue where people can submit interesting news articles that they find around the Internet. There is a fairly simplistic URL validator on the URL field of the form which attempts to load the URL supplied by the user, making sure the host resolves, can be connected to, and doesn't return a 404.

The problem was that after a while, instead of resolving hosts, my log would fill up with UnknownHostExceptions. I could do an nslookup from the command line and see that these were legitimate hosts... some of the obviously so, like washingtonpost.com.

It looked like a classic negative caching problem at first, and at first it probably was, in part. In Java, address lookups, both positive and negative, get cached. My initial hypothesis was that some transient lookup failure was causing a negative lookup to be cached, and that the caching was lasting forever, despite my configuring a networkaddress.cache.negative.ttl value in $JAVA_HOME/jre/lib/security/java.security. This hypothesis seemed reasonable in part because I could see by snooping network traffic on port 53 that the JVM was not making any further DNS requests for the hosts in question. Also, restarting the JVM seemed to clear the problem every time, suggesting that once the host was in the nameserver's cache, everything was fine.

I began trying various things including using some old, deprecated Sun-specific caching options. That didn't work. I tried hacking at the InetAddress source code to completely remove its ability to cache. That seemed to work at first, but later the old behaviour somehow returned. Then I discovered using truss and ktrace that my JVM wasn't reading java.security at all, and -Djava.security.debug=properties didn't print anything. I rebuilt the JVM from the FreeBSD port after first removing the entire work directory and indicating that it should use the precompiled "Diablo" JVM to bootstrap the new JVM.

The rebuilt JVM seemed to read java.security, so I figured the problem was solved. Not so. It still happened after Tomcat ran for a while.

I wrote a simple command line tester which attempted a name lookup, waited for a keypress, and then tried the name lookup again. Then I'd restart named, firewall the name servers for that host, and run the test code. I could verify that it retried the host and did not negative-cache it forever when run from the command line. So something was different in what was happening inside Tomcat.

It was then that I noticed that Tomcat was running with its native libraries. I've seen strange things happen before whenever JNI was involved, so I poked around a bit and noticed with ldd that the native libraries had been built with gcc 4.2.1. Knowing that gcc 4.2.1 has serious problems, I rebuilt the native libraries and restarted Tomcat. I repeated the same steps I used in my command line test with my submission form via Tomcat, and saw that things seemed to be working now.

Hours went by, and the same damn exception flew up my log again. What the hell? It was then that I was breathing fire, my entire being converted into pure fury.

I decided to dive a level deeper, running ktrace against the running Tomcat process so I could see every system call it made. One red herring I dispensed with fairly quickly was that in the cases where hosts seemed to resolve properly, the JVM was reading /etc/resolv.conf, and in cases where they didn't it wasn't. But looking at the source code for ResolverConfigurationImpl, it was clear that this was probably due to its internal caching (hard coded to 5 minutes, mind you).

One thing in the kdump did catch my eye though:

 31013 jsvc     CALL  socket(0x2,0x2,0)
 31013 jsvc     RET   socket 1414/0x586
 31013 jsvc     CALL  close(0x586)

That file handle sure seems awfully big for a servlet container with a maximum of 256 simultaneous connections. Somewhere along this time I had also noticed that everything was fine when Tomcat had been recently restarted, but went bad after a while. I had also noticed at some point that caching seemed to have nothing to do with it. Once the failure mode had been entered, it didn't matter what the address was-- the resolver would throw an UnknownHostException for every host, immediately, without ever attempting a lookup to begin with.

So now I had a new hypothesis. That file handle number was awfully high. I was able to develop a test case that demonstrated that name resolution failed as soon as 1024 file handles were in use:

import java.net.*;
import java.util.*;
import java.io.*;

public class Test3 {
        public static void main(String[] args) throws Exception {

                ArrayList files = new ArrayList(1024);

                System.out.println("Opening lots of files");
                for(int i=0; i < 1024; i++) {
                        files.add(new FileInputStream("/dev/null"));
                }

                System.out.println("Trying to resolve freebsd.org");
                InetAddress.getByName("freebsd.org");   // throws exception!

        }
}

(It's actually only necessary to open 1020 files; stdin, stdout and stderr bump that number up to 1023, and on the 1024th file handle, it breaks).

My friend Alfred recalled that this is a FreeBSD libc bug, which has been corrected since my fairly ancient compilation of FreeBSD 6.2. At some time in the distant past, some library calls would refuse to cooperate with file handles > 1023 because they couldn't be used with select(2). My test case runs to completion on his host, but always fails with an UnknownHostException on my host. (On Linux it dies and complains about 'too many open files'. Teehee.)

So why was Tomcat leaking all these file descriptors? My first suspicion was the NioConnector, since it's new and known to be a bit buggy. I reconfigured Tomcat to use the older HTTP/1.1 connector. I waited a while, and ran ktrace on the process. No good, it was still using hundreds more file descriptors than it should have.

I decided to run fstat on the Tomcat process, and saw that it wasn't leaked sockets at all, but leaked file descriptors. Fstat, despite what the manual page claims about it showing open files, only shows the inode numbers of open files. (What dork thought that would be useful?) I downloaded and compiled lsof, which actually does list the files being held open by a process.

It was then that I saw the real root of all of this trouble: the directory structure used by Lucene, the search engine used by Tally Ho, was not being closed. Apparently each time the article search form was used, it was leaving a number of files opened. This was easy to fix by correcting a try/catch/finally block in the Java code to ensure that the Directory and IndexReader objects were always closed after use.

So to make a long story short, because the Directory object in the search engine wasn't getting closed, the application was overusing file handles, which was tickling a bug in FreeBSD that prevented socket writes from working correctly, which prevented hostnames from resolving, only after Tomcat had been running for a sufficient time to exhaust 1023 file handles, and this was after correcting a problem with a JVM that didn't read the java.security network address cache settings and a native library that was compiled with a bad version of gcc.

Holy f-ing crap.

The key lessons to learn from all of this are lessons for any debugging experience:

1. Develop a hypothesis, but don't get attached to it. (Your initial hypothesis may be wrong.) Revise your hypothesis as you get closer to the answer.
2. Eliminate unnecessary variables. (Like getting rid of native libraries.)
3. Check and recheck your assumptions. (Is java.security even getting read?)
4. Eliminate red herrings. (Reading resolv.conf has nothing to do with it.)
5. Collect more information any way you can. (ktrace, debugging statements inserted into the API, etc)
6. Compare and contrast the information about what happens when things go right with what happens when things go wrong. What's different? What's the same? (What is up with those huge numbered file handles?)
7. Devise the simplest possible test case or isolate the code path that always replicates the problem.
8. Investigate what happened that got you to that code path.

And maybe the all-important last step: tell other people about your experience on your blog, so others can benefit from your nightmare.

Labels: convolution, dns, FreeBSD, java, leak, Lucene, sockets, software, unknownhostexception

Today marks one year since I did the first commits on Tally Ho, the software behemoth that runs morons.org. Well, at least part of it.

I guess I didn't figure on it taking well over a year to port the whole site over to a new architecture, one that would scale nicely and could be maintained without agony. Life has this tendency to get complicated.

When I first started thinking about doing the massive morons.org refactoring (before it even had a name), I didn't even have a boyfriend. I hadn't yet started taking classes in the Hendrickson Method of Orthopedic Massage. I certainly hadn't decided to aim myself in the direction of Chiropractic School and to begin walking.

Yeah, life gets complicated. And as we get older, our priorities change too. I think large, open-source projects may be better suited for kids in their 20's, who still have lots of energy, free time, and don't get laid. What a perfect combination for free software development!

Now that I'm on in years, I want other things out of life. I want to travel, to see new things. I want to climb Mount Shasta. I want to maintain a healthy relationship. I want to always be learning things, especially things unrelated to computer science, so I'm always challenged. I want to spend time in the gym, working on my strength, cardiovascular fitness and endurance. I want to watch many more years of Doctor Who. I want to try to compose some music, even if I don't share it with anyone.

Now don't take this to mean that Tally-Ho is dead or won't be completed. I just committed some code yesterday to bring back the Partners system. Instead, take this to mean that Tally Ho is taking longer than expected, because life gets complicated.

Happy first birthday, Tally Ho! Maybe we'll get to version 1.0 within another year!

Labels: life, software, tally-ho

Consider a common polling loop:

while(true) {
    runSomeQuery();
    doSomeStuff();
    sleepBeforeNextIteration();
}

I've seen and used this pattern probably a thousand times, without really thinking about it. It just stood to reason that the sleeping bit should be some constant number of seconds, or maybe in a fancy case, a computed number of seconds taking into account the time that was spent running a query and doing stuff.

But why should a heartbeat by so regular? It may be worth taking some time to consider the characteristics of doSomeStuff(), especially in the case that doSomeStuff() might be altering the results of runSomeQuery().

I recently had a situation where this was true. The stuff-doing was creating a new row to be returned by the query. Yet even knowing that I had just placed something into my queue, it was going to still be some regular sleep duration before I queried again.

As it happens, where this loop exists I don't necessarily get to know whether I will be producing new rows or not or whether new rows have been produced (without going into the details, it's a highly decoupled architecture). But a fairly simplistic optimization can improve performance significantly without increasing the average number of queries per minute: instead of having one, regular sleep duration, I introduced two durations.

To see how this works, consider a job that creates a second job:

time (s)	event
0	All is quiet
1	Job 1 is inserted
3	*heartbeat* Job 1 is pulled from the queue
3.01	Job 1 executes and places Job 2 in the queue
6	*heartbeat* Job 2 is pulled from the queue and executes

Now consider what happens with a rhythmic, yet non-regular heartbeat:

time (s)	event
0	All is quiet
1	Job 1 is inserted
4	*heartbeat* Job 1 is pulled from the queue
4.01	Job 1 executes and places Job 2 in the queue
6	*heartbeat* Job 2 is pulled from the queue and executes

As you can see, in this case, both methods perform exactly the same way. But consider what happens in the case where the job happens to get inserted just before the first query runs:

time (s)	event
0	All is quiet
3	Job 1 is inserted
3	*heartbeat* Job 1 is pulled from the queue
3.01	Job 1 executes and places Job 2 in the queue
6	*heartbeat* Job 2 is pulled from the queue and executes

Now consider what happens with a rhythmic, yet non-regular heartbeat:

time (s)	event
0	All is quiet
4	Job 1 is inserted
4	*heartbeat* Job 1 is pulled from the queue
4.01	Job 1 executes and places Job 2 in the queue
6	*heartbeat* Job 2 is pulled from the queue and executes

In the lucky case, a full second is chopped off the amount of time it takes from job insertion of the first job to execution of the second job.

Then there's the most unlucky case:

time (s)	event
4	All is quiet
4.1	Job 1 is inserted
6	*heartbeat* Job 1 is pulled from the queue
6.01	Job 1 executes and places Job 2 in the queue
10	*heartbeat* Job 2 is pulled from the queue and executes

In this case, the first job happens to hit right after the query that followed the long sleep. But it still gets to executing the second job within 6 seconds. The regular heartbeat is guaranteed to complete within 6 seconds, but could go as quickly as just over 3 seconds in its most lucky case.

By alternating between a long sleep and a short sleep, we can shave off 1 second in the most lucky case, but in the least lucky case, we're no worse off than we would be with a regular heartbeat. In the real world, jobs will come in at any random time, so sometimes you'll get lucky and sometimes you won't... but when you do get lucky, the payout is considerable. This is especially true given that the cost is effectively nothing: the same number of queries will be performed over a minute (or indeed over 6 seconds in this case) and the cost of alternating the sleep times is negligible.

Remember, of course, that this won't help all situations. It only applied in my case because the result of performing processing on something I get from my query could be to cause something new to be returned from running the query again.

There might be additional tweaks that could be made to this general idea; for example, speeding up the heartbeat when rows are found by the query, then slowing it back down when they aren't, perhaps bounded by some rules to ensure that over a given time interval, some maximum number of queries are performed.

The moral of the story is to know what your workload tends to look like and make reasonable and simple trade-offs that might improve your performance.

Labels: heartbeat, polling, queue, queueing, software

You can always tell an overengineered API when you try to do something really simplistic and it takes you about 30 lines of code and 8 different classes.

For example: I want to read an image, make it smaller, and write it out as a JPG with a given quality. This is a fairly common and simple task. Here's the code:

        BufferedImage sourceImage
        try {
            sourceImage = ImageIO.read(new ByteArrayInputStream(imageData));
        } catch (IOException e) {
            return new ServiceResult(ServiceResult.STATUS.FAIL_HARD,
                    "Unable to read image; image is unreadable or an unsupported type", e);
        }

        // figure out the target width and height
        
        BufferedImage newImage =  getScaledInstance(sourceImage, targetWidth, targetHeight, RenderingHints.VALUE_INTERPOLATION_BILINEAR, true);

        ByteArrayOutputStream output = new ByteArrayOutputStream();

        Iterator imageWritersByMIMEType = ImageIO.getImageWritersByMIMEType("image/jpeg");
        if (imageWritersByMIMEType.hasNext()) {
            ImageWriter writer = imageWritersByMIMEType.next();
            writer.setOutput(new MemoryCacheImageOutputStream(output));
            ImageWriteParam iwp = writer.getDefaultWriteParam();
            iwp.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
            iwp.setCompressionQuality(COMPRESSION_QUALITY);
            
            
            IIOImage tmpImage = new IIOImage(newImage, null, null);
            try {
                writer.write(null, tmpImage, iwp);
            } catch (IOException e) {
                return new ServiceResult(ServiceResult.STATUS.FAIL_HARD, "Exception while creating jpeg content", e);
            }
        } else {
            return new ServiceResult(ServiceResult.STATUS.FAIL_HARD, "Couldn't find a jpeg encoder!");
        }

        return new ServiceResult(ServiceResult.STATUS.OK, "Created avatar successfully", output.toByteArray());

So we had to use: BufferedImage, ImageIO, ByteArrayInputStream, ByteArrayOutputStream, Graphics2D (inside the getScaledInstance method), ImageWriter, ImageWriteParam, MemoryCacheImageOutputStream and IIOImage. (Not counting any java.lang, java.util or exception stuff.)

Why can't ImageIO read a byte array? (I can almost forgive the design decision to work mainly with streams.) Why does an ImageWriter need to write to a MemoryCacheImageOutputStream (an ordinary OutputStream won't do)? What's with ImageWriter#setOutput taking an Object? Do we really expect to have multiple image writers for a given MIME type such that we need an iterator? Why can't an ImageWriter write a BufferedImage? Why is there a getDefaultWriteParam, if that's the only WriteParam there is to get?

javax.imageio has got to be the absolutely most retarded API I have seen to date, with the closest runner-up being jTidy.

Labels: api, javax.imageio, overengineering, software, stupid

In any relational setting, it is wise to avoid this situation if you can:

TABLE_ONE
---------
object_id serial primary key not null
foo_id integer not null references TABLE_TWO(object_id)
bar_id integer not null references TABLE_TWO(object_id)

TABLE_TWO
---------
object_id serial primary key not null
some_field varchar(80) not null

There is no way to perform a single join from table_one to table_two to get a complete set of information, because of the references to multiple different rows in table_two. It's probably better to reorganize the relationship if you can.

The exception is if you don't necessarily need both fields. For example, if you can do without knowing anything about bar_id in most cases, you could always lazy load that field. That is, if lazy loading for 1:1 fields works in your JPA provider.

It still doesn't work with Toplink, and these bugs are STILL open:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=2546
https://glassfish.dev.java.net/issues/show_bug.cgi?id=2554

The consequence is that I cannot static weave my model, so my only choice for now is to mark the fields as @Transient and ignore them for now. (I have a feature that tracks the changes made to every entry in a table with a changer that references an Account... for now, I just won't track the changer until I have time to come up with a better way of doing it or the glassfish folks fix their shit.)

Labels: glassfish, JPA, software, toplink

When Toplink says "Trying to get value for instance variable [foo] of type [bar] from the object The specified object is not an instance of the class or interface declaring the underlying field" what it means is you specified the wrong mappedBy class in your many:many relationship, either explicitly or by inference from your generic type.

But it's much, much clearer to give their error message, isn't it.

Labels: software, toplink

I've used Lucene for a really long time now. Practically since its Java-based inception, morons.org has searched articles using Lucene. My initial effort to add Lucene support to Tally-Ho failed miserably due to some unexpected behaviour from Lucene that I resolved by zeroing in on the problem with more unit tests.

Since all Article manipulation for the site goes through the ArticleService, that makes it a very handy place to automatically index Articles as they are created and updated and as they go through the lifecycle (from submitted, to approved, accepted, and so-on).

Adding an article to the index is trivial. We create a Directory object that points to where we'd like Lucene to write its files, create and IndexWriter to write them there, create several Field objects to represent the names of fields and their contents that we'd like to search on, add those fields to a Document, and add the Document to the IndexWriter. We can then query on any combination of these fields. Great.

I had a problem come when it becomes necessary to *change* an article. Lucene does this via the updateDocument method, or you can call deleteDocument and addDocument yourself. The advantage to updateDocument is that it's atomic. But for me, neither strategy worked at first.

First of all, even though Lucene said it was performing a delete based on a Term (which in our case contains the primary key of the Article), it didn't actually do it unless the Field referenced by the Term was stored as Field.Index.UN_TOKENIZED. If I stored it TOKENIZED, Lucene claims to be deleting, but the deleted Document would still show up in search queries.

Secondly, when I tried to delete a document, it looked like I could never add another document with the same fields ever again.

The first case turned out to be caused by using the StopAnalyzer to tokenize the input. When you index a term as UN_TOKENIZED, Lucene skips the Analyzer when storing the term to the index. The StopAnalyzer tokenizes only sequences of letters. Numbers are ignored. This differs from the StandardAnalyzer, which also uses stop words, which tokenizes letters as well as numbers. Since we delete based on the id term, which is numeric, Lucene was never finding the document it was supposed to delete, as the term had been tokenized into nothing by the StopAnalyzer... so the old document was not found and consequently not deleted.

The second case turned out to be caused by a fault in my unit test. I was doing an assertion that an updated article was in the database by doing a search on its id field. But I didn't assert that it was there before the update by searching the same way. For this reason it appeared that the article disappeared from the database and stayed away, because other unit tests worked (but those other tests also searched on other terms). Once I realized that the search on id was always failing, everything began to fall in place. Note also that you specify a tokenizer on search queries as well, so even when I stored the id term as UN_TOKENIZED, the StopAnalizer applied to the query would effectively eliminate the value of the search term (such that it could only ever find documents that had an empty id).

Lucene 2.2 has a great feature that lets you find documents in its index that are similar to a given document. The given document doesn't even need to be in the index, but it's very easy to do if it is. Since Tally-Ho automatically includes articles in the index as soon as they are created, this case applies. The code is very simple:

    directory = FSDirectory.getDirectory(indexDirectory);
    reader = IndexReader.open(directory);
    searcher = new IndexSearcher(directory);

    MoreLikeThis mlt = new MoreLikeThis(reader);
    mlt.setFieldNames(new String[]{"combined"});
            
    TermQuery findArticle = new TermQuery(new Term("id", String.valueOf(id)));
           
    Hits hits  = searcher.search(findArticle);
    int luceneDocumentId = hits.id(0);

    org.apache.lucene.search.Query query = mlt.like(luceneDocumentId);      
    hits = searcher.search(query);

I probably should be checking the first call to searcher.search to make sure the article for comparison is found (it should always be found, but sometime strange things happen).

Labels: Lucene, software

Last night, the first early release of Tally-Ho hit morons.org. As one might expect, a few small problems turned up at the last minute, and most of these have been worked through. One of them was a strange Internal Error message, but there was no exception in my log file. It was getting late, and I was getting tired, so I fired off a message to the Wicket-Users list to see if anybody had advice.

The problem turned out to be that although my development container is Tomcat, which uses log4j for its logging and consequently configures a log4j root logger and appender, my deployment container is Resin Opensource, which does not.

The answer was to create a log4j.properties file in src/main/resources (so it is automatically included in the .war by Maven 2) with this in it:

log4j.rootLogger=WARN, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n

log4j.category.wicket=INFO
log4j.category.resource=INFO
log4j.category.wicket.protocol.http.RequestLogger=INFO
log4j.category.wicket.protocol.http.WicketServlet=INFO

Now my logging goes to stdout and is happily recorded by Resin.

Now if I could just get somewhere with WICKET-506.

Labels: log4j, resin, software, wicket

Working with Toplink Essentials via JPAQL is quite a bit different than working with the commercial version of Toplink using its Expression class. With the commercial Toplink software, you generally get associated 1:1 objects fetched for you (ie eagerly rather than lazily) when you issue a query. In JPAQL, you get exactly what you ask for, which means if you want to get the associated objects in one query, you must use the JPAQL JOIN FETCH operator.

In my case, I needed LEFT JOIN FETCH, which works like an outer (left) join. My query ends up looking like this:

Select x from Article x LEFT JOIN FETCH x.messageBoardRoot where x.createDate > ?1 and not(x.status = ?2) order by x.createDate desc

Sometimes Articles won't have a message board associated with them, though usually they will. For example, there's no point in putting a message board on an article that is in a Pending state, since nobody can see it anyway.

Without the LEFT JOIN FETCH, Toplink issues one query to get the Articles, and then one query for every associated object. So if you're requesting 10 articles, you're going to get 11 queries. With the LEFT JOIN FETCH, it is supposed to consolidate everything into just enough queries to get what you ask for, and in fact the query it issues is reasonable:

SELECT t0.object_id, t0.thumbs_down, t0.spam_abuse, t0.MAILED, t0.change_summary, t0.VISIBLE, t0.ADJECTIVE, t0.BODY, t0.md5, t0.VIEWS, t0.fuzzy_md5_1, t0.VERSION, t0.fuzzy_md5_2, t0.thumbs_up, t0.create_date, t0.TITLE, t0.SUMMARY, t0.STATUS, t0.section, t0.changer, t0.creator, t1.object_id, t1.post_count, t1.last_post, t1.posting_permitted, t1.source_id, t1.post_count_24hr FROM ARTICLE t0 LEFT OUTER JOIN article_message_root t1 ON (t1.source_id = t0.object_id) WHERE ((t0.create_date > ?) AND NOT ((t0.STATUS = ?))) ORDER BY t0.create_date DESC
bind => [2007-04-14 14:46:15.593, P]


Unfortunately, Toplink's behaviour upon handling the results of running this query is NOT reasonable:


java.lang.NullPointerException
at oracle.toplink.essentials.mappings.ForeignReferenceMapping.buildClone(ForeignReferenceMapping.java:122)
at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.populateAttributesForClone(ObjectBuilder.java:2136)
at oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.populateAndRegisterObject(UnitOfWorkImpl.java:2836)


I've filed this one as https://glassfish.dev.java.net/issues/show_bug.cgi?id=2881. If past behaviour is any indication, the Glassfish people will change the priority on the bug to a P4 and decide not to fix it until we're all very old, despite it being a significant breakage of the API. They even pull that crap when the one-liner fix is already given in the bug report, and it would take longer to reset the priority and update the bug than it would to actually fix the damn problem.

Labels: glassfish, software, toplink, toplink essentials

On the off chance that other people find this helpful, here's how I set up Tally-Ho to work in Eclipse with the Sysdeo Tomcat plugin and Maven 2.

First, obviously, you need to install your prerequisites. Download and install Tomcat. Install the Sysdeo Tomcat plugin. You also want the Maven 2 Eclipse plugin. Installation of these is outside the scope of this post. It is also outside the scope of this post to explain Maven, Tomcat, Servlets and so-on. Use Google.

Next, bootstrap your project. I found it easiest to change into my Eclipse workspace directory, use mvn to create my archetype for my project, and then run mvn eclipse:eclipse inside the project directory it created. Then go to File | Import in Eclipse and import the project. Finally, enable the Maven 2 plugin for your imported project from the project's context menu, Maven 2 | Enable.

I found that the only way to make working with Maven bearable was to follow its default layout. This means that web.xml is going in src/main/webapp/WEB-INF and that all of the library dependencies are defined in pom.xml and all of the libraries will download into the Maven 2 Dependencies collection the first time you run mvn on the project.

Edit pom.xml and make sure you have your dependencies defined how you want them and that your project name and version are what you'd like.

Now is a good time to run a build of the project just to set up all of the remaining directories, like target. I did this by configuring an m2 build from the External Tools menu using my project's location as the Base directory with the goal "install".

Now set up the Tomcat plugin. From Window | Preferences | Tomcat, configure the appropriate Tomcat version and Tomcat home. From the context menu of your project, select Properties and then Tomcat. Check "is a Tomcat project." Set the context name to "/" and check "Can update context definition" and "Mark this context as reloadable." Set the subdirectory to "/target/your_project_name-your.project.version". The project name and version here must match what you've defined in pom.xml.

Now set up Eclipse to build directly to the Maven output directory. This allows you to avoid running a mvn build every time you make a change to a class or resource file. You will still need to run a mvn build if you add or change dependencies or if you change web.xml, however. From the project's properties context menu, choose Java Build Path and set the default output folder to target/your_project_name-your.project.version/WEB-INF/classes.

Lastly, from your project's context menu, choose Tomcat Project | Update Context Definition.

So to recap, here are the steps:

1. Download and install Maven 2, Eclipse, Tomcat, the Maven 2 plugin for Eclipse and the Tomcat plugin for Eclipse.
2. Create a project using Maven. Consult Maven's documentation for more detail or use an existing project that already has a pom.xml.
3. Run mvn eclipse:eclipse to generate Eclipse's metadata files from the project's pom.xml.
4. Import the project into your Eclipse workspace.
5. Edit pom.xml to define the version number and your dependencies.
6. Put web.xml in src/main/webapp/WEB-INF
7. Create a m2 external build and run it to create your target directory structure.
8. Configure the Tomcat plugin to look in Maven's target directory for your webapp's directory structure.
9. Configure Eclipse to build directly to Maven's target directory structure.

To make the setup play nicely with Wicket, you only need to define Wicket as a dependency in pom.xml (step 5). This is a matter of adding:

    <dependency>
      <groupId>wicket</groupId>
      <artifactId>wicket</artifactId>
      <version>1.2.5</version>
    </dependency>

Labels: eclipse, maven, software, tomcat, wicket

Maven is, hands-down, the absolute worst piece of crapware I have had the misfortune of using in the last 4 years. This collection incidentally includes all versions of Microsoft Internet Explorer, including IE7, which only crashed every time I started it for a week due to an incompatibility with the Google Toolbar. It's worse than Norton Antivirus. It's worse than Microsoft Outlook. It is a complete waste of bits.

The terrible, tragic thing about Maven is that there's a kernel of a really good idea behind it. Building stuff, handling dependencies, running tests, producing reports. Great! Fantastic! If only it weren't to software development what Mr Garrison's "It" was to transit.

First, those who get excited about XML configuration need to die in a fire. A sewage fire. You know what? XML blows. The XML fad is over. Stop using XML for all kinds of garbage that it was never intended for. What the hell is wrong with you? People do not like writing this crap, and they like reading it even less. I don't give a damn that it makes your crapware XML/Object mapping tool spit out nice little objects that are easy for YOU to deal with when handling configuration. It's not about YOU if you want people to use your diarrhea soup.

Next, why does everything in this obtuse XML configuration HELL have to be

nested

and nested

and nested

and nested

and nested?

Seriously, if I need to get a file included in my output, why does it have to be in a structure 4 levels deep? And why do some of the bottom-layer elements allow file globbing? Don't you realize that if you can handle file globbing, you could just one ONE DAMN TAG ONE LAYER DEEP and be done with it? Die!

Want to see the results of your unit tests? Go look in a bunch of individual files! Because the build can only scream FAILURE!!! at you (just like that) and doesn't bother to tell you which assertion failed at which line in which class and method.

What a horrible pile of dung. Maven has been around for well over 4 years and in that time the only thing that appears to have improved is its startup time.

I don't know why anyone puts up with this crap.

Labels: crapware, dumbassware, maven, maven 2.0.6, shitware, software, turdware

In my previous post, Making MD5 Fuzzy one thing I noted was a problem at the time was the capability of being off-by-one completely changing the outcome, such that certain small changes in the right places could cause the fuzzy md5 to no longer match up.

I struggled with the solution to this for quite a while, and then it dawned on me: I was looking at the problem the wrong way. It's fine if an off-by-one changes the outcome, if we're prepared to handle it.

The answer is to produce two checksums, not one! In the first, we begin at the beginning, and skip the last n/2 characters for an averaging length of n. In the second, we begin n/2 characters from the beginning and work all the way to the end.

Then instead of comparing one sum to another sum, we perform four comparisons:

object1.sum1 == object2.sum1
object1.sum2 == object2.sum1
object1.sum1 == object2.sum2
object1.sum2 == object2.sum2

If any of these statements returns true, we consider the objects to be "similar".

Here's the code. I've also simplified the way the distance between words is caculated and left room for non-english words to be handled at some point in the future (ie, there's no longer any special significance given to vowels).

package net.spatula.tally_ho.utils;


public class FuzzySum {
    
    private static final int SLOP = 3;

    private static FuzzySum instance;
    
    private static final int SAMPLE_SIZE = 10;
    
    private FuzzySum() {
        
    }
    
    public static synchronized FuzzySum getInstance() {
        if (instance == null) {
            instance = new FuzzySum();
        }
        return instance;
    }
    
    public String[] getSums(String text) {
        text = TextUtils.stripTags(text).toLowerCase().replaceAll("[^\\w\\s]", "").trim();
        
        if (text.length() < SAMPLE_SIZE * 1.5) {
            String md5 = TextUtils.md5(text);
            return new String[] { md5, md5 };
        }

        String[] words = text.split("(?s)\\s+");
        
        String md5_1 = calculateFuzzyMd5(words, 0, words.length - 1 - (SAMPLE_SIZE / 2));
        String md5_2 = calculateFuzzyMd5(words, SAMPLE_SIZE / 2, words.length - 1);
        
        return new String[] {md5_1, md5_2};
    }
    
    private String calculateFuzzyMd5(String[] input, int startIndex, int endIndex) {
        StringBuilder builder = new StringBuilder();
         
        int distanceSum = 0;
        for (int i = startIndex + 1; i<= endIndex; i++) {
            String thisWord = input[i];
            String lastWord = input[i - 1];
            
            distanceSum += calculateDistance(thisWord, lastWord);
            if (i % SAMPLE_SIZE == 0) {
                if (builder.length() > 0) {
                    builder.append("\n");
                }
                builder.append(distanceSum / SAMPLE_SIZE);
                distanceSum = 0;
            }
        }
        
        if (distanceSum != 0) {
            builder.append("\n");
            builder.append(distanceSum / (endIndex + 1 - startIndex % SAMPLE_SIZE));
        }
        
        return TextUtils.md5(builder.toString());
    }

    private int calculateDistance(String word1, String word2){
        int word1Sum = calculateWordSum(word1);
        int word2Sum = calculateWordSum(word2);
        return Math.abs(word1Sum - word2Sum) / SLOP;
    }
    
    private int calculateWordSum(String word) {
        
        if (word.length() == 1) {
            return (int)(word.charAt(0)) & 0xffff;
        }
        
        int wordSum = 0;
        for (int i = 1; i < word.length(); i++) {
            int prevChar = (int)(word.charAt(i-1)) & 0xffff;
            int thisChar = (int)(word.charAt(i)) & 0xffff;
            wordSum += Math.abs(thisChar - prevChar);
        }
        
        return SLOP * wordSum / word.length();
    }
    
}

As you can see, this code has been committed as part of the Tally-Ho project, https://tally-ho.dev.java.net/

Labels: checksum, fuzzy, java, md5, software, tally-ho

I've been tweaking the network stack on my FreeBSD host for many moons now, trying to get everything "just right" for optimal network performance. Many of the defaults are a bit pessimistic, assuming a network that experiences a good deal of packet loss and transmits data over a twisted pair of doorbell wire from a PDP-11 in the damp basement of some godforsaken computer lab to a VAX machine surrounded by nerds in a Physics building 2500 miles away. Sure, that may have been a common scenario back in 1982 or whatever, but these days most networks are much more reliable, delivering far more porn at faster rates than ever before.

My tuning is focused mainly on high-performance web serving on a host that also makes connections via localhost for database access and to front-end Resin OpenSource (a Java Servlet container) with Apache. The host has plenty of RAM and CPU available. These tunings may not be appropriate for all situations, so use your head.

First, enable polling on your interface. While you're at it, compile in zero copy sockets and the http accept filter. In fact, just add this crap to your kernel config if it isn't already there:

options         HZ=1000
options         DEVICE_POLLING
options         ACCEPT_FILTER_HTTP
options         ZERO_COPY_SOCKETS

To make sure your device actually polls, edit /etc/rc.conf and add "polling" at the end of ifconfig_{yourInterface}; eg:

ifconfig_bge0="inet 192.168.1.234 netmask 255.255.255.0 polling"

You probably also will want to tune polling a bit with sysctl:

kern.polling.burst_max=1000
kern.polling.idle_poll=0
kern.polling.each_burst=50

Idle poll tends to keep your CPU busy 100% of the time. For best results, keep kern.polling.each_burst <= the value of net.inet.ip.intr_queue_maxlen, normally 50.

Now sit down and think about what bandwidth and latency you want to plan for. This kinda depends a bit on who typically accesses your host. Are they coming from broadband connections mainly? About how far away are they usually? You can get some assistance with this determination by doing a sysctl net.inet.tcp.hostcache.list. Starting in FreeBSD 5.3, hostcache began keeping track of the usual RTT and Bandwidth available for all of the IP addresses it heard from in the last hour (to a limit of course, which is tuneable... more on that later).

We would be interested in the RTT and BANDWIDTH columns, if the number in the BANDWIDTH column had any bearing on reality whatsoever. Since my hostcache routinely suggests that there's more bandwidth available to a remote host than is actually possible given my machine's uplink, it isn't really reasonable to use this number. You can, however, average the RTT to get a rough idea of the average RTT to the current set of users in your hostcache. You can also get a rough idea of the average TCP congestion window size (CWND). Note that this will be bounded by what you have set for net.inet.tcp.sendspace and net.inet.tcp.recvspace. To make sure you're not the bottleneck, you could try setting these two to an unreasonably high number, like 373760, for an hour to collect the data. You can do a sysctl -w net.inet.tcp.hostcache.purge=1 to clear the old hostcache data if you decide to do this.

Here's a dumb little perl script for calculating your average and median RTT, CWND and Max CWND:

open(IN, "/sbin/sysctl net.inet.tcp.hostcache.list |");

while (<IN>) {
    @columns = split(/\s+/, $_);
    next if ($columns[0] eq '127.0.0.1');
    next if ($columns[0] eq 'IP');

    next if ($columns[9] < 2 || $columns[10] < 2);  # skip if few hits and few updates

    push(@rtts, int($columns[3]));
    push(@cwnds, $columns[6]);

    $rttSum += int($columns[3]);
    $cwndSum += $columns[6];
    $cwndMax = $columns[6] if $columns[6] > $cwndMax;

    $entries++;
}

print "Average RTT = " . int($rttSum / $entries) . "\n";
print "Average CWND = " . int($cwndSum / $entries) . "\n";
print "Max CWND = $cwndMax \n";

@rtts = sort { $a <=> $b } @rtts;
@cwnds = sort { $a <=> $b } @cwnds;

print "Median RTT = " . getMedian(@rtts) . "\n";
print "Median CWND = " . getMedian(@cwnds) . "\n";

sub getMedian {
    my @list = @_;
    if (@list % 2 == 1) {
       return $list[@list / 2];
    }  else {
       return ($list[@list / 2 - 1] + $list [@list / 2]) / 2;
    }
}

It's up to you how to use the information the script provides. For me, the most interesting thing to note is that my median RTT is around 100ms and that my max CWND looks to be 122640, at least for the hosts currently in my host cache.

I want to optimize my site for the best possible experience for high speed broadband users.. My home broadband connection is 8Mbps, but it can burst up to 12Mbps for a short time. If we split the difference, that's 10Mbps. This is probably a bit optimistic for most home broadband users. Also note that there's no point in optimizing for more bandwidth than your host actually HAS. In my case, my uplink is 10Mbps, so there's no point in trying to optimize for a 45Mbps connection.

In all probability I won't be able to actually push 10Mbps because I share that connection with some other folks. So let's be just a little bit pessimistic and optimize for 6Mbps. Many home cable services provide between 4 and 8 Mbps downstream, so 6Mbps is a nice "middle of the road" approximation.

To calculate the bandwidth delay product, we take the speed in kbps and multiply it by the latency in ms. In this case, that is 6144 * 100 or 614400. To get the number of bytes for a congestion window that many bits wide, divide by 8. This gives us 76800, the number of bytes we can expect to send before receiving an acknowledgment for the first packet. That's higher than both the median and average congestion window sizes for the folks currently in my hostcache, and about 2/3 of the max. Remember this number.

The next thing to look at is the net.inet.tcp.mssdflt. This is the maximum segment size used when no better information is available. Normally this is set pessimistically low. These days, most networks are capable of moving packets of 1500 bytes, so let's set this to 1460 (1500 minus 40 bytes for headers). sysctl -w net.inet.tcp.mssdflt=1460. This could make the first few packets fail to transmit should MSS negotiation at the start of a TCP connection not happen for some reason or if a network cannot support a packet of that size. I suspect this is quite rare. And we're trying to optimize for the most common case, not the most pessimistic case.

Now we want to make sure that our congestion window size is an even multiple of the default MSS. In fact it isn't. 76800 / 1460 is 52.6027. We round up to the nearest even number - 54 - and multiply by the MSS to get 78840. (I'm not sure why, but many sites recommend that one use an even multiple of MSS.) I round up rather than down because I'm optimistic that I will not have lost that first packet in transit. Rounding down might mean stopping and waiting for the first acknowledgment rather than continuing with one (or two) more packets while awaiting that first reply.

Now that we have our desired window size, let's set it:

sysctl -w net.inet.tcp.recvspace=78840
sysctl -w net.inet.tcp.sendspace=78840

Since we're being optimistic, let's assume that the very first time we talk to our peer, we can completely fill up the window with data. Recall that we can fit 54 packets into 78840 bytes, so we can do this:

net.inet.tcp.slowstart_flightsize=54

Granted, immediately jamming the pipe with packets might be considered antisocial by cranky network administrators who don't like to see retransmissions in the event of an error, but more often than not, these packets will go through without error. I never minded being antisocial. If it really bothers you, cut this number in half. Note that having RFC3390 enabled (as it is by default) and functioning on a connection means that this value isn't used on new connections.

Next, turn on TCP delayed ACK and double the delayed ACK time. This makes it more likely that the first response packet will be able to have the first ACK piggybacked onto it, without overdoing the delay:

net.inet.tcp.delayed_ack=1
net.inet.tcp.delacktime=100

Now enable TCP inflight. The manual page recommends using an inflight.min of 6144:

net.inet.tcp.inflight.enable=1
net.inet.tcp.inflight.min=6144

Finally some tuning for the loopback. Hosts (like mine) that do a lot of connections to localhost may benefit from these. First I modify the ifconfig entry for lo0 to include "mtu 8232" (programs commonly use 8192-byte buffers for communicating across localhost, add 40 bytes for header). Using a similar strategy to what we did above, I tune the following in sysctl.conf:

net.local.stream.sendspace=82320
net.local.stream.recvspace=82320
net.inet.tcp.local_slowstart_flightsize=10
net.inet.tcp.nolocaltimewait=1

The 10 is arbitrary, but it's also the smallest even multiple that makes the loopback window equal or greater in size than the LAN interface window. There might be some small advantage in doing this if there are programs which may copy the incoming request to some other program via the loopback.

Adding net.inet.tcp.nolocaltimewait frees up resources more quickly for connections on the loopback.

Finally, make the host cache last a bit longer:

net.inet.tcp.hostcache.expire=3900

The reason I do this is that some hosts may connect once an hour automatically. Increasing the time slightly increases the chances that such hosts would be able to take advantage of the hostcache. If you like, you can also increase the size of this hash to allow for more entries. I do this for the TCP TCB hash as well. These have to be changed in /boot/loader.conf as they can't be changed once the kernel is running:

net.inet.tcp.tcbhashsize="4096"
net.inet.tcp.hostcache.hashsize="1024"

So that's it. If these settings are applicable to you, you can just add this to /etc/sysctl.conf:

net.local.stream.sendspace=82320
net.local.stream.recvspace=82320
net.inet.tcp.local_slowstart_flightsize=10
net.inet.tcp.nolocaltimewait=1

net.inet.tcp.delayed_ack=1
net.inet.tcp.delacktime=100

net.inet.tcp.mssdflt=1460
net.inet.tcp.sendspace=78840
net.inet.tcp.recvspace=78840
net.inet.tcp.slowstart_flightsize=54

net.inet.tcp.inflight.enable=1
net.inet.tcp.inflight.min=6144

kern.polling.burst_max=1000
kern.polling.idle_poll=0
kern.polling.each_burst=50

net.inet.tcp.hostcache.expire=3900

And don't forget to edit /etc/rc.conf and add "mtu 8232" for your ifconfig_lo0 line and "polling" for your LAN adaptor.

Labels: software

Well, somewhat impossible depending on what you need to do.

If you need dynamic weaving of your classes at runtime, give up. It won't work, not with a J2SE servlet container.

The problem seems to be that the javaagent (for both OpenJPA and for Toplink Essentials) is incapable of coping with classes that are loaded by a different classloader. If you manage to get your javaagent JVM argument to work (which will require annoying experiments with quotation marks and adding stuff to your classpath), then all of your Entity classes will fail to load because the agent can't find them. They don't exist as far is it is concerned, because they are loaded by a different classloader.

So if you want to use lazy loading of 1:1 fields with a J2SE container, you must either do static weaving or give up hope. Your other option is to use a full-blown J2EE container in all its bloated glory.

Screw this. I'm going to bed.

Labels: j2ee, javaagent, JPA, servlet, software, weaving

Since Toplink isn't up to the task, tonight I decided to get all my tests working with OpenJPA. Normally this should be simply a matter of plugging in a different implementation and everything would work swimmingly. Ah, but we all know it's never that easy.

The trouble is that the JPA spec, being an early revision, leaves a lot of things up to the implementer. This includes all cache control. So it is actually fairly easy to find yourself relying upon the caching semantics of a particular provider, and you may find that unit tests that work correctly with one provider begin to fail when you plug in another one.

A classic example of this is the simple 1:Many relationship. Say you have two objects called Root and Message. One Root contains many Messages. Now you want to perform these operations:

1. Create Root


2. Create Message that refers to the Root


3. Read Root


4. Get messages from Root

Now you may already see the obvious problem with this chain of events, but pretend for a moment that you don't.

In Toplink, this works. The "Get messages from root" step initiates a query to the database to get all of the Message objects associated to the root (lazy loading).

In OpenJPA, this fails. OpenJPA has been tracking the Root object in its cache, and it already knows that the Root object has no message objects. So it performs no query, and the list of Messages is empty.

Of course had I adhered to a best practice of dealing with ORM, this would never have happened. When there is a 1:Many relationship, one should always perform operations on the owning side of the relationship. That means you don't set the root ID from the Message; instead, you add the Message to the list of children that belong to the Root.

Complicating matters, that strategy tends to work in Toplink, but OpenJPA will not set up the owned side of the relationship (ie, the root reference). So you actually have to do both: set the root reference in the Message, and add the Message to the Root's children list. JPA implementations will automatically persist the owned side of the relationship at least, if you set the right value(s) for cascade in the @OneToMany annotation (to CascadeType.PERSIST or CascadeType.ALL). (OpenJPA has its own annotation, @InverseLogical, to indicate that a field should be maintained bidirectionally.) Otherwise you have to explicitly persist both objects.

The safest thing is to maintain both sides of the relationship, though this isn't a particularly intuitive thing to have to do.

And what night would be complete without opening a bug report? Tonight it's OPENJPA-164. I kept getting an exception from OpenJPA tonight that was a rethrowing of a PostgreSQL exception that "f" was not valid data for the BigDecimal type when reading in an instance of class Article. And that was all the information I had to go on. It later turned out that the culprit was the way I had previously mapped ArticleMessageRoot, which had a field called "posting_permitted" mapped as a long instead of a boolean. This worked with Toplink, because Toplink read the field as a long, and PostgreSQL evidently had a conversion that was workable for a boolean to a long, but not a BigDecimal. The good news is that the problem has been caught- it really should have been mapped as a boolean. The bad news is that OpenJPA gave me absolutely nothing to go on for a fairly commonplace mapping error, even with logging cranked all the way up to TRACE.

All the service unit tests are passing now, so it's progress.

Labels: software

I really really like the Toplink commercial product. I think it's tremendously powerful, flexible, well-supported, and rock-solid. It does exactly what it is supposed to do, every time.

I wish I could say the same for Toplink Essentials. It feels like the open source branching was done as a rush-job, leaving large portions of the necessary code and tooling undone. My simple test case to take advantage of "weaving" (bytecode instrumentation) fails. My attempt to make the instrumentation work using static weaving (post-compile and pre-run) also fails, but in a different way. These are not elaborate test cases. My code isn't terribly complex yet, and my schema is fairly straightforward. Yet Toplink falls down and goes splat. Some of their tools can't even handle spaces in the pathname.

For now I'll keep using OpenJPA until the Glassfish people can make Toplink Essentials stop sucking so damn much. OpenJPA is a bit more pedantic anyway, and that's a good thing. My code should end up tighter as a result.

Much of this JPA stuff just isn't ready for the real world yet, though it's been a final spec since last May and a proposed final draft since December of 2005. They don't even have support for @OneToOne in Eclipse WTP 2.0 yet, and every edit to the persistence.xml file brings up some lame dialog box whining about a ConcurrentModificationException. Granted, it's just a milestone build. But this is the type of frustration one encounters when trying to use JPA.

And no, I'm not going to use sucky Hibernate and their dozens of goddamn dependencies, lack of integrated object caching, and mental-defective nerd community.

Labels: JPA, open JPA, software, toplink

https://issues.apache.org/jira/browse/OPENJPA-162

I just can't win tonight.

UPDATE: it turns out that @OrderBy with no options means to order by the identity value of the related object, and this does work correctly in OpenJPA.

Labels: open JPA, shitware, software

I'm reminded tonight of why I hate things that happen by "magic" in code. They are great when they're working correctly, but an impossible pain in the ass to debug when things go wrong. This is the primary reason why I dislike AOP; how do you debug something that isn't in your code? Toplink's "weaving" is very AOP-ish; it's effectively intercepting Java bytecode for setting the values for fields.

Or rather, it's not doing it very effectively, if I may overload usage of the word, because it's totally broken in a very simple and stupid way which even a primitive test case should reveal. It fails to correctly instrument put field operation for a field that has an associated Entity. In this case, that means that when I try to set my MessageRoot on my Message, I get this:

java.lang.NoSuchMethodError:
net.spatula.tally_ho.model.Message._toplink_setroot(Lnet/spatula/tally_ho/model/MessageRoot;)V
 at net.spatula.tally_ho.model.Message.setRoot(Message.java:169)

As you can probably imagine, I did not write a method called _toplink_setroot. The problem is, neither did Toplink, even though it should have.

Since the Toplink instrumentation code doesn't appear to do any logging of any kind, it is utterly impossible to debug this problem. I have exactly nothing at all to go on, other than the name of a method that doesn't exist, which Toplink should have supplied if it wanted to call it.

Much as I wanted to use Toplink for this project, this problem is a complete deal-breaker. What's particularly startling to me is that such a simple test case can fail without anyone noticing.

Next I guess I'll try BEA's Open JPA (Kodo) and see if they can manage to do proper lazy loading of a 1:1 relationship AND let me create that relationship too... since that's too much to ask of Toplink.

Labels: AOP, kodo, NoSuchMethodError, open JPA, software, toplink, weaving

The trouble with recursive tree structures is that databases tend to give poor performance implementing them. Further, JPA doesn't really natively do trees (so far as I can tell). Fortunately, morons.org solved this problem a very long time ago. My solution was to associate each collection of messages with a single "root" (though it isn't a root in the true sense of the word) and with each other. If you happened to have "connect by" syntax with your database, you could take advantage of it by virtue of the parent_id column. If you didn't, you could always do a fairly quick pass through the full collection of messages you got off the single root. The latter is exactly what Tally Ho will be doing with JPA.

Key to making this work is denoting the 1:1 relationship to the parent as fetch=FetchType.LAZY.

When I first tried this with Glassfish Persistence (aka Toplink Essentials) it seemed to be ignoring my request to use FetchType.LAZY. As soon as the first object loaded, it loaded its parent. And the parent of that object. And so on, recursively. Toplink would issue one SELECT statement for every single object in the database for the "root" in question. Obviously the performance of this strategy really sucked.

It turns out that for proper behaviour you need to turn on something the Toplink folks call "weaving", which I suspect is actually just a fancy Oracle term for Java bytecode instrumentation. To do this, you add the property toplink.weaving to persistence.xml and set its value to true. You must also add the JVM argument -javaagent:{path to libs}/toplink-essentials-agent.jar to your running configuration.

Once I did this, Toplink would happily wait to fetch any parent objects until they were asked for. And by virtue of all parent/child relationships being collected inside the same "root", when you first ask for a parent, it has already been loaded in memory, so an additional trip to the database is not required.

The downside is that the "children" List on the Message won't get populated, and if you let JPA handle the population, it will again make a lot of trips to the database to figure out which children belong to which parent... it has no way of knowing my rule that all of the relationships among the children are contained within the confines of that one root.

Fortunately for the needs of Tally Ho, this problem is easily mitigated. We simply mark the children List with @Transient to prevent JPA from trying to do anything with it at all. Then we pull this trick in MessageRoot:

    public List getMessages() {
        if (!messagesFixed) {
            fixMessages();
        }
        return messages;
    }
    
    private void fixMessages() {
        for (Message message : messages) {
            if (message.getParent() != null) {
                message.getParent().getChildren().add(message);
            }
        }
        messagesFixed = true;
    }

The only reason we can get away with this is that we never look at a collection of Messages without getting it from the MessageRoot. In the cases where we do look at a single Message, we don't look at its children.

It's a tiny bit hackish, but the performance improvement is tremendous, so it's worth it. Over the Internet, through my ssh tunnel to my server, 180 message objects were loaded and linked together correctly in about 650ms. I suspect this will improve substantially when the database is on the local host.

Labels: JPA, software, toplink, tree

Tonight came the next logical step in the progression for tally-ho: migrate the old messages database to the new layout standards (and UTF-8 of course) and map the tables to entities using Dali.

The data migration was fairly easy to do, especially now that I've done it before with accounts and articles. Unfortunately when I went to map the new tables to Java classes, Dali pulled its disappearing act again. I don't recall whether I mentioned this here before, but Dali 0.5 has a tendency to just disappear from Eclipse for no apparent reason. It fails to load, silently of course. Why log an error, when you can silently fail?

Luckly, though, Dali has been integrated into the Eclipse Web Tools Project (WTP) version 2.0 and there's a reasonably functional version of it in 2.0M5. Of course the problem with milestone builds is that they tend to lack any documentation whatsoever, so you're kind of on your own.

This became immediately frustrating. When I opened the JPA perspective for the first time, I couldn't add a connection. The drivers list was empty. It turns out you have to first populate a connection via WTP from the Window/Properties/Connectivity/Driver Definitions menu. It was easy to add a definition to connect to my PostgreSQL 8.1 database.

(At some point during this process, there was a magnitude 4.2 earthquake about 30 miles away on the Calaveras fault that shook the house and made a good loud noise. It was a little mild shake followed by one nice big jolt and then some rolling. All told it lasted maybe 5 seconds and was probably the most startling jolt I've felt since I've lived here, with the exception of the magnitude 6.5 earthquake near Paso Robles on December 22, 2003 that caused the lights in my office in San Jose to sway and nearly sent me scrambling under my desk. That time I had about 30 seconds of warning from my coworker in Salinas who I was talking to on the phone at the time.)

(As a side note, I got tired of poking holes in my host machine's firewall, adding the IP to PostgreSQL's allow pg_hba.conf and reloading PostgreSQL every time my dynamic IP changed at home, so I did the most reasonable thing: I added an SSH tunnel from my local machine's loopback to my remote machine's loopback on port 5432. Now I just connect to localhost for PostgreSQL, and it magically works. Using the loopback address on both ends mitigates potential security problems from creating such a tunnel.)

Finally I was able to add a database, but nothing worked with JPA. This is apparently because the necessary settings and natures defined for JPA with earlier versions of Dali are outdated. To make matters worse, they did not create a context or other menu (yet) to add JPA to the current project as they had with Dali 0.5. So I did what any impatient geek would do: I created a new JPA project, and then manually copied the missing natures in .project and missing files in .settings to my existing project. This seems to have done the trick.

The next hurdle is that the Java Persistence/Generate Entities... context menu is broken. First it was popping up a dialog box asking for the connection and schema to use. I could select my connection, but the drop down for schemas was empty. Entering the schema name manually was fruitless. The Next and Finish buttons remained disabled.

So then, having experienced weirdness related to missing configuration before, I went on the hunt again. This time I found the JPA settings in the project properties. Here I could select a connection, so I did. While this did change the behaviour of Generate Entities, it was for the worse: now the menu does nothing. Silent failure. I have reported the problem on the Eclipse site and asked for advice, but for now this has me stuck. It's time for bed anyway.

Labels: software

It seems like things are always easier to do when you've had a good night's rest. This morning I was able to finish adding AJAX to the message tree, with fallback and anchors, in about an hour.

Wicket provides a semi-magical component called AjaxFallbackLink that transparently allows your application to do AJAX if it is capable and falls back to a normal link if it is not. In typical Wicket fashion, the intrusiveness of AJAX is minimal, with much of the magic being handled behind the scenes. The one thing you have to watch out for is the case where the AjaxRequestTarget passed to onClick is null (which is what happens when you fell back).

Then the general paradigm is this:

    public void onClick(AjaxRequestTarget target) {
        doStuff();
        if (target != null) {
            doAjaxStuffToo();
        }
    }

The doAjaxStuffToo would typically involve doing some kind of replace operation in your Wicket page structure and then adding the modified part of the page structure to the target. In the event that AJAX isn't available, you don't have to worry about this, because the entire page is going to be re-rendered anyway.

Thanks to Igor Vaynberg for his help this morning in figuring that out.

Now one might naturally say, "but what about when AJAX isn't available? Won't the page refresh and put me way back up at the top of the page? That will be annoying!" Ordinarily it would, but Wicket makes this easy to deal with. You simply anchor your link to the Component you want (and so that Component needs to have setOutputMarkupId(true) called on it). It's as easy as calling setAnchor(parent) on your AjaxFallbackLink (or other type of Link).

Unfortunately, the setAnchor method defined by Link has the return type of void, so you can't be super lazy and define your Link and call setAnchor on it inside an add method on another component. This is easily worked around by just assigning your Link to something, calling setAnchor and then adding it. I have added WICKET-238 as a wishlist item for Wicket so that this operation is congruent with so many other Wicket operations that are "chainable".

Here's what a final bit of code looks like for handling navigation links. For the full source, look to the repository on tally-ho.dev.java.net.

        private void addNavLinks(final TreeNode node, Fragment nodeFragment) {

            if (!node.hasChildren()) {
                nodeFragment.add(new WebMarkupContainer("nav").setVisible(false));
                return;
            }

            Fragment navFragment = new Fragment("nav", "navigation");

            if (node.hasOpenKids()) {
                AjaxFallbackLink closeLink = new AjaxFallbackLink("closeLink") {

                    private static final long serialVersionUID = 1L;

                    public void onClick(AjaxRequestTarget target) {
                        node.closeKids();
                        if (target != null) {
                            parent.replace(new Entry("entry", root.getChildren(), parent));
                            target.addComponent(parent);
                        }
                    }

                };
                closeLink.setAnchor(parent);
                navFragment.add(closeLink);

                navFragment.add(new WebMarkupContainer("openLink").setVisible(false));
                navFragment.add(new WebMarkupContainer("openAllLink").setVisible(false));

            }

Labels: software

I decided after a bit of a struggle that it probably would be faster and easier just to go ahead and migrate the message board system for morons.org to use Wicket than it would be to continue to fight with trying to integrate two basically incompatible systems.

I like to deal with the most difficult problems first, which in this case meant the message viewing system. The morons.org message board has received much praise from users over the years for being fairly intuitive and easy to deal with. It has been evolving for many years based on reader feedback and on my needs and desires at the time. (It started its life as an unthreaded system where the messages where stored in flat text files!)

Wicket makes it fairly easy to get the recursive display part going, and my first step was to just display a full tree with no navigation. This I accomplished using Panels with ListViews. The ListView was used to display a set of Panels, and each Panel contained a span element where a nested item could be placed. Naturally this nested item can be another Panel of the same type, so you can build this huge recursive structure quite easily.

It was while doing this that I learned a very basic trick that comes in quite handy. In a previous entry I talked about how one can use Fragments to display one glob of HTML vs another depending on conditions in the program. If all you need to do is make some bit of HTML disappear, there's an even easier way. Just add a Label for that wicket:id, and call setVisible(false) on it. This is what I do to my nested item from above if there are no children to show. (Otherwise Wicket would surely complain that I had an element in my HTML that had no corresponding Component in my Java structure).

If you need to make an entire section of HTML disappear (ie, multiple tags/Components) it's still better to use a Fragment so you avoid calling setVisible many times as opposed to once.

Another thing I noticed is that I'm starting to see a relationship between nice looking code and relationships with the HTML components. I don't have enough information to really quantify this yet, but I'll give an example. I observed tonight that it was really nice to have one method that corresponded to one wicket:fragment. Rather than include a bunch of code inside my populateItem method, I could just say addNavLinks(node, nodeFragment); to deal with the corresponding navigation fragment. I probably could (and maybe should) do the same thing with creating the nodeFragment itself (it's a different bit of HTML based on whether the node is expanded).

Lastly on the topic of Fragments, note that you don't nest them in your markup. Even if you're only going to use a wicket:fragment from inside another fragment, the markup goes at the top level of the HTML document.

I got a good tip tonight in ##wicket on freenode from JonathanLocke: when you start getting into complicated containment with recursive structures, get a sheet of paper and draw out what's happening. As I learned in the last few days, it can get very complicated to mentally conceptualize what the hell is going on. I wish he and I had had that conversation before I got all the way through it :) but it is good advice nonetheless.

Another handy Wicket trick is to use setRenderBodyOnly. This comes in handy when you need to use some tag in your HTML as a placeholder for something else, but you don't really want the tag itself to show up in the final product. (Do you have tons of useless span elements all over the place?) You can't do this if you're using AJAX on the Component/tag in question however (for mostly-obvious reasons; the Javascript would not be able to find your tag if it wasn't rendered to the output, and so it would be impossible to act on it.)

Next I got the basic navigation working. This was easy to do using Wicket's Link class. It's really handy to be able to say things like this:

                navFragment.add(new Link("openAllLink") {
                    private static final long serialVersionUID = 1L;
                    public void onClick() {
                        node.openAllKids();
                    }
                });

... and have it just work. In my TreeNode structure, I have a method for recursively opening all of the children of the TreeNode, and in my ListView implementation, my populateItem method takes care of doing the right thing based on whether a TreeNode claims to be open, have children, etc. It all fits together so perfectly!

One thing that I did have to struggle with for a bit is the unfortunate fact that you can't really mix a ListView with a Fragment. In the case of the morons.org comments system, when a thread isn't expanded in the view, it has a different style than when it is. It would have been really nice to have been able to conditionally use a Fragment from within populateItem. The simple workaround for this is to wrap the contents in a div element, which turns out to be needed for the AJAX stuff anyway.

Speaking of AJAX, that was the next step. I just wanted to make something really simple happen using AJAX at first, so I figured I'd just make the whole subsection of the tree get replaced with a Label. This seems like it should be easy to do, but in actuality, it is difficult to replace something on the page with something that is not on the page. This is because Wicket waits to assign the markup IDs until page rendering time. The AJAX code will complain that the markup ID is empty on the new element, because the markup ID will not be assigned until later. This means you can't take something like a ListView and replace it with a Label

What you can do is have Wicket replace the contents of something that is contained on the page. In other words, you can get the Panel containing your ListView to replace it with a new ListView. The difference is subtle, but it's a question of AJAX replacing what exists inside the thing it is acting on versus replacing the thing it is acting on itself. The actual code looks a little bit like this:

            navFragment.add(new AjaxLink("ajaxLink") {

                public void onClick(AjaxRequestTarget target) {
                    node.openAllKids();
                    //myTree.replaceWith(new Entry("entry", root.getChildren()));
                    parent.replace(new Entry("entry", root.getChildren(), parent));
                    target.addComponent(parent);
                }
                
            });

Note that one uses parent.replace (where parent is the Panel), not someListView.replaceWith(someNewListView). The latter will appear to work once, but the second time it won't work anymore, because it has effectively been orphaned by not getting reattached to its parent container properly. The second time the link is used, Wicket would return an error about applying the method to an entity that hadn't been added to the parent in the latter case.

There's still a bit of work to be done, especially in tidying up the code and moving it from an experiment to playing an actual role on a proper article page, but much progress has been made.

In other news, java.net has finally approved our new home. I can't really say anything more about Google Code than Hani Suleiman hasn't already said. I briefly looked at Codehaus and decided that it's at least as retarded as Google Code, except that you can actually use Google Code; you can't use Codehaus until you prove that you're 31337 enough for them. You can't request a project with them without a login, and you can't have a login. I defy anyone to find somewhere on their site that one can create an account. Moreover, they seem to only want projects that are already complete. What the hell is the point of setting up a project and using a repository, etc., for projects that are already done? Go back to IRC, you boar-arse-licking nitwits. I'm too old for that shit...

...but I digress. Our new home is https://tally-ho.dev.java.net/

That's all for tonight. More to come.

Labels: software

IPv6 in FreeBSD 6.2-RELEASE is broken. It's also broken in 6.2-STABLE. It has been known to be broken since September. It was known to be broken in December. It was still broken when FreeBSD 6.2 was released on January 15. This isn't something that's just a tiny nuisance with a trivial workaround. This is a major piece of system functionality that is totally broken. With a stock release of FreeBSD 6.2 or 6.2-stable, you cannot use an IPv6 tunnel at all. Everything will report "no route to host".

Yet, despite this being a known problem, there's no mention of it at all in the release notes, and as of this writing there is still no mention of it in the errata.

This is the type of BS I'd expect from Linux or something. This is not the kind of low-quality, rush-job cockup that I'd expect out of the FreeBSD project. I'm sorely disappointed. I would never have built 6.2 and installed it had I known that IPv6 was in a nonworking state.

There's a fairly easy fix/workaround for those who want 6.2 (or have already gone to the trouble of installing it) and still need IPv6 to work:

First, go grab the diff between versions 1.48.2.15 and 1.48.2.16 of src/sys/netinet6/nd6.c. Save this in a file in your home directory called nd6.c.patch starting with the line that says "--- src/sys/netinet6/nd6.c 2006/10/07 18:31:27 1.48.2.15" (or if you save the whole thing, just delete the lines that precede this one).

Next, cd /usr/src/sys/netinet6 . Run patch -R < ~/nd6.c.patch. This has the effect of reverse patching the differences between those two versions of the file, backing out the last change that broke IPv6. This will also re-break the bug that the commit was trying to fix, but since that fix also completely broke IPv6, that's probably okay for now.

Finally, rebuild your kernel. Do a make clean first. I went as far as doing an rm -rf on the compile/KERNELNAME directory too, just to make sure everything got cleaned out.

Maybe the FreeBSD project can gets its head out of its collective ass and add documentation about this major malfunction to the errata, and then for an encore, perhaps someone can see fit to fix this longtime breakage. IMO, 6.2 should never have been released with major feature set breakage of this severity.

And why aren't there test suites catching this kind of crap?

This is so disheartening. If I wanted to suffer through major feature set breakage, I'd run Linux. I have used FreeBSD for a good 11 years because of its former dedication to stability and quality, not because I liked to spend my afternoons chasing problems.

Labels: software

Hash functions like MD5 are terribly handy when you want to verify that one file is exactly the same as another very quickly, but what if you want to just check that two files are pretty close to the same?

There are probably lots of approaches, but one that I found that works fairly well for catching small spelling changes and the additions and removals of single words is a rounded average distance algorithm. Each word is converted into X and Y coordinates: the sum of the vowels makes the X coordinate and the sum of the consonants makes the Y coordinate. These values are rounded to the nearest 17 (a completely arbitrary number). The cartesian distance from the previous word is calculated, and every 11 words (also completely arbitrary) the average cartesian distance between the words is calculated, rounded to the nearest 17 and added to a list of distances. Finally each distance is averaged with the following distance and used to update an MD5 digest.

One can probably play with the arbitrary constants in this one to make the algorithm more or less selective. 11 and 17 were the values I needed to make my test pass.

Here's the code. It's in an early stage of development, so it's fairly sloppy I'm afraid. I'm sure there's a way to revise things so that the list of sums isn't needed, but it's late and I need to eat and sleep.

package net.spatula.utils;

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.regex.Pattern;

public class FuzzySum {

    private static final int ROUND_TO = 17;
    private static final int AVG_SIZE = 11; // our amp goes to 11.

    public static String checkSum(String text) {
        text = text.toLowerCase();
        text = text.replaceAll("<[^>]*>", ""); // strip HTML tags
        Pattern pattern = Pattern.compile("\\s+", Pattern.DOTALL);
        String[] wordlist = pattern.split(text);

        long x = -1;
        long y = -1;
        double distanceSum = 0;
        long sumCount = 0;
        ArrayList sums = new ArrayList(wordlist.length / AVG_SIZE);

        for (int i = 0; i < wordlist.length; i++) {
            String word = wordlist[i].replaceAll("\\W", "");
            String vowels = word.replaceAll("[^aeiouy]", "");
            String consonants = word.replaceAll("[aeiouy]", "");
            long thisx = round(sumChars(vowels) * 3);
            long thisy = round(sumChars(consonants));
            if (thisx == 0 && thisy == 0)
                continue;
            if (x > 0 && y > 0) {
                distanceSum += Math.sqrt(Math.pow(thisx - x, 2) + Math.pow(thisy - y, 2));
                sumCount++;
            }
            if (sumCount > 0 && sumCount % AVG_SIZE == 0) { 
                sums.add((double) round(distanceSum / (double)AVG_SIZE));
                distanceSum = 0;
            }
            x = thisx;
            y = thisy;
        }
        
        // just drop in the last value if there is one
        if (distanceSum != 0 && sumCount % AVG_SIZE != 0) {
            sums.add((double) round(distanceSum / (double)AVG_SIZE));
        }

        sumCount = sums.size();

        MessageDigest md5 = null;
        try {
            md5 = MessageDigest.getInstance("md5");
        } catch (NoSuchAlgorithmException e) {
            return null;
        }

        for (int i = 0; i < sumCount; i++) {
            if (i + 1 < sumCount) {
                md5.update(String.valueOf((int) ((sums.get(i) + sums.get(i + 1)) / 2F)).getBytes());
            }
        }
        
        byte[] data = md5.digest();
        StringBuffer output = new StringBuffer(data.length * 2);
        for (int i = 0; i < data.length; i++) {
            if ((data[i] & 0xFF) < 16) {
                output.append("0");
            }
            output.append(Integer.toHexString((int) data[i] & 0xFF));
        }

        return output.toString();

    }

    private static final long round(double value) {
        return ROUND_TO * (long) (value / (double)ROUND_TO);
    }

    private static final long sumChars(String word) {
        long sum = 0;
        for (int i = 0; i < word.length(); i++) {
            sum += word.charAt(i) - 'a';
        }
        return sum;
    }

}

(One obvious problem is that crossing an 11-word boundary by even 1 word will cause two different sums to be produced... there is probably a way to cope with that, but I haven't given it much thought yet. More to come.)

Labels: software

It seems to be the weekend for stupidity out of Wicket.

God only knows how they managed to introduce this bug: component.setModelObject(foo) will truncate everything after the first semicolon. Yes, that's right: the first semicolon. Why? Your guess is as good as mine. Fortunately, component.getModel().setObject(component, foo) doesn't arbitrarily dislike semicolons and works fine.

More stupidity: ExternalLink is not a Link, though PageLink, BookmarkablePageLink, ResourceLink and so-on are all Links (that is, they extend Link). To Wicket's credit, ExternalLink at least appears in wicket.markup.html.link, although PagingNavigationIncrementLink and PagingNavigationLink, which do extend Link do not. I'm sensing some problems with the project's layout.

Another particular nuisance: there is no nice way of linking from one application to another. This may be because of limitations imposed by the Servlet spec, but it is annoying no less. This manifests itself if you want to move from one application that doesn't do anything special with Sessions to one that does. For example, I have an application for reading articles that doesn't need a session at all (but it implicitly gets a WebSession) with a link to another application for doing an article update. The latter uses an ArticleSession, but the moment you click the link to the homepage of the article update application, you get a class cast exception because the type of WebSession is different from one application to the other.

The solution is to loosely couple the two pages by directing the update link to the other application by its servlet path (e.g., /app/updateArticle) via an ExternalLink (which isn't a Link), but where does that path come from then? I had to settle for getting it out of a properties file for now, until some better way comes along.

I think Wicket could possibly figure this out in the Wicket Servlet by tracking which applications are mapped to which context paths, though this would probably require an additional init-param in the servlet config (since you don't get to know the context path in a servlet's init, though it's coming in the Servlet 2.5 spec... some day). In the near term there doesn't seem to be any other way than the ugly way.

Labels: software

The time came to add "action" functionality to my article display, so that the owner of an article (or an editor) can easily click to update an article. I decided to make actions dynamic and determined by the article service, then display the appropriate links using a ListView.

Getting a list of links to display in a ListView is easy enough. The downside is that to change the contents of the text of a link, you have to include inside it a <span> element and assign to that a Label and then change the contents of that. This seemed like a lot of running around just to change the contents of a PageLink to me, so I figured I'd have a go at simply extending PageLink to something that can set the contents of the body of the link tag to any arbitrary string.

The normal way one would do this is by overriding the onComponentTagBody method, but unfortunately doing this for PageLink is impossible. For whatever braindead reason, Wicket makes onComponentTag and onComponentTagBody final inside Link.

This is incredibly short-sighted if not outright stupid. There are a myriad of reasons why someone (like me) might want to extend the behavior of a Link, including the way the <a> tag and its body are rendered. Maybe one wants to add some random Javascript behavior, change the contents of the tag body, add a dynamic title attribute, etc. All impossible because someone made these methods final (for no good reason that I can see).

Stupid.

Labels: software

Sometimes it happens that you want a page to allow for a very small variation in markup based on something that happens at runtime. For example, you might want to include just a simple phrase like "No items found" in the case that nothing was found, or some more complex markup in the case that you had something to show. Or in my case, it's desired to conditionally include an <img> tag should the poster have a PNG file containing a mugshot stored on the local filesystem, or to include an ad panel if one has been requested but leave the space blank otherwise.

Without fragments, the only way to have conditional markup where the contents of that markup are manipulated by wicket would be to have multiple classes and multiple markup files. That's a lot of work for an insignificant difference in the content of the page. Wicket provides, however, and we have a construct like this:

      <span wicket:id="mugshot">(mugshot)</span>
 
      <wicket:fragment wicket:id="hasMugshot">
      <img src="/images/posters/Gorthak.png" alt="[Gorthak mugshot]" class="mugshot" wicket:id="mug" />
      </wicket:fragment>

In the code we can say something like this:

        if (mugPath != null) {
            Fragment mugfrag = new Fragment("mugshot", "hasMugshot");
            mugfrag.add(new Label("mug", "") {

                private static final long serialVersionUID = 1L;

                @Override
                protected void onComponentTag(ComponentTag tag) {
                    
                    System.out.println("onComponentTag called!");
                    tag.getAttributes().remove("src");
                    tag.getAttributes().add("src", mugPath);
                    tag.getAttributes().remove("alt");
                    tag.getAttributes().add("alt", mugUser + " mugshot");
                }
            });
            add(mugfrag);
        } else {
            add(new Label("mugshot","").setEnabled(false));
        }

Now I probably could have accomplished something similar in this simple case by just setting up the img tag with a wicket:id and setting that element visible or invisible. Notice what the fragment lets us do, however- something we can't do with plain markup in Wicket. We can ignore the fragment completely in the Java code (as in the else block above). Similarly, we can conditionally use one <wicket:fragment> from an arbitrary number of them if we choose.

One thing to keep in mind is that when you're adding components in the Java class that are encapsulated by a <wicket:fragment>, you're adding them to a Fragment not to the page. As you can see, adding a Fragment involves giving the name of the place you want the wicket:fragment to go and the name of the fragment itself.

You can see I'm doing some crazy inner-class trickery for my img tag. Wicket's Image class expects to refer to an image file that's contained within the framework. In my case, the image is actually out on the filesystem as part of the static content of the web site. So I've basically extended the Label class to deal with the needs of an img tag, in this case replacing the contents of the src and alt attributes when the tag is rendered. I'll probably refactor this out into its own class at some point- there's really more code than I'd like to see in an anonymous inner class there, and I could see reusing it in other places.

Another thing I want to work on is a strategy for copying in/out values from the data model to the service bean classes. I'm thinking that annotations+reflection might be an effective and clean way of doing it.

I resolved my earlier pondering about the article table trigger in PostgreSQL. It is as simple as this:

        ELSIF (TG_OP = 'UPDATE') THEN
            IF (NEW.section != OLD.section OR
                NEW.title != OLD.title OR
                NEW.adjective != OLD.adjective OR
                NEW.summary != OLD.summary OR
                NEW.body != OLD.body) THEN
                INSERT INTO article_audit SELECT nextval('article_audit_audit_id_seq'), now(), 'U', NEW.*;
            END IF;
            RETURN NEW;

Duck soup!

Labels: software

Tonight I got the basic article submission process working completely. One can now fill out the form, get through validation, check a preview and submit the results for creation via the ArticleService.

Stuff that isn't working yet includes checking for duplicates, handling conditional creation based on a user's status (ie, have they been approved to skip the approval process?) and UID/open-proxy checks.

I also want to make the Preview page use a Wicket Panel for the article; that way I can reuse that panel to display the article.

It dawned on me today that my plan to use triggers to create an audit history for articles needs a second look. The trigger is going to fire every time someone votes an article up or down, mails the article, etc. I don't really want these events to cause an insert in the audit table... only changes to the text of the article should do that. Then there's the question about whether these belong in the service layer or on the database.

One thing that's disappointingly missing from JPA is a way to assign the sequence numbers to new objects. Toplink provides this facility in the full product, but as far as I can tell, JPA offers no way to do it. This is annoying from a service standpoint because it means the only way I can get at the new object ID is to commit the transaction, which kind of prevents chaining together multiple activities in one transaction.

I've seen this discussed before among people trying to figure out service-oriented architectures. One school of thought is that you just don't try to make several activities part of one transaction, but instead provide a method to undo any action that was taken. So for every create, there is a destroy, and for any update, you keep the old state in case you need to update it back to the original values.

Another possibility is to provide multiple versions of calls- one that's public facing that takes no transaction as an argument, and then internal versions that allow a running transaction to be passed in. Then any chaining of events must happen at the service level (ie, there's a single separate service call for the chain of events) with the transaction still managed by the service.

I had to do this because I needed to get back an ID for the article that was just created, and the only way was to commit the transaction, due to the absence of an assignIds() call.

Labels: software

In my last post I talked about passing the model around by virtue of keeping it in the WebApplication's session, but there's another, less memory-intensive way of handling this as well. (Thanks to the folks in ##wicket on freenode for this one.)

Rather than calling setResponsePage(PreviewBeforeSubmit.class), it is also possible to say setResponsePage(new PreviewBeforeSubmit(modelObject)), then define a constructor in the PreviewBeforeSubmit class that takes the model object. This does mean that to go back your originating page must also define a similar constructor, and its no-arg constructor may need to do the original instantiation (or your WebApplication could).

This strategy saves a few bytes in your session, since the model object is already in there. It means you won't need a second class in the session to contain the first.

Labels: software

So before bed last night I thought I'd be clever with my WebApplication and require authentication for all instances of WebPage.class. It seemed to work at the time, but that was because I had already logged in, so the required stuff was already in my session.

The problem is that redirecting to Signin.class means redirecting to a WebPage, and that web page comes under the control of the same WebApplication that demands having authentication for all WebPages. You can imagine what happens. If you can't, I'll tell you: an infinite redirect, as it demands authentication for the authentication form.

The nice solution to this is to create a subclass of WebPage, call it AuthenticatedWebPage, implement all of its constructors to call super(...) and make the pages that require authentication extend AuthenticatedWebPage instead of WebPage, and configure the SigninAuthorizationStrategy class to work on classes that are assignable from AuthenticatedWebPage.

(Naturally Signin should not extend AuthenticatedWebPage.)

Labels: software

Before I could get to the point of actually adding an article to the database through my web form, I needed to have an authenticated user. Wicket provides a way to interject another page into the normal flow of an application; this can be done to show an ad prior to the desired page, and it can also be used to require authentication before proceeding.

Unfortunately the wicket example page for sign-in is very single-application-centric. It uses a Session object created via overriding the WebApplication's getSessionFactory() method and then it performs authentication via methods in that Session object. This works if you only have one application on your web site, but it falls down pretty quickly if you want to do web-site-wide authentication. I want my users' logins to work for all applications once they've signed in once. I accomplish this now by sticking a User object in the session once a login is completed successfully.

Unfortunately, Wicket doesn't have a notion of a thing higher in its hierarchy than an WebApplication. There are WebApplications which are generally made up of one or more WebPages which in turn are generally comprised of many WebComponents. (I'm simplifying of course.) There is no notion of a global thing that exists outside the set of WebApplications. Furthermore, a Session object is not a javax.servlet.Session, but rather a Wicket invention that abstracts away things like getting the HttpSession from the request and putting things in it. These Session objects are, again, very WebApplication-specific; you create them by coding up a subclass and returning it via a session factory.

So how do we get something to live in the HttpSession outside of the influence and control of any single application but still usable from those applications? The answer lies in this rather ugly piece of code:

HttpSession session = ((WebRequest)(WebRequestCycle.get().getRequest()))
    .getHttpServletRequest().getSession();

Even though the Wicket framework tries to abstract away many things and handle a lot of menial plumbing for you, it doesn't completely tie your hands in the process. If you need to stick something in the user's HTTP Session, you still can... but beware-- once you step outside the framework, you're on your own for things like type checking, null handling, serialization, etc.

Because Wicket finds WebPages via their class, it's quite easy to create a new package and put the class and HTML markup needed for the signin form in it. I started by taking these directly out of the Wicket example.

The next step was to create an implementation of IAuthorizationStrategy (there's that stupid I prefix again). This is the way authorization is wired into an Application:

getSecuritySettings().setAuthorizationStrategy(instance of IAuthorizationStrategy);

That goes in the WebApplication's init() method. In the Wicket example they create this on-the-fly with an anonymous inner class. I think anonymous inner classes are hideous. What, was it too hard for you to click "New --> Class" in your IDE? Do you have to pay for your disk storage by the bit? No tabs in your editor? Honestly, make a damn concrete class. It won't kill you, and it makes your code far easier to read.

I created a class called SigninAuthorizationStrategy. I didn't stick the abbreviation "impl" in the name, because, like prefixing interface names with I, that behaviour is stupid and pointless. You know by the fact that the class declaration says "implements" that it's a bloody implementation.

As Wicket adds Components to your WebApplication, it calls your public boolean isInstantiationAuthorized(Class componentClass) method, passing in the class of the component that it's adding. This happens recursively, so you'll get a call for the WebPage, then every WebComponent on the web page. We don't want to have to authorize every single component on the page, so we pass in the class of components we do want to authorize and ignore the others:

 private Class componentTypeToAuthorize;

 public SigninAuthorizationStrategy(Class componentType) {
  this.componentTypeToAuthorize = componentType;
 }

 public boolean isInstantiationAuthorized(Class componentClass) {
  if (!(componentTypeToAuthorize.isAssignableFrom(componentClass))) {
   return true;
  }

We return true because we are assuming that we're doing authorization at a high level of containment; that is to say, we're authorizing at the page level, and if the page is authorized, then its contained components are assumed to be authorized as well. There's no point in spending the cycles authorizing them too. The code goes on to say:

  UserBean userBean = Utils.getUserBean();
  if (userBean != null && userBean.isAuthenticated()) {
   return true;
  }
  throw new RestartResponseAtInterceptPageException(SignIn.class);
 }

Utils.getUserBean() is just a convenience method to handle that ugly HttpSession bookkeeping we saw earlier. The rest is pretty straightforward: if there's a UserBean in the HttpSession and it has been properly authenticated, then we authorize this component (in this case a WebPage). Otherwise, we throw the quite-verbosely-named RestartResponseAtInterceptPageException passing it the class name of the page we want to use as the intercepting page.

SignIn.class in turn handles the web form for signing in, authenticating via the UserService and adding the resulting UserBean to the user's session. Once this is complete, Wicket automatically resumes control where it left off via this call:

    if (!continueToOriginalDestination())
    {
     setResponsePage(WebRequestCycle.get().getApplication().getHomePage());
    }

If for some reason we're unable to continue to the original destination, we divert to the application's home page instead. This should probably not ever happen.

You can find the full source on the Tally-Ho project homepage. Look in net.spatula.news.ui.signin and net.spatula.news.article.create.

Labels: software

I'm pleased to announce that the new morons.org codebase finally has a name:



Named, of course for The Village newspaper from the classic television series, The Prisoner.

The code will be released under the Apache 2.0 license, a departure from the former Spatula Public License. Much as I like my SPL, it has some flaws. My main goals in creating that license were to make my code GPL-phobic, because I despise the GPL and any other license whose creators intend to spread like a virus.

Of the free, open source licenses, the Apache 2.0 license comes the closest to my goals, requiring anyone deriving something from my project to leave my copyright notice intact and provide a NOTICE file indicating that they've made changes, though for their own contributions and code they may use whatever license they choose. I hope they choose to also use the Apache license, not the GPL abomination.

Having the Apache license allows me to manage the project with Google Code using Subversion. This is the project's new home. I've checked in the code I've written so far. Keep in mind that this is very much a work in the very early stages of progress. It has quite some way to go before it's in a state that's usable by anybody.

Labels: software

As the Article form progressed it came time to address the HTML question. It's easy enough to do a simple regex match to disallow the use of HTML entirely, but allowing limited HTML presents a more challenging problem. Moreover, allowing limited HTML tags with a limited set of attributes (since we don't want to see anyone injecting onmouseover or onclick in their tags) is a bit trickier still. Getting HTML from end users who may not be terribly familiar with HTML constructs and don't always write the cleanest HTML is even more of a challenge.

My rules for HTML in articles are pretty simple. You must enclose paragraphs with the <p> tag. The allowed tags are p, a, em, strong and blockquote. The only one of these that can have attributes is the <a> tag, and the only attribute I'll allow is href. It's fairly restrictive, but articles are mostly just paragraphs with the occasional link anyway. Anything else might interfere with the flow or style of the page.

To deal with the malformed HTML question, I turned to JTidy, as I have in the past. JTidy is good at dealing with many common HTML coding problems, like dropped end tags, inlines that span blocks, and other common errors. It is not good at dealing with dropped angle brackets and dropped quote marks. For my purposes it is normally adequate. I wrote a small wrapper class around Tidy which also allows users to leave out the paragraph tags and leave blank lines to denote paragraphs... paragraph tags are inserted on their behalf. Additionally, it sets up some options:

  tidy.setMakeClean(true);
  tidy.setWord2000(true);
  tidy.setLogicalEmphasis(true);
  tidy.setDocType("strict");
  tidy.setShowWarnings(false);
  tidy.setQuiet(true);
  tidy.setXHTML(true);
  tidy.setDropEmptyParas(true);

There's also a "demoronize" method to change instances of <br /><br /> to paragraphs.

Unfortunately, Tidy does not remove 100% of the cruft; it can let attributes and tags with namespaces sneak through when you're outputting a document in XHTML or XML, which we need for proper page parsing. Tidy also won't remove arbitrary tags or attributes. A second phase of cleaning is needed.

For my second phase, I decided to try SAX. I have worked with DOM before, but the lightweight nature of SAX appealed to me for this part of the project. SAX can operate on a stream and you never need to have the entire XML document in memory. The way one commonly interacts with the SAX parser is to override the DefaultHandler class, implementing methods that are called by the parser when particular events occur during the parsing of a document. Typically you'd implement startDocument, startElement, endElement, characters, ignorableWhitespace and possibly endDocument.

SAX doesn't really give you a way to alter the document it's parsing (to do so would generally require holding the document in memory, something for which you'd want to look at DOM). If you need to construct a new document, that's up to you. It does provide another mechanism for coding filters which can be chained, and that chain could certainly end with an all-purpose document-writing filter. But this is left up to you.

I decided to take the simplest approach possible and created a class called AllowedTag, which holds a tag name and the list of associated attributes I'll allow on that tag. I didn't go to the effort of differentiating between allowed and required attributes or attributes which might be mutually exclusive, leaving that work up to JTidy. I just want to strip the tags which offend me.

As SAX reads the end user's HTML, it makes calls to my implementation of startElement, passing in the namespace URI, fully qualified tag name, raw tag name, and a set of attributes in an Attributes object.

Then I check the tag name against my map of AllowedTags. If it's allowed, I append a tag opener to my buffer and step through the Attributes, adding those which are allowed by the AllowedTag object. For anything which is not allowed, I record an error and increment an error count as I ignore the entity when building the new document.

Characters and ignorable whitespace are simply appended to the buffer. For end tags, I perform the same check as in the start tag method, with one additional gotcha: self-closing tags like <br /> fire startElement and immediately fire endElement. In order to preserve these tags as-is without changing them to something like <br></br>, it is necessary to track the last tag seen and the position in the output buffer after that tag was inserted. Then when handling an endElement one can check to see if the buffer position has not changed and that the tag name is the same; if so, just change the last character (which will be >) to  />.

Normally when operating with SAX and trying to do anything any more complicated with this, one would be pushing elements onto a stack with startElement and characters and then popping data with endElement. In this case what we're doing is so incredibly simple that a stack of depth > 1 is not necessary (the self-closing trick is functionally equivalent to a depth 1 stack).

After finishing these classes (jUnit test-driven of course) I wired them into a Wicket validator, grossly abusing the validation framework by again letting it rewrite the data model if errors were found and changes were necessary (like illegal tags or attributes used) and by calling error(String) on the form's article body component to insert the very-specific error messages generated by the Tidy wrapper and the SAX tag cleanser.

I also found a flaw last night in my plan to wrap the EntityManagerFactory in a singleton ResourceManager. I'm not completely foiled, but some additional work may be needed. It can happen that an EntityManagerFactory fails to initialize properly or may become invalid due to database connectivity problems. When this happens, that EMF refuses to cooperate any further; you have to destroy it and create a new one. So we can't blindly hold this in a singleton without doing some checks to ensure that it's still valid and functioning. The thing to do might be to move the createEntityManager to the ResourceManager and include the option to destroy and reconnect with some number of retries in that part of the code.

Labels: software

The AJAX code for showing the section description was all well and good, but is it really worth a trip back to the server to fetch that data every time someone changes the drop down choice? A better way would be to send the mappings to the client and have the client update the description itself, without any server intervention.

My first attempt at this was to write out the necessary Javascript by giving a <script> tag a wicket:id and then defining a corresponding Label in my form code. If you go this route, you must set label.setEscapeModelStrings(false); otherwise, Wicket will turn instances of " into &quot;. Using this strategy, you can dynamically generate Javascript to be included in-place on the page. This is all well and good, but it means that any time you want this behaviour, you have to remember to include a <script> tag, sort out the HTML ids yourself, and then worry about multiple drop-downs defining the same function multiple times. This strategy is also not very reusable.

A better solution is to add a behaviour or as the Wicket folks spell it, a behavior. (Years upon years of watching and reading British science fiction have made their impression upon me.) I came across behaviors initially when I added the AjaxFormComponentUpdatingBehavior and again when I was wondering how to get my Javascript into the <head> of a document. It turns out that wicket provides HeaderContributor.forJavascript just for this purpose... and as it happens, HeaderContributor is itself a subclass of AbstractBehavior.

Implementing a behavior is a matter of extending AbstractBehavior and implementing the methods that you need. Typically one will implement bind(Component) and onComponentTag(Component, ComponentTag). It becomes somewhat important to understand the internal Wicket lifecycle, as you can guess from these names. bind is called when your behavior is initially bound to a Component. onComponentTag is called during rendering. It's important to realize that during rendering, you can no longer change the Component, or you'll get a ConcurrentModificationException or the like... but you can make changes to the ComponentTag.

Another "gotcha" is that the markup ID for a component is not generated until rendering time. I am told this is changing in Wicket 2. It's a bit of a limitation for this case; to inject Javascript into the document head, we have to modify a Component. The Javascript needs to know the HTML id of the tag (which corresponds to a Wicket Label) it is updating. For Wicket 1.2, we have to settle for specifying the HTML id of the Label's tag manually and supplying it to our behavior when we add it to our DropDownChoice. This also means that the Label cannot have setOutputMarkupId(true), because that would override the manually-specified HTML id. We can still figure out the id of the DropDownChoice via the onComponentTag method, because the getMarkupId method will return the correct markup id by that time.

Initially I had just used StringHeaderContributor to contribute both the static Javascript function and the dynamically-generated per-drop-down mappings, but Wicket offers a better way to inject the Javascript, using the forementioned HeaderContributor.forJavascript(Class class, String path) method. This allows you to place your Javascript file in the same package as your behavior class and pass that class and the filename of the Javascript. Wicket handles inserting the Javascript code reference and the Wicket servlet takes care of extracting and sending the file from your package in response to a request for it. This is probably a tiny bit less efficient than referencing an absolute path on the server, but the benefit is including the code in the same place and being able to easily edit it on the fly from your IDE.

The Javascript is fairly straightforward, though I had to get some help writing it. Here it is:

var updateMap = new Object();
function updateLabel(list, label) {
  var mylist = document.getElementById(list);
  var mylabel = document.getElementById(label);
  var selectedItem = mylist.options[mylist.selectedIndex].text;
  mylabel.innerHTML = updateMap[label + '.' + selectedItem];
}

Right now it doesn't guard against conditions like an entry not being found in that updateMap, but that should be easy enough to add. As you can see, it takes the name of a drop-down list and the name of the HTML entity it should update, looks up the current value from the drop-down in the updateMap, and sets the value. The mappings are dynamically generated by DropDownLabelUpdateBehavior, which is supplied with a List of beans, the names of the fields to access in those beans to get the value for the map and the data associated with that value, and the HTML id of the field it should update. After all this explanation, the code itself seems pretty simple:

package net.spatula.news.ui.behaviors;

import java.lang.reflect.Method;
import java.util.List;

import wicket.Component;
import wicket.behavior.AbstractBehavior;
import wicket.behavior.HeaderContributor;
import wicket.behavior.StringHeaderContributor;
import wicket.markup.ComponentTag;

public class DropDownLabelUpdateBehavior extends AbstractBehavior {

 private static final long serialVersionUID = 1L;

 private static final String javaScriptStart = "<script type=\"text/javascript\">\n";

 private static final String javaScriptEnd = "</script>\n";

 private String labelName;
 
 private String value;

 private String id;

 private List beans;

 public DropDownLabelUpdateBehavior(List beans, String labelName,
   String idName, String valueName) {
  this.beans = beans;
  this.id = idName;
  this.value = valueName;
  this.labelName = labelName;

 }

 public void bind(Component component) {
  component.add(HeaderContributor.forJavaScript(this.getClass(),
    "dropDownLabelUpdateBehavior.js"));
  component.setOutputMarkupId(true);
  component.add(new StringHeaderContributor(javaScriptStart
    + getMappings() + javaScriptEnd));
 }

 public void onComponentTag(Component component, ComponentTag tag) {
  tag.getAttributes().remove("onchange");
  tag.getAttributes().add(
    "onchange",
    "updateLabel('" + component.getMarkupId() + "', '" + labelName
      + "')");

 }

 private String makeGetterName(String field) {
  String upCaseFirstLetter = field.substring(0, 1).toUpperCase();
  String rest = field.length() > 1 ? field.substring(1) : "";
  return "get" + upCaseFirstLetter + rest;
 }

 private String getMappings() {

  if (id == null || id.length() < 1 || value == null
    || value.length() < 1) {
   return "";
  }
  String idMethod = makeGetterName(id);
  String valueMethod = makeGetterName(value);

  StringBuffer map = new StringBuffer(100);

  for (Object object : beans) {
   Method getId;
   Method getValue;
   String thisid, thisvalue;
   try {
    getId = object.getClass().getMethod(idMethod, (Class[]) null);
    getValue = object.getClass().getMethod(valueMethod,
      (Class[]) null);
    thisid = (getId.invoke(object, (Object[]) null)).toString();
    thisvalue = (getValue.invoke(object, (Object[]) null))
      .toString();
   } catch (Exception e) {
    // Just make a best effort, and skip the element if anything
    // goes wrong.
    continue;
   }
   map.append("updateMap[\"" + labelName + "." + thisid + "\"] = \""
     + thisvalue + "\";\n");
  }

  return map.toString();
 }
}

A few things to note: we turn on setOutputMarkupId on the component so that it'll be inserted in the HTML in the same way as we see it in onComponentTag, and any existing onchange attribute in the generated ComponentTag has to be removed first; calling tag.getAttributes().add does not seem to overwrite an existing value. One thing that would be good to add is a check in bind to ensure that the Component is compatible with this behavior.

Once all of this is in place, the code to fetch the Section list and add this behavior looks like this:

  EntityManager em = ResourceManager.getInstance().getEMF()
    .createEntityManager();
  List sectionsList = SectionService.getInstance()
    .getSections(em);
  em.close();

  sectionDescription.setOutputMarkupId(false); // absolutely don't want
              // it overwritten
  if (article.getSection() != null && article.getSection().getId() != 0) {
   sectionDescription.setModelObject(article.getSection()
     .getDescription());
  }
  add(sectionDescription);

  DropDownChoice sectionChooser = new DropDownChoice("section",
    new PropertyModel(article, "section"), sectionsList,
    new ChoiceRenderer("name", "id"));
  sectionChooser.add(new DropDownLabelUpdateBehavior(sectionsList,
    "sectionDescription", "id", "description"));

  add(sectionChooser);

There you have it: a reusable behavior for updating an HTML field when the selection in a DropDownChoice changes.

Labels: software

I wanted to take a second look at the way I was doing validation, in part because I still wanted to figure out how to get my validators to affect change in the model objects (e.g., if I want my "NoHtmlValidator" strip the HTML tags). I had also noticed that the way I constructed my validator was not strictly in-keeping with the way the Wicket folks wanted it done. There's a big, all-caps message for the validate method saying "THIS IS NOT PART OF THE PUBLIC API". One wonders why the method is public in that case.

The Wicket API for IValidator suggested extending CustomValidator to do a custom validator, but CustomValidator is deprecated. I guess their Javadoc is a bit outdated. Instead, the thing to do is to extend StringValidator, a class for which you can override the onValidate method. The main difference I can see is that this passes you both the FormComponent and the String data associated with the model for that component... saving you having to call component.getInput() yourself.

After hunting through the available completions for a bit, I found the methods setModelValue(String) and setModelValue(String[]) for setting a single-value attribute or a multi-value attribute respectively. This is one of those few times one is reminded that what we're actually always dealing with are strings sent via HTTP. Some things are single strings, like text fields and check boxes; some things are arrays of strings, like multi-select boxes.

Now the code for my NoHtmlValidator looks like this:

package net.spatula.news.ui.validators;

import java.util.regex.Matcher;
import java.util.regex.Pattern;

import wicket.markup.html.form.FormComponent;
import wicket.markup.html.form.validation.StringValidator;

public class NoHtmlValidator extends StringValidator {

 private static final long serialVersionUID = 1L;
 private static final Pattern pattern = Pattern.compile(".*</?[-\\w]+[^>]+>.*", 
                                                 Pattern.DOTALL);

 public void onValidate(FormComponent component, String data) {
  
  Matcher matcher = pattern.matcher(data);
  if (matcher.matches()) {
   error(component);
   data = data.replaceAll("</?[-\\w]+[^>]+>", "");
   component.setModelValue(data);
  }
 }

}

You can see I've moved the Pattern up into the class; there's no sense in spending the overhead of compiling that Pattern more than once. The key change here is that if you did include HTML tags in your field, the validator now strips those tags for you.

My next task was to get the Article Section select box (DropDownChoice in Wicket terms) populated. For this, I created a SectionService singleton class. Borrowing an idea from an Oracle JPA tutorial I also created a singleton ResourceManager class for creating the EntityManagerFactory. Provided anything needing an EntityManagerFactory gets it from the ResourceManager class, this has the effect of ensuring that across the application, I only ever create one EntityManagerFactory. This is definitely desirable while testing, since with the JPA RI (Glassfish), the JDBC connection pool is managed by the EMF. There's no sense in creating more than one of these.

I was able to fairly quickly create my service. Initially I had it return a List of KeyValuePairs for the section ID and the section name. It was trivial to populate the DropDownChoice. There was still a problem, however. My section names are fairly terse, and if I'm to have my readers able to submit articles at-will, they are going to need some guidance about which section they should use for their articles.

This seemed like a good job for AJAX.

Wicket has some built-in AJAX goodies that are perhaps not quite as easy to use as Google's web toolkit, but they have the advantage of being integrated into the framework. To get AJAX behaviour out of the Sections DropDownChoice, we add AjaxFormComponentUpdatingBehavior to the choice. It turns out that the easiest way to do this is with an anonymous inner class, as demonstrated on the Wicket Wiki. My code looks like this:

// In the Form class, declare the objects you are going to change from an AJAX action,
// in this case, a Label called sectionDescription.
 private ArticleBean article = new ArticleBean();
 private Label sectionDescription = new Label("sectionDescription", "");

// Later on as you're building your form... 

  EntityManager em = ResourceManager.getInstance().getEMF()
    .createEntityManager();
  List sectionsList = SectionService.getInstance()
    .getSections(em);
  em.close();

  sectionDescription.setOutputMarkupId(true);
  if (article.getSection() != null && article.getSection().getId() != 0) {
   sectionDescription.setModelObject(article.getSection()
     .getDescription());
  }
  add(sectionDescription);

  DropDownChoice sectionChooser = new DropDownChoice("section",
    new PropertyModel(article, "section"), sectionsList,
    new ChoiceRenderer("name", "id"));
  sectionChooser.add(new AjaxFormComponentUpdatingBehavior("onchange") {

   private static final long serialVersionUID = 1L;

   protected void onUpdate(AjaxRequestTarget target) {
    sectionDescription.setModelObject(article.getSection()
      .getDescription());
    target.addComponent(sectionDescription);
   }

  });

  add(sectionChooser);

So what's going on here? First, I get my List of sections from my service. You can see that I had to abandon KeyValuePairs for SectionBeans, since I needed not only the ID and name, but the description also. Then, we have to setOutputMarkupId(true) so that Javascript can manipulate the HTML object by name. Next there is code to statically render the section description if it is already known; for example, after a form submission in the case that the form needs to be re-displayed.

Then there is code to create the DropDownChoice. Because I want the rendered option values to correspond to the section IDs on the database, I use a ChoiceRenderer, giving it the names of the fields to access from the objects in the list to get the display text and the value to associate with that text as the DropDownChoice iterates through the list.

Finally, we add the AjaxFormComponentUpdatingBehavior to the DropDownChoice component. The "onchange" Javascript event is specified, and in the anonymous inner class extending AjaxFormComponentUpdatingBehavior, onUpdate is overridden. Just as we did when rewriting a field with a validator, we can rewrite the contents of the simple String model of the sectionDescription Label to the description of the SectionBean held in the Article. Notice that we're asking the article-- the backing model-- itself for this SectionBean object; we're not querying the component.

I happily loaded my web page and clicked on the drop-down. Nothing happened. I looked at the page source, and indeed there was Javascript markup on that <select> tag. I popped open the Javascript console and found it full of errors: "wicketAjaxPost is not defined." I looked closer at the source and sure enough, there was no script loading to define that function. After some digging, I came to realize that the culprit was Sitemesh. Wicket puts the declarations to load the Javascript code (which it serves out via its servlet) in the <head> section of the HTML document, which Sitemesh happily strips. The solution is to include a <decorator:head /> tag in the Sitemesh decorator at the end of its <head> section.

One thing I tried to do with AJAX in Wicket initially proved not to work. I wanted to make a <br /> tag come and go depending on whether a choice had been made in the drop-down. Unfortunately, there seems to be some problem with changing the value of setVisible() with AJAX, at least on a <br>. I ended up just putting the section description after the drop-down box, which turned out to be nicer anyway, since it doesn't change the vertical size of the table when the text gets rendered. UPDATE: it doesn't work because AJAX is operating on the client side; in other words, because the initial state is not visible, there's no markup sent for the <br /> to the client, which in turn means it's impossible for AJAX to tell where to put it when it's added. The solution is to wrap it in something that IS sent, like a containing <span> and repaint the <span>.

Next I want to start looking deeper at WicketTester, which helps you write unit tests for Wicket pages and forms, helping you test validators and page contents without having to run a servlet container. If it works, this could be the holy grail of web application UI testing, something that has never had a great solution. (And if you think HTTPUnit is a great solution, there is something the matter with you.) Maybe I can get a decent unit test for NoHtmlValidator going. It won't have been test-first, but the other validators can be now.

Labels: software

There are a lot of things I've come to love about Toplink over the years, and by the transitive property I love them in Glassfish. But there's one part of Toplink I absolutely cannot stand: the error messages, or lack thereof.

Consider this one: "Unknown abstract schema type [Foo]" where "Foo" is some class. What does that actually mean? It means Toplink couldn't find a class called Foo defined in persistence.xml. So why couldn't the bastards just SAY that? Why did they have to say "unknown abstract schema type"?

Attention Toplink Authors, or at least those of you who write these fucked-up error messages. I have a little task for you. There's something I want you to do. Ready for it?

Die in a fire as you eat my ass.

Seriously though, this got me thinking about error messages- what kinds of error messages I find useful and what sort of error messages I hate. Obviously I hate messages like "unknown abstract schema type". Worse still is the *absence* of error messages (aka, the "silent failure") and only slightly better is the NullPointerException (or any other exception with a broad, generic name) with the stack trace.

So what makes a good error message good and a bad error message bad? I have narrowed my scope to four things: Context, Clarity, Detail and Action-ability.

• Context: this error didn't just happen out of the blue. What was the program doing when the error occurred? Was it trying to locate a class defined in persistence.xml? Was it trying to find META-INF/persistence.xml on the classpath? Was it attempting a network connection? A disk write? Was this action part of some larger task? Just what was going on in the system when the error occurred?


• Clarity: errors should be concise, plain-English descriptions of the fault condition that give the developer (or the end user) a concrete idea of what actually went wrong. "Unknown abstract schema type" doesn't do that. "Unable to find a <class> entry for [Classname] in persistence.xml" would give a much clearer idea of what went wrong.


• Detail: hand-in-hand with clarity, the error must give enough detail so that the problem can be resolved, but not so much detail that the user or developer is buried in irrelevant minutia. "Class not found" isn't enough detail. "Unable to locate class net.spatula.news.GoatCheese on the classpath" is better. Along with Clarity, "Unable to locate class net.spatula.news.GoatCheese during initialization" is better still.


• Action-ability: this goes beyond simply giving an error message to which the developer or end-user can respond with some action; it's better still when the error message can offer suggestions. I was pleased to see this show up in PostgreSQL 8.1. Not only did it tell me what my error was, it offered a HINT about what I might do to correct it. How much easier my morning might have been had Glassfish Persistence said, "Could not find an entry for class [Article] while initializing mappings. Make sure the class is specified by EJB or by a <class> entry in META-INF/persistence.xml."

Yeah, it's a little more work for the framework developer, but it makes the framework a lot more usable and developer-friendly.

Labels: software

So tonight I was going to do some work on my JPA stuff, and found that the Dali plug-in for my Eclipse installation has vanished. The perspective is gone. Eclipse no longer lists it among the installed plug-ins. It has simply vanished.

I would try to figure out what the hell happened, but it's been a long day already. I think I'll just download and unpack Dali again and re-install it.

Weird.

Update: reinstalling didn't work. The files are all there, but Eclipse simply refuses to see or read them. My project is dead in the water until I can figure out what the hell is going on.

Update #2: it looks like somehow Eclipse also lost WTP, which is a dependency for Dali. No idea how it could have lost WTP either.

Update #3: was it something I upgraded? *burp* Indeed, when the Dali page says that it is targeted against a particular release of WTP, it really means it. The natural thing for a programmer geek to want to do is to upgrade plugins and packages to their newest, most stable releases, but it turns out that one of the upgrades I had done broke Dali. It's a bit frustrating that even with -debug -console -consoleLog -vm {path to java.exe}, the plugin failed *silently*.

The fix is to unzip the targeted version of WTP and use that only. The all-in-one bundle includes Eclipse as well, so don't even bother grabbing Eclipse first. Dali is apparently so fragile still that it is targeted against one particular build of the Eclipse SDK. Once WTP all-in-one is installed, then unzip Dali to get the plugin.

I look forward to the day when: Dali works with the latest stable versions of things and plugins don't fail silently.

Labels: software

Today I decided that rather than reindex everything so that all my object IDs can be unique, it would be easier to create a few views in PostgreSQL. That would solve my earlier problem of doing a join with JPA between the article table and the message roots table.

Fortunately, this is easy in PostgreSQL:

CREATE VIEW article_message_root(root_source, root_id, posting_permitted, post_count, post_count_24hr, last_post)
    AS SELECT root_source, root_id, posting_permitted='y', post_count, post_count_24hr, last_post
       FROM roots
       WHERE root_type='article';

As you can see, we can even convert one type to another with a view; when I originally made this table, eons ago, the boolean datatype wasn't available (initially this schema was from an ancient version of MySQL) or I was having some other problem with it. Now I can compensate for that in the view. PostgreSQL determines the data types of each of the resulting columns based on the data types given in the SELECT.

The view seems to be implemented by appending the conditions in the view's WHERE clause to any queries issued against the view:

=> explain select * from article_message_root where root_source = 1234;
                                     QUERY PLAN
------------------------------------------------------------------------------------
 Index Scan using roots_root_source_key on roots  (cost=0.00..5.99 rows=1 width=29)
   Index Cond: ((root_source = 1234) AND ((root_type)::text = 'article'::text))

Compare that to:

=> explain select * from roots where root_source = 1234 and root_type='article';
                                     QUERY PLAN                                     
------------------------------------------------------------------------------------
 Index Scan using roots_root_source_key on roots  (cost=0.00..5.99 rows=1 width=40)
   Index Cond: ((root_source = 1234) AND ((root_type)::text = 'article'::text))

Eventually I'll restructure the whole site codebase and these views will go away in favor of a system where every object has a unique identity across the entire system.

I'm also pondering using the insert trigger to create an entry in a centralized table when that time comes to record the type of object associated with each object identity. Makes it easier to hunt these things down should it become necessary.

Labels: software

The next thing I wanted to figure out was the "Wicket Way" of doing form validation. I've made the mistake in the past of going off and doing things on my own when an existing framework already provided a particular best practice for accomplishing the task. Fortunately, Wicket makes it quite easy to add standard form validators and to add your own.

I decided to start with my "headline" field from my simple article form. A headline can be from 1 to 80 characters, must be provided, and must not contain HTML code. I decided to bite off the hard part first: Wicket does not provide a validator class for rejecting input that contains HTML.

I created a new package, net.spatula.news.ui.validators, figuring I'll want to reuse these and keep them all in one place. I looked at one of the standard validators in the Wicket API to see what it extended and/or implemented, and it turns out that it both extended AbstractValidator and implemented IValidator. Mind you, I still think it's dumb to start Interface class names with an I, especially since nobody who does this does it consistently. Nobody shortens "Abstract" to "A" or spells out the word "Interface." Nobody starts every normal class name with a "C" or the word "Class." This is all superfluous fluff. I know by the fact that the name follows the "implements" keyword that it's a bloody interface. But I digress.

I asked Eclipse to create for me a class called NoHtmlValidator, and have it extend AbstractValidator and implement (gag) IValidator. I was pleased to see that all I had to do was supply the method body for public void validate(FormComponent component). Easy.

FormComponents have a method called getInput(), which returns the value you might get from request.getParameter("name"). It's the String value of the component's data. Fortunately for me, a TextField is just one value, so for now, this will suffice. I coded up something really simple:

 public void validate(FormComponent component) {
  String data = component.getInput();
  Pattern pattern = Pattern.compile(".*<[-\\w]+[^>]+>.*", Pattern.DOTALL);
  Matcher matcher = pattern.matcher(data);
  if (matcher.matches()) {
   error(component);
  }
 }

I'll refactor the Pattern to be a static member of the class later, so it doesn't have to be compiled every time. As you can see, the match is pretty dumb, but if HTML is detected, I call error(component). Notice that I don't actually give the error message here. It turns out that Wicket wants to get your error text from a resource (.properties) file.

Having had a little experience with Wicket so far, I figured that surely this meant that the name of the resource file it would want would be the class name of my page, followed by .properties. Sure enough, that's what it is looking for. Inside this file, Wicket wants a line like this:

formName.componentName.ValidatorClassName=Message to use when validation error occurs

After fighting with it initially, I found that the formName is the name you used when creating the form component. It is NOT the form's class name (if you created a separate form class). Now you might be thinking, "you're going to end up having to create one of these for every single field that doesn't allow HTML. You'll have messages coming out your ears in this file!"

Not so! Wicket provides this fantastic way of generalizing your messages. You can create a properties file for your application name (which unsurprisingly uses the ApplicationClassName.properties as a name) and say something like this:

NoHtmlValidator=HTML is not allowed for ${label}.

Now any time the NoHtmlValidator issues an error in the application, if there isn't a more specific error for that field, the generic "HTML is not allowed for ${label}." is used. ${label} is substituted with the name of the form component.

But Nick, you say, sometimes I use weird names for my components, and I would rather send a more friendly version of the name to the end user! For example, I might want to say "the headline" instead of "headline." Wicket provides for this too. Inside your PageClassName.properties file, you can specify user-friendly names for your components:

formName.componentName=friendly name

For example:

articleForm.headline=the headline

Now that we've created the error, how do we get it on the page? There are two good ways. You can either put all your feedback at the top of the page, or you can put your errors adjacent the form components to which they're related.

The first method is the easiest. In your HTML page, you create a span or a paragraph with a wicket:id as a placeholder for your error messages. For example:

<span wicket:id="messages">Errors and messages go here</span>

Then you add that component to your page. Inside your page class:

add(new FeedbackPanel("messages"));

That's really all you have to do. When a validator calls error(component), the error will show up in a list where your placeholder is in the HTML.

But suppose you'd rather put the headline errors with the headlines. This turns out to be fairly simple as well. There's a subclass of FeedbackPanel called ComponentFeedbackPanel. As you might imagine, you need to put a span or paragraph tag next to your component in the HTML and give it a wicket ID, just as you would with a regular FeedbackPanel. The only difference is that you're putting it where you want it to appear.

Next, add the ComponentFeedbackPanel to your form. Here's how all of this stuff looks in my code:

  TextField headline = new TextField("headline", new PropertyModel(article, "headline"));
  headline.add(new NoHtmlValidator());
  headline.setRequired(true);
  headline.add(StringValidator.lengthBetween(1, 80));
  add(headline);
  add(new ComponentFeedbackPanel("headlineErrors", headline));

That's all there is to it. The messages now show where they belong.

One other thing I discovered is that the RequiredValidator is deprecated in favor of setRequired, but a field that fails to pass that validation still uses the resource text for RequiredValidator.

Want that error message to show up in red? Add this to your page's CSS:

.feedbackPanelERROR { color: red }

Labels: software

Everything was going so well... I could make Wicket say "hello world" and get Sitemesh to decorate the page into my style, and then I thought, "now I should try a simple form!" That way, I could fail fast if form handling wasn't going to work, and I could learn what the "Wicket way" is for handling form submission and validation.

Since the first thing I want to replace on morons.org is article submission, I decided to come up with a simple article form. After one tiny hitch (if you include a wicket:id tag in the HTML, you must define a corresponding element in your WebPage; this is a good thing because it means you can't forget to define everything properly) I got the form displayed.

Then disaster struck when I hit the submit button! Not only did the processed form not render, but only a small amount of the decorator showed up. What the hell? Not only that, but hitting the Reload button made the page display as-expected!

After much digging, it came to light that there was a problem with the Content Length. The actual number of bytes in the HTTP Response buffer did not match the number of bytes it claimed to hold, by a difference of about 9KB. As it happened, the size it claimed to have roughly matched the size of the rendered Wicket form... and the difference was about the size of the decorator. I posted to the Wicket Users mailing list and the Sitemesh forum and went to bed, wondering if I could use my SevenBitFilter (which turns literal UTF8 characters into their HTML entity equivalents so the whole page is 7-bit ASCII) to work around the problem, since the SevenBitFilter would reset the Content Length (because it changes the length in most cases).

In the morning, I was pleased to find the answer in my email. Wicket uses different Render Strategies to deliver a rendered page to the end user. The default strategy is to render to a buffer stored in the user's session, then send the client a redirect to another URL; when the redirect is completed, Wicket then dumps the contents of that buffer. The reason for doing this is apparently to prevent the user from double-posting-- clicking "submit" twice in a row.

For some reason, Sitemesh doesn't like that strategy, though I'm at a bit of a loss to understand why it would notice or care. Probably there is some characteristic particular to a redirect that upsets it.

The solution is to change the render strategy for the application by calling getRequestCycleSettings().setRenderStrategy(IRequestCycleSettings.ONE_PASS_RENDER) from the WebApplication's init() method. This does mean that I'll have to handle double-posts myself, but I was planning to do that anyway, so it's not really a setback.

I'm starting to build up a set of settings that I like for my applications, so I may factor out my own WebApplication parent class (a subclass of WebApplication but a parent class to my other applications) that defines those settings by default next.

I'm also thinking about coding up a servlet filter that dumps the state of the Request and Response objects that can be inserted before and/or after any other servlet filter. Then maybe I could see what it was that caused Sitemesh its intestinal discomfort and provide a more detailed bug report. Unfortunately, development on Sitemesh seems to have all but ground to a halt 14 months ago despite promise of an impending new release. That's a bit worrysome.

Labels: software

Today I decided to divert my attention for a little while from JPA and focus on installing Tomcat, the Tomcat Eclipse plugin and then Wicket and Sitemesh. I decided to use Tomcat for development mainly because of the Tomcat plugin support. Normally I hate Tomcat because of the standard Apache Open Sores way of handling errors: e.printStackTrace(). If I had a nickel for every time Tomcat shat out 300 lines of stack trace because one line was malformed in a JSP, I'd have a lot of nickels. Fortunately, it looks like a lot of the performance problems I'm accustomed to feeling with Tomcat are greatly mitigated in version 5.5.20. It no longer takes the better part of a day to start up, for example... now it's down to about 5 seconds, which I can tolerate. The first page load is still a little sluggish, but nowhere near as bad as it used to be.

My next task was to get Wicket going. For some reason the 'binary' distribution of Wicket 1.2.2 contains all of the source to it, including all of its tests. I don't really need all of that crap. Fortunately, in the root of the zip file is wicket-1.2.2.jar. I wish I had noticed that before I unpacked the entire thing.

It was, quite literally, more difficult to set up the directory structure for my web apps than it was to get Wicket to display my first wicket application. I'm really impressed with how well it works out of the box and how few dependencies it has (basically none). Granted, my first attempt was your typical "Hello, World" application, but I'm really impressed at how easy it was and how I didn't have to wade through 60 exceptions generated by my not knowing what I'm doing. It's really easy to do something simple, just as it should be!

Next it was Sitemesh's turn. I elected to grab the Sitemesh example war file for version 2.2.1. Since I was just experimenting, I dumped the whole thing, except the web.xml file, into my web app root. (I think I forgot to turn on auto-deployment so I just unzipped the whole thing myself.) Then I copied the bits of web.xml that I needed, reordered web.xml (though why Sun decided that the order of web.xml should be significant I'll never understand), restarted Tomcat and ... it just worked! My "hello" now appeared inside the Sitemesh example page.

My only complaint about Sitemesh is its relative lack of thorough documentation. Yes, the basic features all have their basic documentation, but what would really be helpful is a Sitemesh Cookbook, showing common layout problems and their solutions. Maybe if I knew what I was doing, this would be a good project to undertake.

Labels: software

I recently undertook a project to begin sorting out the mess that has become of the morons.org back-end code. Much of it was written when I had little idea what I was doing in Java, and a lot of it was written to the limitations present in Java 1.3 (including the absense of regular expressions!).

I decided to learn from some of my past mistakes and separate out the model code from the forms code from the business logic. As it stands today, some of the model (which loosely follows the DAO pattern) contains code for form validation.... and some of it doesn't. There's code in the JSPs to handle deciding whether a form is needed, whether validaton was successful, and whether inserting/updating/deleting happened correctly as well as dumping the errors to the page should anything go wrong. Basically, the thing's a mess, and I've learned a lot since those days.

For several years at my day job, I've been working with Toplink. Prior to that, I did some work with Hibernate and found it frustrating and not ready for serious work (though in fairness to Hibernate, this was with early version 2 stuff). It required patching just to make it barely functional in our environment. Toplink, on the other hand, is a very mature product, having its roots in Smalltalk in the early 1990's, and its mapping workbench is many orders of magnitude easier to work with than Hibernate's XML configuration files. (I despise XML configuration files passionately. They are hideous to look upon and even harder to read and impossible to work with. Or at least impossible for me. I have no tolerance for that garbage at my age.)

As an aside, the Hibernate documentation once answered the FAQ, "Why doesn't Hibernate have a mapping GUI?" with this:

Because you don't need one. The Hibernate mapping format was designed to be edited by hand; Hibernate mappings are both extremely terse and readable. Sometimes, particularly for casual / new users, GUI tools can be nice. But most of the time they slow you down. Arguably, the need for a special tool to edit mapping documents is a sign of a badly-designed mapping format. vi should be a perfectly sufficient tool.

And if you believe that, I have a nice bridge for sale that I'd like to tell you about. Incidentally, I like vi as much as the next guy, but the statement "vi should be a perfectly sufficient tool" calls into question the very sanity of whoever wrote that answer. Twiddling with XML configuration files by hand is not a reasonable or sane way to manage an object model. Maybe there are some people who get a little ego stroking and feel terribly studly and elite working that way, but I am waaaaay too old for that nonsense.

But I digress.

The main thing holding me back from using Toplink in a restructuring of the morons.org back-end was its rather expensive commercial license. Luckily, EJB3 Persistence, now better known as JPA (the Java Persistence API) achived its first release this year, and the reference implementation was done by Oracle (which bought Toplink a few years back) and is largely Toplink code. Around the same time, FreeBSD got an official binary Java 1.5 release, which made annotations available. Annotations aren't required for working with JPA (you can use... a big honking XML configuration file!), but they're by far the easier and cleaner way.

So it was easy to decide to use JPA to manage the object model. I also chose Wicket to handle the view and controller components of the system. Wicket is surprisingly non-stupid, which is a heck of a lot more than I can say for frameworks like Struts. The trouble with a lot of these MVC frameworks is that for the sake of saving you perhaps two if statements, they introduce 30 lines of convoluted XML configuration with a similarly complex API to go with it. I've always believed that frameworks should save you time. If a framework's design is such that complexity is increased-- that is, you replace a few lines of flow control logic written in Java with a large API with interfaces you must implement and XML configuration you must write-- ostensibly for cleanliness that in the end is not realized, then you haven't got a good framework... you've got an exercise in neurotic time-wastery that would turn off even most obsessive-compulsive autistic people. Somehow that doesn't stop some folks.

But I digress.

I also decided to use Sitemesh to handle the overall site layout, abandoning my old system of a customized XSLT filter. Realistically, there are two main categories of browser that access my site anymore: Microsoft Internet Explorer 6 and Firefox 1.5. There are a pittance of other browsers like Safari, Kmeleon, a couple Opera users and the occasional text browser or HTML fetcher. I get a rare hit from an old version of Netscape. The bottom line is that I no longer see value in designing customized XSLT stylesheets for ancient browsers that rarely hit my site. Also the XSLT is (relatively) slow. Sitemesh offers a way of using different files for merging for IE versus Firefox. Currently Firefox is the only browser I know of that can handle exclusively using DIV for layout and that handles display: block correctly. So I will take advantage of Sitemesh to give table-based layout to IE and DIV-based layout to Firefox. The latter saves a bit of bandwidth and is really the better way to do things.

For my code structure I decided I'd put my model in one package, my form handlers in another package, and a service layer in a third package. One discussion that happens with every new project is "where does the business logic go?" and "what operations will exist in the object model?" The trouble with putting intelligence in your object model is that these things tend to be auto-generated by mapping tools and you have to figure out some way to get this business code re-injected each time you generate the model. Further compounding the issue, many business operations involve complex relationships between multiple classes from the object model; how do you decide which part of the model in which to include these operations? Moreover, do operations of that nature even belong in the object model?

I elected to find the most reasonable compromise I could, which was to put these business operations in a service layer, which acts as an intermediary between things like GUI forms and the data model. In this way, business operations are kept out of the GUI, and things which wish to interact with the model are decoupled from directly operating on it.

(Mind you, this is not a service layer implemented as a web service, though it would be easy to tack on a web service using the service layer classes. Web services are nice for some things, but they would add an unnecessary level of indirection, complexity and latency.)

Having made all of these decisions, it was time to begin. I decided to start with the model, since that was the area where the most things could go wrong. And go wrong they did! The trouble with using a bleeding-edge technology like JPA is that sometimes things are bleeding because they've been cut by a sharp object, which will cut you if you aren't careful.

I decided to try the Dali plug-in for Eclipse. Dali helps you create object-relational mappings by marking up your code with annotations and providing a simple interface for supplying the right annotations and the right data to go in those annotations. Unfortunately, Dali is at version 0.5 and doesn't always play as nicely as it could with Eclipse and persistence. Dali also lacks support for PostgreSQL, a glaring oversight considering the popularity of that open-source database (and the one I use). No matter; it does support generic JDBC. I was able to create entities based on classes in my database, with the caveat that some types did not translate well to native Java types (they probably would have if Dali supported PostgreSQL). For example, a few chars and boolean database types became Java longs. These I edited by hand.

I had to struggle a little with the mapping between an Article and a MessageBoardRoot. In the previous incarnation of the morons.org backend code, entities become associated with their corresponding message boards through a table called "roots" which records the entity's type and primary key. For example, a board with the root_id 23654 might be associated with an "article" with the primary key "6745". The composite of "article" and "6745" is unique and associated to all of the messages bearing the root_id of "23654." I wanted to map this as a 1:1 relationship so I could ask an Article for its MessageBoardRoot. Unfortunately, JPA does not allow you to create a join based on a primary key and a constant; only persisted fields can participate in the join. So it was either include in every single article a field with the word "article" in it, or make every object on the entire system have a unique ID and drop the "root_type" field from the roots table. I decided on the latter; every object on the system would share a single sequence generator and name the primary key the same thing: object_id. This meant I would have to renumber some old data, but I think the change is worth it. I can also still record the creating class name in the MessageBoardRoot objects for the sake of being able to easily locate the corresponding class for administrative purposes later.

Then I tried to persist the simplest record I could- an Account, which corresponds to a user login account. JPA exploded immediately with a NullPointerException:

Exception in thread "main" java.lang.NullPointerException
at oracle.toplink.essentials.ejb.cmp3.EntityManagerFactoryProvider.createEntityManagerFactory(EntityManagerFactoryProvider.java:120)
at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:83)
at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:60)

Ah, I just love the intuitive errors provided by Open Sores code sometimes. After failing to find a conclusive answer with Google, I came to realize that this error occurs when JPA is unable to locate the persistence.xml file, which normally resides inside the META-INF directory of a jar file, but may also reside in META-INF in a directory on your classpath. After a lot of digging, and using FileMon from Sysinternals.com, I discovered the problem: JPA prepends META-INF onto your classpath entries, so if you think you're going to add your META-INF directory to your classpath, you're barking up the wrong tree. You actually need one directory above it.

Normally, Dali would have taken care of this for me by putting META-INF in the right place. What happened was that I had added Java Persistence to the Eclipse project before I added a source folder to the project. Eclipse apparently drops the project root as a source folder after you add your first source folder to a project, which meant that my classpath entry for source went from net.spatula.news to net.spatula.news/src/java, but the META-INF directory did not move along with that change. I elected to just have Dali create a new persistence.xml file for me by removing the persistence nature from .classpath and adding Java Persistence to the project again. This got me past the NullPointerException-- progress!

The next trouble was with sequences. Whereas Dali does not support PostgreSQL, Toplink Essentials (aka Glassfish persistence) does... sort-of. It turns out that sequence generation for primary keys for PostgreSQL is currently broken in Toplink Essentials. If you specify a sequence generator for a primary key field that isn't of type serial, Toplink completely ignores sequence generation and attemps to insert a null for the primary key. The workaround is not to specify a sequence generator and instead set up a table for the default sequence generation strategy. This means issuing the commands

create table sequence(seq_name varchar(40) not null primary key, seq_count integer not null);
insert into sequence values('SEQ_GEN', 1);
commit;

I anxiously await Toplink correcting sequence generation for PostgreSQL.

My next challenge was a new exception: "cannot persist detached object." In my Account model, an Account has a member called "changer" which is also an Account. The idea is that whoever caused the change in the model will be recorded as the changer. (I use triggers on the database to keep an audit trail for every record change in the account table.) The mapping is fairly straightforward- the changer field maps to the object_id field in the same table. I had created a row in the account table for an account called "System" with an object_id of 0 to bootstrap the table and handle initial inserts. It turned out that my choice of object_id for this special account was the culprit in the exception. Dali mapped the object_id as a Java native type long rather than an object type Long. This means that the only way JPA can tell whether an object is new (has a primary key) is whether the primary key is non-zero, since native types default to 0 (except booleans which default to false). JPA thought that I was trying to persist a new "changer" object by reachability because the primary key was set to 0. I haven't checked yet, but I suspect that were the primary key mapped as an object rather than a native type, that determination would have been done based on whether the reference was null, and 0 would then be a legitimate primary key. The solution was to give System an object_id of 1, rather than 0.

I also discovered that JPA has one weird deficiency: there's no way to annotate that an entity is read-only. This is desirable in situtions like the audit trail example above; one might want to read these audit records but prevent the developer from accidentally writing back a change or inserting into the audit table directly. One partial workaround is to specify that the primary key field is read-only.

After all of this, I managed to get a row inserted in the database. The next stop: getting it back out again.

Labels: software