Tomcat JSP Precompilation is SLOW!!

Here at Rojo we have recently started precompiling all of our JSPs to improve both the speed and reliability of our system on production servers. We have about 200 JSP files, which is a large web app but not necessarily gargantuan. The entire JSP compilation process, using Ant and the jasper2 ant task, takes about 30 minutes! This is far too long. I mean, there are systems that are doing more than just compiling text, such as advanced 3D graphics and speech recognition, that are far faster than Tomcat's JSP compilation.

I've been working on solving this problem, but to no avail, which is why I'm posting this to see if there is a solution in the blogosphere. Here's what I've done so far:
  • Switched from JspC to Eclipse's JDT compiler by moving from Tomcat 5.0 to Tomcat 5.5.4, which replaces JspC with JDT. This resulted in no real speed improvement but JDTool is less buggy, which is nice.
  • Thinking that maybe the JSP precompilation process was IO bound, I setup a 400 MB ram disk on Linux, moved our entire source tree onto that, and setup the JSP precompilation process to write into the ram disk during its operation. This resulted in no real performance improvement.
  • I configured the JSP compiling process to use Jikes as its compiler, but unfortunately Jikes produced a long string of errors when attempting to compile our source code. Jikes might improve the performance, but I was not able to find out because of the difference in compilation between Jikes and the Sun JDK.
  • I attached JProfiler to a running JSP precompilation process to see if there are any significant hotspots. I was expecting to find a single method that took a majority of time, such as 70% of the time was spent in a single method, due to an unoptimized data structure or tight loop; if I found this I would then be able to fix it. Unfortunately, most of the time was spent amongst several methods; there was one method that consumed 20% of the time, but spending the time optimizing and only getting a 20% improvement was not enough for me; I need more bang for the buck since engineering time is tight here. The results might be useful for Tomcat JSP developers, so I've generated some HTML reports from JProfiler: Hot Spots, Invocation Tree, Method Graph, CPU Statistics
So, I'm at a loss; does anyone out in the blogosphere have any ideas on how to move forward on this issue?

Comments

Gordon Mohr said…
Just some thoughts...

Are you compiling just the 200 JSPs, or a bunch of other code at the same time? How much total JSP content (in KB) is involved? If you alter the script to only compile 100 of them, does the time taken decrease by half?

I might suspect something wrong in your build script that causes the same work to be done many times redundantly. (Should there be 70 invocations of Compiler.compile?) Or, something amiss with the build machine configuration pushing it into swapping the whole time. Is there a chance any part of the build process requires access to network resources (loading classes by net) that might be laggy?

(What is the build machine CPU/RAM? What do 'top' and various options to 'sar' show during the 30-minute build -- is the CPU maxed?)
Brad Neuberg said…
Gordon, great comment. In response to your questions:

> Are you compiling just the 200 JSPs, or a bunch of other code at the same time?

Well, the Ant task compiles other things before reaching the JSP compilation point. Those tasks execute quickly; its the JSP compilation task that is the slow down.

> How much total JSP content (in KB) is involved?

Good question; I don't have an exact answer, since I'm responding to this from home, but in general I've been very surprised at the size of our generated JSP files; it seems like many of them are 1.5 to 2 MBs in size! We use lots of JSP includes, which seem to replicate the includes across every JSP rather than generating a single class that might be shared across all JSPs.

> If you alter the script to only compile 100 of them, does the time taken decrease by half?

Good idea; I will try that on Monday to see if the time is proportional to the number of JSPs

> I might suspect something wrong in your build
> script that causes the same work to be done many times
> redundantly. (Should there be 70 invocations of
> Compiler.compile?) Or, something amiss with the build
> machine configuration pushing it into swapping the
> whole time.

The 70 invocations of Compiler.compile are Java invocations rather than entire re-invocations of the JVM, so they shouldn't be too bad. We shouldn't be going into virtual memory and therefore swapping from the JProfiler results; the process doesn't seem to take that much memory since the total heap doesn't seem to be garbage collected continually. I will attach VisualGC to the process on Monday to get some more indepth results on the structure of the heap (permament generation, tenured generation, and eden generation), but my feeling is the process isn't necessarily memory bound and therefore isn't forcing things into virtual memory. We are also not using anything across the network, such as mounted drives through NFS.

> What is the build machine CPU/RAM?

Unfortunately I can't go into specifics on our machine environment, but the machine is definently loaded with enough memory and CPU power.

> What do 'top' and various options to 'sar' show during the 30-minute build -- is the CPU maxed?

Good point; I'll try running top and sar to observe the CPUs process as JSP compilation occurs.

Thanx for the help. By the way, is this Gordon from the P2P Dim Sums?
Gordon Mohr said…
Yep, same Gordon.

Wasn't so worried about the 70 invocations meaning 70 JVM launches, but since 70 != 200, it could be a symptom more work is being inadvertently repeated by the build script than necessary.

Multi-MB JSPs sound atypical and worthy of investigation (and possibly refactoring!). Are you using jsp:include or %@ include?
b0b0b0b said…
Unless it needs to be a clean build, have your build script leave intermediate files around where it can re-use them if it detects the sources aren't dirty.

If you do that, preserving generated .java and compiled .class files, then you just need to make sure that timestamps are rolled back on jsp's whose contents haven't changed.

And if you do a static import, tomcat should be smart enough to check timestamps on imported resources.
Unknown said…
We'd love to help. Have you tried Tomcat 5.5.12 yet? It includes an improvement to precompilation of many JSP files that was done as a Google Summer of Code project. Give it a shot and let us know (dev@tomcat.apache.org) if things have improved.
John said…
Did you ever figure this one out? I encountered this when randomly looking for something else, and I have to tell you, I routinely compile 500+ .JSP files and it takes less than 5 minutes. This is with Jasper2 shipped with Tomcat 5.0.28 on a 2GHZ machine with 1 GB RAM.

I'm really curious about this; e-mail me (I think my posting should reveal my e-mail address) if you have more information. I'd be glad to share with you my ant build script if it would help.