One problem with 64-bit Java is the amount of Memory it occupies. As of 2011 we feel that all application profiles we have encountered do not require 64-bit Java as they often do not take advantage of using more than than 1-2GB of memory. Also, we have observed that when a Java Application has access to a large amount of memory, the application gets bogged down when garbage collection occurs.
This has improved due to newer algorithms such as concurrency, parallelization and generational collection. However we observe there is still significant impact.
As such, in many cases it is better to scale horizontally (running multiple JVMs) as opposed to using one giant JVM.
Just to be clear, this is not to say not to user 64-bit servers. In fact we are moving to use entirely 64-bit systems these days most importantly to take advantage of all the memory available (most of our servers use over 4 GB).
More details explaining the mechanics can be put here or linked in another document.