In the example, the updateBatchStatus method makes a "RequiresNew" EJB call to actually update the batch_status database table so the status change is immediately visible, even though the effects of the current transaction aren't yet visible. The call to executeUpdate isn't an EJB call, so it executes in the same transaction as the rest of bulkLoadData.

As described, this code will cause a deadlock even in the absence of concurrency. When bulkLoadData calls the executeUpdate method, it updates an existing database row, which involves acquiring a write-lock on that row. The nested EJB call to updateBatchStatus will execute on a separate database connection and try to execute a very similar query, but it will block because it can't acquire the necessary write-lock. From the database's perspective, this second connection will be allowed to proceed as soon as the first connection's transaction is committed or rolled back, which is why the database doesn't detect this as a deadlock. The VM, however, won't allow the call to bulkLoadData to complete before each call to updateBatchStatus completes, so we have a deadlock.

This example shows one update blocking another update, so it causes a deadlock on any database. If the initial update query was a simple select query instead, this example would only cause deadlocks on databases that use lock-based concurrency control, where one connection's read-lock can block another connection from acquiring a write-lock. In any case, a deadlock of this type is neither timing-dependent nor load-dependent, and the thread dump will show one Java thread waiting for a database response, but that thread will be associated with two active database connections. And one of those database connections will be idle, but blocking the other connection.

This scenario has many subtle variations, where more than one thread and more than two database connections may be involved. For example, the outer EJB call's database connection may have acquired a database lock that blocks another unrelated database connection from proceeding, but that unrelated database connection has acquired a lock that blocks the nested EJB call's database operation. This particular case is timing-dependent, and will show several Java threads waiting for database responses. At least one of those Java threads will be associated with two active database connections.

Cross-Resource Deadlock #3: JVM Locks Crossed with Database Locks
A third deadlock scenario occurs when mixing database locks with JVM locks. In this case, one thread holds a database lock and is trying to acquire a JVM lock (trying to enter a synchronized block), and another thread holds that JVM lock and is trying to acquire a database lock. Again, the database sees one connection blocking another, but nothing is stopping that connection from proceeding, so it won't detect a deadlock. The JVM sees one thread inside a synchronized block and another trying to enter, so even if the JVM could detect deadlocks and handle them somehow, it wouldn't detect this case.

For an example that illustrates this deadlock scenario, consider a simple (flawed) read-through cache. This cache is a HashMap backed by a database table. If a cache hit occurs, it will simply return the value from the HashMap; on a cache miss however, it will read the value from the database, add it to the HashMap, and return it as shown in Listing 2.

It's a simple read-through cache. Note that the get() method is synchronized, since we access a non-thread safe container, and we want the containsKey/put combination to be atomic in the case of a cache miss.

This cache is fairly straightforward: the contract is that if we change the data in the table backing the cache, we should call clearCache(), so the cache can avoid handing back stale data. The resulting cache misses will repopulate the cache appropriately.

Let's now consider some code that changes this data and clears the cache:

public void updateData(String key, String value) {
    executeUpdate("update cache_table set value='" + value +
       "' where key='" + key + "'");
    SimpleCache.getInstance().clearCache();
}

In a simple case, this will work with no problems. However, on a database that uses lock-based concurrency control, the query in updateData will prevent the select query in queryForValue from executing, because the update statement will have acquired a write-lock that prevents the select query from acquiring a read-lock on the same row. If the timing is just right, one thread can try to read a given value out of the cache and get a cache miss while another thread updates that value in the database. If the database executes the update statement first, it will block the select statement from continuing. However, the thread executing the select statement came through the synchronized get method, so it has acquired the lock on the SimpleCache. For the thread in updateData to return, it has to call clearCache(), but it can't acquire the lock (clearCache() is synchronized).

When dealing with an instance of this scenario, there will be one Java thread waiting for a response from the database, and one waiting to acquire a JVM lock. Each thread will be associated with a database connection, with one connection blocking the other. The fix is to avoid doing database operations while holding JVM locks: the get() method of our SimpleCache could be rewritten like this:

public Object get(String key) {
    synchronized(this) {
       if (cache.containsKey(key)) {
          return cache.get(key);
       }
    }
    Object value = queryForValue(key);
    synchronized(this) {
       cache.put(key, value);
    }
    return value;
}

Since we now know that this deadlock case can happen, we can add a check to our queryForValue method to try to avoid the deadlock condition by using Thread.holdsLock():

private Object queryForValue(String key) {
      assert(!Thread.holdsLock(this));
      return executeQuery(...);
}

While Thread.holdsLock() can be useful in this case, it only works if we know which locks we're specifically worried about. It would be useful to have a similar method that could determine whether the current thread holds any JVM locks. Then, any piece of code that made any kind of RPC call, database access, etc., could throw an exception or log a warning to indicate that these operations can be dangerous while holding a JVM lock.

Note: even though we've fixed the deadlock problem in this example, it's still flawed because the cache is cleared before updateData's transaction has been committed. If a cache miss happens after the clearCache call but before updateData's transaction is committed, the cache will load the old data because the new data isn't visible yet. The fix here is to clear the cache only after the changes have been committed. Observe that this would only happen in an MVCC database; in a lock-based database, the pending update would block the cache's read, so the cache would read the correct value after the update's transaction was committed.

Rules of Thumb
These guidelines should help you avoid these problems, or at least diagnose and fix them if they do happen.

• Keep your transactions short and simple.
• Understand the locking behavior of your database (and your transaction isolation level).
• Assume any database access has a chance of failing with a database deadlock, and retry correctly.
• Don't update any non-transactional state (in-memory state, caches, etc.) until after the transaction has completed.
• Make sure resource pools are large enough for your peak concurrency.
• Try not to acquire multiple resources at the same time. If you must, acquire them in the same order each time.
• Learn how to get a full thread dump from your application server and a list of database connections from your database (including which connections are blocked by each), and how to know which Java thread each active database connection is associated with. The easiest way to understand the mapping between Java threads and database connections is to add some logging to your connection pool access code.
• When making nested EJB calls, understand which ones will use the same database connection as the caller, and which ones will use new connections. Even if the nested call runs in the same global transaction, it may still use a different database connection, and this can cause cross-resource deadlocks.
• Avoid making database calls and EJB calls, or doing other off-JVM operations while holding JVM locks. Use assert(!Thread.holdsLock(...)) if there are specific JVM locks you're concerned about to prevent future code changes from violating this rule unintentionally.

Resources

• How to get "RequiresNew" and "NotSupported" transaction behavior when using bean-managed transactions (BMT): www.onjava.com/pub/a/onjava/2005/07/20/transactions.html
• Database deadlocks in J2EE applications: www.theserverside.com/patterns/thread.tss?thread_id=14482
• WebSphere documentation's discussion of scenario #1: http://publib.boulder.ibm.com/infocenter/wasinfo/v5r1/ index.jsp?topic=/com.ibm.Websphere.base.doc/info/aes/ae/cdat_conpool.html
• WebSphere discussion of connection sharing across EJB calls: www-128.ibm.com/developerworks/Websphere/library/techarticles/0404_tang/0404_tang.html