Concurrent Modification Exception

I ran into a ConcurrentModificationException (CME) during stress testing.
What does CME actually mean?
It means that you've modified (add, remove, update) your Collection while you've been iterating over it (usually in a multi-threaded fashion, but it can occur in a single thread that modifies while iterating).

A few more things to note about CME:
Best effort detection - If you see a CME printout, first off, consider yourself lucky, CMEs are thrown only in best effort. In another universe, the concurrent modification would not have been detected, causing your collection to become corrupted, instead of fast-failing with a CME.

IDing the problem - Like deadlocks, CME's are easy to pinpoint once you inspected the exception's stack trace.

Avoiding CME:

1. ListIterator
   To modify a collection by the same thread that is currently iterating on it, use a ListIterator that will allow you to perform both.
   Drawbacks - single thread solution only.


2. Naive solution: Synchronizers
   Use locks to for mutually excluding traversal and modification operations.
   Advantages - easy to code.
   Drawbacks - very long lock periods while iterating.


3. CopyOnWrite
   Take advantage of the Java.util.concurrent collections like: CopyOnWriteArrayList, CopyOnWriteArraySet. If you require a map then grab CopyOnWriteMap from Apache (this guys have been doing Sun's dirty work for years now).
   Advantages - very good reading performance (no locks are used, instead visibility is obtained via map member volatility).
   Drawbacks - very bad write performance on large maps.
   Conclusion - use for seldom mutating collections.


4. toArray()
   toArray will create a new array holding a copy of your Set (Map.keySet() for a Map).
   You can then iterate over the array, freely modifying the original collection (the array doesn't change of course).
   Advantages - write operations are cheap.
   Disadvantages - copying the entire set could be expensive if it occurs too often, and/or the set is very large.


5. Concurrent Collections
   If you want to go heavyweight, consider using: ConcurrentHashMap (or one of its package friends).
   Once you create an iterator over a ConcurrentHashMap (CHM), it does not freeze the collection for traversal, updates to the collection may or may not appear during the traversal (weakly consistent).

The approach I ended up taking
My use case was seldom modifying a ~ten items cache. A copyonwrite map was what I used.
In other cases I had, ConcurrentHashMap was the easiest solution (though make sure your code can live in peace with the CHM weak consistency property).

[caption id="attachment_227" align="alignleft" width="200" caption="Best pic idea I could think of to visualize Threads :)"][/caption]

Why catch Throwable is evil - A real life story

Disclaimer: Now I know that this is an old idiom, I'm just presenting my own real life incident taken straight away from the bloody Java trenches.

Exceptions can be threads assassins
when running on top of Websphere thread pool, any Runtime exception that isn't caught by the applicative code, will bubble up in the stack, ending up killing the specific thread. WAS helps here, by automatically creating a new thread that will take the place of the murdered one, but still, killing and immediately creating a thread is everything but the thread pool rational.

Hiring a thread bodyguard
A simple way to avoid thread death is wrapping the first applicative layer (e.g., Run() method) with a try block that catches and swallows any Exception that's thrown from anywhere in the application code.
Our project's code also used this concept, but instead of catch (Exception e), it had a catch (Throwable t), When I noticed that I didn't rushed to fix it, just in case someone before me had done funky stuff with dynamic class loading that might throw ClassNotFoundError (although this should be caught at a very localized resolution), or maybe it's there for some other historical reason that not being one the code's forefathers I’m just not aware of. In any case, I did promise myself that I'll revisit this piece of code in the future.

Getting some bulls to do correct things
today I finally got the excuse I needed in order to change the catch Throwable in a catch Exception:
We were running stress tests, when the server had an OOME (out of memory error). Since the catch Throwable caught and swallowed the OOME (as OOME is a subclass of Error which is a subclass of Throwable), the thread that generated the OMME kept on living, instead of dieing right there, and so, the JVM continued running, crippled and limping, instead of turning to an honorable solution like hara-kiri. Choosing the quick death route would have been rewarded with a quick resurrection to be provided by the gracious NodeAgent and its watchdog mechanism, and the end result would have been a newly born healthy server ready to get back in business. A retreat in order to attack, you might put it.
Instead, the server had to limp for long minutes, suffering from a series of consecutive strokes (OOME), until the OOME was so bad that the JVM just had to exit.

Conclusions
The Catch Throwable was causing down time, by preventing an imminent restart of the JVM due to an OOME.

Open Questions

1. I know that an uncaught exception kills only the specific thread does the JVM treats an error differently? Put other words, if the OOME is not caught, will the entire JVM die or only the specific thread? I assume that the answer is the entire JVM, maybe this is implemented by the JVM itself, or maybe it's implemented somewhere in the WAS bedrock. If for some reason it's not the case, one could catch an Error and then execute System.exit(1); in order to hasten the process imminent death.