Why catch Throwable is evil - A real life story

Disclaimer: Now I know that this is an old idiom, I'm just presenting my own real life incident taken straight away from the bloody Java trenches.

Exceptions can be threads assassins
when running on top of Websphere thread pool, any Runtime exception that isn't caught by the applicative code, will bubble up in the stack, ending up killing the specific thread. WAS helps here, by automatically creating a new thread that will take the place of the murdered one, but still, killing and immediately creating a thread is everything but the thread pool rational.

Hiring a thread bodyguard
A simple way to avoid thread death is wrapping the first applicative layer (e.g., Run() method) with a try block that catches and swallows any Exception that's thrown from anywhere in the application code.
Our project's code also used this concept, but instead of catch (Exception e), it had a catch (Throwable t), When I noticed that I didn't rushed to fix it, just in case someone before me had done funky stuff with dynamic class loading that might throw ClassNotFoundError (although this should be caught at a very localized resolution), or maybe it's there for some other historical reason that not being one the code's forefathers I’m just not aware of. In any case, I did promise myself that I'll revisit this piece of code in the future.

Getting some bulls to do correct things
today I finally got the excuse I needed in order to change the catch Throwable in a catch Exception:
We were running stress tests, when the server had an OOME (out of memory error). Since the catch Throwable caught and swallowed the OOME (as OOME is a subclass of Error which is a subclass of Throwable), the thread that generated the OMME kept on living, instead of dieing right there, and so, the JVM continued running, crippled and limping, instead of turning to an honorable solution like hara-kiri. Choosing the quick death route would have been rewarded with a quick resurrection to be provided by the gracious NodeAgent and its watchdog mechanism, and the end result would have been a newly born healthy server ready to get back in business. A retreat in order to attack, you might put it.
Instead, the server had to limp for long minutes, suffering from a series of consecutive strokes (OOME), until the OOME was so bad that the JVM just had to exit.

Conclusions
The Catch Throwable was causing down time, by preventing an imminent restart of the JVM due to an OOME.

Open Questions

1. I know that an uncaught exception kills only the specific thread does the JVM treats an error differently? Put other words, if the OOME is not caught, will the entire JVM die or only the specific thread? I assume that the answer is the entire JVM, maybe this is implemented by the JVM itself, or maybe it's implemented somewhere in the WAS bedrock. If for some reason it's not the case, one could catch an Error and then execute System.exit(1); in order to hasten the process imminent death.

My attempts with IP Spoofing

Why did I wanted to spoof source IP addresses? and why did I failed? Here's the story before you:

------------
UPDATE Sep/2010: Dear Filipe (see comments below) had proven to me that spoofing over the internet is indeed possible, read all about it on the continuation post: My attempts with IP Spoofing – Revisited. Now back to the original story:
------------

When customers install our product, they often forget to setup firewall rules to accept incoming connections from public IM (instant messaging) providers. Without the firewall rules in place the product does not function properly, of course, and the customer rushes to open a support trouble ticket. Troubleshooting to pinpoint the problem to a missing firewall rule isn't trivial. When we try to validate whether the customer defined the required firewall rule, we need the external entity (that we have no control on) to open a connection to the customer's IP, but the external entity will only do so following the successful completion of a handshake sequence that must be initiated by the customer (consider for example: XMPP Dial-Back mechanism), since this handshake by itself is prone to failures, you can see how reproducing the problem is a combursum process.

I started looking for a simple, independent, and reliable, troubleshooting procedure that would be able to give a clear-cut answer to whether or not the customer defined the firewall correctly.
Here's what I've concocted:

1. Assume that the customer IP is 1.1.1.1 and they were suppose to configure their firewall to allow incoming connections from 2.2.2.2.


2. I'll send a single TCP SYN packet (the 1^st of the standard three messages TCP handshake) from my computer (say it's IP is 9.9.9.9), but I'll spoof the IP datagram's source address field to be 2.2.2.2 instead of what normally should have been my actual machine address (9.9.9.9).


3. I'll ask the customer to run a network sniffer on the IM Gateway machine. Waiting for the single packet to arrive at the destination socket.


4. If the sniffer had recorded the incoming IP message, then it means that the firewall is setup correctly and the problem is else where.
   But, If the sniffer didn't record any incoming SYN packet, then we shell blame the firewall guys.

Pretty simple, eh? Now, in order to spoof the TCP SYN packet I needed a something that could generate and send raw IP packets, since you can't just fiddle with the source IP address if you choose to ride on the good'ol TCP/IP stack. I found this IP spoofing perl script on the net, and it does the job.

[caption id="attachment_103" align="alignnone" width="300" caption="Visualization of the various routes through a portion of the Internet. Took it from Wikipedia."][/caption]

I did my first test on the office LAN, I sent a message from machine (IP 9.9.9.9) to to machine 1.1.1.1 claiming the message source was 2.2.2.2, it worked! Machine 1.1.1.1 registered an incoming packet from 2.2.2.2.
It seems that the office router went along with the scam, perhaps it thought that the machine switched IP it IP, or the DHCP server went crazy, or that it's ARP cache is just stall.

In the next test I tried sending the packet over the Internet, I tried sending a packet to my home computer from the office, with a source IP of some foreign entity, to my dismay, it never got to my home computer. Other IP variations didn't work either.
My guess is that some router along the way noticed that it's getting a packet with a source IP address that the part of the network it is looking can't can't possibly generate (imagine CIDR based ACLs), and that caused it to immediately drop the packet. This failure caused me to give up on the whole spoofing troubleshooting procedure idea.

Some thoughs about what I've seen:

1. Evidently, It's quite trivial to spoofe IP addresses on a LAN.


2. Spoofing  IP addresses over the Internet doesn't seem to be trivial.


3. A side note: If the customer has a reverese proxy, or any form of entity that delegates TCP handshakes, deployed before the actual IM Gateway machine, then the procedure is not applicable, as the first TCP SYN message will never reach the IM Gateway machine.


4. I would assume that the closer you inject the packet into the Internet backbone blood stream, the better the chances of not getting a rejection of the spoofed packet. The backbone routers communicate with many difference parts of the network, and might not have rational of where certain packates should be coming from or not.
   IP Packets tend to travel in different routes, making it harder to judge what IP CIDR is ligit from each fellow router.


5. I'm guessing that the biggest problem for spoofing is the first or the second router (the ISP's), since the ISP knows exactly what is your assinged address. Thereby knowning that the packet is spoofed.


6. If any one knows a better method of spoofing source IP, please step forward and share your secret :)

Increasing the site's posting rate - new paradigm

I've been promising myself to post and publish much more frequent than the current rate of publishing.

Except from reserving more time for the actual posts authoring. I'm also counting on changing the nature of the post as a primary means of increasing the posts rate to once a week or more. From now on, I'll publish stories from my day-to-day work as a software developer, interesting technical things I come across, questions that I don't always have an answer for, general discussion, etc. The post will be less educative, less articles like, less accurate, with less checked facts, but on the other hand, much more real life related, much more up-to-date, and presented in an open discussion inviting format.

So, I'll be writting to you soon as promised :)

Saving on memory usage in Java #1 - the Byte.valueOf method

Say you wanna keep in memory a list of martial arts experts and their respective shoe size. One way to implement it would be to populate a Map structure with the following sets of key and value:

Map map = ...
map.put("Jean-Claude Van Damme", new Byte(45));
map.put("Jet Li", new Byte(45));
map.put("Chuck Norris", new Byte(112));
...
map.put("person number million", new Byte(45));

What if your JVM runs on a Lego mechanical computer that has a very limited amount of memory, you would probably want to save on memory wherever possible.

[caption id="attachment_65" align="alignnone" width="300" caption="A state of the art 3Hz Lego computer"][/caption]

Autoboxing anybody?

Keeping in mind that an object instance weights much more than just the primitive it holds, as it hold additional "plumbing" data (monitor, etc). Even an Object class instance weighs 8 bytes while not holding to any application information. What about keeping only primitives as the map value?
Autoboxing, introduced in Java 5 onwards, allows to pass a byte primitive argument instead of a Byte object instance argument in the following manner:

map.put("Bruce Lee", 42);

Does this help us avoid the costly Byte Objects? Not really, the auto-boxing feature, as the name hints, just statically replaces the 42 literal with a new Byte object instance, this is done during compilation. So there's no real saving opportunity here, and we're back where we started.

[caption id="attachment_66" align="alignnone" width="300" caption="AutoBoxing"][/caption]

How about a plain old cache?

Examining the code above, you notice that you are creating one million unique Byte objects to hold the fighters' shoe size, even though there are only 256 different shoe size values. Is this a venue for saving?
Considering the fact that Byte objects are immutable, why not have just a single Byte object for each distinct byte value (we'll need only 256 instance to cover all values). This way we'll pass the same Byte instance to all people with a 45 shoe size, Jean-Claude and Jet-Li map in our case. This will reduce the number of Byte instance from a million to only 256. Sounds super!

[caption id="attachment_67" align="alignnone" width="201" caption="Too much objects in memory"][/caption]

How do you implement this? You'll might rush into initializing an array of 256 Byte objects during application start-up, giving birth to something of this sort:

// init instances array
int RANGE_OF_VALUES = 2^8; // we don't care about negatives
Static Byte constShoeSizes = new Byte[RANGE_OF_VALUES];
for (byte b=0; b<RANGE_OF_VALUES; b++) {
constShoeSizes[b] = new Byte(Byte.MIN_VALUE + b);
}
map.put("Jean-Claude Van Damme", constShoeSizes[45]);
map.put("Jet Li", constShoeSizes[45]);
map.put("Chuck Norris", constShoeSizes[112]);


Enter the valueOf() method

WHOA! Hold you horses! Doesn't this use case seems to be just too common and trivial?! haven't the Java language designers and implemented came accross the same problem? Surely, some of the JRE classes themselves must have Byte instances data members. In an effort to reduce the JRE memory footprint, won't the JRE programmers cache instances using something very much like the static Byte array we implemented ourselves?
The short answer of course is YES! Java 5 presents a new overloaded Byte.valueOf(byte b) method. This method returns a reference to a Byte instance taken from a shared cache. This trivial cache strategy save memory and CPU, as there's no need to construct new objects and later on garbage collect them.
Here's the relevant Byte.valueOf method source code taken from Byte.java source:

private static class ByteCache {
private ByteCache(){}

static final Byte cache[] = new Byte[-(-128) + 127 + 1];

static {
for(int i = 0; i < cache.length; i++)
cache[i] = new Byte((byte)(i - 128));
}
}
...
public static Byte valueOf(byte b) {
final int offset = 128;
return ByteCache.cache[(int)b + offset];
}

Using the valueOf method, here's how the final version of our code will look like:

map.put("Jean-Claude Van Damme", Byte.valueOf(45));
map.put("Jet Li", Byte.valueOf(45));
map.put("Chuck Norris", Byte.valueOf(112));


Wrapping up quickly:

1. From Java 5 onwards, use the valueOf method for Number extenders like: Byte, Short, and Integer. Notice that as the Integer object has 2^32 different values, only the (-128) to 127 values range is cached. Meaning that expression (Integer.valueOf(129)==Integer.valueOf(129)) will always be false, since it returns a new Integer object on every call.
   Other object types (Double,Float, etc...) valueOf method does not implement a cache at all. If your value range is limited in nature, you might choose to create a caching scheme of your own.


2. Always be on the lookout and Inspect repetetive Instance creation closely, see if you can avoid it by referencing an shared immutable object, or by borrowing an instance from an object pool.


3. Strings can have an even larger space and time performance gains than numbers objects, though at the same time they are inherently harder to reuse. You might want to take time to learn about Strings instances reuse strategies; start with the String.intern() method.

Java Podcasts - tune in to the Java posse

I haven't wrote anything new for a long time now, part due to spending most of October vacationing in Thailand, part due to the massive work load that landed at my doorstep, when returning from Thailand islands.

Until I'm finished with clearing my desk, so I could get back to write here, I thought I might leave you with some quality Java development babbling, to help drive a way the boredom of the daily commute.
Now, these four dudes must have a lot of free times on their hands, delivering an hour long quality podcast every week. Check the Java posse here.

Miniature scale pro-active IT - Creating windows services dependencies

Yesterday night we had a scheduled power shutdown in the Dev lab. Today morning, the lab manager rose up early to get all servers running before the armies of developers arrive to the office. My work includes interacting with a Lotus Sametime server (IBM's Instant Messaging (IM) server), so I run my own private IM server. starting the day's work, I quickly noticed that my IM client fail to log in to the IM server. In fact, all of the developers could not log in to their own private IM servers.
Remembering that during the client log in process the IM server validates the client supplied log in credentials against a central LDAP server, the LDAP server became an immediate suspect.

The LDAP server we're using is an old IBM ITDS LDAP server running on Win2000. It's comprised out of two processes: the ITDS process that parses and execute LDAP queries, and a DB process (DB2) that takes care of data persistency. Both processes are registered as Windows services.

The investigation commenced! Maybe the LDAP server is down? I went a head and checked the ITDS and DB2 services status, both were running. Hmm... I moved on to inspect the ITDS log, and saw that during its start up stage it failed to create a connection to the DB2, therefore it resorted to starting in a, "crippled", configuration only mode. That means that it just sits there, wasting random CPU cycles, giving the illusion that it's there to provide service, but not actually answering any queries.
To remedy the situation I simply re-started the ITDS service. It started up normally and began servicing incoming LDAP queries from the IM servers.
At this point, you'll be tempted to announce world wide: "I fixed it!", but before you do that, stop and think about it for a minute; what is it exactly that you fixed? In did, the ITDS began servicing queries, and the client can log in, meaning that the current manifestation of the problem was eliminated, but did you fix the problem itself? Part of being pro-active means that you solve future problems before they actually occur. What stops the problem from re-occurring the next time someone decides to restart the server? in order to solve it for good, you first need to understand what was the cause of the problem.

So, what happens during a server start up? The ITDS and DB2 services startup-type is set on Automatic, thus they start when the OS starts. The db connection error message fits a scenario in which the ITDS service started and tried to connect to the DB2 before the DB2 service finished starting up.
We would like to instruct the ITDS service to be less hasty, and wait for the DB2 service to finish starting, before stepping into itself start up process. Educating it can be achieved by defining a service dependency, stating that the ITDS service is dependent on the DB2 service.

Implementing it: Dependencies can't be created using the windows MMC "computer manager" snap-in GUI, so you'll have to get your hands dirty with registry mud using the following procedure.

Problem uprooted! You won you pro-activity badge.

Google I/O 2008 - Josh Bloch talk - Effective Java 2nd edition

Now days, technology eager and innovation craving programmers can find abundant amounts of learning material online, served in an easy to swallow and digest form, such as video casts. One of the most prolific sources of technical info is Google, which now posted video sessions from the "google I/O", may 08, dev gathering.
In this video Josh Bloch (formally at sun) gives an hour long session about his new second edition of the Effective Java book. I found the session to be only somewhat interesting (Enum sets are not my main point of interest), plus the video quality is not ideal for reading through source code.
The discussed book is a well gathered compilation of 78 Java best-practices (although, to be honest, I've only read about 20% of it). Another great book of his, that I've read and planning to post here about, is Java Puzzlers.

Listed below are other sessions I watched, or plan to watch (sadly, most sessions are about web programming and client side - not my cup of tee).

Google I/O 2008 - Underneath the Covers at Google - GFS, big table, and the parallelism library MapReduce.
I wonder what similar constructs for parallelism IBM have up their sleeve...

Best Practices - Building a Production Quality Application on Google App Engine (Production stuff - I like the news from the front)

Dalvik VM Internals (That's Google implementation of Java to avoid paying Sun royalties for JME)

How Open Source Projects Survive Poisonous People (programmers intrigues always interesting)

Painless Python for Proficient Programmers (I'm starting to work my way through Python these days)