My attempts with IP Spoofing – Revisited

One upon a time (Jan 2009) I've written this post, basically saying that you're not likely to be able to spoof IP address over the Internet.
Turns out I was dead wrong!

It happened so the very experienced Mr Filipe, from Brazil, came across the post and left me a comment saying that Spoofing over the internet is quiet possible.
I replied surprised, and after a number of comments ping-pongs, we started chatting online, and Felipe had agreed to give me a live spoofing demo:
On my end, I've configured my home router to forward TCP/UDP packets to my desktop, where I ran a wireshark network capture to monitor any incoming packets.
Then Felipe sent a burst of packets from random IP source addresses. Proving me that IP spoofing over the Internet is a reality indeed.

(What do you think? Isn't this kind of stuff is what makes the Internet so amazingly wonderful? two people from two different parts of the world, united by joint interest and kindness :))

So, Thank you Filipe!

A few notes on why spoofing might *not* work:

1. According to Filipe, the recipient's ISP is much more likely to block the spoofed packet, than the sender's ISP. For example if the recipient's ISP see a bogon source IP.
   That's a bit counter-intuitive, because, assuming the ISPs really do care about preventing spoofing, it's a very easy job for the sender's ISP to tell if the packet's source IP is one of the IPs that it handed out to customers, or moreover, to the particular customer (sender).


2. If you are behind a NAT device, then any source address you are planning to use (be it spoofed or real) will be overwritten by the NAT anyway, so make sure you are on a real public IP.


3. No reason to get excited. TCP spoofing is very limited as you won't make it across the TCP handshake, because the recipients will send their ACK,SYN response to the spoofed IP, which you probably don't have much control over.
   In a LAN things are a bit different, if you can manipulate the recipient's ARP table to think that the spoofed IP MAC address is yours. I haven't dag deep.

Feel free to comment.

Concurrent Modification Exception

I ran into a ConcurrentModificationException (CME) during stress testing.
What does CME actually mean?
It means that you've modified (add, remove, update) your Collection while you've been iterating over it (usually in a multi-threaded fashion, but it can occur in a single thread that modifies while iterating).

A few more things to note about CME:
Best effort detection - If you see a CME printout, first off, consider yourself lucky, CMEs are thrown only in best effort. In another universe, the concurrent modification would not have been detected, causing your collection to become corrupted, instead of fast-failing with a CME.

IDing the problem - Like deadlocks, CME's are easy to pinpoint once you inspected the exception's stack trace.

Avoiding CME:

1. ListIterator
   To modify a collection by the same thread that is currently iterating on it, use a ListIterator that will allow you to perform both.
   Drawbacks - single thread solution only.


2. Naive solution: Synchronizers
   Use locks to for mutually excluding traversal and modification operations.
   Advantages - easy to code.
   Drawbacks - very long lock periods while iterating.


3. CopyOnWrite
   Take advantage of the Java.util.concurrent collections like: CopyOnWriteArrayList, CopyOnWriteArraySet. If you require a map then grab CopyOnWriteMap from Apache (this guys have been doing Sun's dirty work for years now).
   Advantages - very good reading performance (no locks are used, instead visibility is obtained via map member volatility).
   Drawbacks - very bad write performance on large maps.
   Conclusion - use for seldom mutating collections.


4. toArray()
   toArray will create a new array holding a copy of your Set (Map.keySet() for a Map).
   You can then iterate over the array, freely modifying the original collection (the array doesn't change of course).
   Advantages - write operations are cheap.
   Disadvantages - copying the entire set could be expensive if it occurs too often, and/or the set is very large.


5. Concurrent Collections
   If you want to go heavyweight, consider using: ConcurrentHashMap (or one of its package friends).
   Once you create an iterator over a ConcurrentHashMap (CHM), it does not freeze the collection for traversal, updates to the collection may or may not appear during the traversal (weakly consistent).

The approach I ended up taking
My use case was seldom modifying a ~ten items cache. A copyonwrite map was what I used.
In other cases I had, ConcurrentHashMap was the easiest solution (though make sure your code can live in peace with the CHM weak consistency property).

[caption id="attachment_227" align="alignleft" width="200" caption="Best pic idea I could think of to visualize Threads :)"][/caption]

Utility Frenzy #1 – The log summarizer

Here's a post I wrote (in the Hebrew language) which tells the story of the log summarizer utility that I've wrote. This story is the first in a line of "utilities stories" I'm planning on writing.
My apologies for those of you whom won't be able to read it. Posts in this site do appear in English..

Extanding your troubleshooting facilities - Always on verbose GC

Getting it right the first time

What happens when customers are experiencing problems with you application in production? The customer would send you the various logs artifacts and, ideally, you should be able to diagnose the problem and provide a resolution. If you find yourself sending the customer back and forth in an effort to gather additional types of log artifacts and system information, then you are, must likely, doing something wrong.

Who should be helping you

If you deploy your application on top of a application server platform, like Websphere Application Server (WAS) in my case, the platform should be assisting with automatic logs generation and collection. Our development team has been increasingly relying on such services provided by WAS, like: FFDC, WAS Collector, hung threads detection. All of which honorably earned their production stripes and badges.

One new serviceability artifact that I have long ago really wanted to have in production was the verbose GC, this feature records the JVM garbage collection activity over time, providing insight for resolving issues such as: stop-the-world performance freezes, memory leaks, native heap corruption, etc.

Until today, I was reluctant to enable the verbose GC in production, since I believed that there's no way to direct the verbose GC output from the native stder (default) to a rotating dedicated file, not doing so might lead to files larger than 2GB (a problem on some file systems), or would cause the system to run out of disk space. I was assuming that the performance implications would be negligible, but still, you have to be extra prudent when it comes to live customers environments.

A trigger for action

Last week I had an issue with a WAS component, after opening a ticket with Websphere support, I was asked to reproduce the scenario in order to generate verbose GC output, I decided that enough is enough! I'm gonna look into the GC output file rollover issue again and see what solutions exist, what the community have to say about it, or whether there might be some other custom solution (with the Apache web server, for example, the file rolling is handled by an external process into which the log output is redirected, the process then does the rolling files management itself).

Following a quick search, I was happy to find that the IBM JVM offers a rolling over verbose GC. I quickly found additional hands on reports, Chris Bailey published verbose GC performance impact results that reassured my gut feeling about any performance impact being a non issue.

Here's the syntax: (quoting the IBM Java 6 diagnostics guide):

-Xverbosegclog[:<file>[,<X>,<Y>]]
Causes -verbose:gc output to be written to the specified file. If the file cannot be found, -verbose:gc tries to create the file, and then continues as normal if it is successful. If it cannot create the file (for example, if an invalid filename is passed into the command), it redirects the output to stderr.
If you specify and the -verbose:gc output is redirected to X files, each containing Y GC cycles.

Final thoughts

1. I don't like having to specify the entire path for the file files, the default path should have been the server's logs directory, or the CWD (CWD is the profile's directory I believe).


2. Rollover threshold parameter - I would rather be specifying it in units of max MBs instead of in units of the number of GC cycles entries. I've empirically found that 1MB of verbose GC log translates to ~700 GC cycle entries (YMMV).


3. Good enough. I'll start doing the preparations to put this into production.

My attempts with IP Spoofing

Why did I wanted to spoof source IP addresses? and why did I failed? Here's the story before you:

------------
UPDATE Sep/2010: Dear Filipe (see comments below) had proven to me that spoofing over the internet is indeed possible, read all about it on the continuation post: My attempts with IP Spoofing – Revisited. Now back to the original story:
------------

When customers install our product, they often forget to setup firewall rules to accept incoming connections from public IM (instant messaging) providers. Without the firewall rules in place the product does not function properly, of course, and the customer rushes to open a support trouble ticket. Troubleshooting to pinpoint the problem to a missing firewall rule isn't trivial. When we try to validate whether the customer defined the required firewall rule, we need the external entity (that we have no control on) to open a connection to the customer's IP, but the external entity will only do so following the successful completion of a handshake sequence that must be initiated by the customer (consider for example: XMPP Dial-Back mechanism), since this handshake by itself is prone to failures, you can see how reproducing the problem is a combursum process.

I started looking for a simple, independent, and reliable, troubleshooting procedure that would be able to give a clear-cut answer to whether or not the customer defined the firewall correctly.
Here's what I've concocted:

1. Assume that the customer IP is 1.1.1.1 and they were suppose to configure their firewall to allow incoming connections from 2.2.2.2.


2. I'll send a single TCP SYN packet (the 1^st of the standard three messages TCP handshake) from my computer (say it's IP is 9.9.9.9), but I'll spoof the IP datagram's source address field to be 2.2.2.2 instead of what normally should have been my actual machine address (9.9.9.9).


3. I'll ask the customer to run a network sniffer on the IM Gateway machine. Waiting for the single packet to arrive at the destination socket.


4. If the sniffer had recorded the incoming IP message, then it means that the firewall is setup correctly and the problem is else where.
   But, If the sniffer didn't record any incoming SYN packet, then we shell blame the firewall guys.

Pretty simple, eh? Now, in order to spoof the TCP SYN packet I needed a something that could generate and send raw IP packets, since you can't just fiddle with the source IP address if you choose to ride on the good'ol TCP/IP stack. I found this IP spoofing perl script on the net, and it does the job.

[caption id="attachment_103" align="alignnone" width="300" caption="Visualization of the various routes through a portion of the Internet. Took it from Wikipedia."][/caption]

I did my first test on the office LAN, I sent a message from machine (IP 9.9.9.9) to to machine 1.1.1.1 claiming the message source was 2.2.2.2, it worked! Machine 1.1.1.1 registered an incoming packet from 2.2.2.2.
It seems that the office router went along with the scam, perhaps it thought that the machine switched IP it IP, or the DHCP server went crazy, or that it's ARP cache is just stall.

In the next test I tried sending the packet over the Internet, I tried sending a packet to my home computer from the office, with a source IP of some foreign entity, to my dismay, it never got to my home computer. Other IP variations didn't work either.
My guess is that some router along the way noticed that it's getting a packet with a source IP address that the part of the network it is looking can't can't possibly generate (imagine CIDR based ACLs), and that caused it to immediately drop the packet. This failure caused me to give up on the whole spoofing troubleshooting procedure idea.

Some thoughs about what I've seen:

1. Evidently, It's quite trivial to spoofe IP addresses on a LAN.


2. Spoofing  IP addresses over the Internet doesn't seem to be trivial.


3. A side note: If the customer has a reverese proxy, or any form of entity that delegates TCP handshakes, deployed before the actual IM Gateway machine, then the procedure is not applicable, as the first TCP SYN message will never reach the IM Gateway machine.


4. I would assume that the closer you inject the packet into the Internet backbone blood stream, the better the chances of not getting a rejection of the spoofed packet. The backbone routers communicate with many difference parts of the network, and might not have rational of where certain packates should be coming from or not.
   IP Packets tend to travel in different routes, making it harder to judge what IP CIDR is ligit from each fellow router.


5. I'm guessing that the biggest problem for spoofing is the first or the second router (the ISP's), since the ISP knows exactly what is your assinged address. Thereby knowning that the packet is spoofed.


6. If any one knows a better method of spoofing source IP, please step forward and share your secret :)