I think the problem has more to do with the browser than the web server. In our application, we have a tomcat servlet engine sending out contents of the jnlp file. It worked fine on IE 7 and 8, on Vista and Windows7 (our testing machines) . Recently it failed at a customer's site, running IE 6 on XP. We had them install Service pack 2 and IE-7 (and Java 6), still no luck. Problem is, if I save that file content on the desktop with some filename and a ".jnlp" extension, double clicking on it starts the application just fine - the computer does understand the jnlp association (checked in Control Panel, too).
In short, IE is not recognizing the .jnlp mime type, and we don't know how to make it do so. This might be the problem you are seeing, too. I would be very interested in a solution to the problem. Good luck to you.