Yah, there's somewhat of a delay in both browsers. But there's a lot of stuph getting downloaded, too; and I notice that the delay goes away on subsequent hits to the page. So, I would guess the problem is mainly just in the amount of stuph the browser has to initially retrieve/parse. Once it's in the cache it's fine.
One other immediate thought: Since you're using conditional logic for IE, you could include the alternate stylesheet block in the reverse condition, to ensure that IE won't try to redownload the same set of sheets. I.E.,
Hey, thanks ! That might be really usefull, indeed.
Something that I just notice (I'm not at home so I can't try any modification right now) is that no stylesheet is downloaded in Safari. It just returns the page without any styling. Do you know if something is messing up with Safari ? I thought the browser was acting like firefox, but apparently it's not