I think the spirit of script loaders is more for dependency management than simply to load libraries, even though people are using script loaders to load just about anything these days.
From my point of view, if you've got no dependencies (except for the one you created by first requiring your script loader so you can use it to require a library) then you've got no reason to use a script loader.
Also, you can configure your Modernizr build to only include what you want - it doesn't come with a loader if you exclude it from your build. The only thing I use modernizr for is HTML5 Shiv and certain CSS tests.
For the record, I'm more than willing to change my point of view if the right argument comes along.