localhost is by definition the hostname of the local computer (or rather a kind of alias for it). It is not something you install on a hard-drive. To access what is 'localhost' on your computer via another computer, you can either use your computers actual hostname (which you can find using the hostname command on *nix boxes), or else it's internal IP address (which you can find using ifconfig).
Perhaps by localhost you actually mean DocumentRoot, ie the root directory of your website? If so you can set this to any mounted location via Apache's config. You can then connect via port 80 (assuming your server is listening on the default port) using either the hostname or IP as mentioned previosly. Obviously in a web browser you dont need to specify port 80, so just http://yourhostname, or http://192.168.0.100 (or whatever your IP is) will work.
Connecting from outside your LAN will obviously require the use of your external IP address, and traffic on port 80 forwarded on to the computer running your web server.
To be clear, I mean... install Apache, PHP, MySQL & PHPmyAdmin on an external drive so I can run PHP/MySQL (locally) from any computer I connect it to (via USB).... oh, and I mean without configuring the machine it's connected to. So, I can plug it in at home and work on my site or take it to work and run PHP without fooling with permissions on my work computer.
You should be able to do this with some applications. However, as you probably know, some applications scatter files and configuration options all over the place. These types of apps may still be installed on a completely separate device--but not likely without a good deal of hassle.
Yes I don't know if Apache, PHP etc make portable versions, but the default installation on my linux box puts files in a few different places. The windows versions may be easier, but I don't use it so can't say.
Otherwise maybe you could install pendrivelinux and run it from there?