Then you should have nothing further to say! If you don't agree with the standard definition, you don't get to just make up a new one that suits your ideas! Vocabulary is a social matter; so, your definitions aren't a basis for an argument. Find a standard definition from another source if you don't like the one I used. I'll even take one of the ones from answers.com if you like ...
I don't agree with that definition ...
By this definition, the necessity of the electronic aspect is sort of replaced with a mathematical ability aspect (a device that computes).
A device that computes, especially a programmable electronic machine that performs high-speed mathematical or logical operations or that assembles, stores, correlates, or otherwise processes information. (answers.com)
Anyway, I do see your concern with the first definition ... So, in terms of quantum computers, I believe that they still rely primarily on electronic operations--they merely use quantum processes as an "input" to some extent. The same could be said in regards to light computing. In fact, electrons are essential in reflecting light, which I presume to be a fairly common need in light computing.
And of course, as our computers evolve (or rather our engineers), we may have to evolve the definition of that word, as a society. Though, for now, we need to work with a standard definition. We can't just create one to suit our arguments.
That's not really important. If the machine even has a rudimentary temporary storage system, perhaps even a single capacitor that can store a bit for only a split second, it still qualifies as storage. There are no special qualifications for the length of storage in the definition. And on that note, even the capacitor may be unnecessary ... Though, I personally believe at least one capacitor (or similarly functioning device), with the purpose of "remembering" a bit, is an essential component.
... lots of computers don't have hard drives and essentially start with a completely blank slate.
I tend to use this "information always exists" idea in God arguments a lot. Though, I don't feel that it's suitable for this discussion. For one, all evidence suggests that "information" is NOT a thing with a physical existence. In terms of storage, all we can do is represent or symbolize that information by changing the orientation of some matter in some way (write it down, turn a capacitor on, magnetize an area of something, etc.). The fact that the "information" is "always out there" is irrelevant. It's a matter of whether the computer is storing a retrievable representation of it.
... information cannot be destroyed, and therefore must always be stored in one form or another.
A neat idea, I guess. Though, the big bang is just an event. I could see it being a boot-up; but not the program. I do, however, see how one might interpret the universe itself as a computer. In terms of our definition, I think it then comes down to whether the universe was intelligently created or "just happened."
Perhaps the big bang might be considered both the ultimate program and ultimate boot-up sequence.
In regards to the Matrix-like universe-computer, I can't really buy into that. I mean, on some level, we already sort of do that. For instance, a farm is just a bio-computer. The end result of the "computation" is generally food (and waste). Though, I have fundamental problems with calling a biological / self-growing system a computer.
I'm hinging primarily on the word FOR in the definition (an electronic device for storing and processing). We can put a seed in the ground for some purpose, but we don't really have control over how it grows. It's output is based on complex genetics, combined with a complex and largely uncontrollable environment, combined with the seed-state (down to the subatomic level), combined with factors we can't even know about. More importantly, the plant grows! It's not created ... it grows! It makes itself--we don't make the plant! That FOR in the definition, if not meant to distinctly say so, at least heavily implies that a computer is a human-made thing (or at made by an intelligent being), an assemblage of naturally occurring elements that will sustain their physical state, except for those changes which are dictated by the program.