Over Optimization happens when your website is considered "too good" by Google - either in terms of a sudden volume of backlinks, or because of heavy on-page optimization. In other words, if Google considers that your website optimization is beyond acceptable limits , your website will be red-flagged and automatically restricted or penalized.
There is a fine line between over optimization and spamming, and it is on this line that Google can appear to err. However, this is not a mistake by the search engine - in fact, Google calculates rankings by considering thousands and thousands of different factors - and a lot of importance is attached to average "trends" within the niche / keyword range that a website is optimizing for.
The bottom line is that over optimization is non-spamming search engine optimization that is misread by Google as being beyond acceptable limits, thus leading to a penalty in search engine rankings.