XHTML is only "obsolete" because Internet Explorer dropped the ball. XHTML 1.x was great to work with (never got to work with XHTML 2.0, sadly).
A lot of web developers whined that the error handling was "draconian", but in reality it was no worse than working with a programming language.
Basically, you have to behave yourself with XHTML (not hard to do, merely make sure all elements are closed and properly nested), quotes around all attributes), while HTML allows you to write slop, and throws in a surprise or two when it comes to the DOM.
For example, in XHTML, if you don't use <tbody> tags, you don't have a <tbody> element. In HTML, the DOM doesn't care. Boy, did that take a while to figure out. Also, in XHTML, your tags have to be written in lower-case letters, and the DOM agrees with that. In HTML, it's case-insensitive, but the DOM says all tag names are in uppercase. That was loads of fun and games, too.
XHTML is only "obsolete" because Internet Explorer dropped the ball.
I think you're absolutely right. We were supposed to use the compatibility guidelines only until browsers were updated to support XHTML, but for IE, that day never came until it was already too late. XHTML was first released in 2000. IE didn't support it until IE9... in 2011.