The answer to this question can be found by searching Wikipedia for "HTML doctype", but, in brief,
With HTML pre-5, the doctype declaration associated the web page with a formal document type declaration document for the specific version of HTML.
With HTML5, the doctype simply declares that the page is to be interpreted as that version of HTML.
Without a doctype declaration, the browser will use "quirks mode". Quirks mode is browser-specific, which can allow anomalies in the page display.