Well, the Joomla! website has lots of documentation, including a Wiki for learning docs. If you're just starting with Joomla! I would recommend learning to style a Joomla! template before anything else. Copy & paste the default template (beez_20) and rename it to beez_20_backup.
Now you can play around with the default template while retaining a backup. The files you'll want to manipulate are:
If you want to start delving into PHP development, once you're comfortable with the back-end (administration) and styling, take a look in the components->com_content folder and start playing with the sub-directories. This is your article data.