As an avid wordpress user, I can definitely vouch that wordpress isn't the way to go, and as I've briefly touched Joomla, I knew the potential that Joomla can bring to you.
However, your requirements seem more tailored towards a custom-made php based website, instead of a turnkey solution such as Joomla and Wordpress.