If your on a "Shared" server, I would complain to your provider because someone else may be causing the server load to slow your site down.
This is a common problem with shared servers, it may be an indication that the new anti-social neighbor is hogging server services at the cost and expense of everyone else ot a neighbors site got hacked and the new squatter is causing mayhem.
Whatever the scenario, your paying for web hosting, you are entitled to some service and a quick email to complain about loading times and that the server is sluggish should prompt the company to look at the bandwidth hog. For all you know it may be a hack attempt on that companies server that eating all the bandwidth.
My server is on a shared server but despite the average load of 1 Meg to 1.5 Meg, load times are not impacted despite the site currently containing nearly 4 Gigbytes of archives, we see no impact on service and our host has not complained about our loading.
IMHO give your host a prod if your close enough otherwise a well crafted email will get some action. You will likely encounter the "Service Requests" and loggin in to the USER CP and have to go several bouts of emails to get refereed to tier 2 support before your complaint is escalated to tier 3 when you have moaned at someone on the phone from lack of movement on this issue.