I completely agree that links from "bad" sources will cost you you ranking rather then improving it, and with PR recalculation taking time. Also, you'd be better off with 300 good links from good sources then 30000 links from just one or two. It is considered an artificial inflation and will harm your ranking.
If you haven't already, I would recommend to use the Google Analytics and diagnostics tools within we webmaster tools, - if Google has any issues with your site, it will be evident there. If you have any issues with your sitemap structure, or any other factors it will affect the ranking.