There are mainly two ways that changing hosting can affect rankings (slightly):
- you switch to a slower server and your load time is worse. Google could potentially penalize you for this as they said loading time of a site is a ranking factor.
- you switch it from one geographical location to another. Google does look at the IP of the host, especially for geo specific website. This is a minor factor and would most likely not affect your rankings if they are quite strong.
I guess there is a third option is that there is a screw up during the transfer, your website goes down for a while.
Other than that, it shouldn't be a big deal. If you move to a faster server, I wouldn't worry about the rankings and do it because there is nothing worse than slow loading sites. You probably loose more visitors because of that than because of a potential unlikely to happen loss of rankings.