There are a lot of web hosting out there that doesn't work straight out of the box with Magento. For long time, I am looking for a fast and affordable hosting to be my play ground. I know Linux really well, but still hate to set it up. Here are my two finalist.
Nexcess SIP plan
SimpleHelix's shared plan
And here is a more comprehensive list:
Find the best web hosts