That's quite a tall order! You'll get lots of web hosting recommendations, basically any Linux/PHP based host will work. Check the host's support forums to see what sort of complaints current users are having before signing up.
For plugins, I'd recommend a Google sitemaps plugin, such as:
Google (XML) Sitemaps Generator for WordPress
along with the ping optimizer (one that fixed Wordpress' bugs with respect to pings and scheduled posts). I review one here:
* A Better Wordpress Ping Plugin*by*Online Opportunity
I use Google Analytics, so a plugin for that is here:
http://cavemonkey50.com/code/google-analyticator/
The one-click install plugin takes the FTP out of installing all the other plugins and themes, and is highly recommended:
OneClick for Wordpress at Thought Outflux | Anirudh Sanjeev
You probably won't need a caching plugin at first, but it's better to start out with one than to try and add it later:
WP Super Cache - Holy Shmoly!
Jay