As a sysadmin I see lots of WordPress websites, it's all too easy for a WP user to introduce slowdowns with extra plugins, and some of them are hideous in terms of their CPU usage - not really the website owners' fault, but causes speed issues all the same.
My go-to first suggestion is to look at caching - the fastest WordPress can be is when you're not running lots of PHP and DB queries on every request! Like dazc, I've seen great results with the "WP Super Cache" plugin - it's generally easy to use and set up and in many cases is set-it-and-forget-it. Web servers can send you a pre-built HTML file in tens of milliseconds rather than a few thousand milliseconds for a dynamic WordPress page.
(Of course, the flip side is caching doesn't work for everybody: some plugins may not work properly if they rely on WP running dynamically on every page load. Some can be tweaked to work, some not)