Few things to think about:
is it on a shared hosting server
are other sites on the server loading slowly
is there any 3rd party content loading on the site, if so can you test load it on another of your domains?
how much traffic is the site getting and is it normal?