Improving the speed of your pages is normally pretty critical. If your pages are slow to load, then people quickly switch over, and even viewers who find the information on your pages useful tend to hesitate to come to your page more often; as for casual viewers, you can be pretty sure that they will switch fast if they find your page taking a lot of time to load.
You can take a number of steps to improve the performance of your page, but before that, you should be spending time to determine how the performance of your page actually is. You can get some idea just by loading your page and comparing the loading times of other pages, and there are a number of other tools that can help you determine the performance of your page. Many of these tools also help in giving you more details about which sections of your page cause overall bad performance.
One tool that can help you evaluate the performance of your sites is called Yahoo! YSlow (link to page):
Details about the tool are below.
YSlow analyzes web pages and suggests ways to improve their performance based on a set of rules for high performance web pages. YSlow is a Firefox add-on integrated with the Firebug web development tool. YSlow grades web page based on one of three predefined ruleset or a user-defined ruleset. It offers suggestions for improving the page’s performance, summarizes the page’s components, displays statistics about the page, and provides tools for performance analysis, including Smush.it™ and JSLint.
Details about performance optimization at Yahoo Developer Tools (link)
80% of the end-user response time is spent on the front-end. Most of this time is tied up in downloading all the components in the page: images, stylesheets, scripts, Flash, etc. Reducing the number of components in turn reduces the number of HTTP requests required to render the page. This is the key to faster pages.
Combined files are a way to reduce the number of HTTP requests by combining all scripts into a single script, and similarly combining all CSS into a single stylesheet. Combining files is more challenging when the scripts and stylesheets vary from page to page, but making this part of your release process improves response times.
A content delivery network (CDN) is a collection of web servers distributed across multiple locations to deliver content more efficiently to users. The server selected for delivering content to a specific user is typically based on a measure of network proximity. For example, the server with the fewest network hops or the server with the quickest response time is chosen.
The article above (link) contains a lot of details about how to increase performance, so make sure that you have read the page and evaluated which of these apply to you.