Figure 11 – Using Application Center Test to stress test a simple page. The result here is from
combined machines – which running around 275 rps. Machine 1 and 2 individually were running 136 and 158 rps respectively.
The script hits only the ASPX page – no images or other static content was hit. I tested each of the machines individually changing the IP Addresses to their dedicated IPs in the ACT script first and then together by changing the script to use the virtual IP. The results for this short 5 minute test are as follows:
Web Store Single Read Page Test
Test Mode
Requests per second
Office Server 111.111.111.2
Laptop 111.111.111.1
Both of them Load Balanced 111.111.111.10
This is a ratio of 91% for the load balanced vs. the machines individually which is excellent given that we are running with a single adapter here.
The second test is a bit more realistic in that it runs through the entire Web Store application site and uses a shared SQL Server on a third machine.
Web Store Full Order Test
Test Mode
Requests per second
Office Server 111.111.111.2
Laptop 111.111.111.1
Both of them Load Balanced 111.111.111.10
Here the ratio is a bit worse: 77%, but the reason for this drop off has little to do with the Load Balancing, but the fact that there are some limits being hit on the SQL Server. Looking at the lock count with performance monitor reveals that the site is hitting the SQL box pretty heavily and the locking thresholds are causing requests to start slowing down significantly.
This application is not heavily SQL optimized and performance could be improved to make these numbers higher both for individual and combined tests. However, this test shows that load balancing can help performance of an app, but that there may still be other limits that can slow down the application as a whole. In short, beware of load issues beyond the Web front ends that can bite you in terms of performance. Still even in this test where an external limit was being approached we still got a significant gain from using Load Balancing.
Port Rules revisited: SSL
Remember I configured my server for HTTPS operation by configuring port 443 earlier? Actually only one of the servers has the certificate installed, so I need to manage the port rules to drive all HTTPS traffic to the SSL enabled server. This must be administered manually through the Network Connections dialog by clicking on the Load Balancing Service and then configuring the Port Rules. Notice that this dialog shown in Figure 12 has a Load Weight option, which is set to 100 in the SSL enabled server and 0 in the other.
Figure 12 – When editing the Port Rules in Network Connections
you can configure the load weight for each server in percentages.
This effectively drives all SSL traffic to the machine that has the certificate installed.
Load Balancing and your Web applications
Running an application on more than one machine introduces potential challenges into the design and layout of the application. If you're Web app is not 100% stateless you will run into potential problems with resources required on specific machines. You'll want to think about this as you design your Web applications rather than retrofitting at the last minute.
If you're using Active Server Pages, you'll have to know that ASP's useful Session and Application objects will not work across multiple machines. This means you either have to run the cluster with
Single Affinity to keep clients coming back to the same machine, or you have to come up with a different session management scheme that stores session data in a more central data store such as a database.
Thankfully ASP.Net has several ways around this problem by providing different options for storing Session state using either a separate State Service that can be accessed across machines or by using Session state stored in a SQL Server database. You should always use session state in one of these mechanisms because these mechanisms can survive Web application restarts which can happen more frequently in ASP.Net due to changes in web.config or simply from the Web Server (IIS 6) recycling an Application Pool.
Finally, load balancing can allow you to scale applications with multiple machines relatively easily. To add more load handling capabilities just add more machines. But remember that when you build applications this way that your weakest link can bring down the entire load balancing scheme. If your SQL backend which all of your cluster nodes are accessing is maxed out, no amount of additional machines in the load balancing cluster will improve performance. The SQL backend is your weakest link and the only way to wring better performance out of it is to upgrade hardware or start splitting databases into separate servers.
Pulling the plug
As mentioned earlier redundancy is one of the goals of a load balanced installation and to test this out I decided to test a failure scenario by pulling the network cable out of one of my servers. With both cluster nodes running one of the clusters went dead and after 10 seconds all requests ended up going to the still active cluster providing the anticipated redundancy. A few requests on the client ended up failing – basically those that had made it into the servers request queue. All others are silently moved over to the other server in the pool.
In another test I decided to turn off the Web service, which resulted expectedly in the network connection still being fed requests that now started to fail. This is to be expected because NLB deals at the network protocol level but doesn’t check for failure of the requests at the network application level (Web Server). For this scenario you will need a smart monitoring application that can tell that your Web services are not responding on port 80 or even better not returning the results that you should be getting back.
The bottom line here is: The service works well for catching fatal failures such as hardware crashes or network failures that cause the network connection to a single machine to die. But application level failures continue to be your responsibility to monitor and respond to.
Just add water… eh, machines
The Windows Server Network Load Balancing service finally makes load balancing affordable and relatively easy to implement. It’s taken a while to get here from two Windows versions back, but now that the tools are integrated into Windows it’s relatively painless to scale out to other machines. It’s good to know that the capabilities are built-in and that you can tackle applications that may require more than a single machine.
Just remember to plan ahead. Just like anything the process of taking an application and making it do something new, spreading apps over multiple machine takes time and some planning to get right. Don’t wait until you really, really can’t live without this feature – start planning for it before you do. Finally make sure you know your bottlenecks in your Web applications. A load balancing cluster is only as good as its weakest link. Pay special attention to data access as that is likely to be the most critical non-cluster component that can potentially snag scalability.
But isn’t that a position we all wish we were in? So much traffic we can’t handle it? Well, hopefully you’ll get to try out this scenario for real – real soon, so you (or your boss) can retire rich…
As always if you have any questions or comments about this article please post a message on our message board at:
http://www.west-wind.com/wwThreads/Default.asp?Forum=Code+Magazine.