Think you're connected? Think again
From LB Wiki
One issue that trips up people when diagnosing load balancer problems is they see a connection has been made to a virtual service, so it looks like they’re getting through to the real server. That’s not always the case. Depending on how your load balancer is configured, it could simply mean you’ve made a connection to the load balancer’s proxy server.
Modern load balancers have actually two different types of virtual services: Layer 4, and Layer 7. Clients can’t really tell the difference, but under the hood, they’re substantially different in the way they operate.
A Layer 4 virtual service on a load balancer operates much like a router doing NAT. It’s just re-writing source and/or destination addresses. Not much more than your basic broadband wireless router. When you connect to a virtual service, you're connecting directly to the real server on the back end.
Layer 7 code is a type of application proxy, and can be aware of HTTP as well as other protocols depending (such as FTP, SIPs) upon the vendor. When a connection is made to a Layer 7 virtual service and a request is sent, a separate TCP connection is opened to the server, and the request is forwarded.
With an operation like cookie persistence, the load balancer can't send the second connection to the server until it receives the full request from the client. It can't because until it sees a cookie (or the absence of one), it doesn't know if the connection is from a previously established session (and thus locked to a specific real server on the back end) or a new session to be load balanced.
The tricky part comes when there’s problem with connectivity between the load balancer and the servers. If the servers are unresponsive, you will still get a connection to the virtual service (in most cases), but you will see no reply to any requests. This can trick you into thinking the servers are hanging, when the issue may very well be elsewhere. If you’ve setup SSL termination, which would fall into the Layer 7 camp, you’ll get non-HTTP encoded error (such as read:errno=104), which won’t show up in the browser, but will show up in a raw TCP connection.
So it’s important to know how your virtual service is configured when you’re troubleshooting an issue.
