The V-Manager web system suffered several brief outages during the month of October. We are currently implementing a comprehensive and extensive backup connection. The following FAQ was designed to assist in understanding why this is not a simple matter of adding an "extra" Internet provider. At the end of the FAQ the new backup system is described.
Q: What causes a break in the connection to a V-Manager server?
A: There are actually four different reasons why an outage occurs.
Q: What is the first reason, and was it responsible for recent outages?
A: V-Manager equipment failure, such as server crash. V-Manager servers and equipment is highly redundant, and the least likely source of an extended outage. None of the outages were caused by equipment failure at V-Manager. Our equipment is monitored 24/7 so should such an incident occur we can respond quickly.
Q: And the second reason?
A: A failure or outage from V-Manager's ISP, or Internet Service Provider. In order for any person or business to connect to the Internet, they must contract with a provider who actually manages the connection from the business location to the Internet. A small portion of the recent outages was caused by a failure of equipment at our ISP. ISP's are large organizations, and typically severely understaffed in order to compete in the price-driven world of Internet connectivity. Thus during an outage obtaining exact ETA's for return to service can be difficult, despite the fact that each outage may affect hundreds or even thousands of businesses. In theory a backup, or secondary, ISP could be installed and used if and when an outage occurs with the primary ISP. Besides being a costly proposition, it does nothing to help with the main causes of outages, reasons #3 and #4.
Q: OK, so what are reasons #3 and #4? Start with #3.
A: It is widely assumed that the Internet is comprised of a solid, reliable internal system of routing information between destinations. Nothing could be further from the truth. Each and every packet, the smallest piece of elctronic information, is send from origin to destination by a route through the Internet determined on the fly at the time the packet is sent. That means that a web page loaded on a PC in Boston from a server in Miami could consist of millions of packets sent over a variety of paths between the two computers. Normally this is seamless. But this routing occurs across the Internet -- and who owns all the cables, pipes, and lines connecting the Internet? The answer is many, many companies own a small part, and each of those parts are connected in a maze-like system of inter-connections. While this sounds styrange, the original goal of the Internet was distributed redundancy so that a nuclear attack could not destroy the entire Internet. Destroy a cable in Texas, and the Internet logic would re-route those packets through Denver. However in practice often times a connection point in the Internet will be down or malfunctioning. This happens numerous times a day. If that malfunctioning point happens to be between your PC and the V-Manager server, it will appear as if there is no connection to V-Manager. The Internet is not able to determine there is a problem and re-route the connection. It is possible to see if this is the case by opening a command window and typing TRACERT XXX.XXX.XXX.XXX, where the X's are the IP address of the destination server. Several recent outages were of this nature, which means they did not affect all users, only ones in certain geographic areas. It is also impossible for V-Manager to do anything about such an outage. Large companies, such as an eBay, work around this problem by having multiple redundant servers in locations across the country or even around the world. Companies smaller than that have little recourse.
Q: Whew! And the fourth reason?
A: The fourth reason also depends on a little known fact about the Internet. Companies have a large competitive marketplace when it comes to selecting the ISP that will provide their connectivity. However _all Internet connections, from any provider_, always rely on the wires or cables that run from each business into the Internet. And that connection, called "the last mile", is _always_ owned by one and only one company. In most locations in the U.S., that company is the phone company - Verizon, AT&T, etc. A hundred ISP's can offer service to a company, but each one of them will connect you over hundreds and thousands of miles of their system -- and each one of them will have no choice but to use the same phone company cable from the business to the ISP. This means every Internet connection has a weak spot of a single point of failure, the phone company, and there is no recourse available. Contracting with every ISP in the world would not help. Further, these phone companies are understaffed and also competing in a vicious price war. For example, we have a contract with a guaranteed response time of two hours. Typically it can take a day just to get an initial response, and that is after the ISP determines that the phone company needs to be involved. Large companies such as eBay use the same method to circumvent this problem. They locate numerous duplicate servers around the world, so when a Verizon cable breaks in New York, their server in Hong Kong takes over seamlessly.
Q: OK, I give up. I never knew the Internet was so easily subject to problems. Given these reasons, expecially #3 and #4, what is V-Manager planning to do to ensure as little down time as possible?
A: Recently an option has become available to companies such as V-Manager that can provide significant failure protection. With the increase in speed and capability of the cellular network, the techology now exists to install a backup Internet connection that will operate outside the normal ISP/phone company system and use the cellular network. The speed of the cellular network is not sufficient to replace the Internet connection, so when there is a failure and the cellular backup takes over all other non-essential Internet activity other than the clients' servers is curtailed. All speed and power is channelled to the servers, and most likely a user would not notice a speed difference at all.
Q: Well, that sounds pretty easy.
A: In theory it is just that easy. The implementation of this backup system is quite technical, and very expensive. V-Manager is investing significant sums of money to try and establish as robust a connection as possible.