The Design of Yale's Campus Network

by Jeremy George and Joe Paolillo

Beginnings

Fewer than 20 years ago every human being on planet Earth involved in tcp/ip networking could, and did, fit around an ordinary conference table. As was the case nearly everywhere, the early adopters of network technology at Yale were the people who used computers as part of their daily academic and professional lives. Isolated pockets of networks soon began to form.

By the mid-80's the network had become interconnected into a nascent campuswide network and Yale, along with CUNY, had pioneered wide area networking in Bitnet. Few people relied on the network but many members of the Yale community beyond those professionally engaged in computing had begun to discover its value. Listservs disseminated information on a rapidly growing number of topics. Email was being explored as a means of communication. Some researchers had already discovered they could collaborate with colleagues on different continents without having to make phone calls at 5:30 am to catch a window of marginal convenience.

During the 90's, data networking seems to have come of age as an integral part of the daily activity of many faculty, students and staff members. Those of us participating may be only vaguely aware that a network is involved at all. The character of Yale's network has, in the past fifteen years, made the transition from interesting technological experiment to necessary daily tool.

Growth Factors

Along with this shift in paradigm has come a growth that at times has been very nearly discontinuous. The network grows in small increments as orders for one or two connections come in and, occasionally, by leaps. Estimating the growth can be tricky. It could be that almost wherever people sit down they will want to connect to the network (e.g., classrooms, the library, residences, study areas).

Just bringing the network physically to the large number of faculty, students and staff at Yale has been a considerable challenge. To get things started in the 1980Ős, Phonenet wiring was selected for many users because of its reasonable cost and, more importantly, because it utilizes existing telephone wire over relatively long distances. Moreover, Phonenet had been developed for the Macintosh, and most computer users at Yale in the late 80's used Macs.

As the network's capacity grew, so did the capability of machines. In each new generation CPUs are faster and network controllers are capable of much higher transmission speeds.

One final growth factor is the so-called "Killer App," the application that everyone "must have" and will be the dominant consumer of network resources. Until two or three years ago the "Killer App" was electronic mail. Today, Web software such as Mosaic and Netscape could be that application, although some may recognize distributed Doom and similar software as the voracious consumers of resources.

Technological Paradigm Shift

With the rapid growth of the network and reliance on its availability have come multiple technological paradigm shifts. The first physical connection points to the Ethernet were large, unwieldy transceivers called H4000s. They were completely unmanaged devices. The next generation of transceivers were 1/4 the size and had LEDs that indicated how they were working. The generation after that changed the model altogether to the hub and twisted pair wiring we see now in which the hubs can be monitored across the network itself. In general, the transition was made from relatively dumb devices to relatively intelligent ones.

The first Ethernets were really just long strands of thick cable built to a specification. Every eight feet a transceiver, and consequently a computer, could be attached. The Ethernet standard prescribed that data be transmitted at 10,000,000 bits per second. If there were 40 machines connected to the cable, in theory each machine would only have 1/40 of the total bandwidth, or 250,000 bits per second (Phonenet speed). In fact not all machines send data at the same time, and as many as 100 machines on a shared cable still experience far better service than on Phonenet. But the limitation of a shared cable was clear, and the coming of hub and twisted pair technology, along with switching technology, produced a way of segmenting Ethernets so that more bandwidth was available to the desktop. A switch segments the network by only sending packets to the areas where they belong. In the previous example, if 40 machines were all connected to individual switch ports, then theoretically 20 machines talking to 20 other machines could use an aggregate bandwidth of 200,000,000 bits per second even though any single conversation would still be limited to 10,000,000 bits per second. Switches are relatively expensive devices, but careful use of them has provided us with a way of leveraging Ethernet bandwidth dramatically.

Even so, at the core of a network and in the case of high demand servers, segmenting Ethernets just does not provide enough bandwidth today. In such cases there is little choice but to change the network architecture to one of the very high speed 100,000,000 bit per second, or faster, technologies. The most mature of the very high speed technologies is FDDI (Fiber Distributed Data Interface). Unlike Ethernet, which is a bus technology, FDDI uses a ring configuration. However, FDDI in its native form is also a shared bandwidth technology. Thus it is sometimes very hard to gain a significant advantage moving from a switched Ethernet environment to a shared FDDI one. Fortunately, just this past fall, vendors began to implement switched FDDI which provides all the bandwidth advantages of switched Ethernet. In the example from the former paragraph, switched FDDI would yield a total aggregate bandwidth of 2,000,000,000 bits per second. While this number is merely theoretical, implementing switched FDDI at the core of Yale's network would mean that, in practice, bandwidth simply ceases to be an issue, at least for the present.

From an architectural perspective then, an efficient and economic paradigm has been found. The design effort now is to continue implementing FDDI (or other appropriate 100,000,000+ technology) as it's needed while pushing switched Ethernets further and further out toward the desktop.

One important limitation of switched technologies is that they are currently based on local area network addresses (e.g., Ethernet addresses) or, in netspeak, layer 2 (where layer 1 defines the network physically and layer 3 defines global addressing e.g., ip addresses). Layer 2 switching floods multicast and broadcast packets throughout the network. In a network the size of Yale's this would mean that multicast and broadcast traffic could consume an unacceptably high level of bandwidth so that many separate layer 2 broadcast domains would need to be formed. There are also security problems associated with large layer 2 environments. To counteract this undesirable result, we place other devices, called routers, strategically throughout the network. Routers operate at layer 3 of the network model and solve many of the problems of flat layer 2 networks. Unfortunately, the immense flexibility and functionality that comes with routing is very expensive. Routers must maintain tables specifying where packets are to go and, in a separate function, switch them through.

The ideal situation would be to have two or three route servers on campus to download the appropriate routing tables to layer 3 switches, thereby breaking out the two functions. The premise is that layer 3 switches will be significantly less expensive than routers but will have the same functionality. This distributed routing technology should begin to be available late in 1996. DNO has purchased switches and routers that will be able to take advantage of distributed routing when it's released for production use.

Developments in machine technology have required bringing greater bandwidth to the desktop than Phonenet permits. As a result substantial fiber optic cable has been, or will be, pulled to all buildings on campus. Much of the University is now connected at Ethernet speed (10,000,000 bits per second or 40 times faster than Phonenet) and almost all of it will be connected by the Fall of 1997.

One other shift, not quite so readily apparent, is the number of different protocols carried across the network. Phonenet is primarily intended for Appletalk but IP (the Internet Protocol) and DECnet can be tunneled through it to gateways. On the other hand, Ethernet is not associated with any particular higher level protocol and therefore allows us to carry the higher level protocols natively (without tunneling) and also to include other routed protocols like IPX, the network protocol used by Novell. Additionally, we carry some of the simpler, non-routable protocols like LAT (Local Area Transport) and NETBEUI (used in some configurations by Microsoft). While this creates a somewhat more complex networking environment, it adds substantially to user functionality.

Current Implementation

Yale's backbone was originally implemented as a shared Ethernet. In 1993 the transition was made to a switched Ethernet at the core of the network.

Implementation of a switched paradigm can be accomplished in various designs. At Yale we have chosen a common design sometimes called a tree of trees. Basically there is an industrial strength layer 2 independent switch at the core of the network connected to a tier of routers or layer 3 switches which are in turn connected to layer 2 switches and/or hubs. Schematically, the picture looks like this.

The critical advantage of maintaining layer 2 independence at the core is that our investment in Ethernet (or any given technology) can be continued until usage actually demands an upgrade. Thus we can increase the bandwidth available to the areas which need it without being immediately required to purchase FDDI devices for every backbone connection point. It's a case of being able to spend money on infrastructure exactly when the services being supported require it and not months and even years in advance. From a narrow engineering perspective this is a considerable blessing. Technology choices in a very rapidly changing field can be postponed as long as possible and the chances of heading down a technological cul-de-sac are minimized. From a University perspective it means that funds are reserved as long as possible, also highly desirable.

Nevertheless, spending money on a just-in-time model does not imply cutting corners. In the last fiscal year Data Network Operations experienced fewer than eight hours of unplanned downtime at the core of the network. Given that there are 8,760 hours in a non-leap year this means our core uptime was about 99.9% and we want to do better next year. Chosen for its reliability and redundancy the core switch is Cabletron's flagship modular MMAC-plus, and we believe unplanned core downtime can be reduced to near 0%. Additionally, we have purchased a mirror image spare that can be brought online in minutes in the unlikely event of failure of the primary switch.

In addition to the core of the network, the Old Campus, the residential colleges and Academic Computing Services have received early design attention. This combination of student residences and services to the residences defines a group that can benefit from a design principal known as "locality of reference." Basically, the idea is to bring the services as close logically to the consumers as possible .

Back to Index Jan/Feb 1996