The panel of an InfiniBand switch

InfiniBand is a switched fabric communications link primarily used in high-performance computing. Its features include quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. It is a superset of the Virtual Interface Architecture. Image File history File links No higher resolution available. ... Switched Fabric is a Fibre Channel topology where many devices connect with each other via Fibre Channel switches. ... It has been suggested that this article or section be merged into Supercomputing. ... In the fields of packet-switched networks and computer networking, the traffic engineering term Quality of Service, abbreviated QoS, refers to resource reservation control mechanisms. ... Failover is the capability to switch over automatically to a redundant or standby computer server, system, or network upon the failure or abnormal termination of the previously active server, system, or network. ... It has been suggested that this article or section be merged with Scale (computing). ... Processor can mean: A central processing unit of a computer. ... Energy Input: The energy placed into a reaction. ... The Virtual Interface Architecture (VIA) is an abstract model of a user-level zero-copy network, and is the basis for InfiniBand and iWARP. Created by Microsoft, Intel, and Compaq, the original VIA sought to standardize the interface for high-performance network technologies known as storage area networks (SANs). ...

Description

Like Fibre Channel, PCI Express, Serial ATA, and many other modern interconnects, InfiniBand is a point-to-point bidirectional serial link intended for the connection of processors with high speed peripherals such as disks. It supports several signalling rates and, as with PCI Express, links can be bonded together for additional bandwidth. Fibre Channel is a gigabit-speed network technology primarily used for storage networking. ... PCI Express (formerly known as 3GIO for 3rd Generation I/O, not to be mistaken with PCI-X) is an implementation of the PCI computer bus that uses existing PCI programming concepts and communications standards, but bases it on a much faster serial communications system. ... SATACO.COM redirects here. ... In telecommunications and computer science, serial communications is the process of sending data one bit at one time, sequentially, over a communications channel or computer bus. ... PCI Express (formerly known as 3GIO for 3rd Generation I/O, not to be mistaken with PCI-X) is an implementation of the PCI computer bus that uses existing PCI programming concepts and communications standards, but bases it on a much faster serial communications system. ... Channel bonding in computer networking is an arrangement in which two or more network interfaces on a host computer are combined for redundancy or increased throughput. ...

Signalling rate

The serial connection's signalling rate is 2.5 gigabit per second (Gbit/s) in each direction per connection. InfiniBand supports double (DDR) and quad data (QDR) speeds, for 5 Gbit/s or 10 Gbit/s respectively, at the same data-clock rate. A gigabit per second (Gbps or Gbit/s) is a unit of data transmission equal to 1,000 megabits per second or 1,000,000 kilobits per second or 1,000,000,000 bits per second. ... Quad data rate (or quad pumping) is a communication signalling technique wherein data is transmitted at both the rising and falling edges of signals much the same way DDR technology works, but with two signals 90° out of phase from each other, effectively delivering 4 bits of data per clock...

Links use 8B/10B encoding — every 10 bits sent carry 8bits of data — so that the useful data transmission rate is four-fifths the raw rate. Thus single, double, and quad data rates carry 2, 4, or 8 Gbit/s respectively. In telecommunications, 8b/10b is a line code that maps 8-bit symbols to 10-bit symbols to achieve DC-balance (see DC coefficient) and bounded disparity, and yet provide enough state changes to allow reasonable clock recovery. ...

Links can be aggregated in units of 4 or 12, called 4X or 12X. A quad-rate 12X link therefore carries 120 Gbit/s raw, or 96 Gbit/s of useful data. Most systems today use either a 4X 2.5Gb/s (SDR) or 5Gb/s (DDR) connection. InfiniBand QDR was already demonstrated during 2007, with expectations of productions systems during 2008. Larger systems with 12x links are typically used for cluster and supercomputer interconnects and for inter-switch connections. An example of a Computer cluster A computer cluster is a group of tightly coupled computers that work together closely so that in many respects they can be viewed as though they are a single computer. ... For other uses, see Supercomputer (disambiguation). ... A network switch is a computer networking device that connects network segments. ...

Latency

The single data rate switch chips have a latency of 200 nanoseconds, and DDR switch chips have a latency of 140 nanoseconds.^{[citation needed]} However, due to the larger effect of the end-points, the total message delivery latency is much larger, from 1.26 microseconds MPI latency (Mellanox ConnectX HCAs) to 1.29 microseconds MPI latency (Qlogic InfiniPath HTX HCAs) to 2.6 microseconds (Mellanox InfiniHost III HCAs).^{[citation needed]} Various InfiniBand host channel adapters (HCA) exist in the market today, each with different latency and bandwidth characteristics. InfiniBand also provides RDMA cababilities for low CPU overhead. The latency for RDMA operations is 0.99 microseconds (Mellanox ConnectX HCAs). Latency is a time delay between the moment something is initiated, and the moment one of its effects begins. ... A nanosecond is an SI derived unit of time equal to 10-9 of a second. ... A nanosecond is an SI derived unit of time equal to 10-9 of a second. ... Message Passing Interface (MPI) is computer software that allows many computers to communicate with one another. ... Remote Direct Memory Access (RDMA) is a concept whereby two or more computers communicate via Direct Memory Access directly from the main memory of one system to the main memory of another. ...

Topology

InfiniBand uses a switched fabric topology, as opposed to a hierarchical switched network like Ethernet. Switched Fabric is a Fibre Channel topology where many devices connect with each other via Fibre Channel switches. ... Ethernet is a large, diverse family of frame-based computer networking technologies that operate at many speeds for local area networks (LANs). ...

Like the channel model used in most mainframe computers, all transmissions begin or end at a channel adapter. Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or quality of service. For other uses, see Mainframe. ... In the fields of packet-switched networks and computer networking, the traffic engineering term Quality of Service, abbreviated QoS, refers to resource reservation control mechanisms. ...

Messages

Data is transmitted in packets of up to 4 kB that are taken together to form a message. A message can be:

• a direct memory access read from or, write to, a remote node (RDMA)
• a channel send or receive
• a transaction-based operation (that can be reversed)
• a multicast transmission.
• an atomic operation

Direct memory access (DMA) is a feature of modern computers that allows certain hardware subsystems within the computer to access system memory for reading and/or writing independently of the central processing unit. ... Remote Direct Memory Access (RDMA) is a concept whereby two or more computers communicate via Direct Memory Access directly from the main memory of one system to the main memory of another. ... Routing Schemes anycast broadcast multicast unicast Multicast is sometimes also used to refer to a multiplexed broadcast, although that is a very different thing and should not be confused. ... In computer science, an atomic operation is an operation during which a processor can simultaneously read a location and write it in the same bus operation. ...

Programming

One caveat is that InfiniBand has no standard programming interface. The standard only lists a set of "verbs," functions that must exist. The syntax of these functions is left to the vendors. The most common to date has been the Open Fabrics syntax, which was adopted by most of the InfiniBand vendors, both for Linux and Windows OpenFabrics Alliance.

History

InfiniBand is the result of merging two competing designs, Future I/O, developed by Compaq, IBM, and Hewlett-Packard, with Next Generation I/O (ngio), developed by Intel, Microsoft, and Sun. From the Compaq side, the roots were derived from Tandem's ServerNet. For a short time before the group came up with a new name, InfiniBand was called System I/O. Compaq Computer Corporation is an American personal computer company founded in 1982, and now a brand name of Hewlett-Packard. ... For other uses, see IBM (disambiguation) and Big Blue. ... The Hewlett-Packard Company (NYSE: HPQ), commonly known as HP, is a very large, global company headquartered in Palo Alto, California, United States. ... Intel redirects here. ... Microsoft Corporation, (NASDAQ: MSFT, HKSE: 4338) is a multinational computer technology corporation with global annual revenue of US$44. ... Sun Microsystems, Inc. ... Tandem Computers was an early manufacturer of fault tolerant computer systems, marketed to the growing number of transaction processing customers who used them for ATMs, banks, stock exchanges and other similar needs. ... Servernet was developed by Tandem Computers Inc. ...

InfiniBand was originally envisioned as a comprehensive "system area network" that would connect CPUs and provide all high speed I/O for "back-office" applications. In this role it would potentially replace just about every datacenter I/O standard including PCI, Fibre Channel, and various networks like Ethernet. Instead, all of the CPUs and peripherals would be connected into a single pan-datacenter switched InfiniBand fabric. This vision offered a number of advantages in addition to greater speed, not the least of which is that I/O workload would be largely lifted from computer and storage. In theory, this should make the construction of clusters much easier, and potentially less expensive, because more devices could be shared and they could be easily moved around as workloads shifted. Proponents of a less comprehensive vision saw InfiniBand as a pervasive, low latency, high bandwidth, low overhead interconnect for commercial datacenters, albeit one that might perhaps only connect servers and storage to each other, while leaving more local connections to other protocols and standards such as PCI. This article is about the computer bus type. ... Fibre Channel is a gigabit-speed network technology primarily used for storage networking. ... Ethernet is a large, diverse family of frame-based computer networking technologies that operate at many speeds for local area networks (LANs). ...

So far InfiniBand has become the de-facto interconnect of choice for high performance computing, and its adoption as seen in the TOP500 supercomputers list is faster than Ethernet. (However, one should note that Top500 uses Linpack for benchmark, which as a neatly parallel computing task tends to be fairly easy on the interconnect; InfiniBand shouldn't be confused with the custom-built interconnects of vector supercomputers. For example, the NEC SX-9 provides 128 GB/s of low-latency interconnect bandwidth between each computing node, compared to the 120 GB/s (96 Gb/s) of an InfiniBand 12X Quad Data Rate link.) Enterprise datacenters have seen more limited use. It is used today mostly for performance focused computer cluster applications, and there are some efforts to adapt InfiniBand as a "standard" interconnect between low-cost machines as well. A number of the TOP500 supercomputers have used InfiniBand including the low-cost System X built by Virginia Tech. In another example of InfiniBand use within high performance computing, the Cray XD1 uses built-in Mellanox InfiniBand switches to create a fabric between HyperTransport-connected Opteron-based compute nodes. The TOP500 project ranks and details the 500 most powerful publicly-known computer systems in the world. ... A supercomputer is a device for turning compute-bound problems into I/O-bound problems. ... LINPACK is a software library for performing numerical linear algebra on digital computers. ... NEC Corporation (Japanese: Nippon Denki Kabushiki Gaisha; TYO: 6701 , NASDAQ: NIPNY) is a Japanese multinational IT company headquartered in Minato-ku, Tokyo, Japan. ... The SX-9 is a supercomputer built by NEC Corporation. ... An example of a Computer cluster A computer cluster is a group of tightly coupled computers that work together closely so that in many respects they can be viewed as though they are a single computer. ... The TOP500 project ranks and details the 500 most powerful publicly-known computer systems in the world. ... A supercomputer is a device for turning compute-bound problems into I/O-bound problems. ... System X is a supercomputer assembled by Virginia Tech in the summer of 2003, comprising 1,100 Apple PowerMac G5 computers. ... The Cray XD1 supercomputer range, made by Cray Inc. ... The Opteron is AMDs x86 server processor line, and was the first processor to implement the AMD64 instruction set architecture (known generically as x86-64). ...

SGI, among others, has also released storage utilizing LSI products with InfiniBand "target adapters". This product essentially competes with architectures such as Fibre Channel, iSCSI, and other traditional storage area networks. Such target adapter-based discs would become a part of the fabric of a given network, in a fashion similar to DEC VMS clustering. The advantage to this configuration would be lower latency and higher availability to nodes on the network (because of the fabric nature of the network). Silicon Graphics, Inc. ... LSI may mean: Large Scale Integration electronic integrated circuit Latent Semantic Indexing LSI: Love Sex Intelligence, a single on the album Boss Drum by The Shamen The IATA airport code for Sumburgh Airport The Life Span Institute at the University of Kansas in the United States LSI Logic LSI Industries... iSCSI is a protocol that allows clients (called initiators) to send SCSI commands (CDBs) to SCSI storage devices (targets) on remote servers. ... In computing, a storage area network (SAN) is an architecture to attach remote computer storage devices such as disk array controllers, tape libraries and CD arrays to servers in such a way that to the operating system the devices appear as locally attached devices. ... The DEC logo Digital Equipment Corporation was a pioneering American company in the computer industry. ... OpenVMS[1] (Open Virtual Memory System or just VMS) is the name of a high-end computer server operating system that runs on the VAX[2] and Alpha[3] family of computers developed by Digital Equipment Corporation of Maynard, Massachusetts (DIGITAL was then purchased by Compaq, and is now owned...

The cable InfiniBand uses (CX4) is also commonly used to connect SAS Serial Attached SCSI HBAs to external (SAS) disk arrays. 2. ...

See also

• InfiniBand Trade Association
• List of device bandwidths

The Infiniband Trade Association is the standards organization that defines and maintains the Infiniband specification. ... This is a list of device bandwidths: the channel capacity (or, more informally, bandwidth) of some computer devices employing methods of data transport is listed by bit/s, kilobit/s (kbit/s), megabit/s (Mbit/s), or gigabit/s (Gbit/s) as appropriate and also MB/s or megabytes per...

External links

• An Introduction to the InfiniBand Architecture
• The InfiniBand Trade Association homepage
• An InfiniBand™ Technology Overview
• Is InfiniBand poised for a comeback?
• InfiniBand edging into storage market
• OpenFabrics Project SVN
• OpenFabrics Alliance
• InfiniBand simulation model