System-level Switched Topologies for Extreme Performance
Through evolutionary enhancements and technology developments, VMEbus still serves as the dominant bus structure for high-performance, real-time embedded systems. When VMEbus was first introduced, interboard transfer rates of 30 or 40 MBytes/sec of its shared bus backplane was more than adequate for most applications.
As requirements grew, VME acquired new interfaces such as VSB, RACEway, RACE++, VME64, VME320, and 2eSST, thereby ensuring a healthy community of suppliers and a new stream of backwardcompatible products. This continues to be very important to risk-averse government agencies selecting products for long-term programs that require support and maintenance for a dozen years or more.
Well into its third decade of widespread deployment, the recent adoption of gigabit serial interfaces for VME clearly represents the most significant leap of performance throughout its entire history.
VXS: Switched Fabric for VME
Motorola's VME Renaissance announcement in 2003 unveiled a new standard called VXS, officially designated VITA 41 by the VITA standards organization. It defines the implementation of gigabit serial technology for VME in a logically layered specification, while carefully preserving the legacy VMEbus form factors and bus operations. At the top layer, VITA 41.0 defines the connectors, pin designations, dimensions and mechanical structures for cards and backplanes—completely free of any mandates for specific protocols or fabrics.
One of the biggest challenges was the choice of a suitable connector that could handle serial signals at bit rates up to 10 GHz, while not interfering with existing mechanical structures such as legacy VME P1 and P2 connectors and PMC modules. It also had to be readily available from one or more vendors at a reasonable price. The solution was Tyco's MultiGig RT2 connector supporting sixteen differential gigabit signal pairs and four singleended signals. The sixteen pairs are usually ganged into two 4X links, each with four transmit and four receive signals. Each link forms a full duplex channel capable of delivering data at 1.25 GBytes/sec in both directions simultaneously when operating at bit rates of 3.125 GHz on each serial lane.
VXS defines two types of cards: the Payload Card and the Switch Card. Both utilize the same mechanical outline as a standard 6U VMEbus card. The payload card fits the new MultiGig RT2 connector in between the existing P1 and P2 connectors in a position designated P0. Most legacy VME backplanes provide clearance for the new P0 connector, thus allowing insertion of VXS cards for backwards compatibility even though the VXS interface is not engaged.
VXS backplanes have one mating MultiGig RT2 connector for each payload card slot. VITA 41 does not dictate any specific backplane topology and leaves plenty of flexibility for various architectures. The simplest configuration is the VXS switchless mesh backplane shown at the bottom of Figure 1.
Here, three VXS cards are joined in a ring configuration, with each card connected to the other two through two 4X links hard-wired in copper as part of the backplane design. At a serial bit rate of 3.125 GHz, this system supports simultaneous data transfers between the cards in both directions at an aggregate rate of 7.5 GBytes/sec.
As a comparison, a traditional three-card VMEbus system shown at the top of Figure 1 shares a common backplane bus that allows one data transfer in one direction at a time. At rates of up to only about 70 MBytes/sec, it is more than two orders of magnitude slower than its VXS counterpart in Figure 1.
VXS also defines an alternative backplane topology that uses a switch card to join cards using the 4X serial links. Although the switch card still retains the VME 6U form factor, it differs from the payload card in that it does not have P1 and P2 connectors. In their place are additional MultiGig RT2 connectors offering as many as 22 serial 4X links to the backplane for connection to multiple payload cards and other switch cards.
Figure 2 shows a typical "star" topology backplane with two 4X links from each payload card hard-wired to one switch card. Each payload card can connect to any two other payload cards according to the routing paths provided by the switch card. The switch card can be dedicated to a particular serial protocol, which determines the switch path routing through inspection of packet header routing information. Alternatively, the switch card can be "transparent" to the serial traffic protocol, where the switch ports are assigned by some other method, such as an on board controller receiving commands over an Ethernet link.
In either case, unlike the dedicated paths of the switchless backplane, the switch card topology offers excellent flexibility in assigning a data flow within the system. This can be a static data flow pattern established during initialization or dynamic routing during runtime. For highreliability applications, two switch cards provide redundancy because payload cards can be joined through either switch.
Maximum data transfer rates in this switched star VXS system are 2.5 GBytes/sec in each direction for each payload slot, for a full-duplex aggregate rate of 100 GBytes/sec for a 21-slot system!
Figure 2 also shows another key extension to VXS: gigabit Ethernet control plane links. Apart from the dual 4X data links, each payload MultiGig connector also supports two 1X GigE-X ports, each using two differential pairs, one for each direction. Multiple GigE-X ports from the payload cards are sent to one of the MultiGig connectors on the switch card, where they are connected with an Ethernet switch or hub. By allowing Ethernet commands to be passed between boards, these GigE-X ports are intended to replace the control plane function of the VMEbus. With this elegant, high-performance solution for both data and control planes, the VXS standard opens up a new range of system applications previously unattainable.
Many extensions to the base VITA 41.0 specification are now released or in working groups at VITA. These extensions are defined by sub-specifications, like the GigE-X control plane now being developed under VITA 41.6. Most of the other extensions implement various switched fabric standards over the 4X data plane links on VXS including Infiniband (41.1), Serial RapidIO (41.2), Gigabit Ethernet (41.3), PCI Express (41.4) and Xilinx Aurora (41.5).
A follow-on standard being defined under VITA 46 draft specification, also known as VPX, extends the number of MultiGig RT2 connectors on the payload cards by completely eliminating the P1 and P2 connectors. As a result, VPX payload cards support a much higher traffic bandwidth than VXS payload cards. Like the VXS specifications, VPX also has sub-specifications for several fabric protocols and other enhancements.
In the five years since its introduction, dozens of embedded computer hardware vendors have developed hundreds of VXS products. They include switchless mesh backplanes, switched star backplanes, complete card cages and chassis, switch cards, A/D and D/A converters, software radio cards, PMC and XMC carriers, single board computers, DSP and RISC array processors, FPGA cards, memory cards, as well as copper and optical LAN, WAN and disk storage adapters.
VXS fully supports PMC mezzanine modules, allowing systems integrators to take advantage of that enormous base of existing I/O products available prior to VXS. Concurrent with the introduction of VXS, the gigabit serial extension to PMC known as XMC was introduced and later defined under VITA 42. XMC modules connect to the VXS carrier card with two or four 4X links, supporting transfer rates up to 5 GBytes/sec and eliminating the legacy PCI bus bottleneck.
As an example, Figure 3 shows the recently introduced Pentek Model 4207 VXS PowerPC I/O processor with dual XMC sites and a wealth of critical interfaces for high-performance embedded systems. At its heart is an on-board fabric-transparent gigabit serial crossbar switch, highlighting the vital role of serial technology for interconnecting resources within the board and to other boards across the backplane.
VXS System Example
Taking advantage of the tremendous bandwidth afforded by VXS, the 16-channel beamforming signal processor and recording system shown in Figure 4 digitizes 16 antenna IF inputs using four 4-channel A/D converter Pentek 7142 XMC modules. installed FPGA IP cores on the XMC modules perform digital downconversion from IF to baseband producing four 15 MHz baseband signals, each requiring a data transfer rate of 75 MBytes/sec, or 300 MBytes/sec for each XMC.
Serving as a dual XMC carrier, the Model 4207 delivers data from the two XMC modules over one 4X VXS link across the backplane to the VXS switch card at 600 MBytes/sec. A second identical 8-channel subsystem completes the front-end functions of digitizing and delivering data for all 16 channels.
Data streams from each 4207 subsystem are alternately delivered through the switch card into one 4X port of a DSP VXS card where the beamforming signal processing algorithms are performed. After processing, the 1200 MByte/sec input data stream is reduced to 600 MBytes/sec and delivered over the second 4X port, through the switch card, into a third 4207 on the right. Here, the data is buffered in memory, formatted into a Fibre Channel stream and then delivered to a Fibre Channel disk array for high-speed real-time storage.
Aggregate bandwidth within this fiveslot VXS system is 3.6 GBytes/sec, far exceeding traditional VMEbus capability, yet comfortably below the 10 GByte/sec limit for this VXS configuration.
VXS or VPX?
While VPX can deliver much higher aggregate transfer speeds than VXS, a vast majority of system requirements, such as the one shown in Figure 4, can be fully satisfied by the tremendous boost in rates that VXS offers over the legacy VMEbus. Compared to VPX, VXS offers lower costs and is compatible with the thousands of VME boards available from an active community of COTS vendors.
With so many VXS products available today and with the VPX standardization still underway, system integrators can be well served by selecting VXS architectures for today's high performance applications. Investments made by board vendors and system integrators in VXS hardware, interfaces, middleware, software and applications will translate easily into VPX, when system needs dictate. For all of these very tangible considerations, VXS and VPX will likely coexist for many years to come.