Introduction

The original design for the PCI bus was to move high bandwidth peripherals closer to the CPU for performance gains. Graphics-oriented operating systems (OS) and other high bandwidth functions such as Ultra2 SCSI, Fibre Channel, Fast Ethernet, and 3D graphics are consuming more PCI bus bandwidth. This need for more bandwidth has compelled system vendors to find ways of increasing the throughput not only on the PCI bus, but throughout the entire system.

Today, heavily loaded systems move about 90 MB/sec of data on the 133-MB/sec PCI bus, the difference being overhead. The majority of system vendors have implemented a 32-bit PCI bus at 33 MHz. We'll talk about implementing 64-bit PCI bus extensions and increasing the PCI clock to 66 MHz as a way of increasing system bandwidth and efficiency. These PCI changes will result in part in increasing overall system performance; however, other components like the OS and the CPU also play a major role in increasing system performance. Let's take a look at the benefits of implementing 64-bit and 66-MHz PCI technologies.

Since the 32-bit PCI bus is currently able to transfer 133 MB/sec of data less the overhead, what results from changing the bus to 64 bit and 66 MHz? How are PCI bus cycles affected by 64-bit transfers? What are the additions to the PCI bus for 64-bit extensions?

PCI Bus Cycles

Address and data transfers are multiplexed over the same lines on the PCI bus, the address is sent first and then the data. A 32-bit PCI bus has 32 data lines and is able to do 32-bit data transfers and 32-bit memory addressing or 64-bit addressing using two 32-bit PCI cycles known as Dual Address Cycles (DAC).

Memory addressing is not what constrains the PCI bus or system performance. 32-bit addressing allows access to 4 GB of memory--systems such as SMP systems need to address more than this range of memory (see the illustration below). More memory can be addressed with 64-bit addressing in one PCI cycle or two 32-bit cycles using DAC, with the first cycle sending the low address and the second cycle sending the high address.

In the illustration to the right, notice the number of PCI cycles it takes to send the same 128 bytes of data over a 32-bit PCI bus versus a 64-bit bus, assuming the PCI bus is not interrupted. 64-bit PCI bus transactions are more efficient, both for addressing and data, because the number of PCI cycles is reduced to half.

64-Bit Extension Pins

A 64-bit extension to the 32-bit bus architecture requires an additional 39 signal pins which are AD[64::32], C/BE[7::4]#, REQ64#, ACK64#, and PAR64. As we said before, data and addressing are multiplexed over the same pins, either AD[31::00] for 32 bits or AD[31::00] and AD[64::32] for 64 bits. During an address phase, AD[64::32] is used to send the upper 32-bits of a 64-bit address or during a data phase, an additional 32-bits of data. To transfer a 64-bit address in one PCI cycle using the 64-bit bus, you must use the DAC command and assert REQ64#. (Asserting or deasserting a signal means that a particular message is present or missing on the line.) To transfer the additional 32-bits of data on AD[64::32], REQ64# and ACK64# must be asserted.

The Bus Command and Byte Enable Commands are multiplexed over C/BE[7::4]# pins; Bus Commands are transferred during the address phase and Byte Enable Commands during the data phase. An even parity bit, PAR64, protects the AD[63::32] pins from data corruption. PAR64 has the same timing as AD[63::32] but is delayed by one clock cycle.

64-bit transactions are negotiated using a transaction between a master and a target asserting REQ64# and ACK64#. Devices determine if they are connected to a 64-bit bus by asserting or deasserting REQ64# when RST# is deasserted. Only memory commands support 64-bit data transfers.

64-Bit Bus Benefits

Many factors play into overall system performance and affect the industry's progress to 64-bit PCI. The 32-bit PCI bus is not in itself slowing system performance. Peripheral devices such as SCSI, IDE, and Fast Ethernet by themselves do not use the full potential of the current PCI bus. Interaction between the PCI bus, the Host Bridge, DRAM, and the CPU commonly slow down PCI transfers.

Let's also consider the transactions in the system. The CPU communicates with Dynamic Random Access Memory (DRAM) and the Host Bridge: the Host Bridge in turn communicates with the PCI bus and DRAM. Direct Memory Access (DMA) devices transfer data directly to DRAM through the PCI bus as shown here.

The burst rate of data throughput on the PCI bus doubles with 64-bit data transfers. 64-bit DMA devices can move data in 64-bit chunks directly between the PCI bus and DRAM if the PCI bus and DRAM are set up the handle 64-bit transfers. The system with a 64-bit PCI bus is less congested; 64-bit devices in the system get on and off the bus in half the time, making the PCI bus more efficient. In essence, since 64-bit transfers achieve better PCI bus utilization and more devices can be added to the bus before realizing the bus' full bandwidth potential. The more heavily weighted your system becomes with peripheral devices the more it benefits from a 64-bit wide PCI bus.

Notice in the following drawings, the number of components that would be affected by increasing the PCI bus to 64-bits, the host bridge, the DRAM, the CPU and even the OS and driver. Since the majority of transactions are data transfers from the PCI bus to DRAM and from DRAM to the PCI bus, you could consider that increasing the bandwidth of the bus would increase the performance of the system. However, the CPU and the OS may become the bottleneck even if implementing a 64-bit PCI bus.

PCI Bus Speeds, 33 MHz Versus 66 MHz

Another way of increasing the throughput of the PCI bus is increasing the PCI clock speed. PCI systems are now at 33 MHz; the PCI spec defines 66 MHz as a way of increasing PCI bandwidth. 66-MHz devices are great for high bandwidth applications and peripherals. Just as 64-bit architecture can double the bus bandwidth, 66 MHz can double the throughput. The diagram to the right illustrates the increase in throughput.

Some hardware modifications need to be made to the PCI devices and motherboard to allow this increase in clock speed. 66-MHz devices are defined by modifying an existing ground pin to a static signal (M66EN) using a single pullup resistor and adding one bit to the Configuration Status register. 66-MHz PCI requires higher maximum clock frequency and modifying timing parameters. Engineers need to pay close attention to maximum trace lengths, loading of add-in boards, and the maximum pin capacitance of all add-in boards. A 66-MHz bus is capable of operating at 0 to 66 MHz speeds. 33-MHz devices operate at 33 MHz in a 66-MHz bus; likewise 66-MHz devices operate at 33 MHz in a 33-MHz bus.

When designing a 66-MHz PCI bus into the motherboard, electrical problems, chip size, and heat dissipation are some issues to deal with. The loading factor in the 66-MHz bus is significantly reduced from that of the 33 MHz. As the bus speed increases, the total distance data can travel decreases, and the number of slots available on the motherboard decreases. To solve the 66-MHz requirements on a motherboard, 66-MHz PCI bus is typically separated out on another bus using a 66-MHz PCI bridge chip.

64-Bit Market Trends

System vendors are looking for ways of creating the fastest and best systems on the market. Who are some of the players in this market segment? How do the OS, the CPU, and other system components affect system performance?

Vendors Implementing 64-Bit Technologies

CPUs -- Currently, Intel, AMD, and Cyrix CPUs manipulate data and addressing internally in 32 bits. HP, DEC Alpha, Sun Sparc, MIPs, NEC, and IBM RS64 processors operate internally in 64 bits. Intel plans to deliver their new 64-bit CPU, code named Merced, in late 1999. According to an article in SunWorld, October 1997, "HP AND INTEL OFFICIALS REITERATED THEIR CLAIM THAT 64-BIT PROCESSORS WILL BE FULLY BACKWARD-COMPATIBLE WITH 32-BIT APPLICATIONS AND OPERATING SYSTEMS."

PCI Chipsets -- Many of the PCI chipsets on the market support only 32-bit transfers. Some earlier PCI chipsets were only capable of supporting transfers of around 40 MB/sec even though the PCI bus is capable of 133 MB/sec. The sustainable transfer rates on the PCI chipsets should be closer to the theoretical throughput of the PCI bus as shown in the chart above. Newer PCI chipsets provide much more bandwidth and features. The advanced graphics port, for example, on Intel's 440LX PCI chipset is designed to remove the video demand from the PCI bus and improve overall system performance for non-video transactions--3D graphics go directly to memory and bypass the PCI bus. Some vendors will be providing 64-bit PCI chipsets by mid 1998.

OSes -- NetWare versions 3.12 through Moab use 32-bit addressing. Windows NTŪ 3.51, 4.0, and the new 5.0 all use 32-bit addressing. A native 64-bit Windows NT is planned for in 1999 when Intel releases its 64-bit CPU (It is 2001 now, this has not been released yet). Some UNIX OSes like HP UX 11.x, Solaris, and IBM AIX already provide support for 64-bit addressing while remaining backward compatible with 32-bit applications. SCO UnixWare also plans on releasing a 64-bit version. Linux currently supports 32 bit only, but 64 bit is being working on.

Since the OS plays a huge role in system performance by addressing and transferring data to and from memory, when the OS supports only 32-bits when the hardware supports 64 bits, the system is still somewhat limited. Changing the OS from 32-bit to 64-bit addressing is an enormous endeavor. The memory management schemes need to be rewritten, and data transfers and structures need to be set up differently. Even the device drivers need to be modified for 64-bit addressing.

Systems -- HP PA-RISC L Class and N Class are all equiped with 66Mhz 64 bit PCI Bus and 64 bit CPU and 64 bit OSes. DEC Alpha systems have 64-bit PCI systems with a 64-bit Alpha processor but the OS (Windows NT 4.0) supports 32 bits. Micron supplies 64-bit PCI systems using Intel's current 32-bit CPUs and 32-bit OSes. SGI supplies 64-bit PCI systems with 64-bit MIPS processor; Toshiba supplies their TX49 systems with 64-bit processors. IBM's RS/6000 S70 Enterprise servers offer full 64-bit processing, 64-bit PCI slots, and 64-bit OS support in AIX.

According to the article in SunWorld, "A NUMBER OF SYSTEMS MANUFACTURERS, INCLUDING HP, COMPAQ COMPUTER CORP., SEQUENT COMPUTER SYSTEMS INC. AND NCR, HAVE PLEDGED SUPPORT FOR THE 64-BIT ARCHITECTURE AND EXPECT TO ROLL OUT PRODUCTS WHEN MERCED APPEARS. IBM ALSO PLANS TO INCORPORATE THE 64-BIT PROCESSOR TECHNOLOGY IN ITS IBM INTELLISTATION AND IN ITS NEW IBM NETFINITY LINE OF PC SERVER PRODUCTS."

Modifying Drivers for 64-Bit OSes

64-bit data transfers from the device to the PCI bus will be transparent to the driver. The 64-bit PCI devices should autosense the PCI slot for 32 bit or 64 bit and operate accordingly. Since the driver interfaces with the OS, if the OS supports 32-bit addressing, the driver must support 32-bit addressing. Likewise, if the OS supports 64-bit addressing, the driver must support 64-bit addressing. To create a 64-bit driver, you must modify the DMA descriptors to address 64 bits of memory so the hardware can access the pointers to the data in this memory range.

Gigabit Ethernet Trends

HP, Alteon and Packet Engines are supporting 64-bit Gigabit Ethernet now. However, Gigabit Ethernet is slow in coming to the market because OS, CPU, and protocol support is lacking. We have tested HP's implementation of 64 bit PCI bus for their Gigabit Ethernet on HP/UX 11.0 with L Class box. We are glad to report the result are significant faster than some others' implementation. Gigabit Ethernet Specification is still in development and it's hard to commit to hardware and software development without a finalized specification. For Gigabit Ethernet and many other emerging technologies to realize their full potential, systems need to increase their overall performance and efficiency with 64-bit and 66-MHz PCI, and 64-bit OS and CPU support.

Overall System Benefits

A 64-bit PCI bus provides higher overall throughput for high-performance adapters and better system efficiency by providing the same data in fewer PCI clock cycles. A 66-MHz PCI bus doubles the data throughput over the same amount of time. The benefits of both 64-bit and 66-MHz PCI implementations are better PCI bus utilization, better overall PCI bus efficiency, and a substantial increase in PCI bus performance.

Intel conducted a test in 1999 for the Gigabit ethernet. Their test result shows the full Gigabit bandwidth can only be achieved by implementing the 66MHz 64bit PCI bus, as well as use CPU and memory that fast enough not causing bottleneck effect. We have conducted the test in Compute Aid, Inc.'s lab for the Intel Gigabit ethernet card and switch (based on chips made by HP), they work flawlessly.

All logo, trademarks and company names are owned by their respective owners.