Subj: Remember it's a prelim. spec. The phys. channel has yet to be added. Title: Computer Interconnect Specification File: HYDRA::WRKD$1:[THOMPSON.CISPEC2]CISPEC.MEM Date: 15-NOV-81 Authors: D. Thompson J. Buzynski J. Hutchison Contributors: R. Casabona R. Giggi R. Stewart W. Strecker Abstract: This document describes and specifies the implementation independent functional characteristics of the Computer Interconnect (CI) and its system interfaces. Revision History: Rev # Description Authors Date 1 Draft Thompson, 1-MAY-81 Buzynski, Hutchison 2 Add priority queued Thompson, 15-NOV-81 model, misc. changes. Hutchison CI SPECIFICATION--COMPANY CONFIDENTIAL Page 2 1.0 SCOPE . . . . . . . . . . . . . . . . . . . . . . . 5 2.0 NOTATIONAL CONVENTIONS . . . . . . . . . . . . . . . 5 3.0 INTRODUCTION . . . . . . . . . . . . . . . . . . . . 6 4.0 GLOSSARY OF TERMS . . . . . . . . . . . . . . . . . 8 5.0 REFERENCE DOCUMENTS . . . . . . . . . . . . . . . . 9 5.1 CI Design Memoranda . . . . . . . . . . . . . . . 9 5.2 Port Architecture And Implementation Specifications . . . . . . . . . . . . . . . . . . 9 5.3 Higher Level Protocols . . . . . . . . . . . . . . 9 5.4 CI Diagnostics . . . . . . . . . . . . . . . . . 10 5.4.1 CI Node Tester . . . . . . . . . . . . . . . . 10 5.4.2 CI Network Exerciser . . . . . . . . . . . . . 10 6.0 OVERVIEW OF THE CI . . . . . . . . . . . . . . . . 11 7.0 ARCHITECTURAL LAYERS . . . . . . . . . . . . . . . 15 7.1 Port Layer . . . . . . . . . . . . . . . . . . . 16 7.2 Data Link Layer . . . . . . . . . . . . . . . . 18 7.3 Physical Channel Layer . . . . . . . . . . . . . 19 8.0 PORT SPECIFICATION . . . . . . . . . . . . . . . . 22 8.1 Path Selection . . . . . . . . . . . . . . . . . 26 8.2 Sequentiality And Priority . . . . . . . . . . . 28 8.3 Datagrams . . . . . . . . . . . . . . . . . . . 29 8.4 Virtual Circuits . . . . . . . . . . . . . . . . 30 8.5 Datagrams (General-purpose) . . . . . . . . . . 32 8.6 Messages . . . . . . . . . . . . . . . . . . . . 33 8.7 Data Transfers . . . . . . . . . . . . . . . . . 34 8.7.1 Writing Data . . . . . . . . . . . . . . . . . 36 8.7.2 Reading Data . . . . . . . . . . . . . . . . . 39 8.8 Configuration . . . . . . . . . . . . . . . . . 44 8.9 Diagnostic And Booting (Maintenance) Operations 48 8.9.1 Loopback . . . . . . . . . . . . . . . . . . . 48 8.9.2 Resetting Remote Systems . . . . . . . . . . . 50 8.9.3 Writing Remote System Memories . . . . . . . . 52 8.9.4 Reading Remote System Memories . . . . . . . . 55 8.9.5 Starting Remote Systems . . . . . . . . . . . 59 8.10 Maintenance/Configuration Datagram Service . . . 61 8.11 Unrecognized Packets . . . . . . . . . . . . . . 62 8.12 Port Performance Counters . . . . . . . . . . . 62 9.0 CI INTERFACE AND SYSTEM STATE . . . . . . . . . . 63 10.0 DATA LINK SPECIFICATION . . . . . . . . . . . . . 70 10.1 Packetization . . . . . . . . . . . . . . . . . 70 10.1.1 Framing . . . . . . . . . . . . . . . . . . . 71 10.1.1.1 Character Sync . . . . . . . . . . . . . . . 71 10.1.1.2 Packet Length/Type . . . . . . . . . . . . . 72 10.1.2 Addressing . . . . . . . . . . . . . . . . . . 73 10.1.3 Packet Integrity . . . . . . . . . . . . . . . 74 10.2 Channel Control . . . . . . . . . . . . . . . . 75 10.2.1 Arbitration . . . . . . . . . . . . . . . . . 75 10.2.2 Acknowledgement And Retransmission . . . . . . 77 10.2.3 Data Link Performance Counters . . . . . . . . 79 10.3 Data Link Structured Specification . . . . . . . 80 10.3.1 Global Definitions . . . . . . . . . . . . . . 80 10.3.2 Transmitter . . . . . . . . . . . . . . . . . 82 10.3.3 Receiver . . . . . . . . . . . . . . . . . . . 86 11.0 PHYSICAL CHANNEL SPECIFICATION . . . . . . . . . . 88 11.1 Channel Character . . . . . . . . . . . . . . . 88 CI SPECIFICATION--COMPANY CONFIDENTIAL Page 3 11.2 Isolation Scheme . . . . . . . . . . . . . . . . 88 11.3 Bit Transmission And Reception . . . . . . . . . 89 11.3.1 Encoding And Decoding . . . . . . . . . . . . 89 11.3.2 Packet Format . . . . . . . . . . . . . . . . 90 11.3.3 Bit Synchronization . . . . . . . . . . . . . 90 11.3.4 Trailer . . . . . . . . . . . . . . . . . . . 91 11.4 Carrier Detection . . . . . . . . . . . . . . . 91 11.5 Media -- Cables And Couplers . . . . . . . . . . 91 11.6 Input/Output Signal Level Specification . . . . 91 11.6.1 Driver Power Output Specification . . . . . . 91 11.6.2 Receiver Input Power Specification . . . . . . 92 11.7 Input/Output Signal Timing Specification . . . . 92 11.7.1 Driver Output Signal Timing . . . . . . . . . 93 11.7.2 Receiver Input Signal Timing . . . . . . . . . 93 12.0 CONFIGURATIONS . . . . . . . . . . . . . . . . . . 95 12.1 Physical Topologies . . . . . . . . . . . . . . 95 12.2 Installation Rules . . . . . . . . . . . . . . . 95 12.2.1 Installation Rules For Signal Cables And Couplers . . . . . . . . . . . . . . . . . . . 95 12.2.2 Power, Grounding And Site Preparation For Installation . . . . . . . . . . . . . . . . . 96 12.2.2.2 Primary Power Distribution: . . . . . . . . 97 12.3 Addressing . . . . . . . . . . . . . . . . . . . 97 12.4 Installation Of A New Node Onto An Operational Cluster . . . . . . . . . . . . . . . . . . . . 98 12.5 Bus Maintenance And Temporary Configurations. . 98 12.5.1 On-Line Maintenance Procedures . . . . . . . . 98 12.5.2 Use Of Extender Cards . . . . . . . . . . . . 99 12.6 Allowed Configuration Exceptions . . . . . . . . 99 12.7 Mechanical Considerations . . . . . . . . . . . 100 13.0 DIAGNOSTIC AND MAINTAINABILITY REQUIREMENTS . . . 101 APPENDIX A CI OPCODE SUMMARY APPENDIX B EXAMPLE OF NODE BOOTING VIA CI B.0.1 Locally-Loaded System . . . . . . . . . . . . . B-1 B.0.2 Remotely-Loaded System . . . . . . . . . . . . . B-2 B.0.3 A Booting Example -- 11/780 . . . . . . . . . . B-4 APPENDIX C MINIMAL IMPLEMENTATION OF DUAL PATH INTERFACE APPENDIX D IMPLEMENTATION FUNCTIONALITY D.0.1 Implementation Functionality Field . . . . . . . D-1 D.0.2 Implementation Functionality Summary . . . . . . D-2 APPENDIX E ACTIVE EXCHANGE HUB SPECIFICATION CI SPECIFICATION--COMPANY CONFIDENTIAL Page 4 APPENDIX F L0100 LINK SPECIFICATION F.1 TABLE OF CONTENTS . . . . . . . . . . . . . . . . F-2 F.2 REFERENCE DOCUMENTS . . . . . . . . . . . . . . . F-4 F.3 GENERAL . . . . . . . . . . . . . . . . . . . . . F-5 F.4 FUNCTIONAL DESCRIPTION . . . . . . . . . . . . . . F-6 F.5 LINK INTERFACE . . . . . . . . . . . . . . . . . F-18 F.6 MECHANICAL . . . . . . . . . . . . . . . . . . . F-41 F.7 APPENDIX A . . . . . . . . . . . . . . . . . . . F-42 F.8 APPENDIX B . . . . . . . . . . . . . . . . . . . F-43 F.9 APPENDIX C . . . . . . . . . . . . . . . . . . . F-44 APPENDIX G CHANGE HISTORY G.0.1 Revision 1 To 2 . . . . . . . . . . . . . . . . G-1 CI SPECIFICATION--COMPANY CONFIDENTIAL Page 5 1.0 SCOPE This document comprises the functional specification of the Computer Interconnect (CI). Included are the system-independent electrical and mechanical characteristics and the specifications of the data link and controller protocols. The higher level protocols for system intercommunication used with the CI are not specified since many such protocols may be accommodated on a single CI connected system. Also not specified are the implementation-specific details of the various interface (port) architecture and option specifications. 2.0 NOTATIONAL CONVENTIONS All numbers are decimal unless otherwise specified. Hexadecimal numbers are noted by a suffix of "(hex)", for example, 1A(hex). All packets (sometimes called "frames") are shown as composed of 8-bit bytes (sometimes called "octets"). Least significant bytes are shown at the top of the diagram (numbered lowest). Bits of a byte are numbered from right to left (bit 0 on right), with the rightmost bit the least significant. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 6 3.0 INTRODUCTION The Computer Interconnect (CI) is an interconnect for intercomputer communication. A computer is defined (minimally) as a memory-processor pair where the processor may be a central or I/O processor. The need for an efficient, cross-product compatible, general-purpose intercomputer communication link comes from several sources: 1. "Intelligent" I/O systems with significant computing capability used for diagnostics, error correction, transfer control, etc. 2. High availability systems structured as a network of closely coupled computers, each with their own independent memory and operating systems. 3. Load sharing systems, where a number of closely coupled computers share, say, a common file system. An example of a CI coupled cluster is shown below in Fig. 3-1: _______________________________________________ | | | | | | +-----+ +-----+ +-----+ | Ici | | Ici | | Ici | +-----+ +-----+ +-----+ +-----+ | +-----+ | +-----+ | | | | | | | | | | | Pio |----------- | Pc |------------ | Pc |----------- +-----+ | | +-----+ | +-----+ | | | | | +----------+ +-----+ +------+ +------+ | | | | | | | | | M(buffer)| | Ms | | Mp | | Mp | +----------+ +-----+ +------+ +------+ Ici --> CI Interface (Port) Pc --> Central Processor (e.g. VAX-11, PDP-10/20, PDP-11) Pio --> I/O Processor (e.g. HSC50) Mp --> Primary memory M(buffer) --> I/O system buffer memory Ms --> Mass storage Fig. 3-1: Typical CI-Coupled Cluster CI SPECIFICATION--COMPANY CONFIDENTIAL Page 7 The primary characteristics of the CI are as follows: o Data Bandwidth: 70 Megabits/second raw bus bit rate (per path) o Topology: Logically multi-dropped, Physically radial o Maximum number of nodes: 16 with "passive hub", 224 with "active exchange" o Maximum distance between nodes: 90 meters, all nodes maximum of 45 meters from central point. o Medium: 2 coaxial cables (per each of two channels), serial data transmission. o Channel Access Control: Carrier-Sense, Multi-access, Round-robin slotted arbitration. Reserved immediate acknowledgement. o Data Format: Packets, 7 to 4100 bytes (excluding bit synch, header and trailer). o High-Availability: Dual-Path access with no single-component failure of both paths. o Intercabinet Isolation: Transformer coupling (high DC isolation). o Communication mechanisms: Variable length buffer to buffer transfers, messages, datagrams. Some of the primary non-characteristics of the CI are: o Security: There is no protection against malicious users in the areas covered by this specification. This specification consists primarily of the implementation-independent architectural and electrical characteristics of the CI. However, implementation is described where it is deemed to be the most appropriate manner of specification. Additionally, implementation examples are given in some sections to increase clarity and ease of understanding. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 8 4.0 GLOSSARY OF TERMS 1. Port Address - 8-bit addressing number on the CI. Each port address is unique on a CI. May be part of higher level addresses. A system may have multiple ports and therefore multiple port addresses. 2. Path - Logical and Physical connection of Data Links over which packets may be transmitted and received. 3. Port - Interface of a system or node to a CI. May be a single or a dual path interface. 4. Host Computer - A computer system, nominally a processor and memory, with or without other components. In this context, systems are the "host" system connected to the CI via a port. 5. System - A host computer and its port(s). 6. Node - Same as a system (used interchangeably). CI SPECIFICATION--COMPANY CONFIDENTIAL Page 9 5.0 REFERENCE DOCUMENTS The following is a list of CI related documents: 5.1 CI Design Memoranda 1. Strecker, W., "ICCS Arbitration", Internal DEC Memorandum, (November, 1979). 2. Casabona, R. and Thompson, D., "ICCS Architecture Specification", DEC Specification, AUG 79 (Original CI Specification). 3. Thompson, D., "Preliminary Results of ICCS Arbitration Study", Internal DEC Memorandum, FEB 80. 5.2 Port Architecture And Implementation Specifications 1. Strecker, W. and Thompson, D., "VAX-11 CI Port Architecture Specification", DEC Specification, MAR 80. 2. Carrafiello, M., "CI 780 Functional Specification", DEC Specification, FEB 81. 3. Dossa, Don, "LCG CI Port/Port Interface Specification", DEC Specification, AUG 81. 4. Documentation on HSC 50 CI architecture and interface (to be supplied), target date unknown. 5.3 Higher Level Protocols 1. Strecker, W., "Systems Communication Architecture", DEC Specification, JAN 81. 2. Kronenberg, Nancy, "VMS Systems Communication Services Specification", DEC Specification, AUG 81. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 10 5.4 CI Diagnostics 1. Sarni, John. "CI Diagnostic Strategy", DEC Memorandum, JAN 82 (target date). 5.4.1 CI Node Tester - 1. Heller, Liz and Giggi, Bob. "CI Node Tester Functional Specification", DEC Specification, 4 MAR 81. 2. Hennesey, Rich. "CINT Control Software Design Specification", DEC Specification, AUG 81. 3. Klumpp, Jim. "CINT Responder Software Functional Specification", DEC Specificaton, MAR 82 (target date). 5.4.2 CI Network Exerciser - 1. McMillen, Dave. "CI Network Test Protocol Specification". 1 MAY 81 (target date). 2. Roemer, Fred. "CI Exerciser Design Specification", DEC Specification, SEP 81. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 11 6.0 OVERVIEW OF THE CI The CI is a logically multi-dropped "computer room" interconnect for building "closely-coupled" multicomputer clusters, where computers may be central processors or dedicated I/O or special purpose sytems. CI nodes usually have dual paths for high availability. However, both paths are regarded as parts of a single CI. All dual-path interfaces, or "ports", must be designed such that a "very low" probability exists that a failure will incapacitate both paths with respect to communication between other pairs of nodes. Rather than specify the actual probability, it has been defined that low probability means that the failure of any single component of the node (at any level) may not simultaneously preclude cluster communication on both paths. Such a failure within a node may, however, preclude it from communicating with any other node. Single path CI nodes are required to be connected to path 0. This ensures that all nodes on a CI may communicate at least on path 0 (under error-free conditions). A logical diagram of a CI cluster is shown below in Figure 6-1. CI PATH 1 <----+----------+---------------------------------------+-----> | | CI PATH 0 | <----------+---------+---------+-------------------+----------> | | | | | | | | | | | | | | +---------+ +--------+ +--------+ +---------+ | Port 0 | | Port 1 | | Port 2 | | Port 3 | |---------| |--------| |--------| |---------| | | | | | | | | |Computer | |Computer| |Computer| |Computer | +---------+ +--------+ +--------+ +---------+ Fig. 6-1: Single and Dual Path Ports The physical implementation of the CI is a radial configuration, with a central hub coupler. The hub coupler is used for increased electrical reliability. Additionally, the "star" type configuration is efficient with respect to cable lengths in a "computer room" environment. The type of hub determines the maximum number of nodes. For up to 16 nodes, a "passive" transformer-coupled hub is used. For extended distance connection of only two nodes, a special coupler may be provided in the future. It will also be possible to use an active "exchange" hub for up to 224 nodes to increase the available cluster bandwidth (by allowing multiple, simultaneous CI SPECIFICATION--COMPANY CONFIDENTIAL Page 12 transfers between node pairs). Special restrictions are likely to be imposed in such configurations. An example of a configuration using the 16-node coupler is shown below in Fig. 6-2. More information in this area can be found in the Configuration section. The bus is multi-access with no single node performing a bus master function. As a result, the removal of any node from the bus does not preclude continued communication between the other nodes. Such removal can be accomplished on-line, that is, while the other nodes of the cluster continue to communicate. Three major types of communication mechanisms are supported. Datagrams, the simplest, provide "best-effort" delivery of single data blocks from one node to the next. Messages provide a more reliable transfer of similar data blocks using "virtual circuits". Virtual-circuit controlled delivery is sequential, non-duplicated, and error-free. The block data transfer service (DMA) is used to move large blocks of buffer data. Large blocks are divided into multiple sub-blocks and non-atomically transferred. The data service also uses virtual circuits and is therefore guaranteed sequential and error-free. The CI is packet-oriented. Packets are integral numbers of bytes from 8 to 4100 bytes in length (excluding bit synch, header and trailer). Transmission is Manchester-encoded serial at 70 Megabits/second. The physical interconnect consists of two coaxial cables per path (one for transmit, one for receive). The cable connections are transformer coupled on the interface card for high inter-cabinet isolation. The central hub is an impedance-matching coupler for reducing reflections and increasing immunity between individual node connections. Dual-path clusters have two separate hubs with each system connected to each hub by two cables (one transmit, one receive per hub). Addressing is on a per node basis, with each node having one address that is unique on the particular CI. Packets are framed by a special start character and a byte count carried in the packet header. A 32-bit CRC character is attached at the end of the packet for detection of transmission errors. The bus is multi-access with equal priority for all nodes (time averaged). The arbitration method is distributed and uses a slotted, dual-priority, round-robin algorithm with carrier detection. Any collisions are detected by the packet integrity check. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 13 +-----------+ | |-----------------+ | SYSTEM 0 | | | | | | | | | |----------------------------+ +-----------+ | | | | +-----+ | +-----------+ | HUB | | | |---------------| 0 | | | SYSTEM 1 | | | | | | +-----+ +-----+ | | | | | | | |-------------------------| HUB | +-----------+ | | | 1 | | | +-----+ | | | +-----------+ | | | | | | | | | SYSTEM 2 | | | | | |-----------------+ | | | | | | | | | | +-----------+ | | | | | | +-----------+ | | | |-------------------+ | | SYSTEM 3 | | | | | | | | | |----------------------------+ +-----------+ Figure 6-2: Dual-path, Hub-coupled (passive) CI Cluster (16 nodes maximum) Acknowledgement of packets is immediate, that is, bus time is reserved immediately after each packet for the receiver to send back an acknowledgement. The type of acknowledgement returned depends on the result of the transmission. If any error in the packet was detected, no acknowledgement is sent and the transmitter detects the problem via timeout. If the packet was correctly received and buffered (at least in the interface), a positive acknowledgement (in the form of a special packet) is sent to the original packet source node. If the packet is correct but the interface is unable to buffer it, a negative acknowledgement is returned. Retransmission occurs in any case except positive CI SPECIFICATION--COMPANY CONFIDENTIAL Page 14 acknowledgement, following a defined algorithm. The limits on the algorithm are designed such that if failure is detected at the limits, it is very likely that a "hard" failure has occurred. Note that that certain transient "errors" can occur in correctly operating clusters due to congestion. These errors are effectively handled by the retry mechanisms. A CI interface, generally referred to as a Port, is conceptualized as an intelligent controller designed to reduce the load on the host processor for intercomputer communication. The functionality of the interface is defined with respect to the CI by this specification and on the host side by the implementation port architecture and hardware specifications. The Port consists of three functional components. These are the Port Processor, Packet Buffer, and Link/Front End as shown in Fig. 6-3. The Port Processor interfaces to the host memory and controls the Link and Packet buffers. The Port Processor is responsible for data mapping, address translation, buffer loading, packet interpretation, and control of the host interconnect and Link. The Packet Buffer is a temporary storage interface between the Link and the Port Processor. It is not imperative that the buffer actually exist. Rather, it is a "virtual buffer". The concept of buffer "fullness" is represented by the temporary inability of the interface to accept the data from the bus at the bus rate. For example, the buffer might actually be a small FIFO. If an implementation does not fully buffer packets, it must be highly likely that the data can be accepted for the entire packet at the bus rate. Cluster bandwidth could be greatly reduced by ports that lost a high percentage of packets due to buffer overflow. The Link is responsible for the implementation of most of the data link protocol and moving the data between the CI and the Packet Buffer. The Front End performs the bit level operations of bit encoding/decoding and carrier detection. +--------------+ +-----------+ +-----------+ CI | | | | | | H --| PORT | | PACKET | | LINK/ |-- PATH 0 O | PROCESSOR |---| BUFFERS |---| FRONT END | S | |---| |---| | T --| | | | | |-- PATH 1 | | | | | | +--------------+ +-----------+ +-----------+ Fig. 6-3: CI INTERFACE FUNCTIONAL COMPONENTS CI SPECIFICATION--COMPANY CONFIDENTIAL Page 15 7.0 ARCHITECTURAL LAYERS The CI is architecturally specified as three layers. The bottom layer, the Physical Channel, describes the transmission medium, bit encoding/decoding and carrier detection functions. The middle layer, the Data Link, encompasses the functions of data packetization and channel control. The top layer, the Port, describes the protocols used between ports to provide the highest level communication mechanisms. The interface of the Port to the next highest layer is implementation dependent and beyond the scope of this document. This necessitates overlap of this specification with the various port architecture specifications, which do, in fact, specify this interface. It is intended that the specification of the internal function of each layer be independent of the other layers such that implementation changes within a layer are effectively isolated. In practice, however, it is recognized that hardware/firmware/software tradeoffs may not dictate such a separation. Ideally, information used in one layer is ignored and untouched by all the lower layers through which it is passed. Information used by a layer is "peeled" off before passing to a higher layer. The exceptions to this are in addressing and framing. Framing at the Port level is implicit in the Data Link framing. Addressing information is also used by both the Data Link and Port levels. These three layers correspond to a portion of the four-layered System Communication Architecture (SCA). The Physical Interconnect layer of the SCA is implemented in the Physical Channel and Data Link layers of the CI. The Port layer of the CI corresponds to the Port portion of the Port/Port Driver (PPD) layer of the SCA. Note that the implementation dependent interface between the Port and Port Driver is shown with a broken line, since it is not included in this specification. Fig. 7-1 shows the correspondence of the architectural layers. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 16 SCA CI | | 3. I/O Applications (SYSAP) | | --------------------------------+ | 2. System Communications | Services (SCS) | | --------------------------------+ | | 1. Port/Port Driver (PPD) +- - - - - - - - - - - - - - - - | 1a. Port | --------------------------------+------------------------------- | 0b. Data Link | 0. Physical Interconnect (PI) +------------------------------- | 0a. Physical Channel | --------------------------------+------------------------------- Fig. 7-1: SCA and CI Architectural Layers 7.1 Port Layer The Port layer provides three main communication services over the CI: Datagram, Message and DMA. The Datagram service provides delivery of a single packet to a single remote destination on a best-effort basis. The packet may be variable size up to 4089 bytes (text length). The actual maximum size may be limited by prior agreement between nodes to less than that. However, all nodes must accept datagrams of a minimum of 58 bytes in length (text). A datagram will be discarded if buffering for it is unavailable in the receiving node. No provision is made for notifying the sender of the loss. However, provisions do exist for counting discards at the receiver. Higher-level protocols using the datagram service must accomodate its characteristics. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 17 The Message service provides a sequential, error-free, delivery service via node-to-node independent virtual circuits. The virtual circuit state is maintained in each node on a per-port basis for all active ports. The circuit state consists of one bit indicating that the circuit is open (on) or closed (off) and two single-bit sequence numbers, one for transmitting message packets and one for receiving message packets. Before a message can be successfully sent from one node to another, the corresponding sending and receiving sequence numbers must be equal and the circuits opened. This is accomplished via higher-level protocols, an example of which is the System Communication Services of the System Communication Architecture. The specification of such a protocol is beyond the scope of this document. The Message mechanism is only to be used for "trustworthy" or highly predictable communications since errors of any type result in circuit closure (requiring reinitialization). The Block Data transfer mechanism provides a reliable, multiple-packet transfer service for moving blocks of data from a buffer in one node to a buffer in another node. This mechanism uses the same port-to-port virtual circuits used for Messages. This guarantees sequential, non-duplicated transfers. Data transfers can be accomplished in both directions, namely a "read" or "write" with respect to one of the nodes. The buffers are named and the name must be passed in prior agreement via a higher-level protocol. Such a protocol is beyond the scope of this document. Note that any errors in the Data transfers close the virtual circuit, disabling both data and message communications. Maintenance and Configuration services are also provided for establishing system configuration/state, booting and aiding in diagnosis of ailing nodes. These mechanisms provide datagram class service. Path selection is necessary in dual-path clusters. This function is seen as a port layer function primarily because it is legitimate for data link transfers to simultaneously occur on both paths (with certain restrictions). The port layer has the duty of multiplexing packets between paths. The path is normally selected on a random, equal-priority basis to balance the load. However, nodes may elect to use a particular path for reasons of path verification or communication with single-path nodes. A node does not necessarily have to support the ability to simultaneously use both paths (for transmission or reception). An example of a minimal implementation of a dual-path interface using a considerable amount of shared hardware is discussed in the Minimal Implementation appendix. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 18 7.2 Data Link Layer The Data Link layer provides the port with reliable delivery of single packets across the Physical Channel. It performs packetization of blocks of data and channel access/control. Packetization includes framing, addressing, and integrity checking. Framing is accomplished by marking the beginning of the packet with a special "start-of-header" character, called the character sync. The end of the packet is determined by a packet length, which is included in the packet and immediately follows the character sync. Addressing is accomplished by following the packet length with destination address. The address is the node number. Each node has one address which is unique on the particular CI to which it is connected. A second copy of the destination address is sent in complemented form to increase the reliability and preclude single component failure sources. The source address is carried to allow the destination port to return acknowledgement. The integrity of the packet is checked by carrying a 32-bit Cyclical Redundancy Check (CRC) Character. The CRC computation is performed in the sending interface and the result is appended to the packet. Upon receipt of the packet, the computation is repeated and the result checked against the value sent with the packet. If the comparison is true, the probability is high that the packet was, in fact, correctly received. Channel access and control consists of arbitration, acknowledgement and retransmission (if necessary). Arbitration uses a slotted, round-robin priority scheme. Node priority is changed on a round-robin basis such that all nodes have equal avcerage priority. The "slot" is the discrete time interval required under worst case configurations to synchronize all nodes. The slot is non-zero due to bus delays. Carrier sense is used to determine when transmission should begin, with each node waiting a unique number of slots in a saturated system. A node receiving a packet must immediately acknowledge the receipt. Immediately following the transmission of a packet on the bus, all nodes wishing to transmit must wait a minimum time for the packet destination node to return an acknowledgement packet. The nature of the acknowledgement is dependent on the results of the transmission. If the packet was not successfully received due to collision, bus error or busy receiver (on other path), the packet is not acknowledged The transmitting node detects this by timing out on acknowledgement receipt (NO_RSP). If the packet was successfully received and buffered in the interface, a positive acknowledgement (ACK) packet is returned. If the packet was correctly received but the interface was unable to buffer it, a CI SPECIFICATION--COMPANY CONFIDENTIAL Page 19 negative acknowledgement (NAK) is returned. In the case of transmission failure (NO_RSP/NAK), the sending node makes an equal probability decision to immediately arbitrate and transmit or wait a delay. If delayed, the same decision is made at the end of the delay period. This is repeated until retransmission occurs. This random delay (exponentially distributed) is used to break possible deadlock situations. Retransmissions are limited. The limits are designed such that under worst case conditions, failure to correctly transmit a packet in a correctly functioning cluster will not occur more than once per year. For further detail, see Data Link subsection on acknowledgement and retransmission. | | The data link layer is logically duplicated in dual path | interfaces. Implementations may actually share a considerable | portion of the hardware in dual path implementations (see data | link specification and minimal dual path implementation appendix). 7.3 Physical Channel Layer The Physical Channel layer is the interface between the data link layers of two nodes. The data packet is conditioned and sent out on the media by line drivers to be received on the "other end" of the media. The data, addressing, crc, leader and trailer are complete when the packet is passed to the Physical Channel. This layer performs media specific tasks in transferring the data over the bus. Included are generating the data clock, manchester encoding and decoding the data, and driving and receiving the media, generation of carrier detect logic signals, and the transport of data signals from node to node. This layer provides the electrical compatibility between nodes and a dependable means of data transport in a hostile environment. A major design goal is to provide the ability to "dock merge" the system, that is to ship working parts to the customer without first testing the assembled configuration (ie, computer cluster). The two important factors are electrical isolation and interface circuit compatibility between nodes. In order to guarantee interface compatibility and "dock mergability" a single implementation for the Physical Channel is intended to be common between all CI options presently planned. The actual circuit design, component layout and media are critical in meeting the interface specifications. Another goal of the design is to provide for reliable and available operation. Reliable operation means that normal environmental affects and component failure (or service technician mistake) will not break the communication channel. Environmental affects considered are radiated/conducted interference, low frequency voltage offsets between nodes, and static discharge. The design also allows any one component to fail (or be yanked CI SPECIFICATION--COMPANY CONFIDENTIAL Page 20 out) without disrupting communications over the remainder of the channel. The Physical Channel is multipath and each path is half duplex in theory. In practice there can be only one path or there can be two paths with logic shared between them giving reduced cost/performance and still maintain CI compatibility. A path of the CI physical channel could be full duplex except for the nature of some types of couplers. Many couplers will combine inputs from all transmit cables and drive this signal on all receive cables, only one driver at a time can use a path and full duplex isn't possible for any node. The complete CI channel has four coaxial cables, two for transmitting and two for receiving data. There is one send/receive pair per path and two independent data paths. Each path transfers 70 Megabit of serial data per second at baseband frequency. In a path the transmit and receive data is on separate media as opposed to being multiplexed onto one coax cable (i.e., TDM or frequency multiplexing). The transmit and receive cables are separate to allow future expandability using an active center hub: an amplifier or packet switching exchange. Redundant paths provide for reliable operation as one failure can only affect one path and for availability as one path can be repaired while the other provides communication over the physical channel. If the redundancy or additional data bandwidth of two path CI clusters is not important for an application only one path is needed. The functional blocks of the Physical Channel at an individual node are: two drivers, two receivers, the carrier detect circuits, encoders, decoders, and the transmit clock. All of these get their power only from the host node and are connected to logic reference of that node (logic ground). The circuits in separate nodes do not share a common logic reference. These functional blocks are described in the next two paragraphs. From a practical design standpoint the driver, receiver and carrier detect circuits must be dedicated to their cables (two of each used for a two path system) while the rest of the parts may be shared between paths to reduce cost. The driver is connected to that path's transmit cable and the receiver/carrier detect circuits to the receive cable with two separate transformers on the circuit board. The carrier detect function is to provide a logic signal that is true while valid data signal levels are present on the receiver input (for each path). Carrier detect cannot tell if the data is sensible or if only one driven signal is present. The carrier detect circuit is intended to provide for "look before talk with no collision detect" Link layer protocol. Circuits at the node which may be easily shared between paths are the encoder, decoder, and transmit clock generator. The encoder/decoder are used to combine/separate the data from the clock in a Manchester scheme. These circuits run at a data rate of 70Mbit/sec. The coding scheme could also be called synchronous baseband biphase encoding and guarantees a data transition in the middle of every bit time by exclusive-or'ing the clock with the CI SPECIFICATION--COMPANY CONFIDENTIAL Page 21 data. The coding scheme not only allows clock and data to share one signal line but also eliminates the low frequency (and DC) components of the signal. The result is the fidelity of the media need not be as high, there is no skew between clock and data, and transformer AC coupling is possible. The final circuit at the node is the transmit clock generator. The Physical Channel layer makes a crystal stabilized 140.00 Mhz. clock which is used to encode the data and is counted down and given to the Link Layer circuits as a system clock. The Physical Channel has a center hub to connect all cables from the individual nodes together in a radial or star arrangement. The center hub is composed of up to two independent "star couplers"; there is one coupler for the each send/receive path (redundancy). The coupler combines the signals from all node transmit cables for a path and then splits the composite signal up equally into all the receive cables going back to the nodes of said path. The coupler also terminates all cables in their characteristic impedance. A single coupler may be a passive device without any power source for up to 16 node systems (inclusive). The shields of all the cables in a path are connected together to the coupler chassis. The cable shields and the coupler chassis are isolated from all the nodes at low frequency. Note that the cable shields are RF decoupled to node chassis (I/O panel) using a capacitor. The coupler chassis may be connected to the earth reference provided by another DEC chassis but this is unnecessary. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 22 8.0 PORT SPECIFICATION This section describes, in detail, the functionality of the Port layer required by all CI ports for each communication mechanism. Not all aspects of the protocols are specified. In particular, the initialization protocols are not specified. This is for the following reasons: Although the CI is specified independent of implementation, due to its high performance, it is very likely that portions of the protocols will be implemented in hardware and firmware, rather than in host software. In fact, the line has been drawn in just such a manner in the first implementations. Therefore, the CI specification includes primarily the aspects of the protocols that are currently implemented in hardware and must therefore not change for compatibility of nodes over the life of the CI. This does not restrict hardware/software partitioning of future implementations, but merely defines certain portions of the protocol that must "appear" to be identical from the perspective of another node on the CI. Obviously, the initialization protocols in communicating nodes must be compatible, but, in fact, multiple protocols may be used, depending on the particular | application. For the purpose of this specification, the port/data | link interface is modelled as a pair of buffers; one transmit, | one receive. A buffer holds the entire body and parameters of a | single packet. The port process performs operations as given by | the port driver by loading the transmit buffer. The data link | layer is responsible for transmitting the buffer and returning | transmission status to the port. | | Whenever the data link receives a packet on the CI, it loads | the receive buffer. The port unloads the packet and either passes | it to the port driver or performs an operation internal to the | port. | | Note that the specification of the port/port driver interface | is beyond the scope of this document. Such information can be | found in the individual port architecture specfications. | | The operations of the port, both those passed from the port | driver and those intiated by received packets, are performed with | certain guarantees of ordering. For further detail see the | subsection on sequentiality and priority. All implementations must be compatible with this model. Not all mechanisms of this layer must be implemented in all ports. A minimum subset as noted in the descriptions, must be provided. The services are specified in terms of actions performed by the port layers in the sending and receiving nodes and the format of packets used in the service. The packets are, in fact, described in the port layer as a packet body with a set of parameters. The subsequent sections describe primarily the packet bodies. of CI SPECIFICATION--COMPANY CONFIDENTIAL Page 23 The general format of the packet BODY passed between the port and data link layers is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | FLAGS | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | | | TYPE SPECIFIC | | | | | :B - 1 +-------------------------------+ + BODY_LEN Where: Byte Bits Field Description B <7:0> OPC Opcode. Indicates type of packet. All opcode values with bit 7 are reserved for local port use. B + 1 <7:0>= FLAGS Flags. Special FLAGS<7:0> miscellaneous opcode modifiers. Flags used for each type are shown in each packet diagram. All unused flags Must Be Zero (MBZ). B + 1 <7>= PF Packing format. Implies FLAGS<7> type of data packing to be used in destination port. Value is implemen- tation specific. Used in DG, LP and MSG packets only. F Force Reset. Value of 1 indicates that reset is unconditionally to be performed. Used in RST packet only. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 24 DSA Default Start Address. Set to indicate that a implementation-specific default starting address should be used. Used in STRT packet only. P Basic packet size for return data transfer. P = 0 for 512 bytes. P = 1 for 576 bytes. Used in data and maint- enance data packets only. B + 1 <6:4>= M Packet size multiple. FLAGS<6:4> Packet data length = (M + 1) * basic size determined by P. Used in data packets only. B + 1 <1>= NS Sending Sequence Number. FLAGS<1> Used for virtual circuit packets. B + 1 <0>= LP Last Packet. Indicates FLAGS<0> last packet of a data transfer. B + 2 TYPE Packet type specific through SPECIFIC information. B + 1 + BODY_LEN The parameters passed to or from the data link layer with each packet BODY are: 1. DST -- node number to receive the packet if transmitting or number of this node if receiving this packet. The value ranges from 0:15 if a passive hub is in use (arbitration enabled) or 0:223 if an active exchange is in use (arbitration disabled). 2. SRC -- Number of this node if sending this packet or number of the node which sent the packet if receiving this packet. The value range is the same as the Destination address. 3. BODY_LEN -- Length of the body of this packet in bytes. Type specific, value specified for each type of packet. Note that BODY_LEN = Packet length minus 5 (see Data Link Specification). CI SPECIFICATION--COMPANY CONFIDENTIAL Page 25 4. STATUS -- Packet status is passed with received packets and returned subsequent to transmission of packets. Possible received packet status values are: - OK_0/1 -- Packet successfully received in entirety on path 0/1. - OVERSIZE_0/1 -- Packet was too large to buffer (in the implementation) but was correctly received by the data link layer on path 0/1. A minimum of the first 48 bytes of the packet body and all other parameters are intact. Possible transmit packet status values (after data link retry) are: - ACK_0/1 -- Successful transmission on path 0/1. - NAK_0/1 -- Receiving interface was unable to buffer the packet within specified retry limits on path 0/1. - NO_RSP_0/1 -- Packet was never successfully acknowledged by destination interface within specified retry limits on path 0/1. The path status values may be mixed with one status for each path used. The data link layer may opt to try one or both paths. The subsequent sections show the type-specific values and formats for each packet. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 26 8.1 Path Selection Dual paths are supported by the CI for implementation of highly reliable distributed clusters using standard systems. Dual path clusters are designed such that a single component failure will not totally prevent communication between other functioning systems. Performance may be degraded until repairs are made. | | The port layer is primarily responsible for the multiplexing | and demultiplexing of packets between paths. The data link layer | is logically duplicated. Note that the parameters previously | specified for the packet include path (data link) selection | information. The port is responsible for maintaining the sequence | of packets on a port-to-port basis. For details, see section on | sequentiality and priority. A dual path cluster must have two sets of CI cables for each port and two couplers or exchanges. However, an interface may range from total hardware (and software) duplication (two separate data links) to a minimal implementation in a single interface (with mostly shared hardware). Note that the interface still consists of only one port. Reduced implementations are discussed further in the Minimal Implementation Appendix. It should be noted that the minimal implementation is specified because of the additional transmission failure mode it introduces. That is, if the interface is only capable of receiving packets on one path at a time, it may ignore packets transmitted to it on the other bus. The parameters of the data link retransmission algorithms were selected to compensate for this effect. Another major implication arises from dual paths. This is the selection of path for transmission. Interfaces must support two types of control of path selection: Select and Automatic. In select mode, the interface must be capable of transmitting on a specified path. This mode is used for controller configuration transactions and requires that packets be returned on the same path in response to a request packet. In addition, it is likely that this feature be required for diagnostic purposes. The majority of packets should be sent in Automatic mode. In this mode, the path for transmission of a packet should be made by a two-way, equally probable, statistical choice ("coin-flip"). This provides a statistical load sharing between paths and results in frequent exercising of both paths for early failure detection. All retransmission attempts of a packet should be made on the same bus until the limits are reached. If a failure occurs, the other path can be tried, following the same retransmission algorithms until the limits are exceeded. At this point, it is most likely that a hard failure has occurred or the destination interface is not in a state where communications are enabled. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 27 A node should maintain a table of path status for each node. Such a table should indicate whether a path to a given node is correctly functioning or not. The table values should be based on prior transmission results. The table should be used to make automatic mode selection from only known good paths. This should prevent known bad paths from being selected with great frequency, since such transmissions waste bandwidth. If a bad path is to be retried, it should be done with much lower frequency (every few seconds or tens of seconds, for example). Single path ports are allowed, but not encouraged. All single path ports must be connected to path 0. This ensures that all nodes on a CI may communicate. The danger that arises from the use of single-path ports is that path load may be unequally distributed to an extent that degrades performance. Such effects must be considered when configuring a mixed single/dual path cluster. Note that the minimal implementation of a dual path port discussed in the appendix has very little incremental cost over a single path interface. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 28 8.2 Sequentiality And Priority | The operations of the port, both those initiated by the port | driver and those intiated by the receipt of packets are performed | at multiple priorities. This is to reduce the latency of | performance-critical transactions. Real time response is not | guaranteed, since latency will be primarily a function of cluster | load. However, this mechanism can be used to disproportionately | divide bandiwdth as desired. | | Sequentiality must be preserved on a per-packet basis between | node pairs. Prioritization is optimally performed for each | packet, but may, in fact, be limited due to pipelining in | implementation. The only guarantees of prioritization are in fact | on a per-operation basis. Operations are service specific, but | consist of sending a single packet unless otherwise specified. | | The following is a set of rules defining the sequence and | priority of operations: | | Two operations become available to the port. Operation OP0 | arrives at time T0. Operation OP1 arrives at time T1. T1 > T0. | | 1. If effective priority of OP0 >= effective priority OP1 then | OP0 is performed in entirety before OP1 is begun. | | 2. If effective priority of OP0 < effective priority OP1, then a | best effort is made to preempt OP0 between packets, perform | OP1 in entirety, and then resume OP0. Best effort must be | limited to 4 packets. That is, no more than 4 packets of OP0 | should be transmitted after T1. | | | There are 5 levels of priority number 0 to 4. Priority 4 is | the highest. The term effective priority is used because all | priorities may in fact not be implemented in certain ports. Most | importantly, priority relationships between operations must be | maintained as specified. A different priority must be implemented | for each supported operation that is specified to have a different | effective priority. The relationship between any two implemented | priorities must be the same as the corresponding effective | priorities. | | If the data link is physically duplicated, it may be possible | to transmit packets on both paths simultaneously. Due to | independent retransmisison, the sequence of arrival could be | different than the sequence of delivery to the data link. To | preserve sequence on a port-to-port basis (required), the | transmission of a packet to a remote port must be completed in | entirety before another packet for the same port is delivered to | the data link. The receiving must process the received packets in | order of real time arrival. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 29 8.3 Datagrams The datagram service provides a best-effort delivery of a single packet from one node to another. A datagram may be successfully transmitted (STATUS = ACK) but be discarded by the port if unable to unload it immediately from the receive buffer due to buffer shortage at subsequent layers. Several of the communication mechanisms of the CI provide datagram quality service, notably those of maintenance and configuration. Additionally, a general purpose datagram is provided for used by higher-level protocols that require such service. Generally, protocols with unpredictable buffering requirements should use the datagram service, since the port-level discard provides some degree of protection against buffer exhaustion problems. | | The receipt of datagrams of any type that would be passed to | the port driver may be controlled on a port-to-port basis. A | control bit, Datagram Queue Inhibit (DQI) should be kept for every | possible port. If this bit is on for a particular port, all | incoming datagrams for that port must be discarded. This bit | should be cleared at initilization so that all datagrams are | initally received. | | In addition to a DQI bit for each possible port, at least one | DQI bit should be provided to control datagram reciept for illegal | port numbers. It is permissible, though not required, to use a | single bit for all illegal port numbers. A performance counter tracks the number of datagrams discarded due to buffer shortage (see Port Performance Counter section). This allows higher-level protocols to determine if sufficient buffering is available for the required performance level. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 30 8.4 Virtual Circuits A virtual circuit is the mechanism used to provide a higher quality of service for a series of packets. Delivery of packets under virtual circuit control is guaranteed to be sequential, error-free and non-duplicated. The circuit is constructed of a set of state variables in the sending and receiving nodes. Virtual circuits are maintained on a per-node basis. That is, each of a pair of nodes have virtual circuit state with respect to the other node. Therefore, in each node, an array of state values is maintained, with one set per node "connected" by virtual circuit. Several of the communication mechanisms specified in the Port use virtual circuits. In fact, the same circuit is simultaneously shared by any of the mechanisms that are in use. The circuit guarantees are on a per-packet basis, independent of the particular packet type (as long as that type uses the circuits). The state of a circuit consists of three bits in each node: Circuit State (CST), Sending Sequence Number (NS), and Receiving Sequence Number (NR). The circuit state reflects whether or not the particular circuit has been initialized. Its values are Open (initialized and "on") and Closed (uninitialized and "off"). The bit value representing each state can be implementation dependent with suggested values being "1" (true) for Open and "0" (false) for Closed. The Sending Sequence Number is the number of the next packet to be sent (or the value of the current packet for which delivery is being attempted). The Receiving Sequence Number is the number of the next packet to be received. On the sending end, when a packet is to be delivered, it is loaded with the current NS value in the defined bit of the FLAGS field. When the data link layer returns successful transmission status, the NS value is incremented modulo 2 (complemented). In the receiving node, when a packet is received for the circuit, the value of the NS bit of the FLAGS field is checked against the current value of NR. If equal, the packet is accepted and NR complemented. If not equal, the packet is discarded. This is the mechanism for discarding duplicates. If an acknowledgement (at the data link level) is lost due to bus error, the sending end will retransmit the packet. If the packet was actually received (and only the acknowledgement corrupted), NR will have be complemented and the packet accepted. Upon receipt of the retransmission, NS will not equal NR and the second packet (a duplicate) will have been discarded. The circuit state determines whether or not packets may be sent or accepted on the circuit. If the circuit is closed in the sending end, no virtual circuit packets may be sent for that circuit. If the receiving node state is closed, then incoming CI SPECIFICATION--COMPANY CONFIDENTIAL Page 31 virtual circuit packets for that circuit will be discarded at the port level. The circuit state should be closed if the transmission of a virtual circuit packet fails. Additionally, a node may close its circuit at any time. In general, any type of error that may result in disruption of sequentiality should result in circuit closure. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 32 8.5 Datagrams (General-purpose) All nodes must provide bidirectional (send and receive) general purpose datagram service. The minimum datagram TEXT_LEN | that a node must be able to handle is 58 bytes. Larger values up to 4089 bytes may be used between nodes based on prior agreement. The prior agreement on increased size limits is left to a higher level protocol. The BODY format of DG is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ |PF | MBZ | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | | | TEXT | | | | | :B + 1 +-------------------------------+ + TEXT_LEN Where: Byte Bits Field Description B <7:0> OPC OPC = 1 for DG. B + 1 <7>= PF Packing format. Implies FLAGS<7> type of data packing to be used by certain types of ports. Value is implementation specific. B + 2 TEXT Datagram text. Passed through to Port layer. 0 - 4089 B + 1 + TEXT_LEN bytes in length. For DG, BODY_LEN = TEXT_LEN + 2. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 33 8.6 Messages The Message mechanism provides highly-reliable delivery of single packets using the virtual circuits. Messages are variable length, ranging from 0 to 4089 bytes in textual length. The maximum size message that may be exchanged between two nodes is determined by prior agreement in a higher level protocol. However, any nodes capable of receiving messages must be able to | receive messages of at least 58 bytes of textual length. Nodes are not required to support the capability of sending or receiving messages. In fact, a node may elect to support the capability of only sending or receiving messages, but not both. The BODY format of MSG is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ |PF | MBZ |NS |MBZ| :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | | | TEXT | | | | | :B + 1 +-------------------------------+ + TEXT_LEN Where: Byte Bits Field Description B <7:0> OPC OPC = 2 for MSG. B + 1 <7>= PF Packing format. Implies FLAGS<7> type of data packing to be used by certain types of ports. Value is implementation specific. B + 1 <1>= NS Sending Sequence Number. FLAGS<1> Current value of NS for destination node circuit. B + 2 TEXT Message text. Passed through to Port layer. B + 1 + TEXT_LEN For MSG, BODY_LEN = TEXT_LEN + 2. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 34 8.7 Data Transfers The data transfer mechanism provides for the transfer of large blocks of data not limited in size to a single packet. Specifically, a block of data is broken into multiple packets which are individually transferred by the data link layer. The state of the transfer need only be maintained in the end sending the data. The mechanism provides for both a read and a write operation. Both operations are confirmed upon completion in the initiating node. All packets involved in data transfers use virtual circuits to provide a high quality of service. Data transfers reference named buffers. Buffer names are 32-bits in length. Mapping of names to actual memory space is implementation specific. The CI packets reference only the name, length (in bytes) and offset (32-bits) into the buffer. The offset mapping is also implementation dependent. The name values, offsets and lengths must be determined prior to the transfer through higher level protocols. To read data, a node sends a special request packet which carries the transfer length, and names and offsets of the source and destination buffers. The data is returned in as many packets as necessary by the requested node. The last packet of the transfer is marked with a special flag. This is the confirmation (to the initiator) that the transfer was complete and successful. To write data, a node merely sends the packets of the transfer to the destination port. The last packet of the transfer is again marked with a special flag. Upon receipt of such a packet, if the transfer was successful, the receiving port must send back a special confirmation packet. Note that, again, the initiator of the transfer receives the confirmation. Any errors in completing the either read or write transfers should result in virtual circuit closure in the node where detected. The closed circuit prevents completion of the transfer. Only the node where the error was detected is aware of it. Higher level protocols must be used to inform the other involved node of the error (if necessary) and reinitialize the circuit. The data packets of a single transfer need not be sent consecutively. They may be interspersed with other packets between the same pair of controllers. They should, however, be sent in order of increasing offset. The length of data packets is discretely variable. Single data packets may contain 1 to 8 integral multiples of either 512 or 576 data bytes. All ports supporting data transfers must accept at least 512 and 576 bytes. Nodes may use sizes larger than 576 bytes only by prior agreement via higher level protocols. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 35 All the packets of a transfer except the last should be of the agreed-upon size. The last packet should carry the remainder and be less than or equal to the first packets in size. All nodes are not required to support either of the data mechanisms. However, if an operation is supported (reading or writing), all the functions required by the operation must be supported. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 36 8.7.1 Writing Data - Data is written from one system to another by sending data packets of the type Sent Data (SNTDAT). The last packet of the transfer has a special flag called the Last Packet (LP) flag set. Each packet carries a destination buffer name, offset and length which determine where the data is written. In the port receiving the packets, the receipt of a SNTDAT packet with the LP bit set indicates conclusion of the transfer. If no errors have occurred, a Confirmation (CNF) packet is sent back to the port which sent the data. If any errors occur in receiving the data or sending the CNF packet, the virtual circuit must be closed, thereby aborting the transfer. | | The operation of returning the CNF packet must occur at an | effective priority of three. Fig. 8-1 shows the data write operation. INITIATOR RESPONDER CI DATA LINK Data read SNTDAT buffer write (ok) from buffer ----------------------> ---> (error -- close into packets . virtual ckt) +-- . | V SNTDAT (LP = 1) +-- ----------------------> -+-> (error -- close | | virtual ckt) | | V buffer write (ok) (error -- close | virtual ckt) | CNF V (ok -- transfer <--------------------- generate CNF (prio 3) confirmed) | +--> (error -- close virtual ckt) Fig. 8-1: Data Write Operation CI SPECIFICATION--COMPANY CONFIDENTIAL Page 37 The BODY format of SNTDAT is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | MBZ | NS| LP| :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 | REC_NAME | | | :B + 13 +-------------------------------+ | | :B + 14 | REC_OFFSET | | | :B + 17 +-------------------------------+ | | :B + 18 | | | DATA | | | | | :B + 17 +-------------------------------+ + DAT_LEN Where: Byte Bits Field Description B <7:0> OPC OPC = 16 for SNTDAT. B + 1 <1>= NS Sequence number. FLAGS<1> B + 1 <0>= LP Last packet flag. Set FLAGS<0> in last packet of transfer. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Value in last packet will be the value of the XCT_ID in the corresponding CNF. B + 10 REC_NAME Receive buffer name. through Byte B + 10 is the least B + 13 significant byte. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 38 B + 14 REC_OFFSET Receive buffer offset. through Byte B + 14 is the least B + 17 significant byte of the byte offset of byte B + 18 of this packet in the buffer. B + 18 DATA Data. Byte B + 18 through is lowest offset byte. B + 17 + DAT_LEN For SNTDAT, BODY_LEN = DAT_LEN + 18. The BODY format of CNF is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | MBZ | NS|MBZ| :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ Where: Byte Bits Field Description B <7:0> OPC OPC = 3 for CNF. B + 1 <1>= NS Sequence number. Current FLAGS<1> value of NS for destination node. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Same value as SNTDAT packet with LP flag set that resulted in its generation. For CNF, BODY_LEN = 10. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 39 8.7.2 Reading Data - Data is read from a remote system by requesting that the data be returned. To request data, a Data Request (DATREQ) packet is sent to the node from which the data is to be read. The DATREQ specifies the names and offsets of buffers to supply and accept the data and the length of the transfer (in byte). (RETDAT) packets in much the same manner as data is sent in SNTDAT packets. The last RETDAT is marked by the LP flag being set. This is the confirmation of transfer success. | | The priority for the data return operation is specified by | the particular DATREQ opcode value. DATREQ0, DATREQ1, and DATREQ2 | correspond to effective priorities of 0, 1 and 2, respectively. | The data return operation consists of sending all the RETDAT | packets of the transfer. Any errors result in virtual circuit closure and therein abort of the transfer. The size of individual packets to be returned (except for the last packet) is specified in the request. The maximum allowable size must be determined by prior agreement between the involved nodes (using a higher level protocol). Fig. 8-2 shows the data write operation. INITIATOR RESPONDER CI DATA LINK Data requested DATREQ ----------------------> -+-> (error -- close | virtual ckt) | | (ok -- Return data) RETDAT V (ok -- buffer <--------------------- --+ Data read from write) . | buffer into packets (error -- close . | virtual ckt) . +-> (error -- close virtual ckt) RETDAT (LP = 1) (ok -- transfer <--------------------- confirmed) Fig. 8-2: Data Read Operation CI SPECIFICATION--COMPANY CONFIDENTIAL Page 40 The BODY format of DATREQ is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | P | M | MBZ |NS |MBZ| :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 | XCT_LEN | | | :B + 13 +-------------------------------+ | | :B + 14 | SND_NAME | | | :B + 17 +-------------------------------+ | | :B + 18 | SND_OFFSET | | | :B + 21 +-------------------------------+ | | :B + 22 | REC_NAME | | | :B + 25 +-------------------------------+ | | :B + 26 | REC_OFFSET | | | :B + 29 +-------------------------------+ Where: Byte Bits Field Description | B <7:0> OPC OPC = 8 for DATREQ0. | OPC = 9 for DATREQ1. | OPC = 10 for DATREQ2. B + 1 <7>= P Basic packet size for FLAGS<7> return data transfer. P = 0 for 512 bytes. P = 1 for 576 bytes. B + 1 <6:4>= M Packet size multiple. FLAGS<6:4> Packet data length = (M + 1) * basic size determined by P. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 41 B + 1 <1>= NS Sequence number. Current FLAGS<1> value of NS for destiation node. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Value to be used in generated RETDAT packets. B + 10 XCT_LEN Transfer length in bytes. through Byte B + 10 is the least B + 13 significant byte. B + 14 SND_NAME Sending buffer name. through Byte B + 14 is the least B + 17 significant byte. B + 18 SND_OFFSET Sending buffer offset. through Byte B + 18 is the least B + 21 byte of the byte offset of the first byte of the transfer to be sent. B + 22 REC_NAME Receive buffer name. through Byte B + 18 is the least B + 25 significant byte. B + 26 REC_OFFSET Receive buffer offset. through Byte B + 22 is the least B + 29 significant byte of the byte offset of the first byte of the transfer in in the receiving buffer. For DATREQ, BODY_LEN = 30. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 42 The BODY format of RETDAT is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | MBZ | NS| LP| :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 | REC_NAME | | | :B + 13 +-------------------------------+ | | :B + 14 | REC_OFFSET | | | :B + 17 +-------------------------------+ | | :B + 18 | | | DATA | | | | | :B + 17 +-------------------------------+ + DAT_LEN Where: Byte Bits Field Description B <7:0> OPC OPC = 17 for RETDAT. B + 1 <1>= NS Sequence number. FLAGS<1> B + 1 <0>= LP Last packet flag. Set FLAGS<0> in last packet of transfer. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Value is same as XCT_ID of DATREQ that resulted in generation of the RETDAT. B + 10 REC_NAME Receive buffer name. through Byte B + 10 is the least B + 13 significant byte. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 43 B + 14 REC_OFFSET Receive buffer offset. through Byte B + 14 is the least B + 17 significant byte of the byte offset of byte B + 18 of this packet in the buffer. B + 18 DATA Data. Byte B + 18 through is lowest offset byte. B + 17 + DAT_LEN For SNTDAT, BODY_LEN = DAT_LEN + 18. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 44 8.8 Configuration | All CI nodes must support the ability to supply configuration | information in response to requests whenever they are | acknowledging packets at the data link level. Though not strictly | required to be implemented in hardware, such capablility should be | provided without CI down-line loading. Configuration information is transmitted in the Identification (ID) packet. This packet contains port type, functionality, and state information. Systems that require knowledge of current cluster configuration can request an ID packet from a node by sending a Identification Request (IDREQ) packet. A node receiving a IDREQ packet is always required to return an ID packet. ID packets must be returned on the path on which the corresponding request was received. A transaction identifier in both the IDREQ and corresponding ID allow the requesting node to differentiate between received ID packets. | | The IDREQ is the preferred method of determining the | functional status of a particular path. The information on which | path the returned ID packet was received must be available at any | layer which intends to verify the status of the path. | Additionally, the ID packet carries bits specifying on which path | it was (alledgedly) transmitted. The correct path (cable) | configuration of a cluster can be completely verified using the | received path and sent path (SP) information. The ability of an interface to send ID packets is generally an indication that the interface is most likely capable of performing other maintenance operations. Some implementations or particular implementation states may not support certain optional fields of the ID packet. In this case, unsupported or inapplicable fields are to be returned as zero. All fields are required unless specifically stated otherwise. All unused fields must be zero to allow compatibility of future expansion with existing interfaces. All nodes are required to respond to received IDREQ packets by returning ID packets. All ports are not required to support the transmission of IDREQ packets, but if supported, the receipt of the returned ID packets must also be supported. The ID and IDREQ packets are delivered with a datagram-class service. They are, therefore, subject to discard. Higher level protocols using this mechanism must accomodate this fact. Discarded incoming ID packets cause the datagram discarded counter to be incremented. | | The operation of returning an ID in response to an IDREQ must | be performed at an effective priority of 4. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 45 It must be considered that IDREQ packets may be discarded at due to limited resources in the port. For details, see the section on Maintenance/Configuration datagram service. The BODY format of IDREQ is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | MBZ | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ Where: Byte Bits Field Description B <7:0> OPC OPC = 5 for IDREQ. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Value to be used in returned ID packet. For IDREQ, BODY_LEN = 10. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 46 The BODY format of ID is: 7 0 +-------------------------------+ | OPC | :B | +---+---+---+---+---+---+---+---+ | | MBZ | SP | MBZ | :B + 1 | +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 +---+ MAINT_ID | | D | | :B + 13 +---+---------------------------+ | | :B + 14 | CODE_REV | | | :B + 17 +-------------------------------+ | | :B + 18 | PORT_FCN | | | :B + 21 +-------------------------------+ | RST_PORT | :B + 22 +---+---+---+---+---+---+---+---+ | | PT_ST |MNT| :B + 23 | SYS_STATE +---+---+---+ | (Implementation-specific) | :B + 25 +-------------------------------+ | | :B + 26 | MBZ | | | :B + 49 +-------------------------------+ Where: Byte Bits Field Description B <7:0> OPC OPC = 11 for ID. | B + 1 <5:4>= SP Sent Path. Path on which | FLAGS<5:4> ID packet was transmitted. | SP = 1 for sent path 0. | SP = 2 for sent path 1. | SP = 0,3 reserved for | local port use. | B + 2 XCT_ID Transaction identifier. through Same value as XCT_ID of B + 9 corresponding IDREQ. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 47 B + 10 <7:0> MAINT_ID Port identification. through Implementation defined B + 13 identifier for use in down- loading, diagnosis and booting. B + 13 <7>= D Dual path. D = 1 for MAINT_ID<31> dual-path interfaces. D = 0 for single path interfaces. B + 14 CODE_REV Code revision. Value through is implementation B + 17 specific. B + 18 <7:0> PORT_FCN Port Functionality. Value through determined by which functions B + 21 the node supports. Values are defined in Implementation Functionality Appendix. B + 22 <7:0> SYS_STATE System State. through Miscellaneous port and B + 25 host state information. Certain fields are defined below. Remainder is implementation specific. | This field is optional. B + 22 <7:0>= RST_PORT Port which last reset SYS_STATE<7:0> the port. B + 23 <0>= MNT Maintenance. Denotes whether SYS_STATE<8> maintenance is enabled in current port state. Value of 1 indicates /Maintenance states. B + 23 <2:1>= PT_ST Port state. Current state SYS_STATE<10:9> of port (see section on Port State). PT_ST = 0 for Uninitialized. PT_ST = 1 for Disabled. PT_ST = 2 for Enabled. PT_ST = 3 reserved. B + 23 <7:3> Must be zero. through B + 53 <7:0> | | For ID, BODY_LEN = 50. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 48 8.9 Diagnostic And Booting (Maintenance) Operations The CI supports maintenance functions for initializing and testing interfaces and systems, reading and writing remote system memories and starting remote host computer execution. These functions are only performed by remote ports in certain specified states (see section on Controller and system States). The maintenance functions are optional and implemented as noted in the Implementation Functionality appendix. It is, however, required that an entire mechanism be supported. That is, all the components of any mechanism must be supported if the mechanism is to be provided. A maintenance control variable (RST_PORT) must be implemented by all resettable ports. RST_PORT limits the maintenance operations to be performed by only a single port. For further details, see description of individual operations. All maintenance packets are delivered with a datagram-class service. Higher level protocols using these mechanisms must accomodate this fact. 8.9.1 Loopback - | | A special datagram is supported to allow a node to verify | correct functioning of the CI to the hub coupler. The loopback | datagram (LB) appears to be equivalent to the general purpose | datagram except in opcode value. Normally, the source port number | is equal to the destination port number. However, any received LB | packet should be passed to the port driver following the | conventions for the general-purpose datagram (with the different | opcode). The implementation of this packet is system dependent. In fact, the standard datagram may serve this function if its implementation includes transmission and receipt on the media and received path information. This special opcode is intended for use by ports whose self-destined operations do not use the physical interconnect (internal loopback). Such ports may require special assistance from the host (such as transmit CRC calculation) for this packet. The received path information should be passed to any higher layer performing path verification. Such information, along with the knowledge of the (alleged) transmit path allows isolation of cabling faults. A node which must verify and isolate faults in cluster path connections must implement both this and the configuration request (IDREQ) functions. This is a datagram-class packet and is subject to discard if buffering is unavailable. The datagram discarded counter is incremented whenever a LB packet is discarded. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 49 The BODY format of LB is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ |PF | MBZ | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | | | TEXT | | | | | :B + 1 +-------------------------------+ + TEXT_LEN Where: Byte Bits Field Description B <7:0> OPC OPC = 13 for LB. B + 1 <7>= PF Packing format. Implies FLAGS<7> type of data packing to be used by certain types of ports. Value is implementation specific. B + 2 TEXT Datagram text. Passed through to Port layer. 0 - 4089 B + 1 + TEXT_LEN bytes in length. For LP, BODY_LEN = TEXT_LEN + 2. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 50 8.9.2 Resetting Remote Systems - A system may send a Reset (RST) packet to a remote system to reset its interface and host to a known state and to halt host computer execution. The RST packet may cause a reset of a host computer only with its interface in one of the three /Maintenance states (see section on Controller and system State). If the interface is not in this state but is receiving packets, the controller will pass the RST packet to the port driver informing of the attempted reset. | | A special maintenance state variable, RST_PORT, is used to | synchronize access to maintenance the maintenance functions of a | port. The RST_PORT variable of a port holds the number of the | last port to reset it. At initialization, RST_PORT is set to the | port's own number. Host control or port-detected host failure may | also set the RST_PORT to the port's own number. If the port is | successfully reset via the CI, RST_PORT is set to the number of | the port initiating the reset. | | A reset is only performed if the RST_PORT value of the | receiving port is equal to the node number of the sender. | Optionally, a special Force Reset (F) flag in the RST packet may | be set. If this flag is set, the RST_PORT field is ignored and | the reset unconditionally performed (if the interface is in a | /Maintenance state). Whenever a reset function is performed, the | RST_PORT field of the port being reset is set to the number of the | node that sent the RST. This offers a synchronization mechanism | wherein a non-forced (conditional) reset may be used to determine | if maintenance operations on a given node are already in progress | by another node. The Reset function (if executed) forces the interface to the Uninitialized/Maintenance state and halts the host computer. Therefore, systems implementing the Reset function must also implement the Start function (which starts execution). For further detail, see the next subsection). RST is a datagram-class packet and is therefore subject to duplication. It will, however, not be discarded if the conditions for resetting are true (since it does not directly require a returned packet). If the conditions for reset are not true, the reset if passed to the port driver. It is only discarded in this case if buffering is not available. If discarded, the datagram discarded counter will be incremented. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 51 The BODY format of RST is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | F | MBZ | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ Where: Byte Bits Field Description B <7:0> OPC OPC = 6 for RST. B + 1 <7> = F Force Reset. F = 1 FLAGS<7> for unconditional reset. F = 0 for reset only if RST_PORT = SRC. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. For RST, BODY_LEN = 10. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 52 8.9.3 Writing Remote System Memories - Remote system memories and interface control storage can be written over the CI by sending maintenance data, in much the same fashion as buffer data is sent. The data is sent in Sent Maintenance Data (SNTMDAT) packets. The receive buffer and offset are actually implementation dependent mapping information for physically accessing the desired data in the system under maintenance. | | The maintenance write operation is limited in length to a | single 512 or 576 byte packet (as is the read). The Last Packet | (LP) flag of the SNTMDAT packet must be set. If the RST_PORT and | state are correct, but the packet is too large or does not have | the LP bit set, it may be discarded. | | Upon receipt of a correct SNTMDAT packet (with LP bit set) | and successful conclusion of the write operation (to the best | knowledge of the port), the port returns a Maintenance Confirm | (MCNF) packet, indicating that the packet was received. Note that | the possible discard implies that multiple packet transfers | (intermediate packets with LP bit not set) are not reliably | tested. Any or all of the intermediate packets may fail or be | discarded and the transfer may still be confirmed (if the last | packet is successful). Data can only be written if the remote interface is in the Uninitialized/Maintenance state and if the RST_PORT value is equal to the number of the node attempting the write. The remote system must therefore have been reset by the node desiring to perform the write. | | For local port testing, self-addressed maintenance writes may | be performed in the Enabled/Maintenance state. Packets received | in the Enabled/Maintenance that are not self-addressed are to be | passed to the port driver as datagrams with unrecognized packet | status. In general, write operations should be verified with corresonding read operations (if supported). A Transaction ID (XCT_ID) is carried with the data and returned with the MCNF. | | The operation of returning the MCNF packet in response to a | SNTMDAT must be performed at an effective priority of 4. SNTMDAT and MCNF are both datagram-class packets. SNTMDAT packets with the LP = 1 may be discarded in the receiving interface. If the MCNF is never received, the data may or may not actually have been written. See section on Maintenance/Configuration Datagram Service for details. MCNF packets may be discarded if buffer space is unavailable in the receiving interface. If discard of the MCNF occurs, the datagram discarded counter is incremented. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 53 The SNTMDAT packet BODY format is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | MBZ |LP | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 | REC_NAME | | | :B + 13 +-------------------------------+ | | :B + 14 | REC_OFFSET | | | :B + 17 +-------------------------------+ | | :B + 18 | | | DATA | | | | | :B + 17 +-------------------------------+ + DAT_LEN Where: Byte Bits Field Description B <7:0> OPC OPC = 18 for SNTMDAT. B + 1 <0>= LP Last packet flag. Set FLAGS<0> in last packet of transfer. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Value is same to be used in the generated MCNF. B + 10 REC_NAME Mapping information through specifying memory area to B + 13 be written. Implementation dependent. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 54 B + 14 REC_OFFSET Mapping information through specifying offset in memory B + 17 to be written. Implementation dependent. B + 18 DATA Data. Byte B + 18 through is lowest address byte. B + 17 + DAT_LEN For SNTMDAT, BODY_LEN = DAT_LEN + 18. The BODY format of MCNF is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | MBZ | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ Where: Byte Bits Field Description | | B <7:0> OPC OPC = 4 for MCNF. | B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Value is same as that of SNTMDAT packet which resulted in its generation. For MCNF, BODY_LEN = 10. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 55 8.9.4 Reading Remote System Memories - Remote system memories and interface control storage can be read over the CI by requesting maintenance data, in much the same fashion as buffer data is requested. The data is requested with the Maintenance Data Request (MDATREQ) packet. The receiving buffer and offset specify the buffer into which the data is to be written. The send buffer and offset are actually implementation dependent mapping information for physically accessing the desired data in the system under maintenance. Data can only be read if the remote interface is in the Uninitialized/Maintenance state and if the RST_PORT value is equal to the number of the node attempting the read. The remote system must therefore have been reset by the node desiring to perform the read. | | For local port testing, self-addressed maintenance read | requests may be performed in the Enabled/Maintenance state. | Packets received in the Enabled/Maintenance that are not | self-addressed are to be passed to the port driver as datagrams | with unrecognized packet status. A maximum of one packet of data may be requested. Packets may only be either 512 or 576 bytes. That is, if P = 0, then 512 bytes is the maximum request length. If P = 1, then the maximum is 576. M must be 0. Interfaces that accept read requests must check the requested length. If the state and RST_PORT are correct, but the request is too large, the request should be discarded. The data is returned in Returned Maintenance Data (RETMDAT) packets. The Last Packet (LP) flag must be set since the maximum transfer length is one packet. The data should only be returned if the memory read is entirely successful. If any part of the operation is unsuccessful, the request should be discarded. The requesting node may detect this via timeout. A Transaction ID (XCT_ID) is carried in the request and returned with the data. | | The operation of returning the RETMDAT packet in response to | a MDATREQ must be performed at an effective priority of 4. MDATREQ and RETMDAT are both datagram-class packets. MDATREQ packets may be discarded in the receiving interface. See section on Maintenance/Configuration Datagram Service for details. RETMDAT packets may be discarded if buffer space is unavailable in the receiving interface. However, the data may or may not have actually been written in the named buffer area. If discard occurs, the datagram discarded counter is incremented. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 56 The MDATREQ packet BODY format is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | P | MBZ | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 | XCT_LEN | | | :B + 13 +-------------------------------+ | | :B + 14 | SND_NAME | | | :B + 17 +-------------------------------+ | | :B + 18 | SND_OFFSET | | | :B + 21 +-------------------------------+ | | :B + 22 | REC_NAME | | | :B + 25 +-------------------------------+ | | :B + 26 | REC_OFFSET | | | :B + 29 +-------------------------------+ Where: Byte Bits Field Description B <7:0> OPC OPC = 14 for MDATREQ. B + 1 <7>= P Basic packet size for FLAGS<7> return data transfer. P = 0 for 512 bytes. P = 1 for 576 bytes. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 57 B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. Value to be used in generated RETMDAT packets. B + 10 XCT_LEN Transfer length in bytes. through Byte B + 10 is the least B + 13 significant byte. Maximum value is 512 if P = 0 or 576 if P = 1. B + 14 SND_NAME Sending buffer name. through Byte B + 14 is the least B + 17 significant byte. Interpretation is implementation specific. B + 18 SND_OFFSET Sending buffer offset. through Byte B + 18 is the least B + 21 byte of the byte offset of the first byte of the transfer to be sent. Interpretation is implementation specific. B + 22 REC_NAME Receive buffer name. through Byte B + 18 is the least B + 25 significant byte. B + 26 REC_OFFSET Receive buffer offset. through Byte B + 22 is the least B + 29 significant byte of the byte offset of the first byte of the transfer in in the receiving buffer. For MDATREQ, BODY_LEN = 30. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 58 The RETMDAT packet BODY format is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ | MBZ | 1 | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 | REC_NAME | | | :B + 13 +-------------------------------+ | | :B + 14 | REC_OFFSET | | | :B + 17 +-------------------------------+ | | :B + 18 | | | DATA | | | | | :B + 17 +-------------------------------+ + DAT_LEN Where: Byte Bits Field Description B <7:0> OPC OPC = 19 for RETMDAT. B + 1 <7> = LP LP = 1 for last (only) FLAGS<7> packet. B + 2 XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. B + 10 REC_NAME Receive buffer name. through Byte B + 10 is the least B + 13 significant byte. B + 14 REC_OFFSET Receive buffer offset. through Byte B + 14 is the least B + 17 significant byte. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 59 B + 18 DATA Data. Byte B + 18 through is lowest address byte. B + 17 + DAT_LEN For RETMDAT, BODY_LEN = DAT_LEN + 18. 8.9.5 Starting Remote Systems - Remote systems which have been halted can be started by sending a Start (STRT) packet. The STRT packet should have no effect on systems that are already running. STRT packets are only accepted by interfaces in the Uninitialized/Maintenance state. The start function will only be performed if the RST_PORT field of the receiving port is equal to the source node number. That is, a node may only be started by the last port to reset it. If this requirement is not met, the packet is passed to the port driver as a datagram of invalid format. An optional start address may be specified. If the host computer does not implement variable start addresses, this field is ignored. For systems that do implement variable start addresses, a default must exist that is specifiable by setting the Default Start Address (DSA) bit. STRT is a datagram-class packet. It is therefore subject to duplication. It will not, however, be discarded if the state is correct, since it does not require a packet to be returned. Interfaces and nodes should not be adversely affected by the receipt of duplicate STRT packets. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 60 The BODY format of STRT is: 7 0 +-------------------------------+ | OPC | :B +---+---+---+---+---+---+---+---+ |DSA| MBZ | :B + 1 +---+---+---+---+---+---+---+---+ | | :B + 2 | XCT_ID | | | :B + 9 +-------------------------------+ | | :B + 10 | STRT_ADDR | | | :B + 13 +---+---+---+---+---+---+---+---+ Where: Byte Bits Field Description B <7:0> OPC OPC = 7 for STRT. B + 1 <7> = DSA Default start address. FLAGS<7> If set, system should use implementation specific default start address. Ignored if not implemented. B + 2 <7:0> XCT_ID Transaction identifier. through Byte B + 2 is the least B + 9 significant byte. B + 10 <7:0> STRT_ADDR Start address. through Byte B + 10 is the least B + 13 significant byte. For STRT, BODY_LEN = 14. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 61 8.10 Maintenance/Configuration Datagram Service It is likely that Maintenance and Configuration operations (with the exception of Loopback) will be performed by systems in states of less than full functionality, since, in fact, these operations are intended for use in down-line loading, diagnostics and bootstrapping. It is therefore probable that such packets will be processed primarily in the interface hardware in the end being "maintained". This implies that the amount of available state storage may be severely limited. | | The processing of certain maintenance/configuration packets, | namely, IDREQ, MDATREQ and SNTMDAT(LP) requires state to be | maintained until the proper packet (ID, MDAT, MCNF, respectively) | is returned. The number of concurrent operations is limited by | the amount of storage available. In the lowest limit, this may be | one operation. These packets are datagram-class; they may be | discarded upon receipt if insufficient state storage exists to | process them. Discard is therefore very likely under certain | conditions, such as consecutively (and rapidly) receiving several | packets of this type. | | For this reason, the maintenance and configuration operations | must have the highest priority (effective priority 4). This | should reduce the number of packets discarded due to storage | limitations. | | This discard must be considered by the node that is sending | the packets. It is also the reason for the single packet | constraint on maintenance read and write operations. In general, it is recommended that maintenance operations be performed one transfer at a time, with verification of completion between transfers. An example down-line load sequence might be: 1. Send a SNTMDAT (LP) packet. 2. Wait for the MCNF packet. 3. Retry previous two steps until successful or give up. 4. Send a MDATREQ for the same memory block. 5. Wait for RETMDAT(LP). 6. Retry previous two steps until successful or give up. 7. Compare data: If successful, do next block. Else retry or give up. For a more complete example, see appendix on remote system booting. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 62 8.11 Unrecognized Packets Packets that are illegal in any respect and are received in the Enabled or Enabled/Maintenance states must be passed to the port driver as datagrams with Unrecognized Packet status. They may, in fact, be discarded by the driver or any higher layer. However, this requirement is necessary to allow the visibility necessary for CI diagnosis. | | Illegal packets are those with OPC, FLAGS or PORT values that | are not correct or are received in states in which their | respective function is not performed. Virtual circuit packets | which are correct in these fields but are for circuits that are | closed or have incorrect sequence numbers are NOT considered | illegal packets and are simply discarded. | | In particular, MBZ fields MUST be checked wherever non-zero | values will cause incorrect system response. Note that, as | datagrams, unrecognized packets are subject to discard if | buffering is not available or if the DQI bit for the corresponding | port number is set. | | If buffering is available, all paramters and a minimum of the | first 58 bytes of the body should be passed intact to the port | driver to aid in determining the source of the error. | | | | 8.12 Port Performance Counters | | For purposes of monitoring cluster performance, a counter is | required to be implemented in all CI nodes. This counter must be | 32 bits in length and resetable and readable by higher layers such | that its value may be collected via higher level protocols. | | The counter counts the number of datagrams discarded due to | lack of buffering beyond the data link (after acknowledgement). | The counter should be incremented for unrecognized packet | datagrams, as well. If a packet is discarded due to the DQI bit | being set, the counter should not be incremented. | | This counter should be controllable to count either all such | events for all remote nodes (loopback should not be included) OR | for a single (specifiable) remote node. The counter should be set | to zero and assigned to count for all ports upon entering an | Enabled (/Maintenance) state. | | Counter overflow should result in the counter being locked at | a value of FFFFFFFF hex. Such a value must always be considered | invalid. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 63 9.0 CI INTERFACE AND SYSTEM STATE The port exists in one of six global states: Uninitialized, Disabled, Enabled, with and without maintenance functions enabled (/Maintenance). Essentially, the interface accepts and processes datagram, message and data packets in the Enabled state. In the Uninitialized state, the interface is essentially off, and not accepting packets. The Disabled state appears on the CI to be identical to the Uninitialized state. It is provided as an implementation-specific intermediate state for "graceful" or otherwise special state transitions. | | In the /Maintenance versions of the three states, special | rules apply for the acceptance or rejection of packets at the port | level. At the data link level, all packets are accepted. | Essentially, ports are enabled to process maintenance commands for | down-line loading and booting (subject to other command-specific | constraints) in the /Maintenance states. | | It is not required that all states be implemented by a port. | However, at least the Uninitialized/Maintenance state must be | implemented if any one of the down-line load or reset/start | functions is supported. The other /Maintenance states are | optional. | | Any state implementation must be within the bounds of the | description in this section. | | Any state may be entered from any other state. However, | certain events should consistently cause specific state | transitions. These transitions are specified in the description | of each state and shown in Figure 9-1 at the end of the section. | | The host system is modeled as existing in either a running or | a halted state. Where applicable, the host state is included in | the port state description. | | 1. Uninitialized | | In this state the interface does not send or acknowledge | packets at the data link level (resulting in NO_RSP). This is | the normal power-up state of most interfaces during which the | host may accomplish bootstrap and local load functions. | | This state can be entered by: | | 1. Power-up. | | 2. The occurrence of certain types of interface-detected | errors. This state might be entered since the higher | layers could not be trusted. | | 3. The occurrence of interface failures. This state would | most likely be entered since the interface cannot be | trusted. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 64 | 4. Hardware control by the host. | | This state can be exited by: | | 1. Hardware control by the host. | | 2. Host failure. The Uninitialized/Maintenance state should | be entered to allow down-line reboot. | | It is advisable that systems that are to be booted or | diagnosed via the CI in cases of failure have a mechanism | by which to enter a maintainable state if the host fails. | An example of such a mechanism could be a sanity timer | which must be "poked" by the host at regular intervals. A | failure of the host to "poke" the timer should result in | the port entering the Uninitialized/Maintenance state. | | The timer value in this state (as the start-up state) | is an implementation-specific value (possibly variable). | It is generally determined by the maximum power-up load | and boot time of the host system. | | | 2. Disabled | | In this state the interface does not send or acknowledge | packets at the data link level (resulting in NO_RSP). This is | normally only a transient state. On the CI, it is | indistinguishable from the Uninitialized state. | | This state can be entered by: | | 1. Hardware control by the host. | | 2. The occurrence of certain types of interface-detected | errors. This state might be entered since the higher | layers could not be trusted. | | This state can be exited by: | | 1. Hardware control by the host. | | 2. Host failure. The Uninitialized/Maintenance state should | be entered to allow down-line reboot. | | The mechanism for detection of host failure is | implementation-specific. However, it should be able to | detect failure within a time of 100 seconds. This allows | a uniform time specification for node reboot (as a failure | correction tool) in cases of failures. | | 3. The occurrence of certain types of interface-detected | errors. Another non-enabled state might be entered since | the higher layers could not be trusted. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 65 | 4. The occurrence of interface failures. The Unitialized | state would most likely be entered since the interface | cannot be trusted. | | | 3. Enabled | | In this state the port is processing packets (both | sending and receiving). ID packets are returned in response | to IDREQ packets. This is the normal operating state of most | interfaces not requiring external maintenance. STRT, RST, | SNTMDAT, MDATREQ and any illegal packets are passed to the | next level from the port as Unrecognized Packets. | | This state can be entered by: | | 1. Hardware control by the host. | | This state can be exited by: | | 1. Hardware control by the host. | | 2. The occurrence of certain types of interface-detected | errors. Some non-enabled state would most likely be | entered since the higher layers could not be trusted. | | 3. The occurrence of interface failures. The Unitialized | state would most likely be entered since the interface | cannot be trusted. | | 4. Host failure. The Uninitialized/Maintenance state should | be entered to allow down-line reboot. | | The mechanism for detection of host failure is | implementation-specific. However, it should be able to | detect failure within a time of 100 seconds. This allows | a uniform time specification for node reboot (as a failure | correction tool) in cases of failures. | | | 4. Uninitialized/Maintenance | | In this state the data link will acknowledge any packets | received. No packets will be sent other than those sent in | response to maintenance or configuration operations. Any | packets other than maintenance or IDREQ packets will be | discarded. ID packets will be sent in response to IDREQ | packets. Maintenance packets will be processed and responded | to as specified for the individual packets. Receipt of a RST | packet under appropriate conditions will result in the host | being reset as specified for the particular implementation. | The port state will not change although internal | initialization may occur, possibly resulting in the abort of | other maintenance operations in progress. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 66 | This could be the power-up state of interfaces of systems | requiring down-line load or bootstrap assistance over the CI. | The value of the RST_PORT field should be equal to the number | of the last node to reset this port. If this state was | entered as the result of a CI Reset, then RST_PORT should | equal the RST source node number. Otherwise, the value should | be equal to the nodes own number. | | This state can be entered by: | | 1. Power-up in systems requiring down-line loading. | | 2. Hardware control by the host. | | 3. Host failure. The Uninitialized/Maintenance state should | be entered to allow down-line reboot. | | The mechanism for detection of host failure is | implementation-specific. However, it should be able to | detect failure within a time of 100 seconds. This allows | a uniform time specification for node reboot (as a failure | correction tool) in cases of failures. | | 4. The occurrence of certain types of interface-detected | errors. This state might be entered since the higher | layers could not be trusted. | | 5. Receipt of a RST packet in any /Maintenance state with | correct RST_PORT and Force bit values. | | This state can be exited by: | | 1. Hardware control by the host. | | 2. The occurrence of certain types of interface-detected | errors. Another non-enabled state might be entered since | the higher layers could not be trusted. | | 3. The occurrence of interface failures. The Unitialized | state would most likely be entered since the interface | cannot be trusted. | | | 5. Disabled/Maintenance | | In this state the interface will acknowledge any packets | received. No packets will be sent other than those sent in | response to configuration operations. Any packets other than | RST or IDREQ packets will be discarded. ID packets will be | sent in response to IDREQ packets. Receipt of a RST packet | under appropriate conditions will result in the host being | reset and halted. The port will enter the | Uninitialized/Maintenance state. | CI SPECIFICATION--COMPANY CONFIDENTIAL Page 67 | This state can be entered by: | | 1. Hardware control by the host. | | 2. The occurrence of certain types of interface-detected | errors. This state might be entered since the higher | layers could not be trusted. | | This state can be exited by: | | 1. Host failure. The Uninitialized/Maintenance state should | be entered to allow down-line reboot. | | The mechanism for detection of host failure is | implementation-specific. However, it should be able to | detect failure within a time of 100 seconds. This allows | a uniform time specification for node reboot (as a failure | correction tool) in cases of failures. | | 2. Hardware control by the host. | | 3. The occurrence of certain types of interface-detected | errors. Some non-enabled state would most likely be | entered since the higher layers could not be trusted. | | 4. The occurrence of interface failures. The Unitialized | state would most likely be entered since the interface | cannot be trusted. | | | 6. Enabled/Maintenance | | In this state the port is processing packets (both | sending and receiving). ID packets are returned in response | to IDREQ packets. Maintenance packets other than RST that are | not self-addressed should be passed to the higher layers as | Unrecognized packets. Receipt of a RST packet under | apprpriate conditions will result in the host being reset and | halted. The port will enter the Uninitialized/Maintenance | state. | | This is the normal operating state of most interfaces | requiring external reset at any time. The value of the | RST_PORT field should be equal to the nodes own number in this | state. | | This state can be entered by: | | 1. Hardware control by the host. | | This state can be exited by: | | 1. Hardware control by the host. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 68 | 2. The occurrence of certain types of interface-detected | errors. | | 3. Host failure. The Uninitialized/Maintenance state should | be entered to allow down-line reboot. | | The mechanism for detection of host failure is | implementation-specific. However, it should be able to | detect failure within a time of 100 seconds. This allows | a uniform time specification for node reboot (as a failure | correction tool) in cases of failures. | | CI SPECIFICATION--COMPANY CONFIDENTIAL Page 69 | Power-up, Host Cntl, Power-up (remote ld), | Interface Failure, Host Control, | Intfc-detected Error Intfc-detected Error | | | | | | | V V | ************* Host Failure ************* | * *---------------------->* *-----+ | * UNINIT * * UNINIT/ * | CI RST | * * * MAINT *<----+ | * * +--------- >* *<--------+ | * * H | +------>* *<---+ C | | ************* o | | ************* | I | | s | | | | | t | | | R | C | | | | S | I | F | | | T | | a | | | / | R | i | | | H | S | Host Control, l | | Host Control, | s | T | Intfc-det Err. u | | Intfc-det Err. | t | / | | r | | | | | H | V e | | V | F | o | ************* | | ************* | a | s | * *-----------+ | * * | i | t | * DISABLED * | * DISABLED/ *----+ l | | * * | * MAINT * | F | * * | * * | a | * * | * * | i | ************* | ************* | l | | | u | | | r | | | e | | | | | | | | | | | | | Host | Host | | Control | Control | | | | | | | V | V | | ************* | ************* | | * * Host Failure | * * | | * ENABLED *---------------+ * ENABLED/ * | | * * * MAINT * | | * * * *---------+ | * * * * | ************* ************* | | | Fig. 9-1: CI Interface State Diagram CI SPECIFICATION--COMPANY CONFIDENTIAL Page 70 10.0 DATA LINK SPECIFICATION This section describes, in detail, the functionality of the data link layer of the CI. Although the implementation is not specified, it is by nature somewhat more restricted than the port layer, due to real-time constraints required for compatibility. With the exception of packet size, the data link layer basically has no optional features. The parameters are specified to be minimally restrictive. However, complete compatibility within specification is required for any interface attached to the CI. The specification of this layer assumes the same model as the port layer, that is, transmit and receive packet buffers and independent packet transmit and receive functions. In fact, implementations are not restricted to providing entirely independent transmit and receive functions. A minimal implementation is described to clarify this aspect. The two primary components of the data link layer are packetization and channel control. Packetization is the translation of port layer body information to and from packets for transmission and reception by the physical channel. Channel control includes such functions as arbitration for use of the channel and retransmission on error. The following subsections describe these functions in detail. 10.1 Packetization Since all data on the CI is passed in packet form for efficiency (low ratio of control information to actual data), one of the primary functions of the data link layer is packetization. This subsection specifies the packets passed (in both directions) across the physical channel/data link interface. The primary components of packetization are framing, addressing, and packet data integrity. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 71 The basic packet format of the data link layer is: +-------------------------------+ | | :0 | CHAR_SYNC | | | +-------------------------------+ | PACK_LEN/TYPE | :1 | | +-------------------------------+ | | :3 | DST/SRC | | | +-------------------------------+ | | :6 | | | BODY | | | | | +-------------------------------+ | | :BODY_LEN + 6 | CRC | | | :BODY_LEN + 9 +-------------------------------+ Fig. 10-1: Data Link Packet Format The various fields an their functions are described in the following subsections. 10.1.1 Framing - The packet is framed by two fields. The beginning is marked by a special starting character and the length is coded in one of subsequent fields. 10.1.1.1 Character Sync - The character sync denotes the first byte (byte 0) of the packet and has a value of 96(hex). The first combination of consecutive bits transmitted to or received from the physical channel that have this value are specified to denote the first byte of the packet. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 72 Any data link that uses shared hardware (cannot operate on both paths simultaneously) should implement a header timeout function (see Physical Channel section for definition of header). This should discontinue the receive process if a correct character synch is not detected within a certain time from the detection of carrier. This time should be a maximum of 32 byte times, or 3.66 microseconds. The minimum time is the minimum header detect time, which is dependent on the size of the header and carrier detection and byte decode times. Note that a header extension is permitted within bounds and may be enabled for special configurations. The maximum header length is 18 bytes. 10.1.1.2 Packet Length/Type - The two bytes of the packet that immediately follow the character sync are the packet length/type field. This field frames the end of packet by supplying the length of the packet immediately following (but not including) the character sync byte up to (but not including) the CRC bytes. The information packet length (PACK_LEN) is always BODY_LEN + 5. The type value identifies whether this packet is an information packet (carrying data) or an acknowledgement packet (transmitted as a result of successfully receiving an information packet, see section on Acknowledgement). The format of the Packet Length/Type field is: Packet Byte Number +---+---+---+---+---+---+---+---+ |TYP|ACK| | PACK_LEN<11:8>| :1 +---+---+---+---+---+---+---+---+ | PACK_LEN<7:0> | :2 +---+---+---+---+---+---+---+---+ Where: Byte Bits Field Description 1 <7> TYPE Packet type. TYPE = 0 for information packet. TYPE = 1 for acknow- ledgement packet. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 73 1 <6> ACK Acknowledgement type. Valid only for acknow- ledgement packets. MBZ for information packets. ACK = 1 for positive acknowledgement of packet receipt. ACK = 0 for negative acknowledgment (NAK) indicating that the packet buffers were full. Bits <5:0> MBZ for Acknowledgement packets. 1 <3:0> PACK_LEN Packet length in bytes. through Byte 1<3:0> = PACK_LEN<11:8>. 2 <7:0> Byte 2<7:0> = PACK_LEN<7:0>. Byte 1<6:4> MBZ for Information packets. Byte 2 does not exist for Acknowledgement packets. A value of 0 indicates a length of 4096 bytes (the maximum). 10.1.2 Addressing - Addressing is accomplished by the source and destination address fields. Addresses are 8 bits, with each node having an address value assigned that is unique within the particular CI. The range of valid addresses is dependent on the type of hub coupler. If a passive coupler is in use, the address values may range from 0 to 15. If an active exchange is in use, the value may range from 0 to 223. The source address is that of the node transmitting the packet. The destination address is that of the node intended to receive the packet. A node should only receive and buffer packets with its address in the destination field and should only transmit packets with its address in the source field. Loopback of packets is legitimate, that is, the source and destinations of a packet may be the same value. The destination field of the packet is replicated (in complement form) to allow duplication of address comparison logic. This permits the design of interfaces free from single component failures capable of prohibitting communication on the CI (on at least one path). CI SPECIFICATION--COMPANY CONFIDENTIAL Page 74 The format of the address field is: Packet Byte Number +-------------------------------+ | DST | :3 +-------------------------------+ | DST_CMPL | :4 +-------------------------------+ | SRC | :5 +-------------------------------+ Where: Bits Field Description 3<7:0> DST Destination node number. 4<7:0> DST_CMPL Destination complement. 1's complement of byte 3. 5<7:0> SRC Source node number. 10.1.3 Packet Integrity - The detection of packet errors due to bus noise or collision is provided by use of a 32-bit Cyclical Redundancy Check (CRC) character. The CRC character is the residue of the division (modulo 2) of the packet bits expressed as a polynomial by a special, defined polynomial. The CRC is generated as the packet is transmitted (or prior to transmission) and appended to the end of the packet. Upon receipt of the packet, the CRC is re-calculated during reception (necessitated by acknowledgement time constraints). If the residue of the calculation in the receiver agrees with the value appended to the packet, the packet is considered to be error free. This technique, along with validity tests of the destination address fields, should provide an undetected error rate of 10**-12. With a typical cluster load, this results in one undetected erroneous packet approximately every 10 years at this level. This assumes that collision and bus errors result in uniformly unpredictable bit values. Note that if many of the other values in the packet, particularly in the body, are not exactly correct, the packet will be discarded at the port layer. A similar rejection also occurs at higher levels. This reduces the probability of an undetected error in the cluster by many more orders of magnitude. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 75 The polynomial used to calculate the CRC is as follows (Autodin-II): 32 26 23 22 16 12 11 10 8 X + X + X + X + X + X + X + X + X + 7 5 4 2 X + X + X + X + X + 1 The transmitter and receiver generator/checker accumulators are preset to all ones prior to generating or checking the CRC character. The transmitter sends the complement of the remainder following the data bytes. In the receiver, an error free reception of the packet should result in a remainder in the accumulator of DEBB20E3(hex). The bytes on which the CRC is calculated include all the bytes after the Character sync (same as included in the packet count). The packet byte counter should be after the Character Sync is detected. It should be incremented for each successive byte received. For acknowledgement packets, CRC accumulator value should be tested when counter has a value of 4. For information packets, the value should be checked when the count is equal to the received packet length value. 10.2 Channel Control The channel control of the data link layer includes all functions employed to transfer a packet between the data link layers of two nodes using the physical channel. Such functions include arbitration for use of the media, selection of path (for dual-path interfaces) and acknowledgement and retransmission of packets. 10.2.1 Arbitration - The arbitration mechanism is implemented in a distributed fashion, with each node having equal priority and capability to arbitrate for the channel for any packet transmission. The method used is a slotted Carrier Sense Multiple Access (CSMA) type referred to as Dual-Count Round-Robin. Its characteristics are: 1. Fairness. Under saturation, each system wins arbitration at least one out of N times, where N is the number of nodes on the cluster. 2. Efficiency. Saturation efficiency is approximately 80% in a 16 node cluster and higher with fewer nodes. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 76 3. Reliable. The frequency of collision under saturation is zero. Collisions may occur under lighter loading but at a maximum rate of approximately 0.5%. The algorithm is based on a slot, or time interval which is the maximum time for any node at any physical location to sense that any other node at any other physical location is transmitting. A node desiring to transmit will examine the bus to determine if carrier is present (another node is transmitting). As soon as the bus becomes idle, the node will begin waiting a unique number of quiet slots based on its address N (ranging from 0 to 16). The number of slots waited may have two values: N + 1 or N + 17. This value is determined by previous transmissions of other nodes on the bus. The initial waiting period is N + 17 slots. If the waiting period elapses without carrier being detected on the bus (no other node attempting to transmit), the node will transmit its packet. If, however, carrier is sensed, then the number of slots waited before carrier detection is counted. The modulo 16 value of this number minus one is the number of the node that transmitted. The node will then set the waiting time value for the next arbitration attempt based on this information. If the number is less than its own, the new waiting value will be N + 1 (high priority). If the number is greater than its own, it the new waiting value will be N + 17 (low priority). Under saturation, this results in each node "waiting its turn", that is, waiting for all nodes with higher numbers to transmit before shifting to low priority. In cases of nodes waiting at equal priorities, the node with the lowest address wins. It should be noted that collisions can only occur when the bus has been idle and two nodes begin arbitrating at times different in slots by the same number as their addresses differ. In this case, they will both attempt to transmit at the same time. This results in corruption of the packet and failure of CRC comparison. A timeout period is specified for cases when the node is unable to win arbitration; that is, when the absence of carrier is never detected for the current arbitration count value. This timeout period is a minimum of 25 ms. with no specified maximum. If the timeout value is larger, it merely takes longer to detect the problem. Arbitration timeouts result in a NO_RSP status for the transmission on that path. For a structured description of the arbitration function, see the Data Link Structured Description subsection. The arbitration "slot" time is nominally 800 nanoseconds. This time, however, may be adjusted for special configurations in the future. If the coupler in use is an active exchange, the | arbitration may be disabled. If arbitration is disabled, | transmission may be begun whenever a packet becomes available. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 77 See Active Exchange appendix for details of operation. 10.2.2 Acknowledgement And Retransmission - The CI uses immediate acknowledgement at the data link level to verify the successful transmission and reception of packets. When a node successfully receives an information packet, it is required to immediately acknowledge the receipt of that packet. Acknowledgement is performed by the transmission of a special packet that carries an acknowledgement type code. Arbitration for transmission of the acknowledgement packet is not required. That is, the packet should be transmitted as soon as the carrier of the transmission is removed from the physical channel. The maximum time between carrier removal and transmission is 600 nanoseconds. The node that transmitted the information packet must attempt to receive the acknowledgement packet as soon as it is finished transmitting. If the acknowledgement is not received within an acknowledgement timeout period, the transmission is considered to have failed. This is termed No Response (NO_RSP). NO_RSP occurs when the intended node did not correctly receive the packet and therefore did not acknowledge it. The minimum arbitration timeout time is a function of the implementation. The carrier from the header of the acknowledgement should be on the cable at the packet sender no later than 800 nanoseconds after the trailer is finished. The maximum value should be limited to 3.66 microseconds for purposes of throughput, especially in interfaces with shared hardware. NO_RSP's may occur as the result of bus noise (CRC compare failure), simultaneous transmission of multiple nodes (collision, resulting in CRC compare failure), or inability of a node to receive the packet on the path on which it was transmitted, either due to malfunction of path or interface hardware, or unavailablity of shared receiver hardware in minimally implemented dual-path interfaces. There are two types of acknowledgements to successfully received packets. The first is positive Acknowledgement (ACK), which indicates that the reception was successful, that is, the transmitted packet is available to the port layer in the receiving node. The second is Negative Acknowledgement, which indicates that the packet was correctly received, but that the interface was unable to buffer it (packet is discarded). Although the actual buffering is implementation specific, the concept of the congested interface that is unable to process a packet applies to all implementations. The probability of congestion in interfaces is not restricted by specification, but should be minimized as it reduces the amount of bandwidth available on the media to all nodes. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 78 | If the transmission results in either NO_RSP or NAK, | retransmission must be attempted according to the following | algorithm: For NO_RSP, if fewer than 64 consecutive NO_RSP on | this packet have occurred, retransmission should be attempted. | For NAK, if fewer than 128 NAK's on this packet have occurred (not | necessarily consecutive), retransmission should be attempted. | | A "coin-flip" decision must be made when the packet is | available for retransmission. If the result is TRUE, | retransmission may be attempted. If FALSE, a delay time interval | should be waited and the decision repeated. The delay time value | should be a minimum of 16 slot times. The normally selected slot | time value is 800 nanoseconds which implies a minimum delay | interval of 12.8 microseconds. The maximum time is unlimited. | Throughput considerations usually limit the maximum. The delay | need not be consistent. This allows for software or firmware | controlled retry with non-constant service latencies (such as in a | polled system). | | The first decision has special properities. If, at the time | of the decision to retransmit, synchronism of the arbiter is | maintained after the transmission, that is, it remains | synchronized to the last deassertion of carrier on the bus, | transmission may occur when the arbitration is completed. | However, if synchronism was lost, a single delay interval should | be waited before the retransmission decison is made. | | This prevents consistent arbitration violation on succesful | retransmit decisions. If an interface always takes a constant | amount of time to determine that it must retransmit, collisions | with the transmissions of another node (with difference in numbers | times slots equalling the retransmit decision time) will | consistently occur. The random choice should be equal probability success/failure. Pseudo random implementations are acceptable with a minimum of 16 bits in the generator. This scheme is designed to break deadlocks. The selection of retry limits was calculated from simulation results to meet the following criteria: The mistaking of failure in a correctly functioning system (due to congestion) should occur no more than once per year with typical factors in heavily loaded clusters. | | A path that has failed in retransmission need not be retried | for the failing packet. Rather, it is appropriate to retry it at | whatever frequency is used for configuration update polling. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 79 10.2.3 Data Link Performance Counters - For purposes of monitoring cluster performance, a set of counters is required to be implemented in all CI nodes. These counters must be 32 bits in length and resetable and readable by higher layers such that their values may be collected via higher level protocols. The following counters must be implemented: 1. ACK_0 -- ACKS occurring on path 0. 2. ACK_1 -- ACKS occurring on path 1 (dual-path nodes only). 3. NAK_0 -- NAKS occurring on path 0. 4. NAK_1 -- NAKS occurring on path 1 (dual-path nodes only). 5. NO_RSP_0 -- NO_RSP's occurring on path 0. 6. NO_RSP_1 -- NO_RSP's occurring on path 1 (dual-path nodes only). | | These counters should be controllable to count either all | such events for all remote nodes (loopback should not be included) | OR for a single (specifiable) remote node. The counter should be | set to zero and assigned to count for all ports upon entering an | Enabled (/Maintenance) state. | | Counter overflow should result in the counter being locked at | a value of FFFFFFFF hex. Such a value must always be considered | invalid. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 80 10.3 Data Link Structured Specification This section specifies the data link transmitter and receiver in PASCAL. Although the transmitter and receiver are specified as procedures (for purposes of test compilation), they are actually processes which pass parameters under flag control to and from the Port layer. 10.3.1 Global Definitions - procedure DATALINK( {Transmit and receive packets} DST: PORT_ADRS; {Destination port number or port received from} SRC: PORT_ADRS; {Source port number -- this port} BODY: PACKET_BODY; {Packet body in a byte array} BODY_LEN: BODY_LEN_TYPE; {Packet body length in bytes} RCD_PACKET: BOOLEAN; {Packet received flag} TX_PACKET: BOOLEAN; {Packet available to transmit flag} PACKET_STATUS: STATUS); {Receive or transmit packet status} const CHAR_SYNC = %X96; {Character Sync value} | ACK_PACK = %XC0; {Packet Type High Codes} | NAK_PACK = %X80; INF_PACK = %X00; OK_CRC0 = %XDE; {CRC remainder} OK_CRC1 = %XBB; OK_CRC2 = %X20; OK_CRC3 = %XE3; type STATUS = (NORSP,NAK,ACK,OVERSIZE,OK); PORT_ADRS = 0..15; BYTE = 0..255; PACKET_BODY = array [0..4089] of BYTE; BODY_LEN_TYPE = 0..4091; TIMER = REAL; {Variable that decrements in time} CRC = ARRAY[0..3] of integer; {CRC accumulator on BUSDATA} OP_STATUS = (SUCCESS,FAIL); {General operation status} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 81 var CARRIER: BOOLEAN; {Carrier sense} BUS_DATA: BYTE; {Bytes passed to/from phys. channel} CRC_ACC: CRC; {CRC accumulator} PACK_TYPLEN_H: BYTE; {Packet Type/Length high order byte} PACK_LEN_L: BYTE; {Packet Type/Length low order byte} PACK_SRC: PORT_ADRS; {Received packet source} PACK_DST: PORT_ADRS; {Received packet destination} PACK_DST_CMPL: BYTE; {1's complement of destination port number} BUF_FULL: BOOLEAN; {True if no receive buffer available} function CMPL( N:INTEGER): INTEGER; begin {CMPL := 1's complement of parameter} end; {CMPL} function RAND: BOOLEAN; begin {return true 50% of time, otherwise false} end; {RAND} procedure STRTCHAN; begin {Transmit header bits on bus and open BUS_DATA for transmit data} end; {STRTCHAN} procedure ENDCHAN; begin {Transmit trailer bits on bus} end; {ENDCHAN} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 82 begin {start transmit and receive processes} {transmit is woken by assertion of TX_PACKET flag by port} {Returns status when it deasserts TX_PACKET} {receive is woken by CARRIER} {returns BODY and parameters when it asserts RCD_PACK} end; {DATALINK} 10.3.2 Transmitter - procedure SEND( {Send a packet} DST: PORT_ADRS; BODY_LEN: BODY_LEN_TYPE; BODY: PACKET_BODY; TX_PACKET: BOOLEAN; PACKET_STATUS: STATUS); const RTX_DELAY = 1.28E-05; {Retransmit delay value} MAX_NAK = 128; {Maximum MAK count} MAX_NORSP = 64; {Maximum consecutive NORSP count} var NAK_CTR: integer; {NAK counter} NORSP_CTR: integer; {NORSP counter} RETRY: BOOLEAN; {Retry flag} DELAY_TMR: TIMER; {Retransmit delay timer} I: INTEGER; {General purpose index} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 83 function ARBITRATE: OP_STATUS; {Returns after arbitration or timeout} const ARB_TIMEOUT = 25E-03; {Minimum arbitration timeout value} SLOT_TIME = 800E-09; {Nominal slot time} var CTR: INTEGER; {Slot counter} INIT_CTR: INTEGER; {Initial slot counter value} ARB_TMR: TIMER; {Arbitration timer} SLOT_TMR: TIMER; {Slot timer} begin ARB_TMR := ARB_TIMEOUT; INIT_CTR := SRC + 17; {Low priority value} CTR := INIT_CTR; while ((CARRIER) and (ARB_TMR <>0 )) do {Wait for carrier to go away} begin end; while (ARB_TMR <> 0) do begin SLOT_TMR := SLOT_TIME; while ((CTR <> 0)AND(NOT(CARRIER))) do begin if (SLOT_TMR = 0) then begin CTR := CTR - 1; SLOT_TMR := SLOT_TIME end end; if (CTR <> 0) then begin if (((CTR - 1) MOD 16) < SRC) then CTR := SRC + 1 {high priority} else if (((CTR - 1) MOD 16) > SRC) then CTR := SRC + 17 {low priority} else CTR := INIT_CTR {Didn't wait -- ACK} end; end; if (CTR = 0) then ARBITRATE := SUCCESS else ARBITRATE := FAIL {Timeout has occurred} end; {ARBITRATE} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 84 function RXACK( {Receive packet acknowledgement} SRC: PORT_ADRS; DST: PORT_ADRS): STATUS; const ACK_TIMEOUT = 800E-09; {Acknowledgement wait timeout} var ACK_TMR: TIMER; {Acknowledgement wait timer} I: INTEGER; {General purpose index} begin for I := 0 to 3 do CRC_ACC[I] := -1; {Initialize CRC accumulator} ACK_TMR := ACK_TIMEOUT; while (((not(CARRIER)) or (CARRIER and (BUS_DATA <> CHAR_SYNC))) and (ACK_TMR <> 0)) do begin end; if (ACK_TMR <> 0) then {Acknowldgement being received} begin PACK_TYPLEN_H := BUS_DATA; PACK_DST := BUS_DATA; PACK_DST_CMPL := BUS_DATA; PACK_SRC := BUS_DATA; for I := 0 to 3 do CRC_ACC[I] := BUS_DATA; if ((CRC_ACC[0] = OK_CRC0) and (CRC_ACC[1] = OK_CRC1) and (CRC_ACC[2] = OK_CRC2) and (CRC_ACC[3] = OK_CRC3) and (PACK_TYPLEN_H >= ACK_PACK) and (CMPL(PACK_DST_CMPL) = DST) and (PACK_DST = SRC) and (PACK_SRC = DST)) then begin | if (PACK_TYPLEN_H = NAKK_PACK) then | RXACK := NAK else RXACK := ACK end else RXACK := NORSP end else RXACK := NORSP end; {RXACK} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 85 begin while (not (TX_PACKET)) do begin {Wait for packet to transmit} end; NAK_CTR := MAX_NAK; NORSP_CTR := MAX_NORSP; RETRY := TRUE; while (RETRY) do begin if (ARBITRATE = SUCCESS) then begin STRTCHAN; for I := 0 to 3 do CRC_ACC[I] := -1; | BUS_DATA := (BODY_LEN + 5) MOD 4096; BUS_DATA := DST; BUS_DATA := CMPL(DST); BUS_DATA := SRC; for I := 0 to (BODY_LEN - 1) do BUS_DATA := BODY[I]; for I := 0 to 3 do BUS_DATA := CMPL(CRC_ACC[I]); ENDCHAN; PACKET_STATUS := RXACK(SRC,DST); if (PACKET_STATUS = NORSP) then begin NORSP_CTR := NORSP_CTR - 1; if (NORSP_CTR = 0) then RETRY := FALSE end else if (PACKET_STATUS = NAK) then begin NAK_CTR := NAK_CTR - 1; if (NAK_CTR = 0) then RETRY := FALSE else NORSP_CTR := MAX_NORSP end end else PACKET_STATUS := NORSP; while ((RETRY) and (RAND)) do begin DELAY_TMR := RTX_DELAY; while (DELAY_TMR <> 0) do begin end end end end; {SEND} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 86 10.3.3 Receiver - procedure RECEIVE( {Receive a packet} DST: PORT_ADRS; BODY_LEN: BODY_LEN_TYPE; BODY: PACKET_BODY; BUF_FULL: BOOLEAN; RCD_PACKET: BOOLEAN; PACKET_STATUS: STATUS); | const HDR_TIMEOUT = 4.8E-06; {Minimum time for 40-bit header} MAX_BODY_LEN = 4091; {Implementation specific maximum} var HDR_TMR: TIMER; {Header timer} JUNK: BYTE; {Byte bucket} I: INTEGER; {General purpose index} procedure TXACK( {Transmit an acknowledgement packet} BUF_FULL: BOOLEAN; SRC: PORT_ADRS; DST: PORT_ADRS); var I: INTEGER; {General purpose index} begin STRTCHAN; for I := 0 to 3 do CRC_ACC[I] := -1; if (BUF_FULL) then BUS_DATA := NAK_PACK else BUS_DATA := ACK_PACK; BUS_DATA := DST; BUS_DATA := CMPL(DST); BUS_DATA := SRC; for I := 0 to 3 do BUS_DATA := CMPL(CRC_ACC[I]); ENDCHAN; end; {TXACK} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 87 begin while (not (CARRIER)) do begin {Wait for carrier} end; RCD_PACKET := FALSE; for I := 0 to 3 do CRC_ACC[I] := -1; HDR_TMR := HDR_TIMEOUT; while ((BUS_DATA <> CHAR_SYNC) and (HDR_TMR <> 0)) do begin end; if (HDR_TMR <> 0) then begin PACK_TYPLEN_H := BUS_DATA; if (PACK_TYPLEN_H <= INF_PACK + 15) then begin PACK_LEN_L := BUS_DATA; PACK_DST := BUS_DATA; PACK_DST_CMPL := BUS_DATA; SRC := BUS_DATA; | BODY_LEN := ((4096 + PACK_LEN_L + | (PACK_TYPLEN_H * 256)) MOD 4096) - 5; if BODY_LEN > MAX_BODY_LEN then BODY_LEN := MAX_BODY_LEN; for I := 0 to (BODY_LEN - 1) do BODY[I] := BUS_DATA; for I := 0 to (MAX_BODY_LEN - BODY_LEN) do JUNK := BUS_DATA; for I := 0 to 3 do CRC_ACC[I] := BUS_DATA; if ((CRC_ACC[0] = OK_CRC0) and (CRC_ACC[1] = OK_CRC1) and (CRC_ACC[2] = OK_CRC2) and (CRC_ACC[3] = OK_CRC3) and (PACK_DST = SRC) and (PACK_DST_CMPL = CMPL(SRC))) then begin TXACK(BUF_FULL,DST,SRC); RCD_PACKET := TRUE; if (BODY_LEN > MAX_BODY_LEN) then PACKET_STATUS := OVERSIZE else PACKET_STATUS := OK end end end end; {RECEIVE} CI SPECIFICATION--COMPANY CONFIDENTIAL Page 88 11.0 PHYSICAL CHANNEL SPECIFICATION | | | The physical channel performs the functions of transmitting | and receiving the bits of packet bytes specified in the data link | layer and detecting the presence of transmissions on the channel. | The channel encompasses the bit encoding and decoding, drivers and | receivers that interface to the media, and the media itself, that | is, the cables and couplers. | | | | 11.1 Channel Character | | The physical channel design is largely determined by two | factors: | | 1. Nodes on the CI needed to have isolated system grounds or | earth references. The signal paths have to be AC coupled with | isolation at primary power frequencies. A large amount of | hardware needed per isolated signal line so bit-serial | transmission techniques are used. | | 2. The bus is to be highly reliable and reconfigurable on line. | The media connecting one node must not be critical to | operation of other nodes. The central hub or star-coupled | approach provides each node with an independant hook-up to the | cluster. A daisy chain media configuration between nodes | doesn't allow this goal. | | | | | 11.2 Isolation Scheme | | Each node is isolated from the media that connects it to the | hub. The shields of the individual node cables are tied together | by the coupler. In a dual path system there may be two | independant couplers which may or may not be isolated from each | other. In any case, the coupler and all the external cables are | "floating" with respect to nodes. The actual connection between | the node and its bus cables is made as follows (a single path | system is discribed): | | 1. The transmit and received data are transfomer coupled into the | end of the cable at the node. The signal path is complete and | requires no further referencing of the cable. | | 2. The transmit and receive cable shields are RF decoupled to the | I/O panel of the node with a 4 to 8 nano-farad capacitor | (value per cable). The capacitor is a filter with two | purposes: 1) keep the internal noise inside the node cabinet, | away from the FCC; 2) keep external noise outside the node | cabinet to minimize susceptabilitly to RF interference or CI SPECIFICATION--COMPANY CONFIDENTIAL Page 89 | static discharge. The capacitive filter is effective above 30 | Mhz and transparrent at lower frequencies. | | 3. The cable shields are connected by a 300 ohm resistor to node | logic reference to dampen common mode resonances of the | internal cables. | | | | | 11.3 Bit Transmission And Reception | | 11.3.1 Encoding And Decoding - | | Bits are encoded for transmission using Manchester phase | encoding. This encoding scheme is used for the following reasons: | | 1. Guaranteed Transitions -- Encoded bits have at least one | transition per bit cell, thereby allowing AC (transformer) | coupling. AC coupling is needed to provide high intercabinet | isolation at power-line frequencies. | | 2. Simplicity of Implementation -- Manchester coding can be | implemented in a "reasonable" number of components. | | 3. Clock Encoding -- Encoding the clock in the data stream | eliminates the problem of clock skew associated with | separately cabled clock and data signals. | | | The nature of bit transmission and the channel are as | follows: | | 1. Serial data transmission with transmit and received data on | seperate cables. | | 2. Raw data rate: 70 Mbit/second. | | 3. Syncronous manchester code (baseband bi-pahase) with data and | clock on same cable. | | 4. Media bandwidth, 10 to 100 Mhz. | | | The manchester code scheme is the same as "X-or'ing" the | clock and the data. A signal transition occurs in the center of | each bit cell. This transition is negative for a zero and | positive for a one, as shown in the figure below. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 90 | | | |<---BIT CELL-->| | | | | | | | | | 1 | 1 | 0 | | | +-------+ +---------------+ | | | | | | | | | | | --------+ +-------+ +------- | | | BIT CELL = 14.286 nanoseconds | | | Fig. 11-1: MANCHESTER ENCODING | | | | 11.3.2 Packet Format - | | The packet must be preceeded by preamble bits and followed by | trailer bits as specified below. The preamble of specified bit | values is needed to start and synchronize the decoder. A string | of trailer bytes follows is appended to ensure the correct | decoding of all packet bits and insure smooth release of the bus. | The format of the bit stream transmitted on the cable is: | | | First Bit Packet Last Bit | Transmitted Byte 0 Transmitted | | | | | V V V | +-------+-----------------------+-------+ | | BIT | PACKET |TRAILER| | | SYNC | BYTES | | | +-------+-----------------------+-------+ | | | | | Fig. 11-2: Bit Format of Packet Envelope | | | | | 11.3.3 Bit Synchronization - | | In order for decoder to synchronize to the center transition | of the bit cell, each packet must start with a preamble consisting | of 32 bits of Bit Synchronization. This Bit Sync consists of four | bytes of value 55 (alternating 1's and 0's). This pattern results | in the fewest possible transitions (one per bit) with every | transition corresponding to the center of a bit cell of the value | appropriate to the direction of transition. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 91 | 11.3.4 Trailer - | | The trailer is needed to ensure that all packet bits can be | successfully decoded, that all bytes can be deserialized (framed) | , and that the bus will become quiet quickly after the end of the | packet. To allow for decoding, a minimum of a 32 bits is appended | to the end of a packet. The maximum number of bits that may be | added is | <***** value needed> | | A longer trailer will interfere with the minimum time between | packets on the bus. The trailer must be composed of all 0 bits to | provide for carrier detect being dropped quickly after the end of | the packet. (All 1's would be suitable as well.) | | | | 11.4 Carrier Detection | | \ TO BE ADDED \ | | | | 11.5 Media -- Cables And Couplers | | \ TO BE ADDED \ | | | | 11.6 Input/Output Signal Level Specification | | 11.6.1 Driver Power Output Specification - | | The power delivered by the driver is specified at the CI | bulkhead connector for convience of measurement. This value | includes assumed properties for the internal cables and | connectors. If measured at the backplane the minimum value would | be 1 db greater and the maximum value would be the same. | | The driver power output at the CI bulkhead connector is: | | 1. Minimum: 13.0 dbm | | 2. Maximum: 16.5 dbm | | The value is specified as "db above a milliwatt" in 50 ohms | characteristic impedance and is a RMS power measurement. In 50 | ohms the dbm value corosponds to roughly 3 volts peak to peak for | a periodic signal (ie, continous all 1's data). | | The driver power is best measured with a true power meter. | Measuring the voltage and converting leads to errors if the output | wave shape is not purely sinusoidal. Under some conditions the | driver may approximate a sine wave although it isn't specified | this way. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 92 | 11.6.2 Receiver Input Power Specification - | | The receiver input power level is specified at the bulkhead | for convience. The same notes given for the driver power spec. | apply here. | | The receiver power is as follows: | | 1. Minimum Power: -19.0 dbm | | 2. Maximum Power: -4. dbm | | Note that "dbm" is explained in the driver power specification | section, above. | | The maximum power level specification is provided to prevent | damage to the receiver input and to allow carrier detect circuit | to detect the end of packet quickly after it occurs. | | | | 11.7 Input/Output Signal Timing Specification | | The data below specifies the timing of waveforms at the | output of the CI driver and the input to the CI receiver. The | difference between these waveforms is caused by the external | cables, couplers, and environmental noise. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 93 | 11.7.1 Driver Output Signal Timing - | | The specification of the driven signal includes the bit | frequency, jitter and the signal rise/fall times. | | The frequency of bits is relatively precise. The tolerance | in the frequecy is +/- .006% and is the same as the transmit clock | frequency tolerance. | | The jitter is relatively low at this point in the bus. The | jitter is attributed to the inter-bit transition below but it is | really equally divided up between all edges. The jitter is | non-accumulating (the frequency is constant) so the jitter may be | attributed to one edge or the other and viewed this way with an | osciloscope. | | The rise/fall time of the driver output signal is nominally | 2.0 nanoseconds. The rise time should be the same as the fall | time to within .30 nanoseconds. | | | |<---BIT CELL-->| | | | | | | | | | 1 | 1 | 1 | | | +------+++ +------+++ +-------- | | ||| | ||| | | | ||| | ||| | | --------+ +++------+ +++------+ | | | | | | |<---- T ------>| ->| |<- J | | J = MAX JITTER = +/- 1.0 nanoseconds | | | BIT FREQ= 70.000 +/- .004 MHZ. | | | T = 1/Freq = 14.286 nanoseconds | +/- .001 | | Fig. 11-3: TIMING SPEC AT DRIVER OUTPUT | | | | 11.7.2 Receiver Input Signal Timing - | | The frequency of bits is unchanged and jitter in the edges is | introduced as shown below. The jitter is attributed to the | inter-bit transition below but it is really equally divided up | between all edges. The jitter is non-accumulating (the frequency | is constant) so the jitter may be attributed to one edge or the | other and viewed this way with an osciloscope. The rise/fall time | of the signal input to the receiver is less than or equal 3.5 | nanoseconds. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 94 | | | |<---BIT CELL-->| | | | | | | | | | 1 | 1 | 1 | | | +------+++ +------+++ +-------- | | ||| | ||| | | | ||| | ||| | | --------+ +++------+ +++------+ | | | | | | |<---- T ------>| ->| |<- J | | J = MAX JITTER = +/- 2.8 nanoseconds | | | BIT FREQ= 70.000 +/- .004 MHZ. | | | T = 1/Freq = 14.286 nanoseconds | +/- .001 | | Fig. 11-4: TIMING SPEC AT RCVR INPUT CI SPECIFICATION--COMPANY CONFIDENTIAL Page 95 12.0 CONFIGURATIONS This section discusses the recommended and allowable physical cluster configurations for CI installation and maintenance. The sections on installation rules give the limits on cable lengths, limits on coupler size, general power and grounding needs for site preparation, and proper node addressing. Next, recommended procedures for maintenance operations involving the signal cables are given. Finally, the limits to which a cluster may be mis-configured and still function, either for maintenance or by mistake, are given as "configuration exceptions". 12.1 Physical Topologies The physical topology of the bus is that of a wheel with computers or other nodes around the circumference and a central coupler in the middle. The node cables radiate outward from the middle, each node is connected "point-to-point" with the coupler. If a dual path CI is involved there will be two separate sets of node cables and couplers. All CI clusters must have a central coupler, even in the two node case. The coupler can be designed to allow any number of nodes from two to the maximum addressable (given below) and in a given coupler all the ports do not need to be used. The most popular coupler sizes are expected to be 2, 8, and 16. For these three sizes the coupler can be a passive device without any power source. The two node coupler is very small and is envisioned as being a lump in the cable between the two nodes. For larger clusters the coupler needs to be housed in a cabinet to help manage the cables in the center. For 8 and 16 node dual path clusters the coupler will usually need to have it's own cabinet. 12.2 Installation Rules The node configuration rules for cluster installation are given below in two categories: those for the signal cables and couplers, and rules covering the "grounding" and cluster power distribution. These rules represent the signal specifications given in the CI Physical Channel section. 12.2.1 Installation Rules For Signal Cables And Couplers - 1. Minimum Cable Length from coupler to node: none | | 2. Maximum Cable Length from coupler to node: 45 meters. 3. Maximum Cable Length inside the node: 3 meters. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 96 4. There must be one coupler per CI path. 5. Unused node ports on a coupler must be terminated. 6. Minimum number of nodes: none. 7. Maximum number of nodes depends on coupler size: For a passive coupler: 16 nodes (32 cables per path). For an active coupler: 224 nodes (548 cables per path) | | 8. Unused cables must be disconnected at the coupler. Cables | must be connected at the node end if connected at the coupler. | The cable must be terminated for proper coupler operation and | the terminator is in the node. Also, to preserve bus | isolation, the end of the cable at a node must not be allowed | to contact the node chassis. Disconnection at the coupler | avoids the problem of cables shorting the coupler to a node. 12.2.2 Power, Grounding And Site Preparation For Installation - CI related cluster power and grounding conforms with DEC STD 186 rules for a "conditioned" signal path between systems. The DC loop in the cluster formed by the power and signal cables is broken in the signal path. The CI bus is "isolated" at low frequencies so no special consideration to power or grounding is needed for the CI signal integrity. Each subsystem or node on the CI may be installed as if they were entirely independent systems. | Note, the installation must comply with all local codes and | National Electrical Codes for safety. 12.2.2.1 Grounding: 1. The installation of a CI interconnection doesn't require special grounding straps, bonding of cabinets, or other special care. 2. If the CI is replacing another bus between two nodes remove ground straps installed for the previous bus if possible. Straps should always be removed when the systems (nodes) are only interconnected by CI cables after the bus change. Straps should not be removed if any non-isolated busses remain between the systems. Removing unneeded straps will improve the reliability of the individual systems. 3. All systems used with the CI must have the standard RF choke in series between the unit chassis and the earth reference wire of the power cord. This is a standard item in all DEC power controllers. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 97 4. The coupler chassis or cabinet may be connected to any single DEC chassis with primary power for safety referencing but this is unadvisable. It may not be connected to more than one such reference. This connection will never improve operation of the bus. 5. The coupler chassis or cabinet must never be "grounded" by connection to building structural metal, water pipes or other random places. A DEC chassis can be used as it provides a necessary RF choke (usually in the power controller). 12.2.2.2 Primary Power Distribution: - Comply with NEC, local safety codes, and the node (computer system) site spec's for power distribution on an individual basis. Refer to DEC STD 186 if there is any doubt as to what is good. The basic idea is that primary power is distributed to a normal system such that ALL power and earth references might be disconnected at one point. When this connection is made the system is safety earth referenced and has power. If this connection is not made there is no earth reference and no power to the system. This point is typically the "breaker box" for large systems (although the safety earth wire never has a circuit breaker in it). All systems used with the CI must have the standard RF choke in series between the unit chassis and the earth reference wire of the power cord. This is a standard item in all DEC power controllers. The actual conditions for proper CI operation are as follows. The most important condition is that all node chassis be within 6 | Vrms or 18 Vpp of each other at frequencies below 10Khz. This | will almost always be the case when primary power and saftey | wiring meet safety codes and site spec's for the individual CI | nodes. (Reference Dec Std 102.7) If the site doesn't meet this | requirement the CI may be prone to bus data errors. A second condition is that the CI cable shields never connect the chassis reference of one node to another node or to building metal. 12.3 Addressing Addresses should be assigned in a cluster consecutively from 0. This maximizes the efficiency of the arbitration algorithm. Addresses are restricted to values of 0:15 in "passively" coupled clusters and 0:223 in clusters coupled with "active" exchanges. The addresses from 223:255 are reserved for future use. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 98 12.4 Installation Of A New Node Onto An Operational Cluster -/to be added/- 12.5 Bus Maintenance And Temporary Configurations. The maintenance procedures given below allow one node and it's associated hardware to be removed from the cluster while the rest of the cluster is using the same path. The configuration rule given in the Installation Configuration section (above) concerning unused ports on the coupler or cables may be violated as given below for the purpose of maintenance. These allowed exceptions to the rules provide for on-line bus repair and an "available" bus. When the bus is configured as per the installation rules no single point of failure exists, providing a "reliable" bus. The configurations given below should not be used as a routine matter of convenience as more than one violation may lead to widespread cluster problems. 12.5.1 On-Line Maintenance Procedures - The following procedures are to be used to repair one path or node while the rest of the cluster is operational. 1. Always remove power from the CI logic backplane before removing CI parts inside the node cabinet. 2. If a cable is to be disconnected at a node it should be disconnected at the coupler first to minimize the effect on the cluster and the chance of the cable shield "short circuiting" between two ground references. 3. Disconnect cables at the coupler first, do one path at a time, and terminate the coupler before breaking the second path. Do not leave unterminated ports on the coupler. Do not leave cables attached at the coupler and disconnected at a node, terminated or not. Cables don't need to be removed from the coupler cabinet while disconnected. 4. Disconnect cables for both paths at the coupler before removing the module containing the Physical Channel circuits (ILI module) for repair. 5. There is never a need to terminate the CI connectors on the node bulkhead panel, only the unused coupler ports need to be terminated. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 99 | 6. To reinstall the node, use the reverse of the above procedure. | First, place the module containing the physical channel | circuits (ILI) in the backplane and connect the internal | cables. Second, install the external cables at the node end. | Finally, install the external cables at the coupler(s). 12.5.2 Use Of Extender Cards - A special version extender card must be used with the ILI or Physical Channel module if CI bus functionality is needed. The special extender card has coaxial cable connecting the CI bus signals on the card. If the bus ports are disabled and not used while the card is on the extender a regular CI extender card may | be used. Check the specification for the particular CI | implementation involved for extender compatability. 12.6 Allowed Configuration Exceptions The CI will always work in many configurations that do not meet the Installation Rules given above. These configurations, given below, are intended to be used for maintenance or by accident only. If a bus is configured as recommended above it is immune to any single media problem while if configured as below a failure may disrupt communication. | | The actual allowed configurations are summed up as follows: | there can be one pair of unused cables or unterminated coupler | ports on an otherwise correctly configured path. This situation | will not disrupt the operation of properly connected nodes on same | path in any way. If a path is correctly configured to begin with, under no circumstances does breaking the path(s) between the coupler and a single node ever disrupt other nodes. Also, anything which happens to only one path cannot disrupt the second path. It is important to insure multiple ground or earth references do not result from disconnected cables in casual contact with metal things. The following are allowed while the cluster operates: 1. Removing and restoring power for an individual node. 2. One unterminated transmit/receive pair of cables or coupler ports per path. The path to a node may be broken in any way without affecting the rest of the nodes using that path. If only one path for a node is disconnected the second path will work. 3. Removal of the Physical Channel logic module while the node is fully connected. This leaves the cables of both paths unterminated but will only disrupt the involved node. CI SPECIFICATION--COMPANY CONFIDENTIAL Page 100 4. Disconnection of any or all cables at any point between the coupler and the backplane in a node. 5. A damaged cable with an internal short or open circuit. 12.7 Mechanical Considerations --- to be added --- CI SPECIFICATION--COMPANY CONFIDENTIAL Page 101 13.0 DIAGNOSTIC AND MAINTAINABILITY REQUIREMENTS ALL CI nodes must comply with the CI Diagnostic Strategy. Reference applicable documents for further information. In addition, all nodes must be compatible with both the CI Network Exerciser and CI Node Tester. For further details, see specifications listed in Reference Documents section. APPENDIX A CI OPCODE SUMMARY CI PACKET DEC HEX OCTAL BINARY DG 1 1 1 0000 0001 MSG 2 2 2 0000 0010 SNTDAT 16 10 20 0001 0000 CNF 3 3 3 0000 0011 | | DATREQ0 8 8 10 0000 1000 | | DATREQ1 9 9 11 0000 1001 | | DATREQ2 10 A 12 0000 1010 | RETDAT 17 11 21 0001 0001 IDREQ 5 5 5 0000 0101 RST 6 6 6 0000 0110 SNTMDAT 18 12 22 0001 0010 | | MDATREQ 14 E 16 0000 1110 | STRT 7 7 7 0000 0111 ID 11 B 13 0000 1011 RETMDAT 19 13 23 0001 0011 | | MCNF 4 4 4 0000 0100 | LB 13 D 15 0000 1101 NOTE: Opcode values with bit <7> = 1 are reserved for local port use. APPENDIX B EXAMPLE OF NODE BOOTING VIA CI The following sections show examples of use of the configuration and maintenance commands. The examples show bootstrapping. Remote system diagnosis is assumed to be similar with respect to loading and starting diagnostic programs. B.0.1 Locally-Loaded System Action Local Port CI Packet Remote Action State Port power-up. Uninitialized : Host executes bootstrap. : Host initializes port Disabled and port driver. : Host enables port and begins on-line Enabled operations. : Host sends IDREQ's to all CI nodes on both paths. IDREQ -> Request is received. : ID packet received. <- ID ID packet returned. Host updates configuration tables. : Higher level protocol initiated to establish communication. EXAMPLE OF NODE BOOTING VIA CI Page B-2 B.0.2 Remotely-Loaded System Action Local Port CI Packet Remote Action State | Port power-up. Uninitialized | RST_PORT = port no. /Maintenance | Host sends IDREQ's | to all CI nodes | <- IDREQ on both paths in | Request is received. configuration poll. | : | ID packet returned. ID -> ID packet received. | Host updates | configuration tables, | recognizes state | and type requiring | down-line load. | : | RST received and Uninitialized <- RST Host sends RST (F=0) | performed. Host /Maintenance : | halted. RST_PORT : | is set to remote : | port no. : | : | : | <- IDREQ Host sends IDREQ | Request is received. to node being reset. | : | ID packet returned. ID -> ID packet received. | Host checks returned | RST_PORT value. | Sequence is repeated | until RST_PORT is | this port's no. If | neccessary, repeat | RST. | : Data is loaded <- SNTMDAT Host loads remote into physically (LP) system program addressed memory. using maintenance : write mechanisms. : Port generates MCNF -> MCNF confirmation confirmation. passed to host. : Host verifies load Port receives <- MDATREQ using maintenance request and responds read mechanims. by returning data with last packet having special RETMDAT -> Confirmation is flag. (LP) passed to host. : EXAMPLE OF NODE BOOTING VIA CI Page B-3 : Host compares data. : Repeat write/read operation until complete. : Load was successful, Port receives start <- STRT Host sends STRT. and responds by starting host execution. : System operation proceeds as in local load and boot case. EXAMPLE OF NODE BOOTING VIA CI Page B-4 B.0.3 A Booting Example -- 11/780 This outlines how an 11/780 system may be booted via the CI. The definition of "booting" (in this case) is the load and starting of a program in the host CPU. Although this description is specifically only for the 11/780, it should be generally extensible to other systems, particularly other VAX systems. This is the sequence as defined from the node controlling the boot: | | 1. Send a RST packet that will perform the reset function in the | remote node. This specifically implies a RST that will result | in the destination port entering the Uninitialized/Maintenance | state. This may, in fact, include sending an IDREQ to | determine the initial state, and sending either a conditional | or forced RST (or a sequence of both). | | 2. Poll for the reset being performed by sending IDREQ packets | and waiting for returned ID. The reset is successful when the | returned state is Uninitialized/Maintenance and the RST_PORT | is the number of the port sending the reset. | | If the correct ID is not received, the RST should be | re-transmitted after a suitable timeout period. Since the RST is a datagram packet, it may be discarded under certain conditions. A suitable timeout period would be 1 second. The reset operation on an 11/780 takes on the order of 100 ms under optimal conditions (no other packets are in the packet buffer). At most, the reset should take on the order of 200 ms. The SNDRST should be retried some number of times. A nominal suggested retry limit is 100. This is an arbitrary limit, as no analytical method exists for determining what this number should be for a given probability of failure. 3. Upon receipt and processing of the RST packet, the port will assert and hold FAIL. 5 ms later, the port will assert DEAD for 2 us. Upon release of DEAD, the console will attempt a WCS reload from the floppy. Upon completion of the load, the console will examine the AUTO RESTART switch. The switch must be in the ON position. Otherwise, the processor will halt. If the RESTART switch is ON, the console will attempt a restart sequence. In the 11/780, this begins with execution of the floppy command file "RESTAR.CMD". Any commands in this file will be executed until a START command is encountered. The start function will not be performed until FAIL is released by the CI780 interface (upon receipt of a STRT packet). Note that the restart command file must not contain any operations that will disturb the state of the port (such as UNJAM or deposits to certain port registers). The standard RESTAR.CMD (after version 2.0) is alledgedly compatible with these requirements. EXAMPLE OF NODE BOOTING VIA CI Page B-5 4. Using SNTMDAT packets (to write) and MDATREQ packets (to read) the memory, load and verify the desired boot program. As with the RST, these packets should retried. The suggested algorithm is to send a data packet and wait some timeout period (like 100 ms) for a MCNF. If not received before the timeout, the packet should be retried some limited number of times (like 100). A unique transaction ID should be used for each operation (not necssarily for each retry). The transaction ID of each response should be checked against the current transaction ID value. If the ID is not correct, the response should be ignored. This allows for duplication or delayed responses. Subsequent to the receipt of the MCNF, the corresponding MDATREQ should be posted. A similar wait and retry algorithm should be used for the RETMDAT packet with the LP flag set. Upon receipt of the RETMDAT packet (LP = 1), a compare of the buffer areas should be made to ensure correct loading. A compare error should, in general, be considered fatal. Such errors could, however, also be retried. Additionally, if the retry limit is ever reached, it also should be considered fatal. A program of any size may be loaded with multiple write/read operations. As part of the load, a Restart Parameter Block (RPB) must be set up at the base of the first usable 64 KB block in memory. The RPB consists of four longwords of the following format: 1. Longword 0 -- Address (physical) of the RPB (Longword 0). 2. Longword 1 -- Address (physical) of the program to be started. 3. Longword 2 -- 32-bit checksum (32 bit additions without carry) of first 31 longwords of program (starting at address specified in Longword 1 of the RPB). 4. Longword 3 -- Bit 0 must be a 0 to restart. Otherwise, cold start will occur. The address of the first usable 64 KB block can be determined via a remote "memory sizing test" using SNDMDAT and REQMDAT for writes and reads. A response will not be received for non-existent or failing memory. 5. Send a STRT packet. When the start packet is received, or when the processor WCS has been loaded and RESTAR.CMD executed (whichever comes first), the processor will begin executing the restart referee of the ROM bootstrap code. If the RPB is correct and no other errors occur, the ROM code will transfer EXAMPLE OF NODE BOOTING VIA CI Page B-6 control to the address specified in the RPB. Otherwise, cold start (from DEFBOO.CMD) will be attempted. 6. The booted program should eventually backload microcode and send a message of some type to confirm success of the boot. Note that if the port is restarted to load microcode, the RST_PORT value that would be returned in an ID packet is destroyed. Therefore, the port does not provide a mechanism to determine which node booted the system. This information could be passed in the loaded program. Otherwise, a boot message could be sent (successively) to all possible node. If the boot message is not received within a reasonable timeout period, the STRT packet should be transmitted until the program start is successful. The reasonable timeout period may be on the order of 10's of seconds. In the case of the 11/780, it is determined primarily by the console terminal speed and the floppy disk operation time. If the down-line load and start is complete before the console restart sequence, execution will begin when the console executes the START of the command file. If the down-line load requires more time than the console restart, execution will begin immediately on receipt of the STRT packet. This algorithm has been tested on an 11/780. The program that was down-line loaded included a copy of the port microcode. Upon starting, it initialized and loaded the port microcode, enabled the port, printed a console message, and sent a boot message (of text) to ports 0:15. The following sequence and times were observed using a console terminal at 9600 baud: EXAMPLE OF NODE BOOTING VIA CI Page B-7 1 EVENT TIME (from 0 in ms) SNDRST 0 IDREC (RST_PORT correct) 110 SNDMDAT 110 MCNFREC 120 REQMDAT 120 MDATREC 130 : 2 : 25,911 bytes (51 operations) : MDATREC 1280 SNDSTRT 1290 MSGREC (booted) 14330 NOTES: 1. Resolution is 10 ms. 2. Load bandwidth = 22.1 KB/s (includes test program overhead. APPENDIX C MINIMAL IMPLEMENTATION OF DUAL PATH INTERFACE CI interfaces are not required to support full operation on both paths simultaneously. It is compatible and permissible to share as much transmit and receive and path hardware as possible. An example of a highly shared hardware interface is one which only has path specific physical channel hardware, that is, duplicated media drivers, receivers and carrier detect circuitry. An example of such an implementation is fully described in the L0100 Link specification (an appendix of this document). Additionally, hardware may be shared for the transmit and receive functions. An example of this is to use the same CRC calculator for both checking and generation. This implies that transmit and receive cannot take place simultaneously, notably in loopback, where the higher layers (such as the port) may be required to perform the CRC calculation and load the result into the transmit buffer. The combination of these reductions requires the careful synchronization of the transmit and receive processes, since both processes are required (due to the required acknowledgement) for any transaction. In cases requiring prioritization decisions, the receipt of packets from the bus should have higher priority, to minimize the wasting of media bandwidth, otherwise available to other nodes. An example of such an implementation is described in detail in the L0100 Link Specification Appendix. All future implementations should use this design as a reference for minimal implementation of functional compatibility. APPENDIX D IMPLEMENTATION FUNCTIONALITY D.0.1 Implementation Functionality Field This field is located in the ID packet and describes the supported functionality of the port. Each bit of this field corresponds to a function. If the implementation supports the function, the corresponding bit value should be one. If not, the bit should be zero. Unused bits are reserved for future expansion and must be zero in current ports for compatibility with future functions. The format of PORT_FCN is: | | 3 2 2 | 1 6 5 8 7 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | STATES | PACKETS | RSVD (MBZ) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Where: Bit Function 31 Enabled state. 30 Enabled/Maintenance state. 29 Disabled state. 28 Disabled/Maintenance state. 27 Uninitialized state. 26 Uninitialized/Maintenance state. 25 Snd MSG/Rcv MSG 24 Snd SNTDAT/Rcv CNF 23 Rcv SNTDAT/Snd CNF | 22 Snd DATREQ0/Rcv RETDAT | 21 Snd DATREQ1/Rcv RETDAT | 20 Snd DATREQ2/Rcv RETDAT | 19 Rcv DATREQ0/Snd RETDAT | 18 Rcv DATREQ1/Snd RETDAT | 17 Rcv DATREQ2/Snd RETDAT | 16 Snd IDREQ/Rcv ID IMPLEMENTATION FUNCTIONALITY Page D-2 | 15 Snd SNTMDAT/Rcv MCNF | 14 Rcv SNTMDAT/Snd MCNF | 13 Snd MDATREQ/Rcv RETMDAT | 12 Rcv MDATREQ/Snd RETMDAT | 11 Snd RST | 10 Snd STRT | 9 Rcv RST-STRT | 8 Snd/Rcv LB | | 7 - 0 Reserved and MBZ. | | D.0.2 Implementation Functionality Summary "X" = Supported, "-" = Not Supported | OPTION | FUNCTION |-----+-----+-----+-------+-----+-----+-----+ |CI780|CI750|HSC50|JUPITER| KL10| CINT| | ------------------+-----+-----+-----+-------+-----+-----+-----+ MAINT_ID | 2 | 3 | 4 | 5 | 6 | 7 | | ------------------+-----+-----+-----+-------+-----+-----+-----+ | | | | | | | | STATES | | | | | | | | ------------------+-----+-----+-----+-------+-----+-----+-----+ Uninitialized | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ Uninit/Maint | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ Disabled | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ Dsbld/Maint | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ Enabled | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ Enbld/Maint | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ IMPLEMENTATION FUNCTIONALITY Page D-3 | OPTION | FUNCTION |-----+-----+-----+-------+-----+-----+-----+ |CI780|CI750|HSC50|JUPITER| KL10| CINT| | ------------------+-----+-----+-----+-------+-----+-----+-----+ | | | | | | | | PACKETS SND/RCV | | | | | | | | ------------------+-----+-----+-----+-------+-----+-----+-----+ MSG/MSG | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ SNTDAT/CNF | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ CNF/SNTDAT | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ DATREQ0/RETDAT | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ DATREQ1/RETDAT | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ DATREQ2/RETDAT | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ RETDAT/DATREQ0 | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ RETDAT/DATREQ1 | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ RETDAT/DATREQ2 | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ IDREQ/ID | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ SNTMDAT/MCNF | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ MCNF/SNTMDAT | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ MDATREQ/RETMDAT | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ RETMDAT/MDATREQ | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ RST/ | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ STRT/ | X | X | - | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ /RST-STRT | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ LB/LB | X | X | X | | | X | | ------------------+-----+-----+-----+-------+-----+-----+-----+ APPENDIX E ACTIVE EXCHANGE HUB SPECIFICATION \ TO BE ADDED IN THE FUTURE \ APPENDIX F L0100 LINK SPECIFICATION TITLE: L0100 CI LINK Functional and Interface Specification STATUS: Released ABSTRACT: This document describes the hardware interface to the CI LINK Interface module. AUTHOR: John Buzynski MAIL STOP: TW/C03 PHONE: 247-2329 REVISION HISTORY: Date Revision Comments ____ ________ ________ 1 OCT 1979 A (Draft) This document replaces and supersedes the PORT to LINK Interconnect (PLI) and the LINK Interface spec of August 7, 1979. 17 DEC 1979 B 18 MAR 1980 C 31 OCT 1980 D 1 MAY 1981 E Incorporate into CI spec. Update with minor changes. 15 NOV 1981 F Corrections from review. Update with CI spec rev 2. L0100 LINK SPECIFICATION Page F-2 F.1 TABLE OF CONTENTS F.1 TABLE OF CONTENTS . . . . . . . . . . . . . . . . F-2 F.2 REFERENCE DOCUMENTS . . . . . . . . . . . . . . . F-4 F.3 GENERAL . . . . . . . . . . . . . . . . . . . . . F-5 F.4 FUNCTIONAL DESCRIPTION . . . . . . . . . . . . . . F-6 F.4.1 PROTOCOL . . . . . . . . . . . . . . . . . . . . F-6 F.4.1.1 INFORMATION PACKETS . . . . . . . . . . . . . F-6 F.4.1.1.1 Bit Synchronization . . . . . . . . . . . . F-8 F.4.1.1.2 Character Synchronization . . . . . . . . . F-8 F.4.1.1.3 Packet Type/Length (High) . . . . . . . . . F-8 F.4.1.1.4 Packet Length (Low) . . . . . . . . . . . . F-9 F.4.1.1.5 Destination . . . . . . . . . . . . . . . . F-9 F.4.1.1.6 Source . . . . . . . . . . . . . . . . . . F-10 F.4.1.1.7 Body . . . . . . . . . . . . . . . . . . . F-10 F.4.1.1.8 Cyclical Redundancy Check (CRC) . . . . . F-10 F.4.1.1.9 Trailer . . . . . . . . . . . . . . . . . F-11 F.4.1.2 ACKNOWLEDGE PACKETS . . . . . . . . . . . . F-11 F.4.1.2.1 Packet Type . . . . . . . . . . . . . . . F-13 F.4.1.3 ARBITRATION . . . . . . . . . . . . . . . . F-13 F.4.1.3.1 PACKET ACKNOWLEDGEMENT . . . . . . . . . . F-16 F.4.2 DUAL PATH CI STRUCTURE . . . . . . . . . . . . F-17 F.4.2.1 HEADER TIMEOUT . . . . . . . . . . . . . . . F-18 F.4.3 MAINTENANCE MODES . . . . . . . . . . . . . . F-18 F.5 LINK INTERFACE . . . . . . . . . . . . . . . . . F-18 F.5.1 LINK INTERFACE SIGNALS . . . . . . . . . . . . F-19 F.5.1.1 XMIT DATA (7:0) (TTL INPUT, ASSERTED HIGH) . F-21 F.5.1.2 XMIT DATA PARITY (TTL INPUT, ASSERTED HIGH) F-21 F.5.1.3 RCVR DATA (7:0) (TTL OUTPUT, ASSERTED HIGH) F-22 F.5.1.4 RCVR DATA PARITY (ODD) (TTL OUTPUT, ASSERTED HIGH) . . . . . . . . . . . . . . . . . . . F-22 F.5.1.5 PORT DATA (7:0) (TTL INPUT, ASSERTED HIGH) . F-22 F.5.1.6 NODE ADDRESS (7:0) (TTL OUTPUT, ASSERTED HIGH) . . . . . . . . . . . . . . . . . . . F-22 F.5.1.7 SELECT (TTL INPUT, ASSERTED HIGH) . . . . . F-22 F.5.1.8 LINK CONTROL (3:0) (TTL INPUT, ASSERTED HIGH) F-23 F.5.1.8.1 TRANSMIT . . . . . . . . . . . . . . . . . F-23 F.5.1.8.2 RESET TRANSMIT STATUS . . . . . . . . . . F-24 F.5.1.8.3 ABORT TRANSMISSION . . . . . . . . . . . . F-24 F.5.1.8.4 ENABLE LINK CONTROL/DISABLE LINK CONTROL . F-24 F.5.1.9 XMIT STATUS (7:0) (TTL OUTPUT, ASSERTED HIGH) F-28 F.5.1.10 XMIT ATTENTION (TTL OUTPUT, ASSERTED HIGH) . F-30 F.5.1.11 XMIT DATA ENABLE (TTL OUTPUT, ASSERTED HIGH) F-30 F.5.1.12 XMIT BUFFER EMPTY (TTL INPUT, ASSERTED HIGH) F-31 F.5.1.13 VALID RCVR DATA (TTL OUTPUT, ASSERTED HIGH) F-31 F.5.1.14 RCVR PACKET END (TTL INPUT, ASSERTED HIGH) . F-31 F.5.1.15 RCVR BUFFERS FULL (TTL INPUT, ASSERTED HIGH) F-31 F.5.1.16 VALID RCVR STATUS (TTL OUTPUT, ASSERTED HIGH) F-31 F.5.1.17 PACKET LENGTH (TTL OUTPUT, ASSERTED HIGH) . F-32 F.5.1.18 ICCS PATH B (TTL OUTPUT) . . . . . . . . . . F-32 F.5.1.19 CRC STATUS (TTL OUTPUT) . . . . . . . . . . F-32 F.5.1.20 PORT CLOCK (TTL INPUT) . . . . . . . . . . . F-32 F.5.1.21 XMIT CLOCK . . . . . . . . . . . . . . . . . F-33 F.5.1.22 RCVR CLOCK . . . . . . . . . . . . . . . . . F-33 F.5.1.23 INITIALIZE (TTL INPUT, ASSERTED HIGH) . . . F-33 L0100 LINK SPECIFICATION Page F-3 F.5.1.24 RCVR A ENABLE (TTL OUTPUT, ASSERTED High) . F-33 F.5.1.25 RCVR B ENABLE (TTL OUTPUT, ASSERTED High) . F-33 F.5.1.26 XTEND HDR (TTL INPUT, ASSERTED Low) . . . . F-34 F.5.1.27 ALTER DELTA TIME (TTL INPUT, ASSERTED Low) . F-34 F.5.1.28 DISABLE ARB (TTL INPUT, ASSERTED Low) . . . F-34 F.5.1.29 MODULE INSERTED LOOP . . . . . . . . . . . . F-34 F.5.1.30 EXTEND ACK TO (TTL INPUT, ASSERTED LOW) . . F-34 F.5.1.31 DISABLE RCVR CLK (TTL INPUT, ASSERTED Low) . F-34 F.5.1.32 RCVR TEST CLK (TTL INPUT, ASSERTED Low) . . F-35 F.5.1.33 XMIT TEST CLK (Low High) (TTL INPUTS) . . . F-35 _ F.5.1.34 XMIT CLK DISABLE (TTL INPUT, ASSERTED Low) . F-35 F.5.2 LINK INTERFACE TIMING . . . . . . . . . . . . F-36 F.6 MECHANICAL . . . . . . . . . . . . . . . . . . . F-41 F.6.1 General . . . . . . . . . . . . . . . . . . . F-41 F.6.2 Module Keying . . . . . . . . . . . . . . . . F-41 F.6.3 Power Requirements . . . . . . . . . . . . . . F-41 F.6.4 Cooling . . . . . . . . . . . . . . . . . . . F-41 F.6.5 LEDS . . . . . . . . . . . . . . . . . . . . . F-42 F.6.5.1 INTERNAL MAINTENANCE LOOP (RED) . . . . . . F-42 F.6.5.2 LOCAL CI ACTIVITY (GREEN) . . . . . . . . . F-42 F.7 APPENDIX A . . . . . . . . . . . . . . . . . . . F-42 F.8 APPENDIX B . . . . . . . . . . . . . . . . . . . F-43 F.9 APPENDIX C . . . . . . . . . . . . . . . . . . . F-44 L0100 LINK SPECIFICATION Page F-4 F.2 REFERENCE DOCUMENTS 1. VAX-11 CI PORT Architecture -(W. Strecker); describes the VAX-11 software interface to the Computer Interconnect (CI). 2. CI Architecture Specification -(D. Thompson, J. Buzynski, J. Hutchison); Describes the structure, electrical characteristics, configuration, information format, and protocol for the CI system. 3. CI780 PORT Specification -(M. Carrafiello) Describes and specifies the interface between the SBI and CI. L0100 LINK SPECIFICATION Page F-5 F.3 GENERAL The PORT PROCESSOR and LINK combine to provide a device with the control mechanism and interface to the CI Bus. The PORT PROCESSOR is the interface to the host device and is responsible for the data mapping, data buffering and the control of the LINK interface. The LINK provides the actual interface to the CI Bus and is capable of servicing a dual path CI Bus. The LINK can only transmit or receive over one CI path at any one time. The LINK interface allows for discrete selection of the transmit path and provides automatic selection of the receive path by monitoring the "initial presence" of carrier on either path. The LINK has only a single CRC circuit which is shared by the transmitter and receiver, therefore, simultaneous transmissions and/or receptions cannot occur at a single node unless the LINK is in one of the maintenance loop modes. The serialization and deserialization of data, 32-bit CRC generation and checking, CI protocol handling, Dual Count-down Round Robin arbitration and LINK Status information to the PORT PROCESSOR are provided by the LINK. Internal and external maintenance loop mechanisms are provided by the LINK. L0100 LINK SPECIFICATION Page F-6 F.4 FUNCTIONAL DESCRIPTION This section describes the LINK functionality. It is expected that the reader has read and understands the "CI Architecture Specification". Some pieces of the above specification are repeated in this section for convenience. F.4.1 PROTOCOL The LINK hardware implements the CI Bus protocol to arbitrate for the CI Bus, to transmit and receive packets, and to confirm the reception of those packets. There are two types of packets handled by the LINK, Information packets and Acknowledge packets. F.4.1.1 INFORMATION PACKETS - The organization and description of the contents of the Information packet is given below. The Information packet is used to transmit both message and data packets across the CI. Some of the CI packet information is inserted by the LINK and some is inserted by the PORT PROCESSOR as described below. L0100 LINK SPECIFICATION Page F-7 7 0 +-----------------------------+ | BIT SYNC (55 HEX) | First byte +-----------------------------+ Transmitted | BIT SYNC (55 HEX) | +-----------------------------+ | BIT SYNC (55 HEX) | +-----------------------------+ | BIT SYNC (55 HEX) | +-----------------------------+ | BIT SYNC (55 HEX) | +-----------------------------+ | CHAR SYNC (96 HEX) | +-----------------------------+ | PACKET TYPE/LENGTH (High)| First byte loaded +-----------------------------+ or read by PORT PROCESSOR | PACKET LENGTH (Low) | +-----------------------------+ | DESTINATION (TRUE) | +-----------------------------+ | DESTINATION (COMPLEMENT) | +-----------------------------+ | SOURCE | +-----------------------------+ | BODY | Last byte loaded +-----------------------------+ or read by PORT PROCESSOR | CRC-0 | +-----------------------------+ | CRC-1 | +-----------------------------+ | CRC-2 | +-----------------------------+ | CRC-3 | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ : TRAILER (00 HEX) : +-----------------------------+ FIGURE F4-1 INFORMATION PACKET FORMAT L0100 LINK SPECIFICATION Page F-8 F.4.1.1.1 Bit Synchronization - Bit Synchronization (55 hexadecimal) is an alternating pattern of 1's and 0's which is used to turn on the carrier detect circuits and to synchronize the delay line decoder circuit in the receiver. The LINK hardware inserts this bit pattern into the data stream. There are normally 5 bytes of Bit Sync transmitted. Bit sync may be extended to 15 bytes by grounding I/O pin B7. F.4.1.1.2 Character Synchronization - Character Synchronization (96 Hexadecimal) is the character used to synchronize the receiver on an 8 bit character basis. The receiver will monitor the serial bit stream for the SYNC Character and will frame the stream into 8 bit characters once the SYNC character is found. The LINK hardware inserts the SYNC character into the data stream. F.4.1.1.3 Packet Type/Length (High) - The Packet Type/Length field contains information specifying whether the packet is an Information type or whether it is an Acknowledge type. Information packets are of variable length in 1 byte increments up to 4K bytes maximum with the minimum packet length being 7 bytes. This byte contains the high 4 bits of the 12 bit packet length. The Packet Type /Length field is described as follows: +-----+-----+-----+-----+-----+-----+-----+-----+ | | | | | | | | | | 0 | 0 | 0 | 0 | L11 | L10 | L9 | L8 | | | | | | | | | | +-----+-----+-----+-----+-----+-----+-----+-----+ 7 6 5 4 3 2 1 0 FIGURE F4-2 PACKET TYPE/LENGTH FIELD FOR INFORMATION PACKETS L0100 LINK SPECIFICATION Page F-9 BIT 7: 0 = Information packet BIT 6: This bit must be zero for information packets. BITS (5:4): These bits are reserved for future use and must be zero. BITS (3:0): These bits form the high 4 bits of the 12 bit packet length field. F.4.1.1.4 Packet Length (Low) - This byte contains the low 8 bits of the 12 bit packet length field. +-----+-----+-----+-----+-----+-----+-----+-----+ | | | | | | | | | | L7 | L6 | L5 | L4 | L3 | L2 | L1 | L0 | | | | | | | | | | +-----+-----+-----+-----+-----+-----+-----+-----+ 7 6 5 4 3 2 1 0 FIGURE F4-3 PACKET LENGTH (Low) BYTE FOR INFORMATION PACKETS The 12 bit packet length includes all data from the Packet Type/Length byte up to and including the last byte of the body. A value of 0 indicates 4096 byte length. F.4.1.1.5 Destination - The Destination is the 8 bit address of the target CI node. There are two bytes; the first being the true node address value and the second being the complement of the true value. The LINK will save the true value during a transmission to be used to verify the Source of the expected acknowledgement packet. The PORT Processor supplies the Destination bytes as part of the packet. The true and complement forms are sent to accomodate high availability systems as follows. One of the goals of a high availability system is that no single component failure may permanently bring down both CI paths. Since this LINK implementation services 2 CI paths with a single transmit and receive path, a failure in the common destination decode logic, causing a node to decode another node's address might have L0100 LINK SPECIFICATION Page F-10 resulted in both nodes transmitting an Acknowledge packet simultaneously. This would result in a collision on the CI and would be seen as a no response by the transmitting node. To overcome this problem the LINK incorporates redundant destination decode circuits, one decoding the true address and the other decoding the complemented address. F.4.1.1.6 Source - The Source is the 8 bit address of the sending node and is supplied by the PORT PROCESSOR as part of the packet. The receiver will save this address and use it as the destination address in the Acknowledge packet. F.4.1.1.7 Body - The Body contains the data and PORT-processed protocol information. The Body contents are specified for Data and Message packet types in the CI Architecture Specification. The Body is supplied by the PORT as part of the packet. F.4.1.1.8 Cyclical Redundancy Check (CRC) - The Cyclical Redundancy Check (CRC, 32 bit) is provided for error detection. The CRC is generated and checked using the polynomial: X32 + X26 + X23 + X22 + X16 + X12 + X11 + X10 + X8 + X7 + X5 + X4 + X2 + X + 1 The transmitter and receiver CRC generators are preset to all 1's prior to generating the CRC characters. The transmitter sends the complement of the generated CRC characters LSB first which is the high order coefficient end of the CRC. An error free packet reception results in the value DEBB20E3 (HEX). CRC is generated by the LINK hardware on the Packet Type/Length, Packet Length, Destination, Source and Body of the packet. Four bytes of CRC are generated and transmitted with CRC-0 being the least significant byte. L0100 LINK SPECIFICATION Page F-11 F.4.1.1.9 Trailer - The Trailer is normally 6 characters of NUL (00 HEX) data which is used to insure that all bits of the packet have been shifted through the LINK front end logic. The Trailer is inserted by the LINK hardware. F.4.1.2 ACKNOWLEDGE PACKETS - The Acknowledge packet is sent by the receiving node to inform the information transmitting node that the packet arrived without collision or data loss (CRC checks OK). The packet contains STATUS information as to whether the packet was accepted. If the receiver's buffer(s) were unable to accept a packet, then a NAK status is returned. If the receiving node successfully accepted the packet into its buffer, an ACK is returned denoting the successful bus transaction. The entire Acknowledge packet is generated and transmitted by the LINK hardware. The format of the Acknowledge packet is shown below. L0100 LINK SPECIFICATION Page F-12 7 0 +-----------------------------+ First byte | BIT SYNC (55 HEX) | Transmitted +-----------------------------+ | BIT SYNC (55 HEX) | +-----------------------------+ | BIT SYNC (55 HEX) | +-----------------------------+ | BIT SYNC (55 HEX) | +-----------------------------+ | BIT SYNC (55 HEX) | +-----------------------------+ | CHAR SYNC (96 HEX) | +-----------------------------+ | PACKET TYPE | +-----------------------------+ | DESTINATION (TRUE) | +-----------------------------+ | DESTINATION (COMPLEMENT) | +-----------------------------+ | SOURCE | +-----------------------------+ | CRC-0 | +-----------------------------+ | CRC-1 | +-----------------------------+ | CRC-2 | +-----------------------------+ | CRC-3 | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ | TRAILER (00 HEX) | +-----------------------------+ : TRAILER (00 HEX) : +-----------------------------+ FIGURE F4-4 ACKNOWLEDGE PACKET FORMAT As shown above, there is no Body in the Acknowledge packet. The descriptions of all other fields except the Packet Type/Length and the Packet Length fields are the same for Acknowledge packets as they are for Information packets. L0100 LINK SPECIFICATION Page F-13 F.4.1.2.1 Packet Type - The Packet Type field is defined as follows: +-----+-----+-----+-----+-----+-----+-----+-----+ | | ACK/| | | | | | | | 1 | NAK | 0 | 0 | 0 | 0 | 0 | 0 | | | | | | | | | | +-----+-----+-----+-----+-----+-----+-----+-----+ 7 6 5 4 3 2 1 0 FIGURE F4-5 PACKET TYPE/LENGTH FIELD FOR ACKNOWLEDGE PACKETS BIT 7: 1 = Acknowledge packet type. BIT 6: 1 = Acknowledge with ACK status. Packet was received correctly and accepted into the packet buffer. 0 = Acknowledge with NAK status. Packet was received correctly but was not accepted into the packet buffer. Requires retransmission. BITs (5:0) These bits must be zero for Acknowledge packets. F.4.1.3 ARBITRATION - The arbitration used for the CI is a Dual Countdown Round Robin algorithm. Consider an N node system where node I has two possible countdown values I and N + I. The choice of which value to use depends on whether (from the node I point of view) the last node (modulo N) to win arbitration had a lower or higher node number than node I. If the last winner had a lower number, node I uses the I countdown value; if the last winner had the same or a higher number, node I uses the N + I countdown value. N is always = 16 = maximum # nodes allowed on the system. I = this node number. L0100 LINK SPECIFICATION Page F-14 The way a node determines who won the last arbitration is as follows. At the beginning of arbitration, a node saves a copy of its own countdown value. When the arbitration ends (either by detection of carrier or countdown to 0), the difference (modulo N) of the initial and final countdown values is the node number of the node which won the arbitration. The above is true only when there is activity on the CI since the de-assertion of carrier is necessary to synchronize each LINK's arbitration logic. Once the arbitration has been successful (count = 0), the transmitter checks to see that the receiver is not busy on the alternate CI path. If it is not, the receiver will be locked to the selected transmit path and the transmission will begin. If the receiver is busy, the transmitter will load 16 into the arbitration counter and continue to arbitrate. Receiver busy is defined as follows: 1. The receiver has detected carrier on the alternate path but has not yet detected its own destination and has not exceeded the header timeout. 2. The receiver has detected its own destination and is accepting the packet on thee alternate CI path. 3. A valid packet has been received and the transmitter is in the process of transmitting the Acknowledge packet. Figure F4-6 below is a flow diagram of this LINK's implementation of the arbitration algorithm. L0100 LINK SPECIFICATION Page F-15 START ----->-<---------------------------------------- | ^ | v | | TRANSMIT | | ? | | |------ NO | YES | | v | COUNT <- N+I+1 | | | ------------------>|<----------------------------------- | | v | | | CARRIER | | | ? | | | |--------------- | | | YES | NO | | | | | v | | | | COUNT <- COUNT - 1 | | | v | | | | LW <,>= I v | | | ----------- ? ------ COUNT = 0 | | | | < >= | ? | | | v v |------------------->| | | COUNT <- I+1 COUNT <- N+I+1 | NO ^ | | | | YES | | | | -------------------- v | | | | RCVR BUSY | | ------------------- ? | | |------------ | | NO | YES | | | | | | | | | | | | | | | | | | | |NO | | | v | | | TAKE OVER RCVR & | | | TRANSMIT PACKET | | | N = Max # nodes on system | v | | I = This node number | COUNT <- 16 | | LW = Last winner (Mod N) | | | | | | | |<-------- ------- | v | | TRANSMISSION DONE | | ? | | |--------- | | NO | YES |__________________________| FIGURE F4-6 ARBITRATION FLOW DIAGRAM L0100 LINK SPECIFICATION Page F-16 The arbitration count is the number of consecutive quiet slots that must be observed before a node can transmit on an CI path. The basic quite slot interval (refer to CI Architecture Spec) is long enough to allow a receiving node to return an Acknowledge packet. Acknowledge packets are the only packets transmitted over the bus without first arbitrating for control. Successful arbitration is defined as successfully transmitting a packet without collision. A collision is detected by the CRC calculation. F.4.1.3.1 PACKET ACKNOWLEDGEMENT - After the LINK has transmitted an Information packet, the receiver waits for an Acknowledge packet to be returned. The Contents of the Acknowledge packet will indicate either an ACK (successful transmission) or NAK status. If an ACK is received, then the transmission was received successfully and stored within the responder's packet buffer. If a NAK status is received, then, the Information packet was received without error. However, the responder did not have a packet buffer available to accept the packet and a retransmission is necessary. In the case when no Acknowledge packet is received (no response) the indication is that the packet became corrupted along the way, that the node was not available, that the node was involved with the other interconnect path at the time of the transmission, or some hardware failure has occurred. In any case, a retransmission is necessary. If an Acknowledge packet is received with its Source field matching the Destination field of the Information transfer and correct CRC within an Acknowledge timeout period of D2 from the end of Information packet transmit, where: D2 >(Delta + Ack Turn-around time + Ack verification time) and Delta = basic quiet slot time Ack turn-around time = skews due to LINK logic then, the receiver (in the transmitting node) notifies the PORT PROCESSOR of successful transmission. D2 is normally approximately equal to 32 byte times. This time may be extended to approximately 256 byte times by grounding I/O pin B10. Delta is normally equal to 7 byte times (800 ns). This time may be extended to 16 byte times (1.83 ns.) by grounding I/O pin in B8. If the Acknowledge packet is not received before the period D2 expires, then, the no response status will be returned by the LINK as described in section F5.1.9. L0100 LINK SPECIFICATION Page F-17 F.4.2 DUAL PATH CI STRUCTURE The LINK provides the capability of servicing two CI paths allowing devices to connect to high availability systems. The two paths are not completely redundant, but, are implemented by providing dual drivers and receivers which are connected to common transmit and receive logic circuits. Simultaneous transmission and reception is not possible unless the LINK is in one of the loop modes described in section F5.1.8.4. The PORT must indicate which CI path it wants to transmit on when it issues the "Transmit" function. The LINK automatically selects which CI path to receive on by constantly monitoring both CI paths to detect the "initial presence" of a carrier on either path and switching its shared logic to the path containing the transmission. If the LINK detects a transmission on both paths simultaneously, it always selects path A (arbitrary decision). The LINK will then decode the Destination field of the transmission. If, when the Destination field is decoded, the LINK determines that the transmission is meant for it, the LINK will continue receiving the information from the selected path. At the completion of the receiving process the LINK will again assume a state that will be able to detect the "initial presence" of a transmission on either path. If, when the destination field is decoded, the LINK determines that the transmission is not meant for it, the LINK will ignore the remainder of the transmission and immediately assume the state that will be able to detect the "initial presence" of a transmission on either interconnect path. The above allows the LINK to detect and receive transmissions under the following conditions with the corresponding results: 1. A single transmission on either path will be accepted. 2. If a transmission on each path is detected by this node at the same time, the packet on path A will be accepted. The LINK always selects path A when it detects the simultaneous "initial presence" of carrier on both CI paths. If the packet on path B was for this node, the packet will not be received. Acknowledge will not be received by the transmitter and a retransmission may be attempted by the Transmitter. 3. If transmissions occur on each path but are not detected by this node at the same time, the transmission that is detected first will be received. If the packet on the other path is for this node, it will be accepted only if the length of time between the two transmissions is longer than: L0100 LINK SPECIFICATION Page F-18 - the time required for a LINK to determine that the first transmission is not meant for it, PLUS - the time required to detect the initial presence of the second transmission. Otherwise, the second packet will be ignored. The above discussion concerning the automatic selection of the CI receive path is valid providing this node is not currently in the process of transmitting a packet. The receiver is locked to the selected CI path during the entire transmission and until an acknowledgement is either received or timed out. F.4.2.1 HEADER TIMEOUT - The LINK receiver includes a Header timeout period of approximately 32 byte times, starting from the time when a message carrier is selected, which will release the receiver from a path if the receiver is unable to detect the SYNC character within the timeout period. The Header timeout is shut off when the SYNC character is detected. Header timeout is disabled for ACK reception where ACK timeout is utilized. F.4.3 MAINTENANCE MODES There are several maintenance features implemented as diagnostic aids in isolating LINK module failures. Internal and External Maintenance loop modes are provided to allow a node to transmit a packet to itself. Wrong receiver parity generation is possible. The ability to force a carrier detect is provided to aid in isolating an arbitration logic failure. Finally, there is provision for swapping the true and complement node address sources. All of the above features are described in detail in section F5.1.8.4. F.5 LINK INTERFACE This section describes the hardware interface between the PORT and the LINK. Detail timing diagrams are included in section F5.2. L0100 LINK SPECIFICATION Page F-19 F.5.1 LINK INTERFACE SIGNALS The following figure lists the signals necessary to interface to the LINK module. All LINK interface signals must be capable of driving a minimum of 2 Schottky loads except for the three clock signals and Initialize which must be capable of driving 4 Schottky loads. LINK interface input signals must be limited to 2 Schottky loads except for the three clock signals and Initialize which may be 4 Schottky loads. In - Out - Assertion Module Pin Signal Put Put Logic Level __________________________________________________________________ XMIT Data 7 x TTL High B81 XMIT Data 6 x TTL High B82 XMIT Data 5 x TTL High B80 XMIT Data 4 x TTL High B78 XMIT Data 3 x TTL High B77 XMIT Data 2 x TTL High B76 XMIT Data 1 x TTL High B75 XMIT Data 0 x TTL High B74 XMIT Data Parity x TTL High B73 ILIF RCVR Data 7 x TTL High B58 ILIF RCVR Data 6 x TTL High B57 ILIF RCVR Data 5 x TTL High B56 ILIF RCVR Data 4 x TTL High B55 ILIF RCVR Data 3 x TTL High B54 ILIF RCVR Data 2 x TTL High B53 ILIF RCVR Data 1 x TTL High B52 ILIF RCVR Data 0 x TTL High B51 ILIF RCVR Data Parity x TTL High B50 Port Data 7 x TTL High B27 Port Data 6 x TTL High B28 Port Data 5 x TTL High B25 Port Data 4 x TTL High B26 Port Data 3 x TTL High B24 Port Data 2 x TTL High B22 Port Data 1 x TTL High B21 Port Data 0 x TTL High B20 ILIK NODE Address 7 x TTL High B19 ILIK NODE Address 6 x TTL High B17 ILIK NODE Address 5 x TTL High B18 ILIK NODE Address 4 x TTL High B15 ILIK NODE Address 3 x TTL High B13 ILIK NODE Address 2 x TTL High B14 ILIK NODE Address 1 x TTL High B11 ILIK NODE Address 0 x TTL High B12 Select x TTL High B29 LINK Control 3 x TTL High B36 L0100 LINK SPECIFICATION Page F-20 In - Out - Assertion Module Signal Put Put Logic Level Pin __________________________________________________________________ LINK Control 2 x TTL High B34 LINK Control 1 x TTL High B32 LINK Control 0 x TTL High B31 ILIL XMIT Status 7 x TTL High B59 ILIL XMIT Status 6 x TTL High B69 ILIL XMIT Status 5 x TTL High B70 ILIB XMIT Status 4 x TTL High B67 ILIL XMIT Status 3 x TTL High B68 ILIL XMIT Status 2 x TTL High B65 ILIL XMIT Status 1 x TTL High B66 ILIL XMIT Status 0 x TTL High B63 ILIL XMIT Attention x TTL High B64 ILIC XMIT Data Enable x TTL High B62 XMIT Buffer Empty x TTL High B60 ILIA Valid RCVR Data x TTL High B49 RCVR Packet End x TTL High B30 RCVR Buffers Full x TTL High B43 ILIC Valid RCVR Status x TTL High B44 ILIA Packet Length x TTL High B41 ILIJ ICCS Path B x TTL High B39 ILIV CRC Status x TTL High B40 Port Clock x TTL High B71 ILIR XMIT Clock x TTL High B85 ILIN RCVR Clock x TTL High B42 Initialize x TTL High B35 ILIM RCVR A Enable x TTL High B37 ILIM RCVR B Enable x TTL Low B38 Extend HDR/TR x TTL Low B7 Alter Delta Time x TTL Low B8 Disable ARB x TTL Low B9 Extend ACK to x TTL Low B10 Module Inserted B5 Indication B6 Disable RCVR CLK x TTL Low B83 RCVR Test CLK x TTL Low B84 XMIT Test CLK x TTL Low B88 XMIT Test CLK x TTL High B87 XMIT CLK Disable x TTL Low B86 CIB XMIT x 10K ECL C11 CIB XMIT Shield x 10K ECL C7, C8, C9, C10 C12, C13, C14 CIA XMIT x 10K ECL C29 CIA XMIT Shield x 10K ECL C25, C26, C27, C28, C30, C31, C32 CIA RCVR x C83 CIA RCVR SHIELD x C79, C80, C81, C82, C84, C85, C86 L0100 LINK SPECIFICATION Page F-21 In - Out - Assertion Module Signal Put Put Logic Level Pin __________________________________________________________________ CIB RCVR x C67 CIB RCVR SHIELD x C63, C64, C65, C66, C68, C69, C70 +5V x A3, A4, A91, B3,B4,B46, B47 B91, B92, C3, C91, C92 -5.2V x C2, C45, C46, C47 C89, C90 GND x A2, A16, A23, A33, A61, A72, A79, A92, B1, B16 B23, B33, B61, B72, B79, B90, B93, B94, C1, C19, C20, C39, C40, C48, C53, C54, C75, C76, C93, C94 ___________________________________________________________________ FIGURE F5-1 LINK Interface Signals F.5.1.1 XMIT DATA (7:0) (TTL INPUT, ASSERTED HIGH) - These lines are used to transfer transmit data from the PORT Processor transmit packet buffer to the LINK Output Buffer Register. The LINK Strobes data from these lines when XMIT DATA ENABLE is asserted and ignores the XMIT DATA Lines at all other times. Data must update each XMIT Clock cycle as long as there is data to be transmitted and XMIT DATA ENABLE is asserted. F.5.1.2 XMIT DATA PARITY (TTL INPUT, ASSERTED HIGH) - Odd parity must be calculated by the PORT Processor on each packet data byte transferred to the LINK. This parity bit must be conveyed to the LINK via the XMIT DATA PARITY line along with its corresponding byte of data on the XMIT DATA lines when XMIT DATA ENABLE is asserted. The LINK will check parity at the output of L0100 LINK SPECIFICATION Page F-22 the LINK XMIT DATA Register and will terminate the transmission in the event of an error. F.5.1.3 RCVR DATA (7:0) (TTL OUTPUT, ASSERTED HIGH) - These lines are used to transfer the data received from an CI Path to the receive packet buffers in the PORT Processor. RCVR DATA is valid and is updated each RCVR CLOCK cycle as long as VALID RCVR DATA is asserted. The first byte of data is valid during the same cycle in which VALID RCVR DATA is first asserted. The PORT Processor is responsible for keeping a byte count to determine when the last valid byte is transferred. F.5.1.4 RCVR DATA PARITY (ODD) (TTL OUTPUT, ASSERTED HIGH) - Data being transferred to the PORT Processor via the RCVR DATA lines will be accompanied by an odd parity bit on the RCVR DATA PARITY line. It is up to the PORT to store and check parity on the received data. Parity will only be valid when VALID RCVR DATA is asserted and there is still data to be transferred. There is a maintenance feature which allows the wrong parity to be generated by the LINK and it is discussed in section (F5.1.8.4). F.5.1.5 PORT DATA (7:0) (TTL INPUT, ASSERTED HIGH) - The PORT DATA lines are used to transfer control information to the LINK. Examples include the CI path selection and LINK control set up information, with the Enable/Disable LINK functions. F.5.1.6 NODE ADDRESS (7:0) (TTL OUTPUT, ASSERTED HIGH) - The 8 bit CI node address for this LINK is available to the PORT via the NODE ADDRESS lines. The address is set up VIA an 8 bit PIANO Dip switch on the LINK. There are two 8 bit dip switches, providing the true and compliment node addresses. F.5.1.7 SELECT (TTL INPUT, ASSERTED HIGH) - The SELECT line must be asserted by the PORT to indicate that a valid control function exists on the LINK CONTROL lines. L0100 LINK SPECIFICATION Page F-23 F.5.1.8 LINK CONTROL (3:0) (TTL INPUT, ASSERTED HIGH) - There are four LINK CONTROL lines supplied by the PORT which are used to control the LINK. Control functions denoted by (*) utilize the PORT DATA lines to pass additional control information to the LINK. The SELECT line must be asserted to make the LINK CONTROL lines valid. Note that all control codes with bit 3 = 1 are null functions. This is because the LINK CONTROL functions are decoded as a subset of the CI 780 Buffer control lines. Other PORT designs may simply ground LINK CONTROL bit 3 on their backpanel if only 3 LINK CONTROL bits are supplied to the LINK. The LINK CONTROL lines are encoded as follows: ----------------------------------------- | LINK CONTROL BIT | FUNCTION | | 3 2 1 0 | | |--------------------|--------------------| | 0 0 0 0 | NULL | | 0 0 0 1 | NULL | | 0 0 1 0 | RESERVED | | 0 0 1 1 | TRANSMIT (*) | | 0 1 0 0 | RESET TRANSMIT | | | STATUS | | 0 1 0 1 | ABORT TRANSMISSION| | 0 1 1 0 | ENABLE LINK | | | CONTROL (*) | | 0 1 1 1 | DISABLE LINK | | | CONTROL (*) | | 1 0 0 0 | NULL | | THRU | " | | 1 1 1 1 | NULL | |____________________|____________________| FIGURE F5-2 LINK CONTROL F.5.1.8.1 TRANSMIT - The "Transmit" function is used to initiate arbitration and transmission on one of the CI paths. The CI path to be used is selected by PORT DATA (7:0) bit 7 as follows. PORT DATA bit 7 DESCRIPTION ________________________________________________ 0 Select CI path A 1 Select CI path B L0100 LINK SPECIFICATION Page F-24 Only one "Transmit" function can be executed at a time. That is, a transmission must have been completed or aborted and the Transmit Status must have been cleared before a second "Transmit" function can be issued. F.5.1.8.2 RESET TRANSMIT STATUS - This function must always be executed at the end of a transmission sequence, i.e., in response to a XMIT ATTENTION after reading the Transmit Status. "Reset Transmit Status" will reset all Transmit Status bits except bits 5 (CDA) and 6 (CDB). F.5.1.8.3 ABORT TRANSMISSION - "Abort Transmission" will cause the currently active transmission to be aborted and will set bit 4 (Transmission Aborted) of the Transmit Status. XMIT ATTENTION will also be asserted. F.5.1.8.4 ENABLE LINK CONTROL/DISABLE LINK CONTROL - These functions are used to enable and disable certain long term controls in the LINK. A particular control may be set by executing "Enable LINK Control" with a 1 in the PORT DATA (7:0) bit position corresponding to that control. A control may be cleared by executing "Disable LINK Control" with a 1 in the proper bit position. Transfer of a 0 in any bit position will have no effect on the corresponding controls. The PORT DATA (7:0) bit layout is as follows: 7 6 5 4 3 2 1 0 +------+-------+------+------+------+------+------+-------+ | RCVR |FORCE | VALID| EXTM | INTM |FORCE |SWAP |RCVR | | B ENA|CDET | RPAR | LOOP | LOOP | ARB |ADR |A ENA | | |ENA | ENA | ENA | ENA | ENA |ENA | | +------+-------+------+------+------+------+------+-------+ FIGURE F5-3 ENABLE LINK CONTROL L0100 LINK SPECIFICATION Page F-25 7 6 5 4 3 2 1 0 +------+-------+------+------+------+------+------+-------+ | |FORCE | VALID| EXTM |INTM |FORCE |SWAP | RCVR | | RCVR |CDET | RPAR | LOOP |LOOP |ARB |ADR | A DIS | | B DIS|DIS | DIS | DIS |DIS |DIS |DIS | | +------+-------+------+------+------+------+------+-------+ FIGURE F5-4 DISABLE LINK CONTROL Bit Description ________________________________________________________________ 7 Receiver B Enable/Disable: "Enable...." activates LINK receiver Path B making the node accessible on CI Path B by remote nodes. "Disable..." will cause this node to ignore all CI traffic on Path B. INITIALIZE will disable receiver Path B. 6 Force Carrier Detect: This function must only be used in conjunction with Internal Maintenance Loop mode. "Enable..." causes the LINK to see a detected carrier. This forced carrier will start the header timeout logic and will stop the arbitration sequence. The normal carrier detect signals from CI paths A and B are disabled when the Link is in Internal Maintenance Loop mode. "Disable..." and Initialize will disable this function. 5 Valid Receiver Parity: "Enable..." function will force the LINK to generate odd parity on the data being received. "Disable..." will cause the receiver to generate even parity on the data. INITIALIZE will set the receiver to odd parity. 4 External Maintenance Loop: "Enable..." will allow the LINK to receive its own transmission by looping on the selected CI path. NOTE: It is illegal to invoke more than 1 loop mode simultaneously. L0100 LINK SPECIFICATION Page F-26 The following requirements must be met: 1. The Destination fields in the CI packet must contain the address of this node. 2. The PORT must load a pre-calculated CRC into the buffer immediately following the data. CRC will be checked by the receiver. 3. The packet byte count must not include the CRC characters. The normal CI arbitration sequence will occur when a "Transmit" function is issued while in External Maintenance Loop mode. If External Maintenance Loop mode is invoked while on line in an operating system, packets may still be received from a remote node. There is, therefore, no guarantee that there will be a receiver buffer available when the local packet is transmitted. ACK/NAK responses are prohibited while in External Maintenance mode, therefore, a remote node will receive no response if it attempts to access a node that is in this node. The LINK will still accept packets from remote nodes and pass them on to the PORT Processor packet buffer but will not send an ACK/NAK in response while in External Maintenance Loop mode. However, the Transmit Status response will be set as though a normal transmission had taken place if a node Transmits to itself in this node.. "Disable..." will take the link out of External Maintenance Loop mode. INITIALIZE will disable this function. 3 Internal Maintenance Loop: "Enable..." will cause the LINK to receive its own transmission by looping inside the CI driver/receiver. This loop will not interfere with CI operation of other nodes. That is, neither CI path will be driven while transmitting in this loop mode. Traffic on both CI A and B will be ignored while in Internal Maintenance Loop mode since the receivers must be disabled when this mode is set. Receptions in progress will be terminated when Internal Maintenance Loop mode is set. Therefore, all packets destined for this node will receive no response. The same requirements (1,2,3) stated above for External Maintenance Loop apply to Internal Maintenance Loop mode. The normal arbitration sequence will occur in this mode. The carrier detectors are disabled when in this mode allowing this node to always win arbitration. The FORCE CARRIER DETECT BIT described above may be used L0100 LINK SPECIFICATION Page F-27 to cause arbitration to stop. Acknowledge responses will occur in this mode. However, CRC will not be calculated or checked on the Acknowledge packet. The Transmit Status responses will be set as though a normal transmission had taken place on the CI. Receiver Enables A & B must be disabled while operating in this mode. "Disable..." will take the LINK out of Internal Maintenance Loop mode. INITIALIZE will enable this function. 2 FORCE ARB: "ENABLE..." function will cause the LINK to force a successful arbitration and to transmit immediately upon receiving the "Transmit" function from the PORT. This mode should not be used in an on-line operating system since there is no arbitration. This may result in collisions on the CI. "Disable..." will allow the LINK to perform normal arbitration sequences in response to the "Transmit" function from the PORT. Initialize will cause the LINK to perform normal arbitration sequences. 1 Swap Address: "Enable..." will cause the true and complement address sources to swap, i.e., true node address becomes complement and complement node address becomes true. This will cause an address mismatch if a packet is sent in either of the maintenance loop modes and will result in a no response status. If, while in Internal Maintenance Loop mode, the PORT Processor enables this function and also complements the address in the packet to be transmitted, then address comparison will take place. This has given the PORT Processor the ability to test all node address bits in both polarities. Note that the later case should only be exercised while in Internal Maintenance Loop mode since the LINK will be using an address which was not assigned to it during system configuration. The Arbitration Logic will also use the complement of the Low 4 address bits if this mode is set. "Disable..." sets the node address sources to their normal positions, i.e., true equals true etc. L0100 LINK SPECIFICATION Page F-28 Initialize also sets the node address sources to their normal positions. 0 Receiver A Enable/Disable: "Enable..." activates LINK receiver Path A making the node accessible on CI Path A by remote nodes. "Disable..." will cause this node to ignore all CI traffic on Path A. INITIALIZE will disable receiver path A. F.5.1.9 XMIT STATUS (7:0) (TTL OUTPUT, ASSERTED HIGH) - The XMIT STATUS is used to report the status of a transmit sequence. Some of the bits, indicated below by (*), are valid only when XMIT ATTENTION is asserted. The remaining bits are valid as described below. Transmit Status operate the same while in any of the loop modes as when the LINK is responding to a packet from a remote node. The status bit allocation is as follows: 7 6 5 4 3 2 1 0 +------+-------+------+------+------+------+------+-------+ | XDAT | CDB | CDA | XMIT | ARB | NAK | ACK | XMTR | | PE | | | ABRT | | | | BUSY | +------+-------+------+------+------+------+------+-------+ * * * * FIGURE F5-5 TRANSMIT STATUS *Indicates that the bit is valid only when XMIT ATTENTION is asserted. BIT DESCRIPTION ---------------------------------------------------------------------- 7 Transmit Data Parity Error: This bit is set if a parity error was detected on the data from the XMIT DATA lines during a transmission. A parity error will cause the XMIT ATTENTION flag to be asserted and the transmission to be aborted. This bit is cleared by INITIALIZE and by the "Reset Transmit Status" function. L0100 LINK SPECIFICATION Page F-29 6 CI B Carrier Detect (CDB): This bit indicates the state of the carrier on CI Bus B. There is a delay between the actual transition of the carrier and this status bit due to synchronizer logic. 0 = Carrier B off 1 = Carrier B on 5 CI A Carrier Detect (CDA): This bit indicates the state of the carrier on CI Bus A. There is a delay between the actual transition of the carrier and this status bit due to synchronizer logic. 0 = Carrier A off 1 = Carrier A on 4 Transmission Aborted: This bit is set when a transmission is aborted by the "Transmission Abort" function. Transmission Aborted will cause the XMIT ATTENTION flag to be asserted. Transmission Aborted is cleared by INITIALIZE and by the "Reset Transmit Status" function. 3 Arbitration : This bit is set when the arbitration count has expired after a "Transmit" function. This does not mean that the transmission has taken place since rearbitration takes place under certain conditions (See section 2.1.3). Arbitration is cleared by INITIALIZE and by the "Reset Transmit Status" function. 2 NAK: This is the response from the destination node that both of its receive buffers were unavailable (busy). NAK will cause the XMIT ATTENTION flag to be asserted. NAK is cleared by INITIALIZE and by the "Reset Transmit Status" function. 1 ACK: This bit is set when a valid positive acknowledgement is received in response to a transmitted packet. ACK indicates that the transmitted packet was accepted and stored by the receiving node. ACK will cause the XMIT ATTENTION flag to be asserted. This bit is cleared by INITIALIZE and by the "Reset Transmit Status" function. Note: If XMIT ATTENTION is asserted with Transmit status bits 7,4,2, and 1 clear, the interpretation is that of a no response, i.e., an acknowledge packet timeout has occurred. L0100 LINK SPECIFICATION Page F-30 0 Transmitter Busy: This bit is set by the "Transmit" function and indicates that a Transmit sequence is in progress or that XMIT ATTENTION is asserted. Transmitter Busy in not asserted synchronously to the Port Clock but to the XMIT Clock. It will be asserted within 2 Port Clock cycles of the PORT PROCESSOR issuing the Transmit function. The PORT Processor, therefore, must not monitor this bit during the 2 PORT Clock Cycles immediately following the Transmit function. Transmitter Busy is cleared by INITIALIZE and by the "Reset Transmit Status" function. F.5.1.10 XMIT ATTENTION (TTL OUTPUT, ASSERTED HIGH) - XMIT ATTENTION is asserted by the LINK when there is a response to a "TRANSMIT" function for normal transmissions as well as for transmissions in any of the loop modes. The response may be any of the following: ACK NAK Transmit Data Parity Error Abort Transmission Acknowledge packet timeout (no response) The responses are described in the Transmit Status section (F5.1.9). XMIT ATTENTION is cleared by INITIALIZE and by the Reset Transmit Status function. F.5.1.11 XMIT DATA ENABLE (TTL OUTPUT, ASSERTED HIGH) - XMIT DATA ENABLE is used to enable data from the transmit packet buffers to the LINK. The first data byte must appear on the XMIT DATA lines during the second XMIT CLOCK cycle after the XMIT DATA ENA line is asserted and on all succeeding cycles until the last data byte is transferred. The LINK will ignore the XMIT DATA lines during the first cycle of XMIT DATA ENA. This allows buffers to read the first byte out of their rams. XMIT data enable will be de-asserted one cycle after the signal XMIT BUFFER EMPTY appears. L0100 LINK SPECIFICATION Page F-31 F.5.1.12 XMIT BUFFER EMPTY (TTL INPUT, ASSERTED HIGH) - This line must be asserted by the PORT Processor during the LINK CLOCK cycle in which the last transmit packet buffer data byte is transferred on the XMIT DATA Lines. F.5.1.13 VALID RCVR DATA (TTL OUTPUT, ASSERTED HIGH) - This signal is asserted by the LINK when the first valid data byte is present on the RCVR DATA lines. VALID RCVR DATA will remain asserted throughout the entire packet, as long as there is a valid packet destined for this node being received. Packet buffer loading should be enabled with this signal. If the packet is not for this node, VALID RCVR DATA will be de-asserted. F.5.1.14 RCVR PACKET END (TTL INPUT, ASSERTED HIGH) - RCVR PACKET END must be asserted by the PORT Processor, on the trailing edge of the RCVR CLOCK that takes the last byte of a packet from the RCVR DATA lines. The PORT Processor is responsible for keeping a byte count to determine when a packet has been completed. This signal is used by the LINK to determine when to check the CRC status. F.5.1.15 RCVR BUFFERS FULL (TTL INPUT, ASSERTED HIGH) - RCVR BUFFERS FULL originates at the PORT Processor and indicates to the LINK that there is no receiver buffer available for storing CI packets. The LINK will return a NAK response to all packets received while this signal is asserted. F.5.1.16 VALID RCVR STATUS (TTL OUTPUT, ASSERTED HIGH) - VALID RCVR STATUS is asserted by the LINK to indicate that valid receive status information may be clocked into the PORT Processor receiver status at the end of the current PORT CLOCK cycle. The LINK supplies a CRC STATUS bit and supplies the CI path identification for received packets. L0100 LINK SPECIFICATION Page F-32 F.5.1.17 PACKET LENGTH (TTL OUTPUT, ASSERTED HIGH) - This signal indicates that the byte currently available to the PORT specifies the length of the incoming packet. This signal will remain asserted for two cycles to allow the two bytes containing the high 4 bits (first cycle) and the low 8 bits (second cycle) to be transferred. The PORT Processor must load its packet byte counter accordingly. F.5.1.18 ICCS PATH B (TTL OUTPUT) - ICCS PATH B indicates on which CI path a packet is received. This line is valid only when the Signal VALID RCVR STATUS is asserted and is interpreted as follows: 0 = CI Path A 1 = CI Path B F.5.1.19 CRC STATUS (TTL OUTPUT) - CRC STATUS is to be used to indicate the status of the CRC check on the Information packet just received. CRC STATUS is valid only when the signal VALID RCVR STATUS is asserted and is interpreted as follows: 0 = Bad CRC 1 = CRC OK F.5.1.20 PORT CLOCK (TTL INPUT) - The LINK requires a clock source from the PORT for some of its operation. The PORT CLOCK will be used to load data such as the LINK mode set up information from the PORT Processor into the LINK and to synchronize LINK communication with the PORT. The LINK interface will operate with a minimum cycle period of 150ns as shown in section F5.2 (figure F5-6). The PORT CLOCK may be single stepped but must stop with the clock asserted low. Signals must be asserted by the trailing edge of the PORT CLOCK and strobed by the leading edge unless otherwise specified by the timing diagrams. INITIALIZE must not stop this clock. L0100 LINK SPECIFICATION Page F-33 F.5.1.21 XMIT CLOCK - The LINK will supply a transmit clock source to the PORT to be used to control the transfer of transmit data and control information between the PORT and LINK. The XMIT CLOCK will be a 28ns pulse occurring every 114ns as shown in section F5.2 (figure F5-7). Signals must be asserted by the trailing edge of the XMIT CLOCK and strobed by the leading edge unless otherwise specified by the timing diagrams. F.5.1.22 RCVR CLOCK - The LINK will supply a receiver clock source (RCVR CLOCK) to the PORT Processor to be used to control the transfer of receiver data and control information between the LINK and the PORT Processor. The RCVR CLOCK is a 28ns pulse occurring every 114ns as shown in section F5.2 (figure F5-8). Signals must be asserted by the trailing edge of the RCVR CLOCK and strobed by the leading edge unless otherwise specified by the timing diagrams. F.5.1.23 INITIALIZE (TTL INPUT, ASSERTED HIGH) - This signal from the PORT Processor is used for system initialization. The LINK will provide a pullup resistor on this signal so that, if the PORT Processor is not installed, the LINK will not interfere with either of the CI paths. The LINK will not respond properly to any commands from the PORT Processor for 2 PORT CLOCK cycles following the de-assertion of INITIALIZE. The LINK will appear as non-existent to remote nodes during the INIT sequence. F.5.1.24 RCVR A ENABLE (TTL OUTPUT, ASSERTED High) - RCVR A ENABLE is asserted when the PORT Processor enables the A receiver path. F.5.1.25 RCVR B ENABLE (TTL OUTPUT, ASSERTED High) - RCVR B ENABLE is asserted when the PORT Processor enables the B receiver path. L0100 LINK SPECIFICATION Page F-34 F.5.1.26 XTEND HDR (TTL INPUT, ASSERTED Low) - This input is provided to allow extending the number of bit sync characters in the header from the normal 5 to 15. Note that if XTEND HDR is asserted, the EXTEND ACK TO input (B10) must also be asserted low. F.5.1.27 ALTER DELTA TIME (TTL INPUT, ASSERTED Low) - When asserted, ALTER DELTA TIME increases the basic quiet slot delta time from 7 byte times ( 800ns) to 16 byte times ( 1.83 us). F.5.1.28 DISABLE ARB (TTL INPUT, ASSERTED Low) - Asserting DISABLE ARB will defeat the normal arbitration sequence and allow the LINK to transmit after waiting only one basic quiet slot (Delta Time). F.5.1.29 MODULE INSERTED LOOP - I/O pins B5 and B6 are connected together on the LINK and may be used to pass a "module inserted" sensing signal through. F.5.1.30 EXTEND ACK TO (TTL INPUT, ASSERTED LOW) - Asserting EXTEND ACK TO will increase the timeout period for an ACK return from the normal x 3.66 us to 25.8 us. EXTEND ACK TO must be asserted if XTEND HDR/TR (B7) is asserted. F.5.1.31 DISABLE RCVR CLK (TTL INPUT, ASSERTED Low) - Asserting this input low will disable the normal RCVR CLK source from entering the TTL portion of the LINK and LINK interface. External RCVR clocks may then be inserted via the RCVR TEST CLK input described below. L0100 LINK SPECIFICATION Page F-35 F.5.1.32 RCVR TEST CLK (TTL INPUT, ASSERTED Low) - The RCVR TEST CLK input may be used (in conjunction with Disable RCVR CLK) to produce RCVR CLK pulses from an external source. One RCVR CLK pulse will be produced for every 4 RCVR TEST CLK pulses. Timing relationships are shown in figure F5-9. F.5.1.33 XMIT TEST CLK (Low High) (TTL INPUTS) - _ The XMIT TEST CLK inputs may be used (in conjunction with XMIT CLK DISABLE) to produce XMIT CLK pulses from an external source. XMIT TEST CLK Low and XMIT TEST CLK High must be asserted in phase with each other. Both signals must be High when not in use. Timing relationships are shown in figure F5-10. F.5.1.34 XMIT CLK DISABLE (TTL INPUT, ASSERTED Low) - Assertion of this input will disable the normal XMIT CLK source from entering the TTL portion of the LINK and the LINK interface. External clocks may then be inserted via the XMIT TEST CLK inputs as described above. L0100 LINK SPECIFICATION Page F-36 F.5.2 LINK INTERFACE TIMING Complete timing diagrams for the above interface signals are shown below in figures F5-6 through F5-10. T0 |<-30ns-->|<------------150ns---------->| | min | minimum | | (6) | | _________ _________ | |<--50ns min (6)--->| | PORT CLOCK (1) ___| |___________________| |__ | | | | |<-45ns(2)>| | | | min | | _____________ ____________________ LINK CONTROL, XXXXXXXXXX XXX PORT DATA (7:0), _____________XXXXXXXXXX____________________XXX SELECT, INITIALIZE | | | -->| 35ns |<-- | | | max | --> | 35ns |<-- | | | max | ________________________ /////// \\\\\ XMIT ATTENTION, _____________/////// \\\\\______ XMIT STATUS 3 | | -->| 35ns |<-- | | max | | _____________ ______________________ XXXXXXXX XXX XMIT STATUS(6,5) _____________XXXXXXXX______________________XXX XMIT STATUS (7,4,2,1) SEE NOTE 3 BELOW XMIT STATUS 0 SEE NOTE 4 BELOW RCVR A & B ENABLE SEE NOTE 5 BELOW FIGURE F5-6 PORT CLOCK TIMING L0100 LINK SPECIFICATION Page F-37 1. PORT CLOCK may be single stepped. However, it must stop asserted low. 2. SET UP Time must not be less than the PORT CLOCK pulse width used. 3. XMIT STATUS Bits 7,4,2,1 are valid only when XMIT Attention is set and will be stable at that time. These bits are not asserted via the PORT CLOCK. 4. XMIT STATUS 0 (XMIT Busy) is asserted within 2 port clock cycles of the Port Processor issuing the "Transmit" function to the LINK. XMIT STATUS Bit 0 (XMIT Busy) is cleared by RESET TRANSMIT STATUS function on the Leading edge of the PORT clock. 5. RCVR ENABLE A & B are set by the PORT "ENABLE LINK" function on the leading edge of the PORT Clock and is passed directly back to the PORT. | 6. Minimum PORT CLOCK cycle time (the sum of the high | low portions) is 150 ns. For example, if the high | portion is the minimum 30 ns., the low portion must | be at least 120 ns. | L0100 LINK SPECIFICATION Page F-38 T0 |<-28ns(1)|<-----------114ns(2)-------->| | | | _________ _________ | | | | XMIT CLOCK ___| |___________________| |__ | | | | |<-20ns->| | | | min | | _____________ __________________ XMIT DATA (7:0), XXXXXXXXXXXX XXX XMIT DATA PARITY _____________XXXXXXXXXXXX__________________XXX | | | |<--20ns-->| | | | max | | | _____________ __________________ XXXXXXXXXXXX XXX XMIT DATA ENABLE _____________XXXXXXXXXXXX__________________XXX | | | |<--------45ns-------->| | | min | _______________________________ /////// \\\ XMIT BUFFER EMPTY _____________//////// \\\ FIGURE F5-7 XMIT CLOCK TIMING 1. Pulse width is actually 1/70mhz x2 = 28.572ns 2. Period is actually (1/70mhz)x8 = 114.286ns L0100 LINK SPECIFICATION Page F-39 T0 |<-28ns(1)|<-----------114ns(2)-------->| _________ _________ | | | | RCVR CLOCK ___| |___________________| |__ | | | |<-35ns-->| | | | max | | | _____________ ___________________ RCVR DATA (7:0), XXXXXXXXXXX XXX RCVR DATA PARITY _____________XXXXXXXXXXX___________________XXX ICCS PATH B, VALID RCVR DATA, PACKET LENGTH | | | |<--55ns-->| | | | max | | | _____________ ___________________ XXXXXXXXXXXX XXX CRC STATUS _____________XXXXXXXXXXXX__________________XXX | | |<--45ns-->| | | min | _____________ ___________________ XXXXXXXXXX XXX RCVR BUFFERS FULL _____________XXXXXXXXXX____________________XXX | | | |<-------45ns------->| | | min | _____________ ___________________ XXXXXXXXXX XXX RCVR PACKET END _____________XXXXXXXXXX____________________XXX FIGURE F5-8 RCVR CLOCK TIMING 1. Pulse width is actually 1/70mhz x2 = 28.572ns 2. Period is actually (1/70mhz)x8 = 114.286ns L0100 LINK SPECIFICATION Page F-40 ____ DISABLE RCVR |______________________________________________ CLK L _________ _ _ _ _ _ _ _ _ RCVR TEST CLK L |__| |__| |__| |__| |__| |__| |__| |__| | | | | | | | | | ____ ____ RCVR CLK H (1) _______| |______________| |___________ FIGURE F5-9 RCVR TEST CLK TIMING ______ XMIT CLK DISABLE L |___________________________________ | _____________ __ __ __ XMIT TEST CLK L |__| |__| |__| |___________ | | | | | | | | ________________ __ __ ___________ XMIT TEST CLK H |__| |__| |__| | | | | | | | | ______ __ __ __ ___________ XMIT CLK H ______|______| |__| |__| |__| FIGURE F5-10 XMIT TEST CLK TIMING NOTES: 1. The first RCVR CLK pulse may appear from zero to four Test CLK pulses depending on where the RCVR CLK Shift Register is when the Disable RCVR CLK Line is asserted Low. L0100 LINK SPECIFICATION Page F-41 F.6 MECHANICAL F.6.1 General The LINK module (L0100) is an extended "L" series module implemented with both Schottky and ECL (both 100K and 10K) logic. The ECL consumes approximately one third of the module in the "C" area of the board with the TTL occupying the remainder of the board. F.6.2 Module Keying The LINK module is notched between I/O fingers C42 and C44 to provide keying which will prevent the module from being inserted backwards and more importantly will prevent any other module from being inserted in the backplane slot used by the LINK. A keying plug (DEC PN 1218624-00) must be inserted into the backplane connector between connector pins C42 and C44. F.6.3 Power Requirements The LINK module requires the following voltages at the specified current ratings. +5V +5%,-5% @ 12.5A = 62.5W Worse Case 8.6A = 43.0W Typical -5.2V +5%,-5% @ 6.5A = 33.8w Worse case 5.0A = 26.0w Typical F.6.4 Cooling Normally, ECL designs require 500 LFM of airflow to insure that operating margins for the ECL circuits are maintained. This is especially necessary if the ECL circuits are not contained on the same module or are spread out on a module such that there may be a temperature gradient across the circuits. If the entire ECL circuit is contained in a small area on a single module where the temperature gradient does not exist, then the requirement of 500 LFM does not have to be met. The ECL circuits on the LINK module are contained in a small area and can be adequately cooled with 300 LFM of airflow. L0100 LINK SPECIFICATION Page F-42 F.6.5 LEDS F.6.5.1 INTERNAL MAINTENANCE LOOP (RED) - The LED will be illuminated when the LINK is in Internal Maintenance Loop mode. F.6.5.2 LOCAL CI ACTIVITY (GREEN) - This LED indicates that this node is either transmitting or receiving on either path of the CI. The level of illumination is relative to the amount of CI activity by this node. F.7 APPENDIX A LINK REGISTERS TRANSMIT STATUS REGISTER: 07 06 05 04 03 02 01 00 _________________________________________________________ | XBUF | CDB | CDA | XMIT | ARB | NAK | ACK | XMTR | | PE | | | ABRT | | | | BUSY | | * | | | * | | * | * | | |_______|______|______|______|______|______|______|_______| * Indicates that the bit is valid only when XMIT ATTENTION is asserted. ENABLE/DISABLE LINK CONTROL: 07 06 05 04 03 02 01 00 ________________________________________________________ | | | | | | | | | | RCVR |FORCE |VALID | EXT | INT |FORCE | SWAP | RCVR | | B |CDET | REC |MAINT |MAINT | ARB | ADRS | A | | ENB | | PAR | LOOP | LOOP | | | ENB | |______|______|______|______|______|______|______|______| L0100 LINK SPECIFICATION Page F-43 F.8 APPENDIX B LINK INITIALIZE STATE CONTROL ENABLED DISABLED RCVR B x FORCE CARRIER DETECT x VALID RCVR PARITY x EXTERNAL MAINT. LOOP x INTERNAL MAINT. LOOP x FORCE ARBITRATION x SWAP ADDRESS x RCVR A x L0100 LINK SPECIFICATION Page F-44 F.9 APPENDIX C LINK MODE SEQUENCES ___________________________________________________________________ CURRENT | NEXT | CONTROL FUNCTION |CONTROL| PORT DATA MODE | MODE | | CODE | BITS=1 | | | | INT.MAINT |ON LINE |1)DISABLE INT.MAINT.LOOP | 0111 | 3 LOOP |(PATH A | | | |and/or |2)RCVR ENABLE (A and/or B)| 0110 |0 and/or 7 (NOTE 1) |PATH B) | | | ____________________________________________________________________ INT.MAINT |EXT. |1)DISABLE INT.MAINT.LOOP | 0111 | 3 LOOP |MAINT. | | | |(PATH A| | | | |or B) |2)ENABLE EXT.MAINT.LOOP | | 4 (NOTE 1) | | and | 0110 | | | RCVR ENABLE (A or B) | | 0 or 7 ___________________________________________________________________ EXT.MAINT.|INT. |1)DISABLE EXT.MAINT.LOOP | 0111 | 4 LOOP |MAINT. | and | | (PATH A |LOOP | | | or B) |(NOTE 1) RCVR DISABLE (A or B) | | 0 or 7 | |2)ENABLE INT.MAINT.LOOP | 0110 | 3 ___________________________________________________________________ EXT.MAINT.|ON LINE |1)UNWANTED RCVR DISABLE | | LOOP |(PATH A | (A or B) and | 0111 | 0 or 7 (PATH A |and/or B)| DISABLE EXT.MAINT.LOOP | | 4 or B) | | | | | |2)RCVR ENABLE (A or B) | 0110 | 0 or 7 | | ___________________________________________________________________ ON LINE |INT.MAINT|1)RCVR DISABLE(A and/or B)| 0111 | 0 and/or 7 (PATH A |LOOP | | | or B) | | | | |(NOTE 1) |2)ENABLE INT.MAINT.LOOP | 0110 | 3 ___________________________________________________________________ ON LINE |EXT.MAINT|1)UNWANTED RCVR DISABLE | 0111 | 0 or 7 (PATH A |LOOP | (A or B) | | or B) |(PATH A |2)ENABLE EXT.MAINT LOOP | | |and/or B)| and RCVR ENABLE (A or B)| 0110 | 0 or 7 ___________________________________________________________________ Note 1: INITIALIZE also puts the LINK in INT.MAINT.LOOP mode. APPENDIX G CHANGE HISTORY G.0.1 Revision 1 To 2 1. Add DQI bit. 2. Change and add opcode values: DATREQ0, DATREQ1, DATREQ2, MCFN, MDATREQ. 3. Add effective priority guarantees. 4. Add multiple effective priority Data Read operations. 5. Remove port state change restrictions. 6. Remove ID broadcast on entering Uninitialized/Maintenance state. 7. Update Implementation Functionality table and field. 8. Add SP and received path information to LB and ID. 9. Miscellaneous corrections from review of revision 1.