Before we talk about hardware and SAN, we still need to start with what a SAN is in principle, where it came from and why.

Once upon a time, at the dawn of the computer era, external devices storage was usually connected directly to servers (DAS - Direct-attached storage) using SCSI, but due to the extremely rapid growth in demand for storage systems this approach has become too ineffective. To change the configuration of storage connected using DAS technology, it was necessary to physically disconnect the server; building fault-tolerant systems was also difficult due to the need to have a physical connection of all servers to all storage devices, and strict restrictions on the maximum distance between devices made such extensions sometimes too complex, and SCSI performance left much to be desired. Changing the approach to connecting external storages allowed us to obtain the following advantages:

• Changing storage configurations does not affect the operation of servers and services
• The distance between devices made it possible to build disaster-proof storage facilities located at remote sites
• Any server can access any storage device on the SAN network

Typically speaking, SAN means networks based on the Fiber Channel protocol, but it is worth noting that the iSCSI protocol allows you to build networks based on IP networks with similar characteristics. Initially, the transition from SCSI to Fiber Channel was driven by the desire to increase connection distance, not throughput. The first version of the protocol, which appeared in 1997, provided a speed of 1Gb/s. Each a new version puncture, constantly "double" the speed of the previous generation. On this moment, the current generation is the 6th generation of the protocol, operating at a speed of 32/128Gb/s.

Fiber channel, as a network protocol, consists of several layers:
FC-0 Physical: which describes the data transmission environment, characteristics of cables, transceivers, HBAs. Physical and electrical characteristics, data transfer speed.
FC-1 Coding: describes how the data will be encoded/decoded (8/10 or 64/66) for transmission
FC-2 Framing and signaling: determines the structure of the transmitted information, monitors data integrity and directly controls the transfer of data. At this level, the data stream is divided into frames and frames are assembled. Defines the rules for data transfer between two ports, classes of service.
FC-3 Common to Service Node: intended for new functionality that can be implemented in the protocol, but at the moment this level is not used
FC-4 Protocol Displays: describes the protocols that FC can use for its operation: SCSI forwarding (SCSI-FCP) or TCP/IP (FC-LE)

Just like in the network protocol, any device in a SAN network has its own unique 64-bit identifier - WWN, set by the manufacturer (analogous to the MAC address of a network device), and each device also receives a 24-bit network address, which is given when connecting the device. The basis of a SAN network is the Factory - a collection of all devices connected to the network. It is worth noting that the factory itself is a single point of failure, so in SAN networks it is normal practice to build several parallel factories (usually two), which are mirror images of each other. This allows you to build fault-tolerant solutions. Although sometimes factories may differ (for example, the connection of only critical systems is duplicated), it all depends on the tasks assigned to them.

The basis of data transmission in FC networks is the frame. The frame contains not only data, but also a header that describes service information from the “from where to where” category, as well as divisions indicating the download and the end of the frame.

Start of Frame - 4 bytes - frame start identifier.
Header - 24 bytes - header. Contains information such as source and destination addresses, frame type, sequence number and serial number frame in it and other service and control information.
Data - 0-2112 bytes - data directly.
CRC - 4 bytes - checksum.
End of Frame - 4 bytes - end of frame identifier.

A sequence is a set of frames that are transmitted from one point to another. For correction possible errors each frame contains a unique sequence counter. Error correction is carried out by a protocol more high level, usually at the FC-4 level. Several sequences make up an exchange. Exchanges are sequences of two-way directions; those. the exchange includes sequences of data transmitted to different directions, although each sequence is transmitted in only one direction. For each exchange, only one sequence can be active at a given time. But, since several exchanges can be active at the same time, different sequences of these exchanges can also be active simultaneously. Each exchange performs one function, for example, implementing the SCSI Read command.

Port types:
Node ports:
N_Port (Node port), device port supporting FC-P2P (Point-to-Point) or FC-SW topology (with a switch).
NL_Port (Node Loop port), a device port that supports FC-AL (arbitrated loop) topology.

Switch/Router Ports (FC-SW topology only):
F_Port (Fabric port), “factory” port (switched fabric - switched communication architecture). Used to connect N_Port type ports to the switch. Does not support loop topology.
FL_Port (Fabric Loop port), a factory port with loop support. Used to connect NL_Port type ports to the switch.
E_Port (Expansion port), expansion port. Used to connect switches. Can only be connected to a port of type E_Port.
EX_port port for connecting an FC router and an FC switch. From the switch side it looks like a regular E_port, but from the router side it looks like an EX_port.
TE_port (Trunking Expansion port (E_port)) was introduced into Fiber Channel by CISCO and is now accepted as a standard. This is an extended ISL or EISL. TE_port provides, in addition to the standard E_port capabilities, routing of multiple VSANs (Virtual SANs). This is implemented using a non-standard Fiber Channel frame (vsan tagging).

General case:
U_Port (Universal port), a port that has not yet been determined in which mode it operates. Usually after initialization it becomes F_Port or E_Port.
L_Port (Loop port), any port of a device that supports the “Loop” topology - NL_port or FL_port.
G_port (Generic port), auto-detection port. Can be automatically detected as a port of type E_Port, N_Port, NL_Port.

SAN consists of:

• Nodes, nodes
  • Disk arrays (data storage systems)
  • Servers
  • Tape Libraries
• Network infrastructure
  • Switches (and routers in complex and distributed systems)
  • Directors - multi-port modular switches with high degree availability.
  • Dedicated switches (standalone switches) are switches with a fixed number of ports.
  • Stackable switches are switches that have additional high-performance ports for connecting independent chassis with each other.
  • Embedded switches are switches built into a blade cage, where there is a separation of ports into functions (ports intended for connecting blade servers cannot be used for interswitch connections).

The heart of the equipment for SAN networks is the ASIC (application specific integrated circuit) - a specialized circuit developed by Brocade to be able to implement most of the functionality of the equipment at the hardware level, which ultimately leads to higher performance and reliability. It is the use of ASIC that makes it possible to maintain such low latency in SAN networks.

It provides a smooth transition between FC-0 and FC-1 while dealing with:

• Encoder/Decoder - Encodes every 8 bits of transmitted data into a 10-bit representation. And decoding the received data back.
• SERDES (Serializer/Deserializer) - Converts a parallel stream of 10-bit chunks of data into a serial stream of 10-bit chunks of data.
• Transceiver - converts electrical impulses into light signals.

ASIC services the ports themselves, while the switch OS runs on separate chips, so when updating the switch firmware you do not have unavailability. At the time of rebooting the OS, the ASIC continues to service current connections, but blocks the establishment of new ones. We will talk more about this in Part 6 of the material.

And since we have SFP in the picture above:
SFPs are separate modules required to connect a cable to a port, but I will talk about them in more detail, their types and differences in the following materials.

I really hope that everything I was able to squeeze out of myself today is digestible and understandable for the reader. After all, without the basics it is very difficult to move on, from the point of view of understanding the processes of the system. For me personally, theory has always been the most difficult part. Not all things can be simply understood as “the grass is green”, some things you just need to accept and remember how they work.

Work on Fiber Channel protocol standards began in 1988. The first standard of the American National Standards Institute (ANSI) was approved in 1994 and called the FC-PH standard (ANSI X3.230:1994). as a transport protocol, was originally developed to overcome the shortcomings of the modern parallel Small Computer System Interface (SCSI) infrastructure, as well as to provide more high speed and great scalability. Although Fiber Channel can carry data from other higher-level protocols, including Intelligent Peripheral Interface (1PI), High-Performance Parallel Interface (HIPPI), IP, and IEEE 802.2, it is currently used primarily for transferring sets SCSI protocol commands and data. The ANSI Tl I standards organization defines Fiber Channel as a protocol that uses a service layer model. The ANSI FC-PH Physical and Signaling Interface specification defines the Fiber Channel Protocol layer service model as shown in Figure 1. 55.3.

Rice. 55.3. Model of level yuzhb

Each level of this model defines a certain set of services, each of which relies on the work of the level below it and serves the level above it. The figure also shows that the Fiber Channel protocol is designed to support multiple upper-layer protocols. However, it is currently primarily used as a transport protocol for transferring SCSI-3 protocol data.

The list below describes all the levels shown in Fig. 55.3.

FC-0. The FC-0 layer defines the physical interface specifications for various transmission media and their associated receivers and transmitters. Various types of copper and fiber optic cables are used as transmission media, as well as the transmission speeds they support. The FC-0 level also defines the types of connectors (plugs and sockets), cable types, media signaling levels and corresponding speeds.

FC-1. At this level, three primary functions are defined. The first of these consists of encoding and decoding data streams. The FC-1 layer defines an 8V/10V encoding scheme that links data symbols and corresponding clock bits. The second function of the FC-1 layer is ordered set management. Ordered sets are unique transfer words that support channel synchronization and control protocols. An example of ordered sets managed by the FC-1 layer are frame delimiters. The third function of the FC-1 layer is protocol management link layer and conditions such as active state, offline status, link failure and link recovery status.

FC-2. The FC-2 layer is responsible for performing functions related to establishing and maintaining communication between two ports. These functions include port login, session communication (the communications element in Fiber Channel SANs) and associated frame ordering, flow control, and error detection and recovery. All these functions will be described in more detail below. The FC-2 layer also defines all data frame and control frame formats used in the Fiber Channel protocol.

FC-3. The FC-3 layer, the Common Services layer, generally provides the basis for future advanced services that are common to multiple ports on a node. Such FC-3 services (some of which have already been developed but are still rarely used) include multicast, compression, and connection request service stacking.

FC-4. The FC-4 layer defines how data from various protocols is converted to Fiber Channel protocol formats. These higher-level protocols include SCSI-3, IP, virtual interface (VI) protocol, and several others. The commands, data, and states of each of these protocols are converted into information blocks transmitted over the Fiber Channel protocol.

The next section discusses the Fiber Channel protocol in more detail.

Fiber Channel protocol topologies

A Fiber Channel SAN can consist of multiple networks with different topologies, such as a fabric topology, a contention loop topology, or a point-to-point network. Each of these topologies defines the associated Fiber Channel port modes that must be supported by the devices within it.

Network topology with point-to-point connections

A point-to-point topology is used primarily to directly connect a station to a disk or tape drive. It provides more bandwidth than parallel connection SCSI, but its use is limited to communication between two devices. This topology is giving way to other, more network-oriented topologies that allow multiple devices to be connected to a network or combined into a structure.

Contention loop topology

A widely used but gradually becoming obsolete topology is the competitive loop topology. An arbitrated loop is a logical loop of up to 126 Fiber Channel devices that compete for data transmission. This loop is typically implemented using a Fiber Channel hub for cable management, resulting in a physical star topology. All devices in such a loop share the available bandwidth. For example, Fiber Channel disk drives are typically grouped into small contention loops within larger disk drive subsystems. The advantage of a competitive loop is the ability to connect multiple devices together.

Private loops

Previous typical topologies are generally referred to as private-loop topologies because all devices in a loop understand only the 8-bit logical address of other devices in that local loop. This addressing scheme prevents devices on a loop from accessing devices on other loops. More modern loop implementations, called public loops, support full 24-bit hierarchical addresses, allowing devices on one loop to communicate with devices on other loops. As a result, when implementing a public loop, multiple concurrent loops can be interconnected using a switched structure.

Switched structure topology

The current topology of choice in Fiber Channel networks is the switched-fabric topology. It combines the best features of previous topologies in terms of number of connections and available bandwidth. The switched fabric topology provides capabilities similar to those found in switched Ethernet/IP networks.

Fiber Channel switches are organized into fabrics and use a 24-bit Fiber Channel ID (FC ID) to route (forward) frames within fabrics from one switch to another. The Fiber Channel protocol fabric can include up to 239 switches, each of which can have ports up to 64 Kbps. Each Fiber Channel switch port can provide up to 1 or 2 Gbps of bandwidth to each connected device. The switched fabric also includes a number of distributed services, such as fabric routing services, naming services, and security services. In Fig. Figure 55.4 shows three of the four topologies described above.

Fiber Channel Port Types

Fiber Channel is a connection-oriented protocol. This means that nodes must establish a communication channel between themselves using the login procedure before exchanging data. The connection is established between logical elements(logical elements), otherwise called ports, which logically communicate with existing physical devices. Ports can be of different types, depending on the physical device and connection topology.

The port type is determined by automatic discussion of which device or topology they belong to. However, the user can limit the possible port modes that will be discussed.

Rice. 55.4. Fiber Channel protocol topologies

The following list lists all the standard and some non-standard port types used in Fiber Channel SANs.

N_Port. The basic port type is N_Port, or host port. All data exchanges in Fiber Channel networks occur between ports of the N_Port or NL_Port types. The N_Port is located on an end device connected to a point-to-point or fabric topology network. Multiple N_Ports can exist on a single physical device.

NL_Port. The node ports found on end devices connected to the contention loop network are called node loop ports or NL_Ports.

F_Port. In a switched fabric, the switch ports directly connected to end devices (N_Ports) are called fabric ports or F_Ports.

FL_Port. The ports in the switched fabric that are connected to the public contention loop are called fabric loop ports or FL_Ports. Using ports of type FL_Port, public concurrent loops can be interconnected to form a switched fabric topology. FL_Ports are connected to looped NL_Ports.

E_Port. When two Fiber Channel switches are connected, the resulting port mode becomes an expansion port or an E_Port. The resulting link between two switches is called an inter-switch link (ISL). E_Ports connect only to similar E_Ports.

B_Port. A bridge port or B_Port is not a generic port. This port extends the Fiber Channel 1SL interswitch link through a port type other than the Fiber Channel port. Ports of type B_Ports are connected only to E_Ports and take part only in basic set channel services. Link extenders on an IP network typically use the B_Port interface to extend a Fiber Channel ISL link onto an IP network.

TE_Port. The special port mode discussed between two Cisco MDS 9000 multilayer switches is the trunk extension port, or TE_Port. Such a port is an add-on or extension of a port of type E_Port; This means that a special tagging mechanism supports the ability of a virtual SAN to create multiple logical structures on top of a common physical structure. TE_Ports can only be connected to similar TE_Ports.

TL_Port. Translational loop ports, or TL_Ports, connect private loops to public loops or switched fabrics. The TL_Port serves as an address proxy server for contention loop devices. The broadcast loopback feature is useful when using legacy Fiber Channel devices that do not support public addressing.

GL_Port. The common loop port or GL_Port is not really the mode being discussed, but rather reflects the port's capability. A port that can negotiate the modes of F_Port, FL_Port, and E Port types is called a GL_Port.

In Fig. Figure 55.5 shows the connection modes and the possibilities of their use.

Fiber Channel Protocol Communication Model

The Fiber Channel protocol uses a connection methodology that requires that a channel be established between the two devices before data can be exchanged. The channel installation process includes several steps. Once the channel is established, data exchange between end devices occurs according to a hierarchical model. Following are the basic steps to establish communication between two devices belonging to different Fiber Channel protocol topologies. For simplicity, some simple steps have been omitted.

Rice. 55.5 Fiber Channel connection modes

For point-to-point connections:

Stage 1: Devices belonging to a point-to-point network behave like two devices belonging to a private contention loop. Such devices must first perform a loop initialization procedure (LIP) to determine whether they belong to a point-to-point network or a concurrent loop network.

Stage 2. One NL Port opens a channel to another NL_Port.

Stage 3. Ports can exchange data.

For a competitive loop:

Step 1: Devices connected to the contention loop must perform a LIP procedure to obtain the Arbitrated Loop Physical Address (AL_PA), i.e. An 8-bit address used for communication with other devices on the network. During this process, NLPort determines whether the loop has a port of type FLPort, the presence of which makes it a public loop. F - If there is no FLPort, then the loop is private.

Step 2: One NL_Port competes for loop access to communicate with another NL Port.

Step 3: Once access is granted, the NL Port opens a connection to another NL Port (or FL Port if communicating with a fabric device).

Step 4: The ports can start communicating with each other.

For a switched structure:

Step 1 Devices connected to the fabric must perform a fabric login procedure (FLOGI) to obtain a Fiber Channel address (FC_ID), which is a 24-bit address used to communicate with other devices. switched network.

Step 2: One N Port must log in to gain access to another N_Port or NL Port with which it will communicate. The port's port login procedure (PLOGI) is performed to establish a channel with the target device.

Stage 3. Ports can exchange data with each other.

Once a communication channel is established between two devices, the Fiber Channel protocol strictly follows a communication model that includes a hierarchy of data structures. At the top of this hierarchy is exchange. Typically, a Fiber Channel protocol exchange is translated into a higher-level protocol command, such as a SCSI-3 protocol read command. Each exchange consists of a series of one-way offers. In turn, each sentence consists of several numbered frames that move from the source to the recipient. Several exchanges can be opened between two devices, each of which has its own set of originator exchange IDs (OXID) and responder exchange IDs (RX_ID). In Fig. Figure 55.6 illustrates this hierarchical relationship and provides an example of a simple SCSI-3 exchange.

Rice. 55.6. Example of SCSI exchange on a Fiber Channel network

Fiber Channel Protocol Addressing

The Fiber Channel protocol has two types of addresses that are used to identify a device or switch port. The first type is a unique, globally assigned address called a world wide name (WWN). The WWN address is assigned by the manufacturer and its global uniqueness is guaranteed. This situation is similar to using the MAC addresses of Ethernet devices.

The second type of address used in the Fiber Channel protocol is a dynamically assigned hierarchical address, which allows a frame to be forwarded from one device to another in a targeted manner. This address is called the Fiber Channel Protocol Identifier (FC_ID). In a Fiber Channel network, the FC_ID is translated into a WWN address so that initiators can use the WWN to contact the device, and that address is then translated into the FC_ID to communicate. The FCJD address assigned to a device depends on the topology type.

Point-to-point topology. Point-to-point connections are actually implemented as a private loop between two devices. Because the devices are on a private loop, they only use the 8-bit AL_PA address. This address ranges from 0x00000 lh to 0x0000Efh.

Competitive loop. Concurrent loops can be implemented as private or public loops, which determines the type of address used. In a private loop topology, the standard AL_PA address is assigned in the same way as in a point-to-point topology. Each private loopback device is assigned an address in the range 0x000001h to 0x0000Efh (however, a maximum of 126 devices can be in one contention group).

Public loop. A public loop contains one or more FL_Ports, which act as gateways into the switched fabric. As such, the address assigned to a public loopback device contains the full 24-bit address. The first octet is the Domain_ID assigned to the switch. The second octet identifies the specific loop on the switch. The third octet is used for the AL_PA address assigned to devices on this loop. The FL_Port always has an AL_PA address equal to OxOOh. Therefore, the actual address range for public concurrent loop devices is represented as Oxddllaa, where dd is the DomainJD identifier of the connected switch from the range OxOlh to OxEFh (1 to 239), // is the loop identifier from the range OxOOh to OxFF, and aa is AL_PA address from the range 0x0lh to OxEFh, where OxOOh is reserved for the FL_Port.

Switched structure. The switched fabric address is based on the FC_ID, which uses the full 24-bit address. Each switch in the fabric is assigned one or more Domain_IDs. This DomainJD can be thought of as a routing prefix that is used by the switch to forward frames to devices connected to other switches. The first octet of the FC_ID is the Domain_ID. It ranges from OxOlh to OxEFh. The second and third FCJD octets of the switched fabric are called AreaJD and Port_ID, respectively. These FCJD components must be locally unique for each switch.

However, when identifying end-fabric devices, they are used differently by different switch manufacturers. Some manufacturers arrange these components based on the physical port to which the end device is connected. Other manufacturers arrange them on a first-come, first-served basis. The standard does not define rules for the placement of these address components. However, the validity range for these addresses is OxOOOOh to OxFFFFh.

15 tab. Section 55.1 summarizes the various FCID models, their address ranges and limitations.

Table 55.1. FCJD ID Models and Limitations

Fiber Channel Protocol Frame Format

The Fiber Channel protocol frame has a standard structure, shown in Fig. 55.7.

Rice. 55.7. Fiber Channel Frame Format

Fiber Channel frame sizes range from 36 to 2148 bytes, depending on the payload size. The following is a description of the main fields of a Fiber Channel frame.

IDLE field IDLE is used for synchronization and word alignment between the transmitter and receiver. IDLE fields indicate readiness for transmission and are constantly transmitted if there is no other data to transmit. IDLE is actually a 4-byte ordered set that is passed from one device to another. According to Fiber Channel standards, each frame transmitted must contain six ordered sets, which often have an IDLE field located in the frame

the last one. Each resulting frame must be filled with at least two ordered sets.

SOF Field The Start of Frame field is a 4-byte ordered set that immediately precedes the content (payload) of the frame. The SOF field also indicates the class of the frame being received.

Frame header The frame header is 24 bytes in size and consists of several control (control) fields. The frame header includes fields such as source FC_ID, destination FC_ID, exchange IDs, routing control, and several other parameters. The complete structure of the frame header in the Fiber Channel protocol is shown in Fig. 55.8.

Data field The data field consists of the actual higher-level protocol data. It can be from 0 to 2112 bytes in length.

CRC (Cyclical Redundancy Check) Cyclic redundancy control. This field is 4 bytes long and is used to check the integrity of the frame. When calculating the value of this field, only the frame header and the Data field are used.

EOF (End of Frame) The end of frame field is a 4-byte ordered set that immediately precedes the contents of the frame. The EOF field also indicates the class of the Fiber Channel frame.

Rice. 55.8 Fiber Channel Frame Header Format

Below is short description all fields of the Fiber Channel frame header.

R_CTL (Routing Control) The Routing Control field contains two 4-bit subfields: a routing type subfield and an information subfield. Routing bits differentiate frames according to function or service, for example, data frames are distinguished from link control frames containing commands or status.

D_ID Fiber Channel ID (FC_ID) Receiver Fiber Channel ID (3 bytes)

CS_CTL (Class-Specific Control) Class-Specific Control field, 1 byte in size, used only in Classes 1 and 4 (Class 1 or Class 4). Fiber Channel protocol classes are discussed in more detail in the next section.

S_ID PC_Fiber Channel source ID (3 bytes).

Type The type field (1 byte) specifies the upper-layer protocol whose data is sent in the payload field.

F_CTL The Frame Control field (3 bytes) contains a number of flags that control the flow of the sequence.

SEQ_ID (Sequence Identifier) ​​The Sequence Identifier field (1 byte) uniquely identifies a given sequence in the context of a single exchange. Each frame is identified by its SEQ_ID.

DF_CTL (Data Field Control) The Data Field Control field (1 byte) indicates the presence of optional headers at the beginning of the Data Field for Device_Data Video_Data frames. The DF_CTL bits have no meaning for Link_Control and Basic Link Service frames.

SEQ_CNT (Sequence Count) The sequence count field (2 bytes) indicates the order in which frames are transmitted in the sequence. It is used by the receiver of the sequence to account for all frames transmitted.

OX_ID (Originator Exchange ID) The originator exchange ID (2 bytes) identifies an individual exchange. This identification is performed by the initiator of the exchange.

RX_ID (Responder Exchange ID) The Exchange Responder ID (2 bytes) identifies an individual exchange. This identification is performed by the exchange responder.

Parameters The Parameters field (4 bytes) depends on the type of the particular frame specified by the R_CTL field.

Fiber Channel Protocol Service Classes

The Fiber Channel protocol defines several classes of service, although in practice only two are typically used. These classes differ from each other in how they implement mechanisms for acknowledgment, flow control, and channel reservation.

Class 1 is a connection-oriented service with delivery confirmation or notification that delivery has not occurred. Before a connection can occur, a channel must be established between the source and destination.

Class 2 is a port-to-port connectionless service with delivery confirmation or notification that delivery failed.

Class 3 is a port-to-port connectionless service with no confirmation of delivery and no notification that delivery has failed. Currently, this class is used most often in practice.

Class 4 is a connection-oriented service that provides a virtual circuit between N_Ports with delivery confirmation or delivery failure notification and guaranteed bandwidth.

Class 6 is a variant of Class 1 for multicast (one-to-many) with delivery confirmation or failure to deliver notification.

In table 55.2 provides an overview of the characteristics of all classes of service. In modern SAN networks, class 3 is most often used. Most structures also support class 2. All other classes are rarely used today.

| Table 55G.2. Protocol Service Classes

Connection-oriented

Bandwidth reservation

Partial

Guaranteed maximum latency

Yes (QoS)

Guaranteed delivery order

Delivery Confirmation

Multiplexing frames on ports

End-to-end flow control

Link Layer Flow Control

Routing in the Fiber Channel Protocol Framework

A fabric topology uses a dynamic routing protocol called Fabric Shortest Path First (FSPF) to route frames in a connected fabric.

The FSPF protocol is primarily based on the Open Shortest Path First (OSPF) IP routing protocol. FSPF is a link-layer protocol that requires all switches to exchange link-layer information with each other, including operational state information and routing metric information for each directly connected ISL. Using this information, which is stored in a local database, each switch runs the well-known Dijkstra algorithm to calculate the shortest path to all other Domain_IDs.

In the case where several routes with equal ratings exist to a certain domain, the load balancing function is performed between these routes. If there is a state change at the link layer, such as a metric change or a link failure, link-state updates (LSUs) are sent and routes are recalculated based on the new information received. In a Fiber Channel network, by setting channel metrics, you can redistribute flows and restore routes.

Flow Control on a Fiber Channel Network

One of the most powerful mechanisms of the Fiber Channel protocol is its flow control capability. Flow control is based on a permission system, which means that a device or port cannot transmit data until it receives credit. The Fiber Channel protocol has two flow control mechanisms: end-to-end and buffer-to-buffer.

End-to-end control is used to control the transmission speed between two end devices and is rarely used. Inter-buffer flow control occurs between all pairs of adjacent device ports along a particular Fiber Channel route and between all pairs of concurrent loop devices.

The concept of buffer credits refers to the number of input buffers available on adjacent connected ports. When executing the login procedure, adjacent devices exchange information about the number of available buffer credits. Buffer credits are replenished when a neighboring device's input buffer becomes free. In this case, a 4-byte R_RDY command is generated and sent to the neighboring device, causing a new credit to appear there. The importance of inter-buffer flow control increases as the distance between adjacent ports increases. As this distance increases, the transit delay or delay also increases. Increasing latency increases the time it takes to receive the return R_RDY message from remote device. If there are not enough credits, a neighboring device may not be able to maintain the transmission speed on the link between these devices, since the transfer will slow down without buffer credits. This scenario is the basis for a planned experiment when expanding a Fiber Channel SAN by adding optical network(SONET/SDH) in order to duplicate data in the event of a failure, and thus restore transmission. The basic rule in this situation is that to support 1 Gbps for every 2 km, one BB_Credit is required. For example, to support a 1 Gbps line on a 100 km optical link, 50 credits must be supported at each end. Any additional delay caused by factors such as data conversion between different protocols, compression or encryption requires an increase in the number of credits. In Fig. Figure 55.9 shows the inter-buffer flow control model as applied to the communication between a station and a disk drive in a fabric.

Distributed Services of Fiber Channel Switched Fabrics

The switched Fiber Channel protocol framework provides a number of distributed services that facilitate management, configuration, and enhance security in Fiber Channel protocol fabrics. This section describes some of these services.

Directory Services

Fiber Channel protocol frameworks support a distributed directory service, often called a name server. Because FC_ID addresses are assigned dynamically in the Fiber Channel protocol, the directory service helps translate a device's static WWN into an FC_ID that is used for routing. When a device logs into a structure, it is automatically registered with the name server along with some of its attributes. This information can subsequently be requested by any end device to find specific device or devices with any specific characteristics.

Zone services

To provide some degree of security within the Fiber Channel protocol design, zone services limit the communications capabilities of certain connected devices. In Fiber Channel, a zone is a logical group of devices that are allowed to communicate and exchange data. Because these devices may be connected to different switches, the zone configuration applies to all switches in the fabric. Zones can be created using a number of identifiers, including FC_IDs, physical switch port indexes, or the most common WWN addresses. There are two types of zones: hard or strict zones (hard zoning) and soft or non-strict zones (soft zoning). In strict zones, the zone configuration provides frame filtering in hardware. This method provides the greatest degree of security. When using non-strict zones, only requests to the directory service are filtered, resulting in only certain devices being visible. Lax zones do not provide complete security because the end device must know the FC_ID of the device at the other end of the link in order to bypass the zone and communicate with the end device.

Management Services

Another very useful service based on the ANSI Til Generic Services Standard (FC-GS and FC-GS-3) is the Distributed Management Service.

It allows you to restore device attributes and configuration information in the structure. This information may include attributes such as software version, device capabilities, device logical names, and management device IP addresses. In addition, it is possible to collect information about connected ports and neighboring devices. Although this information is very useful for management purposes, many manufacturers are slow to take advantage of this opportunity. However, a growing number of manufacturers are planning such support for the next version, called FC-GS-4, which will allow even more configuration information to be recovered from structures.

Status Change Notification Services

In most traditional data networks, information about network failures was transmitted to other network elements such as switches and routers. However, the Fiber Channel protocol extends this service to end devices. Using the State Change Notifcation Service, and later the Registered State Change Notifcation Service (RSCN), end devices can register to receive notifications about fabric events. When an event occurs on the network, whether accidental or intentional, RSCN service messages are generated to notify other devices on the network. By using the RSCN service, devices can respond to network failures much faster than if they were waiting for timers to expire.

Literature:

Internetworking Technology Handbook, 4th Edition. : Per. from English - M.: Publishing house "William", 2005. - 1040 pp.: ill. - Paral. tit. English

ANSI in 1988. Fiber Channel currently competes with both Ethernet and SCSI. (See http://www.prz.tu-berlin.de/docs/html/EANTC/INFOSYS/fibrechannel/detail, http://www.fibrechannel.com/technology/physical.htm and http://www .ancor.com, http://www.iol.unh.edu/training/fc/fc_tutorial.html.) It easily interfaces with local and regional network protocols. Fiber Channel has a unique physical interface system and frame formats that allow this standard to provide easy interfacing with channel protocols IPI(Intelligent Peripheral Interface), SCSI, HIPPI, ATM, IP and 802.2. This allows, for example, to organize a high-speed channel between a computer and a RAID disk storage system. The speed of Fiber Channel networks is n x 100 MB/s for channel lengths of 10 km or more. It is also possible to operate at lower speeds (for example, 12.5 MB/s). Maximum transmission speed today is 4.25 Gbaud. Single-mode or multimode optical fiber can be used as a transport medium. The use of copper coaxial cable and twisted pairs is allowed (at speeds up to 200 MB/s). Fiber Channel has six independent service classes (each class representing a specific communication strategy) that facilitate a wide range of application problems (Table 14.11). Table 14.11.

Class 1	A point-to-point circuit-switched connection between ports of type n_port. The class is suitable for audio and video applications such as video conferencing. Once a connection is established, all available channel bandwidth is used. This guarantees that frames will be received in the same order in which they were sent.
Class 2	Connectionless packet-switched exchange that guarantees data delivery. Since no connection is established, the port can communicate with any number of n_ports simultaneously, receiving and transmitting frames. There can be no guarantee that the frames will be delivered in the same order in which they were transmitted (except in the case of a point-to-point or arbitrated ring connection). Buffer-to-buffer and point-to-point flow control schemes are valid in this class. This class is typical for local networks where data delivery time is not critical
Class 3	Exchange of datagrams without establishing a connection and without guarantee of delivery. Buffer-to-buffer flow control scheme. Applicable for scsi channels
Class 4	Provides allocation of a certain share bandwidth channel with a given quality of service (QoS) value. Works only with fabric topology, where two n_port type ports are connected. In this case, two virtual connections are formed that serve counter data flows. The capacity of these connections may vary. As in class 1, the order of frame delivery is guaranteed here. Simultaneous connections to more than one port of type n_port are allowed. A buffer-to-buffer flow control scheme is used. Each virtual connection controlled independently using the fc_rdy primitive signal
Class 5	Assumes isochronous service
Class 6	Provides multicasting service within a fabric topology. The standard address 0xfffff5 is used. n_port becomes a member of the multicast group by registering at address 0xfffff8

Fiber Channel uses variable length packets (up to 2148 bytes) containing up to 2112 bytes of data. This packet length significantly reduces the overhead associated with forwarding headers (98% efficiency). From this point of view, ATM is in the worst position (83% efficiency of 48 bytes of data in a 53-byte packet). Only FDDI beats Fiber Channel in this parameter (99%). Unlike other local networks that use 6-octet addresses, fiber channel operates with 3-byte addresses that are dynamically allocated during the login operation. Address 0xffffff is reserved for broadcast addressing. Addresses in the range 0xfffff0-0xfffffe are allocated for accessing the fabric structure, the multicasting server and the alias-server. n_port transmits frames from its source_id (s_id) to destination_id (d_id). Before performing the fabric login operation, the port s_id is not defined. In the case of an arbitration ring, the 3-octet addresses al_pa are used, specified when the ring is initialized. To uniquely identify nodes, 64-bit identifier names are used.

The packet format in Fiber Channel networks is shown in Fig. 14.7. It uses 24-bit addresses, which allows up to 16 million objects to be addressed. The network can build connections using a point-to-point scheme; ring architecture with the possibility of arbitration (FC-al) and other schemes (for example, fabric, allowing a large number of independent exchanges simultaneously) are also allowed. The ring connection diagram is shown in Fig. 14.8. Up to 128 nodes can be connected to the ring. The Fiber Channel protocol has 5 layers that define the physical medium, transmission rates, encoding scheme, packet formats, flow control and various types of services. The physical layer (FC-ph, 1993) has three sublayers. FC uses optical fibers with a diameter of 62.5, 50 microns and single mode. To ensure safety, an optional optical connector connection control (OFC) is provided. To do this, the transmitter periodically sends short light pulses to the receiver. If the receiver receives such a pulse, the exchange process continues (Table 14.12).

Table 14.12.

FC-0	Defines the physical characteristics of the interface and environment, including cables, connectors, drivers (ECL, LED, lasers), transmitters, and receivers. Together with FC-1, this layer forms the physical layer
FC-1	Defines the encoding/decoding method (8B/10B) and transmission protocol where the transfer of data and timing information is combined
FC-2	Defines the rules of the signaling protocol, classes of services, topology, segmentation methodology, sets the frame format and describes the transmission of information frames
FC-3	Defines the operation of multiple ports on a single node and provides common services
FC-4	Provides implementation of a set of application commands and higher-level protocols (for example, for SCSI, IPI, IEEE 802, SBCCS, HIPPI, IP, ATM, etc.)

Rice. 14.7.

FC-0 and FC-1 form physical layer, corresponding to the ISO standard model.

The FC standard allows point-to-point, arbitrated ring, and fabric connections (top, middle, and bottom of Figure 14.8). Ring architecture provides the cheapest connection. The arbitration system only allows exchanges between two nodes at a time. It should be noted that the ring structure does not imply the use of a token access scheme. When a device connected to the network is ready to transmit data, it transmits the ARBX primitive signal, where X is the physical address of the device in the arbitration ring (al_pa). If the device receives its own ARBX primitive signal, it gains control of the ring and can begin transmitting. The initiator of the exchange sends an open primitive signal (OPN) and establishes a connection with the recipient. There is no time limit for maintaining control over the ring. If two devices attempt to seize control of the ring at the same time, the X values ​​of the ARB signals are compared. A device with a smaller al_pa gets priority, a device with a larger al_pa is blocked.

Before using the ring, it must be initialized (LIP procedure) so that each port receives its own physical address (al_pa - one octet, which determines the maximum number of ports in the arbitration ring). The initialization procedure begins immediately after turning on the power sending a signal-LIP primitive via port l_port. Then the device that will manage the al_pa selection process is selected.

Before transmission, the octets are converted into 10-bit code sequences called transmission characters (IBM 8B/10B encoding). A logical one corresponds to a higher level of light energy.

Rice. 14.8.

Fiber Channel has two communication modes: buffer-to-buffer and point-to-point. Data transfer occurs only when the receiving party is ready for it. Before sending anything, the parties must perform a login operation. During the login operation, the upper limit on the amount of data transferred (credit) is determined. The value of the credit parameter specifies the number of frames that can be received. After the next frame is transmitted, the credit value is reduced by one. When this variable reaches zero, further transmission is blocked until the receiver has processed one or more frames and is ready to continue receiving. There is a fairly close analogy here with windows in the TCP protocol. The buffer-to-buffer exchange mode involves establishing communication between N_Port and F_Port ports or between two N_Ports. When a connection is established, each party informs the partner how many frames it is ready to accept (the value of the BB_Credit variable). Point-to-point mode is implemented between ports of the N_Port type. The limit on the number of frames that a party can accept is set by the EE_Credit variable. This variable is set to zero upon initialization, incremented by one when a frame is transmitted, and decremented when an ACK Link Control frame is received. An ACK frame may indicate that the port has received and processed one frame, N frames, or an entire sequence of frames. (See also Definitions of Managed Objects for the Fabric Element in Fiber Channel Standard. K. Teow. May 2000, RFC-2837.)

14.2. HIPPI parallel network interface

All information transmission systems considered so far have used exclusively serial code. On different stages During the evolution of telecommunications, preference was given to both parallel and serial methods of data exchange. At the moment, the parallel interface is preserved only for connecting printers. The main advantage sequential circuits transmission of information means saving on cables. Below we describe another standard where a parallel interface is used (the beginning of development dates back to 1987). HIPPI ( High Performance Parallel Interface, see ftp://ftp.network.com ; http://www.cern.ch/hsi/hippi/spec/introduc.htm ; RFC-2067, IP over HIPPI, J. Renwick; RFC-1374, IP and ARP on HIPPI, J. Renwick, ANSI x3t9.3/90-043, 1990 and X3t9.3/91-005) is a high-speed parallel interface rated at 800 Mbit/s (but possible versions with 100, 200 400 and 1600 Mbit/s). The interface was developed in Los Alamos. Later, on the basis of this interface, the ideology of the network was prepared.

The length of the code transmitted per clock cycle in HIPPI is 32 bits (the HIPPI version, designed for a speed of 1600 Mbit/s, has a code length of 64 bits). All shipments are simplex. There is a Superhippi standard (HIPPI -6400, 6.4 GB/s), which describes a data transmission system that is 8 times faster than HIPPI. A serial HIPPI version has been developed for an exchange rate of 1.2 Gbaud for coaxial and fiber optic cables (up to 10 km; version HIPPI -FC - fiber channel). The maximum distance between a station and a switch is 25 m. The maximum distance between stations (station-switch-station) is 50 m. The limit on the number of stations depends on the type of switches used. Switches can communicate with each other (HIPPI-SC), providing information exchange between stations. An example of a HIPPI network topology is presented at

¦ Gigabit interface converters(Gigabit interface converters - GBIC) support serial and parallel translation of transmitted data. GBIC converters provide hot pluggability, i.e. Enabling/disabling GBIC does not affect the operation of other ports. The converters use a 20-bit parallel interface.

Gigabit line modules(Gigabit link modules - GLM) provide functions similar to GBICs, but require the device to be disconnected for installation. On the other hand, they are somewhat cheaper than GBICs.

Media Interface Adapters(Media Interface Adapters) are used to convert signals between copper and optical media and vice versa. Media interface adapters are typically used in HBAs, but can also be used on switches and hubs.

Small form adapters(Small Form Factor Adapters - SFF) allow you to place a larger number of connectors for various interfaces on a board of a certain size.

4.7.4 Interface devices

Interconnection devices connect the components of storage networks. These devices range from low-cost Fiber Channel hubs to expensive, high-performance, managed fabric switches. These devices are discussed in sections 4.7.4.1 through 4.7.4.3.

4.7.4.1 Fiber Channel split ring hubs

FC-AL hubs are a cost-effective option for connecting multiple Fiber Channel nodes (storage devices, servers, computer systems, other hubs and switches) in a ring configuration. Hubs typically provide between 8 and 16 ports. The hub can support different transmission media, such as copper or optical.

Fiber Channel hubs are passive devices, i.e. any other device in the ring cannot detect their presence. Hubs provide the following capabilities:

internal connections, which allow any port to connect to any other port;

the ability to bypass the port to which a malfunctioning device is connected.

The most a big problem in the operation of the ports is due to the fact that at the current time they can only support one Fiber Channel connection. In Fig. Figure 4.7 shows that if port 1 is given control to establish a session with port 8, no other port will be able to transmit data until the established session ends.

Hubs can be connected to Fiber Channel fabric switches (discussed in Section 4.7.4.3) without modification. You can also create a cascade of hubs by connecting two hubs with a cable.

FC-AL hubs dominate the Fiber Channel market, but Fiber Channel fabric switches are becoming increasingly popular as costs drop.

FC-AL hubs are created by companies such as Gadzoox Networks, Emulex and Brocade.

4.7.4.2 Fiber Channel split ring switches

The Most Significant Advantage of FC-AL Switches

in front of hubs is to simultaneously support several connections, while hubs support only one connection at a given time (Fig. 4.8).

Rice. 4.7. Fiber Channel hub

Rice. 4.8. Fiber Channel Switch

The ability to support multiple connections simultaneously comes with its own challenges. Devices connected to the ring switch are not even “aware” of their role. Ring switches are involved in both data transmission and ring addressing. Below is more information on this topic, as well as a look at the role of switches in SANs and how vendors are adding new features to their products.

Ring switches and data transmission

A server that intends to access a storage device must send an arbitration request to control the ring. In a normal hub-based FC-AL ring, each device receives

arbitration packet before it is returned to the server HBA, giving the server control of the ring. The ring switch will send a success response immediately without sending requests to other nodes. At this point, the HBA will send a basic Open packet destined for the storage device port, which will be forwarded by the ring switch. If the port is not transmitting data at this time, there should not be any problems. Otherwise, conflict situations may arise. To solve this problem, the ring switch must provide buffers to temporarily store frames destined for port 7. Some switch vendors provide 32 buffers per port for this purpose.

Ring switches and FC-AL addressing

FC-AL hubs do not play a role in assigning addresses to devices, but only transmit basic address frames around the ring. The same can be said for most switches. However, some devices may insist on receiving a specific address. Some hubs have the ability to control the order of port initialization, which allows a specific port to initialize first, after which the device will be connected to the required port.

Switches and Ring Initialization

The FC-AL protocol requires ring reinitialization when a device is connected, disconnected, or reinitialized. Initializing the ring in this way may disrupt existing communication between the other two devices. Some switch manufacturers provide the ability to selectively screen and forward packets LIP(Loop Initialization Primitives). This operation is intended to minimize problems, reduce ring reinitialization time, and preserve existing data sessions where possible. At the same time, it is necessary to ensure the uniqueness of device addresses.

If all devices participate in ring reinitialization, no duplication of addresses occurs because the devices “protect” their addresses. However, if some devices do not participate in ring reinitialization, it is necessary to prevent the assignment of already allocated addresses to devices that participate in ring reinitialization. Address uniqueness is ensured by additional ring switch logic. When adding a storage device, a LIP packet must be sent to the server, but LIP does not need to be sent to storage devices that never communicate with other storage devices.

Some storage devices can communicate directly with other storage devices, which is used to back up data. Additional Information Copy operations are described in Chapter 5.

Ring switches and fabric architecture

If all devices in the ring are aware of the fabric architecture, the ring switch transmits the necessary frames, such as Fabric Login frames, in the normal manner. If the devices on the ring do not support a fabric architecture, the ring switch must perform quite a lot of work on its own.

Some vendors' ring switches do not support cascading. Additionally, some ring switches require a firmware update before connecting to fabric switches. Some switches must be upgraded to fully support the fabric architecture before connecting them to the SAN.

FC-AL switches are manufactured by companies such as Brocade, McDATA, Gadzoox Networks, Vixel and QLogic.

4.7.4.3 Fiber Channel switches

Fiber Channel Fabric Switches (FC-SW) provide multiple high-speed communication sessions simultaneously with all devices. At the moment, the main switches support speeds of about 1 Gbps, while speeds of 2 Gbps are also no longer a wonder. In general, fabric switches are more expensive per port than hubs and FC-AL switches, but they provide much more functionality.

Fabric architecture switches are more efficient than hubs and FC-AL switches. For example, switches provide the special services described above, provide flow control using basic control packets, and, more importantly, some switches are capable of emulating FC-AL functions to provide backward compatibility with older devices.

Some fabric switches support routing without buffering. The idea is that when a frame header is received, the switch quickly finds the destination header while the frame is still being received. The advantage of this approach is the reduction of delays in frame delivery and the absence of the need to store the contents of the frame in buffer memory. The disadvantage is the immediate transmission of all frames, including damaged ones.

Fabric switches play an important role in the security of Fiber Channel storage networks, which is discussed in more detail in Chapter 7.

4.7.4.4 Comparison of three connection devices

In table Table 4.5 outlines the functionality and differences between the three types of Fiber Channel devices.

4.7.4.5 Bridges and routers

Both in this chapter and throughout the book, terms bridges(bridges) and routers(routers) are not traditional Ethernet bridges and IP routers. IN in this case Bridges and routers refer to devices for Fiber Channel, and not for Layer 2 and Layer 3 network protocols.

Bridges are devices that provide interoperability between Fiber Channel and legacy protocols such as SCSI. Fiber Channel to SCSI bridges allow you to preserve your existing SCSI storage investment. Such bridges support SCSI and Fiber Channel interfaces and convert data from the two protocols. This way, a new server with a Fiber Channel HBA installed can access existing devices SCSI storage. Bridges provide an interface between the parallel SCSI bus and the Fiber Channel interface. Routers have similar capabilities, but for multiple SCSI buses and Fiber Channel interfaces. Storage routers, or smart bridges, provide additional capabilities such as LUN masking and mapping, and support SCSI Extended Copy commands. As data transfer devices, routers use Extended Copy commands to be used by storage libraries, allowing data to be copied between a specified target device and the attached library. This function is also called independent backup(no server).

Examples of router and bridge manufacturers include Crossroads Systems, Chaparral Network Storage, Advanced Digital Information Corporation (ADIC after acquiring Pathlight), and MTI.

4.8 Fiber Channel Control Methods

The previous sections examined the hardware elements that form storage area networks. The operation of a SAN also involves many different programs, mainly designed for management, security, backup and data recovery. Sections 4.8.1 and 4.8.2 cover a number of concepts needed to manage a SAN and ensure data security. Essentially, these concepts provide the "heart" of the SAN.

In a situation where one network contains multiple computers and storage units, it is desirable to limit the influence of some computers (in Fiber Channel terminology, they are called nodes) to certain storage subsystems and certain units within those subsystems. This makes particular sense if the host is running Windows NT, which requires that every detected device be mounted. On the other hand, UNIX has a mount table, which ensures that only devices directly listed in the table are mounted. Even when using UNIX-based hosts, it is advisable to restrict access for security reasons and to reduce the likelihood of data corruption. Access can be restricted by three different types of display and zoning features.

Basic function, implemented within the node; possibly by means of the bus adapter driver software.

Switch function.

Function at the storage subsystem level.

4.8.1 Zoning

Term zoning connected to switches. Zoning allows certain switch ports to connect only to predefined ports. In some cases, zoning may limit the propagation of Fiber Channel control frames; for example, when a new storage device enters the ring, you can limit the propagation of the LIP frame to other devices.

From a functional point of view, zoning allows a computer to connect directly to a specific storage subsystem. The disadvantage of this approach is that all SAN resources are allocated to a single computer, which usually cannot fully utilize them. In particular, zoning does not allow sharing of network access or storage resources.

Rice. 4.9. SAN Zoning

Zoning can be thought of as analogous to configuring an IP port on a firewall-enabled router. Another example would be setting up virtual local area networks (VLANs) on an existing physical LAN. In a VLAN, only some devices “see” each other, even if there are other devices on the same physical LAN. Likewise, zoning limits the capabilities of SAN components (especially initiators) by providing limited data about certain units storage and the ability to access them, even if in the same physical networks data storage and other storage devices are located.

In Fig. 4.9 is demonstrated zoning concept. The SAN has three servers and three storage units. Different shades indicate different zones.

LUN names can be shared software file systems SAN. In this software, one or more servers act as metadata servers. The software is installed on the client computer (the computer that wants to access files on the storage network) and on the metadata server. Metadata provides the client computer with information to map the logical offset in the file to the physical block number of the specified device. This allows the client computer to directly access the file over the SAN, without transferring data through the server. With sufficiently competent organization, the usual permissions for files on

on the client computer will also apply to files stored remotely, which does not require the administrator to take additional steps to configure file sharing permissions.

Multiple zones can be defined, with one node having the ability to enter multiple zones simultaneously; thus, some zones will overlap. Zoning is done in several ways.

Zoning by port number. The advantage of this approach is efficiency. If the device connected to the port is replaced by another device, reconfiguration is not required.

Zoning by WWN name. This is done by specifying WWN names that are part of the same zone. Some WWNs may be listed in multiple zones. The advantage is safety, which, however, comes at the expense of efficiency. Configuration changes may require a server reboot.

Software zoning. It is carried out using a name server (software) that runs on the switch. Software zoning can use port numbers, WWN, or a combination of these parameters. The name server contains a database that stores WWNs, port numbers, and zone IDs.

Hardware zoning. This is done using a routing table that is stored on the switch. Hardware zoning is based on WWN and does not take into account port numbers.

4.8.2 LUN masking

Storage resources can be "partitioned" into multiple nested units (subunits) called logical device number(logical unit number – LUN). The SCSI-2 standard supports up to 64 LUNs per device.

From a functional point of view, LUN masking allows a specific computer to access a specific subunit on a storage system. However, more importantly, this method can prevent access to certain LUNs for some computers or servers. LUN masking allows you to share storage resources and (implicitly) network bandwidth, but the LUN itself cannot be shared. To share the same LUN between multiple computers, you need a file system with additional capabilities, which are described in Chapter 6.

LUN masking is necessary to guarantee data integrity in a SAN environment. Please note: LUN masking is a disk-level security feature, not necessarily a file-level security feature. In the latter case (at the file level), additional software will be required.

LUN masking provides additional functionality such that LUNs can be reassigned to other computers. There are several ways to achieve LUN masking. Each method has its own advantages and disadvantages. Typically, camouflage is performed by means of:

bus adapter hardware;

Fiber Channel switch hardware;

Fiber Channel storage device hardware;

node software.

These options are discussed in sections 4.8.2.1–4.8.2.4.

4.8.2.1 LUN masking using HBA BIOS

The HBA BIOS masks all LUNs that are not listed in the HBA BIOS table. Thus, the node (with the HBA installed) simply does not “notice” the existence of LUNs that it should not “see”.

The disadvantage of this method is the need to correct settings; In addition, the method is not required to be used. Any systems whose HBAs are not configured correctly or do not support this feature may be able to access LUNs that they do not actually want to access. Another problem is the complexity of dynamic control and reconfiguration similar systems.

4.8.2.2 Masking LUNs with Fiber Channel switches

With Fiber Channel switches, zoning is quite simple. An incoming packet is forwarded or not forwarded, depending on the source port and destination port addresses. LUN masking places additional overhead on Fiber Channel switches because the switch must inspect the first 64 bytes of each data packet. This degrades the performance of most Fiber Channel switches, so this feature is typically not implemented.

4.8.2.3 LUN masking by Fiber Channel storage controllers and routers

This method of LUN masking is forced on connected hosts or requires minimal input from the host. LUN masking is implemented by the storage controller or router (using appropriate firmware). These devices are configured to maintain a table of HBA WWNs mapped to the LUNs they (the controller or router) are allowed to access. A significant advantage of this approach is that it creates a configuration that is independent of intermediate switches or hubs.

The disadvantage of this method is the closed implementation of this technology by each vendor and the difficulty of creating a single management console for reconfiguration or even obtaining information about current parameters, although each vendor provides interfaces for managing WWN-LUN bundles.

System vendors supporting this technology include Crossroads Systems, EMC, Dot Hill, and HP (in Storage Works products). Suppliers give their technology implementations their own names; for example, the company Crossroads calls it Access Controls and HP chose the name for StorageWorks products Selective Storage Presentation.

4.8.2.4 LUN masking by host software

LUN masking is in progress. host software, in particular device driver code. The code must run in kernel mode because the main idea is to prevent the operating system from accessing the LUN, and the operating system will do this before the first user-mode application even starts.

Such masking can be performed as a function of the operating system or outside the system. In the absence of a specific solution from Microsoft, some vendors have added the necessary code to the HBA driver. Typically the driver issues the command Report LUNs each device connected to the bus, and before providing the list of LUNs to the Windows NT system, the driver “cuts” the LUNs from the list based on additionally requested data (for example, information system registry Windows NT), thus “hiding” some LUNs from Windows.

The main problem with this method is optional setting, and therefore the need for partial host participation in the LUN masking process. This means that computers that do not have a modified HBA driver do not participate in LUN masking. In addition, there are scaling issues, since in particularly large SANs it is difficult to configure each server and each server bus adapter. In terms of benefits, a LUN can be effectively used by multiple servers.

The described function is implemented in products from Emulex, Dell and JNI.

4.8.2.5 LUN Masking and the Future of Windows NT

At the moment, there is information that Microsoft is working on implementing LUN masking capabilities in the port driver. However, this feature is not available in Windows Server 2003. The advantage of using a port driver is that the driver is always present. port in memory, so the time during which the computer will not take part in LUN masking is significantly reduced. The chance of loading the wrong port driver is much lower than the chance of loading the wrong port and miniport driver. Judging by preliminary forecasts, if the described function is implemented in Windows, the administrator will be able to independently determine and change the list of LUNs visible to the server; however, the list may be changed temporarily. In the latter case, the changes will not be saved after the server is rebooted.

4.9 Interoperability between Fiber Channel devices

The message “Buyer beware!” describes well the state of device communication in the Fiber Channel world.

It can be said that most of the problems in interoperability of FC-AL configurations are related to storage devices, HBAs, FC-AL switches and router vendors. Device vendors do a lot of testing on their products, but while in theory interoperability with other devices should be guaranteed, in practice it requires a lot of additional testing and tweaking to get results. various parameters. It is recommended to use configurations that have been tested by the SAN vendor or vendor.

The biggest problem is the lack of guaranteed compliance with industry standards. Moreover, even compliance with standards also does not ensure 100% interaction.

Suppliers of ready-made solutions, such as IBM, HP and EMC, create laboratories for testing the interaction of various devices and conduct their own certification. To a certain extent, other suppliers do the same. It is recommended to use such certified solutions, which allows you to avoid problems that often arise when adding new devices that are not certified by the supplier.

Although many Fiber Channel SANs offer 1 Gbps speeds, devices that support 2 Gbps speeds have recently become available on the market. New devices mean new problems. The standards followed by manufacturers support 2 Gbps speeds, but devices automatically move to 1 Gbps speeds if other devices on the network are running at that speed. The fact is that storage networks based on Fiber^Channel must operate at the speed of the most slow device online. Thus, even a single device running at 1 Gbps will force the entire SAN to operate at that level of performance.

4.10 Difficulties in practical implementation

Fiber Channel SANs emulate a direct connection from a storage device to a server, even if the device is actually connected through a switch. Thus, in the context of Windows, Fiber Channel devices are accessed using the SCSIPort or Storport drivers described in Chapter 2. Thus, the features of working with direct-attached storage (DAS) are relevant to the SAN.

The new Storport driver model provides a ton of functionality, including I/O optimization and network bandwidth management, but system administrators and IT decision makers should be aware that the Storport driver model is supported exclusively on Windows Server 2003. Those deciding to use the Windows platform should review their storage vendor's plans for migrating to the Storport model. At the same time, it is necessary to pay attention to the implementation of support for these devices based on the Windows 2000 platform, including the details of the device driver implementation. This is especially important to determine the adequacy of the throughput of the legacy SCSIPort driver model if the vendor continues to use it. In addition, you need to find out whether the vendor provides a native SAN architecture, without the SCSIPort driver model, and whether the solution is certified and supported by all stakeholders. Finally, note the vendor's plans to migrate to the Storport driver model for Windows Server 2003.

LUN masking is not currently supported in commercially available Windows versions, and the release of Windows Server 2003 did not change the situation. Before purchasing new software and hardware, find out what technology the vendor uses to implement LUN masking and how suitable it is to work in your environment. Windows environment.

4.11 Summary

Fiber Channel storage networks form a significant part of enterprise storage subsystems. Fiber Channel technology can be deployed in low-cost ring-based configurations or in the increasingly popular switched fabric architecture topology.

operating system Windows Server 2003 supports Fiber Channel devices using the Storport driver provided by the hardware vendor. The vendor may provide a mini SCSI port driver instead, but in this case the benefits of the Storport driver (such as improved performance and error handling) will not be available to users. The Windows 2000 operating system and previous versions support Fiber Channel devices through the SCSIPort minidriver provided by hardware vendors.

Although Windows NT supports LUN masking and zoning technology, there is no basic support for LUN masking in Windows NT. LUN masking in Windows NT can be implemented in a driver from the hardware vendor.

Notes:

ISO - International Organization for Standardization;

OSI - Open System Interconnection (open systems interaction).

IN currently There are several different physical standards, and the fact that only three are used basic type cables (copper, single-mode, and multimode) does not mean there are three types of physical connectors. In addition, these types are also used in other interfaces, such as Gigabit Ethernet.

Dmitry Ganzha
executive editor LAN

Brief general review Fiber Channel technologies.

In short, Fiber Channel is an ultra-high-speed (up to 1 Gbit/s and higher) scheme for full-duplex data transmission with low latency (10-30 μs) over distances of up to 10 km. It can be used equally as an input/output technology and as a local network technology.

In the name of the technology (“fiber channel”, as Fiber Channel could be translated into Russian), both words do not quite correspond to reality. The physical transmission medium can be not only optical fiber, but also coaxial and twisted pair, and the architecture is a mixture of channel and network topologies!

LEVEL MODEL

In fact, Fiber Channel makes up a whole set of standards, many of which are developed independently. They are presented in the form of a five-level model (see Figure 1), and each of these levels, according to the developers, should be implemented in the form of separate hardware components. This model does not have a direct correspondence with the OSI reference model. However, as we will see below, the first and second layers (more precisely, zero and first - FC-0 and FC-1) of Fiber Channel correspond to the OSI physical layer, and the third (second - FC-2) layer corresponds to the MAC sublayer of the OSI data link layer.

The FC-0 layer describes the physical characteristics and possible types of interfaces and transmission media, including cables, connectors, emitters, transmitters, and receivers. FC-1 defines an 8B/10B signal encoding and decoding scheme. FC-2 performs the basic functions of Fiber Channel, including signaling, i.e. establishing a connection between the sender and the recipient; segmentation, assembly and ordering of transmitted frames; flow control using a sliding window scheme, error detection and correction; implementation of service classes. Together, these three layers form the so-called Fiber Channel Physical layer (FC-PH).

FC-3 describes general procedures (though perhaps it would be more accurate to call them special ones) for such special situations as striped data recording to a disk array or multicasting through a video server. FC-4 enables the conversion of various network protocols and applications to be implemented over Fiber Channel. As can be seen from Figure 1, Fiber Channel is capable of supporting a wide variety of network protocols, I/O interfaces, and applications.

TOPOLOGY

Fiber Channel defines three topologies (see Figure 2), namely Point-to-Point, Arbitrated Loop, and Fabric.

The simplest topology is obviously point-to-point. It consists of two Fiber Channel devices and a direct connection between them. One fiber connects the receiver on one device to the transmitter on another device, and the second connects the transmitter to the receiver. (In this article, by fiber we will mean both the optical fiber and the individual twisted pair and strand of coaxial cable.) Both devices can naturally use the full bandwidth of the connection, but they must operate at the same speed.

The most common and at the same time the most complex topology is the arbitration loop. It allows you to connect up to 127 ports in a ring without using a switch. However, unlike the other two topologies, the bandwidth is shared, meaning that only two devices can communicate with each other at any one time. In case of competition for access to the transmission medium between several devices, the device with the lowest address wins arbitration. All devices in the loop must operate at the same speed. The loop can connect to a switch port, but only to one.

For lack of a better Russian-language term, we will call the Fabric topology a switching structure. The switched topology involves the use of a switch(es), but thereby allows you to connect over 16 million devices. Devices with different transmission speeds and over different physical media can connect to the switch.

PORT TYPES

Depending on the type of device, its purpose and the supported topology, ports are divided into several types. The Fiber Channel port on an end device (server, disk array, printer, etc.) is called a Node Port (N_Port). The port on the switch to which the hub port connects is called the Fabric Port (F_Port). If these ports can be connected to an arbitration loop, then they are additionally marked with the letter L from the English loop, i.e. “loop”. Thus, the corresponding ports on the host and switch will be designated as NL_Port and FL_Port.

In addition to F_Port, the switch may also have an expansion port (Expansion Port, E_Port). This port is designed to connect one switch to another. If not only another switch, but also a node can be connected to an expansion port, then such a port is called a generic port (Generic Port, G_Port). Provided that it supports loop arbitration, a generic port may be labeled as GL_Port.

VARIETIES OF EQUIPMENT

In addition to sharing bandwidth, arbitration loops have other disadvantages. In particular, if the adapter on any device fails or there is a break in the connecting cable, the loop becomes completely inoperable. Additionally, when a new device is added, the entire loop must be reinitialized (so that the connected device can obtain an address), which can be quite time-consuming.

These problems can be solved by using Fiber Channel hubs. Additionally, a physical star topology (though logically still a ring) is generally much more convenient in terms of connecting nodes than a ring. Typically hubs have no more than 10 ports. However, this limitation can be easily overcome by cascading hubs. However, as practice shows, the arbitration loop functions optimally when the number of nodes does not exceed 30.

Fault tolerance of hubs to loop breaks is achieved through the use of a port bypass circuit (PBC). PBC allows you to automatically detect the presence of a node and include it in the loop. Similarly, PBC detects a node failure and removes it from the loop (PBC can also be implemented at the disk array internal bus level). The most advanced hubs support remote management and other advanced features.

As with others network technologies, Fiber Channel switches are significantly more expensive devices than Fiber Channel hubs. Unlike hubs, they allow you to provide dedicated bandwidth to a node and, as already mentioned, create topologies with an incomparably larger number of nodes (224). In addition, switches can have ports that support different speeds and transmission media.

A Fiber Channel switch, in fact, combines two types of switches in one device, since it supports both connection-oriented and connectionless switching (relatively speaking, it has the features of both a telephone circuit switch and a local network frame switch). Some switches produced are circuit-only (like Ancor Communications' first commercially available switch), while others are frame-only.

Fiber Channel switches are easy to install and use because they are self-configuring and self-managing. When a node connects to a switch, it registers with the switch and negotiates mutually acceptable parameters with it. When connecting a switch to a switch, they define the configuration and addresses. All operations are carried out automatically. In the case of a universal port (GL_Port), the switch also determines whether it is connected to another switch, to a loop, or to a node.

However, to organize interaction between devices in several loops, it is cheaper to use not a switch, but a switching (or hybrid) hub. The most rarely seen device is a Fiber Channel router (although it might be more accurately called a bridge). It allows you to connect a Fiber Channel network to another transmission medium, such as SCSI or Ethernet.

So far we have talked about, so to speak, structure-forming Fiber Channel devices. However, the most common devices are, naturally, Fiber Channel adapters. Without them, no node would be able to communicate with the Fiber Channel switch fabric. The same adapters can be used to connect both to the local network (other nodes) and to the periphery. This allows, in particular, to reduce the number required slots input/output. Most adapters are available for PCI buses. Gigabit Interface Converters are often used together with adapters. They serve to convert optical signals into electrical signals and vice versa.

SERVICE CLASSES

Switches and nodes can support one or more types of service. No manual configuration is required because the common services supported by switches and nodes are determined during the registration procedure. With services, Fiber Channel can support many different applications. Services are divided into classes. The main ones are Classes 1, 2 and 3. In total, Fiber Channel has 6 or 7 different types service (such uncertainty is due to the fact that Class 5, apparently, will never be defined, and Class Intermix does not have its own number and is often not considered as a separate type of service).

Class 1 corresponds to a connection-oriented service with guaranteed delivery. A connection through a switching structure (a set of switches) is established in a few microseconds. The connection is dedicated so that no other device can communicate with the destination and source ports until the connection is closed. Guaranteed delivery is provided by confirmation of receipt. This class of service is best suited for exchange large volumes data, in particular for backup, graphics applications and communication between supercomputers.

Class 2 represents a connectionless service, but with guaranteed delivery (as in the previous case, using confirmations). Each incoming frame is switched independently of the others, and end ports can transmit or receive frames from several other nodes. Essentially, a switch multiplexes traffic from the host ports, which is why this class of service is sometimes called multiplexed. Frames may not be delivered in the order in which they were sent. This class of service is best suited for transmitting irregular (burst) or interactive traffic similar to local network traffic.

Class 3 is similar to Class 2, except that it does not guarantee frame delivery (acknowledgment of receipt). It allows you to achieve slightly higher actual throughput due to the lack of confirmations. In essence, it is analogous to datagram transmission. This class of service is best suited for multicast and broadcast.

The remaining classes are often not distinguished as independent ones, but are considered subspecies of those listed. The Intermix class is a combination of Class 1 and Class 2 (3). It allows Class 2 or 3 frames to be transmitted when Class 1 frames are not transmitted, and Class 2 or 3 frames do not necessarily have to be addressed to the same recipient as Class 1 frames.

Like Class 1, Class 4 assumes connection establishment, delivery guarantee, fixed delay, and maintaining the original frame order. However, it requires only a portion of the bandwidth to be reserved, meaning the hub port may have other connections. A node can reserve up to 256 Class 4 connections simultaneously, each with its own QoS parameters. This class of service is sometimes called isochronous. It is best suited for transmission digital video and audio.

Like Intermix and Class 4, Class 6 is a variation of Class 1. It is used when a node needs to transmit frames to several nodes at the same time, i.e. in the case of multicast. To do this, the node establishes a dedicated connection with a multicast server, the address of which is fixed (FFFFF5 in hexadecimal format), and it takes on the task of replicating and forwarding frames to all recipients in the multicast group.

CHARACTERISTICS OF FIBER CHANNEL

Concluding the description of Fiber Channel, one cannot fail to mention the main characteristics of this technology. Fiber Channel allows you to support a wide range of speeds - from 133 Kbps to 4.252 Mbps and even more. One of the design goals of Fiber Channel was, in particular, to support HIPPI at 100 MB/s. Therefore, the main data transfer speed - the so-called full speed - is 100 MB/s (other speeds are often indicated as fractions of the main speed - one-eighth, quarter, second, double, quadruple). However, taking into account the overhead of 8B/10B encoding, frame headers, etc., the actual bit rate is 1.063 Mbps. Thus, manufacturers usually give two speeds - “useful”, in bytes per second, and “net”, in bits per second.

Supported distances and transmission rates depend on the type of transmission medium and signal generators used. As mentioned, Fiber Channel can operate over both optical and copper transmission media, with one fiber dedicated to transmitting the signal and the other for receiving. In the case of optics, this can be 50/125 µm and 62.5/125 µm multimode fiber and single-mode fiber with SC connectors. In the case of copper, this could be coaxial cable, specifically video cable with TNC (receiver) and BNC (transmitter) connectors, as well as shielded twisted pair cable with DB-9 connectors.

The highest speeds (up to 4 Gbit/s) and distances (up to 10 km) are achieved when using single-mode optical fiber and low-frequency lasers. Multimode fiber is capable of supporting the same speeds but over much shorter distances, particularly 100 MB/s over distances of up to 500 m in the case of 50/125 µm multimode fiber with a high-frequency laser. The copper transmission medium allows you to maintain speeds no higher than the main one over short distances (100 m or less).

INTERMEDIATE FINISH

Although not as complex as ATM, Fiber Channel technology is covered by several standards (some even believe that expanding its capabilities and, as a result, making it more complex could negatively affect its prospects). Obviously, in one small introductory article you can only give general description technology, which is what we tried to do. However, many important details had to be omitted, in particular, how arbitration and flow control are carried out, what Fiber Channel frames and 8B/10B encoding are, etc. Therefore, we intend to continue covering this topic in the next issue.