Appendix C. Multipath-usage.txt File for Red Hat Enterprise Linux 4 Update 3

This appendix contains the Multipath-usage.txt file. The file is included with the dm-multipath RPM and provides guidelines for using dm-multipath with Red Hat Cluster Suite for Red Hat Enterprise Linux 4 Update 3:

    RHEL4 U3 Device Mapper Multipath Usage

Device Mapper Multipath (DM-MP) allows nodes to route I/O over
multiple paths to a storage controller. A path refers to the
connection from an HBA port to a storage controller port. As paths
fail and new paths come up, DM-MP reroutes the I/O over the
available paths.

When there are multiple paths to a storage controller, each path
appears as a separate device.  DM-MP creates a new device on top of
those devices. For example, a node with two HBAs attached to a storage
controller with two ports via a single unzoned FC switch sees four
devices: /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd. DM-MP creates a
single device, /dev/mpath/mpath1 that reroutes I/O to those four
underlying devices. 

DM-MP consists of the following components:

o dm-multipath kernel module -- This module reroutes I/O and fails
  over paths and path groups.

o multipath command -- This command configures, lists, and removes
  multipath  devices. The command is run in rc.sysinit during startup,
   and by udev, whenever a block device    is added.

o multipathd daemon -- This daemon monitors paths, checking to see
  if faulty paths have been fixed. As paths come back up, multipathd
  may also initiate path group switches to ensure that the optima
  path group is being used. Also, it is possible to interactively
  modify a multipath device. 

o kpartx command -- This command creates Device Mapper devices for the
  partitions on a device. It is necessary to use this command for DOS-
  based partitions with DM-MP.

DM-MP works with a variety of storage arrays. It
auto-configures the following storage arrays:

o 3PARdata VV
o Compaq HSV110
o Compaq MSA1000
o DDN SAN DataDirector
o FSC CentricStor
o HP HSV110
o HP HSV210
o HP A6189A
o HP Open-
o IBM 3542
o IBM ProFibre 4000R
o SGI TP9100
o SGI TP9300
o SGI TP9400
o SGI TP9500
o STK OPENstroage D280
o SUN StorEdge 3510
o SUN T4

Storage arrays not included in the list may require entries in the
/etc/multipath.conf file.

NOTE: Some storage arrays require special handling of I/O errors and
      path-group switching. Those require separate hardware handler
      kernel modules. 

Terms and Concepts

Hardware Handler:
    A kernel module that performs hardware-specific actions when
    switching path groups and dealing with I/O errors.

    The connection from an HBA port to a storage controller port for
    a LUN. Each path appears as a separate device. Paths can be in
    various states (refer to "Path States").

Path States:
    ready  - Path is able to handle I/O requests.
    shaky  - Path is up, but temporarily not available for normal
    faulty - Path is unable to handle I/O requests.
    ghost  - Path is a passive path, on an active/passive
        NOTE: The shaky and ghost states only exist for certain
              storage arrays.

Path Group:
    A grouping of paths. With DM-MP, only one path group--the
    active path group--receives I/O at any time. Within a path
    group, DM-MP selects which ready path should receive I/O in
    a round robin fashion. Path groups can be in various states
    (refer to "Path Group States").

Path Group States:
    active   - Path group currently receiving I/O requests.
    enabled  - Path groups to try if the active path group has no paths
               in the ready state.
    disabled - Path groups to try if the active path group and all
           enabled path groups have no paths in the active state.
        NOTE: The disabled state only exists for certain storage arrays.

Path Priority:
    Each path can have a priority assigned to it by a callout program.
    Path priorities can be used to group paths by priority and change
    their relative weights for the round robin path selector.

Path Group Priority:
    Each path group has a priority that is equal to the sum of the
    priorities of all the non-faulty paths in the group. By default, the
    multipathd daemon tries to ensure that the path group with the
    highest priority is always in the active state.

    When I/O to a path fails, the dm-multipath module tries to switch to 
    an enabled path group. If there are no enabled path groups with
    any paths in the ready state, dm-multipath tries to switch to a
    disabled path group. If necessary, dm-multipath runs the hardware
    handler for the multipath device.

    At regular intervals, multipathd checks the current priority of
    all path groups. If the current path group is not the highest
    priority path group, multipathd reacts according to the failback
    mode. By default, multipathd immediately switches to the highest
    priority path group. Other options for multipathd are to (a) wait
    for a user-defined length of time (for the path groups to stabilize)
    and then switch or (b) for multipathd to do nothing and wait for
    manual intervention.  Failback can be forced at any time by
    running the multipath command.

Multipath device:
    The multipath device is the device mapper device created by
    dm-multipath. A multipath device can be identified by either
    its WWID or its alias. A multipath device has one or more path
    groups. It also has numerous attributes defined in the
    following file:

    The alias is the name of a multipath device. By default, the
    alias is set to the WWID. However, by setting the
    "user_friendly_names" configuration option, the alias is set to a
    unique name of the form mpath<n>. The alias name can also be
    explicitly set for each multipath device in the configuration file.

    NOTE: While the alias in guaranteed to be unique on a node, it
          is not guaranteed to be the same on all nodes using the
          multipath device. Also, it may change.

    The WWID (World Wide Identifier) is an identifier for the
    multipath device that is guaranteed to be globally unique and
    unchanging. It is determined by the getuid callout program.

Using DM-MP

Initial setup:

1. If it is not already installed. Install the device-mapper-multipath

2. Edit /etc/multipath.conf. For new installations, all devices are
   blacklisted. The default blacklist is listed in the commented out
   section of /etc/multipath.conf.  If you comment out or delete
   the following lines in /etc/multipath.conf, the default blacklist 
   takes effect:

   devnode_blacklist {
           devnode "*"

   For some conditions, that may not be sufficient. If DM-MP is
   multipathing devices that you do not want it to work on, you can
   blacklist the devices by either device name or WWID.

   NOTE: It is safest to blacklist individual devices by WWID, because
         their device names may change. 

   Several other configuration options are detailed later in this
   document. To check the  effects of configuration changes, you can
   do a dry run with the following command:

   # multipath -v2 -d

3. Set the multipathd init script to run at boot time. by issuing
   the commands

   # chkconfig --add multipathd
   # chkconfig multipathd on

4. start dm-multipath (This is only necessary the first time.  On
   reboot, this should happen automatically).

   # multipath
   # /etc/init.d/multipathd start

After initial setup, all access to the multipathed storage should go
through the multipath device.

Configuration File:

Many features of DM-MP are configurable using the configuration file,

For a complete list of all options with descriptions, refer to

The configuration file is divided into four sections: system defaults,
blacklisted devices (devnode_blacklist), per storage array model settings
(devices), and per multipath device settings (multipaths).  The per
multipath device settings are used for the multipath device with a
matching "wwid" value. The per storage array model settings are used
for all multipath devices with matching "vendor" and "product" values.
To determine the attributes of a multipath device, first the per
multipath settings are checked, then the per controller settings, then
the system defaults.  The blacklisted device section is described
setup step 2.

NOTE: There are compiled-in defaults for the "defaults",
      "devnode_blacklist", and "devices" sections of the
      configuration file. To see what these are, refer to the
      following file:


If you are using one of the storage arrays listed in the preceding
text (in "Overview"), you probably do not need to modify the "devices"
subsection. If you are using a simple disk enclosure, the defaults
should work. If you are using a storage array that is not
listed, you may need to create a "devices" subsection for your array.

Explanation of output
When you create, modify, or list a multipath device, you get a
printout of the current device setup. The format is as follows.

For each multipath device:

action_if_any: alias (wwid_if_different_from_alias)

For each path group:

\_ scheduling_policy [path_group_priority_if_known]

For each path:

 \_ host:channel:id:lun devnode major:minor [path_status]

NOTE: The preceding lines for path group and path
      were broken because of print limitations.

The dm status (dm_status_if_known) is like the path status
(path_status), but from the kernel's point of view.  The dm status
has two states: "failed", which is analogous to "faulty",
and "active" which covers all other path states. Occasionally,
the path state and the dm state of a device will temporarily
not agree. 

NOTE: When a multipath device is being created or modified, the
path group status and the dm status are not known.  Also, the
features are not always correct. When a multipath device is being 
isted, the path group priority is not known.

DM-MP cannot be run on either the root or boot device.

Other Sources of information
Configuration file explanation:

Upstream documentation:

mailing list:
Subscribe to this from
The list archives are at

Man pages:
multipath.8, multipathd.8, kpartx.8 mpath_ctl.8