Skip to main content

Removing multipath device – map in use

I got in trouble when I tried to remove a multipath device from my servers. This device is on top on some lvm volumes that I am not using it any longer. I tried to remove with multipath -f, but it was not possible, it said that the map was in use:

~# multipath -f /dev/mapper/2554b454e79496758
Dec 05 12:22:31 | 2554b454e79496758: map in use
Dec 05 12:22:31 | failed to remove multipath map 2554b454e79496758

You can view how many processes are using this map with the dmsetup tool, see the open count field:

~# dmsetup info  /dev/mapper/2554b454e79496758
Name:              2554b454e79496758
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        4
Event number:      1086846
Major, minor:      251, 4
Number of targets: 1
UUID: mpath-2554b454e79496758

First remove any LVM active devices on this device, ex: if you have a volume group “vggroup” and a lvm “vol1” on this device remove them:

~# lvremove /dev/vggroup/vol1
~# vgremove vggroup
~# pvremove /dev/mapper/2554b454e79496758

and if the device file is still mapped under /dev remove it:

~# dmsetup remove /dev/vggroup/*

At this point there shouldn’t be any processes accessing this device and we should be able to remove it with the command above, but it some cases there are still processes blocked waiting for the device. We can try to find out which processes are with lsoft command filtering by device mayor and minor number:

~# lsof | grep "251,4"

In my case there was some vgs processes blocked trying to access the device. We cannot kill these processes, because they are already waiting for a signal from the kernel.

~# ps aux | grep sbin/vgs
root     1206972  0.0  0.0  32444  4288 ?        D    dic02   0:00 /sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free
root     1213321  0.0  0.0  32444  4308 ?        D    dic02   0:00 /sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free
root     1248170  0.0  0.0  32444  4196 ?        D    dic02   0:00 /sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free
root     2542017  0.0  0.0  32444  4252 ?        D    10:46   0:00 /sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free

We can try to suspend the multipath device to force timeout to the processes:

~# dmsetup suspend /dev/mapper/2554b454e79496758
~# dmsetup info /dev/mapper/2554b454e79496758
Name:              2554b454e79496758
State:             SUSPENDED
Read Ahead:        256
Tables present:    LIVE
Open count:        4
Event number:      1086846
Major, minor:      251, 4
Number of targets: 1
UUID: mpath-2554b454e79496758

And try to clear the device table:

~# dmsetup clear  /dev/mapper/2554b454e79496758
~# dmsetup wipe_table  /dev/mapper/2554b454e79496758

We are lucky and finally the device is not in use any longer:

~# dmsetup info  /dev/mapper/2554b454e79496758
Name:              2554b454e79496758
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      1086846
Major, minor:      251, 7
Number of targets: 1
UUID: mpath-2554b454e79496758

Now, we can remove it without problems:

~# mutipath -f  /dev/mapper/2554b454e79496758

To avoid multipath rediscover the device again we can blacklist it. Remove device from already discovered devices:

~# sed -i '/2554b454e79496758/d' /etc/multipath/wwids

In the multipath configuration file add an entry in the blacklist section withe the wwid of the deivce, if the file does not exist create it:
/etc/multipath.conf

blacklist {
   wwid 2554b454e79496758
}

And finally reload multipath:

~# systemctl reload multipath-tools

Setting up multipath iscsi targets on debian

In this little tutorial I’m going to explain the basic steps to set up an iscsi target on a debian machine.

Install required packages:

# apt-get install open-iscsi multipath-tools

I always use multipath to get path availability.

Configure /etc/iscsi/iscsid.conf file according to your storage server. It is very common the use of chap authentication for logging into a specified target. In order to configure the user and password, edit /etc/iscsi/iscsid.conf file and change these parameters:

node.session.auth.authmethod = CHAP
node.session.auth.username_in = user
node.session.auth.password_in = password

Discover iscsi targets:

# iscsiadm -m discovery -t sendtargets -p <PORTAL_IP>

You are going to get an output like this:

192.168.0.50:3260,1 iscsi-target1
192.168.0.150:3260,1 iscsi-target1

In this case we have two paths to the same target. It can be, because your storage server has two differents controllers, that can be accessed to reach same target or because your storage server has two network cards but only one internal controller. Any way, we have two paths to reach the same target.

Now, that we have the target discovered we can log in on each portal:

# /usr/bin/iscsiadm --mode node --portal 192.168.0.50:3260 --targetname iscsi-target1 --login
# /usr/bin/iscsiadm --mode node --portal 192.168.0.150:3260 --targetname iscsi-target1 --login

To view your current iscsi connections:

# iscsiadm -m session
tcp: [2] 172.16.64.10:3260,1 iscsi-target1
tcp: [3] 172.16.66.10:3260,1 iscsi-target1

or

# /etc/init.d/open-iscsi status
Current active iSCSI sessions:
tcp: [2] 172.16.64.10:3260,1 iscsi-target1
tcp: [3] 172.16.66.10:3260,1 iscsi-target1

To see what luns provides each iscsi connection you can run the following command, with the parameter -r we specify the session number we want to see:

~# iscsiadm -m session -P3 -r3
iSCSI Transport Class version 2.0-870
version 2.0-873
Target: iscsi-target1 (non-flash)
	Current Portal: 192.168.0.50:3260,1
	Persistent Portal: 192.168.0.50:3260,1
		**********
		Interface:
		**********
		Iface Name: default
		Iface Transport: tcp
		Iface Initiatorname: iqn.1993-08.org.debian:01:6a8c80ddf449
		Iface IPaddress: 192.168.0.20
		Iface HWaddress: <empty>
		Iface Netdev: <empty>
		SID: 8
		iSCSI Connection State: LOGGED IN
		iSCSI Session State: LOGGED_IN
		Internal iscsid Session State: NO CHANGE
		*********
		Timeouts:
		*********
		Recovery Timeout: 120
		Target Reset Timeout: 30
		LUN Reset Timeout: 20
		Abort Timeout: 15
		*****
		CHAP:
		*****
		username: chapuser
		password: ********
		username_in: <empty>
		password_in: ********
		************************
		Negotiated iSCSI params:
		************************
		HeaderDigest: None
		DataDigest: None
		MaxRecvDataSegmentLength: 262144
		MaxXmitDataSegmentLength: 1048576
		FirstBurstLength: 65536
		MaxBurstLength: 1048576
		ImmediateData: Yes
		InitialR2T: No
		MaxOutstandingR2T: 1
		************************
		Attached SCSI devices:
		************************
		Host Number: 9	State: running
		scsi9 Channel 00 Id 0 Lun: 0
			Attached scsi disk sda		State: running

If you check the available devs under /dev/disk/by-path you are going to see your new disks provided by iscsi luns. You can see how each lun is detected twice, one for each available path, for example lun0 can be accesed through sda and sdb:

# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 dic 13 23:47 ip-192.168.0.150:3260-iscsi-target1-lun-0 -> ../../sda
lrwxrwxrwx 1 root root 9 dic 13 23:47 ip-192.168.0.150:3260-iscsi-target1-lun-1 -> ../../sdc
lrwxrwxrwx 1 root root 9 dic 13 23:47 ip-192.168.0.150:3260-iscsi-target1-lun-2 -> ../../sde
lrwxrwxrwx 1 root root 9 dic 13 23:47 ip-192.168.0.50:3260-iscsi-target1-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root 9 dic 13 23:47 ip-192.168.0.50:3260-iscsi-target1-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 dic 13 23:47 ip-192.168.0.50:3260-iscsi-target1-lun-2 -> ../../sdf

We are going to use multipath tool to unify the two paths in an unified one. To do so check the properties of the new device:

~# udevadm info -q property /dev/sda
DEVLINKS=/dev/disk/by-id/scsi-2616c304142756451 /dev/disk/by-path/ip-192.168.0.50:3260-iscsi-target1-lun-0
DEVNAME=/dev/sda
DEVPATH=/devices/platform/host8/session3/target8:0:0/8:0:0:0/block/sda
DEVTYPE=disk
ID_BUS=scsi
ID_MODEL=al0ABudQ263abHG3
ID_MODEL_ENC=al0ABudQ263abHG3
ID_PATH=ip-192.168.0.50:3260-iscsi-target1-lun-0
ID_PATH_TAG=ip-172_16_68_50_3260-iscsi-target1-lun-0
ID_REVISION=220
ID_SCSI=1
ID_SCSI_SERIAL=731ab7e9
ID_SERIAL=2616c304142756451
ID_SERIAL_SHORT=616c304142756451
ID_TYPE=disk
ID_VENDOR=SCST_BIO
ID_VENDOR_ENC=SCST_BIO
MAJOR=8
MINOR=128
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=40848863230

And configure multipath accordingly adding this section to the file /etc/multipatch.conf

devices {
   device {
                vendor                          "SCST_FIO|SCST_BIO"
                product                         "*"
                path_selector                   "round-robin 0"
                path_grouping_policy            multibus
                rr_min_io                       100
        }
}

To check the unified paths run the following command:

# multipath -ll
mpath12 (2507a637562517a71) dm-4 SCST_BIO,PzcubQzqZx3JHfm1
size=9.3T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 3:0:0:2 sdf 8:80   active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 2:0:0:2 sde 8:64   active ready  running
mpath11 (26b59586b61536156) dm-2 SCST_BIO,kYXkaSaVkZZalcKP
size=4.9T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 2:0:0:1 sdc 8:32   active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 3:0:0:1 sdd 8:48   active ready  running
mpath10 (24b4f4575564c6b4b) dm-3 SCST_BIO,KOEuVLkKIxommozh
size=4.9T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 2:0:0:0 sda 8:0    active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 3:0:0:0 sdb 8:16   active ready  running

We can see multipath topology, mpath12 is the multipath device for sde and sdf accesed through dm-4. Now, we can partition /dev/mapper/mpath12 as needed and use it as other disk in our system. Multipath has some parameters that can be changed for each device, for example the path policy (see multipath help). In our example the policy is set to round-robin.

if you don’t see any multipath devices with the above command try running multipath -v3 to see verbose messages and see if are showing up messages like this:

Apr 12 20:07:51 | sda: blacklisted, udev property missing

if so, add these lines to multipath.conf and run again multipath -v3 command

blacklist_exceptions {
        property "(ID_WWN|SCSI_IDENT_.*|ID_SERIAL)"
}

And now the paths should be discovered.


Deprecated: Function create_function() is deprecated in /home/elkano.org/blog/wp-content/plugins/seo-ultimate/includes/jlfunctions/arr.php on line 64