{"id":1320,"date":"2016-12-05T14:22:50","date_gmt":"2016-12-05T13:22:50","guid":{"rendered":"https:\/\/elkano.org\/blog\/?p=1320"},"modified":"2016-12-08T00:26:56","modified_gmt":"2016-12-07T23:26:56","slug":"removing-multipath-device","status":"publish","type":"post","link":"https:\/\/elkano.org\/blog\/removing-multipath-device\/","title":{"rendered":"Removing multipath device – map in use"},"content":{"rendered":"

I got in trouble when I tried to remove a multipath device from my servers. This device is on top on some lvm volumes that I am not using it any longer. I tried to remove with multipath -f, but it was not possible, it said that the map was in use:<\/p>\n

\r\n~# multipath -f \/dev\/mapper\/2554b454e79496758\r\nDec 05 12:22:31 | 2554b454e79496758: map in use\r\nDec 05 12:22:31 | failed to remove multipath map 2554b454e79496758\r\n<\/pre>\n
You can view how many processes are using this map with the dmsetup tool, see the open count field:<\/p>\n
\r\n~# dmsetup info  \/dev\/mapper\/2554b454e79496758\r\nName:              2554b454e79496758\r\nState:             ACTIVE\r\nRead Ahead:        256\r\nTables present:    LIVE\r\nOpen count:        4\r\nEvent number:      1086846\r\nMajor, minor:      251, 4\r\nNumber of targets: 1\r\nUUID: mpath-2554b454e79496758\r\n<\/pre>\n
First remove any LVM  active devices on this device, ex: if you have  a volume group “vggroup” and a lvm “vol1” on this device remove them:<\/p>\n
\r\n~# lvremove \/dev\/vggroup\/vol1\r\n~# vgremove vggroup\r\n~# pvremove \/dev\/mapper\/2554b454e79496758\r\n<\/pre>\n
and if the device file is still mapped under \/dev remove it:<\/p>\n
\r\n~# dmsetup remove \/dev\/vggroup\/*\r\n<\/pre>\n
At this point there shouldn’t be any processes accessing this device and we should be able to remove it with the command above, but it some cases there are still processes blocked waiting for the device. We can try to find out which processes are with lsoft command filtering by device mayor and minor number:<\/p>\n
\r\n~# lsof | grep "251,4"\r\n<\/pre>\n
In my case there was some vgs<\/strong> processes blocked trying to access the device. We cannot kill these processes, because they are already waiting for a signal from the kernel.<\/p>\n
\r\n~# ps aux | grep sbin\/vgs\r\nroot     1206972  0.0  0.0  32444  4288 ?        D    dic02   0:00 \/sbin\/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free\r\nroot     1213321  0.0  0.0  32444  4308 ?        D    dic02   0:00 \/sbin\/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free\r\nroot     1248170  0.0  0.0  32444  4196 ?        D    dic02   0:00 \/sbin\/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free\r\nroot     2542017  0.0  0.0  32444  4252 ?        D    10:46   0:00 \/sbin\/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free\r\n<\/pre>\n
We can try to suspend the multipath device to force timeout to the processes:<\/p>\n
\r\n~# dmsetup suspend \/dev\/mapper\/2554b454e79496758\r\n~# dmsetup info \/dev\/mapper\/2554b454e79496758\r\nName:              2554b454e79496758\r\nState:             SUSPENDED\r\nRead Ahead:        256\r\nTables present:    LIVE\r\nOpen count:        4\r\nEvent number:      1086846\r\nMajor, minor:      251, 4\r\nNumber of targets: 1\r\nUUID: mpath-2554b454e79496758\r\n<\/pre>\n
And try to clear the device table:<\/p>\n
\r\n~# dmsetup clear  \/dev\/mapper\/2554b454e79496758\r\n~# dmsetup wipe_table  \/dev\/mapper\/2554b454e79496758\r\n<\/pre>\n
We are lucky and finally the device is not in use any longer:<\/p>\n
\r\n~# dmsetup info  \/dev\/mapper\/2554b454e79496758\r\nName:              2554b454e79496758\r\nState:             ACTIVE\r\nRead Ahead:        256\r\nTables present:    LIVE\r\nOpen count:        0\r\nEvent number:      1086846\r\nMajor, minor:      251, 7\r\nNumber of targets: 1\r\nUUID: mpath-2554b454e79496758\r\n<\/pre>\n
Now, we can remove it without problems:<\/p>\n
\r\n~# mutipath -f  \/dev\/mapper\/2554b454e79496758\r\n<\/pre>\n
To avoid multipath rediscover the device again we can blacklist it. Remove device from already discovered devices:<\/p>\n
\r\n~# sed -i '\/2554b454e79496758\/d' \/etc\/multipath\/wwids\r\n<\/pre>\n
In the multipath configuration file add an entry in the blacklist section withe the wwid of the deivce, if the file does not exist create it:
\n\/etc\/multipath.conf<\/strong><\/p>\n
\r\nblacklist {\r\n   wwid 2554b454e79496758\r\n}\r\n<\/pre>\n
And finally reload multipath:<\/p>\n
\r\n~# systemctl reload multipath-tools\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
I got in trouble when I tried to remove a multipath device from my servers. This device is on top on some lvm volumes that I am not using it any longer. I tried to remove with multipath -f, but it was not possible, it said that the map was in use: ~# multipath -f […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[192,117,191],"_links":{"self":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1320"}],"collection":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/comments?post=1320"}],"version-history":[{"count":0,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1320\/revisions"}],"wp:attachment":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/media?parent=1320"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/categories?post=1320"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/tags?post=1320"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}