1. Introduction
The following article explains how to resolve and provide hints on a possible resolutions for device mapper that cannot be removed because open count reference is still 1 and it seems no other device or process is still holding a reference to it.
2. Steps to resolve dmsetup remove failed device or resource busy
-
First, we start with an initial status where we want to remove/retrire a disk (maybe an OSD replace, disk failed, etc), and dmsetup info shows something like this:
>> dmsetup info ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6
Name: ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6
State: ACTIVE (DEFERRED REMOVE)
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 9
Number of targets: 1
UUID: LVM-ssXcRqn4X2g58nR4kGWMuTzlQgGJ5yyOhHrPPc5PVs8ac22eVfY1cJEV3z8Ko7wX -
See how ACTIVE (DFERRED REMOVE) is signaling that we have already attempted to remove, force remove and force deferred remove:
>> dmsetup remove ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6
device-mapper: remove ioctl on ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6 failed: Device or resource busy
Command failed.>> dmsetup remove --force ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6
device-mapper: remove ioctl on ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6 failed: Device or resource busy
Command failed.>> dmsetup remove --deferred ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6
device-mapper: remove ioctl on ceph–0e626bfd–bc45–4768–bb5a–ec9454cfc15e-osd–block–85d0d531–f4b3–4bc7–acbc–1501b1790de6 failed: Device or resource busy
Command failed. -
In this sitaution you can try to use lsof to find device/process holding reference to this device mapper:
>> lsof -n 2>&1 | grep ‘253,8’ | head
-
If no useful information is found there, you can also use the following couple of commands to get extended information:
>> dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
fXGZ6r-8935-ebAJ-cdGA-HAlp-QnNO-L50Rj7 253 15 L–w 24 1 0 CRYPT-LUKS2-665a03ebd7b342d3a77870f061730418-fXGZ6r-8935-ebAJ-cdGA-HAlp-QnNO-L50Rj7
ceph–96e1875d–3fb2–49d6–bc55–0c5b1b244cd7-osd–block–6b945141–5181–4059–9e9e–ac86daf9fb05 253 2 L–w 1 1 0 LVM-kKvVvnt7fe8F3OJyKKonTio3IgSmELfRheLUQ9KhsnmuP49jpw3P09dOwY208ayn
storage_group-storage_logic 253 0 L–w 1 1 0 LVM-Jbcql4dfnsc1jHgJTcgdUCzMZQizAov5UeVL0dioE6pfZ43gShNTdQU07OWohkZV
ceph–9d0c6ad6–787a–4d69–b9a6–718087cdb882-osd–block–5919b444–d82a–4955–999f–dc6861eb6923 253 7 L–w 1 1 0 LVM-938QteQpmbmq0C80Cv0J55PE0C1hfKyKTefmQmIGz9zZlhcRb07eBnurRY3cXMNU
ceph–5fb9d482–5d28–4ef5–bbe2–3c2929a28279-osd–block–945bb07e–c7ab–4093–b943–a56d1ba40e36 253 16 L–w 1 1 0 LVM-mCo3nz9XBDo051HzF4PemH2FlVc1jcgvulBsI5ys4yTRC0DKtoeUusQ80obSuFYu
JNq1fC-cZLj-5AvD-KmSN-blx9-XDAE-Ue0mxZ 253 12 L–w 24 1 0 CRYPT-LUKS2-1a9b285d490e45e18287afb8565b7205-JNq1fC-cZLj-5AvD-KmSN-blx9-XDAE-Ue0mxZ
ceph–486a518f–6f84–4f39–8da7–024cfeb3684a-osd–block–2802174b–88a2–447f–b05d–a65578f0bfc7 253 4 L–w 24 1 0 LVM-GsInNjdPjOLluYDFHXl6PeU0mbSlfEbKtSRHCOHNprVCqpfScr8waiWXihQkQ1cQ
ceph–23aec3fb–e30c–4818–9e17–0f0845076127-osd–block–d5a09a18–0aec–4812–82a2–edb8eccddf25 253 3 L–w 1 1 0 LVM-hAOBr7EGzPwyeZvRQLT6BvaBBXSD42hWfXGZ6r8935ebAJcdGAHAlpQnNOL50Rj7
4lfA8B-wXvl-MoEB-N8qx-zFe7-uGSe-7gBKb7 253 13 L–w 24 1 0 CRYPT-LUKS2-cddab57bedb44da6bc5b7fbe703eee66-4lfA8B-wXvl-MoEB-N8qx-zFe7-uGSe-7gBKb7
TefmQm-IGz9-zZlh-cRb0-7eBn-urRY-3cXMNU 253 14 L–w 24 1 0 CRYPT-LUKS2-68cd68db85f24672b0926af23ad8bcf5-TefmQm-IGz9-zZlh-cRb0-7eBn-urRY-3cXMNU
heLUQ9-Khsn-muP4-9jpw-3P09-dOwY-208ayn 253 10 L–w 24 1 0 CRYPT-LUKS2-9f4cb9f713084b748304b7f4324b469e-heLUQ9-Khsn-muP4-9jpw-3P09-dOwY-208ayn
ulBsI5-ys4y-TRC0-DKto-eUus-Q80o-bSuFYu 253 17 L–w 24 1 0 CRYPT-LUKS2-80955e714fe1453c85473a49308d60e8-ulBsI5-ys4y-TRC0-DKto-eUus-Q80o-bSuFYu
ceph–a05f1d5a–5b74–46dc–ad11–5038b2956150-osd–block–ad5c7379–8c07–40b4–bc00–c7a3cc499833 253 5 L–w 24 1 0 LVM-UXTYox6MnZMS1lJ1jspGEB0UagT5CarYwdF0xp8Z6SouNZj4oediGNCkFtxozGsl
ceph–53fb00c9–baa2–4684–a818–6811dc28e1a9-osd–block–a301683e–93ef–47c2–b6ce–1e3c858b255b 253 9 L–w 1 1 0 LVM-AkHrCWzxaVEL7eRnakW9QEPtW88rPRaiJNq1fCcZLj5AvDKmSNblx9XDAEUe0mxZ
ceph–fad14131–c90a–40e2–8e44–b9e4bc76090d-osd–block–713858f4–6589–4b39–a3c5–0bbdac956b91 253 8 L–w 24 1 0 LVM-Zy7p8Nx8RfUH6Baar2KPwvfondpg4QLykQAPcv9H08X25uzkv68IWkMNJxukacrv
ceph–fad883b9–ef30–414c–9263–5a120a1c2d41-osd–block–f7179597–0e1e–4078–87b5–cb61d73cde87 253 1 L–w 1 1 0 LVM-mtV5tt6bMNfGsmn2VSxRjqtESGvAzeea4lfA8BwXvlMoEBN8qxzFe7uGSe7gBKb7
iRrC1j-3bMT-ldFj-qtUu-Q1Ak-1xOu-gG52IQ 253 11 L–w 0 1 0 CRYPT-LUKS2-6bb655efef664641a70dd459d231391a-iRrC1j-3bMT-ldFj-qtUu-Q1Ak-1xOu-gG52IQ
ceph–2750f9af–90cd–4c93–93f4–1acc3d579c25-osd–block–08869209–1c51–4c06–85b8–872d5bd0187d 253 6 L–w 1 1 0 LVM-Gt5hwxd6JzNokrrxr466MZ0BMd8itp1PiRrC1j3bMTldFjqtUuQ1Ak1xOugG52IQ…and also, the following which provides information about possible parent device/process holding reference to our selected device for removal:
>> dmsetup ls --tree
fXGZ6r-8935-ebAJ-cdGA-HAlp-QnNO-L50Rj7 (253:15)
└─ceph–23aec3fb–e30c–4818–9e17–0f0845076127-osd–block–d5a09a18–0aec–4812–82a2–edb8eccddf25 (253:3)
└─ (8:145)
storage_group-storage_logic (253:0)
└─ (8:7)
JNq1fC-cZLj-5AvD-KmSN-blx9-XDAE-Ue0mxZ (253:12)
└─ceph–53fb00c9–baa2–4684–a818–6811dc28e1a9-osd–block–a301683e–93ef–47c2–b6ce–1e3c858b255b (253:9)
└─ (8:81)
ceph–486a518f–6f84–4f39–8da7–024cfeb3684a-osd–block–2802174b–88a2–447f–b05d–a65578f0bfc7 (253:4)
└─ (8:49)
4lfA8B-wXvl-MoEB-N8qx-zFe7-uGSe-7gBKb7 (253:13)
└─ceph–fad883b9–ef30–414c–9263–5a120a1c2d41-osd–block–f7179597–0e1e–4078–87b5–cb61d73cde87 (253:1)
└─ (8:129)
TefmQm-IGz9-zZlh-cRb0-7eBn-urRY-3cXMNU (253:14)
└─ceph–9d0c6ad6–787a–4d69–b9a6–718087cdb882-osd–block–5919b444–d82a–4955–999f–dc6861eb6923 (253:7)
└─ (8:17)
heLUQ9-Khsn-muP4-9jpw-3P09-dOwY-208ayn (253:10)
└─ceph–96e1875d–3fb2–49d6–bc55–0c5b1b244cd7-osd–block–6b945141–5181–4059–9e9e–ac86daf9fb05 (253:2)
└─ (8:113)
ulBsI5-ys4y-TRC0-DKto-eUus-Q80o-bSuFYu (253:17)
└─ceph–5fb9d482–5d28–4ef5–bbe2–3c2929a28279-osd–block–945bb07e–c7ab–4093–b943–a56d1ba40e36 (253:16)
└─ (8:161)
ceph–a05f1d5a–5b74–46dc–ad11–5038b2956150-osd–block–ad5c7379–8c07–40b4–bc00–c7a3cc499833 (253:5)
└─ (8:33)
ceph–fad14131–c90a–40e2–8e44–b9e4bc76090d-osd–block–713858f4–6589–4b39–a3c5–0bbdac956b91 (253:8)
└─ (8:97)
iRrC1j-3bMT-ldFj-qtUu-Q1Ak-1xOu-gG52IQ (253:11)
└─ceph–2750f9af–90cd–4c93–93f4–1acc3d579c25-osd–block–08869209–1c51–4c06–85b8–872d5bd0187d (253:6)
└─ (8:65) -
Review all this information, especially (dmsetup ls --tree) to ensure there are no parent top device holding reference. In our case, see above explain (bold) where there is a device (JNq1fC-cZLj-5AvD-KmSN-blx9-XDAE-Ue0mxZ (253:12)) holding a parent reference to (ceph–53fb00c9–baa2–4684–a818–6811dc28e1a9-osd–block–a301683e–93ef–47c2–b6ce–1e3c858b255b (253:9)).
-
In our case, we had to remove that parent holding device (due to CryptLuks installed) using regular “dmsetup remove” on it. After that, pending device to removed dissapeared without error.