I was replacing a failed drive in an existing array in OMV. After I connected a replacement disk, I went to Multiple Device -> Recover.
However the list of available drives for recovery was blank. The next step I tried was to format the replacement drive under Storage -> Disk -> select the drive -> Wipe button.
Do you really want to wipe the device /dev/XXX ? All data will be lost.
I clicked Confirm.
At that point the GUI threw an error:
Connection Lost
500 – Internal Server Error
Failed to execute command ‘export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; blockdev –rereadpt ‘/dev/XXX’ 2>&1′ with exit code ‘1’: blockdev: ioctl error on BLKRRPART: Device or resource busy
In the notification area, an error was also logged:
Failed to execute command ‘export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; blockdev –rereadpt ‘/dev/XXX’ 2>&1′ with exit code ‘1’: blockdev: ioctl error on BLKRRPART: Device or resource busy
OMV\ExecException: Failed to execute command ‘export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; blockdev –rereadpt ‘/dev/XXX’ 2>&1′ with exit code ‘1’: blockdev: ioctl error on BLKRRPART: Device or resource busy in /usr/share/openmediavault/engined/rpc/diskmgmt.inc:295
Stack trace:
#0 /usr/share/php/openmediavault/rpc/serviceabstract.inc(622): Engined\Rpc\DiskMgmt->Engined\Rpc\{closure}()
#1 /usr/share/openmediavault/engined/rpc/diskmgmt.inc(227): OMV\Rpc\ServiceAbstract->execBgProc()
#2 [internal function]: Engined\Rpc\DiskMgmt->wipe()
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod()
#5 /usr/sbin/omv-engined(535): OMV\Rpc\Rpc::call()
#6 {main}
The reason for these errors is because the replacement disk had an existing superblock on it from another array. When it was inserted into the OMV system, the system interpreted it as another array. You can verify this by pulling up an OMV console and running the command:
cat /proc/mdstat
You should see listed your primary array marked as active, but also a second array marked as inactive and with only one device in it. The device name will match the errors above and it represents the replacement drive.
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
mdXXX : inactive XXX[20](S)
Now if you go back to Multiple Deive -> Recover, the list of drives is no longer blank and the GUI presents you with the replacement drive. However, if you select the drive and click the Save button, the GUI will throw a new error:
500 – Internal Server Error
Failed to execute command ‘export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; mdadm –manage ‘/dev/mdXXX’ –add /dev/XXX 2>&1′ with exit code ‘1’: mdadm: Unknown keyword INACTIVE-ARRAY mdadm: Cannot open /dev/XXX: Device or resource busy
And in the notification area:
Failed to execute command ‘export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; mdadm –manage ‘/dev/mdXXX’ –add /dev/XXX 2>&1′ with exit code ‘1’: mdadm: Unknown keyword INACTIVE-ARRAY
mdadm: Cannot open /dev/XXX: Device or resource busy
OMV\ExecException: Failed to execute command ‘export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; mdadm –manage ‘/dev/mdXXX’ –add /dev/XXX 2>&1′ with exit code ‘1’: mdadm: Unknown keyword INACTIVE-ARRAY
mdadm: Cannot open /dev/XXX: Device or resource busy in /usr/share/php/openmediavault/system/process.inc:247
Stack trace:
#0 /usr/share/openmediavault/engined/rpc/mdmgmt.inc(419): OMV\System\Process->execute()
#1 [internal function]: Engined\Rpc\MdMgmt->add()
#2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod()
#4 /usr/sbin/omv-engined(535): OMV\Rpc\Rpc::call()
#5 {main}
You can see here the superblock on the replacement drive is causing an issue with the scripts due to the presence of an inactive array.
The solution is pretty straightforward:
- Disconnect the replacement drive and wait for it to spin down.
- Reattach it to OMV and wait for it to fully spin up.
- Find it in the list of Disks
- Use the GUI to perform a Wipe, either Quick or Secure.
- The wipe should completely successfully
- Navigate to Multiple Device and click Recover
- Select your replacement Disk
- Click Save to begin the recovery
At this point if you perform another scan of your arrays:
cat /proc/mdstat
You will still see the inactive array in the list. It wont cause any immediate problems to leave it, but you can flush it from the list by rebooting OMV.