MDADM – cannot open /dev/*: Device or resource busy

So, what a faff.. I recently had a RAID card fail in my server, causing 2 of my 3 disks in an array to dissappear, which MDADM didnt take too kindly too. After deducing it was this and not a double disk failure, I got them back online (Confirmed by BIOS) but now came the problem of trying to recover my RAID 5 array – which was a ROYAL faff..

So, here are my notes!

After getting an error around – ‘cannot open /dev/sdc: Device or resource busy’ and ‘mdadm exited with non-zero exit status 2: (udisks-error-quark, 0)’ via the disks-utility on Ubuntu, the following steps worked for me:

1. Stop the array – this will free up the disks from being ‘resource busy’. Bit daft given they arent actually running, but anyway:

mdadm --stop /dev/md127

2. Next, run a forced assemble (be sure to get the disks right!):

root@server:/var/log# mdadm --assemble --force /dev/md100 /dev/sdc /dev/sdd /dev/sdg
mdadm: forcing event count in /dev/sdg(0) from 142 upto 152
mdadm: forcing event count in /dev/sdd(1) from 142 upto 152
mdadm: /dev/md100 has been started with 3 drives.
root@server:/var/log# pvs
  PV         VG        Fmt  Attr PSize   PFree
  /dev/md100 VG001     lvm2 a--    3.64t  1.20t
  /dev/md127 VG002     lvm2 a--    2.73t  1.32t
  /dev/sdf5  server-vg lvm2 a--  465.52g 48.00m

And there you have it, your back up and running. Hopefully this helps if your stuck in the mire of syslog, lsof, etc trying to figure out what is keeping your disk open!