So, what a faff.. I recently had a RAID card fail in my server, causing 2 of my 3 disks in an array to dissappear, which MDADM didnt take too kindly too. After deducing it was this and not a double disk failure, I got them back online (Confirmed by BIOS) but now came the problem of trying to recover my RAID 5 array – which was a ROYAL faff..
So, here are my notes!
After getting an error around – ‘cannot open /dev/sdc: Device or resource busy’ and ‘mdadm exited with non-zero exit status 2: (udisks-error-quark, 0)’ via the disks-utility on Ubuntu, the following steps worked for me:
1. Stop the array – this will free up the disks from being ‘resource busy’. Bit daft given they arent actually running, but anyway:
mdadm --stop /dev/md127
2. Next, run a forced assemble (be sure to get the disks right!):
root@server:/var/log# mdadm --assemble --force /dev/md100 /dev/sdc /dev/sdd /dev/sdg mdadm: forcing event count in /dev/sdg(0) from 142 upto 152 mdadm: forcing event count in /dev/sdd(1) from 142 upto 152 mdadm: /dev/md100 has been started with 3 drives. root@server:/var/log# pvs PV VG Fmt Attr PSize PFree /dev/md100 VG001 lvm2 a-- 3.64t 1.20t /dev/md127 VG002 lvm2 a-- 2.73t 1.32t /dev/sdf5 server-vg lvm2 a-- 465.52g 48.00m
And there you have it, your back up and running. Hopefully this helps if your stuck in the mire of syslog, lsof, etc trying to figure out what is keeping your disk open!