At first, we must create partitions on disks with the SAME size in blocks:
fdisk /dev/sdc
> n (new partition)
> p (primary type of partition)
> 1 (partition number)
> 2048 (first sector: default)
> 1953525167 (last sector: default)
> t (change partition type) - selected partition nb. 1
> fd (set it to Linux raid autodetect)
> w (write end exit)
fdisk /dev/sdd
> n (new partition)
> p (primary type of partition)
> 1 (partition number)
> 2048 (first sector: default)
> 1953525167 (last sector: default)
> t (change partition type) - selected partition nb. 1
> fd (set it to Linux raid autodetect)
> w (write end exit)
fdisk -l /dev/sdc
Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Device Boot Start End Sectors Size Id Type
/dev/sdc1 2048 1953525167 1953523120 931.5G fd Linux raid autodetect
root@cl3-amd-node2:~# fdisk -l /dev/sdd
Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Device Boot Start End Sectors Size Id Type
/dev/sdd1 2048 1953525167 1953523120 931.5G fd Linux raid autodetect
Now, we can create raid using a mdadm. Parameter –level=1 defines raid1.
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1
We can watch the progress of building the raid:
cat /proc/mdstat
md1 : active raid1 sdd1[1] sdc1[0]
976630464 blocks super 1.2 [2/2] [UU]
[>....................] resync = 1.8% (17759616/976630464) finish=110.0min speed=145255K/sec
bitmap: 8/8 pages [32KB], 65536KB chunk
Now we can add a spare disk:
fdisk /dev/sde
> n (new partition)
> p (primary type of partition)
> 1 (partition number)
> 2048 (first sector: default)
> 1953525167 (last sector: default)
> t (change partition type) - selected partition nb. 1
> fd (set it to Linux raid autodetect)
> w (write end exit)
mdadm --add-spare /dev/md1 /dev/sde1
And now we can see detail of the raid:
mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Tue Mar 14 11:56:28 2017
Raid Level : raid1
Array Size : 976630464 (931.39 GiB 1000.07 GB)
Used Dev Size : 976630464 (931.39 GiB 1000.07 GB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Mar 14 12:00:49 2017
State : clean, resyncing
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Resync Status : 3% complete
Name : cl3-amd-node2:1 (local to host cl3-amd-node2)
UUID : 919632d4:74908819:4f43bba3:33b89328
Events : 52
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 65 - spare /dev/sde1
And we can it see here too:
cat /proc/mdstat
md1 : active raid1 sde1[2](S) sdd1[1] sdc1[0]
976630464 blocks super 1.2 [2/2] [UU]
[=>...................] resync = 7.5% (73929920/976630464) finish=103.3min speed=145533K/sec
bitmap: 8/8 pages [32KB], 65536KB chunk
unused devices: <none>
After reboot, if we can not see our md1 device like this:
root@cl3-amd-node2:/etc/drbd.d# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[0] sde1[2](S) sdb1[1]
976629760 blocks super 1.2 [2/2] [UU]
bitmap: 1/8 pages [4KB], 65536KB chunk
unused devices: <none>
We can recreate (assemble) it with this command without resync:
mdadm --assemble /dev/md1 /dev/sdc1 /dev/sdd1
mdadm: /dev/md1 has been started with 2 drives.
root@cl3-amd-node2:/etc/drbd.d# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdc1[0] sdd1[1]
976630464 blocks super 1.2 [2/2] [UU]
bitmap: 0/8 pages [0KB], 65536KB chunk
md0 : active raid1 sda1[0] sde1[2](S) sdb1[1]
976629760 blocks super 1.2 [2/2] [UU]
bitmap: 1/8 pages [4KB], 65536KB chunk
unused devices: <none>
If we want to automatically start this raid with the boot, we must add this array to mdadm.conf. At first, we scan for our arrays and add it to /etc/mdadm/mdadm.conf.
root@cl3-amd-node2:/etc/drbd.d# mdadm --examine --scan
...
ARRAY /dev/md/1 metadata=1.2 UUID=94e2df50:43dbed78:b3075927:401a9b65 name=cl3-amd-node2:1
ARRAY /dev/md/0 metadata=1.2 UUID=2c29b20a:0f2d8abf:c2c9e150:070adaba name=cl3-amd-node2:0
spares=1
cat /etc/mdadm/mdadm.conf
...
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=2c29b20a:0f2d8abf:c2c9e150:070adaba name=cl3-amd-node2:0
spares=1
echo "ARRAY /dev/md/1 metadata=1.2 UUID=94e2df50:43dbed78:b3075927:401a9b65 name=cl3-amd-node2:1" >> /etc/mdadm/mdadm.conf
And the last step is update the initramfs to update mdadm.conf in it:
update-initramfs -u
If there is a need to replace bad missing disk, we must create a partition on new disk with the same space.
fdisk -l /dev/sdb
Disk /dev/sdb: 233.8 GiB, 251000193024 bytes, 490234752 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 488397167 488395120 232.9G fd Linux raid autodetect
Degraded array:
mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Fri May 27 09:08:25 2016
Raid Level : raid5
Array Size : 488132608 (465.52 GiB 499.85 GB)
Used Dev Size : 244066304 (232.76 GiB 249.92 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Apr 20 11:33:11 2017
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : cl2-sm-node3:1 (local to host cl2-sm-node3)
UUID : 827b1c8a:5a1a1e7c:1bb5624f:9aa491b1
Events : 692
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 65 1 active sync /dev/sde1
3 8 49 2 active sync /dev/sdd1
Now we can add new disk to this array:
mdadm --manage /dev/md1 --add /dev/sdb1
mdadm: added /dev/sdb1
And its done:
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 sdb1[4] sde1[1] sdd1[3]
488132608 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
[>....................] recovery = 0.3% (869184/244066304) finish=197.5min speed=20515K/sec
bitmap: 0/2 pages [0KB], 65536KB chunk
If we have a problem with some disk, we may remove it during work. At first, we must mark it as failed. So look at good and working raid-1:
mdadm --detail /dev/md0
/dev/md0:
Raid Level : raid1
Array Size : 976629760 (931.39 GiB 1000.07 GB)
Used Dev Size : 976629760 (931.39 GiB 1000.07 GB)
Raid Devices : 2
Total Devices : 3
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
active sync /dev/sda1
active sync /dev/sdb1
spare /dev/sde1
Now mark disk sda1 as faulty:
mdadm /dev/md0 -f /dev/sda1
mdadm --detail /dev/md0
/dev/md0:
Array Size : 976629760 (931.39 GiB 1000.07 GB)
Used Dev Size : 976629760 (931.39 GiB 1000.07 GB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 1
Spare Devices : 1
Rebuild Status : 0% complete
spare rebuilding /dev/sde1
active sync /dev/sdb1
faulty /dev/sda1
cat /proc/mdstat
md0 : active raid1 sda1[0](F) sde1[2] sdb1[1]
976629760 blocks super 1.2 [2/1] [_U]
[>....................] recovery = 0.2% (2292928/976629760) finish=169.9min speed=95538K/sec
I waited until finish this operation. Then I halted this server, remove the exact drive and insert a new one. After power-on, we create a new partition table on /dev/sda exactly as old one, or as active disks now. The we re-add it as spare to the raid:
mdadm /dev/md0 -a /dev/sda1
mdadm --detail /dev/md0
/dev/md0:
Raid Level : raid1
Array Size : 976629760 (931.39 GiB 1000.07 GB)
Used Dev Size : 976629760 (931.39 GiB 1000.07 GB)
Raid Devices : 2
Total Devices : 3
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
active sync /dev/sde1
active sync /dev/sdb1
spare /dev/sda1
cat /proc/mdstat
md0 : active raid1 sda1[3](S) sde1[2] sdb1[1]
976629760 blocks super 1.2 [2/2] [UU]
bitmap: 1/8 pages [4KB], 65536KB chunk
Total Page Visits: 166650 - Today Page Visits: 8