在国产Kylin操作系统中，iscsi多路径挂载磁盘遇见的bug
通过tgt-iscsi实现RBD的多路径挂载测试
- 参考资料
环境准备
- tgt节点和ceph节点准备与之前文章类似，不再重复。tgt-iscsi对接rbd的多路径挂载测试验证
问题描述
- 在Kylin操作系统中，使用iscsi+multipath挂载过来的磁盘，磁盘上已经有lvm配置。
- 在multipath聚合之前，lvm先扫描到上面的pv-lv并激活。
- 后续multipath聚合失败，并且多块盘有相同的pv-lv，触发报错
- 该现象暂未在非国产操作系统上遇见（debian redhat centos）
规避办法
- 在
lvm.conf中增加过滤规则，iscsi设备不要自动激活pv-lv
操作记录
-
配置lvm过滤规则，这个在
/etc/lvm/lvm.conf也有详细说明。- 打开配置文件，找到过滤规则配置说明，并添加过滤规则
filter = [ "r|/dev/disk/by-path/ip-*|" ]
# Configuration option devices/filter. # Limit the block devices that are used by LVM commands. # This is a list of regular expressions used to accept or reject block # device path names. Each regex is delimited by a vertical bar '|' # (or any character) and is preceded by 'a' to accept the path, or # by 'r' to reject the path. The first regex in the list to match the # path is used, producing the 'a' or 'r' result for the device. # When multiple path names exist for a block device, if any path name # matches an 'a' pattern before an 'r' pattern, then the device is # accepted. If all the path names match an 'r' pattern first, then the # device is rejected. Unmatching path names do not affect the accept # or reject decision. If no path names for a device match a pattern, # then the device is accepted. Be careful mixing 'a' and 'r' patterns, # as the combination might produce unexpected results (test changes.) # Run vgscan after changing the filter to regenerate the cache. # # Example # Accept every block device: # filter = [ "a|.*|" ] # Reject the cdrom drive: # filter = [ "r|/dev/cdrom|" ] filter = [ "r|/dev/disk/by-path/ip-*|" ] # Work with just loopback devices, e.g. for testing: # filter = [ "a|loop|", "r|.*|" ] # Accept all loop devices and ide drives except hdc: # filter = [ "a|loop|", "r|/dev/hdc|", "a|/dev/ide|", "r|.*|" ] # Use anchors to be very specific: # filter = [ "a|^/dev/hda8$|", "r|.*|" ] # # This configuration option has an automatic default value. # filter = [ "a|.*|" ] # 筛选出通过iscsi挂载的磁盘 root@client:~# ls -l /dev/disk/by-path/ip-* lrwxrwxrwx 1 root root 9 Oct 28 11:48 /dev/disk/by-path/ip-192.168.1.141:3260-iscsi-iqn.2014-04.rbdstore.tgt1.com:iscsi-lun-1 -> ../../sdb lrwxrwxrwx 1 root root 9 Oct 28 12:02 /dev/disk/by-path/ip-192.168.1.141:3260-iscsi-iqn.2014-04.rbdstore.tgt1.com:iscsi-lun-2 -> ../../sdc lrwxrwxrwx 1 root root 9 Oct 28 11:48 /dev/disk/by-path/ip-192.168.1.142:3260-iscsi-iqn.2014-04.rbdstore.tgt2.com:iscsi-lun-1 -> ../../sdd lrwxrwxrwx 1 root root 9 Oct 28 12:02 /dev/disk/by-path/ip-192.168.1.142:3260-iscsi-iqn.2014-04.rbdstore.tgt2.com:iscsi-lun-2 -> ../../sde # 配置过滤规则 root@client:~# grep -v \# /etc/lvm/lvm.conf | grep -v ^$ config { } devices { filter = [ "r|/dev/disk/by-path/ip-*|" ] } allocation { } log { } backup { } shell { } global { } activation { } dmeventd { } -
清理其中一个multipath配置和映射
- 检查当前配置，准备清理 sdc/sde + tgt_vg2/tgt_lv2 这一组
root@client:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 50G 0 disk |-sda1 8:1 0 487M 0 part /boot |-sda2 8:2 0 1K 0 part `-sda5 8:5 0 49.5G 0 part |-debian12--vg-root 254:0 0 16.8G 0 lvm / |-debian12--vg-swap_1 254:1 0 976M 0 lvm [SWAP] `-debian12--vg-home 254:2 0 31.8G 0 lvm /home sdb 8:16 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sdc 8:32 0 10G 0 disk `-data2 254:4 0 10G 0 mpath `-tgt_vg2-tgt_lv2 254:6 0 10G 0 lvm sdd 8:48 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sde 8:64 0 10G 0 disk `-data2 254:4 0 10G 0 mpath `-tgt_vg2-tgt_lv2 254:6 0 10G 0 lvm sr0 11:0 1 1024M 0 rom root@client:~# multipath -ll data (360000000000000000e00000000010001) dm-3 IET,VIRTUAL-DISK size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:1 sdb 8:16 active ready running `- 34:0:0:1 sdd 8:48 active ready running data2 (360000000000000000e00000000010002) dm-4 IET,VIRTUAL-DISK size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:2 sdc 8:32 active ready running `- 34:0:0:2 sde 8:64 active ready running- 清理
#移除multipath记录 root@client:~# multipath -w data2 wwid '360000000000000000e00000000010002' removed #移除挂载的磁盘 root@client:~# echo 1 > /sys/block/sdc/device/delete root@client:~# echo 1 > /sys/block/sde/device/delete #清理dm root@client:~# dmsetup ls data (254:3) data2 (254:4) debian12--vg-home (254:2) debian12--vg-root (254:0) debian12--vg-swap_1 (254:1) tgt_vg-tgt_lv (254:5) tgt_vg2-tgt_lv2 (254:6) root@client:~# dmsetup remove tgt_vg2-tgt_lv2 root@client:~# dmsetup remove data2 root@client:~# dmsetup ls data (254:3) debian12--vg-home (254:2) debian12--vg-root (254:0) debian12--vg-swap_1 (254:1) tgt_vg-tgt_lv (254:5)- 检查
# 检查多路径 root@client:~# multipath -ll data (360000000000000000e00000000010001) dm-3 IET,VIRTUAL-DISK size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:1 sdb 8:16 active ready running `- 34:0:0:1 sdd 8:48 active ready running # 检查盘符 root@client:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 50G 0 disk |-sda1 8:1 0 487M 0 part /boot |-sda2 8:2 0 1K 0 part `-sda5 8:5 0 49.5G 0 part |-debian12--vg-root 254:0 0 16.8G 0 lvm / |-debian12--vg-swap_1 254:1 0 976M 0 lvm [SWAP] `-debian12--vg-home 254:2 0 31.8G 0 lvm /home sdb 8:16 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sdd 8:48 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sr0 11:0 1 1024M 0 rom # 检查pv-lv root@client:~# pvs PV VG Fmt Attr PSize PFree /dev/mapper/data tgt_vg lvm2 a-- <10.00g 0 /dev/sda5 debian12-vg lvm2 a-- <49.52g 0 -
重新扫盘，并识别lvm
- 重新扫盘，磁盘正常识别，lvm未注册。multipath因为配置被移除，也未聚合
# 扫描 scsi_host,lsblk正常识别磁盘 root@client:~# host_id=$(ls /sys/class/scsi_host/) root@client:~# for i in $host_id ; do echo "- - -" > /sys/class/scsi_host/$i/scan ; done root@client:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 50G 0 disk |-sda1 8:1 0 487M 0 part /boot |-sda2 8:2 0 1K 0 part `-sda5 8:5 0 49.5G 0 part |-debian12--vg-root 254:0 0 16.8G 0 lvm / |-debian12--vg-swap_1 254:1 0 976M 0 lvm [SWAP] `-debian12--vg-home 254:2 0 31.8G 0 lvm /home sdb 8:16 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sdc 8:32 0 10G 0 disk sdd 8:48 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sde 8:64 0 10G 0 disk sr0 11:0 1 1024M 0 rom # multipath 未聚合 root@client:~# multipath -ll data (360000000000000000e00000000010001) dm-3 IET,VIRTUAL-DISK size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:1 sdb 8:16 active ready running `- 34:0:0:1 sdd 8:48 active ready running root@client:~# multipath -d -v3 2>/dev/null ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st 0:0:0:0 sda 8:0 -1 undef undef VMware,Virtual disk unknown 360000000000000000e00000000010001 33:0:0:1 sdb 8:16 1 undef undef IET,VIRTUAL-DISK unknown 360000000000000000e00000000010002 33:0:0:2 sdc 8:32 1 undef undef IET,VIRTUAL-DISK unknown 360000000000000000e00000000010001 34:0:0:1 sdd 8:48 1 undef undef IET,VIRTUAL-DISK unknown 360000000000000000e00000000010002 34:0:0:2 sde 8:64 1 undef undef IET,VIRTUAL-DISK unknown # pv未识别 root@client:~# pvs PV VG Fmt Attr PSize PFree /dev/mapper/data tgt_vg lvm2 a-- <10.00g 0 /dev/sda5 debian12-vg lvm2 a-- <49.52g 0- 聚合
sdc/sde，并激活lv
# 添加 multipath信息，重启服务 root@client:~# multipath -a 360000000000000000e00000000010002 wwid '360000000000000000e00000000010002' added root@client:~# systemctl restart multipathd.service # multipath 正常聚合，data2正常识别 root@client:~# multipath -ll data (360000000000000000e00000000010001) dm-3 IET,VIRTUAL-DISK size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:1 sdb 8:16 active ready running `- 34:0:0:1 sdd 8:48 active ready running data2 (360000000000000000e00000000010002) dm-4 IET,VIRTUAL-DISK size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:2 sdc 8:32 active ready running `- 34:0:0:2 sde 8:64 active ready running root@client:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 50G 0 disk |-sda1 8:1 0 487M 0 part /boot |-sda2 8:2 0 1K 0 part `-sda5 8:5 0 49.5G 0 part |-debian12--vg-root 254:0 0 16.8G 0 lvm / |-debian12--vg-swap_1 254:1 0 976M 0 lvm [SWAP] `-debian12--vg-home 254:2 0 31.8G 0 lvm /home sdb 8:16 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sdc 8:32 0 10G 0 disk `-data2 254:4 0 10G 0 mpath sdd 8:48 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sde 8:64 0 10G 0 disk `-data2 254:4 0 10G 0 mpath sr0 11:0 1 1024M 0 rom # pv vg 正常识别 root@client:~# pvs PV VG Fmt Attr PSize PFree /dev/mapper/data tgt_vg lvm2 a-- <10.00g 0 /dev/mapper/data2 tgt_vg2 lvm2 a-- <10.00g 0 /dev/sda5 debian12-vg lvm2 a-- <49.52g 0 root@client:~# vgs VG #PV #LV #SN Attr VSize VFree debian12-vg 1 3 0 wz--n- <49.52g 0 tgt_vg 1 1 0 wz--n- <10.00g 0 tgt_vg2 1 1 0 wz--n- <10.00g 0 # 激活lv，lsblk正常识别 root@client:~# vgchange -a y tgt_vg2 1 logical volume(s) in volume group "tgt_vg2" now active root@client:~# vgs VG #PV #LV #SN Attr VSize VFree debian12-vg 1 3 0 wz--n- <49.52g 0 tgt_vg 1 1 0 wz--n- <10.00g 0 tgt_vg2 1 1 0 wz--n- <10.00g 0 root@client:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 50G 0 disk |-sda1 8:1 0 487M 0 part /boot |-sda2 8:2 0 1K 0 part `-sda5 8:5 0 49.5G 0 part |-debian12--vg-root 254:0 0 16.8G 0 lvm / |-debian12--vg-swap_1 254:1 0 976M 0 lvm [SWAP] `-debian12--vg-home 254:2 0 31.8G 0 lvm /home sdb 8:16 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sdc 8:32 0 10G 0 disk `-data2 254:4 0 10G 0 mpath `-tgt_vg2-tgt_lv2 254:6 0 10G 0 lvm sdd 8:48 0 10G 0 disk `-data 254:3 0 10G 0 mpath `-tgt_vg-tgt_lv 254:5 0 10G 0 lvm sde 8:64 0 10G 0 disk `-data2 254:4 0 10G 0 mpath `-tgt_vg2-tgt_lv2 254:6 0 10G 0 lvm sr0 11:0 1 1024M 0 rom
测试结论
- 全流程符合预期，scsi_host扫盘后，lvm未自动注册磁盘上pv-lv。只有当multipath聚合后，才自动识别pv-lv。
粤公网安备案：