# LVM2 on removable storage

## nsoveiko

i have an external USB drives that is used for rsnapshot backups. the drive (/dev/sdb) has partition (/dev/sdb2), which is a physical volume used in volume group backup. a logical volume rsnapshot in this group is the filesystem where increments are stored. it is formatted with reiserfs and mounted under /mnt/backup in fstab.

this sytem has been set up like this for a while and everything was working fine. today i was plugging a network cable and accidentally unplugged the USB cable without unmounting the external drive. an attempt to mount the filesystem back resulted in utter failure.

```
# sfdisk -l -uS /dev/sdb

Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track

Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System

/dev/sdb1   *        63   1992059    1991997  82  Linux swap / Solaris

/dev/sdb2       1992060 976768064  974776005  8e  Linux LVM

/dev/sdb3             0         -          0   0  Empty

/dev/sdb4             0         -          0   0  Empty

```

so far, so good. /dev/sdb1 is unused and past sdb2 is empty space.

```
# /sbin/vgscan

  Reading all physical volumes.  This may take a while...

  /dev/dm-9: read failed after 0 of 4096 at 268435390464: Input/output error

  /dev/dm-9: read failed after 0 of 4096 at 268435447808: Input/output error

  /dev/dm-9: read failed after 0 of 4096 at 0: Input/output error

  /dev/dm-9: read failed after 0 of 4096 at 4096: Input/output error

  /dev/dm-9: read failed after 0 of 4096 at 0: Input/output error

  Found volume group "backup" using metadata type lvm2

  Found volume group "raid0" using metadata type lvm2

  Found volume group "storage" using metadata type lvm2

  Found volume group "raid1" using metadata type lvm2

```

hmm, don't like the "read failed" part.

```
# ls -al /dev/dm-9

lrwxrwxrwx 1 root root 23 2008-08-21 16:00 /dev/dm-9 -> mapper/backup-rsnapshot

```

```
# vgdisplay backup

  /dev/dm-9: read failed after 0 of 4096 at 0: Input/output error

  --- Volume group ---

  VG Name               backup

  System ID

  Format                lvm2

  Metadata Areas        1

  Metadata Sequence No  2

  VG Access             read/write

  VG Status             resizable

  MAX LV                0

  Cur LV                1

  Open LV               0

  Max PV                0

  Cur PV                1

  Act PV                1

  VG Size               464.81 GB

  PE Size               4.00 MB

  Total PE              118991

  Alloc PE / Size       64000 / 250.00 GB

  Free  PE / Size       54991 / 214.81 GB

  VG UUID               87seY5-Vm0I-WoCU-sH1n-bQ2Q-qyc3-kJHsZh

```

```
# vgck -v backup

    Using volume group(s) on command line

    Finding volume group "backup"

  /dev/dm-9: read failed after 0 of 4096 at 0: Input/output error

```

```
# lvdisplay backup

  /dev/dm-9: read failed after 0 of 4096 at 0: Input/output error

  --- Logical volume ---

  LV Name                /dev/backup/rsnapshot

  VG Name                backup

  LV UUID                XV66Gp-3Nnn-db2R-nZaX-Im7o-FzDI-Sj5nW2

  LV Write Access        read/write

  LV Status              available

  # open                 0

  LV Size                250.00 GB

  Current LE             64000

  Segments               1

  Allocation             contiguous

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:9

```

so, it appears that /dev/dm9 a.k.a. /dev/backup/rsnapshot a.k.a. /dev/mapper/backup-rsnapshot got corrupt. it can not mount and fsck.reiserfs can not fix the problem either:

```
# fsck.reiserfs /dev/mapper/backup-rsnapshot

reiserfsck 3.6.19 (2003 www.namesys.com)

*************************************************************

** If you are using the latest reiserfsprogs and  it fails **

** please  email bug reports to reiserfs-list@namesys.com, **

** providing  as  much  information  as  possible --  your **

** hardware,  kernel,  patches,  settings,  all reiserfsck **

** messages  (including version),  the reiserfsck logfile, **

** check  the  syslog file  for  any  related information. **

** If you would like advice on using this program, support **

** is available  for $25 at  www.namesys.com/support.html. **

*************************************************************

Will read-only check consistency of the filesystem on /dev/mapper/backup-rsnapshot

Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

The problem has occurred looks like a hardware problem. If you have

bad blocks, we advise you to get a new hard drive, because once you

get one bad block  that the disk  drive internals  cannot hide from

your sight,the chances of getting more are generally said to become

much higher  (precise statistics are unknown to us), and  this disk

drive is probably not expensive enough  for you to you to risk your

time and  data on it.  If you don't want to follow that follow that

advice then  if you have just a few bad blocks,  try writing to the

bad blocks  and see if the drive remaps  the bad blocks (that means

it takes a block  it has  in reserve  and allocates  it for use for

of that block number).  If it cannot remap the block,  use badblock

option (-B) with  reiserfs utils to handle this block correctly.

bread: Cannot read the block (2): (Input/output error).

Aborted

```

reiserfs thinks that it's an underlying problem, which makes me think it is on the LVM level, rather than filesystem. is there a way to fix this and recover the data?Last edited by nsoveiko on Thu Oct 02, 2008 8:02 pm; edited 1 time in total

----------

## nsoveiko

 *Quote:*   

> reiserfs thinks that it's an underlying problem, which makes me think it is on the LVM level, rather than filesystem.

 the problem appears to be in the stale device nodes. man page for vgscan says that it'll take care of that, but it doesn't.

```
       --mknodes

              Also checks the LVM special files in /dev that  are  needed  for

              active  logical volumes and creates any missing ones and removes

              unused ones.

```

```
# /sbin/vgscan --mknodes

  Reading all physical volumes.  This may take a while...

  /dev/dm-3: read failed after 0 of 4096 at 268435390464: Input/output error

  /dev/dm-3: read failed after 0 of 4096 at 268435447808: Input/output error

  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error

  /dev/dm-3: read failed after 0 of 4096 at 4096: Input/output error

  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error

  Found volume group "raid0" using metadata type lvm2

  Found volume group "backup" using metadata type lvm2

  Found volume group "storage" using metadata type lvm2

  Found volume group "raid1" using metadata type lvm2

```

now, if i explicitly remove the offending node, vgchange will recreate it and mount goes without a hitch:

```
# dmsetup remove backup-rsnapshot

# vgchange -a y backup

  1 logical volume(s) in volume group "backup" now active

# mount /mnt/backup

# df /mnt/backup

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/backup-rsnapshot

                      250G  117G  134G  47% /mnt/backup

```

it looks to me like a bug in vgscan man pages and/or in /etc/udev/rules.d/64-device-mapper.rules. am i right?

----------

