# Reiserfsck : --rebuild-tree hangs on pass 2

## Daivil

Hi everyone,

Very unlucky week as far as I'm concerned.

On one of my servers, 2/4 disks died at the same time on a reiserfs raid-5 array and no backup is available.

To recover it, I did run a "dd" copy of the "least dead" disk to a new one. Then I rebuild the raid-5 array with the two "good" drives and the third new one (--assume-clean).

The problem is that reiserfsck --rebuild-tree hangs on pass 2 :

 *Quote:*   

> 
> 
> *************************************************************
> 
> ** Do not  run  the  program  with  --rebuild-tree  unless **
> ...

 

As you can see, pass 2 is VERY slow... (0/sec).  With 38406 items left to read, it could take weeks (supposed it'd work to the end).

My server is booted over a rescue debian and I build reiserfsprogs locally (chroot on the local disk).

 *Quote:*   

> 
> 
> rescue / # reiserfsck -V
> 
> reiserfsck 3.6.21 (2009 www.namesys.com)
> ...

 

Anyone with a brilliant idea?

Thanks,

Daivil

----------

## idella4

Daivil

before going any further I would get into an external system, some live cd, gentoo, knoppix, whatever and view the state of the system at that point.

I've had reiserfsck reduce s systems into a thousand pieces and saved to lost&found.

----------

## sashametro

 *Daivil wrote:*   

> 
> 
> On one of my servers, 2/4 disks died at the same time on a reiserfs raid-5 array and no backup is available.
> 
> ...
> ...

 

No brilliant ideas, except that just this weekend I have run into this slowness with a 3TB external RAID-5 (hardware) volume on a RHEL5 system with reiserfs-utils 3.6.19-2.2.  The actual corruption was quite minor (no lost disks, just some failed I/Os) and I have to say, the overkill of rebuild-tree is extremely annoying - enough so that I doubt I will ever use reiserfs again for a large RAID filesystem.

Running strace on reiserfsck I can see it making lots of seeks and 4K writes - I think that this is triggering RAID-5 read/modify/write cycles as the writes are smaller than the RAID chunk/stripe size and that is making the thing dog dog slow (I only have "left 100 0/sec" but it has taken over 24 hours on this stage).  As far as I can tell, it has already made more than one (very slow) pass through the entire 3 TB - the seek offsets were increasing monotonically (and slowly) and appear to have wrapped from 1.5TB to 600GB offsets (although I could be wrong about the monotonic scan, and it is still doing a single pass).

Avoiding reiserfs seems like a good idea.  If that's not possible/desirable, making it use a larger block size that is a multiple of the RAID stripe/chunk would seem helpful too.  But it is already too late for either of us to do anything about that.  If you have any possibility to copy the data to a non-RAID-5 volume (or one with a much smaller chunk/stripe) you might be able to speed it up that way, but are you sure it is worth having to start again from scratch?

I think the 0/sec is somewhat misleading as it seems (from output listed at http://osdir.com/ml/file-systems.reiserfs.general/2002-10/msg00219.html) that pass 2 is trying to find space to put (move?) some of the items in the tree, but it is doing it by scanning the entire tree or volume - on a fairly full filesystem there may be very few places to put the items and so it can take a long time?  The percentage output makes it seem like your pass 2 is about half done, so perhaps it won't take a week, just another day?

@alex

----------

## sashametro

 *I wrote:*   

> 
> 
> Running strace on reiserfsck I can see it making lots of seeks and 4K writes - I think that this is triggering RAID-5 read/modify/write cycles as the writes are smaller than the RAID chunk/stripe size and that is making the thing dog dog slow (I only have "left 100 0/sec" but it has taken over 24 hours on this stage).  As far as I can tell, it has already made more than one (very slow) pass through the entire 3 TB - the seek offsets were increasing monotonically (and slowly) and appear to have wrapped from 1.5TB to 600GB offsets (although I could be wrong about the monotonic scan, and it is still doing a single pass).
> 
> 

 

Just did a progress analysis on this, and if it is scanning the entire 3TB volume, the apparent rate of ~5MB/minute (from seek offset differences on strace samples over 22 hours, ignoring the 1.5TB -> 600GB jump) means that it will take over 400 days to complete.

I guess I just decided to give up, reformat the volume as ext4 and restore from backups....sigh.  Hope you had better luck with yours.

----------

