# Problem with filesystem after power outage

## cgits

Hello, I have a HDD named /dev/sdc which hosts a filesystem other than my root. I boot after a power outage, and I cannot mount it. I wonder if it is possible for the filesystem to be corrupted beyond repair. I don't think it was even used at the moment of the failure. What can I try to make it mount again?

```
user@gandalf ~ $ sudo fdisk -l /dev/sdc

...

Device     Start        End    Sectors  Size Type

/dev/sdc1   2048 5860532223 5860530176  2.7T Linux filesystem
```

From /etc/fstab:

/dev/sdc1		/mnt/data_c	ext4		defaults	0 0

```
user@gandalf ~ $ sudo mount /dev/sdc1

mount: wrong fs type, bad option, bad superblock on /dev/sdc1,

       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try

       dmesg | tail or so.

user@gandalf ~ $ dmesg|tail

...

[154150.717274]  sdc: sdc1

[154564.927084] EXT4-fs (sdc1): VFS: Can't find ext4 filesystem

user@gandalf ~ $ sudo fsck -t ext4 /dev/sdc1

fsck from util-linux 2.26.2

e2fsck 1.42.13 (17-May-2015)

ext2fs_open2: Bad magic number in super-block

fsck.ext4: Superblock invalid, trying backup blocks...

fsck.ext4: Bad magic number in super-block while trying to open /dev/sdc1

The superblock could not be read or does not describe a valid ext2/ext3/ext4

filesystem.  If the device is valid and it really contains an ext2/ext3/ext4

filesystem (and not swap or ufs or something else), then the superblock

is corrupt, and you might try running e2fsck with an alternate superblock:

    e2fsck -b 8193 <device>

 or

    e2fsck -b 32768 <device>

```

I tried again twice (with the -b parameters suggested by fsck above) with no result.

----------

## NeddySeagoon

cgits,

Try mount with alternate superblock locations. 

Don't use fsck. It can make a bad situation worse.  If it runs at all, it will make assumptions in the face of missing or conflicting data about your filesystem.

It will make the filesystem metadata self consistant but that says nothing about the user data that was on the filesystem.

At first, do no harm, so don't do anything that will change the filesystem.  Ideally, make an image of it, so you have an undo.

Try mount with alternate superblock locations. 

mount -t ext4 -o sb=131072,ro /dev/sdc1 /mnt/someplace

That's a read only mount, using the first alternate superblock.

See man mount, the bit about ext2. That's common to ext3 and ext4 too.

```
Superblock backups stored on blocks: 

    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000
```

These numbers are in filesystem blocks, which will be 4k for you.

Mount wants its sb= in terms of 1k blocks, so you need to multiply all the fs block numbers by 4.

Hence fs block 32768*4 is sb=131072 to mount.

----------

## cgits

wow, thank you for the very detailed analysis of the situation. Unfortunately none of the alternate superblock locations worked. I always got the same error and (dmesg)

```
EXT4-fs (sdc1): VFS: Can't find ext4 filesystem
```

----------

## NeddySeagoon

cgits,

Now it gets more difficult.  

Its time to ask about backups, if you have space to make an image of the drive and how much time do you want to spend recovering the data.

At this stage, it looks like that in place data recovery isn't going to happen.

Knowing what the data is (photos, videos, flac etc and how it was written (write mostly, lots of additions and deletions) may be useful too.

There are tools like photorec that find files by the file headers, then make assumptions about the disk layout to recover the files.

Some background questions are also overdue.

Are you attempting the data recovery on the same machine using the same install that the drive was normally used on?

We don't want to have GPT support normally and be missing it for data recovery.

Posting the output of 

```
dumpe2fs -h /dev/sdc1
```

may be useful.

Read 

```
man dumpe2fs
```

 and play with the -o superblock=superblock -o blocksize=blocksize and even the -f options.

This is all harmless, as its only reading things.

Healthy output (from my /boot) looks like:-

```
$ sudo dumpe2fs -h /dev/sde1

Password: 

dumpe2fs 1.43.3 (04-Sep-2016)

Filesystem volume name:   <none>

Last mounted on:          /boot

Filesystem UUID:          c400b18c-0210-4338-a0fd-f437ecbaaf99

Filesystem magic number:  0xEF53

Filesystem revision #:    1 (dynamic)

Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize

Filesystem flags:         signed_directory_hash 

Default mount options:    user_xattr acl

Filesystem state:         clean

Errors behavior:          Continue

Filesystem OS type:       Linux

Inode count:              30976

Block count:              123904

Reserved block count:     6195

Free blocks:              61564

Free inodes:              30931

First block:              1

Block size:               1024

Fragment size:            1024

Reserved GDT blocks:      256

Blocks per group:         8192

Fragments per group:      8192

Inodes per group:         1936

Inode blocks per group:   242

RAID stride:              4

RAID stripe width:        4

Flex block group size:    16

Filesystem created:       Sun Dec 28 12:54:24 2014

Last mount time:          Fri Oct 14 16:09:46 2016

Last write time:          Fri Oct 14 16:27:02 2016

Mount count:              30

Maximum mount count:      -1

Last checked:             Sun Dec 28 12:54:24 2014

Check interval:           0 (<none>)

Lifetime writes:          113 MB

Reserved blocks uid:      0 (user root)

Reserved blocks gid:      0 (group root)

First inode:              11

Inode size:             128

Default directory hash:   half_md4

Directory Hash Seed:      02028996-f637-4b92-91e3-0ae757d3347d
```

----------

## cgits

Thank you again NeddySeagoon,

I have backups of most files, but not all of them. I must be missing more than 200 files. Yes, I am using the same machine to recover the data, I did not touch the physical disk drive

```
user@gandalf $ sudo dumpe2fs -h /dev/sdc1

dumpe2fs 1.42.13 (17-May-2015)

dumpe2fs: Bad magic number in super-block while trying to open /dev/sdc1

Couldn't find valid filesystem superblock.
```

Also the same with -o superblock=... and -f options (but I did not try changing blocksize)

BUT some good news. I emerged photorec and it did recover some hundreds of files. But it does not recover filenames and also the playback of the videos is not always smooth. Minor glitches like pauses, freezes, jumps, fast-forwards, that I don't think were present in the original videos. This needs a huge amount of time and effort, and I also don't have enough free space on another drive to copy the files at the moment.

BUT then I tried testdisk, which comes together with photorec. It searches for lost partitions, it finds my partition, it lists all (I think all) my files on that disk. I must have done something not entirely correctly though, because I told it to write the changes to the partition table, it told me like "done, but you will have to reboot" and after reboot I still cannot mount /dev/sdc1. (But testdisk still finds the partition and sees my files)

Anyway, maybe I should check Testdisk's documentation deeper.

You have been unbelievable help

EDIT:

I guess the partition table was fine to start with, I just got excited to see my files and stopped thinking. Anyway, I went to "[Advanced] Filesystem Utils", then "Locate ext2/ext3/ext4 backup superblock" and I got

```
Disk /dev/sdc - 3000 GB / 2794 GiB - CHS 364801 255 63

     Partition                  Start        End    Size in sectors

  MS Data                     2048 5860532223 5860530176 [data3]

superblock 0, blocksize=4096 [data3]

superblock 32768, blocksize=4096 [data3]

superblock 98304, blocksize=4096 [data3]

superblock 163840, blocksize=4096 [data3]

superblock 229376, blocksize=4096 [data3]

superblock 294912, blocksize=4096 [data3]

superblock 819200, blocksize=4096 [data3]

superblock 884736, blocksize=4096 [data3]

superblock 1605632, blocksize=4096 [data3]

superblock 2654208, blocksize=4096 [data3]

To repair the filesystem using alternate superblock, run

fsck.ext4 -p -b superblock -B blocksize device
```

EDIT2:

I found this quote somewhere on the internets  *Quote:*   

> I can't make myself believe that you were unlucky enough to have all copies of the superblock wiped. So there must be something wrong with the partition table, which in turn is throwing off the logical block offsets in the filesystem causing fsck to not be able to find the alternate superblocks

  I wonder if it applies in my case.

----------

## NeddySeagoon

cgits,

All the right answers so far.

I agree, I think your partition table was fine to start with, its GPT so there are two copies. That both agree is checked.

You were also getting the first/only entry listed.  In this case, the overwrite is harmless.

Even if it wasn't, you don't actually need a partition table to mount filesystems within partitions.

If testdisk found some more superblock locations you can try feeding them to the mount command I outlined earlier.

Don't forget the *4 for the 4k block size.

Testdisk isn't reading your filesystem to show you your files.  Its taking hints but if it could read the filesystem, so could mount.

I'll guess the the jumps and other replay issues in your recovered videos are where testdisk guessed wrong and blocks of data are missing or are entirely unrelated to the file that they were recovered as a part of.

If you really want to get your hands dirty 

```
emerge  app-forensics/sleuthkit
```

Description: A collection of file system and media management forensic analysis tools.  How much time you want to spend on this depends on how valuable your missing data is.

----------

## cgits

Thank you again,

And now for the next episode in this series: So over the last few days, I used testdisk to compare which files I already had in my backups and which needed to be recovered. I recovered several hundred files until I was satisfied I was not missing anything important. I went ahead making a new filesystem on the device /dev/sdc1, but this time I chose XFS, because I was angry at ext4 for this trouble. After all, in the past I was using all kinds of filesystems: reiserfs, JFS, XFS and never had such problems.

After making the xfs filesystem, I mount it, populate it with data, I unmount, I remount, the data is there. But next day I reboot and I can no longer mount it (!)

```
XFS (sdc1): Invalid superblock magic number
```

----------

## NeddySeagoon

cgits,

Start a new topic for XFS. I've never used it.

I understand that its fast and scales well but it keeps a lot of data in RAM cache (to get the speed) and doesn't take kindly to unclean shutdowns.

----------

