# Using dd to ntfs-3g: How much slower than with kernel fat32?

## FuriousGeorge

A friend's raid0 array failed and we need professional help to restore it.

Thanks to some simple linux tools, we found a company that will do it on the cheap provided I can create images of the drives and copy them to a computer running Windows.  They will then remote in, install their software, and attempt to work their magic.

Unfortunately, it seems that ntfs-3g is painfully slow.  As of 8 hours into the process I've only been able to copy about 24 of 160G.  Looking at top I can see that ntfs-3G is using ~90% of the cpu cycles, with dd using only around 5%.

I have two questions:

#1  How much faster would this be if the destination media was fat32 or even if I'd used the kernels ntfs module with write support (not sure if it's included in the 2008.0 Beta livecd), rather than ntfs-3g?

#2  It's important that if dd encounter a bad block that it not 'skip it'.  It must write 0's or whatever, so long as the image file is equivalent to the original media in terms of number of blocks and their sequence.  So far so good, but will a simple dd command like this work...

```
dd if=/dev/sda of=/mnt/Windows/DISK0.img
```

...  or will it just skip a bad block, making this all for naught?  If there are X blocks in there must be X blocks out, know what I mean?

I'm kind of in a bind with this because it looks like it will take at least another 20 hours, but I don't want to cancel it and experiment if the time savings will only be nominal.

----------

## fangorn

First, FAT32 cannot handle files bigger than 4 GB, so forget it!

Second, Write support in the kernel nfts driver is marked experimental for a reason. Stick with ntfs-3g!

Third, dd stops at first read error! That is why there is dd_rescue. 

dd could also be so slow because it uses a improper block size. I use bs=32k or bs=64k normally when using dd. Speeds up the process tremendously. 

But as I said: dd stops at first read error. So I would stop the process and use dd_rescue instead. It uses good block sizes by default and nullifies blocks it cannot read, but continues afterwards. 

Your command line would be

```
dd_rescue /dev/sda /mnt/Windows/DISK0.img
```

----------

## FuriousGeorge

dd_rescue isnt really an option, unless I install linux on a fourth hd.  isnt there some way to make dd not stop

Overnight it did only four gigs, so I'm thinking of sticking a fourth hd in there, and then copying to ntfs.

Alternatively, and in the interest of time, I suppose I could install *buntu on the fourth hard drive, and use dd rescue to ext3, then copy it bulk to the other hd.

I guess...

----------

## FuriousGeorge

So what I'm doing now is as follows:

I put another hard drive, large enough to hold both images, in the computer, which is formatted with ext3.  I'm then using dd_rescue as follows to copy them both to the new hard drive:

```
sudo dd_rescue /dev/sda /home/user/DISK0.img
```

When that completes I'll try to use ntfs-3g to copy them to the Windows drive so that the Data Recovery people can work on the images.

It remains to be seen if ntfs-3g will be any better when not used with dd.

I'll let everyone know.

----------

## FuriousGeorge

UPDATE:  One of the drives throws an error around GB 76 of 80.  This error hard lcoks the kernel on the ubuntu system I installed.  I'm now trying from a gentoo livecd from which I chroot into the ubuntu system and run dd_rescue.

I made sure to not let the screensaver come on, and I wrote it down the exception:

```
ata 6.00:  exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0

ata 6.00:  CPB resp_flags 0x11 CMD error

ata 6.00:  cmd c8/00:f8:08:ce:d9/00:00:00:00:00/e8 dma 126976

ata 6.00:  { DMRDY ERR }

ata 6.00:  { UNC }

```

I'm now investigating to see if that means anything other than disk hardware failure

UPDATE II:  It seems this could be at least partially related to a known kernel bug.  http://kerneltrap.org/mailarchive/linux-raid/2008/9/22/3366734 , which I'm guessing is teased out by a bad block, or something.

This would make some sense, since the whole point of using dd_rescue was to handle possible bad blocks, and I wouldn't expect it to hardlock the entire system like it does.  I don't think a kernel lock like this should be expected for unmounted media which I am reading at the bit level...

Seeing if the gentoo boot CD makes a differnece, if not I could use some advice...  perhaps I could pick up where it left off somehow, manually skipping this block...

----------

## FuriousGeorge

Disaster:

Same thing happens when using the kernel from the 2008.0 beta LiveCD.

There are a few differences, such as at the bottom where it says:

Kernel Panic - not syncing:  CPU Context Corrupt.

I'm gonna google around a bit and try an older kernel.

Any input is MUCH appreciated.

UPDATE:  Now trying kernel 2.6.19 AMD_64.  This is the first 64bit kernel I've tried.  It is probably also the oldest with the kernel from the Ubuntu 8.10 LiveCD (out this month), being the newest.Last edited by FuriousGeorge on Wed Nov 26, 2008 10:14 pm; edited 1 time in total

----------

## dmpogo

 *fangorn wrote:*   

> 
> 
> But as I said: dd stops at first read error. So I would stop the process and use dd_rescue instead. It uses good block sizes by default and nullifies blocks it cannot read, but continues afterwards. 
> 
> 

 

dd has 'noerror' option, or you can skip damaged blocks. How else did I deal witn my magnetic tapes that somebody chewed on  :Smile:  ?

----------

## FuriousGeorge

Great Success:  The 64-bit kernel got to the bad blocks on the media, threw some errors, crawled along for a few seconds, then finished.

Both disk images are of identical sizes.  It looks like it worked.

Now all I have to do is use ntfs-3g to copy 160GB of data to a NTFS partition...  that should be easy right?    :Confused: 

Thanks dmpogo and fangorn for the responses.  I'll keep you guys posted.

----------

## cyrillic

 *FuriousGeorge wrote:*   

> Now all I have to do is use ntfs-3g to copy 160GB of data to a NTFS partition...  that should be easy right?    

 

If the array was originally formatted NTFS, then your backup is still NTFS because dd does not remove the filesystem ...

----------

## FuriousGeorge

 *cyrillic wrote:*   

>  *FuriousGeorge wrote:*   Now all I have to do is use ntfs-3g to copy 160GB of data to a NTFS partition...  that should be easy right?     
> 
> If the array was originally formatted NTFS, then your backup is still NTFS because dd does not remove the filesystem ...

 

I was refering to the disk images I've just made and copying to 'doze so that the Data Recovery guys could do their thing.  I meant I was worried about ntfs-3g and how long it would take to copy.

For some reason dd if=/dev/sda of=(ntfs destination mounted with ntfs-3g) was going real slow.  In about 20 hours I had copied only about 34 gigs, and it surely would have failed at about 90% completion, as per the kernel bug I uncovered above.

otoh, dd or dd_rescue where the destination was ext3 would do about 33 megs a second.

copying to ntfs now is going much quicker.  about the same speed as dd_rescue if=/dev/sda of=(ext3 destination), maybe a bit slower.

----------

## dmpogo

 *FuriousGeorge wrote:*   

>  *cyrillic wrote:*    *FuriousGeorge wrote:*   Now all I have to do is use ntfs-3g to copy 160GB of data to a NTFS partition...  that should be easy right?     
> 
> If the array was originally formatted NTFS, then your backup is still NTFS because dd does not remove the filesystem ... 
> 
> I was refering to the disk images I've just made and copying to 'doze so that the Data Recovery guys could do their thing.  I meant I was worried about ntfs-3g and how long it would take to copy.
> ...

 

dd works below any file system, it copies block by block, its probably very painful if the block size of the output should be converted into ntfs-3g requirements

----------

## FuriousGeorge

 *dmpogo wrote:*   

> 
> 
> dd works below any file system, it copies block by block, its probably very painful if the block size of the output should be converted into ntfs-3g requirements

 

Just to be clear, the cause of the slowness was that dd and ntfs-3g were using different block sizes, which the ntfs-3g software was accounting for, and thus slowing down the progress?

----------

## cyrillic

 *FuriousGeorge wrote:*   

> For some reason dd if=/dev/sda of=(ntfs destination mounted with ntfs-3g) was going real slow. 

 

That is because if you want an exact copy of the original disk, you would normally use dd like this :

```
# dd if=/dev/sda of=/dev/sdb (or some other empty block device that is not mounted) 
```

Then the resulting copy on /dev/sdb would have the same filesystem (NTFS) that was on the original /dev/sda

----------

## FuriousGeorge

 *cyrillic wrote:*   

>  *FuriousGeorge wrote:*   For some reason dd if=/dev/sda of=(ntfs destination mounted with ntfs-3g) was going real slow.  
> 
> That is because if you want an exact copy of the original disk, you would normally use dd like this :
> 
> ```
> ...

 

Sure, I've done that when 'cloning' systems with varying degrees of success when the disks are of identical size.

What I don't really understand is why...

```
dd if=/dev/sda of=(ntfs partition mounted with ntfs-3g)/IMAGE.IMG
```

is an order of magnitude SLOWER than...

```
dd if=/dev/sda of=(ext3 partition mounted normally)/IMAGE.IMG
```

I think the reason is:

 *dmpogo wrote:*   

> 
> 
> dd works below any file system, it copies block by block, its probably very painful if the block size of the output should be converted into ntfs-3g requirements

 

----------

## dmpogo

 *FuriousGeorge wrote:*   

>  *dmpogo wrote:*   
> 
> dd works below any file system, it copies block by block, its probably very painful if the block size of the output should be converted into ntfs-3g requirements 
> 
> Just to be clear, the cause of the slowness was that dd and ntfs-3g were using different block sizes, which the ntfs-3g software was accounting for, and thus slowing down the progress?

 

It is my guess.   

You can use ibs=  and obs=   options to specify input and output block sizes.   Ext2/3 as far as I remember uses 4096 byte blocks,

while you should check which one NTFS uses.

dd has some default block size and it reads that size and outputs SAME size if no other options specified.

Let us guess it is 4096 byte as ext2 has and ntfs has different. Then, since you are writing into the file and not raw device, it is now

up to ntfs driver to deal with partial blocks that it receives from dd.  In worse case scenatio (imagine the block is 16k) it may be

rewriting the same 16k block 4 times on each receive of 4k (with continuing return of the disk head back).

While if it matches etx2 block it is easily recorded as received by ext2 driver.

Moreover once the writing goes through filesystem, it activates all io_schedulers in the kernel, and if a process has to wait it may

be penalized in io_queues or whatever

So to check that you should  find what block size ntfs uses and give dd obs=NTFSBLOCKSIZE option.

----------

## dmpogo

Actually I see that NTFS  uses 4k block size,  while ext2/3 uses 1k,  2k, 4k depending in filesystem, and on ~ 30Gb partitions I have it defaulted also to 4k.

----------

## FuriousGeorge

 *dmpogo wrote:*   

> Actually I see that NTFS  uses 4k block size,  while ext2/3 uses 1k,  2k, 4k depending in filesystem, and on ~ 30Gb partitions I have it defaulted also to 4k.

 

So the mystery is unresolved?

I have a theory, but it seems inadequate:

When I did the original:

```
dd if=/dev/sda of=(ntfs partition mounted with ntfs-3g)/IMAGE.IMG
```

I was actually doing two at a time.  As in...

```
dd if=/dev/sda of=(ntfs partition mounted with ntfs-3g)/IMAGE0.IMG & dd if=/dev/sdb of=(ntfs partition mounted with ntfs-3g)/IMAGE1.IMG
```

I thought this was an SMP system, but as it turns out I was wrong.

Still, as I said before, for some reason, in about 20 hours I had copied only about 35 or 36 gigs aggregate.  

That comes out to about 500K/sec compared to about 33MB/sec when the destination was ext3 AND I was doing one disk at a time.

The fact that I was doing two drives at a time in the first scenario, while I was doing one drive at a time under the second scenario doesn't seem to account for a speed difference of 6600%.Last edited by FuriousGeorge on Thu Nov 27, 2008 6:00 am; edited 2 times in total

----------

## dmpogo

so what was the result when you copy one disk at a time to ntfs system ?

----------

## FuriousGeorge

 *dmpogo wrote:*   

> so what was the result when you copy one disk at a time to ntfs system ?

 

You caught me mid edit.  

It was about 66x faster when I did one disk at a time AND the destination was ext3.

----------

## FuriousGeorge

 *dmpogo wrote:*   

> so what was the result when you copy one disk at a time to ntfs system ?

 

I misunderstood.  The answer is:  I don't know because I didn't try.  Maybe I'll give it a whirl tomorrow and report back.

----------

## dmpogo

[quote="FuriousGeorge"] *dmpogo wrote:*   

> so what was the result when you copy one disk at a time to ntfs system ?

 

I misunderstood.  The answer is:  I don't know because I didn't try.  Maybe I'll give it a whirl tomorrow and report back.[/quote

As you understand speed of IO has little to do with number of processors  :Smile: 

----------

## irgu

Use dd .... bs=1M. It should be about 10 times faster. Also defrag your NTFS. More info: http://www.ntfs-3g.org/support.html#dd

----------

