# Problem with 3.6TB large drive.

## viol8r

I'm having a problem with a Promise VTrack 15100 storage server (scsi external storage). We recently bought 4 of these, but we are having difficulties accessing them in Linux.

The setup is a Promise vtrack 15100 with 15 300GB Seagate 7200 SATA drives configured in a big RAID5 with 14 drives and 1 hotspare (resulting in a big 3.6TB drive). The SCSI controller is an Adaptec 29160. We tried using Gentoo (and Red Hat Fedora, Suse and Knoppix) - all with the same result and running both 32 and 64-bit architectures.

We tried several configurations, different partitioning programs, but in the end it all gives the same problem - we can only use 1.6TB of the device.

This is what gets detected during boot using a 2.6.12 based linux kernel:

```

scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36

        <Adaptec 29160 Ultra160 SCSI adapter>

        aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs

  Vendor: Promise   Model: 14 Disk RAID5     Rev: V0.0

  Type:   Direct-Access                      ANSI SCSI revision: 04

scsi0:A:0:0: Tagged Queuing enabled.  Depth 32

 target0:0:0: Beginning Domain Validation

WIDTH IS 1

(scsi0:A:0): 6.600MB/s transfers (16bit)

 target0:0:0: Domain Validation skipping write tests

(scsi0:A:0): 160.000MB/s transfers (80.000MHz DT, offset 62, 16bit)

 target0:0:0: Ending Domain Validation

SCSI device sda: 3795896832 1024-byte hdwr sectors (3886998 MB)

SCSI device sda: drive cache: write through

SCSI device sda: 3795896832 1024-byte hdwr sectors (3886998 MB)

SCSI device sda: drive cache: write through

 sda: sda1

Attached scsi disk sda at scsi0, channel 0, id 0, lun 0

```

Using fdisk (we calculate the number of cylinders as 3795896832 / 16065

= 236282, since fdisk can't figure it out):

```

livecd ~ # fdisk /dev/sda

Note: sector size is 1024 (not 512)

You must set cylinders.

You can do this from the extra functions menu.

Command (m for help): x

Expert command (m for help): c

Number of cylinders (1-1048576): 236282

The number of cylinders for this disk is set to 236282.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

   (e.g., DOS FDISK, OS/2 FDISK)

Expert command (m for help): r

Command (m for help): p

Disk /dev/sda: 0 MB, 0 bytes

255 heads, 63 sectors/track, 236282 cylinders

Units = cylinders of 16065 * 1024 = 16450560 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1               1      236282  3795870267   83  Linux

Command (m for help):

```

All things look good, but now comes the real problem. No matter what

file system we choose (reiserfs, xfs, jfs etc.) only gives us 1.6TB.

```

livecd ~ # mkfs.jfs /dev/sda1

mkfs.jfs version 1.1.7, 22-Jul-2004

Warning!  All data on device /dev/sda1 will be lost!

Continue? (Y/N) Y

   \

Format completed successfully.

1648386619 kilobytes total disk space.

```

As you can see there is 2TB missing. df and other programs says the

same.

Are there any reasons for this? And more importantly - what's the

solution?

Output of blockdev command:

```

livecd ~ # blockdev

Usage:

  blockdev -V

  blockdev --report [devices]

  blockdev [-v|-q] commands devices

Available commands:

        --getsz (get size in 512-byte sectors)

        --setro (set read-only)

        --setrw (set read-write)

        --getro (get read-only)

        --getss (get sectorsize)

        --getbsz        (get blocksize)

        --setbsz BLOCKSIZE      (set blocksize)

        --getsize       (get 32-bit sector count)

        --getsize64     (get size in bytes)

        --setra READAHEAD       (set readahead)

        --getra (get readahead)

        --flushbufs     (flush buffers)

        --rereadpt      (reread partition table)

livecd ~ # blockdev --getsz /dev/sda

livecd ~ # blockdev --getsize64 /dev/sda

3886998355968

livecd ~ # blockdev --getsize /dev/sda

BLKGETSIZE: File too large

livecd ~ # blockdev --getss /dev/sda

1024

```

Note that the running kernel has "Support for Large Block Devices"

enabled.

Any ideas?  :Wink: 

----------

## .:chrome:.

do a test with another file system (for example XFS)

----------

## viol8r

 *k.gothmog wrote:*   

> do a test with another file system (for example XFS)

 

As I wrote - I get the exact same results with reiser, jfs and xfs - only 1.6TB free.

----------

## .:chrome:.

 *viol8r wrote:*   

> As I wrote - I get the exact same results with reiser, jfs and xfs - only 1.6TB free.

 

sorry   :Embarassed: 

----------

## Dlareh

Is it possible for you to make individual filesystems on, say, four .9 TB partitions?  Combine the partitions with a linear (dm?) raid or lvm ?

----------

## viol8r

 *Dlareh wrote:*   

> Is it possible for you to make individual filesystems on, say, four .9 TB partitions?  Combine the partitions with a linear (dm?) raid or lvm ?

 

Perhaps, but it would be much easier to have it as one big partition - and really - it's 2005 - Linux should be able to handle a partition bigger than 2TB...  :Wink:  Windows has no problem with it   :Evil or Very Mad: 

----------

## Dlareh

 *viol8r wrote:*   

>  *Dlareh wrote:*   Is it possible for you to make individual filesystems on, say, four .9 TB partitions?  Combine the partitions with a linear (dm?) raid or lvm ? 
> 
> Perhaps, but it would be much easier to have it as one big partition - and really - it's 2005 - Linux should be able to handle a partition bigger than 2TB...  Windows has no problem with it  

 

Well it might narrow down the problem, if you would try...

----------

## viol8r

 *Dlareh wrote:*   

> 
> 
> Well it might narrow down the problem, if you would try...

 

 :Wink: 

Ok - i tried doing it with two 1.8TB partitions and it worked. 

```

livecd ~ # mkfs.jfs /dev/volume_group/logical_volume

mkfs.jfs version 1.1.7, 22-Jul-2004

Warning!  All data on device /dev/volume_group/logical_volume will be lost!

Continue? (Y/N) Y

   \

Format completed successfully.

3795861504 kilobytes total disk space.

```

Now - anybody with any idea of why it doesn't work when theres one big regular partition? 

I really don't want to use LVM and other workarounds on a production server...   :Crying or Very sad: 

----------

## Sachankara

As always, people forget to check their kernel configuration. If one wants to run partitions larger than 2TiB, it must be enabled in the kernel.

Device Drivers --> Block devices --> Support for Large Block Devices

----------

## viol8r

 *Sachankara wrote:*   

> As always, people forget to check their kernel configuration. If one wants to run partitions larger than 2TiB, it must be enabled in the kernel.
> 
> Device Drivers --> Block devices --> Support for Large Block Devices

 

As always, people forget to read the first post...  :Very Happy: 

 *viol8r wrote:*   

> Note that the running kernel has "Support for Large Block Devices" 
> 
>  enabled.

 

I tried hooking it up to a running Gentoo box with it surely enabled and I do believe that the kernel on the 2005.1 live cd has it enabled by default too, but I can't get the filesystem bigger than 2TB (if I make it 3.6TB it gets 3.6TB-2TB=1.6TB, so it seems to be a signed int problem or something somewhere   :Crying or Very sad:   )

----------

## Dlareh

I didn't think it would be a 2GB issue since it topped out at 1.6GB, but the fact that you were able to create two 1.8GB pretty much guarantees that the guy below me is correct.

----------

## frenkel

```

$ man fdisk

<snip>

       In a DOS type partition table the starting offset and the size of  each

       partition  is  stored  in  two  ways:  as an absolute number of sectors

       (given in 32 bits) and as a Cylinders/Heads/Sectors  triple  (given  in

       10+8+6  bits).  The former is OK - with 512-byte sectors this will work

       up to 2 TB. The latter has two different problems. First of all,  these

       C/H/S fields can be filled only when the number of heads and the number

       of sectors per track are known. Secondly, even if we  know  what  these

       numbers  should be, the 24 bits that are available do not suffice.  DOS

       uses C/H/S only, Windows uses both, Linux never uses C/H/S.

<snip>

```

So this is a problem with fdisk not being able to handle it, I'm searching for a solution now.

http://www.google.nl/search?hl=en&q=fdisk+2TB

Edit:

Ok, so I found out you need to make a GPT disk label or something. This can currently only be done with parted (emerge parted, also installed on the livecd). Here's some more info about it:

http://www.wlug.org.nz/GPT

Also have a look at this discussion, this guy solved it by using parted:

http://lists.debian.org/debian-user/2005/07/msg03555.html

----------

## viol8r

 *Frenkel wrote:*   

> 
> 
> Ok, so I found out you need to make a GPT disk label or something. This can currently only be done with parted (emerge parted, also installed on the livecd). Here's some more info about it:
> 
> http://www.wlug.org.nz/GPT
> ...

 

Ok - it seems to work!

Using parted:

```
livecd ~ # parted /dev/sda

You found a bug in GNU Parted.  Please email a bug report to bug-parted@gnu.org containing the version (1.6.20), and the following message:

The sector size on /dev/sda is 1024 bytes.  Parted is known not to work properly with drives with sector sizes other than 512 bytes

Ignore/Cancel? I

GNU Parted 1.6.20 with HFS shrink patch 16

Copyright (C) 1998 - 2004 Free Software Foundation, Inc.

This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS

FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

Using /dev/sda

(parted) mklabel gpt

(parted) mkpart primary 0 -0

(parted) p

Disk geometry for /dev/sda: 0.000-3706930.500 megabytes

Disk label type: gpt

Minor    Start       End     Filesystem  Name                  Flags

1          0.017 3706930.483 

(parted) q

Information: Don't forget to update /etc/fstab, if necessary.

```

Putting JFS on it:

```
livecd ~ # mkfs.jfs /dev/sda1

mkfs.jfs version 1.1.7, 22-Jul-2004

Warning!  All data on device /dev/sda1 will be lost!

Continue? (Y/N) Y

   \

Format completed successfully.

3795896798 kilobytes total disk space.

```

Mounting it to check df and free space.

```
livecd ~ # mount /dev/sda1 /mnt/gentoo

livecd ~ # df

Filesystem           1K-blocks      Used Available Use% Mounted on

tmpfs                   257740      3672    254068   2% /

/dev/cdroms/cdrom0      695512    695512         0 100% /mnt/cdrom

/dev/loop/0              42176     42176         0 100% /mnt/livecd

udev                    257740       700    257040   1% /dev

tmpfs                   257740      3584    254156   2% /lib/firmware

tmpfs                   257740         0    257740   0% /usr/portage

/dev/sda1            3795747980    463936 3795284044   1% /mnt/gentoo

```

Looks like it should!

Let's just check what fdisk now says

```

livecd ~ # fdisk /dev/sda

Note: sector size is 1024 (not 512)

You must set cylinders.

You can do this from the extra functions menu.

Command (m for help): x

Expert command (m for help): c

Number of cylinders (1-1048576): 236283

The number of cylinders for this disk is set to 236283.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

   (e.g., DOS FDISK, OS/2 FDISK)

Expert command (m for help): r

Command (m for help): p

Disk /dev/sda: 0 MB, 0 bytes

255 heads, 63 sectors/track, 236283 cylinders

Units = cylinders of 16065 * 1024 = 16450560 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1               1      267350  4294967295   ee  EFI GPT

Command (m for help):

```

Nice - THANKS!

Next problem is probably getting grub (or something else) to boot from GPT  :Wink: 

----------

## frenkel

Indeed, or you might stick a small drive in it, or even a floppy, with grub and a kernel, when the kernel is loaded, it can mount the GPT label and us it as root filesystem.

Glad it finally works  :Smile: 

----------

