# P35 + >4GB issue?

## xibo

Hi,

I just inserted another 4GB of memory into my DP35DP based machine ( resulting in 6 GB total ). Afterwards I started Windows Vista and it took considerably longer as expected to start up ( abouth 10x longer ), but then ran at acceptable speed. Then I rebooted into linux ( vanilla kernel 2.6.22.9, vanilla gcc-4.2.1, emerge-patched glibc-2.6.1, everything compiled with '-march=nocona -O2 -fno-unit-at-a-time -pipe' ) ...

don't get me wrong, I do think it's a good thing to be able to read one's kernel's output, but I also think that is what dmesg is good for..... instead of 20 ( twenty ! ) minutes to reach init 5. I restarted and took a look at the bios setup hoping it would just have screwed the frequencies or timings but it appeared to be correct. Next, memtest told me everything was fine, too, same did windows' memory tester.

Now while 20 minutes startup times are disgusting me, I wouldn't bother if it were just startup times that increased... unlinke windows, here everything is slow...

```

dhcppc2 / # echo 'main(){printf("hello world!");}' > test.c

dhcppc2 / # time gcc test.c -O2

test.c: In function 'main':

test.c:1: warning: incompatible implicit declaration of built-in function 'printf'

real    0m8.939s

user    0m8.672s

sys     0m0.048s

```

I am used to see something that simple compile in ... at least faster than I can notice...

Anyone knows what I'm doing wrong?

Here's some foo:

 *cat /proc/version wrote:*   

> 
> 
> Linux version 2.6.22.9 (root@dhcppc0) (gcc version 4.2.1 (Gentoo 4.2.1 p1.0)) #6 SMP Mon Oct 8 11:59:53 Local time zone must be set--see zic m
> 
> 

 

 *cat /proc/cpuinfo wrote:*   

> 
> 
> processor       : 0
> 
> vendor_id       : GenuineIntel
> ...

 

 *cat /proc/meminfo wrote:*   

> 
> 
> MemTotal:      6038768 kB
> 
> MemFree:       5829456 kB
> ...

 

and finally I cant post /usr/src/linux/.config due to scrollback buffer limitations...

----------

## frenkel

What happens if you compile a file without warnings? Like this one:

```

#include <stdio.h>

int main() {

    printf("hello world!");

    return 0;

}

```

----------

## xibo

You mean if access to stdout/stderr or fid 1/2 are slow? It doesn't appear to be that:

Started with just the 4GB of memory ( 2x2048 Great Skill CL5-5-5-15 800 Mhz Dual DDR Mode ):

```

 cat test.c

#include <stdlib.h>

#include <stdio.h>

int main( int n_args, char **args )

 {

  printf("hello world!\n");

  return EXIT_SUCCESS;

 }

dhcppc2 / # time gcc -O2 test.c

real    0m0.044s

user    0m0.032s

sys     0m0.012s

dhcppc2 / # time ./a.out

hello world!

real    0m0.001s

user    0m0.000s

sys     0m0.000s

```

Now again with 6 GB  ( 2x2048 G.Skill CL5-5-5-15 800 Mhz Dual DDR Mode + 2x1024 G.Skill CL4-4-4-12 running as CL5-5-5-15 ( this the problem? ), 800Mhz Dual DDR Mode ):

```

dhcppc2 / # time gcc -O2 test.c

real    0m7.927s

user    0m7.040s

sys     0m0.046s

dhcppc2 / # time ./a.out

hello world!

real    0m0.317s

user    0m0.293s

sys     0m0.001s

```

----------

## frenkel

If you're using 32bit, a lot of memory mapping needs to be done to access anything beyond the 4GB barrier. That might be what is happening here.

----------

## xibo

Nope. It's running in 64 bit mode. At least sizeof(size_t)=sizeof(void*)=8, and I can allocate 3 gigabytes of memory for a single application.

----------

## eccerr0r

Are you sure your cache didn't get inadvertently turned off?  Check BIOS settings?

What's in your /proc/mtrr with 6G installed?

----------

## xibo

 */proc/mtrr wrote:*   

> 
> 
> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> 
> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> ...

 

The BIOS Setup does not feature anything about caching ( or at least I wasn't able to find it )

----------

## kernelOfTruth

is there a "memory remap" option ?

enable that, 

I just added 2 GB to existing 2 GB and running 32bit being able to access / use all 4 GB   :Smile: 

----------

## frenkel

 *kernelOfTruth wrote:*   

> is there a "memory remap" option ?
> 
> enable that, 
> 
> I just added 2 GB to existing 2 GB and running 32bit being able to access / use all 4 GB  

 

He's running in 64bit mode, so that's not needed.

----------

## kernelOfTruth

in my bios it says:

disable on 32bit (== 3 GB   :Sad:   )

enable on 64bit OS   :Laughing: 

----------

## a7thson

I feel your pain.  I'm experiencing similar performance degradation (from Core 2 Duo to Pentium Pro 200) on my MSI P6N E6600 running x86_64.  Doing some research online suggests this is an issue with MTRR configuration and the solution or workaround seems to be to make manual changes by directly writing to /proc/mtrr, and the problem occurs with RAM >=4GB.  Unfortunately it appears that system configurations will vary so I am not expecting this will be an easy issue to remedy.

https://forums.gentoo.org/viewtopic-t-563647-highlight-mtrr+mismatch.html is the related forum thread but I've been unable to just copy verbatim from that or another source for my system.  Also note that incorrect order in disabling MTRR areas can hard-lock and/or corrupt anything on-screen (confirmed by experience) so it could be a painstaking process.  At this point I'm considering simply yanking 2GB and holding off until I see an acceptable solution if I don't have better luck.  AFAIK there are no kernel configuration parameters that will affect or fix this.

Here are a few related threads, including the main one referenced in the forum post above:

http://ubuntuforums.org/showthread.php?s=bf9bfbe854cc786dd0ec20e8b122832b&t=303013&page=2

http://www.rage3d.com/board/showthread.php?t=33821469

http://osdir.com/ml/linux.gentoo.amd64/2005-10/msg00110.html

If you make any progress or find any of the above threads helpful please let us (me) know!  I'll do the same.

 *xibo wrote:*   

> Hi,
> 
> I just inserted another 4GB of memory into my DP35DP based machine ( resulting in 6 GB total ). Afterwards I started Windows Vista and it took considerably longer as expected to start up ( abouth 10x longer ), but then ran at acceptable speed. Then I rebooted into linux ( vanilla kernel 2.6.22.9, vanilla gcc-4.2.1, emerge-patched glibc-2.6.1, everything compiled with '-march=nocona -O2 -fno-unit-at-a-time -pipe' ) ...
> 
> don't get me wrong, I do think it's a good thing to be able to read one's kernel's output, but I also think that is what dmesg is good for..... instead of 20 ( twenty ! ) minutes to reach init 5. I restarted and took a look at the bios setup hoping it would just have screwed the frequencies or timings but it appeared to be correct. Next, memtest told me everything was fine, too, same did windows' memory tester.
> ...

 

----------

## stask

I had similar problem with x64 and 4G ram on DG965WH motherboard. This patch fixed the problem for me. Using it with 2.6.22-r8 now.

----------

## a7thson

Thanks for the link; unfortunately I haven't had a chance to try it as I'm testing 2.6.23 (gentoo-sources-2.6.23) but didn't find a corresponding patch, and the patch linked to didn't apply cleanly on .23

My configuration is a 256MB XFX 7900GS (nvidia) and MSI P6N (nforce-650 chipset) with MSI's shipped AMI BIOS.  Booting (and finding something to do for 20-30 minutes) the kernel results in this for /proc/mtrr:

```

reg0: base=0xc000 0000   (3072MB) size=1024MB uncacheable

reg1: base=0x0000 0000  (0MB)       size=4096MB write-back

```

This is terrible (and I wonder at the fact it is default behavior), as it 

1) creates an overlapping area at 3072MB mark, which furthermore is flagged UC

2) is not able to create the necessary 256MB write-combining cache for the 7900GS

I was able (after doing quite a bit of reading) to write to /proc/mtrr, first removing the reg1 mapping and replace it with a 2GB allocation (leaving a giant 2GB hole at 2048MB that would hopefully have removed the overlap):

```

echo "disable=1" >| /proc/mtrr && echo "base=0x0000000 size=0x80000000 type=write-back"

```

And this worked, displaying the new 2GB addressing on reg1.  Any attempt, though, to disable and rebuild the reg0 information results in a hard lock immediately on "disable=0".  This happened regardless of order, before or after setting up reg1.  I did manage to redefine reg0:

```

echo "base=0xc0000000 0x40000000 type=write-back" >| /proc/mtrr

```

And this was verified to have changed in /proc/mtrr, but still no improvement.  I even tried booting into it with kernel option "nomtrr" and not loading the nvidia drivers, but saw exactly the same allocation, with the exception that there was no error in dmesg for failure to set up the 256MB area.  System performance was just as slow, before and after doing the above manipulations again in that configuration.  If I'm doing something wrong, it's rather tough to fix, due to the hard-lock occurring.  For the time being I pulled the 2G of DDR2 and am again running happily with 2GB until I see that there's a solution to this.  If I decide to try 2.6.22 series I will let you know the results.

 *stask wrote:*   

> I had similar problem with x64 and 4G ram on DG965WH motherboard. This patch fixed the problem for me. Using it with 2.6.22-r8 now.

 

----------

## eccerr0r

Since it could be a mtrr issue and MTRRs should be setup by BIOS, try upgrading your BIOS/firmware to see if it helps.

I just got 4G into my G965 (Foxconn G9657MA, E6700) - it seems to take it in fine and I don't see much of a slowdown with PAE enabled.  Its MTRRs look like:

```
reg00: base=0x100000000 (4096MB), size= 512MB: write-back, count=1

reg01: base=0x120000000 (4608MB), size= 256MB: write-back, count=1

reg02: base=0x00000000 (   0MB), size=2048MB: write-back, count=1

reg03: base=0x80000000 (2048MB), size=1024MB: write-back, count=1

reg04: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1

reg05: base=0xcf800000 (3320MB), size=   8MB: uncachable, count=1

reg06: base=0xcf700000 (3319MB), size=   1MB: uncachable, count=1

```

Apparently there are no overlaps here.

Kind of weird with that PCI address hole.  Memory map looks like (for this G965)

0-3.25GB - RAM.  PAE not needed for this segment, visible with a 4G kernel

3.25G-4GB - PCI IO (includes uncacheables)

4GB-4.75GB - rest of RAM, needs PAE to access

Note that this is likely different from machine to machine.

----------

