# Hyperthreading and SMP

## yohnson

I recently have installed Gentoo 2007.0 on a SuperMicro X5DP8-G2 Dual Xeon 2.8Ghz computer.

Being a multiprocessor I had enabled SMP and Hyperthreading while building the kernel 2.6.24-r4, everything worked great for about 3 hours then a system freeze would occur (system idle during this time).  Ironically enough /proc/cpuinfo was reporting 4 processors and there were no errors to be seen anywhere (dmesg or *.log).

Looking around I discovered in the BIOS Hyperthreading was enabled.  So with a kernel having Hyperhreading and the BIOS enabled hyper threading I guess there was a conflict of 'interest'.

The question is.  Performance & stability wise, which makes more sense:

 *BIOS Hyperthreading enabled and build a SMP kernel w/o HT  

 *BIOS Hyperthreading disabled and build a SMP kernel with HT 

Logic would say having the hardware disseminate the threads would produce higher performance although, this is countered by my intuitive nature that the BIOS does not know of the tasks at hand.  Therefore the kernel has the ability to schedule more appropriately so turning off HT in the BIOS is best.

Some background on this particular study, the computer's purpose is an email and LAMP server

```
vendor_id       : GenuineIntel

cpu family      : 15

model           : 2

model name      : Intel(R) Xeon(TM) CPU 2.80GHz

stepping        : 7

cpu MHz         : 2799.312

```

Interesting question and I wonder what the community thinks, what are your thoughts or experiences?

----------

## pilla

The BIOS setting turns on/off the HT. The kernel setting turns on/off the kernel support for HT. It doesn't make sense to have BIOS on and kernel off.

----------

## yohnson

 *pilla wrote:*   

> The BIOS setting turns on/off the HT. The kernel setting turns on/off the kernel support for HT. It doesn't make sense to have BIOS on and kernel off.

 

I'm confused because with the BIOS HT off and Kernel HT on, wouldn't this mean HT is not going to happen because the BIOS won't support it?

So it makes sense to have the HT in BIOS off and the kernel HT on but never both?? I say this because I had researched the freezing issue and found many discussions where people found turning off the BIOS HT fixed crash/freeze issues on SMP kernels.

With this computer, to have the BIOS and Kernel with HT enabled, system freeze occurs hours later and number of CPUs is reported by cpuinfo is 4. 

With HT off in BIOS and kernel with SMP and HT enabled cpuinfo reports 2.

I guess I misunderstood the purpose of the BIOS HT option then...I was thinking the motherboard would handle the HT for the OS installed.

----------

## Carnildo

Hyperthreading is a "fake" second CPU: it takes CPU cycles where one program is waiting on something like memory access, and uses them to execute a second program.  It looks like and can be treated as a second CPU, but only provides the performance of about an additional 5% of a CPU.

Turning off BIOS HT will disable this fake second CPU, at the cost of that 5% extra performance.  I'm not sure what the kernel HT support does -- it either tells the kernel that there's a second CPU per core that it can use, or it tells the kernel that two of the four CPUs it sees are actually HT slight-of-hand, I'm not sure which.

----------

## Mad Merlin

You need HT enabled in both the BIOS and the kernel to take advantage of it, turning off one but not the other doesn't make any sense.

The BIOS option is provided mainly for legacy OSes where you can't toggle HT support for the kernel with a simple recompile.

----------

## yohnson

 *Mad Merlin wrote:*   

> You need HT enabled in both the BIOS and the kernel to take advantage of it, turning off one but not the other doesn't make any sense.
> 
> The BIOS option is provided mainly for legacy OSes where you can't toggle HT support for the kernel with a simple recompile.

 

Ok so the answer is both in the BIOS and Kernel HT must be enabled to have HT work.

Circling back to the instability of having it enables in both the BIOS and Kernel. Provided that you must have HT enabled in both places...  Why would the OS then become unstable and freeze (which happened in my case) with /proc/cpuinfo reporting a different number of CPUs?

----------

## pilla

With HT enabled, it is OK to have twice as much processors in cpuinfo as you really have -- the OS sees HT as an extra processor for each processor. 

As for the instability, that can be millions of things.

----------

## eccerr0r

AFAIK the HT option in the kernel will tell the scheduler that the cores that it can schedule upon, to not choose the same physical core when running two threads.  I imagine this will not have way too much effect (other than to try to avoid cache thrash when scheduling lightweight threads) on a single socket/single core machine, but on a multi-socket/multi-core machine, submitting two threads to the same core is not optimal at all - submitting the two threads to different sockets/cores will improve throughput.  I think Linux will still show both threads as distinct cores without kernel HT support, but be oblivious to the scheduling problem.

I've found a lot of sudden machine lockups turn out to be mainboard issues but it's not always the case.

BIOS hyperthreading enabling has nothing to do with Linux hyperthreading awareness, it just enables hyperthreading in the CPUs.  Theoretically, if your machine hangs with Linux HT=on and BIOS HT=on, it should also fail with Linux HT=off and BIOS HT=on.  See if this is the case, this should also show up as 4-cores but overall performance may be lower.

I haven't seen any issues with my P4-Proc 650 in hyperthreading mode (I see two threads on my 1-socket/core system), so Linux should be sound and likely pointing towards a hardware issue.

----------

## pilla

 *eccerr0r wrote:*   

> AFAIK the HT option in the kernel will tell the scheduler that the cores that it can schedule upon, to not choose the same physical core when running two threads. 

 

Not true. HT is the buzz word from Intel for SMT (Simultaneous Multi Threading), a kind of processor where you use your available and mostly idle resources to run more than a single thread in a processor. Thus, you can hide latencies from one thread by running stuff from the other threads while it sits waiting for a load, for example. If you have two cores, each one with the capability of running two threads, then it's wise to tell the scheduler to keep two threads that will cooperatively use cache on the same processor, and not the contrary, if there are threads from other processes to run on the other processor. However, keep in mind that running two threads in the same processor will make them share/compete for resources, so for single-thread applications SMT/HT may not incur in performance gains -- it may even mean losses.

This site has many papers, slides and other material about hyperthreading and simultaneous multithreading.

----------

## eccerr0r

Right, it depends on the threads you're trying to run.  I shouldn't have said "core" in that first part of the statement as it may confuse people - I actually meant the "thread execution unit" or "virtual processor." - specifically on HT/SMT machines where physical cores is less than virtual processors.  But what I said is correct, the HyperThread "SCHED_SMT" option in the kernel is to help the scheduler decide which resources are truly 'free' and use that to run additional workload instead of using the same core and thus lowering throughput.  SMT looks a lot like SMP to the software, so it's easy for it to get confused.

For a typical user workload where you're running lots of different programs, sending the coarse threads ("Processes") to distinct cores is the more optimal solution due to cache thrash and execution unit resource sharing.  If however your lightweight threads do a lot of Inter-Process Communication then *potentially* running them on the same socket *may* be faster than running it with snoop invalidates spread across sockets/cores.  It all depends on the workload, and running two processes on the same core with HT is always slower than running two processes on different cores, if available.

----------

## richard.scott

Hi,

I have an issue with the same CPU and HT on in both the bios and the kernel that seems to resolve with switching it off in the BIOS.

My issue resolves only about uncompressing tar.bz2 files where I get this sort of error:

```
p4 ~ # emerge hardened-sources

Calculating dependencies... done!

>>> Verifying ebuild Manifests...

>>> Emerging (1 of 1) sys-kernel/hardened-sources-2.6.25-r8 to /

 * hardened-patches-2.6.25-9.extras.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...   [ ok ]

 * genpatches-2.6.25-11.extras.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...           [ ok ]

 * linux-2.6.25.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...                                   [ ok ]

 * genpatches-2.6.25-11.base.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...             [ ok ]

 * checking ebuild checksums ;-) ...                                                                [ ok ]

 * checking auxfile checksums ;-) ...                                                               [ ok ]

 * checking miscfile checksums ;-) ...                                                              [ ok ]

 * checking linux-2.6.25.tar.bz2 ;-) ...                                                             [ ok ]

 * checking hardened-patches-2.6.25-9.extras.tar.bz2 ;-) ...                             [ ok ]

 * checking genpatches-2.6.25-11.base.tar.bz2 ;-) ...                                       [ ok ]

 * checking genpatches-2.6.25-11.extras.tar.bz2 ;-) ...                                     [ ok ]

>>> Preparing to unpack ...

>>> Unpacking source...

>>> Unpacking linux-2.6.25.tar.bz2 to /var/tmp/portage/sys-kernel/hardened-sources-2.6.25-r8/work

bzip2: Data integrity error when decompressing.

        Input file = /var/tmp/portage/sys-kernel/hardened-sources-2.6.25-r8/distdir/linux-2.6.25.tar.bz2, output file = (stdout)

It is possible that the compressed file(s) have become corrupted.

You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover

data from undamaged sections of corrupted files.

tar: Unexpected EOF in archive

tar: Unexpected EOF in archive

tar: Error is not recoverable: exiting now
```

How can the file downloads cleanly, pass the integrity checks, yet bzip2 can't unpack it??

----------

