# System is very slow since a few month (complex setup, RAID)

## doublehp

Hello.

I installed the machine in 2010. Was fast and good.

In 2014 I changed the disks to get more space. I move files, and data, without reinstalling any software. But, I changed one detail: the HOME partition was changed from EXT4 to ZFS. I now have 5 HDD.

Since that disk change, the system got a bit slower. But it is now significantly slower than before. And I don't understand why.

Long story short: Gentoo is on RAID6 over 5 disks. HOME is on ZFS raidz2 over 5. BIG (say, /mnt/tmp) is on a more complex setup; but in short, it's also raid6 over 5 disks.

1: all apps are slower and slower. But ... deadly slow. Rox-filer used to be able to open a folder with 200 pictures in 3 or 5s. Now, it needs about 1s to generate each preview for each picture. That's about 3mn for a 200 pics folder, while it used to be below 10s.

2: some apps freese for long time; E17 shows blinking red decorations

3: some times, the whole X will freese (mouse wont move) for 20s, or up to 8mn. 

When things are slow, or frozen, the load increases hugely (from 0.6, in average, to 2, 5, or  :Cool: . But ... the CPU remains usually 75, or even 92% iddle. And the HDD led blinks very slowly. Like 3 dots per seconde.

```
519# for n in a d e ; do hdparm -tT /dev/sd$n ; done

/dev/sda:

 Timing cached reads:   6230 MB in  2.00 seconds = 3115.74 MB/sec

 Timing buffered disk reads:  402 MB in  3.01 seconds = 133.72 MB/sec

/dev/sdd:

 Timing cached reads:   6414 MB in  2.00 seconds = 3207.49 MB/sec

 Timing buffered disk reads:  454 MB in  3.00 seconds = 151.29 MB/sec

/dev/sde:

 Timing cached reads:   7010 MB in  2.00 seconds = 3506.11 MB/sec

 Timing buffered disk reads:  368 MB in  3.00 seconds = 122.50 MB/sec

0 0 2015-12-04_17-21-14 17:20:21 @pts/3 root@uranus:/tmp 

520# hdparm -tT /dev/mapper/Big_2014-Big_2014 

/dev/mapper/Big_2014-Big_2014:

 Timing cached reads:   8128 MB in  2.00 seconds = 4065.51 MB/sec

 Timing buffered disk reads:  750 MB in  3.03 seconds = 247.82 MB/sec

0 0 2015-12-04_17-22-09 17:21:48 @pts/3 root@uranus:/tmp 
```

During these tests, the HDD led was  100% on.

These were raw hardware values.

Now, let's see what's after filesystem:

```
511# dd if=/dev/zero of=/tmp/plop conv=sync bs=1M count=1k   # SYSTEM

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 11.2483 s, 95.5 MB/s

0 0 2015-12-04_17-14-24 17:14:12 @pts/3 root@uranus:/tmp 

512# o=/mnt/big/tmp/plop ; dd if=/dev/zero of="$o" conv=fsync bs=1M count=1k ; rm "$o"        # BIG

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 68.9786 s, 15.6 MB/s

0 0 2015-12-04_17-16-12 17:15:03 @pts/3 root@uranus:/tmp 

513# o=/home/plop ; dd if=/dev/zero of="$o" conv=fsync bs=1M count=1k ; rm "$o"       # HOME

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 7.20762 s, 149 MB/s
```

Each disk can perform around 150MB/s; or, at least 120MB/s. Considering raid6, the theoretical speed should be between 450 and 360MB/s. BIG at array level got 240MB/s, and ... about 30MB/s after file system (ext4).

How can such a loss be possible ?

In the end, ZFS is the fastest one, with 150MB/s (instead of 360-450 !!!).

I have also booster my RAM, from 4GB, to 20GB.

The system was deadly faster than this in 2010. Any filesystem could perform at least 180MB/s, but, more genrally 230MB/s. In 2010, no test on filesystems were below 160MB/s. 

My use ratio are: (space / inodes)

system: 47% / 23%

BIG:      63% / 1% 

Home:    45% / 1%

What the hell could make the system so slow, when ...

- it used to be 1000% faster in the past

- it's still reasonably fast on block level

- I have no SMART error (or any other kind of error in any log I have)

- I have free space (and free inodes)

- I am not strugnling on CPU

- I am not strugling on bus

- I have very low IO

- I don't know which apps may need to write, or read

- some times, I have heavy disk access, but I have no clue which app are making them.

My typical use is: Thunderbird, Firefox, Pidgin, Chrome. Sometimes OpenOffice. The whole lot used to be fluent 5 years ago with 4GB RAM, and 4 disks in raid 5; and is now slow with 5 times more RAM.

I constantly look at top, and always have at least 8GB RAM free. Thunderbird has leaks, I restart it every 4 or 8h. And I reboot the whole system at least once a day, as soon as the free RAM comes down to 0.

This is not a 486 with 64k RAM; if the issue was apps eating ressources, I should see low free memory, or low iddle CPU.

I can run vmstat, and most other IO monitoring apps, but not iotop, I don't have the symbol in my kernel, and can not update my kernel.

I am getting mad with this slow machine. When it's freesing, even network traffic stops. Wait 3mn, and things come back to normal.

I need any help to track the root issue.

----------

## Keruskerfuerst

Detailed hardwareinfo?

----------

## eccerr0r

And what kernel version?

Why can't you compile the needed code into your kernel for iotop?  It should be there in recent kernels...

----------

## doublehp

 *Keruskerfuerst wrote:*   

> Detailed hardwareinfo?

 

What details do you want ?

----------

## doublehp

 *eccerr0r wrote:*   

> And what kernel version?
> 
> Why can't you compile the needed code into your kernel for iotop?  It should be there in recent kernels...

 

2.6.34-xen

Rebuilding is very complicated. If I try to change kernel, I have little chances to make new kernel bootable, and, also possibilities to make the system stop booting at all. Fixing a broken system may take days of work. And I don't have days to loose on this.

----------

## eccerr0r

Only guess now is fragmentation, if your disk has a lot of turnover from bittorrenting or something...?

----------

## doublehp

 *eccerr0r wrote:*   

> Only guess now is fragmentation, if your disk has a lot of turnover from bittorrenting or something...?

 

ZFS can not fragment.

For ext4: 0.2%

----------

## TigerJr

If you have xen kernel, than you can have xen guests, many guests - many iops. why you can't debug io rates with snmp or other monitoring systems? Or even with sarg and shell script mrtg or gnuplot with simple webserver to understand what is making high IO rate or latency bottleneck or raid had degraded due to disk fail?

----------

## eccerr0r

 *doublehp wrote:*   

>  *eccerr0r wrote:*   Only guess now is fragmentation, if your disk has a lot of turnover from bittorrenting or something...? 
> 
> ZFS can not fragment.
> 
> For ext4: 0.2%

 

Yeah, right.  Fragmentation resistance does not equate to "can not fragment".  Unless there's an in-OS/in-kernel auto defragmenter there is always a degenerate write pattern that forces fragmentation.

----------

## Keruskerfuerst

ZFS has a fragmentation program.

Detailed hardware info means:

CPU(s)

Mainboard

RAM; type

Graphics card

Harddisk controller

Harddisks (HDD or SSD)

----------

## doublehp

 *TigerJr wrote:*   

> If you have xen kernel, than you can have xen guests, many guests - many iops. why you can't debug io rates with snmp or other monitoring systems? Or even with sarg and shell script mrtg or gnuplot with simple webserver to understand what is making high IO rate or latency bottleneck or raid had degraded due to disk fail?

 

I don't have any guest. Usually, just after boot, if I don't touch anythink, the disk may stay iddle for seconds, or tens of second (apart from the 5s sync of ext). I don't have background heavy work.

ZFS works a way that does not fragment files. And, it has a background daem that cleans the filesystem when system is iddle (daemon is always on, but does work only when no writes are made for a long time). So, ARC is preventing fragmentation, and, ARC has a design that can not make it responsible for system slowliness.

 *Quote:*   

> Yeah, right. Fragmentation resistance does not equate to "can not fragment". Unless there's an in-OS/in-kernel auto defragmenter there is always a degenerate write pattern that forces fragmentation.

 

Yes, ZFS is very special about this (and many more interesting aspects). I have askes ZFS people, and ZFS is correctly configured in my machine. Maybe I could increase the memory afforded to ARC (I could double it, since I have x5 my SDRAM).

 *Quote:*   

> Detailed hardware info means:
> 
> CPU(s)
> 
> Mainboard
> ...

 

To make it quick, I take this from lshw (not the most detailed, but there will be short answers to your question)

 *Quote:*   

> 
> 
>   *-core
> 
>        description: Motherboard
> ...

 

All 5 disks are identical, with consecutive serial numbers. They are pluged in hotplug sata mode (not IDE compatible), all on MB, and hardware RAID manager (of motherboard) is not used.

 *Quote:*   

> 
> 
> # lspci | grep VGA
> 
> 01:05.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RS880 [Radeon HD 4200]
> ...

 

Configured with Xinerama, and partial Xrandr (on first card). I forgot details of Xcrossover configuration.

Xinerama is the part that took me 6 months to configure, and get working, because ... X team still not implemented in 2010 what they promised to to in 2004. I spent 3 years in finding cards that would work, and let me have 6 monitors for a reasonable price. That's the point that makes the setup VERY FRAGILE. Any change in kernel configuration, BIOS configuration, or file or driver or firmware change around VGA will break the system, from ... getting freese (double panic) when starting X, to ... kernel not booting at all. If won't try to rebuild the kernel; because, if I just forget any small detail during the rebuild process, X will refuse to start (and may even freese the kernel, what end up in breaking the file systems: when you have more than 40 double panic per day, and have to press RESET button each time, the file system end up in getting corrupt). I reported all these issues against Linux, X, ATI drivers, and nobody care, because I am obviously the only guy on earth who want's to have 6 monitors in Xinerama.

X will also break on udev or sysinit update.

Staring alternate distribution also breaks the system. Booting Arch Linux and Debian usually messes MDADM chains. They are easier to fix than X; only takes half an hour to fix, and don't have side effect.

And this was the short version. For the long story, search all the bugs I created, or joined, in Gugzilla ... and don't forget the closed ones. In there you will find my detailed X configuration, logs, and crash logs.

But once again: the whole system was fast and fluent untill 2013. Things slown down after disk change; and since then, they get worst and worst each month. I stopped updating Gentoo in octobre 2010. It was working; no reason to change stuff that work.

Right now, my load is quiet low: 0.16, 0.25, 0.27 . System is up since 90mn. Few things are running (TB, FF, Chrome), and system is fluent.

----------

## TigerJr

mdadm? you didn't use hardware raid???

----------

## doublehp

I had an idea.

There is one thing that grows up with time: my Gmail box. Each email sent is copied into SentFolder on local, and, due to SMTP relay being gmail, it's also in Sent folder on TWO Gmail accounts. I am cleaning the online account for the box used for outgoing messages (the account configured for the SMTP relay). The account (on local disk, for cache) was 18G. And is now 5G. But there are still surprises. After heavy cleaning, thunderbird tells me the AllMail folder has a "size on disk 65MB", but, via terminal, after compacting ... it's rather doing ... 2.5G. 

I have yet to understand why Thunderbird things that 2.5GB is 65MB ... 

 *Quote:*   

> 2.5G    [Gmail].sbd/All Mail
> 
> 16M     [Gmail].sbd/All Mail.msf
> 
> 2.5G    [Gmail].sbd/Sent Mail
> ...

 

Since my sync are done at random times ... and since this is stored on ZFS, and since ZFS has a strange mood for duplicating files (to be able to take snapshots), for every single email sent, and stored by Gmail online, if Thunderbird wants to add a message in the middle of an mbox file (MSF), it MAY be possible that ZFS was trying to duplicate a 9GB file ... for every single message new. This could be done in the background, when TB triggers a sync timeout (every 10mn from memory).

I have tried "compacting" and "repair folder"

Also, panacea.dat refers to both 

ImapMail/imap.gmail.com/[Gmail].sbd/All Mail.msf

and

ImapMail/imap.gmail.com/[Gmail]-1.sbd/All Mail.msf

and does not explain the difference.

But there is an account I can not clean up because Gmail will not let me filter (search) messages that are in AllMail, but not in any other directory. You can not search "unlabelled" messages.

And Even a thunderbird issue would not explain why my filesystem is so slow (when TB is closed).

 *Quote:*   

> mdadm? you didn't use hardware raid???

 

No; Hardware Raid is not flexible enough, has too many bugs, and is not portable at all. If your MB dies, you loose all data. Can not expand (move from RAID6/5 to raid6/6) or change raid profile. If one sata plug dies, can not move disk to IDE or USB plug. And it masks disks completely, so you can not monitor SMART anymore.

Right NOW, E17 shows thunderbird with red blinking decoration, but, rest of system is fluent, and, disk blinks slowly: 1 flash per second. So, app is rozzen, but, one core of my CPU is at 99%; all other cores are 75% iddle. I don't know what it's doing. That never happened before 2014. ... and 3mn later, hop, app works normally.

----------

## doublehp

Took 315 seconds to flush bin that contained 132,862 messages ... from Gmail Webmail.

Even the largest computer in the world was in trouble to flush my account. No wonder my small desktop was slow managing it  :Smile: 

Never saw Google spend more than 5s on any request.

One account left ... but don't know how to clean it. I have too many messages, in too many folders to make a copy of it.

----------

## schorsch_76

It doesnt help to cry. If you want to do something, do it. 

Take a full backup and keep your current kernel as a fallback. Then try a newer kernel and try to improve. If you cant do a full backup, then try to backup the system and keep your data save. 

It is always a bad sign, if the one who installed/build something doesnt want to touch it, because it could break.  :Wink: 

----------

## doublehp

 *schorsch_76 wrote:*   

> If you want to do something, do it. 

 

- I want to know why the system is slower than before.

- I want to know why the system freeses from time to time (for 10s to 10mn).

- I want to know why writing on filesystem is significantly slower than writing on block (I am used to loose 10 or 20% speed; I have lost 90 or 95%).

- I want to know why the system may be unresponsive for about 5mn, when CPU is 75% iddle, and disk flashes two or three short flashes per second (heavy writes produce a continuous light on the HDD LED).

I just have no clue how to dig these issues.

I am shy rebuilding a kernel, because the last 3 times I did it, it took me a full week to fix the system afterwards (a full week spending 12h per day on my computer), because of udev and X bugs (that have not been fixed since 4 to 15 years - yes recent Xorg still trigger bugs I reported in 2002, dispite they pretend a complete rewrite ... twice).

And yes, I am out of nerves. Some days, I have work to do  on computer, and it's so exhausting for me to wait 5mn doing nothing looking foe mouse refusing to move, I just go to bed without doing work. And next day ... work is not done.

And I won't do a complete reinstall of the system, because it took me 6 months to set it up (2 full months not doing anything else than building packages and configuring /etc, 10 to 12h a day, not doing any thing else, 6 days per week).

New kernel means ... udev update, means, break the whole box. Reverting to revious kernel will required udev revert, and considering udev update implies messing many files in /etc ... you know it's not something that can be done. Ortherwise, I would need to make a complete snapshot of the system at ZFS level, stop mounting user volumes, spend several days on trying to make it work, and if it does not, revert to previous snapshot of filesystem. This means ... several days without logging in as user, and not going out for work. I have too much work out of home to tell people I will take 3 days to fix my computer. 

Yes I have lost patience. I want to fix the issue. Not shoot blindly with a tank and hope that some random actions may help. I want to dig the root cause of problem, and fix the issue where it is. I am pretty certain there are a lot of things that can be done without rebuilding kernel. I just don't know what. Not sure if pushing syslog on a network server could help; pushing kernel logs to serial port is usually much more reliable (directly talk to hardware, less software layers; better chances to receive messages). But I don't know how to enter heavy debug mode, or how to debug modules or scheduler. But during system freeses, kernel is still up and running, because I see slow disk activity; so there are some things to do at least at kernel level.

----------

## TigerJr

 *doublehp wrote:*   

>  - I want to know why the system is slower than before. 
> 
>  - I want to know why the system freeses from time to time (for 10s to 10mn). 
> 
>  - I want to know why writing on filesystem is significantly slower than writing on block (I am used to loose 10 or 20% speed; I have lost 90 or 95%). 
> ...

 

Hm, i want to know too... but here i can only read =) But for diagnose disease you have only LED =)

 *Quote:*   

> - it's still reasonably fast on block level 
> 
>  - I have no SMART error (or any other kind of error in any log I have) 
> 
>  - I have free space (and free inodes) 
> ...

 

Ok - you have all of this issue but how can we help you? You are in panic?

You need handkerchief, doll whipping, girls, alcohol, atomic bomb, drugs??? Just ask...

You didn't do anything to monitor issues past years and than problem appears - you are in panic...

----------

## Keruskerfuerst

1. You should update the bios of the mainboard. Here: http://www.gigabyte.com/products/product-page.aspx?pid=3154#bios

2. I only see one HDD in you setup.

3. ZFS does fragment and there is a utility to defragment the partitions.

----------

## doublehp

 *Quote:*   

> Hm, i want to know too... but here i can only read =) But for diagnose disease you have only LED =)

 

Many times, I did not even have one LED to help debugging.

One LED is enough to make mose. Those who tried to play with Android boot sequence will understand.

 *Quote:*   

> You didn't do anything to monitor issues past years and than problem appears

 

You did not ask anything else than HW configuration. Maybe I have. But ... did not spent hours on mentionning every single thing I have done to track the issue. To talk about the biggest tool I have, Munin; but it shows logs only for 14 months; what is not enough. I started to use MRTG in 2005; and Munin around 2009. MRTG was too heavy. Munin has only 1y worth of logs.

I also have kept all my syslogs since 2008.

I also run top all day long. I have also tried various combinations of smartctl, vmstat, and lsof ... and got nothing conclusive. I did dozens of things to log, monitor, track, and analyse, and nothing was conclusive. I am running out of ideas.

I said I don' want to rebuild my kernel; but, maybe I already have everything needed to dig the issue. Up to now, nobody mentionned any application, tool, or procedure. You don't know, because you didn't ask.

 *Keruskerfuerst wrote:*   

> 1. You should update the bios of the mainboard. Here: http://www.gigabyte.com/products/product-page.aspx?pid=3154#bios
> 
> 2. I only see one HDD in you setup.
> 
> 3. ZFS does fragment and there is a utility to defragment the partitions.

 

This is a forum; I am trying to make messages informative, short, and remove all redundant parts. I said the 5 disks are identical with consecutive serial numbers. If I wanted to make very long messages, I could paste my kernel conf, and all my syslogs from the 5 last years, and merge logs, and why not ... a complete hexdump of my disks. I have answered every single question that was asked.

I have dug the ZFS possibility with ZFS people from IRC. They said my system is fine; but I could update my ZFS when next update will be available, and increase my ARC cache, but this may not help.

My biggest performance loss is on EXT4 anyway ...

Still waiting for suggestions, or commands to run, monitoring apps to install ... if installing them does not required to update too many things.

----------

## schorsch_76

How about perf? [1] 

[1] https://perf.wiki.kernel.org/index.php/Main_Page

Do you at least have the souce and config of your kernel? With that you could include the functions for iotop. 

Without iotop it is difficult to trace io troubles. You dont even know when the hdd runs ... (no LED).

----------

## TigerJr

For analyze we need what you need - information about running you system. But information must be helpful. You say that all information you monitor doesn't helpful, now i don't know what information to request. IOPS monitor than error appears, CPU load, memory load, /proc/mdstat - but you say there is nothing useful 

MRTG -not so heavy... (otherwise prtg, cacti, zabbix, watsup and even gnuplot) and can be used via crontab script for each graphs, quite easy - it generates html pages with png images those you monitor via http webserver. You can use all the information you need with mrtg(disk load, network load, cpu load, iops, processes, memory, swap). If server have faced with DoS problem you can understand than error appears and what indications was before the problem appears and even what was source of DoS. That is good for diagnosis. Analyzing only LED gives you small amount of information and haven't LED history to understand what LED rates was hour ago or past day.

Did you check your disks for bad blocks?

----------

## doublehp

 *schorsch_76 wrote:*   

> How about perf? [1] 
> 
> [1] https://perf.wiki.kernel.org/index.php/Main_Page
> 
> Do you at least have the souce and config of your kernel? With that you could include the functions for iotop. 
> ...

 

Perf built. What's next ?

Yes I have my kernel tree. But I never understood how to compile drivers without rebuilding the whole tree. I need help on this. 

I have one LED on the front of my tower for disks.

I have increased my ARC cache from 300-400 to 4GB.

Updated from "ZFS: Loaded module v0.6.3-181_gb0cf067, ZFS pool version 5000, ZFS filesystem version 5" to latest 0.6.5 (it requires to have a valid Linux tree) butI don't see the difference: I still have zpool ver 28 and zfs ver 5.

 *Quote:*   

> 
> 
> You say that all information you monitor doesn't helpful, now i don't know what information to request. IOPS monitor than error appears, CPU load, memory load, /proc/mdstat - but you say there is nothing useful 

 

When slowlyness occurs, the system is so slow that I can not launch manually any of those commands; when system get's fluent, there is no trace of any issue. Things like Munin are just useless; they run once every 5mn, and just miss the train. In best case, I can see the load getting high; but even then, CPU remains at 75% iddle, and disk blinks slowly (1 to 3 short flashes per second). And when I type anything in consoles, letters take 20s to 3mn to appear on screen; when process is initiated, the whole system is fluent again, and the command also missed the train. When some commands run on long term, like top, they just freese for 1 to 10mn, and I can't see anything usefull. Last time, I watched my screen carefull for 12; in 12 minutes, the time on clocks moved 5 times (top, and gkrellm). Top was also refreshed 5 times in 12mn. I could write pages about what I have tried to track the issue. I am not here to talk about what failed, but learn new methods to track problems.

I will try to rebuild kernel to add iotop support; but it may take days to fix all side effects it will imply.

MRTG Cacti Munin and brothers ... can't help. Any thing based on cron is useless: as said, they miss the train. I know Thunderbird is a part of the problem; but freeses did not occur between jan 2010 and may 2014. Stopping using TB is not an option.

hmmm ... after remerging a few things, iotop now works. So, what are we looking for in there ?

iotop only gives the process that is doing work; what I would like to see is which file it's reading or writing. Lsof does not do that either.

We have been one step forward today.

 *Quote:*   

> Did you check your disks for bad blocks?

 

Cron month does zfs scrub and mdadm full resync.

----------

## schorsch_76

You need to set the kernel options according to [1]

CONFIG_TASKSTATS

CONFIG_TASK_DELAY_ACCT 

CONFIG_TASK_IO_ACCOUNTING

[1] http://linux.die.net/man/1/iotop

These changes would require to rebuild your kernel and your out of tree kernel modules (maybe ZFS). To get a backup net, backup your kernel, initrd and the modules Folder. 

```

mkdir /root/kernel-backup

cp /boot/*`uname -r`* /root/kernel-backup

cp -R /lib/modules/`uname -r`/root/kernel-backup

```

Please verify that vmlinuz, initrd,system map and your module folder are in the kernel backup.

After this, you can save the folder "kernel-backup" to a USB Flash drive. 

Then recompile the kernel with the above options, install it and it's modules, build out of tree modules (maybe zfs), rebuild initrd and reboot.

If something wents wrong, you can use sysrec cd to redeploy the saved kernel files and modules.

With this approach you can use your current kernel and dont need to update it. Just include the needed options.

perf itself is described on the linked page.

[2] https://perf.wiki.kernel.org/index.php/Main_Page

----------

## doublehp

```
# zcat /proc/config.gz  | grep -i TASK

CONFIG_TASKSTATS=y

CONFIG_TASK_DELAY_ACCT=y

CONFIG_TASK_XACCT=y

CONFIG_TASK_IO_ACCOUNTING=y

# CONFIG_IDE_TASK_IOCTL is not set

```

My setup is way more complex than a simple vmlinuz, due to RAID, and other details. I give a specific version name to each kernel, to avoid having rebuilds overwrite each other. Before doing any change in the conf, I manually edit the kernel name:

```
CONFIG_LOCALVERSION="-Gentoo-uranus-1-50"
```

Most of the rest is done by scripts; but for example, due to raid and other small details, I have four boot partitions. Build time scripts put the kernel in /boot (on top of RAID); then, shutdown scripts sync this to a BIOS accessible non raid partition. This is designed to reduce break of MBR. And since 2010 my Gentoo never suffered from broken MBR. But breaks further in the boot process; usually at startx.

Next ?

iotop running; waiting for the next freese to see how it will behave.

----------

## schorsch_76

You see, when i do something like this, 

```
ls /boot/*`uname -r`*
```

i get

```
ls /boot/*`uname -r`*

/boot/System.map-4.1.3-slim      /boot/initramfs-genkernel-x86_64-4.1.3-slim

/boot/System.map-4.1.3-slim.old  /boot/initrd-4.1.3-slim.img.gz

/boot/config-4.1.3-slim          /boot/vmlinuz-4.1.3-slim

/boot/config-4.1.3-slim.old      /boot/vmlinuz-4.1.3-slim.old

```

That is why i posted these two liner to backup your whole linux stuff. Even when it has a "more complex name" uname -r will tell it to me. 

It is not complex. It is just a kernel which follow the same rules. Even when it is from 2010. 

My output is:

```
zcat /proc/config.gz | grep LOCALVERSION

CONFIG_LOCALVERSION="-slim"

# CONFIG_LOCALVERSION_AUTO is not set

```

and now guess what will your uname -r tell you?

----------

## doublehp

```
$ ls /boot/*`uname -r`*

/boot/config-2.6.34-xen-r4-Gentoo-uranus-1-50

/boot/System.map-2.6.34-xen-r4-Gentoo-uranus-1-50

/boot/vmlinuz-2.6.34-xen-r4-Gentoo-uranus-1-50

$ uname -r

2.6.34-xen-r4-Gentoo-uranus-1-50

```

Where 50 is the part I always update manually. 

Building a kernel is not just build the kernel. Need to update all misc modules and drivers (including ... some not handeled by Gantoo official scripts). Updating grub.cfg ... considering the fact my kernel changes name on each rebuild. Plus, /boot is not my bootable partition; scripts who try to be smart and assume that /boot is bootable may break things. But, there is no way to have my /boot on a non RAID volume; and there is no way to boot a kernel from a raid volume from BIOS; so a caching system / sync mechanism is required to get benefit of RAID security, and push vmlinuz on a place accessible to BIOS. In the end, I don't need initrd (what is rare for people doing raid).

So ... now I have the good flags in my kernel, what's next ?

Thinderbird did not really bore me today; maybe did not spend enough time working with it.

But there is an issue I can easily reproduce. Open a directory, with Rox-Filer, that contains hundreds of pictures or movies. In 2011, it used to generate previews very fast; way more than 10 pictures per second, or 5-3 movies per sec. Now, it does a bit more than one picture per second, and less than one film per second. Yes, I am also having cache issue with ROX, but, that is my fault; I disabled rox preview cache somewhere, and can't find where I dit that. But, preview generation used to be very fast. And, this is easy to reproduce and track. I just open any large folder; can predict the moment, and repeat action. It's a better tracking process than waiting for TB to freese.

For example, I have been looking iotop and top while opening a picture folder: iotop reported total disk rate of 32K/s, and actual disk read 125K/s, traffic generated by ROX. While I had 75% iddle CPU, where heaviest process was X at 75%. It's a quad core processor, so the 75% iddle is average over the whole processor (value between 99% and 0%), while 75% used by X is for one core (value between 0% and 400%). During iddle time, my iddle is about 90%, so X 75% of core used by X is in fact only 15% of the whole processor. This means, dorung the preview generation, X does not even saturate one core; very far from it. So, the issue is not about thumbnails algo not being multi-tasked, or multihtreaded. CPU is not an issue, algorythm is not limited by core; and disk is hardly an issue (still, I wonder how it can generate previews of pictures that are about 900KB each, one per second, while reading disks are 32 or 125K/s ... )

----------

## krinn

if you want a fs that need no defrag, it must copying always file in sectors free and adjacent to each other, something doable with a lot of free adjacent space, something that will get hard with a few (not free space, but free adjacent sectors space, you can have 10g of free space and 0 adjacent sector free).

and doing such work is of course done by the fs, and your cpu is handling it, so writing any file on such fs in software raid, mean that for each sectors the cpu is balancing on all disks and fs is looking for free adjacent sector space to avoid fragmentation....

you should avoid adding work to work on things that should be done fast, it's good to add some workload to get raid software feature or to get compression (or no defrag feature in your case), but if you start stacking them, you're making it too complex for an operation that should remain quick.

If i were you, i would goes with a simpler fs (ext3, ext4) with no extra feature, and count only on raid for speed up, many fs features are wanking feature, that works with simple scheme, but in complex scheme like you (lot of disks and files) show their limit to their claims.

fs should do fs task (read/write file to disk), it should not goes into doing anything complex that would slow its handling for the fun of claiming "we don't use defrag", "we do mirroring", "we compress datas" or "we use md5 hash to make sure content is 100% valid". As while your fs is doing that it, other queries are waiting for it to end.

It's a balance of feature vs time to do the feature, of course non fragment files mean a disk will take less time to reach sector1 and sector2 for a file instead of sector1 and sector100 to read the same file, but on raid, disk1 answering to sector1 and disk3 answering to sector100 query change the rule and is less of a problem, making such non fragment fs feature more a problem than a good feature to use when you use software raid.

----------

## doublehp

How can I have fragmentation with 50% free space, and talking about 33G free for system, 400G free for home, and 2.5T free for storage disk ? 

So, after a few days with iotop, I have found that, at some times, thunderbird or Firefox may be responsible for very large disk usage. It tells about 2.6GB/s for 1 to 3 seconds. During this time, the whole system is slow. And top keeps talking about 75% iddle CPU, and very low sy and wa values, and the top process is usually ... none of those two. 

Next step: how to know which files they are reading, writing, or copying ?

This does not happen just after boot. It starts happening after at least 2 to 4h of use.

Since I started this thread, I did not have any very long freese (mouse refusing to move for 30s to 10mn). Never been more than 5s not moving at all, or 10-20s moving slowly (during a very heavy write, when iotop reports between 2GB and 3GB read rate).

Now, I can be more specific about what I want:

1: which files are Thunderbird messing with ? in particular, I want to know if they are system files (/usr), user files (/home) or temp files (/tmp).

2: why is rox-filer so slow to generate miniatures (thumbnails) of pictures and movies ?

3: offtopic: why doesn't rox keep thumbnails over time ? As soon as process is killed, all miniatures are lost, and need to be recomputed. I know I messed with this 2 years ago; maybe I have changed some settings, or changed some folder to point to /dev/null or /dev/shm. I just can't remember what I did. I have recently checked the settings, and cache folders, and they all seem fine to me.

----------

## krinn

 *doublehp wrote:*   

> How can I have fragmentation with 50% free space

 

record file 1, 2 & 3 of 10 sectors.

remove file2

add file4 of 12 sectors

file4 get record at free sectors 11-20 and the 2 that remains get records after file3, so at sector 41,42

file4 is now fragment ; and nobody cares how much space you have left.

----------

## doublehp

I am not using FAT12. ext4 and ZFS are much more clever than this.

----------

## doublehp

I have been watching my system very carefully those last hours. It's about 4h my machine is deadly slow; every 5 or 10mn, it gets slow. Mouse moves very slowly (about 70% of movement information is lost; I have to move my hand 4x more than expected). When mouse is slow, or frosen, disk blinks slowly, iotop reports no traffic (below 300kB/s), and top usually reports X to use 25% of the system (if it saturated one core, it would use 100%; if it saturated the whole CPU, it would use 400%), iddle time is about 70%, wa below 3%.

And this is happening a dozen times per hour, since 4h. Doing desktop work (Firefox, Chrome, OOo).

3 days ago, I could identify TB to be responsible for a specific issue: at some times, the whole box is completely frozen for 3 to 5s. Not slow: frozen. And then, HDD led is fully ON, and iotop reports traffic to be between 2 and 3GB/s. => I know what my machine looks like when some process want to have fun with my storage. So, this is not what have been boring me today.

And, after finishing my work, I still have 9GB RAM free. So, I am very far from running out of RAM. And my used SWAP is 0k out of 21GB.

I need more ideas to dig.

----------

## schorsch_76

Hm, that sounds strange. 

Just some ideas:

- cpu cooler. If the cpu gets too hot, it clocks down. Very common problem on old hardware (dust on the fans) and in the case. 

- check PSU: Try to exchange it against a new one

- memtest

- SATA cables could be bad. Bad contact (oxyd on connectors over time)

As you said, it shows up after a few hours of working, my best bet is dust on the coolers. The CPU clocks down to prevent damage.

https://www.google.de/search?q=dust+on+the+cooler&client=ubuntu&hs=iFp&channel=fs&source=lnms&tbm=isch&sa=X&ved=0ahUKEwi4uM-5p-LJAhVHtBoKHVItAIcQ_AUICSgD&biw=1680&bih=921

----------

## C5ace

 *schorsch_76 wrote:*   

> Hm, that sounds strange. 
> 
> Just some ideas:
> 
> - cpu cooler. If the cpu gets too hot, it clocks down. Very common problem on old hardware (dust on the fans) and in the case. 
> ...

 

Add to this:

CPU thermal paste, miss matched RAM, corroded motherboard ground posts. dried electrolytic capacitors, undersized powersupply. 

I has a similar problem in 2009 with a new Gigabyte MB. Can't recall the model number. Fixed by replacing the MB with a later model GA-MA790FXT-UD5P using 4x2GB matched Kingston RAM sticks and 5x 500GB SAMSUNG HD502IJ drives configures as RAID-1 for /boot and RAID-5 for /, /home and /VirtualBox on EXT4. The box need vacuming once a year and is still running like new.   

I personally would stop wasting my time and nerves with something that is very complex and probably not fixable and instead get a new box with matched memory sticks and a good graphics card with HDMI output. Then get the largest LCD TV you can afford and use this as a monitor.  Then sell your old stuff on e-bay or to a pawn shop.

----------

## doublehp

At last, good ideas.

I disagree, but I will need to proove you are wrong. So, which tests could proove you wrong ?

- machine is 5 years old; but I did a complete checkup when I changed my disks in mid 2014: changed several fans, removed all dust everywhere. Machine was not very dusty; I don't think it could accumulate more dust in 1 year, than during the first 4. 

- CPU cooler is oversized. My processor is a black edition: all people who bought thismodel bought it to overclock it by 5%, without touching voltages. After adding a second fan to my heat sink (Noctua NH-U12P) and adjusting voltages, people usually do +10%

- my Noctua is oversized; and my fan profile is to keep fan speed as low as possible to reduce aging of ball bearings, total air flow, and dust accumulation (and noise). I would have a hotter CPU.

- I draw my CPU temp since years; and honestly ... 42°C average over the year, with an absolute max at 58°C during the summer, hardly call that hot. 49° max over the last week, 54 the last month. Si, I am pretty sure my paste is good.

- cpu-cooler thermal paste: I would have a hotter CPU

- CPU clocking down for any misc reason: don't know how to check this. Could be an idea.

- dead PSU ... that is probably the least stupid idea. It's a very long time I know my +5V is very low (4.85V), and +12 not really better: 11.4V.

- memtest was good recetly. My RAM is not matched; I mean, I have 2x2G + 2x8G. But sticks are inserting correctly, to respect the topology required for my AMD64. When sticks are not inserted correctly, my BIOS complains about it. They are different models (2G and 8G are not the same model at all) but BIOS is usually responsible to downclock the whole system to the best possible configuration. Any way, I have slowliness issues since mid 2014; and bought 2x8G only 2 months ago; so, it may be a contribution, but can not be the root problem.

- bad Sata cables; worth speaking about it; I had problems due to sata cables, HDD firmwares and so on; but the problem does not sound like this to me: I have very good read/write rates on raw disks, and on raid blocks; perf come down when writing files on filesystem. I could kill my swap if you want, and show you that at disk, and MDADM levels, it's all good (tell me if you prefer JBOD, raid0, raid5 ... I have 4G swap partition on each disk; tests should not take more than 2mn if disks are good). Plus, bad sata cables usually produce messages in syslog or dmesg or SMART.

- RAM/HDD/cables do not seem to me a good explanation: when X is slow, I have 2 led flashes per sec and about 500k/s; when an app freeses completely, iotop reports 3GB/s for 2s. But, the first case could match a CPU frequency problem ... due to ... undervoltage. Undervoltage could be dead PSU, or dead VCU (voltage control unit inside the MB). This is a point which I can not counter-argue.

- sata cables: you say they could have oxydation. Which part could get oxyded ? why oxydation should have to be on cables, cable plugs, and not MB or HDD plug ? is there a way to clean that ? I am not going to change my MB for just a layer of oxyde on sata plugs.

- what I had once ... on a dual P3 800MHz machine was that one CPU had a bad contact on L2 cache line; some times CPU started with 512k, sometimes with 128k. Booting with 128k was always fine. But if Linux started with 512k, and, later the cache got away, loosing 372k of cache on the run was pretty bad. At the beginning, I had kernel panics; because the faulty CPU was CPU #0; because kernel had severe memory corruptions; after inverting CPUs, I only had app crashes; kernel could continue to manage the system from CPU #0, but most apps having tasks on CPU #1 had core dumps. After the partial crash, once the cache had gone away, I just had to kill all old process running on #1, and restart them. Once cpu had lost the bad cache line, it could not go worst; and app started after the loss would work 100% untill bed time. By that Linux, desktop Linux kernel did not yet have the capability to migrate process over cores. It was an electrically bad connection inside the die.

- dried capacitor: inside MB, I hope not; this model claims to have special capacitor that don't dry. PSU ... can hardly be so proud.

- undersized PSU ... I don't think so. I will dig.

Things are way better since I have made a huge cleanup in Thunderbird last week. No more "10mn freese".

What I am going to do:

- record CPU freq

- print CPU temps on screen permanently

- memtest, costs nothing

- cpuburn

- deeper write tests of disks, after killing my swap space; but 4GB per disk is not really that much; I will look for larger free partitions. Depending on results, maybe buy new cables.

- will measure input power with an Amp meter, at 220V side; will give me the total input power. Then, with clamp meter, measure 3V, 5V, and 12V currents, to see if any of them is exceding the PSU specs. I hope I can clamp wires by voltage; will be impossible if tubing of cables come too close to PSU. Ex this: http://www.overclock3d.net/gfx/articles/2009/06/05070513238l.jpg i can't clamp it. Or I will have to cut all tubings (dirty, not nice, but should not affect performances).

- I will setup a permanent external monitoring of all voltages (easy and cheap for me). I don't trust values measured by MB.

- not sure how to dig CPU faults on long term. The P3 was easy to study, and reading /proc/cpuinfo gave different results with time. My actual Athlon may not be as cooperative about details as P3 was.

- change PSU But if +5 and +12V are low, maybe 3.3 and 1.8 are also too short.

I don't need a more powerfull machine. This one is already much more powerfull than I need. And during freeses, disks don't work, and CPU is 75% iddle. Changing the hardware would be useless if issue is in software. And I don't have time to reinstall and configure the whole software now.

----------

## C5ace

 *doublehp wrote:*   

> At last, good ideas.
> 
> I disagree, but I will need to proove you are wrong. So, which tests could proove you wrong ?
> 
> - machine is 5 years old; but I did a complete checkup when I changed my disks in mid 2014: changed several fans, removed all dust everywhere. Machine was not very dusty; I don't think it could accumulate more dust in 1 year, than during the first 4. 
> ...

 

Use a precision digital voltmeter and adjust the voltages to the correct values and verify with a precision oscilloscope that the noise levels on all motherboard supply lines are less then 0.1 microvolts. Especially with the memory supply voltages. Or borrow a top quality powersupply.

----------

## doublehp

 *C5ace wrote:*   

> Use a precision digital voltmeter and adjust the voltages to the correct values and verify with a precision oscilloscope that the noise levels on all motherboard supply lines are less then 0.1 microvolts. Especially with the memory supply voltages. Or borrow a top quality powersupply.

 

Please, only quote the part you reply to.

I have 3-1/2 and 4-1/2 multimeters, and an old scope; and ... I have already notived that many projects get poluted by the PSU of my computer: have never been possible to have a flat line on my scope of my computer is plugged (even ATX power off: noise goes away when I remove the plug). It's an Enertec Schlumberger 5224, 4 ways 100MHz. But, after reading your message 4 times, I thing you exagerate a bit: all devices should accept a 0.1 microvolt ripple. I think you meant 0.1 mV (mili), or 100uV (micro). And even 100µV should be accepted by all devices. Plus ... 0.1µV ripple is not the same on 3.3 and 5.0V. My wonder is: what are the official acceptable extreme values for each voltage line ? An ATX PSU is performing DCDC conversion, which by definition generates squares waves at some point, and thus high frequency ripple. I don't think any PSU can have a ripple as low as 0.1µV (and even 0.1mV would seem amazing to me).

The only value I have now is: USB voltage 4.8V (same in front and back plugs, which are not on the same root). No phone can charge with so few.

But I add your point to my list: check ripple.

----------

## C5ace

 *doublehp wrote:*   

> 
> 
> The only value I have now is: USB voltage 4.8V (same in front and back plugs, which are not on the same root). No phone can charge with so few.
> 
> But I add your point to my list: check ripple.

 

Sorry, my error. I wanted to say 0.1 millivolt. 

Some time ago I had a intermittend HD and tested the voltaged on the motherboard. They where OK with a noise level of less than 0.1 millivolt. The fault was caused by a bad SATA cable.

The output of the DC-DC converter in the power supply is usually filtered by high capacity electolytic capacitors and the powerlines on the mother board by tantalium and ceramic capacitors they are for suppressing low and high frequency noise genrated by the chips on the motherboard. Mesure the voltages and noise direct on the memeory chips sockets, CPU, Host Controller, SATA and HD power sockets when your system is hot and runs slow.

----------

## doublehp

 *C5ace wrote:*   

> The output of the DC-DC converter in the power supply is usually filtered by high capacity electolytic capacitors and the powerlines on the mother board by tantalium and ceramic capacitors they are for suppressing low and high frequency noise genrated by the chips on the motherboard. Mesure the voltages and noise direct on the memeory chips sockets, CPU, Host Controller, SATA and HD power sockets when your system is hot and runs slow.

 

Will be complicated to not poluate the signal with ambiant noise (CRT, LCD, power lines ...).

----------

## C5ace

 *doublehp wrote:*   

> 
> 
> Will be complicated to not poluate the signal with ambiant noise (CRT, LCD, power lines ...).

 

The other option I can think off is to run stress testing software like described here:

https://wiki.archlinux.org/index.php/Stress_Test

----------

