# Shake your files for their own good

## 98111

EDIT : The latest shake  (0.29: tarball, Changelog, ebuild) is available at http://vleu.net/shake/.

(Please forgive me my bad English, if I say something arrogant or insulting, it's an probably an error)

I recently hacked something which could help to reduce fragmentation, without using dangerous and low level, and filesystem specific black magic.

It's main idea is that the lack of defragment software on GNU/Linux is mainly due to the lack of interest, due to the fact that filesystems do their job right.

But, even making their best efforts, FS can't perfectly anticipate years of usage (in fact, tests seems to show that just filling sparses files is a problem for ReiserFS 3).

That's when Shake intervene : using a few criteria, such as fragments number, or distance between two files used at the same time, it tries to detect which files are  fragmented or misplaced. Then it writes them again to make the FS re-allocate the space.

It's stupid, but seems to works.

In fact, it's even not so bad in comparison to other systems : if the filesystem is journalized, then shake will be. And one can shake only a folder, even while the rest of partition is used. It also avoid useless moves that would occur if filesystem and defragment software were using conflicting algorithms, and risks of corrupting the FS.

If you want to test^Wuse it, you only need to get it's tarball, call "make",  and then invocate..

```
./shake -pvv a_file
```

 as root (a --pretend and two --verbose).

There is a manpage, but I can already say that "-p or --pretend" put Shake in read only mode, and that "-vv"; show statistics like this  *Quote:*   

> FILENAME: start=FILE_START, ideal=IDEAL_START, end=FILE_END, fragc=FRAGMENTS_NUMBER, crumbc=CRUMB_NUMBER, age=FILE_AGE, CULPABILITY

 

Where FILE_START, FILE_END refers to file position's on the disk, IDEAL_START is where the file should be (according to the position of files used at the same time), CRUMB are small fragments (by default less than 0.95% of the file, 2 seconds of a 210 seconds OGG Vorbis).

Adding "-o0" allow you to force shake to force file "shaking" (rewrite).

Adding an extra "-v" allow you to see the position of each fragment, to see by yourself how ReiserFS to do his job (very well).

A side effect of shake is that it makes files sparse, so you could gain some disk space.

If you don't want to risk your files, can test it on a special disk image created to show it to my teachers (it was a project for my university at Lyon). If you speak French, you can an also read le dossier.

I don't see it as something you should use every month... maybe a time each year, unless you use intensively sparses files.

It use locks, make backups and some extra checks to detect by himself eventual corruptions, but you should not use it on important datas without reading by yourself the source code.

It is also less efficient on partitions mounted with notail, due to a limitation of FIBMAP, but you should not use notail anyway.

Don't hesitate if you have questions ! I am thinking to a new way of presenting statistics, and to the way I could port it on non-GNU systems.Last edited by 98111 on Thu Feb 01, 2007 9:40 pm; edited 3 times in total

----------

## steveb

Two things:What about a ebuild?Is that usefull for me (I use SGI XFS and xfs_fsr already does a defragmentation for me)?

cheers

SteveB

----------

## 98111

 *steveb wrote:*   

> What about a ebuild?

 

Good idea.

http://vleu.net/shake/shake-0.17.ebuild

 *steveb wrote:*   

> Is that usefull for me (I use SGI XFS and xfs_fsr already does a defragmentation for me)?

 If I understood xfs_fsr, it makes a job similar to the one of shake, excepted that it use a file system specific call to swap extents, so applications which don't honor locks won't have problems.

So, if I have understood, Shake is probably more efficient, because instead of just copying the 10% most fragmented files during 2 hours, it takes in account many parameters wich should represent more accuratly the effect of fragmentation on the performance. It also takes in account the age of the file, and so should reorganize free space.

To resume : Shake should be smarter and do less/better job. But it's a 0.17 release, and at least "magic values" probably could be improved...

One could also try to write an XFS-specific version of shake_reg() (in executive.c) wich would use XFS-specific calls on this filesystem...

----------

## steveb

 *98111 wrote:*   

>  *steveb wrote:*   What about a ebuild? 
> 
> Good idea.
> 
> http://vleu.net/shake/shake-0.17.ebuild

 Now be nice and post that ebuild to bugs.gentoo.org  :Wink: 

cheers

SteveB

----------

## Braempje

Do you have a project page? This seems interesting...

----------

## p0uLp3

Has anyone done a benchmark for this soft, before and after ? It seems interesting but I haven't my PC avalaible today :/

++

----------

## jim_crow

MAN! YOU'RE A GENIUS!!!

i've benchmarked it with one file (700MB).

the file was on reiserfs partition, very fragmented due to constant lack of space.

first i had problems making this little prodigy to work...

finaly i came to this: 

```
# shake -o0 -n0 -vv -S1024000 lola_rennt.avi
```

```
output: lola_rennt.avi: start=14011849, ideal=0, end=14011840, fragc=9713, crumbc=9712, age=7, guilty
```

so! this is it! some hd activity during the process.

then a test: 

```
# shake -pvv lola_rennt.avi
```

```
lola_rennt.avi: start=193514, ideal=0, end=7390636, fragc=633, crumbc=617, age=0, innocent
```

and a benchmark:

copying the file with mc to /dev/null

35MB/s (before shake it was ca 9,5MB/s)

this isn't  a slight improvement, it's a rocket!!!

write a man, Man :D like i said, i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...)

----------

## jim_crow

second thoughts:

(oh, there is a man, though, i couldn't make it to work right now)

-S0 forces it to work with each file.

-c1 - even less fragmentation? but further position, if i read it right...

my question is: could this program be used to defragment huge amout of files on a partiton, like for example /usr ?

how does it affect partition when used on greater amount of files? i mean, how does it change free space and positions of files?

would it still be performance increase when used on large number of small files?

i can hardly wait to test it :)

----------

## 98111

 *jim_crow wrote:*   

> 
> 
> -S0 forces it to work with each file.
> 
> -c1 - even less fragmentation? but further position, if i read it right...

 

No, in fact -S0 make it thinks that a "big file" is a file of at least 0 bytes.

So all files are considered big, and by default, big files are not shaked (to change that, set -T to something else).

If you want to shake all files, you should use "-S0 -n0 -c1", to tell him that no files are too big (-S0), no files are too young (-n0) and that 0 fragment is already too much (-c0). But I don't think it would be usefull, you should better let (-n) at it's original value, or at 1, because rewriting files that have been wrote this morning is probably useless.

Also, if you don't change -S and you "cd /dev/shm" before shaking, shake will use the "dynamic ramdisk" that is there instead of you harddisk for temporary copy, so it will be faster. But you won't be hable to shake files bigger than your available ram space (in fact, you can change -S to make it equal to your amount of free ram space in kB).

I know it's really not intuitive and that this project should have a decent documentation, interface, and web page.

I'll do all that after my exams (3 June). For now i just changed main.c in a such way that -c0 is considerd as valid (please delete /usr/portage/distfiles/shake-0.17.tar.bz2 and re-emerge it).

 *jim_crow wrote:*   

> my question is: could this program be used to defragment huge amout of files on a partiton, like for example /usr ?

 

Yes ! Shaking the whole partition is the best thing to do IMHO, because it will reorganise free space.

 *jim_crow wrote:*   

> 
> 
> how does it affect partition when used on greater amount of files? i mean, how does it change free space and positions of files?
> 
> 

 

In order to reorganize free space, it shakes files of the same directory that are far from each other, and files that are "old".

To change position of files, it just write them again and let the filesystem do his job... it's stupid but it's seems that xfs_fsr do the same (should check again, but don't have time until 3 June - i would also be interested if someone points me to others defragment software source code). I will change it if I find a better way, even if this way is not available on all filesystems (by doing filesystem specifics version of shake_reg() ? (executive.c))... but only after my  exams.

 *jim_crow wrote:*   

> 
> 
> would it still be performance increase when used on large number of small files?
> 
> 

 

I think so: at least it should reorganise free space and put together files that are used at the same time.

But if you're thinking to /usr/portage, the better way IMHO is to shake the whole partition to reorganize free space, then do a 

```
rm -Rf /usr/portage/* # supress ebuilds

emerge-webrsync # get them again
```

 so you're sure that they will be new files.

 *jim_crow wrote:*   

> i can hardly wait to test it 

 

Thanks   :Smile:  ! But be carefull, it hasn't yet been tested as it should.

By the way if someone has idea to improve shake (such as secret ioctl's, a better syntax, a better idea for the text output...), he is welcome   :Wink: Last edited by 98111 on Mon May 22, 2006 7:53 am; edited 1 time in total

----------

## 98111

 *Braempje wrote:*   

> Do you have a project page? This seems interesting...

 Thanks !   :Very Happy: 

I'll make one soon after my exams (3 June) !

----------

## 98111

 *jim_crow wrote:*   

> first i had problems making this little prodigy to work...

 My user interface sux... I will try to improve that after my exams.

 *jim_crow wrote:*   

> 
> 
> finaly i came to this: 
> 
> ```
> ...

 Glad it worked for you !   :Cool: 

 *jim_crow wrote:*   

> i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...)

 You should find better explanations in my previous post. And Shake should work recursively. If something like "shake -pvv /etc/" don't go through subdirectories, it's a bug, so please test again and tell me !

----------

## gentoobunny

 *jim_crow wrote:*   

> and a benchmark:
> 
> copying the file with mc to /dev/null
> 
> 35MB/s (before shake it was ca 9,5MB/s)
> ...

 

Now try it again with another file and copy the file with mc twice before using "shake". You will find that it's caching that made the file go 35MB/s the second time.

----------

## jim_crow

i get your point. but notice that the file size is over 700MB and it's definately too much for cache on my 512MB machine. so i tried it three times before (time after time), each time getting transfer ca 9,5MB/s. so - it really works, try for yourself... but, the file you try must be really fragmented, or you won't see the real difference.

oh, and i know it reads hd not the cache, because hd led is on almost all the time.

well, i wonder about one more thing:

i have some folders on my reiserfs partition with huge amount of files, and i've noticed that access to those folder with mc and such is very slow and quite loud. i wonder why...

----------

## 98111

 *jim_crow wrote:*   

> i have some folders on my reiserfs partition with huge amount of files, and i've noticed that access to those folder with mc and such is very slow and quite loud. i wonder why...

 What does 

```
shake -n0 -pv your_folder | grep guilty | wc -l # count fragmented files
```

, 

```
find your_folder -type f | wc-l # count files
```

 and 

```
mount | grep reiser | grep -v notail # show reiserfs with packed files
```

 return ?

Note it's important to use "shake FOLDER" and not "shake FOLDER/*", because if the argument is a folder name, shake will be hable to take distance between two files in account.

----------

## jim_crow

```
shake -n0 -pv /usr/lib | grep guilty | wc -l # count fragmented files
```

returns lots of  *Quote:*   

> No such file or directory

 

and then 13325.

```
find /usr/lib -type f | wc -l # count files
```

returns 26389.

```
mount | grep reiser | grep -v notail # show reiserfs with packed files
```

returns

 *Quote:*   

> /dev/hda5 on /data type reiserfs (rw)

 

well, it seems my /dev/hda1 uses reiserfs without notail option.

----------

## 98111

 *jim_crow wrote:*   

> 
> 
> ```
> shake -n0 -pv /usr/lib | grep guilty | wc -l # count fragmented files
> ```
> ...

 

I should filter "No such file or directory" (they correspond to invalid symlinks, try to look at app-misc/symlinks)... Anyway, the fact is that you have 13325 misplaced file in this dir (~50% of them). It should be better after a 

```
 cd /dev/shm/

shake -n0 /usr/lib
```

. If it is not, this either mean that shake is too severe with files, or that your empty space is really fragmented (plausible as you don't use notail).

 *jim_crow wrote:*   

> 
> 
> ```
> mount | grep reiser | grep -v notail # show reiserfs with packed files
> ```
> ...

 You should add it to your /etc/fstab (and maybe shake your whole partition one or two times with -n0) before doing anything else. tail mode kill reiserfs performance.

  :Exclamation:  shaking /usr/lib or a whole partition may be dangerous  :Exclamation: 

----------

## jim_crow

i checked i do use notail option in fstab for /dev/hda1 and not for /dev/hda5. maybe thats why /dev/hda5 is so slow in folders with lots of files.

i've tried

```
cd /dev/shm/ 

shake -n0 /usr/lib
```

and...

it has to be quite dangerous, well it crashed my machine in very weird way (static on display, freeze).

maybe it's because /dev/shm/ ran out of space?

anyway i hope no file was harm. i'm glad to still have my gentoo alive :) (after small fsck at boot)

i've tried to shake -n0 /data/mp3 (over 30GB, /dev/hda5) ...

well, 776 fragmented files before, 702 after... shouldn't it be 0? what's wrong?

oh, btw, it took over 10 minutes, so it would be nice, if you could add some % thing or something...

is it safe to ctrl+c working shake?

--

ok, i did it.

```
# shake -n0 -vv /usr
```

and took a nap, than a bath. results:

booting time: non-x - shortened from 1:10 to 1:00, total: 1:45 to 1:27.

so, definately - it works.

still i wonder why does it take so damn long for kdm to load :/ fonts?

----------

## beatryder

So, you wrote this from scratch? I am really impressed. I am watching this topic, I am interested in trying shake, however my computer working for me is very important so I think I will wait for a few versions.

----------

## shebang_line

This is very impressive, I cant wait for future releases  :Smile:  keep up the good work!

----------

## Jinidog

I would want to let this run over my filesystems. 

Can you suggest a command, that defrags all files of the root-directory and is as secure as it can be?

----------

## 98111

 *Jinidog wrote:*   

> I would want to let this run over my filesystems. 
> 
> Can you suggest a command, that defrags all files of the root-directory and is as secure as it can be?

 

```
shake /
```

 should do that. But I found many bugs, so don't use it before the 3 June, when I'll release the new version.

The main problem on the actual version is that it update the ctime to avoid to shake again recent files. That could conflict with emerge !

The consequence is that emerge *may* let shaked files after an unmerge. If that happened to you, you should use a tools like findcruft. If more than one person had the problem, i'l try to code something to revert ctimes. Else I'll try to find a way to fix it by hand.

Ask me by mail if something gone wrong. I'm really sorry.

To conclude DO NOT USE SHAKE BEFORE V 0.18.

----------

## smurfer

Hi all,

sounds very promising, I think I will give it a try. For those of you who want to defrag whole partitions and have enough free space, there's the old way of

- Copy

- Format

- Copy back

I usually keep one free partition with the size of the largest partition used to do this job. Using two drives makes copying faster.

Regards, smurfer...

----------

## Allochtoon

Sounds good, will try it with more stable version  :Smile: 

----------

## Merlin-TC

I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.

I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.

Then I added "user_xattr" as option for the partitions I want to use it on.

Am I missing anything?

----------

## 98111

 *Merlin-TC wrote:*   

> I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
> 
> I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.
> 
> Then I added "user_xattr" as option for the partitions I want to use it on.
> ...

 Sorry, the version you tested wasn't mean to be distributed, I didn't think that someone would try to find the webpage  :Embarassed: 

But it wasn't harmfull.

From now, version distributed on the page http://vleu.net/shake/ will be stable one.

Direct link to the ebuild : http://vleu.net/shake/shake-0.19.ebuild

THE SEVERE BUG IS FIXED  :Very Happy:  (Changelog)

Also, the output should now be readable   :Wink: .

If it works well, I'll try to submit the ebuild for inclusion in Gentoo.

----------

## Merlin-TC

Well, I do have version [ebuild   R   ] sys-apps/shake-0.19  0 kB [1] emerged so that's not it?

Any clues?

----------

## 98111

 *Merlin-TC wrote:*   

> Well, I do have version [ebuild   R   ] sys-apps/shake-0.19  0 kB [1] emerged so that's not it?
> 
> Any clues?

 Yes: the bugged tarball had the same name as working one, as the buggy wasn't mean to be released (it were just for testing the webpage)   :Confused: 

if you get the new ebuild and then 

```
rm /usr/portage/distfiles/shake-0.19.tar.bz2

emerge shake
```

 all should works.

----------

## Merlin-TC

Ah ok, seems to work now. Using it on a 200gig drive now  :Wink: 

----------

## beatryder

 *98111 wrote:*   

>  *Merlin-TC wrote:*   I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
> 
> I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.
> 
> Then I added "user_xattr" as option for the partitions I want to use it on.
> ...

 

I just installed shake now, and it gave me the very same problem.

I even tried remounting the partition with the user_xattr flag like you suggested on your website, and it gives me the following:

 *Quote:*   

> Septu Windows 98 # shake -p ./
> 
> Septu Windows 98 # shake -p ./*
> 
> Septu Windows 98 # shake -p --old 0 ./
> ...

 

Also this is on a reiserfs partition. does that matter?

----------

## 98111

 *beatryder wrote:*   

>  *Quote:*   IDEAL   START   END     FRAGC   CRUMBC  AGE     GUILTY  NAME
> 
> shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
> 
> 0       692098  692120  1       0       32      1       .//vmware.log
> ...

 There is 2 possibility : either your kernel doesn't support user_xattr, or you use the old version (as Merlin-TC was doing).

To know about this, please remove "/usr/portage/distfiles/shake*", make sure you have the latest ebuild, and emerge shake again.

If it still don't works, I would be really happy to see the output of "mount", and the last line of dmesg  :Smile: .

The next version (which will probably be released this night, or tommorrow  :Wink: ) will have a switch to disable user_xattr, but I would like to have them tested before, because they are really usefull.

----------

## Merlin-TC

I have a little bit off topic question about xattr. Does it costs speed or does it improve the speed for programs that support it?

Does it help "gamin" as well?

----------

## 98111

 *Merlin-TC wrote:*   

> I have a little bit off topic question about xattr. Does it costs speed or does it improve the speed for programs that support it?
> 
> Does it help "gamin" as well?

 eXtented ATTRibutes consists in pair of name/values attached to a file.

By exemple, Shake use them for recording the time of space allocation, using the pair "shake_ptime"/time (with time coded in little endian on four bytes).

A file manager could use them to associate preview or mimetype with files, but I don't see how it could help gamin.

And it doesn't cost speed, just a bit of HD space, 5megs for all ptimes of files on my system.

----------

## Merlin-TC

Is it possible that xattr takes up a huge amount of space when the partition consists of many small files?

I have one partition where the portage tree is and I just formated it and the tree takes up 122MiB but df -h shows /sda5             4.7G  559M  4.2G  12% /usr/portage.

So it must have a huge overhead.

Is there any way to remove the xattr stuff?

----------

## 98111

 *Merlin-TC wrote:*   

> Is it possible that xattr takes up a huge amount of space when the partition consists of many small files?
> 
> I have one partition where the portage tree is and I just formated it and the tree takes up 122MiB but df -h shows /sda5             4.7G  559M  4.2G  12% /usr/portage.
> 
> So it must have a huge overhead.
> ...

 I don't think that the difference is due to xattr. Have you tried to do a "df -h" when there's no files ?

I can write something to remove xattr, but before I would like to ask you something... could you delete this portage tree, mount this partition without user_xattr, re-get the portage three and tell if "df -h" return something different ?

I think that wasted space is not wasted due to xattr, but due to internal structures of reiserfs.

----------

## bunder

i tried running it on /usr/portage and i got a whack of "too many open files".  is that directory too big to use shake on?

edit: i have a feature request.  can you put in a --one-file-system option?  say i wanted to run it on just /usr (not /usr/portage, or /usr/portage/distfiles which is running over nfs).

nice script, i'll play around with it on my systems to see if it makes anything faster.

----------

## Merlin-TC

You were right, it had nothing to do with xattr.

I was mounting the partition with the "notail" option (reiserfs) and this caused the huge overhead. There is also another option calles atime, is this somehow connected with xattr?

----------

## 98111

 *Merlin-TC wrote:*   

> You were right, it had nothing to do with xattr.
> 
> I was mounting the partition with the "notail" option (reiserfs) and this caused the huge overhead. There is also another option calles atime, is this somehow connected with xattr?

 tail mode spare space, but it is however recomended to disable it on reiserfs<4, which cannot manage it properly.

Atimes helps shake to see the correlation between files.Last edited by 98111 on Sat Jun 10, 2006 10:09 am; edited 2 times in total

----------

## 98111

 *bunder wrote:*   

> i tried running it on /usr/portage and i got a whack of "too many open files".  is that directory too big to use shake on?

   :Shocked:  ! No, shake open only three files at once, whatever is the size of the directory. I have already used it on /usr/portage...

 *bunder wrote:*   

> edit: i have a feature request.  can you put in a --one-file-system option?  say i wanted to run it on just /usr (not /usr/portage, or /usr/portage/distfiles which is running over nfs).

 Good idea, it will be done arround tomorrow.

 *bunder wrote:*   

> nice script, i'll play around with it on my systems to see if it makes anything faster.

 OK ^^. If you have some time to experiments with default values... I know they could be improved because I just set them arbitrary   :Confused: 

----------

## Cintra

Very nice, well done 98111  :Very Happy: 

I just ran it on the /root folder of my 2nd Gentoo copy and it works like a charm! It cleaned up a lot of 'guilty' files, and everything still seems to be working fine on that copy  :Wink: 

Btw, one thing that could make shake even nicer is a summary after 'shake -pvv <whatever>' (like XP's analyze).. 

Mvh

Amost forgot.. does it work only on Linux partitions?

----------

## bunder

i found out why i was getting the "too many open files" errors.  i didn't happen to look at the path of the files before, but those were the ones on the nfs share.   :Embarassed: 

----------

## 98111

 *bunder wrote:*   

> i found out why i was getting the "too many open files" errors.  i didn't happen to look at the path of the files before, but those were the ones on the nfs share.  

 Effectively, I was handling badly the failure of get_testimony... it should be fixed !

I also added --one-file-system, fixed some little bugs, and trapped signals during the file copy (Changelog ebuild).Last edited by 98111 on Sat Jun 10, 2006 10:36 pm; edited 1 time in total

----------

## 98111

 *Cintra wrote:*   

> Very nice, well done 98111 

 Thanks   :Very Happy: 

 *Cintra wrote:*   

> Btw, one thing that could make shake even nicer is a summary after 'shake -pvv <whatever>' (like XP's analyze).. 

 In long term (newt weeks), I will try to contact authors of davtools; or to make a python module that will wrap arround shake and allow to add better UIs while keeping the core source code light.

During this time, adding some statistics in the main source code to remove it just after seems a bad idea...

 *Cintra wrote:*   

> Amost forgot.. does it work only on Linux partitions?

 I would love to have it running on all posix-compliant systems, but that will require a little works.

And it's efficacity depends of the quality of the filesystem implementation, so using it on vfat or ntfs from Linux is definitively a bad idea.

----------

## bunder

my box crashed when running the new version.  i'm not sure if it was because i ran out of /dev/shm or if it was because the filesystem was mounted.  just something to look for   :Wink: 

edit: maybe it didn't crash.  it still responds to pings, so i bet the kernel killed sshd.

----------

## beatryder

 *98111 wrote:*   

>  *beatryder wrote:*    *Quote:*   IDEAL   START   END     FRAGC   CRUMBC  AGE     GUILTY  NAME
> 
> shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
> 
> 0       692098  692120  1       0       32      1       .//vmware.log
> ...

 

I have not used shaker since AFTER you said the major bug was repaired. I have the xattr in my kernel, and have mounted with the user_xattr option

----------

## Uncle_Psychosis

```
 # ebuild shake-0.20.ebuild compile

>>> md5 files   ;-) shake-0.17.ebuild

>>> md5 files   ;-) shake-0.20.ebuild

>>> md5 files   ;-) files/digest-shake-0.20

>>> md5 src_uri ;-) shake-0.20.tar.bz2

>>> Checking shake-0.20.tar.bz2's mtime...

>>> WORKDIR is up-to-date, keeping...

gcc -O2 -march=athlon-xp -pipe -std=gnu99 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Werror -pedantic-errors -Wconversion -Wpointer-arith -Wbad-function-cast -Wcast-align -D_XOPEN_SOURCE=600 -O2 -D_POSIX_C_SOURCE=200112L -O0 -ggdb3  -DNDEBUG -c linux.c -o linux.o

linux.c:23:43: attr/attributes.h: No such file or directory

linux.c: In function `set_ptime':

linux.c:44: warning: implicit declaration of function `attr_setf'

linux.c:44: error: `ATTR_DONTFOLLOW' undeclared (first use in this function)

linux.c:44: error: (Each undeclared identifier is reported only once

linux.c:44: error: for each function it appears in.)

linux.c: In function `get_ptime':

linux.c:56: warning: implicit declaration of function `attr_getf'

linux.c:56: error: `ATTR_DONTFOLLOW' undeclared (first use in this function)

make: *** [linux.o] Error 1

!!! ERROR: sys-apps/shake-0.20 failed.

!!! Function src_compile, Line 566, Exitcode 2

!!! emake failed

!!! If you need support, post the topmost build error, NOT this status message.
```

Any ideas? I'd like to give this a shot! 

Sam [/code]

----------

## 98111

 *beatryder wrote:*   

> I have not used shaker since AFTER you said the major bug was repaired. I have the xattr in my kernel, and have mounted with the user_xattr option

 

Could you try

```
touch testfile

attr -s testattr -V testval testfile
```

 ?

Anyway, the new release of xattr have --no-xattr, but this is not optimal.

http://vleu.net/shake/shake-0.21.ebuild

 *Uncle_Psychosis wrote:*   

> 
> 
> ```
>  # ebuild shake-0.20.ebuild compile
> 
> ...

 You probably didn't installed sys-apps/attr. By the way, are you sure that manually calling "ebuild" is a good idea ?

----------

## Uncle_Psychosis

Ok, when I get home I'll try and install atrr. 

As for ebuild, I'm new to trying to install using ebuilds that aren't in the portage repository. I was following the guide here:

http://gentoo-wiki.com/HOWTO_Installing_3rd_Party_Ebuilds

Cheers

Sam

----------

## Lokheed

So how do I know this works? What I mean is, how do I know it did anything? I notice no speed improvements. I also don't understand what "guilty" means. Windows defrag programs show clearly that there is less fragmentation after running them. With shake, I have no clue it did anything.

Maybe you should make things a little more nonambiguious. Not using words like "guilty" because most people won't know what you are talking about. The layout is pretty nonfunctional, seeing tens of files fly past you with no overall rundown on the directory. Have you tried looking into ncurses? It could add a lot of functionality to shake.

Cheers.

----------

## 98111

 *Lokheed wrote:*   

> So how do I know this works? What I mean is, how do I know it did anything?

 

You can make a diff between two 

```
shake -pvv
```

, before and after a 

```
shake
```

.

 *Lokheed wrote:*   

> I notice no speed improvements. I also don't understand what "guilty" means. Windows defrag programs show clearly that there is less fragmentation after running them. With shake, I have no clue it did anything.

 guilty mean "fragmented". I should add something about this in the documentation.

 *Lokheed wrote:*   

> Maybe you should make things a little more nonambiguious. Not using words like "guilty" because most people won't know what you are talking about. The layout is pretty nonfunctional, seeing tens of files fly past you with no overall rundown on the directory. Have you tried looking into ncurses? It could add a lot of functionality to shake.

 The problem is that showing some nice graphics suppose to scan the whole partition to find the block layout (else, there will be "gaps" everywere).

That said, if you wants to see the disk before and after, try using a tool like http://davtools.sourceforge.net/ .

I *will* add statistics, but IMHO those things should not go in the core of shake. I will provide them as seperate tools.

And it's complectely normal that you don't see a performance improvement : most FS don't fragment significantly, excepted after a long time or in pathologic cases (many bittorent clients).

----------

## Lokheed

If guilty means fragmented, why not simply use fragmented? You are putting more cognitive strain on people using "fancy" words when you could simply state what you mean. There is also no semantic connection between guilty and fragmented.

I guess shake is limited to a tiny select few if it is "normal" to not see a performance improvement. I guess this is why no one using Linux really cares about fragmentation...

Shake seems to be more of a specialty tool than a must have system utility. Not that I miss defragging  :Wink: 

Cheers

----------

## 98111

 *Lokheed wrote:*   

> Shake seems to be more of a specialty tool than a must have system utility. Not that I miss defragging 

 Yes, Shake is not something to use daily... that is the problem of being a defragmenter on a system that do not fragment  :Smile:  .

But for old partitions, or very used one, or for those wich download a lot via P2P (try "shake -pvv" on something downloaded via ktorrent), I think it could be usefull.

----------

## ruben

 *Lokheed wrote:*   

> I guess shake is limited to a tiny select few if it is "normal" to not see a performance improvement. I guess this is why no one using Linux really cares about fragmentation...
> 
> Shake seems to be more of a specialty tool than a must have system utility.

 

It's not a critical tool, but I believe it is or has the power to become a valuable tool. I imagine you won't see much of a performance improvement on recent hardware with fast harddisks, unless the file system is very, very fragmented. On the other hand, I believe it can make a big difference on laptops with slow 4200rpm/5400rpm harddrives. I've experienced this myself with a ReiserFS (v3) on a laptop with Gentoo. Performance on ReiserFS noticeably degrades over time, especially when you frequently update your system. I remember defragmenting that partition by copying the data to another partition and then copying it back in a clean, re-formatted partition. This operation halved the time from GDM login to usable Gnome desktop (45secs to 23secs). I've since switched to ext2/3, which seems to suffer a lot less from fragmentation. I think this would be a useful tool for more people than one would initially expect.

I have to admit I didn't yet use this myself, but it seems smarter than standard defragmenter tools by using a heuristic to place files together, which will likely be needed at the same time. I've been following this thread a bit and I'll definitely try it some time, but I first wanna make sure I have a working backup.

----------

## nordpolcamper

Hi,

nice work. But if I execute "shake /usr" (shake-0.21) it stops working after a few minutes and does not quit. No error message and no consumption of cpu time. It just stops working without quitting.

So are there any ideas what I may have done wrong. user_xattr are enabled. I'm using reiserfs on a amd64 machine.

rgds

Bernhard

----------

## Dralnu

I read through part of this, and the idea I like/love. Personally, seems like a cron job to run weekly/monthly on high-use HDD (like /home maybe?), and then maybe annually on something like / to defrag everything.

I'll check things out when my system finishes updating, and then give you my thoughts/problems with it.

P.S. I think this is the first I've heard of a defrag program for Linux. I didn't know the filesystem EVER fragmented, but I guess anything will.

----------

## Cintra

 *nordpolcamper wrote:*   

> Hi,
> 
> nice work. But if I execute "shake /usr" (shake-0.21) it stops working after a few minutes and does not quit. No error message and no consumption of cpu time. It just stops working without quitting.
> 
> So are there any ideas what I may have done wrong. user_xattr are enabled. I'm using reiserfs on a amd64 machine.
> ...

 Yes I just had the same thing.. it simply went to sleep when 'shake -pvv /root' got to  /root/.kde3.5/share/apps/juk/playlists - always at the same spot!

Re-emerging 0.19 allows it to complete ok

Mvh

----------

## whitesouls

1. I'm sorry but does this wonderfull app works on reiser4?

2. Are the files which are guilty will be defragged?

----------

## nordpolcamper

[quote="Cintra"] *nordpolcamper wrote:*   

> 
> 
> Re-emerging 0.19 allows it to complete ok
> 
> Mvh

 

Many thanks, that did it  :Smile: . It 

rgds 

Bernhard

----------

## 98111

Sorry for the lag, the bug is fixed.

Shake was waiting for the lock when it couldn't get it.

Shake 0.22 (tarball, Changelog, ebuild) fix this problem.

----------

## 98111

 *Lord Vader wrote:*   

> 1. I'm sorry but does this wonderfull app works on reiser4?

 Yes, it should.

 *Lord Vader wrote:*   

> 2. Are the files which are guilty will be defragged?

 Yes.

----------

## BlueFusion

Does shake work on the ext2/3 filesystems?  What about a vfat filesystem, say, on an iPod?

Forgive me if these were asked in any prior posts, but I read through it and didn't see any mention of them.

EDIT: I tried it on ext3 and love it!  I did not try vfat, yet, however.

----------

## 98111

 *BlueFusion wrote:*   

> Does shake work on the ext2/3 filesystems?  What about a vfat filesystem, say, on an iPod ?

 It should works on ext2 and ext3, but probably won't on vfat.

Knowing this, if you want to try shake on vfat, you can by adding the --no-xattr option. But the better is probably to make a backup of the ipod content, then format the ipod hard drive, and copy back the data.

 *BlueFusion wrote:*   

> Forgive me if these were asked in any prior posts, but I read through it and didn't see any mention of them.

 No problem !   :Smile: 

----------

## BlueFusion

I shook my entire Gentoo install.  I didn't backup only because I'm planning on wiping and re-installing soon to use a RAID10 setup and if it died, it'd just expedite that.  But I'm happy to say, after running shake -v / a few times (until nothing else was listed as being shook) while chrooting from a LiveCD, I am happily using my system with no problems, thus far.

I did try running with the --old=0 option, however I recieved an error on a single file every time it ran through.  The file was a Battlefield 2 game file installed with Cedega.  I'm not sure yet if it's a problem with the file or the size or what.  And I'm sorry but I don't have that output available at the moment.

I am definately adding this topic/project to my favorites list  :Very Happy:  .  Very cool program!

----------

## 98111

 *BlueFusion wrote:*   

> I did try running with the --old=0 option, however I recieved an error on a single file every time it ran through.  The file was a Battlefield 2 game file installed with Cedega.  I'm not sure yet if it's a problem with the file or the size or what.  And I'm sorry but I don't have that output available at the moment.

 I would really like to read that error, if you can.

 *BlueFusion wrote:*   

> I am definately adding this topic/project to my favorites list  .  Very cool program!

 Thanks !  :Very Happy: 

----------

## BlueFusion

shake -v /home/rich/.cedega:

```
shake: /home/rich/.cedega/Ghost Recon/c_drive/Program Files/InstallShield Installation Information/{D89EF3B3-6F17-4665-B7A9-A4235A6DC787}/setup.ilg: temporary copy failed
```

It goes through quite a few files and then fails on this one.  I'm wondering if it has anything to do with the { }?

----------

## Jinidog

Trying this now on a reiserfs. But for every file it tells me:

 *Quote:*   

> 
> 
> failed to set position time, check user_xattr: Operation not supported
> 
> 

 

I compiled reiserfs extendend attributes in the kernel. Does this message harm and/or how can I solve it?

----------

## BlueFusion

Do you have user_xattr in the mount options in /etc/fstab?

----------

## Jinidog

no, should I have?

----------

## lnxz

Yes

----------

## saluk

0.22 wont compile.  I haven't downloaded any of the previous releases, and I've tried both the source and the ebuild with the same error:

```

linux.c:23:43: attr/attributes.h: No such file or directory
```

Did you forget to pack that directory?

----------

## BlueFusion

Ensure you have attr installed.  You should have gotten that if you emerged shake from portage with the ebuild in this topic.

----------

## saluk

Oops, I was using ebuild .. merge instead of emerge.  Now it works!

I am going to run it on my drive tonight, I've been pretty hard on my system for the past 2 years or so, with games and downloads etc, so I expect to see many many guilty's.

----------

## P0w3r3d

vleu.net is down??

----------

## Sourcecode

Very Nice Work  :Smile: 

Can i know run this over the whole partition?  :Smile:   i don't have a maschine for testing  :Sad: 

----------

## Utoxin

How safe is it to run this on the root of a running system? Will it properly skip /dev/ and /proc/, since they're not real filesystems? Is it likely to kill my machine?

I understand that it would be best to run it on a non-system partition, but are there specific reasons /not/ to run it on my active system partitions?

----------

## BRPXQZME

 *Utoxin wrote:*   

> How safe is it to run this on the root of a running system? Will it properly skip /dev/ and /proc/, since they're not real filesystems? Is it likely to kill my machine?
> 
> I understand that it would be best to run it on a non-system partition, but are there specific reasons /not/ to run it on my active system partitions?

 It should roll right over /dev and /proc just fine.  Just don't run it over any libs that you need to run... ESPECIALLY libc.  In fact, just avoid doing it on a running system.  It's safer  :Smile: 

I just made that mistake and it took me a day to fix it.  If you want to know how I got over it, here's an ugly process to restore your system from a livecd (yeah, kind of had to figure it out myself, unfortunately... hope anyone can elaborate on this or get a more elegant way; after borking my libc I couldn't so much as emerge anything! Python couldn't load the libs!  Also, need it be said, this applies only to Linux, as shake only works on Linux.  But it's probably a generic way as any to restore a borked libc on a Gentoo box):

mount -o bind /dev /mnt/gentoo/dev (of course, replace 'gentoo' with the appropriate partition)

mount -o bind /proc /mnt/gentoo/proc

Get the stage3 tarball, and extract it to a seperated directory, say, /mnt/gentoo/stage3 for example.

chroot /mnt/gentoo /bin/bb (note: bash WILL PROBABLY NOT work with a @#$%ed up libc, this is what busybox is for  :Very Happy:  )

cd /stage3/lib

cp * /lib (yeah, don't need to bother going for recursive)

cp /stage3/usr/lib/gcc/<arch>-pc-linux-gnu/<version>/libstdc++.so* /usr/lib/gcc/<arch>-pc-linux/<version>/

cp /stage3/usr/bin/python2.x /usr/bin

By this point you should be able to re-merge glibc, gcc, and python and boot into your normally scheduled Linux.  I recommend a hearty double dose of emerge -e system followed by an emerge -e world just to be safe.  Use cruft to intelligently get rid of the extra crap you have just put in (if it bothers you, anyway; it shouldn't hurt anything) 

If someone could improve this mini-howto and put it in the wiki that'd be a good thing for anyone who borks their libc, I think

----------

## 98111

Sorry, I wasn't at home   :Confused: 

 *Quote:*   

> It should roll right over /dev and /proc just fine. Just don't run it over any libs that you need to run... ESPECIALLY libc. In fact, just avoid doing it on a running system. It's safer 

 Are you sure that your problems resulted from a Shake bug ? smartctl and memtest didn't repported anything ?

I'm about to mark the new version of shake (0.23) as a stable one (no longer alpha), and proposed it for inclusion in Portage, so it's important for me that it works flawlessly (old bugs are fixed btw)

If it still had a bug, I am really confused   :Embarassed:  . I'll do all that I can to repair systems broke by Shake, if there is such. Just drop me a mail at the address given in "shake --help".

That said, those who wants to use the latest version can do it through the ebuild available there (Changelog).  It should be safe to run it while the system is used (except maybe by nfs, due to locks issues).

 *Quote:*   

> I understand that it would be best to run it on a non-system partition, but are there specific reasons /not/ to run it on my active system partitions?

 Some files used by Shake (like parts of the glibc) cannot be shocked while the system run.

 *Quote:*   

> Will it properly skip /dev/ and /proc/, since they're not real filesystems

 Yes, but it's greately advised to run shake with "--one-filesystem" to also avoid fat/ntfs partition. "--one-filesystem" will be the default for Shake 0.24Last edited by 98111 on Fri Aug 18, 2006 10:56 pm; edited 1 time in total

----------

## BlueFusion

I ran over a running system and it went fine.  The only problems were with the files in use, and that was just saying it could not move them.  When running chroot'd from a LiveCD, all but like 3 or 4 files could not be shook (bash, shake, and a few libs).  No issues with killing the syste, however.

----------

## Dralnu

Possibly you could have it load needed files to swap before running, so that you could run it on an active partition to keep it from going nuts over a file being in use.

----------

## Janne Pikkarainen

This is a "thank you" message.   :Very Happy: 

Most of the time I'm a Gentoo user, but my laptop is running Ubuntu Dapper. I just tried to shake its HD with shake v0.23 and cool, it works! I took a brave route and shaked *everything* (ok, just with sudo shake --verbose / without fancy flags, but still). Since HD of this poor thing has every now and then been practically full and I test all kind of stuff with it all the time, and this system has even been upgraded from Hoary to Breezy to Dapper, filesystem (ext3) had became a bit cluttered. 

Previously during bootup HD light would flash and HD was seeking a lot, at least it sounded like that. After running shake a reboot was a lot smoother: light was practically lit and background seeking noise was almost gone. I don't know about the boot time though, it's always been pretty snappy. 

e17 and all the applications I've loaded so far (xterm, Firefox, Thunderbird) seemed to load faster and with lot less seeking, too. Yay!

----------

## slick

Has somebody test it on a (dm-) encrypted partition? I think it's safe (because it is transparency), but I want not to try it if there are risks (for the important data on the encrypted partition).

----------

## 98111

 *Dralnu wrote:*   

> Possibly you could have it load needed files to swap before running, so that you could run it on an active partition to keep it from going nuts over a file being in use.

 File used by Shake are locked when it run, so it won't broke them, at least   :Razz: 

Linking static should do the trick. I don't think that's really usefull but I added a shake-static target to the future 0.24 Makefile  :Wink:  .

 *Janne Pikkarainen wrote:*   

> Previously during bootup HD light would flash and HD was seeking a lot, at least it sounded like that. After running shake a reboot was a lot smoother: light was practically lit and background seeking noise was almost gone. I don't know about the boot time though, it's always been pretty snappy. 
> 
> e17 and all the applications I've loaded so far (xterm, Firefox, Thunderbird) seemed to load faster and with lot less seeking, too. Yay!

 Thanks  :Very Happy: 

 *slick wrote:*   

> Has somebody test it on a (dm-) encrypted partition? I think it's safe (because it is transparency), but I want not to try it if there are risks (for the important data on the encrypted partition).

 It should be the same for Shake, but I did not tried. Please tell me if you do.

----------

## Diredicker

I tried to emerge shake but it wont compile, getting this error  :Sad: 

Anyone know what to do?

```
executive.c: In function `fcopy':

executive.c:64: error: passing arg 2 of `ioctl' with different width due to prototype

executive.c:146: error: passing arg 3 of `write' with different width due to prototype

executive.c: In function `shake_reg':

executive.c:232: error: passing arg 3 of `fcopy' with different width due to prototype

executive.c:246: error: passing arg 3 of `fcopy' with different width due to prototype

executive.c: In function `list_dir':

executive.c:346: error: passing arg 2 of `qsort' with different width due to prototype

executive.c: In function `list_stdin':

executive.c:405: error: passing arg 2 of `qsort' with different width due to prototype

judge.c: In function `judge_list':

judge.c:245: error: passing arg 1 of `abs' with different width due to prototype

judge.c:247: error: passing arg 1 of `abs' with different width due to prototype

judge.c:252: error: passing arg 1 of `abs' with different width due to prototype

make: *** [executive.o] Error 1

make: *** Waiting for unfinished jobs....

make: *** [judge.o] Error 1

!!! ERROR: sys-apps/shake-0.23 failed.

Call stack:

  ebuild.sh, line 1539:   Called dyn_compile

  ebuild.sh, line 939:   Called src_compile

  ebuild.sh, line 609:   Called die

!!! emake failed

!!! If you need support, post the topmost build error, and the call stack if relevant.

!!! This ebuild is from an overlay: '/usr/local/overlays'

```

----------

## 98111

 *Diredicker wrote:*   

> I tried to emerge shake but it wont compile, getting this error 
> 
> Anyone know what to do?
> 
> ```
> ...

 That bug again... sorry. It was an abusive use of -Wconversion .

Shake 0.24 (tarball, Changelog, ebuild) fix this problem, add "static" useflag and makes "--one-file-system" the default.

Please confirm me that you are on a non-x86_32 system, and that it fixed the bug.

----------

## zxy

I would like to try shake.

I have a list of files that I precache while booting. Is it possible to shake just these files. Will they be put together on the disk? Then boot should be done in a flash  :Very Happy: , I hope.

What is the command line for this. I have all files listed in a file. Each file in it's own line.

----------

## zxy

I just tried shake.

1. user_xattr makes my reiser4 partitions not being able to mount

2. reiserfs partitions boot ok with user_xattr

3. I tried shake -pvv on a file that is on a reiserfs partition. No results. 

I checked first with -pvv

```
shake -pvv /var/cache/http-replicator/apache-ant-1.6.5-src.tar.bz2

IDEAL   START   END     FRAGC   CRUMBC  AGE     SHOCKED NAME

0       1058804 1077680 6       0       0       0       /var/cache/http-replicator/apache-ant-1.6.5-src.tar.bz2
```

after without -pvv

```

localhost # shake  /var/cache/http-replicator/apache-ant-1.6.5-src.tar.bz2

localhost # shake -pvv /var/cache/http-replicator/apache-ant-1.6.5-src.tar.bz2

IDEAL   START   END     FRAGC   CRUMBC  AGE     SHOCKED NAME

0       1058804 1077680 6       0       0       0       /var/cache/http-replicator/apache-ant-1.6.5-src.tar.bz2

```

Why are there still 6 fragments?

----EDIT -----

I had to be on the same partition when starting shake.This way it works. 

But it still lets 17 fragments on other 236K file. It does nothing even if I run shake many times.   :Sad: 

----------

## 98111

 *zxy wrote:*   

> 
> 
> ```
> 
> localhost # shake  /var/cache/http-replicator/apache-ant-1.6.5-src.tar.bz2
> ...

 As you can see, SHOCKED=0 . This is because the file is new (AGE=0 days), and therefore they would be little or no benefits shaking it (as a fragmented new file generally mean a fragmented free space).

If you want to force it to be shocked, call 

```
shake --old=0 FILE
```

 and files older or as old than 0 day, that is all files, will be shocked. But the results may be negative.

For more information about advanced use of shake, please refer to http://vleu.net and to the manpage.

EDIT: If you have a list of file to be shocked, you can do 

```
cat list | shake
```

 to tell shake that you want them near each others, or 

```
shake `cat list`
```

 to tell it that they are independent.

But IMHO you won't see a great improvement between {prefetch} and {prefetch + shake}, as prefetch will reduce seeking to the lowest possible.

----------

## zxy

When I use reiser4 partition which cannot be mounted with xattr, I get a lot of negative numbers for start or end in the output. Is this ok?

----------

## indianiec

You can also check this stuff from Con Kolivas.

----------

## 98111

 *zxy wrote:*   

> When I use reiser4 partition which cannot be mounted with xattr, I get a lot of negative numbers for start or end in the output. Is this ok?

 No definitively. I would like to be sure that it is not a bug in reiserfs4. Could you download http://vleu.net/shake/fb_r4.c, compile it with 

```
gcc fb_r4.c -o fb_r4
```

 and run it as root on a file for which ShaKe detect the start at a negative value ?

If it return a negative value, I will contact the reiser team. In both case, I would like the output of "shake -pvv" on this file.

----------

## zxy

@indianiec

I know for his defrag.sh, but it doesn't put files together on the disk. I want this because of faster boot.

@98111

Here you have the outputs

```
localhost shake # ./fb_r4 /usr/portage/distfiles/Sunken.tar.gz

FIBMAP=-22, FIGETBSZ=4096

localhost shake # shake -pvv /usr/portage/distfiles/Sunken.tar.gz

IDEAL   START   END     FRAGC   CRUMBC  AGE     SHOCKED NAME

0       -22     -22     1       0       1       0       /usr/portage/distfiles/Sunken.tar.gz
```

Here is also the output from fsck.reiser4:

```
localhost shake # fsck.reiser4 /dev/md7

*******************************************************************

This is an EXPERIMENTAL version of fsck.reiser4. Read README first.

*******************************************************************

Fscking the /dev/md7 block device.

Will check the consistency of the Reiser4 SuperBlock.

Will check the consistency of the Reiser4 FileSystem.

Continue?

(Yes/No): yes

***** fsck.reiser4 started at Mon Aug 21 22:56:27 2006

Reiser4 fs was detected on /dev/md7.

Master super block (16):

magic:          ReIsEr4

blksize:        4096

format:         0x0 (format40)

uuid:           ddccfb51-f64d-44c5-9e3e-a86ea0d243f1

label:          <none>

Format super block (17):

plugin:         format40

description:    Disk-format for reiser4.

magic:          ReIsEr40FoRmAt

flushes:        0

mkfs id:        0x4437aaf2

blocks:         1469888

free blocks:    1433177

root block:     36675

tail policy:    0x2 (smart)

next oid:       0x1233b

file count:     1378

tree height:    3

key policy:     LARGE

CHECKING STORAGE TREE

        Read nodes 741

        Nodes left in the tree 741

                Leaves of them 726, Twigs of them 14

        Time interval: Mon Aug 21 22:56:27 2006 - Mon Aug 21 22:56:28 2006

CHECKING EXTENT REGIONS.

        Read twigs 14

        Time interval: Mon Aug 21 22:56:28 2006 - Mon Aug 21 22:56:28 2006

CHECKING SEMANTIC TREE

        Found 1378 objects.

        Time interval: Mon Aug 21 22:56:28 2006 - Mon Aug 21 22:56:28 2006

***** fsck.reiser4 finished at Mon Aug 21 22:56:28 2006

Closing fs...done

FS is consistent.

```

----------

## 98111

 *zxy wrote:*   

> Here you have the outputs
> 
> ```
> localhost shake # ./fb_r4 /usr/portage/distfiles/Sunken.tar.gz
> 
> ...

 For me it's a bug in Reiserfs4, as FIBMAP should never return negative values.

I'll tell them when I'll be sure (I don't have the time to look to that before tomorrow), but I'm afraid to have to tell you that you should not use shake on reiserfs4 before the problem is fixed.

----------

## Diredicker

 *98111 wrote:*   

>  *Diredicker wrote:*   I tried to emerge shake but it wont compile, getting this error 
> 
> Anyone know what to do?
> 
> ```
> ...

 

Yes i forgot to mention i'm using an AMD64 based system, and it emerged just fine now   :Laughing: 

----------

## 98111

 *zxy wrote:*   

> @98111
> 
> Here you have the outputs
> 
> ```
> ...

 It was a bug in reiserfs4. Gurus from namesys just fixed it there.

The latest shake  (0.27: tarball, Changelog, ebuild) handle the bug, but you should wait for a new version of reiserfs4.

By the way the project is now registered at Savannah (http://savannah.nongnu.org/projects/shake/). There you'll find a bugzilla, CVS and all that   :Wink: .

----------

## zxy

thanks

I'm looking forward to new reiser4

----------

## dalek

OK.  I saw where someone emerged this, where is it?

```
root@smoker / # emerge -s shake

Searching...

[ Results for search key : shake ]

[ Applications found : 0 ]

root@smoker / # equery list -p shake

[ Searching for package 'shake' in all categories among: ]

 * installed packages

 * Portage tree (/usr/portage)

root@smoker / #   
```

Just curious.  

 :Very Happy:   :Very Happy:   :Very Happy:   :Very Happy:   :Very Happy: 

----------

## yabbadabbadont

 *dalek wrote:*   

> OK.  I saw where someone emerged this, where is it?
> 
> ```
> root@smoker / # emerge -s shake
> 
> ...

 

You will have to download the ebuild from http://savannah.nongnu.org/projects/shake/ and setup a portage overlay to use it.

----------

## dalek

 *yabbadabbadont wrote:*   

> 
> 
> You will have to download the ebuild from http://savannah.nongnu.org/projects/shake/ and setup a portage overlay to use it.

 

Oh.  Are there plans to put it in portage?

 :Very Happy:   :Very Happy:   :Very Happy:   :Very Happy:   :Very Happy: 

----------

## Bigun

Would it be better to run shake when the system is booted?  Or boot from a live cd, chroot in, and run shake?

*bookmarked*

----------

## BlueFusion

I ran it from both a lice OS and a LiveCD.  The problem with running it from a live OS is that alot of files are in use and can't be shaken (shook?) by the program.  Running from a LiveCD, only 2 or 3 files were in use (shake and some libs).

----------

## Bigun

Has someone tested this on an AMD64 system yet?

----------

## GenYetiToo

 *bigun89 wrote:*   

> Has someone tested this on an AMD64 system yet?

 

What difference could a particular processor architecture possibly make?

In other words: do you have specific problems in mind that could occur?

M.

----------

## Bigun

ebuild problems...

----------

## Merlin-TC

Yep, I did it and it just works fine on an AMD64

----------

## Bigun

I just borked my libs during use of shake while the system was booted.

Running it, and the screen went nuts.

Rebooted, and my libc was gone... yay

Seriosuly folks... try to do this from a live CD if you can...

----------

## devsk

 *bigun89 wrote:*   

> I just borked my libs during use of shake while the system was booted.
> 
> Running it, and the screen went nuts.
> 
> Rebooted, and my libc was gone... yay
> ...

 why doesn't shake refuse to shake files which are in use? It should shake what it can and leave the rest alone. Isn't that what any sane program should do? This, IMO, is a major problem.

----------

## Phenax

I refused to shake the files that were in use on my system. Great utility IMO.

----------

## devsk

so, is that a bug then? or is he using an old version?

----------

## Phenax

I used version 0.27, the latest. I shook my whole harddrive while it was all mounted, no problems. If it was in use it gave a warning and then passed the file.

----------

## BRPXQZME

 *devsk wrote:*   

>  *bigun89 wrote:*   I just borked my libs during use of shake while the system was booted.
> 
> Running it, and the screen went nuts.
> 
> Rebooted, and my libc was gone... yay
> ...

 I think I know what's going on here (at least this part I know I @#$%ed up, I hope bigun89 figured out how to fix it... did my method work for you?):

-o0 (--old 0) option considered harmful on partition with libs being used!

Do it without that option, or get another "partition" to run shake from, like a livecd.  ...well, perhaps a "defrag diskette" is in order here.  Of course, it'd help if shake figured out what was going on.  As if watching all the programs you might be using die as shake clobbers their libs wasn't enough of a clue.

----------

## Bigun

 *BRPXQZME wrote:*   

>  *devsk wrote:*    *bigun89 wrote:*   I just borked my libs during use of shake while the system was booted.
> 
> Running it, and the screen went nuts.
> 
> Rebooted, and my libc was gone... yay
> ...

 

My commandline to run shake looked like this:

```
shake /
```

So, no commandline options.

To note, I ran it exactly the same way on another machine with no problems, it seems that somehow this little problem is left to 'chance', I'm not sure.

At any rate, your solution got me to the point that I could emerge glibc, but beyond that, emerge refused to work due to some function missing... I'm not sure.  So I'm emerging crap in reverse:  python, gcc, then glibc.  It's been working on it since I've left for work, I'll let you know how it went when I get back home.

----------

## Bigun

I seem to be able to get my system to emerge python and gcc just fine, but as soon as it gets to glibc, it fouls up.  Emerge and a bunch of other programs stop workingas soon as I do.  And I'm getting the same error this fella is getting:

https://forums.gentoo.org/viewtopic-t-473318-highlight-libpthread+glibcprivate.html

----------

## W3BMAST3R101

I know this is alpha software, but it seems that i can't really use it. It runs through and it seems like it's working for about 5 min, but then it just kicks out an error on a random file and just stops. Is there any way for it just to skip files and move on if it doesn't like them? Also, how's the software going. I like the idea. I hope development continues.

thnx,

the w3b

----------

## mudrii

 *W3BMAST3R101 wrote:*   

> I know this is alpha software, but it seems that i can't really use it. It runs through and it seems like it's working for about 5 min, but then it just kicks out an error on a random file and just stops. Is there any way for it just to skip files and move on if it doesn't like them? Also, how's the software going. I like the idea. I hope development continues.
> 
> thnx,
> 
> the w3b

 

try to trace the execution of the shake

```
#emerge strace

#strace shake
```

----------

## AlphaHeX

What are the shake parameters to shake all files no matter of size, age and and number of fragments ?

----------

## nowinter

0.27 doesn't compile on my system due to lack of attr/attributes.h. What is it?? The only attributes.h I have are xine's and libxslt's   :Rolling Eyes: 

actual error message:  *Quote:*   

> 03:21:47|nwhost shake-0.27 # make
> 
> gcc -std=gnu99 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -O2 -D_POSIX_C_SOURCE=200112L -Wall -pedantic-errors  -Wcast-align -Wpointer-arith -Wbad-function-cast -DVERSION=\"0.27\" -DNDEBUG -c executive.c -o executive.o
> 
> gcc -std=gnu99 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -O2 -D_POSIX_C_SOURCE=200112L -Wall -pedantic-errors  -Wcast-align -Wpointer-arith -Wbad-function-cast -DVERSION=\"0.27\" -DNDEBUG -c judge.c -o judge.o
> ...

 

----------

## AlphaHeX

you need to emerge attr  :Smile: 

----------

## nowinter

 *AlphaHeX wrote:*   

> you need to emerge attr 

 

 :Razz:   :Razz:   :Razz:   :Idea:   :Embarassed:  i knew it, i swear )))))))))))

----------

## 98111

 *bigun89 wrote:*   

> I just borked my libs during use of shake while the system was booted.
> 
> Running it, and the screen went nuts.
> 
> Rebooted, and my libc was gone... yay
> ...

 This sort of problems are more likely to result from a system/hardware problem triggered by sHaKe.

That being said I would really like to help you, but I don't understand what's going on  :Sad:  . Maybe you should copy a working /usr/lib and /lib from another system into /tmp/lib, change LD_PATH to make it include /tmp/lib and bootstrap from that.

 *W3BMAST3R101 wrote:*   

> I know this is alpha software, but it seems that i can't really use it. It runs through and it seems like it's working for about 5 min, but then it just kicks out an error on a random file and just stops

 Could you tell me the name of that file, and of the error ?

 *W3BMAST3R101 wrote:*   

> I hope development continues.

 Yes it do ! It makes a long time since I hadn't recieved a bug report, so sHaKe don't changed two much over last weeks, but I'm ready to fix yours.

----------

## Gregoire

Hello,

I would like to shake all my partitions on my amd64 system.

Before I do it, I would like a comment on my fstab :

```

/dev/sda1 swap swap defaults,pri=2 0 0

/dev/sda2 /mnt/cooker xfs noatime 1 1

/dev/sdb1 swap swap defaults,pri=3 0 0

/dev/sdb2 / reiserfs user_xattr,noatime 0 0

/dev/sdc2 swap swap defaults,pri=2 0 0

/dev/sdc3 /boot ext2 noatime 1 1

/dev/sdc4 /mnt/gentoo  xfs noatime 1 1

none /dev/shm tmpfs defaults 0 0

none /proc proc defaults 0 0

```

Should I change anything before shaking ?

Thank you very much  :Smile: 

----------

## 98111

 *Gregoire wrote:*   

> [...]
> 
> ```
> /dev/sdb2 / reiserfs user_xattr,noatime 0 0
> ```
> ...

 Yes, you should change the above line into

```
/dev/sdb2 / reiserfs notail,user_xattr 0 0
```

, add user_xattr and remove noatime to others partitions that you want to shake, like in

```
/dev/sdc4 /mnt/gentoo  xfs user_xattr 1 1
```

Then use your system a day or two (to generate atimes) before shaking it.

----------

## Gregoire

 *98111 wrote:*   

> add user_xattr and remove noatime to others partitions that you want to shake, like in
> 
> ```
> /dev/sdc4 /mnt/gentoo  xfs user_xattr 1 1
> ```
> ...

 

Thank. I don't have user_xattr for XFS and I can't find it in the kernel menuconfig, maybe that's not for amd64 ?

----------

## 98111

 *Gregoire wrote:*   

>  *98111 wrote:*   add user_xattr and remove noatime to others partitions that you want to shake, like in
> 
> ```
> /dev/sdc4 /mnt/gentoo  xfs user_xattr 1 1
> ```
> ...

 Sorry  :Confused:  I forgot that user_xattr were enabled by default on xfs.

```
/dev/sdc4 /mnt/gentoo  xfs default 1 1
```

 should do the trick.

----------

## Drewgrange

From now on I'm going to ignore the issue of disk fragmentation. I ran shake and got tons of errors, then a kernel panic when I rebooted. 

I know, this isn't helpful information for you to fix a possible bug, but I didn't write anything down. 

Anyway, I'm writing this from an ubuntu install (until I have enough time too reinstall gentoo eventually) and using photorec to see what I can recover.

I think if this program has the ability to mess things up, it should come with a warning and some safeguards

----------

## 98111

 *Drewgrange wrote:*   

> I think if this program has the ability to mess things up, it should come with a warning and some safeguards

   :Exclamation:  The problem is that Shake use only documented syscall()s, call them on regular files and sp should not be hable to do that.

For now I think that problems (two if I remember well) occured on systems which would have failed any other intensive tasks on the whole hard disk.

I could add a warning to tell that shake should be run only after a fsck, but if fsck fail instead of shake it will be worst.

That said, I am really sorry for your hard disk. Magicrescue usually give good results, but installing Ubuntu probably wrote over your old datas.

If you installed it on another partition, the old one could be saved by fsck... (mail me if you want some help about that).

If all cases, the next tools you should try are "smartcl -t long", to test your hard disk, and memtest86+ . If they succeed, then the problem was in the filesystem and should not occur again.

If you have time to loose, you could run Shake again on your ubuntu before installing gentoo over it.

And by the way : which filesystem/kernel was it ? on which hardware ?

----------

## sobers_2002

on reiser4, I don't see any option for xattr in kernel and shake give operation not permitted errors for user_xattr, so what's the issue?

EDIT: okay, tried with -X, but now shake segfaults at two places and after that glibc is kind of gone. why is that?

----------

## mrsaccess

Hello, congratulations for this amazing piece of software you wrote!

I used it yesterday with very good results! The speedup was really big in apps that their libraries aren't usually loaded in my ram, ie wine and gtk apps. In kde apps (kde is my desktop so it's libraries are always in the ram)  the speedup was small but noticeable. Also I noticed a nice speedup in apps that use the filesystem as a cache, for example digiKam which creates thumbnails of your photos in order to load faster.

Unfortunately I ran into a small -for my occasion- problem and I want to share it in case anyone else comes across it.

Before running shake I closed the X server and switched to a virtual terminal. From there I used shake to defrag my /:

```
$ shake /
```

My fstab line for / (and most of my partitions):

```
/dev/sda3               /               reiserfs        defaults,noatime,notail,user_xattr      0 1
```

Although noatime is used it worked well.

At some point of the process (I guess 1 to 2 hours after it started) the disk activity stopped entirely and my computer wouldn't accept any input except Alt+Fnumber to switch virtual terminals. Also it didn't respond to ssh connection attempts (but it responded to ping).

After a fair amount of waiting and since there wasn't any disk activity for many minutes I reset my computer.

Upon boot I had a kernel panic because the file /lib/libc.so.6, which is part of glibc, was corrupted. I guess that is what "froze" the pc in first place.

Luckily for me I had another machine with gentoo so I copied it's libc.so.6 to my machine and everything went fine. Till now I haven't noticed any other problems (emerging deep update world right now).

Unfortunately if someone hasn't another gentoo installation near will be in a difficult position as with a corrupted glibc you can't even chroot to your system from a livecd in order to fix things.

Hope my comments will help a bit.   :Very Happy: 

----------

## 98111

You're right. And the problem is serious.

I'm really sorry about damaged systems. Hopefully, it should'nt occur on all libc, but that do not excuse me.

The problem is that writing to /lib/libc-2.4.so crash the system. Even if you respect the correct order (take a write lock, then shake, then release)(imho the write lock should be refused).

Difficulty is that, in order to ensure I spot the bug, I wrote a short code wich done that  :Razz:  . So my computer no longer boot.

If I remind well, older version of ld.so aquired a lock() on open libs. I don't know if the new behaviour is a bug or a feature, but shake need a workaround.

I have some ideas about it, but I first need to find a glibc guru. I must go now, but will do the update this evening (arround 19hour GMT).

I lack english words to ask you to forgive me. Thanks for your clear bugreport which allowed me to spot the problem. Also I hadn't the time to check for grammar  and spelling... as you see  :Confused: 

Shame on me

----------

## mrsaccess

I really appreciate your fast response to the problem report.   :Very Happy: 

Don't worry for the "damage", no harm done! Besides I'm too happy with the result to care about libc.   :Razz: 

PS. English isn't my native language so I have to worry about the mistakes I make instead noticing other people mistakes.   :Very Happy: 

----------

## 98111

Version 0.29 (tarball, Changelog, ebuild) is available an include a quick and dirty workaround.

Will investigate more and make a 0.30 release ASAP.

----------

## Dralnu

Something to think about would be to require the target fs to be unmounted (i.e. run it from a LiveCD). I would think there would be more problems running on a mounted FS then not, so that might be worth looking into.

----------

## getchoo

I'd like to try this program, but it's giving me the following error:

```

tux shake-0.29 # ls

COPYING    GPL.txt   distrib  executive.c  executive.o  judge.h  linux.c  main.c  msg.h

Changelog  Makefile  doc      executive.h  judge.c      judge.o  linux.h  msg.c   unattr.c

tux shake-0.29 # make

gcc -std=gnu99 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_XOPEN_SOURCE=600 -O2 -D_POSIX_C_SOURCE=200112L -Wall -pedantic-errors  -Wcast-align -Wpointer-arith -Wbad-function-cast -DVERSION=\"0.29\" -DNDEBUG -c linux.c -o linux.o

linux.c:23:43: error: attr/attributes.h: No such file or directory

linux.c: In function 'set_ptime':

linux.c:42: warning: implicit declaration of function 'attr_setf'

linux.c:43: error: 'ATTR_DONTFOLLOW' undeclared (first use in this function)

linux.c:43: error: (Each undeclared identifier is reported only once

linux.c:43: error: for each function it appears in.)

linux.c: In function 'get_ptime':

linux.c:55: warning: implicit declaration of function 'attr_getf'

linux.c:55: error: 'ATTR_DONTFOLLOW' undeclared (first use in this function)

make: *** [linux.o] Error 1

```

 :Shocked: 

----------

## 98111

 *getchoo wrote:*   

> I'd like to try this program, but it's giving me the following error:
> 
> ```
> linux.c:23:43: error: attr/attributes.h: No such file or directory
> ```
> ...

 Please install sys-apps/attr . You can also use the ebuild.

----------

## Jkay

Hi. I haven't used Shake since last summer, so I grabbed 0.29 and shook:

/home

/var/db/pkg

/var/cache/edb

/usr/portage

Am I missing any other directory which Portage makes a mess in? 

I only ran Shake in non-system dirs, for now ... until things settle, and we're non-alpha!! Just getting 

```

emerge --sync && emerge -uavDN world 

```

to be a lot faster is good enough for now!!

Thanks again for the great work.

EDIT - I should mention that I have /var and /usr and /home and / all on separate hard drives or partitions.

----------

## Dralnu

 *Jkay wrote:*   

> Hi. I haven't used Shake since last summer, so I grabbed 0.29 and shook:
> 
> /home
> 
> /var/db/pkg
> ...

 

I think /var/tmp and /tmp get messed up real good at times.

----------

## psyqil

 *Dralnu wrote:*   

> I think /var/tmp and /tmp get messed up real good at times.

 I think those should be deleted rather than 'shaken'...

----------

## Dralnu

 *psyqil wrote:*   

>  *Dralnu wrote:*   I think /var/tmp and /tmp get messed up real good at times. I think those should be deleted rather than 'shaken'...

 

True, but he asked what he missed that portage makes a mess with. If you don't reboot all the time (like some of us), it could make a very minor diff  :Smile: 

----------

## sobers_2002

 *Dralnu wrote:*   

> Something to think about would be to require the target fs to be unmounted (i.e. run it from a LiveCD). I would think there would be more problems running on a mounted FS then not, so that might be worth looking into.

 

that should be the case, but the problem reported by mrsaccess happened even when I ran it via livecd   :Crying or Very sad:   :Crying or Very sad:   :Crying or Very sad:   I tried it with v0.28 on reiser4. I am not sure about the performance increase because I was too afraid to experiment after that corruption   :Razz:   :Razz: . If it's fixed in v0.29 I'll try it again and report back. 

and it really does help to have another box of gentoo around   :Very Happy: 

----------

## Bones McCracker

This looks interesting.  I am looking forward to a stable version.

----------

## mudrii

Hei

Big Thx for grate pice of soft 

Just waiting for 0.30

----------

## Bones McCracker

So what's the status?  Have you "submitted it for inclusion in gentoo"?

----------

## AssociateX

Trying it now... yippy, this is fun.

Is there a way to make the columns line up?

So the stuff for the following are really under them: IDEAL   START   END     FRAGC   CRUMBC  AGE     SHOCKED NAME

Thanks a ton!

----------

## tabascoz

Hi!

First, thanks for this great program ! 

I have been using shake with no problems in reiser4 until 2.6.22 series. 

Shake gives  errors like that in all files: 

```
shake: /mnt/incoming//firmware/licencas/00059e80d0a1-v5_0-8186.dat: FIBMAP failed: Invalid argument
```

I am using reiser4 + criptocompress + ragnarok-2.6.22-r1

i think this is a reiser4-related bug, in others partitions on the same system with ext3 i have no problems.

It can be fixed in a new version?

Thanks!

----------

## hitachi

And please remember: your file system needs you!

Like if you are using something on torrent, lets say downloading the debian bluray installation iso you might end up with one iso file separated in 79425794 pieces. Consider using cp the file from your scratch/torrents/ to where you want the file for future use instead of mv. Well I know you need to have the space on the disk an then to remove the original file but it might be worth it.

----------

## albright

sorry here's a dumb question: how do I tell the extent of

fragmentation of a whole directory or partition?

(I know that fsck shows this but can I run this on a live

mounted filesystem?)

----------

## a.b.

 *albright wrote:*   

> 
> 
> (I know that fsck shows this but can I run this on a live
> 
> mounted filesystem?)

 

```
fsck -n /dev/foo
```

----------

## albright

Thanks for not just saying RTFM (which would have

been fair enough I must admit)

----------

## albright

this is getting off topic, but ...

that -n option works on ide drive but on sata there is

no report of fragmentation:

```
wes64@gothmog ~ $ fsck -n -C /dev/sda5

fsck 1.41.2 (02-Oct-2008)

e2fsck 1.41.2 (02-Oct-2008)

Warning!  /dev/sda5 is mounted.

Warning: skipping journal recovery because doing a read-only filesystem check.

/dev/sda5: clean, 473522/1531072 files, 2397328/3058366 blocks
```

Also, everytime I run fsck -n on my ide notebook, it reports file system

errors (even after running fsck for real at bootup). I'm wondering if there

is a serious problem with this drive, or what.

Like this:

```
wesfuj@thorin ~ $ fsck -n -C /dev/hda4

fsck 1.41.2 (02-Oct-2008)

e2fsck 1.41.2 (02-Oct-2008)

Warning!  /dev/hda4 is mounted.

Warning: skipping journal recovery because doing a read-only filesystem check.

/dev/hda4 contains a file system with errors, check forced.

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

Free blocks count wrong (3551097, counted=3551093).

Fix? no

Free inodes count wrong for group #679 (15354, counted=15357).

Fix? no

Free inodes count wrong (13278975, counted=13278964).

Fix? no

/dev/hda4: ********** WARNING: Filesystem still has errors **********

/dev/hda4: 631041/13910016 files (9.1% non-contiguous), 24257418/27808515 blocks
```

----------

## SlashBeast

I tried to use this soft but..

```
slashbeast@mizore build % ./shake ~/foo

./shake: /home/slashbeast/foo: FIBMAP failed: Operation not permitted
```

Any idea?

----------

## gimpel

 *SlashBeast wrote:*   

> I tried to use this soft but..
> 
> ```
> slashbeast@mizore build % ./shake ~/foo
> 
> ...

 

 *http://vleu.net/shake/ wrote:*   

> How to use it ?
> 
> [...]
> 
> As root, call shake my_dir

  :Wink: 

----------

## as.gentoo

```
$> shake --pretend foo.bin

shake: foo.bin: FIBMAP failed: Operation not permitted
```

Maybe "You have to execute shake as root" would be a more meaningful error message.

If it's root only, shouldn't it be placed into /usr/sbin?

```
$> which shake

/usr/bin/shake
```

Well, the website says

```
[…]

Short version

As root, call shake my_dir , and go do something usefull or pleasant until it completes.

[…]
```

IMHO the man page should point out that you can only use it as root!

----------

## Ant P.

It would probably be more productive to use the project's issue tracker than bumping a 9 year old thread.

----------

