# How to recover a lost world file

## fonic

Hi everybody,

over the years, the Gentoo forum helped me out quite a few time and therefore I think it's time to give something back. So here's my first post - hopefully it proves to be useful for others. If I chose the wrong category for this post, please move it to where it belongs.

How your world file may get lost

I guess there are countless ways of how to loose the portage world file. In my case, two simple but serious mistakes lead to an empty world file. As many others, I'm using rpm in Gentoo for the sole purpose of installing drivers for printers and scanners made by Brother (e.g. the MFC series). Yesterday, I wanted to update those drivers. Therefore, I issued rpm -e brscan[...] in order to remove the old drivers prior to installing the new ones.

This was the first serious mistake, since you must NEVER ever use rpm without the --nodeps option if rpm is not the main package manager on your system. Anyway, issuing said command left me with a completely messed up system: several important files like 'libstdc++.so' and 'stdio.h' were missing, gcc was unable to create executables and so on. I wasn't able to emerge anything, basic programs like eix and locate stopped working - long story short, the system was almost unusable. However, I was able to use links and wget to download a current stage3 tarball which I planned to use to get the basic system functions back online.

I knew that extracting the tarball would probably overwrite a lot of important files, so I created a backup of /etc. That was the second serious mistake I made - before extracting a stage3 tarball in order to recover a broken system, one should backup EVERYTHING, not only /etc. So, after extracting the tarball most parts of the system like portage, gcc and actually most other applications as well were back online, but the portage world file located at '/var/lib/portage/world' had been overwritten and was now empty.

All of this took place after 2am. As it is mentioned in How I met your mother, "No good things happen after 2am. After 2am, just go to bed". They were totally right... Gentooish high five...

Recovering a lost world file

There is that well-known command called regenworld, which attempts to recover lost world files. In my case, that did not help at all, since regenworld tries to find out about package installations that took place in the past by parsing portage's log files. Those were gone too by extracting the tarball, therefore regenworld did not report a single package on my system.

After trying several administrative commands, I noticed there are at least two ways of getting a list of currently installed packages: equery and eix. Both of them use their own databases and seem to be unaffected by a missing world file. Sadly, the package lists provided by equery and eix contain ALL packages currently installed on the system, i.e. programs, libraries, fonts, codecs etc. The world file, however, should only contain packages that were installed on demand.

```
# equery list -i

# eix-installed -a
```

But then I came up with an idea: packages that are installed on demand most likely have no dependencies to other packages - if they had, they would have already been pulled in by other packages during the emerge process. I verified that thought with a few equery depends <package> on packages like vlc, amarok, kdebase-meta etc. which I knew had been installed on demand. The results were very promising.

Using a script to automate the recovery process

Based on the idea described above I created a small but effective recovery script which collects a list of all currently installed packages and then automatically checks the dependencies for each package. The script creates a text file containing all packages without dependencies, which may directly be used as a replacement for the lost world file. On my system, the script recovered over 90% of my world file (80 entries out of 85 total). The recovery process takes some time (about 30 min. in my case) since equery depends queries are very slow - if you have an idea on how to speed things up, feel free to comment accordingly.

Some packages won't be reported by the script because of their special dependency structure. One example is 'sun-jre-bin', which depends on 'virtual/jre' and is therefore omitted by the script. But those few missing packages may easily be spotted by checking the result of emerge -pv --depclean after copying the recovered world file to '/var/lib/portage/world'. In my case, I only had to add five packages reported by depclean in order to have my world file recovered to 100%.

All in all, my system was up and running again after about three hours (including the time it took to write the script) which is well below the amount of time it would have taken to install Gentoo and all my applications from scratch. By using the script, other users with a similar problem might be able to recover their system in less than an hour, I guess.

Find the recovery script I created attached below. Feel free to comment if you have any suggestions on how to improve it.

Best,

fonicLast edited by fonic on Sun Mar 20, 2011 11:43 am; edited 4 times in total

----------

## fonic

Since there seems to be no way of attaching files to posts, here's the script using the code tag:

```
#!/bin/bash

#

# recover_world.sh by fonic

# Created: 03/19/11

# Modified: 03/19/11

#

# Configuration

dwork=./rebuild_world

fstep1="1_world_raw"

fstep2="2_world_stripped"

fstep3="3_world_recovered"

# Create working directory and flush files

echo -n "Creating working directory... "

mkdir -p $dwork

cd $dwork

> $fstep1

> $fstep2

> $fstep3

echo "ok"

# Step 1:

# Get a raw list of the currently installed packages. Both eix-installed 

# and equery will provide such lists. Note that either list will include

# all sorts of packages (i.e. libraries, fonts, codecs etc.), not only the

# main programs we're looking for.

echo -n "Grabbing a raw list of currently installed packages... "

#eix-installed -a > $fstep1

equery -q list -i > $fstep1

numpackages=$(cat $fstep1 | wc -l)

echo "$numpackages packages found"

# Step 2:

# Process raw list and remove version numbers. The code also accounts for

# packages with additional release information (e.g. qt-webkit-4.6.3-r1).

echo -n "Stripping version information from package list... "

while read package; do

   echo ${package%-[0-9]*} >> $fstep2

done < $fstep1

echo "ok"

# Step 3:

# Process package list and determine the number of dependencies for each

# package. Packages WITHOUT dependencies are most likely packages that were

# installed on demand (and therefore need to be put in the world file).

while read package; do

   # Determine number of depedencies of the current package

   echo -n "Processing \"$package\"... "

   numdeps=$(equery -q depends $package | wc -l)

   if [ $numdeps -eq 1 ]; then

      echo "$numdeps dependency"

   else

      echo "$numdeps dependencies"

   fi

   # If there are no dependencies, a world entry is found

   if [ $numdeps -eq 0 ]; then

      echo "-> Packages was added to the list of recovered world entries."

      echo $package >> $fstep3

   fi

done < $fstep2

# Done, show results and depclean notice

numpackages=$(cat $fstep3 | wc -l)

echo -e "\nThe recovery process is completed. $numpackages world entries were identified and stored in '$fstep3'.\n"

echo -e "Make sure to check 'emerge -pv --depclean' for packages that were not found by this script.\n"

```

----------

## nenemsis5

does the command: "regenworld" not doing the same ??

----------

## dwbowyer

 *Quote:*   

> does the command: "regenworld" not doing the same ??

 

For those that didn't actually bother to read the entire original post:

 *Quote:*   

> 
> 
> Recovering a lost world file
> 
> There is that well-known command called regenworld, which attempts to recover lost world files. In my case, that did not help at all, since regenworld tries to find out about package installations that took place in the past by parsing portage's log files. Those were gone too by extracting the tarball, therefore regenworld did not report a single package on my system.
> ...

 

Now, I do wonder that extracting the tarball over a live system would have overwritten log files though... is that the default portage /var/log/build.log rather than the individual logs written to /var/log/portage/<package-group>:<package-name>-<packageversion><datetime>.log?

----------

## fonic

 *dwbowyer wrote:*   

> 
> 
> For those that didn't actually bother to read the entire original post:
> 
> 

 

Thank you! I can't stand people commenting on something they actually haven't read...

 *Quote:*   

> 
> 
> Now, I do wonder that extracting the tarball over a live system would have overwritten log files though... is that the default portage /var/log/build.log rather than the individual logs written to /var/log/portage/<package-group>:<package-name>-<packageversion><datetime>.log?

 

Well, I can't tell exactly what happened or how it happened. My focus that night was to get the system running again, therefore I didn't spend much time on finding out what lead the data loss. It might have also been rpm that caused the missing portage log files since it destroyed a lot on the system. Fact is, regenworld could not find anything in that situation while my way (or: my script) did. And even if regenworld helps in 90% of those cases, I think it never hurts to have a backup solution.

----------

## Nicias

I trimmed your script, and just used the last stage to clean up my world file. I saw I could delete 25 of the 99 entries from my world file.

 Then I uncovered a problem with it. If a package has a conditional dependency, it will flag it as a dependency. For instance I have gimp installed from world, and gtkam installed. gtkam has the following line in its RDEPEND  "gimp? ( >=media-gfx/gimp-2 )" Since I don't have the gimp use flag set, it doesn't acually depend on gimp, but equery sees that it does. So your script flags gimp as non-world-worthy.

----------

## Nicias

After doing more work, I'd be very cautious about using this. I had 99 packages in my world file, it only said I needed 74. Upon further inspection, I actually need 92. So it missed almost 25% of the packages that should be in my world. 

Having said that, it's still better than nothing, if you have no choice. 

I would just suggest backing up.

----------

## drutten

Thank you for this post and script! Did the same thing (unpacked stage3 tarball) one evening when everything was going wrong. Should have gone to bed instead. I figured a solution like this should work and thought of writing a script, but then realised it must already exist somewhere   :Laughing: 

Unfortunately they changed the use-pattern of equery -q list

```
# equery -q list -i

List installed packages matching the query pattern

!!! Default action for this module has changed in Gentoolkit 0.3.

!!! Use globbing to simulate the old behavior (see man equery).

!!! Use '*' to check all installed packages.

!!! Use 'foo-bar/*' to filter by category.

Usage: list [options] pkgspec

options

 -h, --help              display this help message

 -d, --duplicates        list only installed duplicate packages

 -b, --missing-binpkgs   list only installed packages without a corresponding binary package

 -f, --full-regex        query is a regular expression

 -m, --mask-reason       include reason for package mask

 -I, --exclude-installed exclude installed packages from output

 -o, --overlay-tree      list packages in overlays

 -p, --portage-tree      list packages in the main portage tree

 -F, --format=TMPL       specify a custom output format

              TMPL       a format template using (see man page):

                         $location, $mask, $mask2, $cp, $cpv, $category, $name, $version, $revision, $fullversion, $slot, $repo, $keywords

```

You should either uncomment the eix-installed command and comment out equery, or change it to 

```
equery -q list '*' > $fstep1
```

Thanks again!

/André

----------

## soltoo

Thanks for the script. 

I did notice that dev-lang/python was not in my world file before and this script didn't include it either.

Running "emerge -p system | grep python" doesn't show it.  Running "emerge -p world | grep python" doesn't show it either.

Plus dev-lang/python was out-of-date until I ran "emerge python" even after I ran "emerge world".

Admittedly I'm running Gentoo Prefix on Solaris 10.  I started using it in July 2011--I don't think I ever had a python 2.6 installed.

```
eselect python list 

Available Python interpreters:

  [1]   python2.7 *
```

nothing else.

Does anyone else see this behavior too?

----------

## mittah

Very good and useful script! Thank you!

My using stats:

found by script: 124

missed: 16

Nice result!

----------

