# emerge -pe world resuming

## Dolio

So I tried an emerge -pe world last night, but it ended up failing on GRUB. However, I thought to myself, nothing depends on GRUB, so it doesn't really matter if it failed to build. Further, I've already got all this stuff installed, so it shouldn't make much of a difference if stuff fails along the way, I want it to go all the way to the end.

Also, I've seen many people in the forums (including myself) that would like to resume an emerge -pe world. So, this morning, I wrote a couple scripts that (hopefully) solve the problem to some degree (and if anyone finds a problem, please correct me here).

This first one builds a list, and then emerges each item on the list, printing the failed packages to a file (/tmp/failedpkg)

Edit: Okay, I'm done re-merging my system. The first script I displayed worked (only two failures on my remerge. Grub and glibc. I guess the new binutils really doesn't work on glibc. I'll have to try the slightly older 2.12 version). So, I tweaked the scripts a little bit. I'll explain the changes from the original after the script. Here goes:

Update: Many thanks to fghellar, who suggested I replace my incredibly ugly method of retrieving the package names from emerge -pe world with something much nicer (I need to retake sed 101  :Embarassed: ) (Also, it should be noted, that this is more correct than my version. My old version deleted the extra stuff around the package name, and there could theoretically be stuff in packages that I didn't anticipate (it deleted _beta, but it wouldn't delete _alpha), so you should update your old scripts even if you like ugly things):

```

#!/bin/bash

PKGNAME=""

if [ -n "$1" ]; then

        PKGNAME=$1

else

        PKGNAME='world'

fi

FNAME="/tmp/$PKGNAME.lst"

FAILNAME="/tmp/$PKGNAME.failed"

#Remove the previous emerge -pe world | ... command and replace it

#with the following:

emerge -pe "$PKGNAME" | egrep 'ebuild' | sed -e 's:\(.*\) \(.*\)/\(.*\)-\([0-9].*\) \(.*\) \(.*\):\3:' > "$FNAME"

for package in `cat $FNAME`

do

        emerge --oneshot $package || echo "Emerge of $package failed" >> "$FAILNAME"

done

rm "$FNAME"

if [ `egrep -c "^.+$" $FAILNAME` -le 0 ]; then

        rm "$FAILNAME"

fi

```

Note also that the stuff between the 'FAILNAME' and the for is on one line, even though it wraps.

Continue Edit: Okay, so, it used to be you'd just type in the script name and it'd remerge your entire system. The way it's set up now, you can do that, and it will still remerge your entire system, or you can do:

./blah.sh PACKAGE

And it will only remerge the specified package.  Similar changes will be posted shortly for the resuming script (along with a handy resume keyword).

----------

## Dolio

And here's the one we've all been waiting for (Similar code initially):

<EDIT>

This code has now been updated.

Update: Again, many thanks to fghellar for making things prettier and more correct (and probably faster, too):

Update 2: The code below is old. However, I'm about 80% sure it works as advertised. I wrote a new, fancier version of this, and it works from what I can tell. However, I haven't tested it at any length as of yet. So, I'm going to keep the old version up here, and post the new version below. Look there if you want something fancy.

```

#!/bin/bash

RESUME=0

PKGNAME=""

if [ -n "$1" ]; then

        if [ "$1" == "resume" ]; then

                RESUME=1

                if [ -n "$2" ]; then

                        PKGNAME=$2

                else

                        PKGNAME='world'

                fi

        else

                PKGNAME=$1

        fi

else

        PKGNAME='world'

fi

FNAME="/tmp/$PKGNAME.lst"

if [ $RESUME -eq 1 ] && [ -e "$FNAME" ]; then

    echo "Resuming previous merge"

else

    echo "Beginning new merge"

    emerge -pe "$PKGNAME" | egrep 'ebuild' | sed -e 's:\(.*\) \(.*\)/\(.*\)-\([0-9].*\) \(.*\) \(.*\):\3:' > "$FNAME"

fi

FAILED=0

cp -f "$FNAME" "$FNAME.tmp"

for package in `cat $FNAME`

do

    emerge --oneshot $package || FAILED=1

    if [ $FAILED -ne 1 ]; then

        sed "s/$package//g" "$FNAME.tmp" > "$FNAME.tmp.2"

        mv -f "$FNAME.tmp.2" "$FNAME.tmp"

    else

        egrep "^.+$" "$FNAME.tmp" > "$FNAME"

        exit

    fi

done

rm "$FNAME"

rm "$FNAME.tmp"

```

I still haven't tested this one as thoroughly as the other one, so if someone could do that, it'd be helpful. Here's how you use this, let's assume you named it 'resmerge':

'resmerge': starts a new 'emerge -pe world' FROM SCRATCH, EVEN IF ONE WAS IN PROGRESS

'resmerge resume': attempts to resume a previous 'emerge -pe world', and, if not possible, begins a new one from scratch

'resmerge PACKAGE': same as the first one, except it attempts to emerge PACKAGE from scratch

'resmerge resume PACKAGE': same as the second one, only with the specified package.

If someone wants me to fiddle around with the logic at the start, I can, but I'm not really a bash scripting wizard by any stretch of the imagination (I kind of learn as I go along). Possible changes would be to have resuming be the default action, having to supply a parameter to start from scratch, being able to specify more than one package, and being able to type 'resmerge PACKAGE resume' instead of 'resmerge resume PACKAGE'. However, at the moment, things are the way they are because I'm lazy, and this was the (relatively) easy way to go.

</EDIT>

<EDIT again>

I should note that these two scripts now use different files than they originally did. The first script uses PACKAGE.lst and PACKAGE.failed, and the second uses PACKAGE.lst, PACKAGE.tmp and PACKAGE.tmp.2. Thus, you shouldn't run them simultaneously on the same package (I'm not sure why you would do that, but...).

</EDIT>

Note: I haven't tested this one as extensively as the previous one (right now I'm remerging with the 'continue at all costs' script, which is easier to test, as well), because I don't really feel like letting my system run for 3 hours until grub dies and see if the package list gets updated correctly. If someone would like to test this out (or can spot a problem right away), I'm sure we'd all be greatful.

Anyway, I hope this is of some use to people, and that I haven't screwed up the code too badly.  :Smile: 

Cheers.

P.S.: How exactly do you use the list and list= tags? I can't figure out what the list= tags are for.

----------

## debian

WooWa! They are definitely useful! I always wondered if I'm the only one having problem with emerge resumption. I wrote my own shoddy script for the purpose but yours look much better!

I haven't tried it yet; anyway, thank you very much!  :Smile: 

----------

## J0rd

that sure would have been usefull like a week ago, =)

i ended up doing the whole thing manually, like 20 builds at a time.

im not sure exactly how that script works, but make sure it doesnt rebuild dependancies that have already been built with the 'world -pe' ones a package fails.

i would suggest ( once again im not sure how that script works ) a 'emerge -pe --nodeps world'

i dont even know if the script has this bug/feature, but i though i might want to give you a heads up.

----------

## Dolio

J0rd: I think I'm safe from that. Basically, what the scripts do is do an emerge -pe world, does various magic things to the output, and dumps all the package names to a file. Then, it goes through each package name in the file, and does "emerge --oneshot $package" which basically emerges each package by name, in order, without adding it to the world file. So dependencies shouldn't enter into it unless a: you're using the first script to emerge your system/a package from scratch (not a terribly good idea) or b: one of your already installed packages has a dependency that isn't merged, and it won't build (in which case, something is wrong with your system).

These should really only be used when you're rebuilding your system with all your dependencies already intact, and that's really the only time resuming is a problem (if you're emerging, and it cuts off, and stuff is missing, portage knows where you left off). However, I could be wrong, and if someone with more expertice out there knows otherwise, let me know and I'll correct the mistake.

debian: You're welcome.

all: Have a nice day.

----------

## debian

 *Dolio wrote:*   

> 'resmerge': starts a new 'emerge -pe world' FROM SCRATCH, EVEN IF ONE WAS IN PROGRESS
> 
> 

 

I recommend calling it 'rmerge' and submit it to gentoo.  :Smile: 

----------

## Dolio

Well, I thought about what debian said, and I think he/she/it is probably right about it being a useful portage package. However, I thought the script was a little rough around the edges to be submitted as it was, so I rewrote it to be a little nicer. This is not yet a finished work, as one part that I want to add (combining, I'll explain later) is not yet done. I just wanted to post this to get it out there, so people can maybe give it a go on a couple small merges to see if they can break it.

Note: I'd especially appreciate it if you have something in your system that breaks when emerging. If you don't mind, use this on it, and then see if the script resumes in the proper place (at the package that broke). As far as I know, this functionality has not been tested yet (because as far as I know, I don't have any package lines that break easily), and it's really the most important part.

So, here's how this script (rmerge, courtesy of debian) works (I'll have to write a usage for it too):

<EDIT>

Here's the 'final' code. I deleted the command line option explanation here, because the code now has a --help feature that explains everything, so you can read it there.

Okay, I think that's it. I'm sure you're all anxious, so here's my code.  :Wink: 

EDIT 7/31: I fixed a little bug in this code. There's two places where it occurs. In the remerging loop (if you have the older version) you'll see a 

```
sed -e "s/$pkg//g" ...
```

which deletes the name of the package just merged from the list of packages that need to be merged. However, if you have a package named "man" and a package named "man-pages" (as there actually is), the sed for "man" will leave "-pages", which is a problem. The fix is to change the above to

```
 sed -e "s/^$pkg\$//g"
```

as I have in the code below (Note again that it occurs twice in the script). This is only a significant bug if you acutally need to resume and packages you haven't merged yet have been mangled by this, but it's probably important, nonetheless.

```

#!/bin/bash

ROOT="/tmp/rmerge"

usage() {

   echo "Usage: rmerge [Options] [packages]"

   echo "       rmerge --help or rmerge -h for a detailed list of"

   echo "           options"

}

usage_long() {

   echo "Usage: rmerge [Options] [packages]"

   echo "Options:"

   echo

   echo "--scratch, -s:"

   echo "    Starts the rmerge from scratch, (i.e. doesn't use any"

   echo "    information about where the previous rmerge for the"

   echo "    specified package left off). This option is cancelled"

   echo "    by the --resume option, and vice versa. --resume is"

   echo "    the default."

   echo

   echo "--resume, -r:"

   echo "    Resumes the rmerge from where the package last left off."

   echo "    --resume cancels --scratch and vice versa. --resume is"

   echo "    the default."

   echo

   echo "--previous, -p:"

   echo "    Every time rmerge is run, the packages to be merged and"

   echo "    combination preference are saved to the following file:"

   echo

   echo "        ${ROOT}/prev"

   echo

   echo "    The --previous option loads this information and re-runs"

   echo "    the previous rmerge (note that --resume/--scratch is not"

   echo "    saved). By default, this is not enabled"

   echo

   echo "--combine, -c:"

   echo "    This option specifies that the packages should be merged"

   echo "    as a unit, similar to:"

   echo

   echo "        emerge -pe pkg1 pkg2 pkg3 pkg4 ..."

   echo

   echo "    this option cancels the --separate option, and vice versa."

   echo "    --combine is the default."

   echo

   echo "--separate, -S:"

   echo "    This option specifies that the two packages are to be"

   echo "    merged separately, so if the merging of one package fails"

   echo "    it will not keep the others from being attempted. For"

   echo "    all intents and purposes"

   echo

   echo "        rmerge [options] -S pkg1 pkg2 pkg3 pkg4 ..."

   echo

   echo "    is equivalent to"

   echo

   echo "        rmerge [options] pkg1"

   echo "        rmerge [options] pkg2"

   echo "        rmerge [options] pkg3"

   echo "        rmerge [options] pkg4"

   echo "        ."

   echo "        ."

   echo "        ."

   echo 

   echo "    Note: using this option, each package will be emerged"

   echo "    separately, so any common dependencies will be remerged"

   echo "    more than once. Use this with caution, as it will probably"

   echo "    take quite some time. This option cancels --combine."

   echo "    --combine is the default."

   echo

   echo "--no-color, -n:"

   echo "    Turns off color in error displays."

   echo

   echo "Packages can be any valid package name that you would feed to"

   echo "emerge. You may specify 0 or more packages to rmerge. If 0"

   echo "packages are specified, world is assumed, unless of course"

   echo "the --previous option is set. specifying one package should"

   echo "have predictable behavior, and see the --combine and --separate"

   echo "options for detailed behavior of specifying multiple packages."

}

SCRATCH=0

PREVIOUS=0

COMBINE=1

PKGNAMES=""

RED='\033[1;31m'

CLEAR='\033[0m'

for arg in $@

do

   if [ -z "${arg%-*}" ]; then

      arg=${arg:1}

      while [ -n "$arg" ]

      do

         case "${arg:0:1}" in

         s)

            SCRATCH=1

            ;;

         r)

            SCRATCH=0

            ;;

         p)

            if [ -z "$PKGNAMES" ]; then

               PREVIOUS=1

            fi

            ;;

         c)

            COMBINE=1

            ;;

         S)

            COMBINE=0

            ;;

         n)

            RED=''

            CLEAR=''

            ;;

         h)

            usage_long

            exit

            ;;

         *)

            echo "Unrecognised parameter: -${arg:0:1}"

            usage

            exit

            ;;

         esac

         arg="${arg:1}"

      done

   else

      case "$arg" in

      --scratch)

         SCRATCH=1

         ;;

      --resume)

         SCRATCH=0

         ;;

      --previous)

         if [ -z "$PKGNAMES" ]; then

            PREVIOUS=1

         fi

         ;;

      --combine)

         COMBINE=1

         ;;

      --separate)

         COMBINE=0

         ;;

      --no-color)

         RED=''

         CLEAR=''

         ;;

      --help)

         usage_long

         exit

         ;;

      *)

         if [ "${arg:0:2}" = "--" ]; then

            echo "Unrecognized parameter: ${arg}"

            usage

            exit

         fi

         

         PREVIOUS=0

         PKGNAMES="$PKGNAMES $arg"

         ;;

      esac

   fi

done

if [ -z "$PKGNAMES" ]; then

   PKGNAMES="world"

fi

if [ $PREVIOUS -eq 1 ]; then

   echo "Attempting to recover previous merge data..."

   if [ -f "${ROOT}/prev" ]; then

      eval `cat "${ROOT}/prev"`

      

      if [ -z "${PKGNAMES}" ]; then

         echo "Previous merge data appears to be invalid."

         exit 1

      fi

   else

      echo "File ${ROOT}/prev does not exist. No previous merge data."

      exit 1

   fi

   echo "Successful."

fi

echo "COMBINE=${COMBINE}" > "${ROOT}/prev"

echo "PKGNAMES='${PKGNAMES}'" >> "${ROOT}/prev"

if [ $COMBINE -eq 0 ]; then

   for package in $PKGNAMES

   do

      FNAME="${ROOT}/${package}.rmerge"

      

      DIE=0

      

      if [ $SCRATCH -eq 0 ] && [ -f "$FNAME" ]; then

         echo "Resuming merge of ${package}..."

      else

         echo "Beginning new merge for ${package}..."

         emerge -pe "${package}" | egrep 'ebuild' | sed -e 's:\(.*\) \(.*\)/\(.*\)-\([0-9].*\) \(.*\) \(.*\):\3:' > "${FNAME}"

         # A test to weed out bad packages

         emerge -pe "${package}" >/dev/null || DIE=1

         if [ $DIE -eq 1 ]; then

            rm -f $FNAME

            echo

            echo -e "${RED}FAILED TO: emerge -pe ${package}"

            echo -e "           Perhaps a bad package name. Check command line.${CLEAR}"

            echo

            continue

         fi

      fi

      

      FAILED=0

         

      cp -f "${FNAME}" "${FNAME}.tmp"

      

      for pkg in `cat ${FNAME}`

      do

         emerge --oneshot "${pkg}" || FAILED=1

         if [ $FAILED -ne 1 ]; then

            sed -e "s/^$pkg\$//g" "${FNAME}.tmp" > "${FNAME}.tmp.2"

            mv -f "${FNAME}.tmp.2" "${FNAME}.tmp"

         else

            egrep "^.+$" "${FNAME}.tmp" > "${FNAME}"

            break

         fi

      done

      

      if [ $FAILED -eq 1 ]; then

         echo "Emerge of ${package} failed at ${pkg}"

      else

         echo "Successfully remerged ${package}."

         rm -f "${FNAME}"

         rm -f "${FNAME}.tmp"

      fi

   done

else

   PKGCOM=`echo ${PKGNAMES} | sed -e "s/ /-/g"`

   FNAME="${ROOT}/${PKGCOM}.rmerge"

   

   DIE=0

   

   if [ $SCRATCH -eq 0 ] && [ -f "$FNAME" ]; then

      echo "Resuming combined merge..."

   else

      echo "Beginning new combined merge..."

      emerge -pe ${PKGNAMES} | egrep 'ebuild' | sed -e 's:\(.*\) \(.*\)/\(.*\)-\([0-9].*\) \(.*\) \(.*\):\3:' > "${FNAME}"

      # A test to make sure we don't accept bad packages

      emerge -pe ${PKGNAMES} 2>&1 >/dev/null || DIE=1

      if [ $DIE -eq 1 ]; then

         rm -f $FNAME

         echo

         echo -e "${RED}FAILED TO: emerge -pe ${PKGNAMES}"

         echo -e "           Perhaps a bad package name. Recheck command line.${CLEAR}"

         echo

         exit

      fi

   fi

   

   FAILED=0

         

   cp -f "${FNAME}" "${FNAME}.tmp"

   

   for pkg in `cat ${FNAME}`

   do

      emerge --oneshot "${pkg}" || FAILED=1

      if [ $FAILED -ne 1 ]; then

         sed -e "s/^$pkg\$//g" "${FNAME}.tmp" > "${FNAME}.tmp.2"

         mv -f "${FNAME}.tmp.2" "${FNAME}.tmp"

      else

         egrep "^.+$" "${FNAME}.tmp" > "${FNAME}"

         break

      fi

   done

   

   if [ $FAILED -eq 1 ]; then

      echo "Emerge failed at ${pkg}"

   else

      echo "Successfully remerged packages ${PKGNAMES}."

      rm -f "${FNAME}"

      rm -f "${FNAME}.tmp"

   fi

fi

```

Some of the code is kind of redundant (lots of copy-paste between the --separate and --combine cases), but I haven't really thought about modularizing it yet.  I guess I'll look into submitting this somehow now (maybe then it will get some testing). I guess I might have to write an ebuild now  :Smile: .

Note that the default is now --combine, unlike what I just replaced, where --separate was the default.

</EDIT>

Again, thanks to fghellar, who contributed the much nicer looking sed command for extracting package names, and who also pointed me to the bash articles in the articles section of this site that basically allowed me to write this new, fancy script.

If you see any/find any errors with this script, please post them so I can make corrections. Once I add the final touches, I'll see about submitting it as debian suggested.

Enjoy.

----------

## Dolio

So I whipped up what I think is a satisfactory ebuild for my little script and submitted it for inclusion in the portage tree. The bug id is 5667, in case you want to track its progress.

----------

## Dolio

I figured I should tell you all about this, because it might cause some confusion if you use this script.

I was rewriting the script to be a lot nicer (I think it's done now), as well as testing some of the existing things, and I came across a small problem. It goes like this. When you do emerge -pe world, emerge tells you what packages you need, as well as what versions. However, in some cases, there are newer versions of the packages you need, and when my script strips the package down to the name, that newest version will get installed, instead of the one you really need (which also tends to install other packages you don't want). The only real offender I've noticed with this is Gnome 2, which installs a bunch of bonobo stuff I don't want, even though I have -bonobo and don't acutally have anything that requires gnome 2.

So, there are currently two solutions to this problem. 1: you can use the script as is and remove the unwanted packages afterwards. See this post https://forums.gentoo.org/viewtopic.php?t=6755 for something that should help with that. 2: I could post the new, huge script here for you to use. It has options for solving this problem as an option (the solution is pseudo-dangerous, so it's not enabled by default). I'm trying to get the script added to portage, but it's kind of taking a while (I haven't really seen much activity), and the version I initially submitted didn't have the extra options, so I'm not sure if I'd be able to get the new script in there very fast.

Anyway, take your pick.

P.S.: Note the bug fix I made in the above script. It's sort of important if you want to keep using this version of the script.

----------

## sven

None of these scripts work for me. I've also tried the newest version from the Bug Report. It seems like the sed command does not edit the stream at all! I've remerged sed but it did not help. Any idea?

----------

## Yama

 *sven wrote:*   

> None of these scripts work for me. I've also tried the newest version from the Bug Report. It seems like the sed command does not edit the stream at all! I've remerged sed but it did not help. Any idea?

 I've got the same problem. Here's what I get:

```
# ./rmerge2 -f -s -e -p world

I would merge the following packages (in order):

#
```

If I leave out the -p, it does nothing. I'm using rmerge2-0.9.6.tar.gz, and I have created a /var/lib/rmerge2 directory as outlined in the bug page.

----------

## DArtagnan

It does not work at all  :Smile: 

```

roth-supp voicu # rmerge2 -p -e world

I would merge the following packages (in order):

roth-supp voicu #

```

Any clue?

----------

## BlackBart

a feature sugestion:

i'm not sure how hard this would be to implement but:

to save all the outputs into a file(just the little message when it's done installing) so they can be looked at later

----------

## Yama

I emerged app-admin/rmerge2-0.9.7 from portage and now everything works okay. Thanks.

----------

## DArtagnan

 *Yama wrote:*   

> I emerged app-admin/rmerge2-0.9.7 from portage and now everything works okay. Thanks.

 

Here this seems to be masked  :Sad: 

----------

## BlackBart

where is the file located that lists failed builds and builds packages that it will build? I can't seem to locate it.

----------

## pilla

Through output redirection:

```

command >& output.txt 

```

 *BlackBart wrote:*   

> a feature sugestion:
> 
> i'm not sure how hard this would be to implement but:
> 
> to save all the outputs into a file(just the little message when it's done installing) so they can be looked at later

 

----------

## BlackBart

wouldn't that dump the whole outputs of all the compiles to output.txt and not let me see it onscreen?

also, whenever i try to run it it complains that my use flags are not set in make.conf, forcing me to run "export CFLAGS="-march=pentium -O3...", is there away around this?

----------

## iancurtis

 *BlackBart wrote:*   

> wouldn't that dump the whole outputs of all the compiles to output.txt and not let me see it onscreen?
> 
> also, whenever i try to run it it complains that my use flags are not set in make.conf, forcing me to run "export CFLAGS="-march=pentium -O3...", is there away around this?

 

I have the same problem.

----------

## hielvc

Yes, you have to 

```

export CFLAGS="your_flags_here "

export CXXFLAGS="${CFLAGS}"    (note use braces, look at make.conf)    

```

hiel

----------

## ozt

emerge -u --deep world works fine for me. Starts to emerge where I cancelled it yesterday  :Razz: .

----------

## BlackBart

 *hielvc wrote:*   

> Yes, you have to 
> 
> ```
> 
> export CFLAGS="your_flags_here "
> ...

 

or

export CXXFLAGS=$CFLAGS

without braces or quotes

----------

