# Howto compile packages with PGO

## xming

Profile guided optimization can sometimes yield better performance, I am going to explain howto to use the portage system to do that without hacking portage itself, so PGO maybe become more popular.

1. mkdir /var/tmp/pgo

2. put the following in /etc/portage/env/cflags.pgo

```
PROFILE_DIR=/var/tmp/pgo/${PN}

CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-arcs -fvpt"

CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-correction"

LDFLAGS_PROFILE_GEN="-fprofile-arcs"

CFLAGS_CUSTOM=""

LDFLAGS_CUSTOM=""

if [ -d ${PROFILE_DIR} ]

then

  CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"

  LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"

else

  CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"

  LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"

fi

CXXFLAGS="${CFLAGS}"

```

3. for packages which you want to compile with pgo, you have to use cflags.pgo, see here for instructions.

4. emerge <pkgs>

5. you need to chown/chmod /var/tmp/pgo/${PN} so that your normal user can write to the files/dirs

6. use the pkgs as you normally would

7. emerge <pkgs> (again)

8. now you have PGO builds

Caveats:

- Well you will have to compile twice (for the first time) and use the application in between compiles, after that it will use the existing profile stats.

- Step 5, sometimes during the compilation, some exec are run (conftest, ...) so there are files generated in /var/tmp/pgo/${PN}, you will have to manually delete those or fix the permissions.

- if you want to regenerate the profile stats then you need to manually delete /var/tmp/pgo/${PN} and start over

It's not fully automatic method, but this is as far as I could get to reduce the manual steps, any suggestions are welcome.

----------

## avx

 *Quote:*   

> use the application in between compiles

 How? Trigger any possible action in the application, only what is used mostly?

----------

## xming

 *avx wrote:*   

>  *Quote:*   use the application in between compiles How? Trigger any possible action in the application, only what is used mostly?

 

I would say use it normally.

----------

## i13m

Hi, I followed this guide, but get an error

```

!!! Problem in 'app-emulation/qemu-kvm' dependencies.

!!! "/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=') portage.exception

... done!

"/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=')

```

As far as I know, /etc/portage/env/cflags.pgo cannot do bash expression, or am I wrong

----------

## xming

I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .

And I am using portage-2.2.0_alpha* .

I don't know which of the two would allow bash expressions (I am guessing the second), but it certainly works here.

----------

## bergelin

 *avx wrote:*   

>  *Quote:*   use the application in between compiles How? Trigger any possible action in the application, only what is used mostly?

 

Wikipedia says

 *Quote:*   

> The caveat, however, is that the sample of data fed to the program during the profiling stage must be statistically representative of the typical usage scenarios; otherwise, profile-guided feedback has the potential to harm the overall performance of the final build instead of improve it.

 

Found a lecture transcript (p. 34 and forwards) with some more info.

I'm gonna give this a try on a few applications. Ran into the same problem i13m did, but solved it by using symlinks as mentioned above, after which everything compiled fine. 

It would be interesting to see benchmarks, or any other information, as to the possible performance benefits in some use cases, so one can determine which (if any) applications are worth the while.

----------

## papu

 *xming wrote:*   

> I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .
> 
> .

 

i don't undestand this,  etc/portage/env/ pkg???, i don't have pkg here... write a real example please.

i have same error 

```
sudo USE="pgo" emerge -a firefox

These are the packages that would be merged, in order:

Calculating dependencies      y

!!! Problem in 'www-client/firefox' dependencies.

!!! "/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=') portage.exception

... done!

"/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=')
```

thanks  :Smile: 

----------

## xming

like 

```
ln -s /etc/portage/env/cflags.pgo /etc/portage/env/media-libs/mesa

```

Or even better (relative instead of absolute)

```
mkdir /etc/portage/env/media-lib && cd /etc/portage/env/media-lib && ln -s ../cflags.pgo mesa

```

----------

## mv

If you use portage-bashrc-mv (available e.g. from the mv overlay), you can simply put into /etc/portage/package.cflags (or, if you prefer, into a file in there):

 */etc/portage/package.cflags/pgo wrote:*   

> category/package PGO=true

 

(if you have installed eix then instead of category/package you can also specify versions/version ranges/slots/overlays as usual in /etc/portage/package.* files).

Then after the first emerge of the correponding packages run them and then re-emerge them a second time. (The mechanism used is quite similar than in this thread and is described in more detail in the README).

You need >=portage-bashrc-mv-12.0 for the correct name PGO (in earlier versions, this was the misnomer GPO which I realized only after reading this thread...).

----------

## ryszardzonk

I just read that using PGO for python should yield extra 10% perfomance boost - http://komodoide.com/blog/2014-06/python-pgo-on-linux/ That having said the only way I found searching for using PGO in Gentoo other than some packages having use flag for it is to use this script. I did have to change location to the PROFILE_DIR=/tmp/portage/${P} as with original one I had some write permissions problems and from ${PN} to ${P} IMHO should be more appropriate as packages like python have more than one version that may be present in the system at the time

Now to questions

- what to do so I would not have to emerge package twice as first "emerge python" emerged package with just profile generated and only second run used generated profile

-march-native -O3 -fprofile-generate

dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 108627,379 KiB

-march-native -O3 -fprofile-use

dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 96819,41 KiB

BTW If you wanted to enable lto on python with GCC 5.3 think again  :Wink:  It balooned in size for me

-march-native -O3 -flto=4 -ffat-lto-objects -fprofile-generate

dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 132411,422 KiB

-march-native -O3 -flto=4 -ffat-lto-objects -fprofile-use

dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 111267,736 KiB

- After having it done in just one run it would be nice for portage to delete that profile as it should not be needed any more right?

----------

## steveL

That is a lovely post, xming.

 *ryszardzonk wrote:*   

> After having it done in just one run it would be nice for portage to delete that profile as it should not be needed any more right?

 

Just DIY, at step 8; script the process if you prefer.

And thanks for the thread necromancy ;) I'd never have seen this topic otherwise.

----------

## ryszardzonk

I would love to write such a script to cover that but it s over my head stuff. Anyways I believe that by now there should be some kind of eclass that one could add to their ebuild to get PGO enabled for ebuilds. Emerging stuff twice is not what tigers like most  :Wink: 

----------

## trippels

For python you can simply edit the ebuild and use the built-in profile-opt target:

```

diff --git a/dev-lang/python/python-2.7.11-r2.ebuild b/dev-lang/python/python-2.7.11-r2.ebuild

index 5e7b2b6..4e708de 100644

--- a/dev-lang/python/python-2.7.11-r2.ebuild

+++ b/dev-lang/python/python-2.7.11-r2.ebuild

@@ -217,7 +217,7 @@ src_compile() {

        touch Include/graminit.h Python/graminit.c

 

        cd "${BUILD_DIR}" || die

-       emake

+       emake profile-opt

 

        # Work around bug 329499. See also bug 413751 and 457194.

        if has_version dev-libs/libffi[pax_kernel]; then

diff --git a/dev-lang/python/python-3.4.3-r7.ebuild b/dev-lang/python/python-3.4.3-r7.ebuild

index e281de4..54c7e37 100644

--- a/dev-lang/python/python-3.4.3-r7.ebuild

+++ b/dev-lang/python/python-3.4.3-r7.ebuild

@@ -176,7 +176,7 @@ src_compile() {

 

        cd "${BUILD_DIR}" || die

 

-       emake CPPFLAGS= CFLAGS= LDFLAGS=

+       emake profile-opt 

 

        # Work around bug 329499. See also bug 413751 and 457194.

        if has_version dev-libs/libffi[pax_kernel]; then

```

----------

## steveL

 *ryszardzonk wrote:*   

> Emerging stuff twice is not what tigers like most ;)

 

Yeah, but you have to build at least twice to profile at all.

Given that, it's an admin-level decision when to profile, since it is best done on typical usage scenarios, and it implies a lack of performance during profiling. Both require human input.

----------

## steveL

 *trippels wrote:*   

> For python you can simply edit the ebuild and use the built-in profile-opt target

 

Hmm nice patch; nicer USE-flag dependent? ie: 

```
if use profile; then ..
```

(already a global USE) which does throw up the option of making it into a more generic thing, at the FEATURE level perhaps.

If someone's already doing that, please share.

----------

## ryszardzonk

 *steveL wrote:*   

>  *ryszardzonk wrote:*   Emerging stuff twice is not what tigers like most  
> 
> Yeah, but you have to build at least twice to profile at all.

 

Yeah I know that. But emerging twice takes even longer than just compiling twice hence the idea of having more generic approach to the build process. I wish one of the devs would add it to the src_compile routines adding lets say "pgo_src_compile" section in one or the other eclass so it would be easier and faster to use for users and package maintainers to use.

----------

## steveL

 *ryszardzonk wrote:*   

> Emerging stuff twice is not what tigers like most ;)

 

 *steveL wrote:*   

> Yeah, but you have to build at least twice to profile at all.

 

 *ryszardzonk wrote:*   

> Yeah I know that. But emerging twice takes even longer than just compiling twice

 

No, it does not. Manually compiling software is much more tedious, and far less useful, than using an ebuild.

 *Quote:*   

> hence the idea of having more generic approach to the build process. I wish one of the devs would add it to the src_compile routines adding lets say "pgo_src_compile" section in one or the other eclass so it would be easier and faster to use for users and package maintainers to use.

 

I'm fine with a more generic approach, though I think that's a bad notion of one.

The thing you cannot get away from is that there MUST be a time lag between the builds, for the profile data to be generated, in order to be used in the subsequent production build.

As such it means a next ebuild run, and nothing should be changed at the ebuild level, imo.

I'd knock up a script in bash, using eix and portage-utils, to begin with.

#bash are your friends.

----------

## ryszardzonk

 *ryszardzonk wrote:*   

> Yeah I know that. But emerging twice takes even longer than just compiling twice

 

 *steveL wrote:*   

> No, it does not. Manually compiling software is much more tedious, and far less useful, than using an ebuild.

 

When I said about compiling twice it would be faster I meant that when using ebuild to do

- unpack & prepare & pgo_compile (2xcompile) & install & qmerge will be faster than 2x ( unpack & prepare & compile & install & qmerge).

 *steveL wrote:*   

> The thing you cannot get away from is that there MUST be a time lag between the builds, for the profile data to be generated, in order to be used in the subsequent production build.

 What lag? If first compile finishes, than second may start. Otherwise those ebuilds in portage like firefox would not have pgo flag.

----------

## Petross404

In my Gentoo I make use of the following trick in /etc/portage/bashrc : 

```
#!/bin/bash

PROFILE_DIR=/var/tmp/pgo/${PN} 

CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -fprofile-arcs -fvpt" 

CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -fprofile-correction" 

LDFLAGS_PROFILE_GEN="-fprofile-arcs" 

CFLAGS_CUSTOM="" 

LDFLAGS_CUSTOM=""

post_src_unpack() {

   #EAPIs < 6 don't provide epatch_user if needed ;)

   if type epatch_user >& /dev/null; then

      cd "${S}"

      epatch_user

   fi

   if [ -d ${PROFILE_DIR} ] 

   then 

        CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}" 

      LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}" 

   else

        CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}" 

      LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}" 

   fi

   CXXFLAGS="${CFLAGS}"

}

....

#This way the profile dir will be writable for programms that run with my user priviliges

pkg_postinst() {

   chown -R petros:petros ${PROFILE_DIR}

}
```

I enable profiling for all packages. If a package failes, I think then  /etc/portage/env/${PN} will come in handy.

----------

## Yamakuzure

 *mv wrote:*   

> If you use portage-bashrc-mv (available e.g. from the mv overlay), you can simply put into /etc/portage/package.cflags (or, if you prefer, into a file in there):
> 
>  */etc/portage/package.cflags/pgo wrote:*   category/package PGO=true 
> 
> (if you have installed eix then instead of category/package you can also specify versions/version ranges/slots/overlays as usual in /etc/portage/package.* files).
> ...

 Actually this is doing the trick already. No need to write something new.  :Wink: 

```
 ~ # cat /etc/portage/package.cflags/IDEs

dev-util/codeblocks PGO=true
```

I am currently testing this...

----------

## steveL

 *ryszardzonk wrote:*   

> When I said about compiling twice it would be faster I meant that when using ebuild to do
> 
> - unpack & prepare & pgo_compile (2xcompile) & install & qmerge will be faster than 2x ( unpack & prepare & compile & install & qmerge).
> 
>  *steveL wrote:*   The thing you cannot get away from is that there MUST be a time lag between the builds, for the profile data to be generated, in order to be used in the subsequent production build. What lag? If first compile finishes, than second may start. Otherwise those ebuilds in portage like firefox would not have pgo flag.

  Whiile what you proposed above looks like it would work, you've missed my point: the profile data must be generated in usage, before the second profile-guided compilation can take place at all. Otherwise there's no profiling data for it to work with.

The first build generates profiling data during usage (making it slower, as mentioned previously); that is used during the second build, to guide how it optimises eg by deciding branches are unlikely or likely.

So there must always be two separate builds, with a time lag between them, each with specific flags: the first to generate the data, the second to use it (for the final "production" build.)

As such, any tweaking must happen at a higher-level than ebuild, and there is no benefit to changing anything at the ebuild level.

It seems obvious to me that it must require human input, as well, though I could envisage automation at package mangler level. (It still would not change anything at ebuild level, mind: there would still need to be that time lag between builds, to gather usage data.)

----------

## Yamakuzure

 *ryszardzonk wrote:*   

> If first compile finishes, than second may start. Otherwise those ebuilds in portage like firefox would not have pgo flag.

 That is something completely different. Packages with "pgo" USE flag have means to do (some) profiling during and/or after the first build.

Just take dev-libs/gmp as an example:

```
multilib_src_compile() {

        emake

        if use pgo ; then

                emake -j1 -C tune tuneup

                ebegin "Trying to generate tuned data"

                ./tune/tuneup | tee gmp.mparam.h.new

                if eend $(( 0 + ${PIPESTATUS[*]/#/+} )) ; then

                        mv gmp.mparam.h.new gmp-mparam.h || die

                        emake clean

                        emake

                fi

        fi

}
```

See?

That is some sort of "pgo", but not what we discussed in this thread.

sys-devel/gcc for example can generate some data by itself during bootstrapping, that's what

```
GCC_MAKE_TARGET="profiledbootstrap"
```

is for.

And firefox has special PGO targets, that actually build firefox and simulate usage using a virtual X session.

----------

## Petross404

I 've edited my bashrc : 

```
#!/bin/bash

PGO_DIR=/var/tmp/pgo

PROFILE_DIR=${PGO_DIR}/${PN}

PROFILABLE=0

CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -fprofile-arcs -fvpt" 

CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -fprofile-correction" 

LDFLAGS_PROFILE_GEN="-fprofile-arcs" 

CFLAGS_CUSTOM="" 

LDFLAGS_CUSTOM=""

post_src_unpack() {

   if type epatch_user >& /dev/null; then

      cd "${S}"

      epatch_user

   fi

   case "${CATEGORY}" in

      *app-editor* | *dev-*  | *kde-* | *mail-client* | *media-video* | *www-client* )

      

      echo "1" > /tmp/profile

      elog "Will try to profile" "${PN}"

      if [ -d ${PROFILE_DIR} ] 

      then 

         elog "Running with PGO for the first time"

           CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}" 

         LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}" 

      else

          CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}" 

         LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}" 

      fi

      CXXFLAGS="${CFLAGS}"

      

   esac

}

pre_src_prepare() {

    [[ ${EAPI:-0} == [012345] ]] || return

    if ! type estack_push > /dev/null 2>&1; then

        local estack_names="eshopts_push eshopts_pop evar_push evar_push_set evar_pop estack_push estack_pop"

        source <(awk "/^# @(FUNCTION|VARIABLE): / { p = 0 } /^# @(FUNCTION|VARIABLE): (${estack_names// /|})\$/ { p = 1 } p { print }" ${PORTDIR}/eclass/estack.eclass)

    fi

    if ! type epatch_user > /dev/null 2>&1; then

        local epatch_names="EPATCH_SOURCE EPATCH_USER_SOURCE epatch_user_death_notice epatch_user epatch"

        source <(awk "/^# @(FUNCTION|VARIABLE): / { p = 0 } /^# @(FUNCTION|VARIABLE): (${epatch_names// /|})\$/ { p = 1 } p { print }" ${PORTDIR}/eclass/epatch.eclass)

    fi

    epatch_user

    for name in $epatch_names; do

        unset $name

    done

    for name in $estack_names; do

        unset $name

    done

}

pkg_postinst() {

   #if PGO was applied, test if it succeded and chown the PROFILE_DIR

   read PROFILABLE < /tmp/profile && rm /tmp/profile

   if [[ "$PROFILABLE" -eq 1 ]]  ; then

      for i in `equery f ${PN} | grep "bin/"`;

      do

         if [[ -z `strings $i | grep profiling` ]]; then

            ewarn "This binary doesn't make use of PGO:" $i

         else

            elog  "This binary does make use of PGO:" $i

         fi

      done

   

   chown -R petros:petros "${PGO_DIR}"

   fi

}
```

It can print to which packages PGO has been applied. Can someone propose another way to read the categories instead of hardcoding them? For example maybe sourcing a file under /etc/portage/env?

----------

## Yamakuzure

 *Petross404 wrote:*   

> Can someone propose another way to read the categories instead of hardcoding them? For example maybe sourcing a file under /etc/portage/env?

 Just do not re-invent the wheel, and use, what is already there: *mv wrote:*   

> If you use portage-bashrc-mv (available e.g. from the mv overlay), you can simply put into /etc/portage/package.cflags (or, if you prefer, into a file in there):
> 
>  */etc/portage/package.cflags/pgo wrote:*   category/package PGO=true 
> 
> (if you have installed eix then instead of category/package you can also specify versions/version ranges/slots/overlays as usual in /etc/portage/package.* files).
> ...

 I have tested it, and it works like a charm! However, Live-Packages are tricky.  :Wink: 

----------

## Rutcha

Although, I'm thankful for your tips

I'm afraid I can't get {PN} variable or any other ebuild variable related to package name to work. They simply get ignored, it's like they never exist.

Do you have any undocumented work around ?

----------

## Hu

For future readers, Rutcha also posted following instructions no longer works and 'Ebuild variable' vanishes during normal portage emerge about this same subject.  In the free thread, it is made clear that the variables are being used in make.conf, where their usage is invalid and has never worked.  The instructions here did not rot over time.

----------

