# Per-package CFLAGS seem already to be part of Portage!

## Guenther Brunthaler

Hi all,

While searching for a solution how to define per-package CFLAGS and sifting through various postings on that issue, I noticed that a simple solution to the problem seems already to be built into Portage.

And it works right out of the box, without adding any custom scripts to /etc/portage/bashrc!

Well, at least it does in the 2006.1 profile, because the feature is actually a part of the profile, not of Portage itself.

But there are good chances it might work for older profiles as well, because the feature is part of the base profile, which all other profiles are based upon.

I found the feature when examining the files in the various profile directories, where I encountered the file

/usr/portage/profiles/base/profile.bashrc

which obviously sources script files from

/etc/portage/env/category/package

For instance, when I added a file /etc/portage/env/app-text/unix2dos

containing the line

```
CFLAGS="-Werror"
```

then this flag was obviously passed to the compiler during the ebuild's compile phase, because then the compiler bailed out with an error. (This was intended for testing: -Werror makes all warnings errors, so I could notice the difference whether or not my CFLAGS have been taken into account. And it seems they were.)

So, if you want to use different CFLAG/CXXFLAGS/FEATURES (or other variables set in /etc/make.conf) for a specific package, create a text file

/etc/portage/env/category/package

and add your overrides in exactly the same way you would otherwise do in the command line manually.

For instance, instead of doing a

```
CFLAGS="-O3" emerge --oneshot gcc
```

you could instead create a file /etc/portage/env/sys-devel/gcc

containing the line

```
CFLAGS="-O3"
```

and then just use

```
emerge --oneshot gcc
```

in order to achieve the same effect.

The next I will try to do is creating symlinks instead of real files in /etc/portage/env.

This should allow me to share a single file with non-standard stronger optimizations settings in the /etc/portage/env subdirectories without a need to copy the same file into several subdirectories.

It will also be easier to modify the settings if they are stored only in a single place.

I will thus create only a single physical settings file for each group of related options (one file for stronger optimization, another one for buggy packages to be compiled with FEATURES="nostrip", etc), and refer to them only via symlinks from within /etc/portage/env.

Has anyone else already found and used that obviously undocumented system profile feature?

I would really like to learn of any experiences, potential drawbacks, and generally the opinons of others before I make heavy use of this /etc/portage/env feature in my live system!

Anyway, it looks quite promising so far.

What I still have to find out is whether the per-package variables set that way are additive, or whether they replace the default values from the current profile or /etc/make.conf.

UPDATE: I have now written a minimalist ebuild for testing the behaviour when specifying flags in /etc/portage/env.

The result is: The settings in the /etc/portage/env files completely replace all settings from the matching variables in /etc/make.conf.

However, at the time when the /etc/portage/env files are called (are sourced as shell scripts), the /etc/make.conf settings are already available in the respective environment variables.

Which means you can use code like

```
CFLAGS="$CFLAGS -O3"

CXXFLAGS="$CXXFLAGS -O3"

```

in order to add options to the end of the /etc/make.conf options. (As far as I know, later options can override previous mutually-exclusive options in the GCC command line. Thus "-O2 -O3" should be the same as "-O3" alone.)

Alternatively, all the power of the shell's scripting language can be unleashed to filter out / replace unwanted flags from the original /etc/make.conf settings.

Or, just choke on the /etc/make.conf settings and simply specify all the CFLAGS/CXXFLAGS you want to use, completely replacing the /etc/make.conf settings.

----------

## NeddySeagoon

Guenther Brunthaler,

According to the guys in #gentoo-portage this is equivelent to modifying your /etc/make.conf with one exception.

When something breaks, the actual CFLAGS used will not be reported in emerge --info

Please do not report bugs that occured while this feature is in use for the failed package.

Clear it first and rebuild with your standard /etc/make.conf CFLAGS.

----------

## truc

yeah this feature is realy handy , especially when it comes to the EXTRA_ECONF variable available for some ebuilds. Really really handy, I think you said enough about it, but, it's not a well known feature (probably due to what NeddySeagoon said about emerge --info), and it's a shame ! ;=

----------

## Guenther Brunthaler

 *NeddySeagoon wrote:*   

> According to the guys in #gentoo-portage this is equivelent to modifying your /etc/make.conf

 

Thanks! That's exactly that kind of feedback I have been waiting for.

 *NeddySeagoon wrote:*   

> do not report bugs that occured while this feature is in use for the failed package.
> 
> Clear it first and rebuild with your standard /etc/make.conf CFLAGS.

 

Thanks also for pointing out this important fact.

----------

## sliwowitz

Is there a way to use this feature for specifying CFLAGS for larger groups of packages? (like setting -O3 for x11-base/*)

----------

## Guenther Brunthaler

 *sliwowitz wrote:*   

> Is there a way to use this feature for specifying CFLAGS for larger groups of packages? (like setting -O3 for x11-base/*)

 

Well, not automatically, I'm afraid.

But of course it is possible to write a script to aid in such cases.

I wrote a simple one named "set-optimization-level":

```
#! /usr/bin/perl -w

# Adds a list of package descriptions in the format "category/package"

# to the list of heavily optimized packages by creating symlinks

# in /etc/portage/env.

#

# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/set-optimization-level $

# $Author: root $

# $Date: 2006-10-15T11:45:12.625789Z $

# $Revision: 448 $

use strict;

my $outdir= '/etc/portage/env';

my $target= '../heavily-optimized';

my($d);

foreach (@ARGV ? @ARGV : <>) {

   s/^\s*|\s*$//g;

   substr($_, 0, 0)= "$outdir/";

   die unless ($d)= m{^(.+)/[^/]+$};

   mkdir $d or die "Cannot mkdir '$d': $!" unless -d $d;

   if (-l) {

      next if readlink eq $target;

      unlink($_) == 1 || die "Cannot remove old '$_': $!";

   } elsif (-e) {

      # Note: '-l' also works for symlinks where '-e' will fail!

      die "'$_' already exists and is not a symlink";

   }

   symlink $target, $_ or die "Cannot symlink '$_': $!";

   print "symlinked '$_'.\n";

}

```

In order to use this script, you have to

Create a directory "/etc/portage/env".

Create a script "heavily-optimized" within that directory. That script should set up the CFLAGS etc as you want them for heavily optimized packages.

Run my script for each "category/package"-name you want to be optimized. The script will automatically create the required symlinks. It also recognizes whether a symlink has already been set up and will not complain if you later specify a category/package accidentally again.

The category/package specifications should not contain a version specification; just the generic category and package name, such as "x11base/xorg-x11".

The script also takes multiple arguments, which means it can be used with xargs.

If no arguments are specified, the script expects a categoy/package list on standard input, one package specification per line. Which means it's easy to redirect something like /var/lib/portage/world as input to my script.

----------

## Guenther Brunthaler

Here is my version of the "heavily-optimized" script, and it's counter-part script "debugging" which sets special CFLAGS for debug-only versions of packages.

There is also a script "default" which simply does nothing. I use it to redirect a symlink to a neutral script, if I do not want special optimization for a package. As my script will never re-create or change an existing symlink, it will not overwrite such a symlink with one pointing to "heavily optimized" if I accidentally specify the package as an argument to my script later.

Script "/etc/portage/env/heavily-optimized":

```
#! /bin/false

#

# Symlink target for "/etc/portage/env/<category>/<package>".

# Changes compiler options for heavy optimization,

# unless the "debug" USE-flag is set for the package, in which case

# the debugging options will be set instead.

# Whether space-separated string $1 contains token $2.

contains_ncm9q70alywv346gx4ug8hwrq() {

   test "${1#$2}" != "$1" && return

   test "${1%$2}" != "$1" && return

   test "${1% $2 *}" != "$1" && return

   return 1

}

# Check for debugging override.

if contains_ncm9q70alywv346gx4ug8hwrq "$USE" "debug" \

   || contains_ncm9q70alywv346gx4ug8hwrq "$FEATURES" "noclean"

then

   unset -f contains_ncm9q70alywv346gx4ug8hwrq

   . /etc/portage/env/debugging

else

   unset -f contains_ncm9q70alywv346gx4ug8hwrq

   optimizer_filter_ncm9q70alywv346gx4ug8hwrq() {

      local S1 S2 OLD NEW T REST M SET

      S1=1; S2=1

      while true; do

         case $S1 in

            1) # Start variable modification.

               case $S2 in

                  1) OLD="$CFLAGS";;

                  2) OLD="$CXXFLAGS";;

                  3) OLD="$FEATURES";;

                  4) OLD="$USE";;

               esac

               SET=

               case $S2 in

                  1 | 2) SET="-O3 -DNDEBUG -fomit-frame-pointer -fno-stack-check";;

               esac

               OLD="$(echo "$OLD" | tr -s '[[:space:]]' ' ' | sed -e 's/^ *//; s/ *$//')"

               NEW=

               S3=1

               S1=2

               ;;

            2) # Modify next token if necessary.

               if [ -z "$OLD" ]; then

                  S1=3

                  test -n "$SET" && NEW="$NEW${NEW:+ }$SET"

                  continue

               fi

               REST="${OLD#* }"

               if [ "$OLD" = "$REST" ]; then

                  T="$OLD"; OLD=

               else

                  T="${OLD%$REST}"; T="${T% }"; OLD="$REST"

               fi

               M=

               case $S2 in

                  1 | 2)

                     case "$T" in

                        -O[0-9]) M=1;;

                        -g*) M=1;;

                        -DNDEBUG | -DDEBUG | -D_DEBUG) M=1;;

                        -fomit-frame-pointer | -fno-omit-frame-pointer) M=1;;

                        -fstack-check | -fno-stack-check) M=1;;

                        -fno-eliminate-unused-debug-symbols | -feliminate-unused-debug-symbols) M=1;;

                     esac

                     ;;

                  3)

                     case "$T" in

                        nostrip | keeptemp | keepwork | noclean) M=1;;

                     esac

                     ;;

                  4)

                     case "$T" in

                        debug) M=1;;

                     esac

                     ;;

               esac

               if [ -n "$M" ]; then

                  # $T matches a debugging-relevant token.

                  test -z "$SET" && continue

                  T="$SET"; SET=

               fi

               NEW="$NEW${NEW:+ }$T"

               ;;

            3) # Complete variable modification.

               case $S2 in

                  1) CFLAGS="$NEW"; S2=2;;

                  2) CXXFLAGS="$NEW"; S2=3;;

                  3) FEATURES="$NEW"; S2=4;;

                  4) USE="$NEW"; return;;

               esac

               S1=1

               ;;

         esac

      done

   }

   optimizer_filter_ncm9q70alywv346gx4ug8hwrq

   unset -f optimizer_filter_ncm9q70alywv346gx4ug8hwrq

   echo "OVERRIDING FLAGS: heavily-optimized"

   #echo "CFLAGS=$CFLAGS"; echo "CXXFLAGS=$CXXFLAGS"; echo "FEATURES=$FEATURES"; echo "USE=$USE"

fi

```

Script "/etc/portage/env/debugging":

```
#! /bin/false

#

# Symlink target for "/etc/portage/env/<category>/<package>".

# Changes compiler options for source-level debugging.

debugging_filter_ncm9q70alywv346gx4ug8hwrq() {

   local S1 S2 OLD NEW T REST M SET

   S1=1; S2=1

   while true; do

      case $S1 in

         1) # Start variable modification.

            case $S2 in

               1) OLD="$CFLAGS";;

               2) OLD="$CXXFLAGS";;

               3) OLD="$FEATURES";;

               4) OLD="$USE";;

            esac

            SET=

            case $S2 in

               1 | 2) SET="-O1 -DDEBUG -ggdb3 -feliminate-unused-debug-symbols -fno-omit-frame-pointer -fstack-check";;

               3) SET="noclean nostrip";;

               4) SET="debug";;

            esac

            OLD="$(echo "$OLD" | tr -s '[[:space:]]' ' ' | sed -e 's/^ *//; s/ *$//')"

            NEW=

            S3=1

            S1=2

            ;;

         2) # Modify next token if necessary.

            if [ -z "$OLD" ]; then

               S1=3

               test -n "$SET" && NEW="$NEW${NEW:+ }$SET"

               continue

            fi

            REST="${OLD#* }"

            if [ "$OLD" = "$REST" ]; then

               T="$OLD"; OLD=

            else

               T="${OLD%$REST}"; T="${T% }"; OLD="$REST"

            fi

            M=

            case $S2 in

               1 | 2)

                  case "$T" in

                     -O[0-9]) M=1;;

                     -g*) M=1;;

                     -DNDEBUG | -DDEBUG | -D_DEBUG) M=1;;

                     -fomit-frame-pointer | -fno-omit-frame-pointer) M=1;;

                     -fstack-check | -fno-stack-check) M=1;;

                     -fno-eliminate-unused-debug-symbols | -feliminate-unused-debug-symbols) M=1;;

                  esac

                  ;;

               3)

                  case "$T" in

                     nostrip | keeptemp | keepwork | noclean) M=1;;

                  esac

                  ;;

               4)

                  case "$T" in

                     debug) M=1;;

                  esac

                  ;;

            esac

            if [ -n "$M" ]; then

               # $T matches a debugging-relevant token.

               test -z "$SET" && continue

               T="$SET"; SET=

            fi

            NEW="$NEW${NEW:+ }$T"

            ;;

         3) # Complete variable modification.

            case $S2 in

               1) CFLAGS="$NEW"; S2=2;;

               2) CXXFLAGS="$NEW"; S2=3;;

               3) FEATURES="$NEW"; S2=4;;

               4) USE="$NEW"; return;;

            esac

            S1=1

            ;;

      esac

   done

}

debugging_filter_ncm9q70alywv346gx4ug8hwrq

unset -f debugging_filter_ncm9q70alywv346gx4ug8hwrq

echo "OVERRIDING FLAGS: debugging"

#echo "CFLAGS=$CFLAGS"; echo "CXXFLAGS=$CXXFLAGS"; echo "FEATURES=$FEATURES"; echo "USE=$USE"

```

Script "/etc/portage/env/default":

```
#! /bin/false

# This is a symlink dummy target.

# It does absolutely nothing.

#

# Use it to temporarily remove special optimizations

# from some target, without a need to remove the

# target's symlink completely.

```

----------

## Guenther Brunthaler

I have also written a simple helper script, which extracts the "category/package"-specifications from the output of "emerge" in a form suitable as input for my "set-optimization-level" script.

Script "/usr/local/sbin/extract-category-and-package":

```
#! /usr/bin/perl

# Extracts <category>/<package> from an emerge-generated package list.

#

# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/extract-category-and-package $

# $Author: root $

# $Date: 2006-10-30T20:34:30.295020Z $

# $Revision: 494 $

while (defined($_= <>)) {

   print if s{^.*] (\S*?)-\d.*?$}{$1};

}

```

This script is handy if you want to optimize a package you are about to emerge as well as all the helper packages it depends on.

For instance, let's assume you wanted to install "games-fps/industri" and make sure it runs fully optimized.

Just do a

```
# emerge --pretend --emptytree games-fps/industri | extract-category-and-package \

  | set-optimization-level
```

and my scripts will mark the package as well as all of its dependencies for subject to heavy optimization.

----------

## Guenther Brunthaler

Always wondered what a specific dependency of a package was good for?

Of course, it is always possible to manually run esearch or a similar utility in order to get a description.

But that's laborious if you want to do this for all the packages in a rather long ist.

That's why I wrote another little filter script for this.

It's called comment-packages, takes category/package list on standard input, writing the same list to standard output, with package descriptions appended to each line.

For instance, to comment all the dependencies of a package "games-fps/industri", just run

```
# emerge --pretend --emptytree games-fps/industri \

  | extract-category-and-package | comment-packages
```

and my script will display:

```
virtual/libiconv:Virtual for the GNU conversion library

sys-devel/gettext:GNU locale utilities

sys-libs/gpm:Console-based mouse driver

sys-libs/ncurses:console display library

virtual/libintl:Virtual for the GNU Internationalization Library

sys-apps/texinfo:The GNU info program and utilities

sys-apps/groff:Text formatter used for man pages

sys-devel/gnuconfig:Updated config.sub and config.guess file from GNU

sys-libs/db:Berkeley DB

sys-libs/gdbm:Standard GNU database libraries included for compatibility with Perl

sys-devel/libperl:Larry Wall's Practical Extraction and Reporting Language

dev-lang/perl:Larry Wall's Practical Extraction and Report Language

perl-core/Test-Harness:Runs perl standard test scripts with statistics

app-shells/bash:The standard GNU Bourne again shell

app-admin/perl-cleaner:User land tool for cleaning up old perl installs

perl-core/PodParser:Base class for creating POD filters and translators

perl-core/Test-Simple:Basic utilities for writing tests

virtual/perl-Test-Simple:Virtual for Test-Simple

dev-perl/Locale-gettext:A Perl module for accessing the GNU locale utilities

sys-apps/help2man:GNU utility to convert program --help output to a man page

sys-devel/autoconf-wrapper:wrapper for autoconf to manage multiple autoconf versions

sys-devel/m4:GNU macro processor

sys-devel/autoconf:Used to create autoconfiguration files

sys-devel/automake-wrapper:wrapper for automake to manage multiple automake versions

sys-devel/automake:Used to generate Makefile.in from Makefile.am

sys-apps/man:Standard commands to read man pages

sys-apps/findutils:GNU utilities for finding files

sys-devel/binutils-config:Utility to change the binutils version being used

sys-devel/binutils:Tools necessary to build programs

sys-devel/libtool:A shared library tool for developers

dev-util/pkgconfig:Package config system that manages compile/link flags

x11-misc/util-macros:X.Org autotools utility macros

```

Caveat: As my script is very simple, it requires that you have the eseach package installed, and will run esearch for each category/package combination in the input list. And as esearch will only list descriptions which it can find, packages without any description will be silently skipped on output by my comment-packages script.

Finally, here is the script.

Script "/usr/local/sbin/comment-packages":

```
#! /bin/sh

# Takes a list of <catagory>/<package> and adds ":<comment>".

#

# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/comment-packages $

# $Author: root $

# $Date: 2006-10-30T20:34:30.295020Z $

# $Revision: 494 $

while read PKG; do

   esearch --fullname --nocolor --own '%p:%d\n' "$PKG" | head --lines 1

done

```

----------

## Guenther Brunthaler

In case you might be wondering what my heavily-optimized and debugging scripts actually do, here is some explanation.

Both scripts work in a similar way: They examine the currently defined CFLAGS, CXXFLAGS, FEATURES and USE settings as specified in you /etc/make.conf.

That is, they will be using the settings from your system if you use the scripts, not mine.

Then the scripts scan those settings for particular items which are considered significant for determining whether we are building an optimized or debugging (or normal) version of a package.

Such items might be CFLAGS such as "-O<something>", "-f-omit-frame-pointer", "-g" etc.

All those items will be removed from the current CFLAGS, FEATURES etc. in the first pass of my script.

Any other settings, such as "-march", will not be touched and be left as they are.

In the second pass my scripts will add the settings which are specific for a "heavily-optimized" or "debugging" build.

For instance, in "heavily-optimized" CFLAGS like "-O3", -f-omit-frame-pointer" and "-DNDEBUG" will be added.

In contrary, the script "debugging" will add CFLAGS like "-O1", "-ggdb", -f-no-omit-frame-pointer" and "-DDEBUG".

If you want to modify the scripts to disable/enable different options, just look at the case with the "SET=" lines: The case specifies which environment variable should be modified (CFLAGS, FEATURES, ...), and the assignment specifies the options to add.

Similarly, the case with those "M=1" statements must be modified to filter out any flags which you want to be modified.

For instance for optimized builds it's a good idea to use CFLAGS "-DNDEBUG" and remove any CFLAGS like "-DDEBUG", "-D_DEBUG".

In order to implement this (which already has been done in my scripts), all those variants have to be filtered out first (in the "M=1"-lines of my script), and then only the variants which refer to optimized builds have to be added in the "SET=" lines of the script.

In any case, it's important that any setting you want to be modified has to have a case in both parts of my script, in the "SET="-section as well as in the "M=1" section.

If you just added an item to the "SET=" section but not also a corresponding case to the "M=1" section, it might happen that both types of CFLAGS are set at the same time: The original ones as well as your modified ones. This will generally be a bad idea.

You might also notice that my script can easily be modified to change items of any environment variable, not just CFLAGS, CXXFLAGS, FEATURES and USE.

In order to modify different environment variables as well, just add more cases to the sections which load and save the environment variables.

You will especially have to change the lines

```
case $S2 in

                  1) CFLAGS="$NEW"; S2=2;;

                  2) CXXFLAGS="$NEW"; S2=3;;

                  3) FEATURES="$NEW"; S2=4;;

                  4) USE="$NEW"; return;;

               esac
```

to something like

```
case $S2 in

                  1) CFLAGS="$NEW"; S2=2;;

                  2) CXXFLAGS="$NEW"; S2=3;;

                  3) FEATURES="$NEW"; S2=4;;

                  4) USE="$NEW"; S2=5;;

                  5) OTHERVAR="$NEW"; return;;

               esac
```

in order to allow modification of $OTHERVAR within the script.

I hope you get the idea.

----------

## Guenther Brunthaler

If my "heavily-optimizing" script detects that the "debug" USE flag is set for a package, it will invoke the "debugging" script automatically instead.

This allows one to toggle beween heavily-optimitzing and debugging via the debug USE flag for all packages on which my set-optimization-level script has been applied already, without a need to change the symlink in such cases.

----------

## mark_alec

Moved from Portage & Programming to Documentation, Tips & Tricks.

----------

## sliwowitz

Guenther Brunthaler, thanks a lot for your scripts. They are exactly what I was dreaming about. Using them gives you imho even more power than a modified portage bashrc. I would send you some "thumbs up" smileys, but they aren't in forum's assortement...

----------

## mudrii

nice scripts I will go and try to check it 

Regards

----------

## Element Dave

 *Guenther Brunthaler wrote:*   

>  *sliwowitz wrote:*   Is there a way to use this feature for specifying CFLAGS for larger groups of packages? (like setting -O3 for x11-base/*) 
> 
> Well, not automatically, I'm afraid.
> 
> But of course it is possible to write a script to aid in such cases.
> ...

 

Here is a cleaner and "safer" modification of your above script:

```
#!/usr/bin/perl -w

use strict;

no warnings qw(uninitialized);

my $outdir= '/etc/portage/env';

my $target= '../heavily-optimized';

while (chomp(my $atom = <>)) {

   next if $atom =~ /^(?:#|\s*$)/;

   $atom =~ s/^\s*|\s*$//g;

   [^\w\+\/-]

   if ($atom !~ /^[\w\+-]+\/[\w\+-]+$/) {

      warn("Skipping invalid package specification: $atom\n");

      next;

   }

   my $category = (split(/\//, $atom))[0];

   my $target_dir = join '/', $outdir, $category;

   if (-!d $target_dir) {

      mkdir($target_dir) or die "Cannot create \"$target_dir\": $!\n";

   }

   my $link = join '/', $outdir, $atom;

   next if -e $link; # don't touch existing links/files

   symlink($target, $link) or die "Cannot symlink $link => $target: $!\n";

   print "symlinked $link => $target\n";

} 

```

I personally prefer to keep everything in a single file in conjunction with a custom /etc/bashrc.  I much prefer external scripting languages to bash/shell scripting, so my bashrc does almost nothing itself, instead calling a perl script to do the real work.

BTW, is there an official specification for what characters may appear in an "atom base" [as described in ebuild(5)]; i.e., in the format category/package, without the version?  The character classes in the expression [\w\+-]+/[\w\+-]+ are sufficient to match all packages currently in portage, but I'd still like to know for the sake of completeness.Last edited by Element Dave on Wed Jan 10, 2007 4:12 am; edited 1 time in total

----------

## steveL

Thanks Guenther. As ever, you provide the glue to make the system stay up!

----------

## swimmer

Hmm - nvidia-drivers/alsa-driver/ndiswrapper all have problems with sandbox violations during emerge on 2.6.19 kernels so I thought I could use this feature and placed this line in /etc/portage/env/media-sound/alsa-driver:

 *Quote:*   

> FEATURES="-sandbox"

 

Same for x11-drivers/nvidia-drivers & net-wireless/ndiswrapper ...

But I still get sandbox violation errors :-/

What am I missing?

Greetz

swimmer

----------

## Guenther Brunthaler

 *swimmer wrote:*   

> 
> 
> FEATURES="-sandbox"

 

Does the FEATURES variable support removing single features by prefixing them with "-" at all?

I'm not so sure. It's similar to USE, of course - but does it really work the same? I don't know.

But even if this might be the case in /etc/make.conf or when running emerge from the command line: In /etc/portage/env scripts, variable settings work differently.

Settings in /etc/portage/env-scripts are not cumulative; they are executed by the shell, not by the python parser of portage.

Which means, if you don't want "sandbox" in $FEATURES, just write something like

FEATURES=""

which does not set "sandbox".

However, it might be better to set FEATURES to the same settings as in your /etc/make.conf, and just omit "sandbox" from that list of features.

----------

## swimmer

Hi Guenther, 

```
FEATURES=-sandbox emerge alsa-driver
```

works flawless - it's more likely another problem ...

Greetz

swimmer

----------

## Guenther Brunthaler

Hi swimmer,

 *swimmer wrote:*   

> works flawless - it's more likely another problem ...
> 
> 

 

You are referring to the paragraph *Quote:*   

> but generally it won't, same for most other variables usually found in make.conf as they are used before /etc/portage/env is parsed (which btw isn't done by portage itself).

 are you?

Well, then we are out of luck regarding FEATURES in /etc/portage/env I guess!

Too bad.  :Sad: 

So we have to look for a better solution it seems.

But at least the /etc/portage/env approach works for normal shell variables like CFLAGS.

BTW, as it seems you have troubles with distcc: Have you ever tried to leave FEATURES=distcc alone and to disable distributed compilation indirectly?

That is, override

```
MAKEOPTS=j1
```

in the /etc/portage/env snippet.

This should serialize compilation even without disabling distcc... perhaps it helps?

However, this can work only if the /etc/portage/env snippet is parsed before make is called... which I don't know.

----------

## swimmer

Hu? I don't know exactly why people think I have problems with distcc but I can assure you I do *not* have any problem with distcc since I don't use it  :Wink: 

And indeed it seems that "FEATURES=-sandbox" is not support in /etc/portage/env  :Sad: 

Out of luck

swimmer

----------

## Guenther Brunthaler

 *swimmer wrote:*   

> Hu? I don't know exactly why people think I have problems with distcc

 

Oops - sorry; I somehow mismatched your posting with those from the referenced thread, where someone was having issues with distcc.

 *swimmer wrote:*   

> Out of luck

 

Yes.

Which, btw, debunks the myth that an application like Portage written in a script language like Python is easier to understand/master than one which has been written in C or some other compiled language.

From a user's perspective, it's pretty much the same (provided the C application is OSS of course): Without proper documentation of certain features, it's very hard to get them working.

Reading the source code is certainly an invaluable tool, but as the complexity or size of an application exceeds some limits, the practical usefulness of this approach is also limited.

But at least, script languages protect against buffer overruns and similar simple but common errors, which certainly is an advantage.

Anyway, if you ever should happen to find a solution to the per-package FEATURES problem, please let me know by posting here.

I'll do the same.

----------

## steveL

AFAIC portage should just be written in bash.

----------

## Dralnu

 *steveL wrote:*   

> AFAIC portage should just be written in bash.

 iirc, one of the first versions was in pure bash.

----------

## Conan

sourcemages package manager is written in pure bash.

It's kind of silly.

----------

## Guenther Brunthaler

 *steveL wrote:*   

> AFAIC portage should just be written in bash.

 

I'm not so sure about that.

Scripting in bash is fun, as long as one artificially limits the file names or inputs which can be processed by the script.

That is, the file names or input strings processed should better not contain any

space characters

tab character

single quotes

double quotes

dollar signs

exclamation points

greater than / less than characters

vertical bar symbol ("or")

ampersand character

literal backslashes

question marks

asterisks

start with a tilde

fancy UTF-8 characters

newline characters

which are all perfectly legal characters in filenames,

or the script is likely to run into problems sooner or later, if the coder is not extremely careful.

Of course, one is free to say "such characters should not be used in filenames", but that won't stop a script from failing if an unaware user still creates such file names by accident.

Depending on what the script does, such failures could be everything from harmless to catastrophic.

In other languages, there are no or at least not that many issues with "special/meta characters" and quoting as in bash/sh.

And then not to forget the problem of all shell languages: The arguments of each command can be seen by anyone using "ps -Af", which is bad fun if passwords or sensitive data have to be processed.

That does not mean bash is bad per se.

But it's not a general purpose language, and will never beat Perl, Python, C, awk or even JAVA for application development.

In my opinion, awk is the least-bloated language capable of actual application development. At least for small applications. But bash isn't.

----------

## Dralnu

Personally, I think a package managment system should be in a hard-coded language (I'm thinking C here), or an easy to use scripting language that can actually do the job, and the devs know well enough to work with (python, perl, ruby). awk and bash, while fun to use and they work for scripts, I think should be left behind for such an effort.

Personally I think making ebuilds in bash is a bad idea (because, well, it requires bash to be installed, which is a wasted shell if you use, say, zsh) since it isn't too hard to setup a syntax in a language to handle such things. If you use a database, then even better.

----------

## Dralnu

 *Conan wrote:*   

> sourcemages package manager is written in pure bash.
> 
> It's kind of silly.

 

Works for a starting point if nothing else. Got to give bash that.

----------

## Conan

err, 

1) You're arguement that it requires bash to be installed doesn't hold weight because portage also requires bash to be installed.

2) Sourcemage's package manager is just as functional as portage, it even does some things that portage can't do currently.  I'd say its more than a starting point  :Smile: 

----------

## Guenther Brunthaler

 *Dralnu wrote:*   

> awk and bash, while fun to use and they work for scripts, I think should be left behind for such an effort.

 

I agree.

Portage is a rather complex tool (as any capable package management system will be), and this beyond the scope of application well suited for bash, and also awk.

I also suppose much of the difficulties with ebuilds originate from the fact that bash, while providing powerful means of expression, does not and cannot actually enforce the ebuild layout guidelines.

For instance, many ebuilds use USE-flags without specifying them in the "IUSE=" lines, although they are supposed to do so.

Just look at the many "QA"-warnings when emerging ebuilds.

Add that to missing or incorrectly specified dependency information, or look at the mess with /etc/init.d where dependencies are verifiably resolved incorrect in some (although rare) situations.

The Linux Standard Base has a clear and well-defined run-level system. Was it really necessary to change this - in favor to a new, more complicated system which, although more elegant, just does not always work correctly, and is totally incompatible with all the other distributions?

I think, Gentoo and the ebuild system are too complicated in this regard, and should be simplified (ebuild scripting) or unified (LSB/FHS conformance).

On the other hand, most of the time it works.

And even if it does not, it's usually rather simple to fix - if you are a programmer and have an understanding of the programming languages involved. Most of the time, it's just the names of header files which have changed, or some declarations are missing. Such problems are rather easy to fix using the "ebuild compile" command and a text editor.

These are also the reasons why I still stick with Gentoo despite of the above facts - there is no other distro except for LFS (which can hardly be called a "distro") where one can do this.

Combined with USE flags and user-defined CFLAGS, this is raw power for the administrator - it's quite the opposite VISTA tries to do.

I only wish ebuilds and Portage were a bit more stable, forcing the administrator to fewer often has to actually wield those raw powers...

And writing ebuilds in a more structured, easier to be verified way (in some other specification language), would be a good starting point.

I wouldn't even use C for this - some sort of expert system might be even more appropriate. There are many good and free expert systems available, some of them completely written in C. And even a full-fledged, C based XPS will hardly ever require as much space as a full Python installation does.

Expert systems could be made intelligent enough to understand all the problems regarding package interdependencies, USE-flags and update build order resolution. And take actions to resolve those problems.

But it's just dreaming - I know this cannot happen. No-one will convert hundreds of thousands of existing ebuilds into whatever different specification language just to make them more reliable.

So ebuilds will continue to be flawed, admins will continue to be able to resolve most of those matters "by hand", and "normal" users without programming experience... will continue to run away and choose UBUNTU/FEDORA instead.

But hey - what am I fussed about? I am a programmer, Gentoo is the programmers/power users club, and so I should feel right at home anyway...  :Wink: 

----------

## gringo

as a side note :

 *Quote:*   

> Hmm - nvidia-drivers/alsa-driver/ndiswrapper all have problems with sandbox violations during emerge on 2.6.19 kernels so I thought I could use this feature and placed this line in /etc/portage/env/media-sound/alsa-driver 

 

if you use gentoo-sources or  2.6.19.2 its already fixed. More info here  :Wink: 

And thanks for sharing your scripts Guenther, very handy  :Smile: 

cheers

----------

## Guenther Brunthaler

 *gringo wrote:*   

> if you use gentoo-sources or  2.6.19.2 its already fixed.

 

Thanks for the info.

 *gringo wrote:*   

> More info here 
> 
>  *Quote:*   "Because then when
> 
> the compilation runs as root some as versions will delete /dev/null on a error.
> ...

 

ROTFL!  :Smile: 

----------

## swimmer

 *gringo wrote:*   

> if you use gentoo-sources or  2.6.19.2 its already fixed. More info here 
> 
> cheers

 

Hmm - I'm using =sys-kernel/gentoo-sources-2.6.19-r3 and still get sandbox violations with those modules:

        =media-sound/alsa-driver-1.0.14_rc1

        =x11-drivers/nvidia-drivers-1.0.9746

        =net-wireless/ndiswrapper-1.34

All that on AMD64 ...

Greetz

swimmer

----------

## Dralnu

 *Conan wrote:*   

> err, 
> 
> 1) You're arguement that it requires bash to be installed doesn't hold weight because portage also requires bash to be installed.

 

Whats your point here? I obviously and missing something here.

 *Quote:*   

> 2) Sourcemage's package manager is just as functional as portage, it even does some things that portage can't do currently.  I'd say its more than a starting point 

 

I still say it would be better to use a diffrent language. Like he said, bash has some fairly hard-set limits, and it, again, requires bash, which is a waste for people who don't use it.

----------

## aidanjt

source based package managers will always have a level of shell script dependency anyway with regards to at least configure.

That said, portage really needs a rewrite to a compiled language since it's now getting too heavy for python to handle efficiently.

I don't mind the ebuilds themselves having a bash dependency but the portage core defiantly needs rewritten.

----------

## Dralnu

 *AidanJT wrote:*   

> source based package managers will always have a level of shell script dependency anyway with regards to at least configure.
> 
> That said, portage really needs a rewrite to a compiled language since it's now getting too heavy for python to handle efficiently.
> 
> I don't mind the ebuilds themselves having a bash dependency but the portage core defiantly needs rewritten.

 

The problem is none of the devs want to spend that kind of time on it. Python might could handle Portage well, but it would still need a rewrite.

Using a script like bash for something like ebuilds it kind of stupid imho. Hell, I think something like xml would do better, with the best option being a database.

Personally I'd prefer a modular design, with things such as emerge --search, the data-handling system (saving ebuilds, the caches, ect), and all that being entered seperatly, with the core making calls to each so that, say, you wanted to use a specific database (we'll say Oracle here) instead of plain text or MySQL, you could emerge a new data-handler. Problem with converting between them I think would be fairly easy to get around (output things into plain text, then reinput them into the new database system), and it would give the user control over what Portage would use for what, with emerge itself being hard-coded into the core of the system (to remove the chance of breakage).

----------

## PaulBredbury

Ebuilds have to be scripts - look at the complexity of e.g. /usr/portage/x11-drivers/nvidia-drivers/nvidia-drivers-1.0.9746.ebuild

----------

## Dralnu

 *PaulBredbury wrote:*   

> Ebuilds have to be scripts - look at the complexity of e.g. /usr/portage/x11-drivers/nvidia-drivers/nvidia-drivers-1.0.9746.ebuild

 

Still doesn't mean you have to use bash. Its not hard to put a scripting language into the code.

Heck, awk may even work well for that.

----------

## gringo

 *Quote:*   

> "Because then when
> 
> the compilation runs as root some as versions will delete /dev/null on a error.
> 
> This has happened in the past."

 

 :Rolling Eyes:   :Wink: 

 *Quote:*   

> Hmm - I'm using =sys-kernel/gentoo-sources-2.6.19-r3 and still get sandbox violations with those modules:
> 
> =media-sound/alsa-driver-1.0.14_rc1
> 
> =x11-drivers/nvidia-drivers-1.0.9746
> ...

 

amd64 here too and no problems. -r4 is available, try out and reopen above bug if it doesnt work.

cheers

----------

## Element Dave

 *Guenther Brunthaler wrote:*   

> Which, btw, debunks the myth that an application like Portage written in a script language like Python is easier to understand/master than one which has been written in C or some other compiled language.
> 
> From a user's perspective, it's pretty much the same (provided the C application is OSS of course): Without proper documentation of certain features, it's very hard to get them working.
> 
> Reading the source code is certainly an invaluable tool, but as the complexity or size of an application exceeds some limits, the practical usefulness of this approach is also limited.

 

Another problem with relying on source code as opposed to proper documentation is that one seldom has any indication whether certain functionality is intended to persist into future versions,  If public hooks are documented, there is at least some sense that they won't disappear tomorrow.

----------

## Element Dave

 *steveL wrote:*   

> AFAIC portage should just be written in bash.

 

I hope you have a good sense of humor.

----------

## steveL

 *Element Dave wrote:*   

>  *steveL wrote:*   AFAIC portage should just be written in bash. 
> 
> I hope you have a good sense of humor.

  :Very Happy:  TBH I can't believe I wrote that!

----------

## Dralnu

 *Element Dave wrote:*   

>  *Guenther Brunthaler wrote:*   Which, btw, debunks the myth that an application like Portage written in a script language like Python is easier to understand/master than one which has been written in C or some other compiled language.
> 
> From a user's perspective, it's pretty much the same (provided the C application is OSS of course): Without proper documentation of certain features, it's very hard to get them working.
> 
> Reading the source code is certainly an invaluable tool, but as the complexity or size of an application exceeds some limits, the practical usefulness of this approach is also limited. 
> ...

 

If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it".

----------

## likewhoa

thx Guenther Brunthaler this looks like a good feature but it would be best if you had implemented per-package not overall cflags for packages that are gonna be merged with custom cflags.

the whole point of the per-package custom package.{cflags,cxxflags,features,ldflags} is to set unique values for packages not one global set of package.{cflags,cxxflags,features,ldflags} per all. anyways, thanks for the scripts and your effort. I guess this would have been something for me if it handle the variables differently. I will stick with setting files with variables instead of your symbolic link method. you gotta remember that not all packages will merge with one global set compile flags.

----------

## Element Dave

 *Dralnu wrote:*   

> If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it".

 

Well, how does one know whether something is a "feature" if it isn't documented?  Do you audit the source of every program on your machine to determine what "features" you have available?Last edited by Element Dave on Tue Mar 06, 2007 11:08 pm; edited 1 time in total

----------

## Dralnu

 *Element Dave wrote:*   

>  *Dralnu wrote:*   If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it". 
> 
> Well, how does one know whether something is "feature" if it isn't documented?  Do you audit the source of every program on your machine to determine what "features" you have available?

 

Apparently some people do.

----------

## Element Dave

 *Dralnu wrote:*   

>  *Element Dave wrote:*    *Dralnu wrote:*   If they put in a feature and released it, it would make little sense to retract it later due to the fact some users WILL keep using that feature (or try to). When developing the system, you should put such experimental features into a seperate repository for testing to see if you will really use it or not before saying "hmm, this looks nice. Lets go ahead an give this to them and not document it. No one will ever touch it". 
> 
> Well, how does one know whether something is "feature" if it isn't documented?  Do you audit the source of every program on your machine to determine what "features" you have available? 
> 
> Apparently some people do.

 

That's unlikely.  It certainly isn't an efficient use of time compared to reading documentation.  Consider the amount of time needed to write documentation, and contrast that with the collective amount of time spent by the users (which also include developers) of that code that would have been saved had  documentation been available.  More progress would be made in many areas.

----------

## Dralnu

Point is, its a generally stupid idea to make a feature avalible if you don't plan on people using it. This is why we have releases and development trees. Make sure the feature works as best you can, THEN release it. Considering this is over a package manager known to be used by coders and oddballs, you don't put anything past them.

----------

## kowal

CFLAGS per package is also being discussed at gentoo-user mailing list.

http://www.mail-archive.com/gentoo-user@lists.gentoo.org/msg54130.html

----------

## steveL

 *Guenther Brunthaler wrote:*   

>  *steveL wrote:*   AFAIC portage should just be written in bash. 
> 
> I'm not so sure about that.
> 
> Scripting in bash is fun, as long as one artificially limits the file names or inputs which can be processed by the script.
> ...

 

Well I didn't see this post before, but basically you need to hang out in #bash on irc.freenode.org. Here's some links to start:

http://wooledge.org/mywiki/BashGuide | http://www.grymoire.com/Unix/index.html | http://wooledge.org/mywiki/BashFaq | and of course | http://www.shelldorado.com/ ;)

File globbing (eg for i in *) handles all those filenames, the only thing you need to be aware of is !quotes: do cmd "$i"; done

BASH is actually an excellent environment to code in imo. awk is nice too tho :D

----------

