# vanilla-sources new stablization policy

## STrRedWolf

For those who don't know, this was in the gentoo repository news:

```
2013-08-07-vanilla-sources-stablization-policy

  Title                     vanilla-sources stabilization policy

  Author                    Mike Pagano <mpagano@gentoo.org>

  Posted                    2013-08-07

  Revision                  1

The Gentoo Kernel Team will no longer be providing stable

vanilla-sources kernels. All currently stabilized vanilla-sources

versions will be dropped to ~arch. The Arch teams, via normal requests

of the Kernel Team, will continue to stabilize gentoo-sources kernels

upon request. This decision is based on the facts that upstream is now

releasing approximately 1-2 vanilla-sources kernels a week. Arch teams,

understandably, are unable to keep up with this rate of release.  As

most vanilla releases contain security fixes, the user who only runs

stable vanilla-sources will consistently be behind and potentially at

risk.  For the latest "upstream kernel unpatched by Gentoo", we

recommend users add 'sys-kernel/vanilla-sources' to their

package.accept_keywords file. gentoo-sources will continue to be a

tested and supported version for Gentoo users.

Note: This news item only applies to gentoo-sources and vanilla-sources.

Other kernels currently maintained in portage have their own policies

and procedures in place today.

```

I must be missing something.  Does this mean that vanilla-sources will be permanently ~arch until someone finds out from Greg K-H that 3.10 is stable, and files a stablization bug? 

If so, I'll just have to file the bug.  Greg K-H saying 3.10 is stable series. Google Plus link to stable kernels listing, with 3.0.89, 3.4.56, and 3.10.5 as stable/long term.

----------

## jimmij

It is already marked longterm on official site

https://www.kernel.org/category/releases.html

ps. BTW, newest 3.10.5 is still terrible buggy so far and freezes very frequently.

----------

## asturm

@STrRedWolf: Upstream stable does not equal downstream stable.

It means it makes no sense to apply the usual stable policy to vanilla-sources when by the time one version is stabilised in Gentoo it is already two or three versions behind latest upstream bump. It will remain ~arch now, but anyone is free to put e.g. =vanilla-sources-3.10* into package.keywords.

@jimmij: Seems your trouble is related to one driver. Others won't be troubled by that at all.

----------

## Hu

OP: the Greg-KH statement you linked indicates that he plans to support it for longer than he supports normal stable trees, not that it is any more or less reliable than a regular stable tree.  As genstorm hinted, and the Gentoo announcement states, no stabilization bugs are needed or wanted.  Gentoo lacks the manpower to keyword vanilla versions fast enough to keep up with Greg's releases.

To me, this calls into question the support of the gentoo-sources packages, since they are vanilla-sources plus some Gentoo-specific patches.  If vanilla is moving too fast, then either gentoo-sources will cease to track GregKH stable kernel releases (and users will have the problems described in the announcement) or it will track GregKH stable kernels and the only benefit of this change is that for each new release, testers need only work on gentoo-sources-N, rather than gentoo-sources-N and vanilla-sources-N.  The announcement says gentoo-sources will continue to be a tested and supported kernel, but it is not clear whether it will continue to be released fast enough to roll out the same security fixes that GregKH is pushing, particularly since upstream Linux developers (not just Greg), have a bad habit of not clearly marking which changes are security related.

----------

## hadrons123

I understand that If a package is called vanilla, then there is not too many patches from the downstream. So people should already know what they are getting into when they choose vanilla-sources. But a documentation or wiki info regardging this should help new people before they dive-in.

----------

## TomWij

Greetings, as a member of the Gentoo Kernel Project I am going to try to clarify any questions or doubts that arrive; answers are from our perspective, such that you know what our intents are and why we make certain decisions. As a disclaimer, not everything I say is necessarily blessed by the rest of the Gentoo Kernel Project or the Gentoo Project; but, I intend to match their thoughts as much as possible.

 *STrRedWolf wrote:*   

> Does this mean that vanilla-sources will be permanently ~arch

 

Yes, vanilla-sources will be permanently ~arch assuming this decision is never revoked; what it means for something to be ~arch means that we can not consider it stable (in terms of Gentoo QA and Security) or cannot ensure that stable packages are provided in a timely fashion (because resources are limited).

Since GregKH wanted the vanilla-sources versions to more closely reflect the versions that are listed on https://www.kernel.org/ most versions only stay in Portage for half a week to a week; this small amount of time, gives no room for stablization with the limited resources we have.

May I remind you that Gentoo does not solely deal with the kernel, there is a whole tree to be stabilized; so, they cannot extensively test vanilla-sources twice a week.

 *STrRedWolf wrote:*   

> ... until someone finds out from Greg K-H that 3.10 is stable, and files a stablization bug?

 

Stable in Gentoo QA and Security terms does not match what upstream deems as stable; to give our users a good service, we intend to act as a buffer to prevent upstream problems from reaching our users while also trying to ensure that important upstream fixes reach our users in a timely fashion. As you can see, this is what we are doing with gentoo-sources.

Users are therefore prevented with the choice to accept the ~ keyworded vanilla-sources and follow upstream or instead follow what we consider stable in terms of Gentoo QA and Security.

 *STrRedWolf wrote:*   

> If so, I'll just have to file the bug.

 

Please spare yourself the time from doing so, unless it is meant as an objection; we are well aware of the upstream situation, we look multiple times at their homepage a day, we follow their ML and so on...

 *jimmij wrote:*   

> It is already marked longterm on official site
> 
> https://www.kernel.org/category/releases.html

 

Correct, we are aiming to stabilize either the EOL version of 3.9 or the next release of 3.10; since 3.10, we are rooting more for that since it is intended to be a kernel with a lot of API compatibility, whereas 3.9 is likely to require us to backport too much fixes for no good.

 *jimmij wrote:*   

> ps. BTW, newest 3.10.5 is still terrible buggy so far and freezes very frequently.

 

Which bugs do you experience?

There are 102 commits coming up with 3.10.6; so, the next release should be much more promising.

Do you know with which version these bugs were introduced? We might be able to get an idea from the commit log and your .config.

If not, we would appreciate if someone is willing to do a git bisect to find the offending commit...

Please file kernel bugs at https://bugs.gentoo.org such that we can keep track of them and try to resolve thim.

 *Hu wrote:*   

> If vanilla is moving too fast, then either gentoo-sources will cease to track GregKH stable kernel releases (and users will have the problems described in the announcement)

 

The decision and announcement isn't because there was a change in speed; well, there was some time ago, but we're still following along at the same rate in terms of pushing them to the Portage tree.

As for stabilization of gentoo-sources, we look at what version works the most "stable" in terms of Gentoo QA and Security for our users; if you look at the different options you have to stabilize, we want to pick the one which contains the most patches and the least refactoring and the least new features.

The first release in a branch comes with the refactoring, new features and fixes that have been gathered in the release candidate phase; any future releases in a branch only come with fixes backported from the next release candidate or branch, there is no refactoring or new features in later releases of a branch. So, we attempt to pick the latest releases of a branch to stabilize.

If you take jimmij's situation as an example, you can see that (likely) 3.10.0 broke it for him whereas 3.9.11 worked fine and that he might see a fix for it soon in a later 3.10 release. So, from these options we either stabilize 3.9.11 or wait for a later 3.10 release if we consider it is no longer worth to pick up 3.9.11.

There's currently some minor issues blocking stabilization of 3.9.11; therefore, we are considering the option to stabilize a kernel like 3.10.6 or later instead.

 *Hu wrote:*   

> or it will track GregKH stable kernels and the only benefit of this change is that for each new release, testers need only work on gentoo-sources-N, rather than gentoo-sources-N and vanilla-sources-N.

 

Since the kernel we intend to stabilize for gentoo-sources is no longer necessary going to be present in the vanilla-sources package, we can't stabilize both with the same version anymore either.

 *Hu wrote:*   

> The announcement says gentoo-sources will continue to be a tested and supported kernel,

 

Confirmed.

 *Hu wrote:*   

> but it is not clear whether it will continue to be released fast enough to roll out the same security fixes that GregKH is pushing,

 

Recently, I'm doing my best to port back CVE commits; as for important security fixes, we immediately do our best to get them backported as fast as possible. For instance, one day I came home to see a root exploit on Reddit; in such case I immediately start investigating and make sure the security fix is backported as fast as possible. For less important security fixes, also those that are not CVE; we expect the user that really needs his or her system to be really secure, to run a hardened kernel.

They make the extra efforts to do what we can't promise with gentoo-sources; but well, we don't ought them as important as they are more local exploits or those that simply are not sure to be vulnerable. As you can see, this is also because we are limited in resources, it's either me or mpagano that's currently pushing new genpatches and gentoo-sources releases; while we have one or two interested to join in the last month, we are yet to see if they actually join and contribute.

People are welcome to become Gentoo Developers and help us out, I guess one could say there is always room for more people on the Gentoo Kernel Project team...

 *Hu wrote:*   

> particularly since upstream Linux developers (not just Greg), have a bad habit of not clearly marking which changes are security related.

 

There was a whole conversation I made with him on the gentoo-dev ML; it appears that this historically comes up, they are at the state they are now and haven't come to a solution yet they consider as one that is useful and does not consume too much of their time.Last edited by TomWij on Sat Aug 10, 2013 5:12 pm; edited 2 times in total

----------

## hadrons123

@Tomwij

Big thank you for your post and clarification of the situation.

----------

## Hu

Tom,

I did not mean to imply that you would delay publishing a kernel when you knew it had a security problem.  Rather, as you discovered in the thread with Greg, upstream is reluctant to call out even obvious security problems on the (IMO ill-founded) belief that by calling out known security problems, it encourages users to assume all security problems are known.  Given that upstream does not highlight security problems, it falls to downstream consumers (whether individual users of vanilla sources or the Gentoo kernel team) to examine the changes and try to determine whether a change fixes an exploitable bug, and how readily the bug can be exploited (remote privilege escalation, local privilege escalation on any system, local privilege escalation if a certain configuration (e.g. CONFIG_USER_NS) is in effect, etc.).

As I understand the referenced thread and others like it from prior discussions, upstream works with the premises that "all fixes could be security fixes" and "all security bugs could be exploited in some way, therefore all security fixes are critical."  The latter is disingenuous in my opinion, because a security fix is only critical if it can be exploited by an unauthorized user.  If the machine operator believes (hopefully correctly) that he has closed off all avenues by which an attacker could trigger the security bug, then the fix is not critical to the machine.  This is especially relevant when the mitigation is simple, such as "unload module Z and remove the corresponding .ko so it cannot be loaded."

----------

## TomWij

Every new feature, refactoring and even fix can possibly introduce another security hole as well; so, if you do not go for the first releases of a new branch and backport known security fixes then you avoid introducing new security holes, basically introducing more security. It doesn't help to choose for a new branch release like 3.10.0 for security fixes if it introduces another set of security holes at the same time. That's yet another reason why we prefer the later releases of a branch.

In my personal opinion there are two ways to give an attacker the chance to attack you; one way is to give him more time, the other way is to give him a bigger set of attack vectors. What we try to do is minimize both to a great extent; by backporting fixes in a reasonable amount of time we do not buy them extra time, and by not taking in new commits that are not stability or security fixes we do not enlarge the attack vector set. What upstream does with its stable releases is not give the attacker any time, yet by introducing a lot of new commits that pertain to new features and refactoring they enlarge the attack vector set. They do the opposite with their LTS releases.

You can not fully minimize both, you must make a choice here: 1) Minimize time. (This is what upstream does with its Stable releases) 2) Minimize the size of the attack vector set. (This is what upstream does with its LTS releases) 3) Try to minimize part of the time and part of the size of the attack vector set, effectively trying to have the best of both worlds; such that neither the delay in time or the size of the attack vector set is a vulnerability. (This is what we do, or at least what I want us to see do; try to stay between Stable and LTS)

----------

