# Automatic Perfect Kernel Building

## Gear.0

I just had an idea while I was customizing my kernel.

I am fairly new to anything involving the kernel so I really hope this idea isn't too ridiculous, but even if it is impossible to do something like this I still think I could learn something from an explanation of why.

But, here is what I was thinking:

The best way I could think to describe it is: "kernel learning"

Basically it would be a program that keeps track of what parts of the kernel are actually being used, or if parts are needed. Now, I wouldn't want it to change the kernel automatically but it could at least make suggestions of what it thinks you need to add (which could help solve many problems) or even suggest things to remove that maybe you thought your computer needed but actually has never been used. For example you could see (X kernel component has not been used once in 2 months) and then you can decide if you want to remove it.

Basically it would keep track of everything that has ever been used by you while you use a kernel for some time and tells you what your computer has actually used and what it has never used.

This way you can be 100% sure that you have everything your computer will ever need, and nothing that it doesn't use.

I understand that some things might only be used rarely and might be missed by such a program.. however, if the time ever comes when that thing is used and it doesn't work, the program would be able to inform you that you need to add something to the kernel as a module or otherwise.

In anticipation of some of the responses I might get...

Yes, this is partly brought on by a newbie to kernel compiling being scared that he might forget something or include too much.

But I still think, if nothing more, this could at least be a very helpful diagnostic tool.

[example] some piece of hardware isn't working, so run this program and it tells you what you need in the kernel..

Perhaps it would only work if you had a kernel with absolutely everything in it and it can only tell you what you aren't using... That would be cool too.

Either way it could be very helpful to some people I think.

----------

## NeddySeagoon

Gear.0,

I'm not sure the overhead in the kernel would be tolrable.  Such a program would have to run as a part of the kernel as it would need to keep track of which code the kernel was executing.  It would also need to tie that back to the source code. That means the kernel would need to be made with debug options. That makes it bigger and slower.

Further, some kernel options are a matter of taste/purpose, which would be very hard to detect.

For example, for a desktop, you would set preemption to low latency desktop, for a server, you would set none.

This trades raw throughput, against a faster response to user requests.

I don't know how ou would detect code that was present but never used, as it would not be executed to be dectected.

All in all, I think the performance penalty would be too great.

There is a good site at kernel-seeds.org to help you build a lean, mean kernel an have it boot first time. I think such configure time helpers are a good compromise

----------

## Muso

 *NeddySeagoon wrote:*   

> There is a good site at kernel-seeds.org to help you build a lean, mean kernel an have it boot first time. I think such configure time helpers are a good compromise

 

++

Though I do not use any kernel seeds I wish kernel-seeds was around when I was first building my own kernels.   Trial and error over many years has tuned my kernel configuring to where it is now, but if I would have had a seed in the beginning I could have saved a lot of headache.  It's a great service to the Linux community and I'm thankful for Pappy and Neddy for making available to all.

----------

## Gear.0

I understand,

I guess I should have figured that building a kernel is not as simple as hardware fitting. That's partly what makes it so worrisome for me   :Laughing: 

But, thank you for the kernel-seeds.org link, that should be very helpful, and thanks for giving such a detailed explanation.

----------

## koan

I was thinking it might be possible to do the hardware fitting part of it by doing a scan of the hardware present and updating the .config to enable the drivers you need and disable the rest.  

I am not sure if even this would be feasible given that not all hardware is necessarily present at the time you run a detection script, and it would mean a kernel rebuild each time you wanted to plug something new in.

----------

## BitJam

 *koan wrote:*   

> I was thinking it might be possible to do the hardware fitting part of it by doing a scan of the hardware present and updating the .config to enable the drivers you need and disable the rest. 

 

What you want is:

```
 lspci --> module name --> .config option
```

The first step is already automated.  The second step needs work.

 Go to this page of the kernel-seed site.  It shows you how to use the lspci command to get a list of all your hardware and then you can enter that information here: http://kmuto.jp/debian/hcl/ to get a listing of the modules that are needed to use your hardware.

The non-automated part is figuring out which .config options correspond to which modules.  ISTM that this would simple require a table (perhaps a table for each kernel version).  I think the problem is that there are many kernel flavors and all the kernels are constantly being updated so it could be a lot of work.    Also remember that none of this is perfect.

----------

## Muso

 *BitJam wrote:*   

>  http://kmuto.jp/debian/hcl/ 

 

Whoa, nice site.  tyvm for the link.

----------

## Ant P.

 *BitJam wrote:*   

> The non-automated part is figuring out which .config options correspond to which modules.  ISTM that this would simple require a table (perhaps a table for each kernel version).  I think the problem is that there are many kernel flavors and all the kernels are constantly being updated so it could be a lot of work.    Also remember that none of this is perfect.

 

`make localmodconfig`?

----------

## BitJam

 *Ant_P wrote:*   

> `make localmodconfig`?

 

That's very cool.  I hadn't heard of it before.  A quick peek showed that it starts with lsmod not lspci so it will remove support for any hardware that is not currently being used.  Even so, it looks like it is a great tool for helping out with kernel configuration.  Thanks for the info.

----------

## Gear.0

Well basically the reason I was even thinking of something like this is not to make things simpler, but only more accurate.

What I mean is that I couldn't care less how difficult something is, as long as there is a right way and a wrong way. I wouldn't even mind if some ideas came along to Gentoo that made it even more of a pain to configure your kernel.

All that concerns me (and perhaps I am wrong, so please correct me if I am) but it seems like even if I had a [professional?] kernel builder who has done this for years, building my kernel for me.. there is still a [extremely?] small chance that there will still be some problems..

Basically it's my philosophy (so far, it may change as I get more familiar with Gentoo)

that the process needs to become more certain. Even if by doing that it is made more difficult and more time consuming. I just think that it needs to move to a place where a very motivated newbie could (maybe after doing hours or days of research and configuring) build a kernel the first time without trial and error and be certain that it will work, and use only what is needed and absolutely nothing more.

Perhaps some people share this view, but maybe it is just not doable...

But anyways... ya that's basically what I had in mind and why I came up with the idea in my OP.

----------

## krinn

well, you can make allmodconfig and everything will be as module

next: pickup the drivers list you need for your hardware to put as include in kernel. You will get that list with lspci -k

and voila, you have a kernel that handle anything on your computer.

and if something isn't use, it's a module and udev won't load it because it can't find an utility for it

and if something is use but you don't want it anymore for a reason: blacklist it.

if something isn't load and you need it, just modprobe and you'll good to go.

the only bad thing with this method: you will eat a little more space for unused module and more time to build the kernel, nothing i care of.

But again, nothing stop you from edit your kernel to disable something you really don't want.

But it's the easiest method to build a kernel that will remain small & fast, but still versatile and easy to run.Last edited by krinn on Mon Jul 19, 2010 6:23 pm; edited 1 time in total

----------

## BitJam

 *Gear.0 wrote:*   

> I just think that it needs to move to a place where a very motivated newbie could (maybe after doing hours or days of research and configuring) build a kernel the first time without trial and error and be certain that it will work, and use only what is needed and absolutely nothing more.

 

Well, I think Pappy's Kernel Seeds come closest to this goal.  I suggest you try starting with a seed and go through the instructions at www.kernel-seeds.org and see how close that comes to your ideal.

----------

## NeddySeagoon

Gear.0,

There is a lot of scope for error, even after you have configured an built your 'perfect' kernel. To name a few ...

1. forget to mount /boot

2. typo in destination file name in /boot

3. typo in grub.conf

Everyone messes up a kernel once in a while.

As well as the above, I've done a few silly things over the years.

1. Removed all partition table support (won't boot)

2. Omitted framebuffer and VGA console support .... works perfectly but the console is black text on a black background.

Some of the nasty surprises come as the kernel evolves and there is little time to react to these changes in such a tool.

Again a few examples,

1. The default configuration had PCI off ... everyone needs that

2. The SATA low level drivers was moved - make oldconfig did not catch that

3. ATA SFF support was added, in the default state of off, so a lot of SATA drivers appeared to vanish.

Its very rarely that you configure an entire new kernel.  Really only for a first install on new hardware.

The config files can be migrated from an old kernel to a newer one.

Copy the .config file over and run make oldconfig in the new kernel tree.

----------

## Gear.0

Well actually I didn't necessarily mean no typos or anything. I do like doing things manually, so goof ups are expected. I was just suggesting that there should be some method of going about it that is definitive.

I should probably point out that I'm a bit paranoid and that's probably why I'm having a hard time swallowing the fact that I might never have everything configured for my needs, perfectly optimized for me.

I'm also somewhat of an idealist. I find it very hard to settle for things.

Again, perhaps I'm just a nub, and I really do feel weird asking so many questions like this... But I just know that they will always be weighing on my mind, so I just have to know these things. I figure it's better that you all get frustrated with me now while I'm new here instead of later.

Please don't misunderstand. I still think custom kernels are a great idea even if it doesn't come out perfectly, and I love Gentoo and everything it stands for.

But, Let's say I use a kernel seed, and even spend hours and hours researching what to put in the kernel. Then I use that kernel for a while and realize I need something else, or that I can take something out.. Ignoring the fact that kernels get updated for now.

Even if I go about this for years, is there really any guarantee that I would have the best kernel for my needs and hardware?

See my point? I am willing to do the research and spend a huge amount of time on this, but (am I wrong?) it seems like there is absolutely no way, no matter how much time I dedicate to this, to ensure that I will ever have the kernel exactly the way I want it.

I'm just saying, even if there is a process that takes months, and is incredibly painstaking and requires you to compile several kernels and test each options individually...

although it would be somewhat impractical, at least it would be something that by the end of it you can say without a doubt that you have the kernel the way you want it, and there is absolutely nothing else you are missing, or have that you don't need.

It's like, if we say the finish line is having the "perfect kernel", there really is no actual direct set way to get there... you sort of just walk in the general direction.. but one thing that sort of highlights my point is, say you did actually get there... would you even know? You might know you are close to having your dream kernel because your system runs smoothly, but even if you did have the perfect kernel you still couldn't be 100% sure that you aren't missing some small little tiny improvement. (even though you aren't).

It just seems very imprecise to me.

----------

## NeddySeagoon

Gear.0,

We won't get frustrated - everyone here started sometime and we all remember it.

Keep in mind that there are no stupid questions, except the one you never asked.  Be on your lookout for stupid answers though. 

 *Gear.0 wrote:*   

> ... perfectly optimized for me.

  Thats the nub of the matter. Optimised means best compromise ... which is only something you can determine. A wee while ago I wrote this post about building kernels. Its still generally applicable. (OMG May 22, 2005)

The modular nature makes it easy to add things and remove things. e.g. you can build and load modules you found you need bit don't have and remove from memory, modules that are loaded but not needed. Then, on the next iteration of your kernel, you can remove things entirely, or add things as built in.

The use of kernel modules males it very easy to play, without breaking anything, once you have a working kernel.

----------

## BitJam

I think just about everybody here has a kernel with nothing they need missing.  If something doesn't work (usb drives for example) and the person can't figure it out for themselves, they post here and get help.

I also imagine most people have a few extra things in the kernel that they don't need.   Getting your kernel "perfect" be getting rid of every single thing you don't need is not a good use of time (unless it is something you enjoy).  The only downside to having something extra is the extra space it takes up (which is usually measured in K not in M) and a little extra time to compile the kernel.   Remember what Voltaire said: *Quote:*   

>  Perfect is the enemy of good.

 

Instead of more theorizing, I again suggest that you jump in and make a kernel starting from one of Pappy's seeds.  Once you've done that, you will be in a much better position to discuss perfect kernels.

----------

## krinn

 *NeddySeagoon wrote:*   

> 
> 
> As well as the above, I've done a few silly things over the years.
> 
> 

 

I don't trust that.

----------

## NeddySeagoon

Moved from Dustbin to Gentoo Chat.

Oops - I clicked on my 'Dustbin' button, instead of reply to.  I hope I've put the thread back in the right forum.

Now what were we saying about doing silly things ?

----------

## NeddySeagoon

Moved from Gentoo Chat to Kernel & Hardware.

Thats where it started

----------

## Gear.0

 *BitJam wrote:*   

> Instead of more theorizing, I again suggest that you jump in and make a kernel starting from one of Pappy's seeds.  Once you've done that, you will be in a much better position to discuss perfect kernels.

 

Perhaps that's the problem. I can't do anything for about 2 weeks as I am out of town.

I guess I'm just getting restless and over thinking things. I guess I'll try to be more patient    :Surprised: 

But I thank you all for being patient with me and giving such high quality responses.

----------

## bus_drivr

Could add to some of the above good info

I've used both grub and lilo and if your Gentoo > /boot/vmlinuz and Gentoo.old > /boot/vmlinuz.old

you can issue 'make install' and it will replace /boot/vmlinuz then for you. Saves some errors with typing  :Wink: 

Recently had bad luck making kernels and putzed 3 times....made a new one Gentoo works> /boot/vmlinuz.wrk

Where vmlinuz.wrk was a known good kernel ...

got to try the make targets mentioned above. But doesn't just 'make' make all those anyway  :Wink: 

for me it's

cd /usr/src

cp linux/.config /new/.config

'eselect kernel list' then 'eselect kernel set (new)'

cd new && make oldconfig

make menuconfig #and just check for new stuff/gotcha's nake sure support for the root filesystem's

hardware is compiled in (see SFF comment above-smh)

make && make modules_install

mount /boot

make install

dolilo

<ctrl-alt-del>

----------

