# (yet another) Gentoo search engine

## Zougloub

Some months ago I asked on the gentoo channels on IRC, how to look for a package in an overlay you don't have installed.

Or how to search for a package whose name is unknown, in an unknown overlay.

A generous soul introduced me with the command update-eix-remote, a new feature of recent eix versions.

But it wasn't quite what I wanted or imagined.

As I had nothing to do I decided to make a portage indexer, with the information of all overlays.

The thing is on http://Gentoo.zapto.org.

EDIT : http://Gentoo.zougloub.eu.

Some features were added over the time, such as reverse dependencies display and search, regexp search, firefox search plugins, overlay information, and an open wiki for comments on packages, overlays, and this website of course.

If you own an overlay, feel free to tell me, and I'll reference it.

I'll be pleased to receive feedback.

Play with it and ask features you want  :Wink: 

Thank you !Last edited by Zougloub on Tue May 04, 2010 5:11 am; edited 4 times in total

----------

## gimpel

Great stuff!

Long awaited something like that.

----------

## golding

 *gimpel wrote:*   

> Great stuff!
> 
> Long awaited something like that.

 

There was something similar a couple of months ago, but it only lasted a week.  May this example (which works quite well) last a little longer.

The search for particular packages is what I have been waiting for, a long time in fact.  I don't know why the Gentoo Overlay crew haven't had the search incorporated...

BTW gimpel, what happened to your site yesterday?  It was down until this morning (my time AU).  I was trying to add your overlay for the google-desktop.    :Very Happy: 

----------

## Zougloub

 *golding wrote:*   

> 
> 
> There was something similar a couple of months ago, but it only lasted a week.  May this example (which works quite well) last a little longer.
> 
> 

 

Fortunately I can keep this site running  :Wink: 

After a few days of testing on my poor laptop, it will move to a "true" web server (keeping its current URL, thanx noip).

Data will be updated every day, as it is now (I can decrease the frequency, but I see no need to increase it).

At the moment I have time and can implement nearly whatever people ask me on this site.

Later, I might not have the time to modify it that easily.

I have a lot of visits and a few comments. Don't be shy  :Exclamation:  and give feedback if you want more.

I can get very busy (or lazy) and can't be involved on a regular basis with projects such as Gentoo ; I don't have ties with gentoo devs, forum or IRC mods, and keeping this thread running might be the only way to keep the site visible.

 :Idea:  Before asking for features, notice that I made this because I wanted to answer basic "end-user questions" such as : «what is the name and where can I find the package doing...», «what are its dependencies», «how many packages are there in Gentoo», and that there are some things that can't be done (mostly what [url]packages.gentoo.org[/url] does : new packages information, link to bugzilla threads, etc.) and I don't want to do either, because it is done very well or is not the purpose of this site.

 :Idea:  I can't remember when I first heard of gentoo overlays.

I think mentionning them (a good paragraph) in the gentoo handbook would be a good idea...

----------

## swimmer

Can you index the package *lists* as well so that I can search to which package a file belongs?

This would be a great feature  :Smile: 

Greetz

swimmer

----------

## likewhoa

 *swimmer wrote:*   

> Can you index the package *lists* as well so that I can search to which package a file belongs?

 

while that being a great feature it would require every single package to be installed on the server and this is something that is easier said than done. on the other hand, some of the features i would like implemented are.

 filter by keywords dropdown. i.e ~x86 only. 

 filter by single or multiple category types dropdown.

 filter by single or multiple USE flags textbox.

 shown results in compact mode checkbox. i.e just show "package full name and version" i.e xorg-x11-7.3

 allow results to come from select overlays or filter by no overlays.

 sort by: version,overlay,name,keyword & popular (log hits on package views.)

 results should include "License, & USE flags" as an option to search results or on them in addition to what's already showing.

That's all for now. Thanks for contributing, and if you need hosting let me know, I'll put you on a Gentoo ded serv running lighty.

likewhoa

----------

## Zougloub

swimmer : as likewhoa said, this feature can't be implemented fully.

( for that I'd ask people in #gentoo to run qfile, it would be simpler   :Confused:  )

Even if likewhoa provided me a dedicated box to spend its life compiling stuff (every version of every package) and storing the CONTENTS file, it would have to use default use flags, (I don't think every version of every package with every use flags combination can be reached) and a fixed architecture.

It would limit the accuracy of the file listings.

What could be done too is using the contents of the binary packages that some guys gracefully provide.

I'm not very motivated to do any of these ; the target audience is IMO very limited.

likewhoa : suggestions noted, some are immediate and are effective (License, USE) ; some don't fit the current design of the site and will be implemented later (ie. not tonight  :Very Happy:  ).

----------

## swimmer

@Zougloub: Don't bother - it was just wishful thinking  :Wink: 

But thanks for your great work in aany case!!!

Greetz

swimmer

----------

## oshecho

Sweet awesomeness. 

I love it.

----------

## Zougloub

The site has moved to its definitive location  :Smile: 

Service will be interrupted for a pair of minutes.

yay  :Wink: 

----------

## kernelOfTruth

 *gimpel wrote:*   

> Great stuff!
> 
> Long awaited something like that.

 

++

love it,

there was a replacement for packages.gentoo.org while it was offline but wasn't up too long

hopefully your site / search will stay up longer   :Smile: 

----------

## zixnub

++

good innitiative, I'll be sure to check it out.

----------

## wolfden

You can add a firefox search plugin for the site too

```
<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/" xmlns:os="http://a9.com/-/spec/opensearch/1.1/">

<os:ShortName>Gentoo Packages</os:ShortName>

<os:Description>Gentoo Packages</os:Description>

<os:InputEncoding>UTF-8</os:InputEncoding>

<os:Image width="16" height="16">data:image/x-icon;base64,AAABAAEAEBAAAAEACABoBQAAFgAAACgAAAAQAAAAIAAAAAEACAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAACXf4IApYGDAJaChQCsiIkAp4mLALGKkgC0jI4AroyRALWNkQCikI8AuI2SAKKQkwCjkJQAn5GVALqSkQDBk5EAq5aYAKyXmQDDlZgAvZaXALyXmwCzmZ0Aw5q

cAMaeoQCtoaYAzaCkALmkpwCwpagAraanAMympADHpaoA1qajANGqqgDQq6oA4qurAPeprAD3qq0A96utAPSssADksrIA5bKyAN2xuQD4sbMA2LS1ANi4tgD/tbUA/rW2AP6

3twD1uLsA+7i5APu5uQD+uboA47rAAP27vADevr4A27+9APy8vADZv8QA/r6/APO/wQDlv8YA57/GAPTAwwD+wMEA9cDGAP3BwAD1wsIA/cPDAObGxQD9xcUA/sXGAP7G

xQDwxsoA5MnFAP3HxgD9yMYA08vNAOzI0ADnys8A88nSANbNzwD9y8wA9MvQAP7MzQDoz8wA887QAPTP0gD8z9AA6tLSAPzR0AD60dEA/tHSAO3T2AD+09QA7dPaAPLV1

gD+1NYA4tfcAP3W1wD92NkA5NvdAPTb2gD+2tsA/dvcAPXc3QD03N8A8NzhAP3d3gDv390A/t7fAPbf3wD+398A99/iAPrg4AD+4OEA/OHiAPzh4wD64+MA9+PnAPvj5gD95O

UA8ObnAPzl5wD+5egA9efmAPfm6QD+5+gA++joAP7o6QD96OoA9OroAPvp6QD66ukA8OvtAPnr7AD96+wA/evtAPfs7QD+7O4A++7uAPbu8AD+7u8A/+7vAP7v8QD/7/EA

+/DyAP3x8AD88vIA/vLzAPrz8gD49PUA+/T2APn19QD69fUA/fb3AP739wD/9/gA+fj4APv4+AD9+PgA+vn6AP75+QD++foA/Pr7AP37+wD++/wA+/z8APz8/AD8/P0A/fz9AP

v9/QD8/f0A/P3+AP79/gD//f4A/P79AP3+/QD7/v4A/v79APz+/gD9/v4A/v7+AP/+/gD9/v8A/v7/AP/+/wD9//4A/v/+AP3//wD+//8A////AAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAvhsNCwwYTJ24vr6+vr6+vhUWKCcfDwMQUKe+vr6+vr4ZW

mdtcnNYIQYRYb6+vr6+KXqHjpCNiItsIAEavr6+vjx9prKupZyRipJUDge+vr6gTpa0uL62oY+Ag3wdBb6+uKhcjLe6va6Ufm9rhCwIvr65rDl5sb26mntmXVd1NxSwhRwAF5i8t5t4

YlFHQ3ErZAITSZW0r6STdFtFOjUyOx42mbS+qlYSCl9TPzMuJG40nre9vYYiCQRlSjgtJVK1Xmqzvr2jgkRoWUEvI1Wtvqk9ibW7vaJ/YEYxJnCzvr6+vk1pp5+BY0sqPpe4ub69vri0

dk9IQjBAd6u+vb6uvgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=</os:Image>

<os:Url type="text/html" method="GET" template="http://gentoo.zapto.org/packages/search?description={searchTerms}"></os:Url>

</SearchPlugin>
```

copy the above and call it gentoo-packages.xml and put it in your firefox/mozilla  searchplugins folder than restart the browser and you should now have a new search engne to select

mod edit: fixed huge line --bunder

----------

## easy target

Nice :-)

I agree with likewhoa's features request and also would like to ask for placing a search form in results page so that it wouldn't be necessary to go back to main page to try a new search :-).

Also, packages coming from overlays should have additional info, e.g. "from java-experimental overlay".

 *swimmer wrote:*   

> Can you index the package *lists* as well so that I can search to which package a file belongs?

 

Try this

regards

----------

## swimmer

Ah nice  :Smile: 

Thank you very much for the link!

Greetz

swimmer

----------

## Zougloub

Thanks wolfden for your search plugin.

I'm putting this link on teh site.

easy target : packages coming from overlays don't belong to the «gentoo» overlay.

But yeah I'll try to make something more visible.

----------

## Zougloub

I added reverse dependencies information for packages, since a friend asked it, and it wasn't very long to implement.

When browsing a package, a link for reverse dependencies is displayed.

By going to the main page : http://Gentoo.zapto.org/ and if you use Firefox, you'll find the search engine "plugins" for the normal search by description, and the one for reverse dependencies.

I only provided the searchplugin for Firefox, but If someone is willing to translate it for other browsers, feel free to go  :Wink: 

----------

## easy target

 *Zougloub wrote:*   

> Thanks wolfden for your search plugin.
> 
> I'm putting this link on teh site.
> 
> easy target : packages coming from overlays don't belong to the «gentoo» overlay.
> ...

 

Thanks for info about overlays :-).

Some other requests:

 Link to changelog

 I think that "Link" title should be "Ebuild", because it is a link to ebuild.

 Tree timestamp (like in packages.gentoo.org) also would be useful (how often do You sync?)

Thanks in advance :-).

----------

## Zougloub

Hi easy target,

 - I renamed "Link" to ebuild following your advice.

 - I wrote "Today xxx there are nnn packages" which is an indirect way to say that packages info is updated daily  :Wink: 

 - I'll do the changelog link later, I'm busy today   :Embarassed: 

Thanks for your suggestions !

I'm making a wiki that should be linked with the packages information... I don't know if it will be used,  but I did write some information in it and thought I could share it on g.z.o.

----------

## easy target

 *Zougloub wrote:*   

> 
> 
>  - I wrote "Today xxx there are nnn packages" which is an indirect way to say that packages info is updated daily ;)
> 
> 

 

Yeah, I've seen it, but didn't think it was so 'indirect' ;-).

----------

## ryker

Not a very pretty website but it is very functional and convenient.  Thank you for creating it.

----------

## alphonce

Nice engine.

But I think it would be better if you showed the packages in reversed order, ie the latest first, because it's often those you want to find.

THanks for a great site.

----------

## Zougloub

Listing order has been "corrected" ; I think everybody would agree with alphonce (and I feel dumb to not having done so at first   :Embarassed: ).

...and I corrected a stupid bug happening when searching for a package in a specific category (bad package url)

----------

## Zougloub

Downtime   :Rolling Eyes: 

There was a power outage this afternoon in the building hosting the server.

Unfortunately the students that were in charge of the network maintenance are on holidays and left with the keys.

And I'm 8000km away.

The server will be back on tomorrow morning (~9:00 UTC).

----------

## Zougloub

Tonight I made some arrangements to have the site display nicely... with links  :Twisted Evil: 

For those who could be interested, the site also has :

 - gentoo man pages (see this topic)

 - a pastebin at http://Gentoo.zapto.org/p/ whose submissions will stay forever (will be removed if you ask).

The wiki is now linked, but information is very sparse. One day I'll check if pages are present before linking them in the search engine pages...

----------

## Zougloub

I've worked a lot on the site yesterday and there were times it was unusable.

It might not be very visible since I was enhancing the structure, but I added an overlay information page, and modified the ebuild parsing to get more information from them.

If you own/know an unofficial overlay and want it referenced, you can send me the layman xml for it (using jabber, IRC, gzo's wiki or here, but here would be noisy).

----------

## Zougloub

I'm rewriting some indexer stuff to benefit from Paludis' features (EAPI stuff, cleaner tools, speed), and make the site more efficient.

----------

## think4urs11

a RSS-Feed would be fantastic

----------

## Zougloub

It's g.z.o's anniversary   :Smile: 

I did some small updates, nothing transcendant and the site is still as ugly as always... but the service is here, so I don't care.

Since I am pulling all the repos, I figured that I could make the data public, so for those who don't want to have all the scms to sync, they can use rsync to sync on g.z.o (see /repos FYI).

Think4UrS11, I'll look into that some day... currently I'm not convinced of the utility of the feature...

I'm testing something quick and dirty with diff-eix right now (there is a public /repos/eix.cache file and now diff-eix output will go to /diffs/).

----------

## alex.blackbit

Zougloub,

please update the text on the website, eix-sync runs layman -S for quite some time now.

----------

## Zougloub

I don't have a lot of time to work on gzo but yesterday night I took a pair of hours to perform some updates.

I updated the category listing, now it's generated from the contents of the portage tree.

If translations are sufficiently added, the site might get multilingual.

At the moment, I see no need to adapt it since only some categories are translated and usage of metadata.xml in packages is marginal.

When I logged on my server yesterday, I saw that I had some days of cron'ed indexing tasks performing at the same time, and the server was under heavy load.

FYI, there are currently 927MB of data on the repositories, and 187MB of caches.

I have not found the source of the lagging yet, I guess some bad ebuilds and bash bugs might cause some trouble...

Noticed that I forgot to have a robots.txt file, leading to unnecessary hits by Google and Cuil search engines.

----------

## MageSlayer

pastebin on site works, but no utf-8  :Sad: 

My Russian letters turn to hex. 

Thanks for effort

----------

## Zougloub

k-nopaste seems not to support UTF-8 out of the box, I quickly patched it to have it working (author notified)

++

----------

## MageSlayer

Still it does not work as expected.

Tested with Firefox and Opera

----------

## Zougloub

Works for me ; I posted a few pages in chinese and saw your "test" in russian.

----------

## MageSlayer

Well my Russian after posting is similar to Chineese, I would say  :Smile: 

That's how it looks before posting

http://img249.imageshack.us/img249/7883/zaptoo.gif

That's how it looks after posting

http://img12.imageshack.us/img12/17/zaptoo2.gif

----------

## MageSlayer

Now it works ok. Thanks

----------

## Zougloub

More than 2 years that the site is online now.

g.z.o was visited by 704 distinct IPs in december (excluding the search engine crawlers), that's not a lot but that's enough to consider that the site is helping people.

Although I don't really want to release the sources because I'm not very proud of that mess, I think I can share a few facts about the site.

Every day,

 - all the overlays are sync'ed using the good old layman (landing in http://gentoo.zapto.org/repos/)

 - the overlays list is transformed to build the http://gentoo.zapto.org/overlays/ page

 - overlays metadata is generated using Instruo (a Paludis tool)

 - the modified metadata are parsed, filling a SQLite3 database

 - an eix cache is built, and the 2 latest caches are diffed to produce the http://gentoo.zapto.org/diffs/ contents

The pages are served by mod_python; the code is quite simple, there is a database interface, something to produce the wanted HTTP, and something to parse the requested URIs.

I've made a C++ rewrite of the search backend, but since there is absolutely no overloading of the 1.2GHz CPU the site is running on with the current Python code, it's still in the drawers.

----------

## Alanw

This is a very nice site! 

The only thing I'd like to see is different color anchor tags <a />.  Trying to view orange over purple is very difficult. I have to select the text to read it.  Maybe a bold white, or deep purple would help?  

-Alan

----------

## Zougloub

The links styles were not specified.

Maybe your style makes them orange.

I specified them.

----------

## Alanw

 *Zougloub wrote:*   

> The links styles were not specified.
> 
> Maybe your style makes them orange.
> 
> I specified them.

 

Ah, you're right, orange is the default in FireFox.

----------

## Zougloub

Dear g.z.o users, please use gentoo.zougloub.eu instead of gentoo.zapto.org.

I'm tired of renewing my dyndns/no-ip memberships...

See ya !

----------

## Zougloub

 :Embarassed:  An horrible mistake had made its way during an update made some times ago to enhance the site performance (> 1 month...), rendering the site quite unusable.

Found about it some days ago but there was not much time to fix it... well, now it's fixed.

----------

