# HOWTO: Use MySQL as a DB backend to Portage

## arkane

I wanted to make sure this was in the tips'n'tricks section.  I didn't write this, however I did make it work following the instructions provided exactly.

This was done by OneofOne.

https://forums.gentoo.org/viewtopic.php?t=175461&postdays=0&postorder=asc&start=0

Here's a copy just to make sure it's documented here.

Instructions:

1. emerge mysql and configure it (if you haven't already)

2. emerge mysql-python

3. /etc/init.d/mysql start

4. put this code in a file and do mysql -u root -p < file.sql

Here's the SQL code to use (file.sql):

```

CREATE DATABASE `portage`;

USE mysql;

REPLACE INTO user (Host,User,Password) VALUES ('localhost','portage','');

FLUSH PRIVILEGES;

GRANT SELECT,UPDATE,DELETE,INSERT,CREATE,LOCK TABLES ON portage.* TO portage@localhost;

FLUSH PRIVILEGES;

USE portage;

CREATE TABLE `path_table` (

  p_id int PRIMARY KEY AUTO_INCREMENT,

  name char(255) NOT NULL UNIQUE

);

CREATE TABLE `category_table` (

  c_id int(7) PRIMARY KEY AUTO_INCREMENT,

  name char(255) NOT NULL UNIQUE

) ;

CREATE TABLE `package_name` (

  name char(255) NOT NULL,

  cat_id  int NOT NULL,

  path_id int NOT NULL,

  data TEXT,

  FOREIGN KEY (cat_id) REFERENCES category_table (c_id) ON DELETE CASCADE,

  FOREIGN KEY (path_id)  REFERENCES path_table (p_id) ON DELETE CASCADE,

  #INDEX p_index (name,cat_id,path_id),

  primary key (name, cat_id, path_id)

) MAX_ROWS=99999999; 

```

5. create /usr/lib/portage/pym/portage_db_mysql.py and put this in it : 

```

# portage_db_mysql.db by oneofone{-a-t-}limitlessfx.com

import MySQLdb,cPickle

import portage_db_template

ex = MySQLdb.escape_string

class database(portage_db_template.database):

   #ought to do ref counting on the number of open connections.  course that's worthless till portage calls database.close().

   connections = {}

   cursor = None

   path_map = {}

   categories = {}

   defaultOptions = {'host':'localhost', 'port': 3306,  'user': 'portage', 'password': "", 'db': 'portage'}

   lastkey = lastval = None # internal caching to make get_values() work a little fast.

   global ex

   def __init__(self,path,category,dbkeys,uid,gid,config_path='/etc/portage/module_configs/'):

      try:

         portage_db_template.database.__init__(self, path,category,dbkeys,uid,gid,config_path)

      except:

         self.path     = path

         self.category = category

         self.dbkeys   = dbkeys

         self.uid      = uid

         self.gid      = gid

         self.config = {}

         self.module_init()

   def __del__(self):

      try:

         self.close()

      except:

         pass

   def module_init(self):

      options = database.defaultOptions.copy()

      if self.config:

         for x in self.config.keys():

            options[x] = self.config[x]

      self.config = options; del options

      from re import sub

      path = sub('/{2,}', '/', self.path)

      self.constr = self.config['host'] + ':' + str(self.config['port']) + '/' + self.config['db']

      if not database.connections.has_key(self.constr):

         try:

            con =  [0, MySQLdb.connect(host=self.config['host'], port=self.config['port'], user=self.config['user'], passwd=self.config['password'], db=self.config['db']) ]

            database.connections.setdefault(self.constr, con)

         except Exception, e:

            raise Exception, "Error connecting to Database using self.config=(%s): exception=(%s)" % (str(self.config), str(e))

      self.con = database.connections[self.constr][1]

      self.db = database.connections[self.constr][1].cursor()

      database.connections[self.constr][0] += 1

      

      try:

         self.db.execute('set AUTOCOMMIT=1')

         if len(database.categories.keys()) == 0:

            self.db.execute('select name,c_id from `category_table`')

            for y, x in self.db.fetchall():

               database.categories[y] = x

      except Exception, e:

         self.check_exception(e)

         raise Exception, "Database Error: failed pulling tables using self.config=(%s), exception=(%s)" % (str(self.config), str(e))

      

      if  not database.categories.has_key(self.category) :

         #create the category table

         try:

            self.db.execute('INSERT INTO `category_table` (`name`) VALUES("%s")' % ex(self.category))

            database.categories.clear()

            self.db.execute('select name, c_id from `category_table`')

            for y, x in self.db.fetchall():

               database.categories[y] = x

         except Exception, e:

            self.check_exception(e)

            raise Exception, "Database error: failed creation table for path('%s'), category('%s'), exception=(%s)" % (self.path, self.category, str(e))

      self.cat_id = database.categories[self.category]

      

      if not database.path_map.has_key(self.path):

         try:

            if len( database.path_map.keys()) == 0:

               if self.db.execute('SELECT name, p_id FROM `path_table` ') > 0:

                  for x,y in self.db.fetchall():

                     database.path_map[x] = y

         except Exception, e:

            self.check_exception(e)

            raise Exception, "database error: %s" % str(e)

         

         if not database.path_map.has_key(path):

            try:

               self.db.execute('INSERT INTO `path_table`(`name`) VALUES("%s")' % ex(path))

               #flush the table, and load it anew

               database.path_map.clear()

               self.db.execute('SELECT name, p_id  FROM `path_table`')

               for path,id in self.db.fetchall():

                  database.path_map[path] = id

            except Exception, e:

               self.check_exception(e)

               raise Exception, "Database error, failed loading path map: exception=%s" % str(e)

      self.p_id = database.path_map[path]

      

   def check_exception(self, e):

      if "Got error 127 from table handler" in str(e):

         if self.db.execute('check table `package_name`') > 1:

            print '`package_name` got corrupted some how, trying to fix'

            self.db.execute('repair table `package_name`')

            check = self.db.execute('check table `package_name`')

            status = self.db.fetchall()

            if ( check == 1 and status[0][3] == "OK"):

               raise Exception, "Database was corrupt be we were able to repair it, please rerun the emerge command"

            else:

               raise Exception, "Database is corrupt, exception=(%s)" % str(e)

      

   def has_key(self,key):

      self.check_key(key)

      try :   

         #self.db.execute('REPAIR TABLE `package_name`')

         return self.db.execute("SELECT data FROM `package_name` WHERE `name` =  '%s' AND `cat_id` = '%d' AND `path_id` = '%d'" %  (ex(key), self.cat_id, self.p_id )) > 0

      except Exception, e:

         self.check_exception(e)

         print "exception in has_key for Key (%s), Error (%s) " % (key, str(e))

         return False

      

   def keys(self):

      ks = {}

      try:

         self.db.execute("SELECT name, data from `package_name` WHERE `cat_id` ='%d' AND `path_id` = '%d'" % (self.cat_id, self.p_id ))

         for x,y in self.db.fetchall():

            ks[x] = cPickle.loads(y)

         return ks

      except Exception, e:

         self.check_exception(e)

         raise KeyError, "Keys are dead :( (%s)" % str(e)

   

   def get_values(self,key):

      self.check_key(key)

      if self.lastkey == key:

         return self.lastval

      try:

         if self.db.execute("SELECT data FROM `package_name` WHERE `name` =  '%s' AND `cat_id` = '%d' AND `path_id` = '%d'" %  (key, self.cat_id, self.p_id )) > 0:

            one = self.db.fetchone()

            if len(one) > 0 and one[0] != None:

               self.lastkey = key; self.lastval =  cPickle.loads(one[0])

               return self.lastval

            return None

      except Exception, e:

            self.check_exception(e)

            raise ValueError, "Value error (%s)" % str(e)

      return None

   

   def set_values(self,key,val):

      self.check_key(key)

      try:

         if self.lastkey == key: self.lastkey = self.lastval = None

         self.db.execute("""REPLACE INTO `package_name` (`name`,`cat_id`, `data`, `path_id`) VALUES ('%s','%d', '%s','%d') """ %  (ex(key), self.cat_id, ex(cPickle.dumps(val)),self.p_id ) ) 

      except Exception, e:

         self.check_exception(e)

         raise Exception, "Error inserting/updating the database (%s)." % str(e)

         

   def del_key(self,key):

      try:

         if self.lastkey == key: self.lastkey = self.lastval = None

         return (self.db.execute('DELETE FROM `package_name` WHERE `cat_id` = "%d" AND `name`= "%s" AND `path_id` = "%d"' % (self.cat_id, key, self.p_id)) > 0)

      except Exception, e:

         self.check_exception(e)

         print "Error deleting key=(%s), exception= (%s)" % (key,str(e))

         return False

         

   

   def sync(self):

      pass

   

   def close(self):   

      if database.connections.get(self.constr, [0])[0] == 0: return 0 # make sure that .close() doesn't get called twice

      try:

         #print 'in self.db.close(), instcount=%d' % database.connections[self.constr][0]

         database.connections[self.constr][0] -= 1

         self.db.close()

         if database.connections[self.constr][0] == 0:

            #self.db.execute("analyze TABLE `category_table` , `package_name` , `path_table`") # optimize the db on close

            database.connections[self.constr][1].close()

            del database.connections[self.constr]

            return 1

      except Exception, e:

         self.check_exception(e)

         print "Exception in self.db.close() == (%s), ignored." % str(e)

      

```

6. Create /etc/portage/modules and add :

```

portdbapi.auxdbmodule="portage_db_mysql.database"

eclass_cache.dbmodule="portage_db_mysql.database" 

```

7. emerge regen (you dont have to, it'll update the cache when you do emerge -s or something)

8. enjoy!

btw portage won't work if unless mysql is started (of course you can always delete /etc/portage/modules and it'll use the default flat file db system).

This is provided with no warrant, if the computer blows up DON'T BLAME ME!

Credits to everyone on #gentoo-portage (specially carpaski and ferringb) for helping me optimize it and to joecool and redeeman (#love-sources) for testing.

the module : http://oneofone.limitlessfx.com/gentoo-stuff/portage_db_mysql.py

the sql file : http://oneofone.limitlessfx.com/gentoo-stuff/init-portage-db.sql

----------

## chipi

Neat trick! thanks.

----------

## John5788

does this make portage faster using a mysql db?

----------

## AllTom

 *John5788 wrote:*   

> does this make portage faster using a mysql db?

 

I just made the switch and verified that it was working by stopping the MySQL server and seeing what complained.

On my AMD64 3000+ (in 32-bit Linux) with 512 MB RAM I have not noticed any great speed increase, if any at all. Searching seems to take the same amount of time (i.e. forever). The only difference I can see is that now portage requires a MySQL server to be running to work at all. Not worth the trouble, in my opinion.

----------

## Torin_

Hi. I have done all with instructions.

And i've got this errors ? Anybody knows what's are these?

```

root@deception /usr/lib/portage/pym # emerge regen

Traceback (most recent call last):

  File "/usr/bin/emerge", line 14, in ?

    import portage

  File "/usr/lib/portage/pym/portage.py", line 6141, in ?

    portdb=portdbapi(settings["PORTDIR"])

  File "/usr/lib/portage/pym/portage.py", line 4424, in __init__

    self.eclassdb = eclass_cache(self.mysettings)

  File "/usr/lib/portage/pym/portage.py", line 4326, in __init__

    self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")

  File "/usr/lib/portage/pym/portage.py", line 1305, in load_best_module

    return load_mod(best_mod)

  File "/usr/lib/portage/pym/portage.py", line 34, in load_mod

    mod = __import__(modname)

  File "/usr/lib/portage/pym/portage_db_mysql.py", line 2, in ?

    import MySQLdb,cPickle 

ImportError: No module named MySQLdb

```

I have password for root in mysql is this it ?

----------

## xces

 *Torin_ wrote:*   

> Hi. I have done all with instructions.
> 
> And i've got this errors ? Anybody knows what's are these?

 

You probably left out 'emerge mysql-python' since the error says it couldn't find the appropriate module.

----------

## Torin_

i'm so stupid.

i'm going sleep.

thx for help  :Smile: 

----------

## AllTom

 *AllTom wrote:*   

> On my AMD64 3000+ (in 32-bit Linux) with 512 MB RAM I have not noticed any great speed increase, if any at all.

 

Actually, I'd like to know if anyone else can verify this. Should using MySQL be speeding things up? Even a little bit?

----------

## chipi

Actually, after giving it some tests - I too can't notice any speed improvments at all  :Sad: 

----------

## AllTom

One would think that the whole job of searching would be passed onto MySQL but I guess that is not the case...

----------

## bravecobra

 *AllTom wrote:*   

> Searching seems to take the same amount of time (i.e. forever).

 

Try esearch

----------

## AllTom

 *bravecobra wrote:*   

>  *AllTom wrote:*   Searching seems to take the same amount of time (i.e. forever). 
> 
> Try esearch

 

I would, but now that I've started using packages.gentoo.org and gentoo-portage.com I haven't needed to search from the command line from a while.  :Smile:  Thanks, though.

----------

## msimplay

what are the benefits to doing this modification anyway ?

----------

## Torin_

Try doing emerge -s something.

In my opinion it speeds the operation many times.

----------

## AllTom

 *Torin_ wrote:*   

> Try doing emerge -s something.
> 
> In my opinion it speeds the operation many times.

 

That's the thing. I haven't noticed any difference. Perhaps this only benefits slow(er) computers?

----------

## arkane

 *AllTom wrote:*   

>  *Torin_ wrote:*   Try doing emerge -s something.
> 
> In my opinion it speeds the operation many times. 
> 
> That's the thing. I haven't noticed any difference. Perhaps this only benefits slow(er) computers?

 

I have it running on my 1.8ghz P4 (512mb ram), and my 700mhz P3 laptop.  It speeds up the emerge -s searches by a large margin.

It's worth trying out.  I'd give it a serious workout.. first doing the "emerge regen" to regenerate all the stuff.

do a "time emerge -s perl" with and without this alteration.

----------

## AllTom

 *arkane wrote:*   

>  *AllTom wrote:*    *Torin_ wrote:*   Try doing emerge -s something.
> 
> In my opinion it speeds the operation many times. 
> 
> That's the thing. I haven't noticed any difference. Perhaps this only benefits slow(er) computers? 
> ...

 

Well I followed the directions exactly and verified that portage is, in fact, using MySQL. I did the following test and, as you can see, I didn't notice any difference.

# emerge -s perl

# time emerge -s perl

real    0m2.398s

user    0m1.645s

sys     0m0.128s

# mv /etc/portage/modules /etc/portage/modules-goaway

# /etc/init.d/mysql stop

# emerge -s perl

# time emerge -s perl

real    0m2.344s

user    0m1.672s

sys     0m0.138s

So then I tried using -S for searching the descriptions (which takes a lot longer). This was right after the previous test with nothing else running. First was without MySQL, then with MySQL.

# emerge -S perl

# time emerge -S perl

real    1m16.150s

user    1m8.357s

sys     0m4.828s

# mv /etc/portage/modules-goaway /etc/portage/modules        

# /etc/init.d/mysql start

# emerge -S perl

# time emerge -S perl

real    1m24.071s

user    1m9.615s

sys     0m4.243s

So what can I conclude from this? That I did something wrong? That reading from my filesystem is faster than MySQL? That using MySQL as a backend slows things down?

If I were one of the people to notice a speed-up in searching, would that also speed up calculating dependencies? That's really what I want this for. emerge world -vuDp takes a while.

----------

## Lews_Therin

Out of curiosity...if the mysql server was running on another (faster) machine, could this be used to greatly increase the speed of portage on a slow machine? For example, on my desktop "emerge -pv $PACKAGE" is done before I can respond, but takes >15 seconds for the same package on my server (P166). Would creating a mysql server on my desktop and hooking this script up to it on my server increase performance?

----------

## duderonomy

I would think that the time spent crunching in python would be the 

bottleneck... Does anyone know how to profile this code in python? 

That would certainly yield the information we are interested in.  :Smile: 

Cheers,

D

----------

## NewBlackDak

I just tried this on my LatitudeCPiA(PII-400, 192MB Ram), and my desktop rig(see sig).  

What I noticed is on the desktop it seems unchanged.

On the laptop(very slow disk, 4200RPM) using MySQL is faster.

 *duderonomy wrote:*   

> I would think that the time spent crunching in python would be the 
> 
> bottleneck... Does anyone know how to profile this code in python? 
> 
> That would certainly yield the information we are interested in. 
> ...

 

This is prolly part of it.  If I actually search the DB it is MUCH FASTER than doing an emerge -S.  If the portage code was updated then this could make searching portage act like a whole new animal.

----------

## chratnox

First off, very very nice! Speeds up emerge sync by about 3 minutes on my box (P2 400 MHz, 256mb ram.. ata66 hdd, mysql running on an ata133 controller)

However, there's a small problem with equery it seems, haven't tried the other commands

```
 # equery size gcc

* sys-devel/gcc-3.3.4-r1

           Total Files : 474

           Total Size  : 29301.69 KiB

Exception exceptions.AttributeError: "'NoneType' object has no attribute 'close'" in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor instance at 0x406c912c>> ignored

Exception exceptions.AttributeError: "'NoneType' object has no attribute 'close'" in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor instance at 0x4062ce0c>> ignored
```

Dunno if it's anything serious as of calling it 'problem' though

Running dev-db/mysql-4.0.20-r1, sys-apps/portage-2.0.50-r10

----------

## MooktaKiNG

The only thing i hate about portage is the whole emerge sync bit.

The blasted "Updating cache" is bloody annoying. It takes forever.

Now, if MySQL can sort out the emerge sync speed. Then that would be a great improvement.

Everything else is fine about portage. I use esearch, and i also don't mind the speed of emerge -s.

----------

## NewBlackDak

It's the downloading that takes the time.  

I think it would be really awesome if there were no emerge sync.  You emerge -p system, it hits your favorite mirror to get a list of the most recent packages, and downloads every thing it needs then.

----------

## MooktaKiNG

 *NewBlackDak wrote:*   

> It's the downloading that takes the time.  
> 
> I think it would be really awesome if there were no emerge sync.  You emerge -p system, it hits your favorite mirror to get a list of the most recent packages, and downloads every thing it needs then.

 

There's a lot of security issues with that.

And you wont be able to do emerge -s, and other search stuff.

You will be a good connection. The server will have to be VERY GOOD, since the whole world is gonna use it.

----------

## eGore911

Two things:

1.) emerge sync is not slow because of downloading, i think. I sync over a 100 mbit lan. The issue is the reading from and writing to the fs. So this MIGHT be an improvement using mysql.

2.) I didn't look at the script, so I might be wrong: Moving portage to a DB is worth nothing. You have to build a database concept around that, optimized for searching. This would just cause that it's not save in the DB of the FS, but in another one, nothing else. But since I'm not a database expert, I'd better shut up.

I'd leave it as it is for now.

----------

## chratnox

 *eGore911 wrote:*   

> Two things:
> 
> 1.) emerge sync is not slow because of downloading, i think. I sync over a 100 mbit lan. The issue is the reading from and writing to the fs. So this MIGHT be an improvement using mysql.
> 
> 2.) I didn't look at the script, so I might be wrong: Moving portage to a DB is worth nothing. You have to build a database concept around that, optimized for searching. This would just cause that it's not save in the DB of the FS, but in another one, nothing else. But since I'm not a database expert, I'd better shut up.
> ...

 

Emerge sync indeed takes a while because of the reading/writing yes, it only downloads approx. 7mb, so that's shouldn't be such a bottleneck. However the reading and writing does speed up through database (from 7 minutes for syncing I went to 3.30 minutes for syncing.. So it does speed up  :Smile: ). Also, searches are alot faster.. Took about 30 seconds, now takes ~10 seconds.

I'm still struggling with that error with equery, I hope It'll get some feedback soon.

----------

## MooktaKiNG

 *chratnox wrote:*   

>  *eGore911 wrote:*   Two things:
> 
> 1.) emerge sync is not slow because of downloading, i think. I sync over a 100 mbit lan. The issue is the reading from and writing to the fs. So this MIGHT be an improvement using mysql.
> 
> 2.) I didn't look at the script, so I might be wrong: Moving portage to a DB is worth nothing. You have to build a database concept around that, optimized for searching. This would just cause that it's not save in the DB of the FS, but in another one, nothing else. But since I'm not a database expert, I'd better shut up.
> ...

 

Is there a speed increase in Updating Cache too?

----------

## Nutterpc

after having got Mysql backend support for portage setup & working correctly, I've rerun regen twice since then......here's roughly how long it went each run

1st run: 5mins or so

2nd run: 1min48sec

3rd run: 

real    0m34.952s

user    0m14.053s

sys     0m1.803s

as u can see, nuff said  :Smile: 

----------

## MooktaKiNG

One more thing...

.. Does this mean i dont need /usr/portage anymore?

I hope thats the case  :Smile: 

----------

## geekmug

I wanted to also note that using the anydbm module got me a doubling of speed for my rather slow (k6-2 500Mhz) machine. I imagine it is largely to do with the anydbm module not being nearly as expensive to manipulate. Didn't try the mysql module.. I don't like it. I'm sure you get similar results, and I'd rather use a module that will be updated along with future portages.. what happens when the portage cache structure changes.. will you remember the cheap hack and who to complain to get an update? I'd rather not worry about it.

----------

## MooktaKiNG

 *geekmug wrote:*   

> I wanted to also note that using the anydbm module got me a doubling of speed for my rather slow (k6-2 500Mhz) machine. I imagine it is largely to do with the anydbm module not being nearly as expensive to manipulate. Didn't try the mysql module.. I don't like it. I'm sure you get similar results, and I'd rather use a module that will be updated along with future portages.. what happens when the portage cache structure changes.. will you remember the cheap hack and who to complain to get an update? I'd rather not worry about it.

 

Thats true.

So how do we setup the anydbm module?

----------

## JimmyW

Is there not a point rather using SQLite, than a multiuser big database like mysql, that requires an own process laying in background, whith network support, support for authentications, and lots of lots,  I think SQLite is the way to go, its fast and scaleable, and i think there are python wrappers.

/ Jimmy.

----------

## geekmug

To use the anydbm module you just do the last step there..

Create /etc/portage/modules:

```
portdbapi.auxdbmodule="portage_db_anydbm.database"

eclass_cache.dbmodule="portage_db_anydbm.database"
```

Again, a "emerge regen" to spend the time now or don't and spend it on the next use of the portage system. On my box, I watched it for a long time, and then decided to go to bed, so I can't say how long that takes, but it is definitely at least an hour on my k6-2 500Mhz.

----------

## Wilhelm

Is it me or is it possible to remote connect to the mysql db using portage by modifying the python file? I'd like to see my pc's all connecting to my central portage db without the excess crap.

Another thing that disturbed me was the following 

```

REPLACE INTO user (Host,User,Password) VALUES ('localhost','portage',''); 

```

This means no password is used allowing anybody with bad intentions to use your portage to emerge a rootkit or what have you. Mysql can be accessed from any username even a guestaccount by just doing a "mysql -u portage -p portage".

Might want to stress people that they should password it by modifying th python script and the SQL statements.

----------

## taveren

 *Quote:*   

> 
> 
> This means no password is used allowing anybody with bad intentions to use your portage to emerge a rootkit or what have you. Mysql can be accessed from any username even a guestaccount by just doing a "mysql -u portage -p portage". 

 

I agree.  Please, at least set a password in the instructions.  Better yet, make a step saying "Replace any instance of $PASSWORD with a randomly generated password."  You can specify the password in the python script so it should work just fine.

Has anyone tried using a single MySQL database for several machines?  I will try as soon as I can find the time.  If this leads to only having to "Emerge sync" once, that would be most helpful.

----------

## Wilhelm

I'm on it too  :Wink: 

I'll report back if it works. 

MyTHEORY: MySQL only allows reading of the database by users and no updating so there is no need for locking or mutual exclusions. 

Locking or mutual exclusions should only be nessecary for updating but since the system itself updates and no multiple updates should occur it should work.

In the unlikely event that an ebuild would be partially put into the DB (which should't happen due to monotonic transactions) the ebuild would just fail and no harm is done.

And if it works i'd like to see if you can connect to portage on a dfferent host when building your system for a first time.

Having a single mysql driven portage system will allow new users to chose if they want to use a private mirror or an online one hosted by a volunteer or maybe gentoo.org itself. This could bring gentoo down in size and make it more attractive for small or obsoleted systems like 486'es.

----------

## Wilhelm

Portage works  :Smile:  Thx m8. I'll test the mirror later when i redo a pc.

For people willing to try themselves here's a few pointers that should work

- Allow the mysql portage user to be accessed from a different host.

- Modify the bind-address of mysql to your network IP address.

- Restart mysql

- Setup the python script and /etc/portage/modules file on your other pc's and modify the python script to point it to your portage server.

For initial install using mysql portage i'd have to look into it since i don't think the bootstrap script will be there or other directories like distfiles etc.. Might work for Stage2 or 3.

HOWTO FOR PASSWORDING PORTAGE

------------------------------------------------

For passworded portage do the following

```

# Select a password you wish to use.

# Replace $PASSWORD in following snippets with a password of your choice

# Modify the following line in the portage.sql file

REPLACE INTO user (Host,User,Password) VALUES ('localhost','portage','');

# To:

REPLACE INTO user (Host,User,Password) VALUES ('localhost','portage',PASSWORD('$PASSWORD'));

# Then modify the following line in the python script

defaultOptions = {'host':'localhost', 'port': 3306,  'user': 'portage', 'password': "", 'db': 'portage'}

# To:

 defaultOptions = {'host':'localhost', 'port': 3306,  'user': 'portage', 'password': "$PASSWORD", 'db': 'portage'}

```

----------

## Wilhelm

 *MooktaKiNG wrote:*   

> One more thing...
> 
> .. Does this mean i dont need /usr/portage anymore?
> 
> I hope thats the case 

 

Me thought the same but however the following seems to apply to me.

When i "emerge sync" i need to have mysql running which is OK since it indicates it's using the portage database.

However if i remove let's say all "app-*" directories they come back after an emerge sync which shouldn't happen. The database is updated aswell (i assume).

So this portage system still uses /usr/portage to store the ebuilds AND it stores it all in mysql so double loss of space.

So basically you can't run off the portage database without having the ebuilds in your portage directory.

Also portage still searches in the /usr/portage directory. If i remove a directory it won't find the apps.

 :Sad: 

/me hopes to find a solution

----------

## MooktaKiNG

 *Wilhelm wrote:*   

>  *MooktaKiNG wrote:*   One more thing...
> 
> .. Does this mean i dont need /usr/portage anymore?
> 
> I hope thats the case  
> ...

 

I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.

There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development.

----------

## Ateo

Choice is wonderful. However, I can't help but wonder why one would want to use mysql to maintain the portage tree. Portage does just fine without it.

Still, a neat idea.

----------

## Wilhelm

 *MooktaKiNG wrote:*   

> 
> 
> I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.
> 
> There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development.

 

Ok well i'm looking for a portage replacement since i don't think a cache is needed unless you work 100% from the DB.

----------

## MooktaKiNG

 *Wilhelm wrote:*   

>  *MooktaKiNG wrote:*   
> 
> I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.
> 
> There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development. 
> ...

 

yeh me too.

I wish someone would rise PortageSQL from the grave  :Very Happy:  :Very Happy:  :Very Happy: 

----------

## Wilhelm

 *MooktaKiNG wrote:*   

>  *Wilhelm wrote:*    *MooktaKiNG wrote:*   
> 
> I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.
> 
> There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development. 
> ...

 

Portage would pwn if it ran directly from a database. You would only need some /tmp directory like /usr/portage/distfiles and you can rock. The world-file etc. can also be put in there. You can easily share and replicate your portage through your network.

emerge sync could be done by replication for example .....  :Razz: 

----------

## Wilhelm

I think the mysql feature is in portage for the gentoo.org peeps to host an online package db.

----------

## xbmodder

ok i benchmarked it w00t w00t:

/proc/cpuinfo

processor       : 0

vendor_id       : AuthenticAMD

cpu family      : 6

model           : 8

model name      : AMD Athlon(tm) MP 2400+

stepping        : 1

cpu MHz         : 2000.991

cache size      : 256 KB

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 1

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow

bogomips        : 3940.35

processor       : 1

vendor_id       : AuthenticAMD

cpu family      : 6

model           : 8

model name      : AMD Athlon(tm) Processor

stepping        : 1

cpu MHz         : 2000.991

cache size      : 256 KB

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 1

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow

bogomips        : 3997.69

/proc/meminfo

MemTotal:       903976 kB

MemFree:        318364 kB

Buffers:         49388 kB

Cached:         311760 kB

SwapCached:          0 kB

Active:         361608 kB

Inactive:       175952 kB

HighTotal:           0 kB

HighFree:            0 kB

LowTotal:       903976 kB

LowFree:        318364 kB

SwapTotal:     5012228 kB

SwapFree:      5012228 kB

Dirty:             120 kB

Writeback:           0 kB

Mapped:         301200 kB

Slab:            31536 kB

Committed_AS:   302708 kB

PageTables:       1872 kB

VmallocTotal:   122804 kB

VmallocUsed:     42920 kB

VmallocChunk:    77044 kB

HugePages_Total:     0

HugePages_Free:      0

Hugepagesize:     4096 kB

uptime:

 14:14:56 up  1:29,  1 user,  load average: 1.54, 0.66, 0.28

MYSQL TIMES:

----------

## codemaker

Hi!

I would like to try this out but I get this error when running emerge:

 *Quote:*   

> 
> 
> delphis codemaker # emerge regen
> 
> Traceback (most recent call last):
> ...

 

I have mysql-python-1.2.0 emerged.

Any ideas?

----------

## Daim

I have the same problem. Any solutions?

Daim

SOLVED: I can't copy&paste  :Wink: 

----------

## nevynxxx

 *Ateo wrote:*   

> Choice is wonderful. However, I can't help but wonder why one would want to use mysql to maintain the portage tree. Portage does just fine without it.
> 
> Still, a neat idea.

 

How about in an enterprise, 500 PCs and just one mysql portage database to be updated.

I think that's about the only advantage.

----------

## b00l

I get this error.... Anyone a clue?

 *Quote:*   

> 
> 
> Traceback (most recent call last):
> 
>   File "/usr/bin/emerge", line 10, in ?
> ...

 

----------

## sylvanus

can this be used with cdb, which is also to speed up portage:

https://forums.gentoo.org/viewtopic-t-261580.html

----------

## shane

With this SELECT:

```
SELECT name, data from `package_name` WHERE `cat_id` ='%d' AND `path_id` = '%d'"
```

Since neither cat_id nor path_id is indexed, this will do a full table scan. (I know they are part of a compound index, but if the leftmost component of the index is not in the WHERE clause, then the index is not used.)

Doing:

```
ALTER TABLE package_name ADD INDEX (cat_id);

ALTER TABLE package_name ADD INDEX (path_id);
```

Should help a bit.

----------

## sokhapkin

This mysql module does not work with portage 2.1... Any chance to make it working with 2.1? Without the module "emerge -uDa world" stays in "Calculating world dependencies" stage forever.

----------

