# udev-197 breaks system (ethX jumping around on boot)

## Dragonlord

Since udev-197 my system became unusable. Upon booting the network interfaces randomly jumped around. on one boot I had eth0 and eth1. On the next boot eth1 and eth2. Yet on another boot eth0 and eth2. It's totally impossible to boot the system with jumping network interfaces like this. Udev claims to fight this problem but before udev-197 I had no such problem and now it's all broken.

By removing the file /etc/udev/rules.d/80-net-name-slot.rules I could prevent the jumping around. But now I've got eth1 and eth2 constantly which doesn't look right. Should udev not use crazy names instead of ethX?

Here an excert of the DMESG:

```
r8169 0000:03:00.0: eth0: RTL8168b/8111b at 0xffffc9000000c000, 00:1f:e2:5c:c7:f8, XID 18000000 IRQ 44

r8169 0000:03:00.0: eth0: jumbo features [frames: 4080 bytes, tx checksumming: ko]

r8169 0000:04:00.0: eth1: RTL8168b/8111b at 0xffffc9000000e000, 00:1f:e2:5c:c7:f9, XID 18000000 IRQ 45

r8169 0000:04:00.0: eth1: jumbo features [frames: 4080 bytes, tx checksumming: ko]

systemd-udevd[2095]: renamed network interface eth1 to eth2

systemd-udevd[2094]: renamed network interface eth0 to eth1

r8169 0000:03:00.0: eth1: link down

r8169 0000:03:00.0: eth1: link down

IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready

r8169 0000:03:00.0: eth1: link up

IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
```

udev is totally craz. it moves eth1 to eth2 and eth0 to eth1. What the fuck is that? How in Gods name is one supposed to fix this f'ing mess?!

----------

## PaulBredbury

 *Dragonlord wrote:*   

> Should udev not use crazy names instead of ethX?

 

You probably want predictable network interface names.

I use the old method, e.g. in /etc/udev/rules.d/10-local.rules:

```
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="00:11:6d:4c:9e:d5", NAME:="inet"
```

Note the filename, and the := assignation method, and the address being lower-case, and the fact that my new name doesn't follow the old style of having a number at the end, so that we know it's taking effect.

Get the address using e.g.:

```
udevadm info -a -p /sys/class/net/eth0 | grep address
```

Edit: Can't decide whether to bother adding ACTION=="add"  :Confused: Last edited by PaulBredbury on Thu Apr 11, 2013 12:06 pm; edited 2 times in total

----------

## Dragonlord

Actually I would not mind the names based on hardware position. The problem is only that udev-197 doesn't do this as advertized. It renames ethX to ethY and that's plain stupid to begin with. I can't even use the command outlined in the /etc/udev/rules.d/80-net-name-slot.rules as it complains. With other words:

```
# udevadm test-builtin net_id /sys/class/net/ifname

calling: test-builtin

=== trie on-disk ===

tool version:          197

file size:         5481459 bytes

header size             80 bytes

strings            1230475 bytes

nodes              4250904 bytes

load module index

unable to open device '/sys/class/net/ifname'

unload module index
```

Did I say already that udev-197 is a fucking broken mess?

----------

## ulenrich

 *Dragonlord wrote:*   

> Actually I would not mind the names based on hardware position. The problem is only that udev-197 doesn't do this as advertized. It renames ethX to ethY and that's plain stupid to begin with. I can't even use the command outlined in the /etc/udev/rules.d/80-net-name-slot.rules as it complains. 

 

/etc/udev/rules.d/80-net-name-slot.rules  

is a totally commented out file intended to just hide the real

/lib/udev/rules.d/80-net-name-slot.rules  

for compatibility reasons. 

Upstream Poettering&Co advise to use a more explicit syntax:

/etc/udev/rules.d/80-net-name-slot.rules -pointsTo- /dev/null

(These /dev/null pointers are commonly used  at /etc/systemd/system by administrators to disable distribution default units at /usr/lib/systemd/system )

----------

## SamuliSuominen

udev can't rename hardware interfaces to already existing names, so when kernel assigns eth0 for the first card it finds, udev can't use it anymore since it's reserved

so what you can do is change the kernel naming before udev even starts using the script (attachment) from https://bugs.gentoo.org/show_bug.cgi?id=453494

but as already referenced in this thread, the real solution is to migrate to the new predictable networking name scheme which obsoletes the need for renaming

----------

## Dragonlord

But deleting the /lib/udev/rules.d/80-net-name-slot.rules doesn't fix it. According to the comments in there this should enable the new names but instead udev gives me the ethX name-clash crap. Udev starts out with the volatile interfaces names before the renaming according to DMESG.

----------

## SamuliSuominen

 *Dragonlord wrote:*   

> But deleting the /lib/udev/rules.d/80-net-name-slot.rules doesn't fix it. According to the comments in there this should enable the new names but instead udev gives me the ethX name-clash crap. Udev starts out with the volatile interfaces names before the renaming according to DMESG.

 

Where did you find the information that you should delete /lib/udev/rules.d/80-net-name-slot.rules?

It should be kept there and /etc/udev/rules.d/80-net-name-slot.rules be deleted to enable it.

/bin, /sbin, /lib, /lib32, /lib64, /usr, should all be safe to mount read only, except /usr/src, /usr/portage, and KDE writing directly to polkit's files in /usr -- there is propably more but it SHOULD be supported, just that I'm not sure how well default'ish Gentoo setup runs like that (yet)Last edited by SamuliSuominen on Tue Jan 29, 2013 8:17 pm; edited 2 times in total

----------

## ulenrich

 *Quote:*   

> Where did you find the information that you should delete /lib/udev/rules.d/80-net-name-slot.rules?
> 
> It should be kept there and /etc/udev/rules.d/80-net-name-slot.rules be deleted to enable it.

 

The old wisdom:

Where something can go wrong it will some time, where is the slightest chance something can be misunderstood ....

Perhaps pointers to /dev/null carry no chance at all to be misunderstood ...

----------

## Dragonlord

Typo. Of course it's /etc/udev/rules.d/80-net-name-slot.rules . Removing that one didn't change a thing.

----------

## ulenrich

 *Dragonlord wrote:*   

> But deleting the /lib/udev/rules.d/80-net-name-slot.rules doesn't fix it. According to the comments in there this should enable the new names but instead udev gives me the ethX name-clash crap. Udev starts out with the volatile interfaces names before the renaming according to DMESG.

  "...before the renaming" - which means there is some effect of

/lib/udev/rules.d/80-net-name-slot.rules

showing up in dmesg?

----------

## Dragonlord

 *ulenrich wrote:*   

>  *Dragonlord wrote:*   But deleting the /lib/udev/rules.d/80-net-name-slot.rules doesn't fix it. According to the comments in there this should enable the new names but instead udev gives me the ethX name-clash crap. Udev starts out with the volatile interfaces names before the renaming according to DMESG.  "...before the renaming" - which means there is some effect of
> 
> /lib/udev/rules.d/80-net-name-slot.rules
> 
> showing up in dmesg?

 

See the first post in this topic. There is the DMESG output.

----------

## sera

 *ssuominen wrote:*   

> udev can't rename hardware interfaces to already existing names

 

Because the code to do those pseudo in-place switches Dragonlord relied upon was removed.

 *Dragonlord wrote:*   

> By removing the file /etc/udev/rules.d/80-net-name-slot.rules I could prevent the jumping around. But now I've got eth1 and eth2 constantly which doesn't look right. Should udev not use crazy names instead of ethX?

 

Quite possibly other old net rules are interfering. Find and remove them?

----------

## Dragonlord

There seems to be a file generated by the udev update that is not listed anywhere. Removing the file the names changed. But there is no way to figure out before the change what the new names will be. A problem for a server with 3 or more network cards. I'm not going to upgrade if I end up with a non-working server box.

----------

## emc

 *PaulBredbury wrote:*   

> 
> 
> I use the old method, e.g. in /etc/udev/rules.d/10-local.rules:
> 
> ```
> ...

 

I did (2 first steps advised from udev-197 ebuild):

- rename 70-persistent-net.rules to 70-net-persistent.rules

- revdep-rebuild --library '/lib64/libudev.so.0' && rm '/lib64/libudev.so.0'

- add 10-local.rules (I just add eth0 and wlan0)

And it work as before udev-197 (from end-user perspective)

----------

## SamuliSuominen

 *emc wrote:*   

>  *PaulBredbury wrote:*   
> 
> I use the old method, e.g. in /etc/udev/rules.d/10-local.rules:
> 
> ```
> ...

 

If you do the renaming in 10-local.rules, then why do you have (the now renamed) 70-net-persistent.rules file at all? If you haven't edited it yourself, you shouldn't have it at all.

And it won't work unless you also grabbed the workaround script from bug 453494. It may seem to work if kernel assigned the names like you would have.

----------

## wcg

 *Quote:*   

> udev can't rename hardware interfaces to already existing names

 

Why does udev think it needs to rename something that the kernel

has already named in bus probe order? Why does it not simply use

the kernel's name, in the case of eth? and wlan? interfaces?

edit:

From http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames

 *Quote:*   

> the driver probing is generally not predictable for modern technology

 

Since when? I have never seen a kernel come up with two network devices

in a different order in dmesg on two successive boots.

In theory, I understand this. It would be cool to be able to have "upstream"

(eth0) hardware fail without changing the device names of "downstream"

(ethX with X > 0) network interfaces on a reboot.

I guess the question is why is udev renaming interfaces without explicit

instructions to name specific mac addresses with specific device names.

----------

## ulenrich

I wish some strict semantics here:

 *Quote:*   

> udev can't rename hardware interfaces to already existing names

 Udev can only ask the kernel to rename. It does only if there is an adminitrators instruction.

The kernel refuses by the same rules as he refuses for example:

ln -s /bin /etc

(But using a bind mount you could implement this by force)

 *wcg wrote:*   

> Why does udev think it needs to rename something that the kernel
> 
> has already named in bus probe order? 

 Udev doesn't think, it only asks what the administrator wants to be asked by his set of rules.

 *Quote:*   

> Since when? I have never seen a kernel come up with two network devices
> 
> in a different order in dmesg on two successive boots.

 You are just in the herd of the lucky ones (like me too). But there are tons of possibilities of undefined situations: 

- ide and sata

- flashed bios with other sequences occuring

- never tried combinations of devices

 *Quote:*   

> I guess the question is why is udev renaming interfaces without explicit
> 
> instructions

 It doesn't, Udev simply doesn't think ...

While we speak about Gentoo Udev we talk about a certain implemented set of rules ...

----------

## dmpogo

 *ulenrich wrote:*   

> It doesn't, Udev simply doesn't think ...
> 
> While we speak about Gentoo Udev we talk about a certain implemented set of rules ...

 

While strictly speaking nothing in computer 'thinks', only carries out instructions put in by programmers (and you can go down to silicon),

in most cases the distinction is irrelevant for the user.  In the case of udev, I suspect 98% of users never ever touched any rules (I am for one, never did), and for them 'rules' are part of udev, rather than configuration options that they chose.  In that sense udev rule are further away from 'user configuration level' than even bash scripts of standard init system.

----------

## sera

 *Dragonlord wrote:*   

> There seems to be a file generated by the udev update that is not listed anywhere. Removing the file the names changed. But there is no way to figure out before the change what the new names will be. A problem for a server with 3 or more network cards. I'm not going to upgrade if I end up with a non-working server box.

 

You still can give those interfaces individual names, just avoid using names the kernel would hand out as clashes are no longer resolved via temporaries. Ie, edit your old persistent net rules to use interface names like dragonegg, dragonhorst, dragonhart - well, you get the idea.

----------

## wcg

I know udev does not actually think. One can imagine it as

a state machine. It has a lot of values from kernel device

probes, programmed in defaults, and rules read from

config files (like /etc/udev/udev.conf and /etc/udev/rules.d/*)

that determine its state in this sense. Various states

determine what code paths the udev code follows.

But I think you get my point. What in this set of variables

is sending udev down a code path that tells it to request

that the kernel rename eth0 and eth1 immediately after

boot in the OP's example?

(I have no issue with the concept of persistent network

interface names. It can save a lot of administrative work

when things go haywire. It is similar to mounting by

filesystem label or UUID instead of by drive and partition

number: you can move drives or raid arrays around

in the box at will and still have the same filesystems

mounted on the same mountpoints at boot without

editing fstab to account for changed device probe

order.)

----------

## Dragonlord

@All:

WTF is this UDev crap? I managed to get the interface auto-named as enp3s0 which is fine for me as I don't need special names. But now I did a weekly emerge and udev got recompiled. What does this bitch do? It broke my system AGAIN! No more network devices. What happened? Udev recreated all the files I deleted to get it working and broke everything again.

Can you please either fix UDev-197 or better mask it since it's a broken mess?

 *sera wrote:*   

>  *Dragonlord wrote:*   There seems to be a file generated by the udev update that is not listed anywhere. Removing the file the names changed. But there is no way to figure out before the change what the new names will be. A problem for a server with 3 or more network cards. I'm not going to upgrade if I end up with a non-working server box. 
> 
> You still can give those interfaces individual names, just avoid using names the kernel would hand out as clashes are no longer resolved via temporaries. Ie, edit your old persistent net rules to use interface names like dragonegg, dragonhorst, dragonhart - well, you get the idea.

 

I don't use custom network names as this is not the way a system should be build to begin with. I could do this but see the problem above the quote for why this doesn't get you anywhere with the mess called udev-197.

----------

## SamuliSuominen

 *Dragonlord wrote:*   

> @All:
> 
> WTF is this UDev crap? I managed to get the interface auto-named as enp3s0 which is fine for me as I don't need special names. But now I did a weekly emerge and udev got recompiled. What does this bitch do? It broke my system AGAIN! No more network devices. What happened? Udev recreated all the files I deleted to get it working and broke everything again.
> 
> 

 

Perhaps deleting the file was a bad advise in a sense that stable udev, 197-r4 always recopies the file there if it's not there, so instead of deleting the file, copy the one from /lib/udev/rules.d/80-net-name-slot.rules to /etc/udev/rules.d/

```

# cp -f /lib/udev/rules.d/80-net-name-slot.rules /etc/udev/rules.d/

```

(For now, deleting the file is only okay for ~arch users due to the ebuild behavior)

----------

## Dragonlord

 *ssuominen wrote:*   

>  *Dragonlord wrote:*   @All:
> 
> WTF is this UDev crap? I managed to get the interface auto-named as enp3s0 which is fine for me as I don't need special names. But now I did a weekly emerge and udev got recompiled. What does this bitch do? It broke my system AGAIN! No more network devices. What happened? Udev recreated all the files I deleted to get it working and broke everything again.
> 
>  
> ...

 

Not a solution. The coments in 80-net-name-slot.rules state explicitly that:

 *Quote:*   

> To activate this function, move this file to a name that doesn't end in.rules or remove it then reboot your system.

 

But if udev-197 keeps reinstalling this file this is totally impossible to get working properly.

With other words udev-197 is seriously broken and needs to be masked to protect innocent users.

EDIT:

To better show why this is broken the full quote of the important comment parts:

 *Quote:*   

> # This file is here to prevent your interfaces from being renamed automatically,
> 
> # because the new names will be drastically different from the eth*, wlan*, etc
> 
> # names you are used to working with.
> ...

 

With other words to get interfaces with names like epXsY you need this file to be deleted... but udev reinstalls it always. That's the major problem.

EDIT: EDIT:

The file in question is empty only having comments. It's thus the mere presence of the file that is important not the content.

----------

## SamuliSuominen

 *Dragonlord wrote:*   

> 
> 
> With other words udev-197 is seriously broken and needs to be masked to protect innocent users.
> 
> 

 

You seriously need to calm down. Not masking working udev because of a "bug" in a comment. Heh.

 *Quote:*   

> 
> 
> +  03 Feb 2013; Samuli Suominen <ssuominen@gentoo.org>
> 
> +  files/80-net-name-slot.rules:
> ...

 

----------

## ulenrich

Why not, if new-net-stable-slot-names are aspired:

ln -sf /lib/udev/rules.d/80-net-name-slot.rules /etc/udev/rules.d/80-net-name-slot.rules

----------

## Bones McCracker

 *Dragonlord wrote:*   

> @All:
> 
> WTF is this UDev crap? I managed to get the interface auto-named as enp3s0 which is fine for me as I don't need special names. But now I did a weekly emerge and udev got recompiled. What does this bitch do? It broke my system AGAIN! No more network devices. What happened? Udev recreated all the files I deleted to get it working and broke everything again.

 

This wouldn't be such a problem for you if you had bothered to read the guidance and figure out for yourself what you need to do for your system, instead of ranting, being obnoxious to the people who maintain this package voluntarily for you, and demanding like a child that other people tell you how to administer your system.  Try to show a little more dignity and basic human courtesy.

----------

## allaboutmike

Perhaps it is sprained? But seriously...

If you upgrade when suggested (as I usually do) and then you follow the instructions given when upgrading (which I have learned to always read carefully) then you usually avoid problems. 

Not this time.

 *Quote:*   

> udev-197 and newer introduces a new method of naming network
> 
> interfaces. The new names are a very significant change, so
> 
> they are disabled by default on live systems.
> ...

 

Ooh, better leave those alone then.

 *Quote:*   

> Upstream has removed the possibility of renaming to existing
> 
> network interfaces. For example, it's not possible to assign based
> 
> on MAC address to existing interface eth0.
> ...

 

Hmm, that sounds like how I sort out my three ethernet interfaces. Better read that 80-* rule file and see what to do!

From that file...

 *Quote:*   

> # This functionality has not been tested with gentoo. In fact, we are aware that
> 
> # things will break if you activate it.
> 
> #
> ...

 

Holy crap! I don't want to break my server, better leave that alone then.

Reboot: renaming doesn't work now depending on what order the interfaces are detected. 

Hmm, the old 70-persistent-net.rules file is making it try to do the (now) impossible! 

Got rid of that and no more complaining or missing interfaces. Also no more knowing what interface will end up where. My system seems to randomly reorder every boot!

So. My choices according to the info so far seemed to be:

A. Leave old rules in place: randomly fails depending on detection order. Some interfaces work some times : No way!

or

B. Remove old persistent-net rules and go with unpredictable kernel names: random ethX devices! : No Way! (They invented that file for a reason.)

or

C. Remove old files and 80-* and go with new predictable names: "Untested on Gentoo. In fact, we are aware that things will break if you activate it." : Doesn't sound like a good idea either!

so...

D. Ask Google: After reading for about an hour, I deciphered the mess and went what I think is a much simpler way.

1. Leave the 80-* file as is. A reinstall of udev will apparently put it back anyway. (Surely that is broken? There must be a permanent way to set this on or off.)

2. Edit the old 70-persistent-net.rules file and simply change all ethX to netX. This gives predictable MAC->netX naming which doesn't trip over preexisting device names, because netX names are not used by the kernel. Detection order is no longer a problem.

3. Edit /etc/conf.d/net and other files (eg /var/lib/iptables/rules.save) that mention ethX and change them to use netX. (Is this what will break if you change to the new names?)

4. Reboot and live happily ever after! (Until the next unexpected udev change anyway.)

This way, everything works just the way it used to, only the eth's are all net's. I found that much easier to understand.

----------

## Bones McCracker

Somebody with a working brain!  Imagine that!   :Surprised: 

----------

## Dragonlord

 *BoneKracker wrote:*   

> Somebody with a working brain!  Imagine that!  

 

For your information mister super-duper I did exactly what has been mentioned above... but only after everythign printed by eselect news, emerge messages and comments in the various udev files not only contradict each other but are flat out incorrect in some places. You really want to call this mess the "right way to do it"? I understand that this is voluntary work but GenToo has the notion of "stable" and "testing". If a mess like this happens with testing stuff I say nothing because it is not supposed to work out of the box. Stable stuff though should be safe enough to use except unforseen consequences which are again fine for me. That's what system administrators are for. But turning a stable package into such a mess is just not worth of the notion "stable" on GenToo. I just think a seriously broken package has to be demoted into "testing" (aka masked) until it is stable enough to deserve the "stable" spot. You can call names and and insult me as long as you want but it doesn't change the fact this package is not working correctly and doesn't deserve the "stable" notion. If I would program a mess like this I would be ashamed down into the ground and fix it and not go around insulting others like you just did.

----------

## SamuliSuominen

 *Dragonlord wrote:*   

>  *BoneKracker wrote:*   Somebody with a working brain!  Imagine that!   
> 
> For your information mister super-duper I did exactly what has been mentioned above... but only after everythign printed by eselect news, emerge messages and comments in the various udev files not only contradict each other but are flat out incorrect in some places. You really want to call this mess the "right way to do it"? I understand that this is voluntary work but GenToo has the notion of "stable" and "testing". If a mess like this happens with testing stuff I say nothing because it is not supposed to work out of the box. Stable stuff though should be safe enough to use except unforseen consequences which are again fine for me. That's what system administrators are for. But turning a stable package into such a mess is just not worth of the notion "stable" on GenToo. I just think a seriously broken package has to be demoted into "testing" (aka masked) until it is stable enough to deserve the "stable" spot. You can call names and and insult me as long as you want but it doesn't change the fact this package is not working correctly and doesn't deserve the "stable" notion. If I would program a mess like this I would be ashamed down into the ground and fix it and not go around insulting others like you just did.

 

You haven't yet mentioned how it's broken. It's not. Not in the business of masking working packages because some admins are unwilling to either read documentation, or change.

----------

## Bones McCracker

I didn't insult you.  You insulted yourself, by behaving like a petulant, self-centered child.  Rather than go on a rant and derogate volunteer developers you could offer helpful suggestions, or at least do your homework before wailing helplessly about your full diaper.  That kind of negativity and insulting tone isn't helpful to anybody.

----------

## Dragonlord

Always funny how those insulting others crank up the insulting level and call others insulting them if poked at their behavior. Sorry to say this but I don't need these kinds of insults. They solve nothing... as pretenting problems don't exist doesn't solve anything. And helping is quite a problem if solutions are brushed aside and problems declared WONTFIX in the hope no extra work to fix it is required. But I guess this is the new GenToo style: problems? WONTFIX :thumbsup:

----------

## Bones McCracker

I didn't insult you there either.  I accurately described your behavior, which you had already exhibited for all to see and recognize as such.

Are you now going to continue to abrasively make a spectacle of yourself?

Or, maybe you should apologize for your hostility and rudeness, thank ssuominen for doing his best to maintain the software for you, thank those who offered useful help, and be on your way.

----------

## Ant P.

 *Dragonlord wrote:*   

> But I guess this is the new GenToo style: problems? WONTFIX :thumbsup:

 

The only way the problem in this thread can be fixed is with liberal application of a LART.

----------

## emc

Back to the topic after compiling udev-197-r8  should I add something like:

```
SUBSYSTEM=="net", ATTR{address}=="54:04:a6:14:c3:ba", ACTION=="add", NAME:="eth1"

SUBSYSTEM=="net", ATTR{address}=="74:2f:68:a8:1d:bb", ACTION=="add", NAME:="wlan0"

SUBSYSTEM=="net", ATTR{address}=="54:04:a6:0a:9d:68", ACTION=="add", NAME:="eth0"
```

to /etc/udev/rules.d/80-net-name-slot.rules file or just leave it untouched and commented. I don't want new udev functionality....

----------

## fbcyborg

 *Dragonlord wrote:*   

>  is totally craz. it moves eth1 to eth2 and eth0 to eth1. What the fuck is that? How in Gods name is one supposed to fix this f'ing mess?!

 

Holy words mate!

I'm still wating before upgrading udev from 171 to 197 on my laptop because it is a mess.

I have a lot of custom device mapping among network and disk devices. 

For example, the /dev/sdX to /dev/foo remapping is no longer working as well on my desktop PC. Fortunately I don't have network card issues on such PC.

But the laptop has many more custom mappings and I certainly don't want to mess up things because of udev!

----------

## lost+found

Here's a trick I use, that prevents the jumping netcards. It might be useful for some people. It works, at least on my system, when the cards use different drivers. The one compiled into the kernel image always gets eth0, because it's detection is faster. The other one compiled as a module gets eth1.

----------

## dch24

Got bitten bad by this today on a production machine with 5 NICs. It took way too much downtime to figure out that the root cause was something done 2 reboots ago - the natural first place to look was the last reboot (when udev was upgraded we always check device mappings but this time device mappings changed on the 2nd reboot without notice).

Moving to option D: *allaboutmike wrote:*   

> D. Ask Google: After reading for about an hour, I deciphered the mess and went what I think is a much simpler way.

 

----------

## tuner23

..best explantation of udev, sysfs and all that stuff i ever read  :Wink: 

http://tuomov.bitcheese.net/b/archives/2008/10/29/T20_34_21

----------

## _______0

mm.... so how to use the new way?

also net.lo no longer exists??

----------

