# PPPoE and static subnet setup

## eccerr0r

Maybe someone here knows more about this:

I have a pppoe account for internet access.  I also have a static IP subnet that I should route to (all my machines should be visible to the outside - no NAT).  What I don't understand is how this should be set up.

So there are a few things:

1. The P-t-P address which is the PPPoE server, say ADDR_X = 200.201.202.203

2. The assigned public IP address of the router, say ADDR_L = 50.60.1.1

3. The subnet size and the local Ethernet port, say subnet size of /16 for 65536 addresses.

So I suspect that two Ethernet ports are needed, one for PPPoE, and other for LAN.

[PPPoE server 200.201.202.203] ------PPPoE Ethernet1 ---- [ Router 50.60.1.1 ] ---- Ethernet 2 ---- [ Other externally visible hosts like 50.60.1.2 ]

where Ethernet1 subnet = 50.60.1.1/32 PtP 200.201.202.203

and Ethernet2 subnet = 50.60.1.1/16

Now for the networking configuration:

The PPPoE link needs to be set up such that the peer is ADDR_X/32.  The local address of the PPPoE link needs to be ADDR_L/32 as the outside world needs to look at it as such and it is point to point.  However, how would the outside world also know that the PPPoE link also is the end point for ADDR_L/16?  Does the PPPoE server need to be made aware that this PPPoE link actually holds a /16 ?

The other Ethernet link needs to be set up as ADDR_L/16, I suspect, but this is a bit weird: now both the PPP link AND the LAN link need to have the same IP address.  Is this normal?  Does the kernel automatically bridge across?

Now I suspect there needs to be some iptables routing rules that would route the other machines on the ADDR_L/16 Ethernet port to route out of the LAN.  But how about the inbound routes, how does this work especially when the PPP link is necessarily set to a /32?  

Does there need to be kernel hacks to allow this setup to work?

(I tried to infer this information from a DSL router/modem box that I want to replace with a Gentoo box... but can't trust that the kernel hasn't been hacked to do this.  Honestly I think there's a GPL violation going on here, as it's not clear for an enduser to update the kernel.)

----------

## nativemad

Hi,

i'm not quite sure if i understand everything correctly...   :Rolling Eyes: 

The ISP should have a route set up for your ADDR_L/32 through your PPP IP. So you don't have to worry about that.

if only the ADDR_L/32 is public-routed subnet, then I wonder why you setup your network to be larger!? Sure you can do that, but you will need some kind of NAT for the devices besides the /32 subnet to reach the outside.

The tricky part is on the router where you should route the /32 subnet cleanly but NAT every other subnet... You can set the internal interface to be in /16 so it sees every host in the net. Then you should be very careful with iptables to only do normal routing for the /32 range while doing NAT for the other parts of that /16 net. Maybe the easiest way would be to split the /16 in /32 parts and use these to do the rules...

HTH, cheers

----------

## eccerr0r

Well, I don't think I have a specific IP address but rather a "range" of IP addresses through PPP... That's what confusing me.  The router is set to one of these addresses and serves as the gateway.

Officially I am leasing ADDR_L/16 (actually much smaller than that, but to make it clear that there's a lot of publicly routed ip addresses involved).  My current, crashy, POS router will route any of my ADDR_L/16 addresses through PPP, which is what I'm trying to emulate through a regular machine so I have full routing control.  The POS router takes one of the ADDR_L/16 addresses, ADDR_L, and assigns it to both the PPP link and the LAN link which baffles me.

To complicate things further, the POS router also routes local IP addresses through NAT as well as the publicly visible IP addresses.  This is the traditional NAT that most people have dealt with and thus has a lot of documentation...

----------

## nativemad

Hmmm... I'm still not sure...

If you hook up a simple PPPoE client, do you get the 200.201.202.203 or 50.60.1.1 assigned? It sounds like it's 50.60.1.1, but what is the 200.201.202.203 then?

Technically you could do a bridge internal/external, but that's not what I would do without ebtables heavily filtering stuff.....

Also, just to be aware: If you use the whole /16 net in your LAN but only a fraction of it is "yours" (publicly routed to you), then you might miss your ISP-neighbors who have those other bits of the /16 public range, as you can not really route traffic to them.

----------

## eccerr0r

The 200.201.202.203 is the P-t-P address, this should be the remote host at the ISP.  50.60.1.1 would be one of "my" external host IP addresses.   (grabbed by "ifconfig ppp0")

Actually the whole /16 would be "mine" on my lan, so 50.60.50.60 would be under my control too.  I guess my example is kind of ridiculous as no mere mortal has a /16 but this is just an example:)

I wish my ADSL modem-router had more clear explanation of the options.  The setup has three routing options: 

"Dynamic IP"

"Single Static IP"

"Block of static IP"

Dynamic IP is the usual dhcp option where the router will pppoe and DHCP an address.  It will then NAT out addresses to the LAN.  Fairly standard setup.

Single Static IP is equally simple to understand.  I suppose pppoe will get an IP address that was preassigned.

The "Block of static IP" is my mystery...

Also there's an option for "ISP protocol" which is set to "Auto".  It autoselected PPPoE but the options are:

PPPoE

PPPoA

Transparent Bridging

DHCP

Static IP

And dangit I hate this modem, its webpages are so slow...  Alas it's the only modem that I have that will do 6rd which I want to also do with my homemade box, but ipv4 needs to work first :)

----------

## NeddySeagoon

eccerr0r,

Your ISP will route your entire block of IPs to your router.

I have a /29, which in 8 IP addresses total, 5 usable

Broadcast

Router

5 Usable IPs

Network Address.

Anything in my network_address/29 gets routed to the IP my router is on.

Its up to me to route them on.

From the port scans on the IPs I don't use, it seems to just work.

----------

## nativemad

ok, if you don't mix up a larger a lan with a smaller public-IP range then you can just do "normal" routing.

I'd take two NICs - both started with "noop". Then start pppoe on one NIC and create a bridge device with ppp0 and the second (lan-facing) NIC.

That way your clients see the IP on ppp0 which will also be their gateway.

Then you can do brouting with ebtables and use iptables on top of it to filter unwanted stuff. http://ebtables.netfilter.org/examples/basic.html#ex_brouter

----------

## eccerr0r

Interesting.  Trying to correlate this with my router, which actually does use a bridge - but only between wlan and lan.  The software bridge does not contain ppp0!

Since there is no physical Ethernet between the subscriber line and the router inside this router-modem, pppoe makes it less clear what's really going on.

```

# brctl show

bridge name     bridge id               STP enabled     interfaces

br0             8000.[ETH3MAC]          no              eth0

                                                        eth1

                                                        eth2

                                                        eth3

                                                        usb0

                                                        wl0

```

----------

## nativemad

Does it has an IP bound on br0? And also on ppp0? Which ones?

----------

## eccerr0r

Yes br0 is bound to ... the LAN subnet with its LAN/nonrouteable IP (10.x.x.x).

br0:0 (alias) is bound to... the WAN subnet with the WAN IP address (ADDR_L)

but ppp0 is not bound to any bridge as far as I can tell, but it has the P-t-P address and the WAN IP address with netmask 255.255.255.255 (/32).

--- and oh... NeddySeagoon:

Well, yes my ISP supplied modem-router "works".  Except it seems flaky.  I was trying to understand and reverse engineer the router so that I can build my own with Gentoo but the two-interfaces-same-IP (br0:0 and ppp0) didn't make much sense to me and how it actually routed between the interfaces.  I have a few other modem-routers that seem more reliable but do not support ipv6, and was hoping to use one of my modem-routers as modem-only, and set up my homemade router (and have 6rd as well) instead of using my flaky modem-router that has built in 6rd...

----------

## NeddySeagoon

eccerr0r,

I have a BT supplied 'white box' that takes VDSL in form the phone line and provides PPPoE out.

My router is a Gentoo KVM. The PPPoE gets to the Gentoo KVM via a bridge on the bare metal host.

The bare metal setup is

# eth interfaces for firewall

# we don't want them getting IP addresses

# as they are being donated to bridges

```

config_eth0="null"

config_eth1="null"

config_eth2="null"

config_eth3="null"

config_eth4="null"

bridge_br0="eth1"

#for ppp0 - fake IP 

#config_br0="1.1.1.1"
```

There is no IP on br0, which exists only because of a hardware bug on my network card that prevents PCI passthrough working.

On the router (KVM) side, its eth2

```
config_eth2="null"

config_ppp0="ppp"

link_ppp0="eth2"
```

The KVM provides the PPPoE end point.

When udev dropped interface renaming all the interface names got scrambled, when I set the system up, the names were consistent.

It all works for my IPv4 /29

My IPv6 /64 for the uplink - yes, I use one IP in that address space

And my IPv6 /48 that my ISP gave me for my internal netwark segments.

My ppp0 has both a IPv4 IP, an IPv6 IP and all my other IPs are routed over them.

ppp0 is never donated to a bridge, its the physical ethX that carries the PPPoE packets that gets donated to the bridge.

----------

## eccerr0r

Interesting, so you don't have any routes/iptables entries between the ppp0 despite it "carrying" more than one IP address (the whole /29) and all machines can be accessed from the outside world?  Weird.

I think I'll be trying to use physical hardware for now (likely will end up being a Firebox X1000 box if an Elan 520 doesn't work.  Then again I might have to forgo the Elan as it only has one Ethernet port, two with the cardbus slot).

Yes it's a bit ridiculous: Though I have a ridiculous IPv6 block for MAC address-determined IP space but the ISP also gave me consecutive addresses outside this space... which I don't quite understand either.  But that's another problem to solve once I get IPv4 working...  It really would be nice to go to IPv6 as then the hackers that step through IP addresses looking for potential targets will have a much more interesting address space to deal with.

----------

## nativemad

 *eccerr0r wrote:*   

> Yes br0 is bound to ... the LAN subnet with its LAN/nonrouteable IP (10.x.x.x).

 

And you have a second LAN (DMZ) with the public IPs? Or are you doing S-/D-NAT to the 10.x adresspace?

 *eccerr0r wrote:*   

> 
> 
> br0:0 (alias) is bound to... the WAN subnet with the WAN IP address (ADDR_L)
> 
> but ppp0 is not bound to any bridge as far as I can tell, but it has the P-t-P address and the WAN IP address with netmask 255.255.255.255 (/32).
> ...

 

Yeah, it doesn't need a bridge if the public gateway-IP is also reachable via br0:0. But I find it weird to use one IP on multiple interfaces!?

You can try it that way... I guess both should work. 

 *eccerr0r wrote:*   

> Interesting, so you don't have any routes/iptables entries between the ppp0 despite it "carrying" more than one IP address (the whole /29) and all machines can be accessed from the outside world? Weird. 

 

That is just how routing works... You tell a router that the subnet X is reachable via the gateway Y (which doesn't even need to be in the subnet X) like "route add -net x.x.x.x/32 via x.x.x.y". The router/gateway then either does NAT and knows which of the public IP is which internal IP (S-/D-NAT) or it doesn't do NAT and routes to your boxen with the Public-IP on it's local LAN. For that, you just need to allow forwarding. 

I always did S-/D-NAT, as there you can easily have more boxen than public IPs and do the forwarding based on ports...  :Wink: 

The rest on my setup looks quite like Neddys.

----------

## eccerr0r

 *nativemad wrote:*   

> And you have a second LAN (DMZ) with the public IPs? Or are you doing S-/D-NAT to the 10.x adresspace?
> 
> 

 

The router does S-NAT to the same interface, but I wanted to push this aside until I understood the main routing where there should be no NAT.  Except the oddness of...

 *nativemad wrote:*   

> 
> 
> Yeah, it doesn't need a bridge if the public gateway-IP is also reachable via br0:0. But I find it weird to use one IP on multiple interfaces!?
> 
> You can try it that way... I guess both should work. 
> ...

 

Yes! That's what bugging me!  This router is set up strangely by my ISP (or Actiontec).  How could this work with two interfaces with the same IP address?

But even worse...

 *nativemad wrote:*   

> 
> 
> That is just how routing works... You tell a router that the subnet X is reachable via the gateway Y (which doesn't even need to be in the subnet X) like "route add -net x.x.x.x/32 via x.x.x.y". The router/gateway then either does NAT and knows which of the public IP is which internal IP (S-/D-NAT) or it doesn't do NAT and routes to your boxen with the Public-IP on it's local LAN. For that, you just need to allow forwarding. 
> 
> I always did S-/D-NAT, as there you can easily have more boxen than public IPs and do the forwarding based on ports... :wink: 
> ...

 

Well, if Linux understands that an incoming packet from PPP that's destined to the LAN side automatically since one side is PPP, then this would make things easy... just that I've never had this working.

Maybe I should just test it and see if it works, thought it still defies what I understand about routing.

----------

## nativemad

 *eccerr0r wrote:*   

> 
> 
> Well, if Linux understands that an incoming packet from PPP that's destined to the LAN side automatically since one side is PPP, then this would make things easy... just that I've never had this working.

 

It's really just normal routing and isn't in any way specific to ppp!   :Wink: 

You just need to allow forwarding between ppp0 and the bridge.

 *eccerr0r wrote:*   

> 
> 
> Maybe I should just test it and see if it works, thought it still defies what I understand about routing.

 

Yes, I would just try it with two times the same ip on different interfaces... If that doesn't work properly I'd try brouting.

Or take the plunge and reconfigure everything to only use private IPs internally and do everything with S-/D-NAT.   :Twisted Evil: 

----------

## eccerr0r

Actually I am trying to do this because I want to get away from NAT, NAT is too easy! :D

But the rule that each interface needs a different IP address is broken here...

Now the next question is: suppose I actually controlled the ISP end PPP "server" ... how is IT set up...  I figure that it would have control of several subnets and needs to delegate each out through a single PPP end point...

----------

## nativemad

 *eccerr0r wrote:*   

> Actually I am trying to do this because I want to get away from NAT, NAT is too easy! 

  Hmm.. I'd say a proper S-/D-NAT is much more dynamic and at least my setup has a quite difficult ruleset to deploy each service individually. -I'm not talking about simple masquerading.   :Wink: 

 *eccerr0r wrote:*   

> But the rule that each interface needs a different IP address is broken here...

  That's why I would vote for brouting.

 *eccerr0r wrote:*   

> Now the next question is: suppose I actually controlled the ISP end PPP "server" ... how is IT set up...  I figure that it would have control of several subnets and needs to delegate each out through a single PPP end point...

  This is really no black magic... Just set up the connection through something like PPP. After that it is just a matter of  setting the proper route for the target subnet through the newly created interface. Basically the same as you would do with a branch-VPN.

----------

## eccerr0r

 *nativemad wrote:*   

> That's why I would vote for brouting.
> 
> 

 

How can I tell if the router that I have is doing this?  And if it is not, how can it be actually working, as my router does have two interfaces with the same IP address (ppp0 and br0:0)?

IIRC the router is using a montavista 2.6 kernel...

----------

## eccerr0r

Okay, this problem has just been escalated.

I just noticed my router got a new kernel without my intervention.  While every belief is that this was done by my ISP,

THEY AREN'T SUPPOSED TO CHANGE SOFTWARE ON MY ROUTER WITHOUT MY KNOWLEDGE AND PERMISSION!

Ugh...

Granted I've always suspected this router had an ISP backdoor.

And at least I still have root access to the router, though I can't do much with it (no source code, hence GPL violation)...

----------

## nativemad

 *eccerr0r wrote:*   

> 
> 
> How can I tell if the router that I have is doing this?  And if it is not, how can it be actually working, as my router does have two interfaces with the same IP address (ppp0 and br0:0)?
> 
> 

  It doesn't do brouting, as ppp0 isn't member of the bridge...

It may work also with a stock kernel to set one ip on two different interfaces - I am really not sure, but you might give it a try.. At least it shouldn't cause the usual problems like spanning tree or arp hickups as the ppp interface isn't physical and not connected to an actual LAN....

Either that, or use brouting if you want to use a documented feature. (Although I'm not sure if there would be an actual documentation somewhere about the double-ip solution...  :Rolling Eyes:   :Laughing:   )

 *eccerr0r wrote:*   

> I just noticed my router got a new kernel without my intervention. While every belief is that this was done by my ISP

 

Ugh... At least they care and do updates! Offering an opt-out would be the best way here I guess. There are enough examples of customers who get a cablemodem or similar which won't get updated at all for years....

If I were your dad I'd say: Stop talking my child and finally get your hands dirty! 

Try the double-ip thing first. That is quite easy as you can more or less just dump the iptables ruleset on your current device and maybe change the interface names within the dump and then hook up a clean box, connect pppoe and apply the dump. If that doesn't work as expected, then first check the ruleset again... If it still doesn't work, go back to the ISP-router and read some documentation about brouting! You might have to adjust the iptables ruleset even a bit more, but essentially it should still be applicable.

Good luck!   :Razz: 

----------

## eccerr0r

A lot of this hesitance to do this is because I want to keep my network uptime high :(  Ironically, one of the main reasons to do this was to keep uptime high, at least understand why the router seems to fail...

So I have to build a mock-up if I can, test it before hand, and hopefully plug it straight into the network and have it work.

Fat chance that will happen? :(

Thanks for the hints though.

----------

## nativemad

Sure, you can start playing with a pppoe-server and simulate the whole setup... It looks not that hard to setup actually.. Just hook another linux box to your gentoo routers WAN port, start the interfaces without setting up an actual IP and connect them via pppoe. You can then use that ppp-serverbox to do your tests with reaching your public subnet.  :Wink: 

----------

## miroR

 *nativemad wrote:*   

>  *eccerr0r wrote:*   Actually I am trying to do this because I want to get away from NAT, NAT is too easy!   Hmm.. I'd say a proper S-/D-NAT is much more dynamic and at least my setup has a quite difficult ruleset to deploy each service individually. -I'm not talking about simple masquerading.  
> 
> 

 

And I read it all. Only after more and more searching and reading did I find this. I couldn't recognize much by the title... And I also wandered a lot... And then went the Nftables way partly... (And there's no mention of Nftables here, though...)

I need to do bridge and NAT and linux firewall, as I tried to explain:

A Firewalled Internet Access to Internal Subnet

https://forums.gentoo.org/viewtopic-t-1041028.html

But I have a simple question: so for bridge and stuff, use the old Iptables, and not Nftables?

And also, and I'll ask about that on Netfiler-users mailing list, it that's what it takes...

What does, from:

http://wiki.nftables.org/wiki-nftables/index.php/Bridge_filtering

 *Quote:*   

> 
> 
>  Limitation
> 
> There is currently no connection tracking available for bridge filtering. 
> ...

 

mean?

Does it mean, there really no serious development of Nftables, as something that important, and fairly often used is not supported, or...

Can one use bridge with Nftables still? (Maybe by having the necessary ct done in the usual filter, or nat table (taken they are created with equivalent purpose as in Iptables) or in some other fashion (these ideas have been settling in my understanding, but they haven't yet got my complete understanding...

So, basically, Nttables or Iptables, esp. if linux bridging is involved?

And BTW. eccerr0r, where did you reach with this story? Will you leave the readers wondering?  :Wink: 

----------

## miroR

 *miroR wrote:*   

> 
> 
> But I have a simple question: so for bridge and stuff, use the old Iptables, and not Nftables?
> 
> And also, and I'll ask about that on Netfiler-users mailing list, it that's what it takes...
> ...

 

Just asked:

Nftables or Iptables/Ebtables for a simple linux bridge?

http://marc.info/?l=netfilter&m=145928820814936&w=2

----------

## eccerr0r

I haven't much time to work on this, have a bunch of other stuff to work on that don't have solutions (and need network access).  At least this currently "does" have the "microsoft" solution (crash? reboot router)...

Sigh... sometimes I wish I never wanted to learn networking and just stuck with a dynamic IP address from my ISP and contract other people to do "cloud services"... and hope they don't do stuff with my data that I don't want them to do...

Grr...

----------

## miroR

 *eccerr0r wrote:*   

> At least this currently "does" have the "microsoft" solution (crash? reboot router)...

 

Yeah...

 *eccerr0r wrote:*   

> Sigh... sometimes I wish I never wanted to learn networking 

 

No! Just keep up instead! I'm struggling too...

 *eccerr0r wrote:*   

> and just stuck with a dynamic IP address from my ISP and contract other people to do "cloud services"... and hope they don't do stuff with my data that I don't want them to do...
> 
> Grr...

 

Thx for the honest reply.  :Wink:   ... Yeah, hoping they don't do stuff with one's data.... Anyway, I'll keep my fingers crossed for you to make it.

----------

## nativemad

Hmm... maybe I'm really just getting old.

I guess it is just easier to stick to stuff that you already know by heart than learning new stuff that might not yet be capable of doing everything you're used to.

I'm also not that much into networking and I haven't followed nftables development enough to judge.

Also, if the existing problem is instability, then I'd recommend something more established/mature/stable....  :Wink: 

----------

## miroR

 *nativemad wrote:*   

> Hmm... maybe I'm really just getting old.
> 
> I guess it is just easier to stick to stuff that you already know by heart than learning new stuff that might not yet be capable of doing everything you're used to.
> 
> I'm also not that much into networking and I haven't followed nftables development enough to judge.
> ...

 

Thanks for the reply.

I'm all for FOSS. I actually hate when projects don't succeed, seeing that honest people initiated them...

FOSS to me is almost a religion, that is how pure it is, developers giving good stuff to the world of kind of a finer class of computer users...

You can see here: Nftables or Iptables/Ebtables for a simple linux bridge? that I offered to test (sadly I may not be a good enought tester...), exactly for that reason.

Also I tried to seek advice over where some people suggest Nftables (but possibly not for a linux bridge involving setups):

ERROR: sys-apps/iproute2-4.5.0::gentoo failed (compile phase

https://forums.gentoo.org/viewtopic-t-1040708.html#7899384

Nftables is a sane project, but if there is nowhere to learn how to do my simple setup which involves linux bridge, I will have to go the more established/mature/stable way as you say. Thanks again.

----------

