# HTB not keeping correct ratio for its classes [UNRESOLVED]

## fls

Hi people,

I´ve set up traffic shaping on our company´s router with HTB classes.

They are working as expected with one glitch on which I´m stuck since a few days.  :Sad: 

Here´s my (simplified) setup and the problem:

```
tc qdisc add dev eth0 root handle 1: htb default 20 r2q 1

tc class add dev eth0 parent 1: classid 1:1 htb rate 360kbit ceil 360kbit [uplink from ISP is 384kbit]

tc class add dev eth0 parent 1:1 classid 1:10 htb rate 340kbit ceil 360kbit prio 1

tc class add dev eth0 parent 1:1 classid 1:20 htb rate 20kbit ceil 180kbit prio 2
```

The packets are classified with the iptables CLASSIFY target and they properly arrive at the correct chain.

The class 1:20 is constantly filled with traffic (gentoo package CD uploads on bittorrent  :Very Happy:  ) and as long as there´s no other traffic it gets those 180kbit as it should be.

But as soon as for example an FTP upload on the 1:10 chains kicks in, I expect the traffic to be shared in the ration 340:20 (=17:1) between class 1:10 and 1:20. This is the understanding I got form the manpage and the LARTC Howto. 

But in reality the ratio is approx. 7:2 when both classes are saturated  :Sad: 

The actual output of "tc -s -d class show dev eth0" after a few minutes of full traffic on both classes is:

```
class htb 1:10 parent 1:1 prio 3 quantum 43520 rate 340Kbit ceil 360Kbit burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b level 0

 Sent 7263338 bytes 5158 pkts (dropped 0, overlimits 0)

 rate 34448bps 24pps backlog 4p

 lended: 5147 borrowed: 7 giants: 0

 tokens: -46580 ctokens: -14579

class htb 1:20 parent 1:1 prio 4 quantum 2560 rate 20Kbit ceil 180Kbit burst 1624b/8 mpu 0b cburst 1829b/8 mpu 0b level 0

 Sent 2070191 bytes 1559 pkts (dropped 0, overlimits 0)

 rate 9997bps 7pps backlog 65p

 lended: 408 borrowed: 1086 giants: 0

 tokens: -931521 ctokens: -39038
```

As you can see class 1:10 gets an average of ~34kb/s and 1:20 gets around 10kb/s which is nowhere near the desired share ratio between both of them.

If I look at the traffic with iftop then I can see that the FTP upload on 1:10 gets around 42kb/s (=340kbit) MOST OF THE TIME but in frequent intervals it drops down to something between 10 and 20 kb/s and then slowly rises again to the normal level.   :Evil or Very Mad: 

I experimented with the burst, cburst, quantum and prio options of HTB and got nowhere, always the same "drops".

Anyone of you has some idea or pointer where those frequent drops come from? I´ll be glad about every hint I can get, as it is imortant for us that as soon as there´s some customer downloading from our FTP server (this is what 1:10 is used for) they gets nearly all of our available bandwidth. Of course I´ve also a class for interactive traffic which doesn´t get blocked by 1:10 but, this is working fine so I didn´t include it here  :Smile: 

The box is running:

grsec-sources 2.4.27 (tried vanilla too)

iproute2-2.4.7.20020116-r1

kernel mode PPPoE with 2048/384 connection.

If you need more details I´ll happy to provide them!

Thank you for ANY ideas  :Smile: Last edited by fls on Wed Sep 08, 2004 7:52 am; edited 1 time in total

----------

## devon

What is the "r2q" in your script? I assume a classless qdisc, but I tried looking at the iproute2/tc docs and did not find that listed. You may also want to check out the HTB User Guide for some additional information.

----------

## fls

Thanks for your reply devon!

the r2q is NOT a qdisc but it is a (fairly undocumented as you noticed) parameter to the HTB qdisc.

It is used to calculate the quantum. Here is a minimalistic explanation from tc itself:

```
tc qdisc add dev eth0 htb help

Usage: ... qdisc add ... htb [default N] [r2q N]

 default  minor id of class to which unclassified packets are sent {0}

 r2q      DRR quantums are computed as rate in Bps/r2q {10}

 debug    string of 16 numbers each 0-3 {0}

<snip>

```

It´s also explained a bit in the link you gave me near the end of chapter 2 (link sharing).

I have to use it because the rate in my 1:20 class is so low, that HTB computes a quantum smaller than my mtu (1492 for PPPoE) and the kernel prints a warning about this in syslog:

```
kernsrc@methusalix HTB: quantum of class 10020 is small. Consider r2q change.
```

I could circumvent the problem by specifying a quantum for every class, but this is easier. And it doesn´t affect my problem  :Sad: 

Thank for the pointer to the user guide. I´ve already read it several times, but can´t seem to find the fault in my setup. It seems like a simple setup and in fact the shaping IS WORKING but HTB doesn´t keep the rate permanently (see first post), thus making it useless.

----------

## fls

Really no one who´s got an idea? I´d be very happy for even the smallest hint or the wildest guess  :Smile: 

----------

## tnt

I've had problems whenever I've used ceil.

Try writting a script without ceil, burst and cburst, but only with priority settings.  :Idea: 

Something like:

```
tc qdisc add dev eth0 root handle 1: htb default 20 r2q 1

tc class add dev eth0 parent 1: classid 1:1 htb rate 360kbit

tc class add dev eth0 parent 1:1 classid 1:10 htb rate 340kbit prio 1

tc class add dev eth0 parent 1:1 classid 1:20 htb rate 180kbit prio 2
```

and see how it works. In theory, prio 1 traffic (ftp upload) should suppress prio 2 traffic, but you should try on your settup to see if that works.

Please, report your results here.

----------

## fls

Thanks for the suggestion  :Smile: 

I tried changed our production setup to only specify rate and prio.

The code looks like this:

```
$TC qdisc add dev $DEV root handle 1: htb default 20 r2q 1

$TC class add dev $DEV parent 1: classid 1:1 htb rate 360kbit

$TC class add dev $DEV parent 1:1 classid 1:10 htb rate 48kbit prio 1

$TC class add dev $DEV parent 1:1 classid 1:20 htb rate 340kbit prio 2

$TC class add dev $DEV parent 1:1 classid 1:30 htb rate 180kbit prio 3
```

Class 1:10 is for TCP SYN's, ACK's and DNS trafficClass 1:20 is FTP trafficClass 1:30 is Bittorrent uploads

When ceil is not given tc seems to set ceil = rate as you can see in the output of tc -s

```
class htb 1:1 root rate 372000bit ceil 372000bit burst 1645b/8 mpu 0b overhead 0b cburst 1645b/8 mpu 0b overhead 0b level 7

 Sent 11556013 bytes 24970 pkt (dropped 0, overlimits 0 requeues 0)

 rate 199584bit 80pps backlog 0b 0p requeues 0

 lended: 0 borrowed: 0 giants: 0

 tokens: -2192 ctokens: -2192

class htb 1:10 parent 1:1 leaf 10: prio 1 quantum 6000 rate 48000bit ceil 48000bit burst 1605b/8 mpu 0b overhead 0b cburst 1605b/8 mpu 0b overhead 0b level 0

 Sent 556282 bytes 10800 pkt (dropped 0, overlimits 0 requeues 0)

 rate 18360bit 44pps backlog 0b 0p requeues 0

 lended: 10800 borrowed: 0 giants: 0

 tokens: 403953 ctokens: 403953

class htb 1:20 parent 1:1 leaf 20: prio 2 quantum 42500 rate 340000bit ceil 340000bit burst 1641b/8 mpu 0b overhead 0b cburst 1641b/8 mpu 0b overhead 0b level 0

 Sent 992851 bytes 678 pkt (dropped 0, overlimits 0 requeues 0)

 rate 96bit 0pps backlog 0b 0p requeues 0

 lended: 678 borrowed: 0 giants: 0

 tokens: 57468 ctokens: 57468

class htb 1:30 parent 1:1 leaf 30: prio 3 quantum 22500 rate 180000bit ceil 180000bit burst 1621b/8 mpu 0b overhead 0b cburst 1621b/8 mpu 0b overhead 0b level 0

 Sent 10070114 bytes 13540 pkt (dropped 0, overlimits 0 requeues 0)

 rate 181312bit 35pps backlog 0b 48p requeues 0

 lended: 13492 borrowed: 0 giants: 0

 tokens: -118218 ctokens: -118218
```

Now the testresults:

When Class 1:30 is fully saturated and traffic on 1:20 kicks in then 1:30 reduces to about 120kbit and 1:20 only gets the remaining ~240kbit

So prio 2 traffic is not really suppressing prio 3 traffic and I have the impression even the prio 1 traffic in class 1:10 gets slowed down by heavy traffic in the other classes.

What where your findings when you experimented with and without ceil??

Thanks for the input  :Smile: 

----------

## tnt

Very sad to hear that.

Well, I've started with similar idealistic script as yours, but latter I've found that ceil thing does not work so reliable and just made 3 prio classes. 

It worked better in my case, but I use very slow (single channel ISDN) line. I guess ceil couldn't work well with classes as slow as 3, 3.5 and 5KB/s.

I hope I'll get ADSL soon and I will be really disapointed if I could not control traffic the way I want. 

And what to do with those prioritys if they don't work?

Don't know what to suggest now... If you have time (and will), try to set some rally agressive setup, something like:

```
$TC qdisc add dev $DEV root handle 1: htb default 20 r2q 1

$TC class add dev $DEV parent 1: classid 1:1 htb rate 360kbit

$TC class add dev $DEV parent 1:1 classid 1:10 htb rate 48kbit prio 1

$TC class add dev $DEV parent 1:1 classid 1:20 htb rate 300kbit prio 2

$TC class add dev $DEV parent 1:1 classid 1:30 htb rate 12kbit ceil 180kbit prio 3
```

to try how it works with two classes without ceil (ceil = rate) and just one class that can borrow bandwidth.

(I've lowered down rates to achieve sum of 360kbps which is rate of your parent class.)

I think I've read somewhere that class that exceeds it's rate (borows some bandwidth) is served last... So, it doesn't cost anything to try out...

I'm very interested in results you will get, so please, post them here.

Any other thoughts about this topic?

----------

## fls

Okay, I tried your aggressive setup and it was equally good as the best result I could get (with ceil, burst, cburst, quantum, etc. all specified)

Class 1:30 was filled by constant bittorrent traffic hovering at around 21kb/s (-> ~180kbit/s)

As soon as FTP upload on 1:20 started 1:30 went down to around 7kb/s and the FTP upload got around 35kb/s (-> 280kbit).

Pings (Class 1:10) were around 130ms - not too much above the idle pings of around 60ms

The ratio 7:35 is not too bad but after reading the theory about traffic shaping I had a much more idealistic view - just like you.

But I think with this approach we cannot get much better - at least have I dont LOTS of parameter tuning without success.

 *Quote:*   

> Any other thoughts about this topic?

 

Yes, two:

After switching the firewall to another host some month ago I noticed that the traffic shaping was much worse. But setup was same so it had to be hardware. I substituted the cheap RTL8139 card with a old 3com 3c905 and traffic shaping was okay again. Maybe it's the size of the transmit buffer in the card.Have you taken a look at the PRIO qdisc? My idea was to have a PRIO qdisc with 3 bands each having a HTB class in it to enforce maximum bandwidth per class.

That way traffic in band 3 would only be sent if band 1 and 2 were empty, band 2 would transmit if band 1 was empty and band 1 would always have top priority.

I experimented on a testmachine with its ethernet card and it worked PERFECTLY. I could have a 10MB/s filetransfer on band 2 and as soon as I started another upload on band 1 the previous upload didn't even transmit a single kilobyte per minute. Once the new upload stopped band 2 was back at its normal bandwidth.

"Wonderful" I thought and went on to implement the setup on the box with the ADSL connection. But it did not work at all - the PRIO qdisc just didn't seem to do anything. I spent some nights hunting for the error but had no success.

Maybe you want to try it and report your results?!

If so, I wish good luck  :Smile: 

----------

## tnt

I'll try PRIO setup but I'm afraid I can't get any results on my 64kbps link - it's just too slow to be shaped correctly. I hope I'll get ADSL next week and do all the testing.

Btw, have you tried to lower MTU to 1000 bytes or even 576 bytes (I've read somewhere about this number).

And, in many traffic shaping scripts I've seen that tx queue on ADSL setups is reduced to 30 (!!!) instead of 100 on older linuxes and 1000 on my current Gentoo.

Maybe one of these or combination (MTU 576 and txqeue length of 30) would help... Just guessing, ofcorse...  :Sad: 

```
ippp0     Link encap:Point-to-Point Protocol

          inet addr:212.124.166.141  P-t-P:212.124.160.54  Mask:255.255.255.0

          UP POINTOPOINT RUNNING NOARP  MTU:576  Metric:1

          RX packets:1486881 errors:0 dropped:0 overruns:0 frame:0

          TX packets:1540685 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:30

          RX bytes:885687810 (844.6 Mb)  TX bytes:197028274 (187.9 Mb)
```

Try setting that by:

```
ifconfig eth0 txqueuelen 30 mtu 576
```

or whatever eth you use

 :Question: 

----------

## bigfunkymo

[edit: wrong thread  :Embarassed:  ]

----------

## fls

 *Quote:*   

> Btw, have you tried to lower MTU to 1000 bytes or even 576 bytes (I've read somewhere about this number).

 

Hm, don't have the time to try it at the moment. I also don't see how it could benefit - but I'll try when I've some time.

 *tnt wrote:*   

> And, in many traffic shaping scripts I've seen that tx queue on ADSL setups is reduced to 30 (!!!) instead of 100 on older linuxes and 1000 on my current Gentoo.

 

Well, with ADSL which is PPPoE you have two devices involved:

First, the ethernet card connected to the ADSL Modem. Here's mine:

```
eth3      Link encap:Ethernet  HWaddr 00:E0:81:2F:E9:3F

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:13559575 errors:0 dropped:0 overruns:0 frame:0

          TX packets:12705530 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:7837124390 (7474.0 Mb)  TX bytes:7399185092 (7056.4 Mb)

          Interrupt:18
```

As you can see this device has the default tx queuelen of 1000

Second, there's the ppp interface:

```
ppp0      Link encap:Point-to-Point Protocol

          inet addr:AAA.BBB.CCC.DDD  P-t-P:AAA.BBB.CCC.DDD  Mask:255.255.255.255

          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1

          RX packets:4499949 errors:0 dropped:0 overruns:0 frame:0

          TX packets:4384279 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:3

          RX bytes:2284424289 (2178.5 Mb)  TX bytes:2219116974 (2116.3 Mb)
```

This interface has a transmit queue of only 3, so I increased it in my experiments but sadly I could not notice any difference  :Sad: 

I'm also not entirely sure wether the big txqueue of the ethernet card is used or only the queue of the ppp link or both ...  :Question: 

----------

## fls

@bigfunkymo:

I read your HOWTO. The egress shaping you're doing is the same as what we have discussed above except that you specify burst = 10k which I also tried in my experiments, but no success.

Then you do ingress shaping and that's the part abour your post I don't understand:

Why could ingress shaping solve the problem that egress shaping doesn't work perfectly?

And sorry for being dumb, but what do you mean with "your internet radio's traffic"?? You mean listening music?

----------

## bigfunkymo

I think I posted that reply in this thread by mistake, actually... there was some guy having trouble listening to music because his brother was downloading.  I had quite a bit to drink that night and I can see I got a little confused, sorry.   :Embarassed: 

I am not a TC expert, but I think what you are experiencing may be related to this:

 *Quote:*   

> I´ve also a class for interactive traffic which doesn´t get blocked by 1:10 

 

if it is not blocked by 1:10, then it may be blocking 1:10.  Do you have too much traffic going into this intactive traffic class?  Do you have stochastic fairness enabled anywhere? 

If you were not on the torrent AND you were not shaping traffic at all, would your FTP uploads work significantly faster?  If not, then your problem may be caused by the remote system.

----------

## fls

 *bigfunkymo wrote:*   

> I think I posted that reply in this thread by mistake, 

 

Okay, no problem  :Smile: 

 *Quote:*   

> I am not a TC expert, but I think what you are experiencing may be related to this:
> 
>  *Quote:*   I´ve also a class for interactive traffic which doesn´t get blocked by 1:10  
> 
> if it is not blocked by 1:10, then it may be blocking 1:10.  Do you have too much traffic going into this intactive traffic class?  Do you have stochastic fairness enabled anywhere?

 

This interactive class has an rate + ceil of 48kbit so it cannot really block the others, or am I wrong?

Yes, I have a SFQ qdisc attached to all HTB classes, so stochastic fairness should be given inside each class.

 *Quote:*   

> If you were not on the torrent AND you were not shaping traffic at all, would your FTP uploads work significantly faster?

 

Yes, then my upload hovers at ~375kbit/s which is nearly the advertised 384kbit/s of my ISP.

Thanks for the sugggestions!  :Smile: 

----------

## tnt

As I already said, I have NO expiriance with ADSL, but now I'm waiting for one and I've read some topics about it, so:

it seems that all filtering and shaping should be done on ppp0 iface and NOT on eth0

Try clearing all tc stuff for your eth0 and set:

```
tc qdisc add dev ppp0 root handle 1: htb default 20 r2q 1

tc class add dev ppp0 parent 1: classid 1:1 htb rate 360kbit ceil 360kbit

tc class add dev ppp0 parent 1:1 classid 1:10 htb rate 340kbit ceil 360kbit prio 1

tc class add dev ppp0 parent 1:1 classid 1:20 htb rate 20kbit ceil 180kbit prio 2
```

 :Idea:   :Idea:   :Idea: 

----------

## fls

 *tnt wrote:*   

> it seems that all filtering and shaping should be done on ppp0 iface and NOT on eth0

 

Sorry if that was unclear, but I do all my shaping on ppp0 and nowhere else.

Just wanted to point out that the packets flow through eth0 too so maybe the txqueuelen there also matters.

Good luck with your new connection, BTW  :Smile: 

----------

## tnt

I've started this thread, so maybe some will come out about those txqueuelens... (at the bottom of the post)

https://forums.gentoo.org/viewtopic-t-348459-highlight-adsl.html

 :Confused: 

----------

## Nossie

Hi,

I use this script, maybe it works for you too...

http://nossie.addicts.nl/qos.txt

It uses sfq qdiscs.

Nossie

----------

