# Possible hardware/software issue, r8169/hostapd [updated]

## jargon777

Over the past few days, the onboard LAN on my motherboard has been behaving strangely, and I'm not sure what to try --

The chip is a realtek 8168 (10ec:8168) the the motherboard is an AM3+ Gigabyte 970A-D3P.

The Gigabyte computer is connected to another x86 machine that acts as a router/switch (both are Gentoo machines)

This seems to happen under load more often, but it's intermittent -- the LAN chip will suddenly stop hearing replies over the Ethernet. Using wireshark to monitor the Gigabyte's interface, this triggers a wave of ARP requests and DHCP requests. Restarting the net.eth service does not fix the problem.

Using wireshark on the router box shows that the router machine is seeing the ARP and DHCP requests from the Gigabyte machine, and that replies are being transmitted, but replies are not observed on the Gigabyte machine.

Reseating the ethernet cord restores normal behaviour for a bit until this repeats.

Things I've tried:

 - rolling back from kernel 5.4.28 to 4.19.97 (though 5.4.28 was working fine for some time)

- switching the port on the router box the machine is connecting to, and connecting it to a dedicated switch (that still routes to the router box)

dmesg on either machine doesn't reveal anything interesting. 

I'm at a loss at this point -- is this some sort of hardware failure?Last edited by jargon777 on Wed Apr 22, 2020 1:08 am; edited 4 times in total

----------

## Jaglover

Try another cable before you do anything else.

----------

## jargon777

 *Jaglover wrote:*   

> Try another cable before you do anything else.

 

When I connected it to the switch it was through a different cable -- but I'll give the cable thing another go since that test wasn't super rigorous.

----------

## jargon777

I had just marked this solved because it didn't do it for a whole day after I changed the cable, but I spoke too soon and the issue just repeated.

Below is the tshark from the Gigabyte motherboard box:

```

    1 0.000000000 fe80::467f:a8ea:344d:29f5 → ff02::1:ff7f:96a8 ICMPv6 86 Neighbor Solicitation for fe80::210:18ff:fe7f:96a8 from 74:d4:35:9e:10:de

    2 0.064046897 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.15.160? Tell 10.0.10.100

    3 0.383984626 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.0.1? Tell 10.0.10.100

    4 1.087974449 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.15.160? Tell 10.0.10.100

    5 1.408076445 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.0.1? Tell 10.0.10.100

    6 2.111969957 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.15.160? Tell 10.0.10.100

    7 2.431981508 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.0.1? Tell 10.0.10.100

    8 3.136054931 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.15.160? Tell 10.0.10.100

    9 3.352442558 fe80::467f:a8ea:344d:29f5 → ff02::1:ff7f:96a8 ICMPv6 86 Neighbor Solicitation for fe80::210:18ff:fe7f:96a8 from 74:d4:35:9e:10:de

   10 3.455980438 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.0.1? Tell 10.0.10.100

   11 4.160969061 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.15.160? Tell 10.0.10.100

   12 4.415987716 fe80::467f:a8ea:344d:29f5 → ff02::1:ff7f:96a8 ICMPv6 86 Neighbor Solicitation for fe80::210:18ff:fe7f:96a8 from 74:d4:35:9e:10:de

   13 4.575077069 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.0.1? Tell 10.0.10.100

   14 5.183977013 Giga-Byt_9e:10:de → Broadcast    ARP 42 Who has 10.0.15.160? Tell 10.0.10.100

```

And from the router box:

```

    1 0.000000000 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.101? Tell 10.0.10.100

    2 0.023255122 AskeyCom_ee:1f:71 → Giga-Byt_9e:10:de ARP 60 10.0.15.101 is at 00:26:b6:ee:1f:71

    3 0.227972719 fe80::9273:5aff:feed:4c6 → ff02::16     ICMPv6 90 Multicast Listener Report Message v2

    4 0.319995195 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.0.1? Tell 10.0.10.100

    5 0.320022078 Broadcom_7f:96:a8 → Giga-Byt_9e:10:de ARP 42 10.0.0.1 is at 00:10:18:7f:96:a8

    6 0.383996484 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.160? Tell 10.0.10.100

    7 0.384058449 ASUSTekC_64:1d:34 → Giga-Byt_9e:10:de ARP 60 10.0.15.160 is at 60:a4:4c:64:1d:34

    8 1.024038634 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.101? Tell 10.0.10.100

    9 1.026356660 AskeyCom_ee:1f:71 → Giga-Byt_9e:10:de ARP 60 10.0.15.101 is at 00:26:b6:ee:1f:71

   10 1.344037030 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.0.1? Tell 10.0.10.100

   11 1.344063347 Broadcom_7f:96:a8 → Giga-Byt_9e:10:de ARP 42 10.0.0.1 is at 00:10:18:7f:96:a8

   12 1.408084019 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.160? Tell 10.0.10.100

   13 1.408149583 ASUSTekC_64:1d:34 → Giga-Byt_9e:10:de ARP 60 10.0.15.160 is at 60:a4:4c:64:1d:34

   14 1.478673983 fe80::467f:a8ea:344d:29f5 → ff02::1:ff7f:96a8 ICMPv6 86 Neighbor Solicitation for fe80::210:18ff:fe7f:96a8 from 74:d4:35:9e:10:de

   15 1.478739894 fe80::210:18ff:fe7f:96a8 → fe80::467f:a8ea:344d:29f5 ICMPv6 86 Neighbor Advertisement fe80::210:18ff:fe7f:96a8 (rtr, sol, ovr) is at 00:10:18:7f:96:a8

   16 1.513412035 Shenzhen_50:99:8b → Spanning-tree-(for-bridges)_00 STP 52 Conf. Root = 32768/0/00:10:18:7f:96:a8  Cost = 0  Port = 0x8001

   17 1.702166701  10.0.15.160 → 224.0.0.251  MDNS 76 Standard query 0x0000 PTR _ipps._tcp.local, "QM" question

   18 1.770184674  10.0.15.160 → 224.0.0.251  MDNS 968 Standard query response 0x0000 PTR Brother HL-L2320D series @ mattlin._ipps._tcp.local TXT, cache flush SRV, cache flush 0 0 631 mattlin.local AAAA, cache flush 2607:f2c0:94f9:e313:3043:12b2:17f3:46bf A, cache flush 10.0.15.160 PTR Virtual PDF Printer @ mattlin._ipps._tcp.local TXT, cache flush SRV, cache flush 0 0 631 mattlin.local

   19 2.048025689 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.101? Tell 10.0.10.100

   20 2.050106826 AskeyCom_ee:1f:71 → Giga-Byt_9e:10:de ARP 60 10.0.15.101 is at 00:26:b6:ee:1f:71

   21 2.432022653 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.160? Tell 10.0.10.100

   22 2.432095054 ASUSTekC_64:1d:34 → Giga-Byt_9e:10:de ARP 60 10.0.15.160 is at 60:a4:4c:64:1d:34

   23 2.496020730 fe80::467f:a8ea:344d:29f5 → ff02::1:ff7f:96a8 ICMPv6 86 Neighbor Solicitation for fe80::210:18ff:fe7f:96a8 from 74:d4:35:9e:10:de

   24 2.496084208 fe80::210:18ff:fe7f:96a8 → fe80::467f:a8ea:344d:29f5 ICMPv6 86 Neighbor Advertisement fe80::210:18ff:fe7f:96a8 (rtr, sol, ovr) is at 00:10:18:7f:96:a8

   25 3.072018240 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.101? Tell 10.0.10.100

   26 3.074257709 AskeyCom_ee:1f:71 → Giga-Byt_9e:10:de ARP 60 10.0.15.101 is at 00:26:b6:ee:1f:71

   27 3.077750712 SmartRG_c3:a0:8d → Broadcast    ARP 60 Who has 10.0.14.194? Tell 10.0.14.2

   28 3.077843304 SmartRG_c3:a0:8d → Broadcast    ARP 60 Who has 10.0.14.195? Tell 10.0.14.2

...

   90 3.081636256 SmartRG_c3:a0:8d → Broadcast    ARP 60 Who has 10.0.15.1? Tell 10.0.14.2

   91 3.169927811 fe80::3e90:66ff:fec3:a08d → ff02::1      ICMPv6 182 Echo (ping) request id=0x600d, seq=48879, hop limit=255

   92 3.170102203 fe80::3e90:66ff:fec3:a08d → ff02::1:ff7f:96a8 ICMPv6 86 Neighbor Solicitation for fe80::210:18ff:fe7f:96a8 from 3c:90:66:c3:a0:8d

...

  118 3.178646381 fe80::3e90:66ff:fec3:a08d → ff02::1:ff5b:b65a ICMPv6 86 Neighbor Solicitation for fe80::a982:8cc1:b35b:b65a from 3c:90:66:c3:a0:8d

  119 3.456015088 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.160? Tell 10.0.10.100

  120 3.456079845 ASUSTekC_64:1d:34 → Giga-Byt_9e:10:de ARP 60 10.0.15.160 is at 60:a4:4c:64:1d:34

  121 3.497410338 Shenzhen_50:99:8b → Spanning-tree-(for-bridges)_00 STP 52 Conf. Root = 32768/0/00:10:18:7f:96:a8  Cost = 0  Port = 0x8001

  122 3.520011880 fe80::467f:a8ea:344d:29f5 → ff02::1:ff7f:96a8 ICMPv6 86 Neighbor Solicitation for fe80::210:18ff:fe7f:96a8 from 74:d4:35:9e:10:de

  123 3.520053825 fe80::210:18ff:fe7f:96a8 → fe80::467f:a8ea:344d:29f5 ICMPv6 86 Neighbor Advertisement fe80::210:18ff:fe7f:96a8 (rtr, sol, ovr) is at 00:10:18:7f:96:a8

  124 3.872173004 fe80::210:18ff:fe7f:96a8 → ff02::1:ff38:6402 ICMPv6 86 Neighbor Solicitation for 2607:f2c0:94f9:e313:6988:3d49:1738:6402 from 00:10:18:7f:96:a8

  125 4.096059115 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.101? Tell 10.0.10.100

  126 4.098033380 AskeyCom_ee:1f:71 → Giga-Byt_9e:10:de ARP 60 10.0.15.101 is at 00:26:b6:ee:1f:71

  127 4.296343763 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.0.1? Tell 10.0.10.100

  128 4.296367121 Broadcom_7f:96:a8 → Giga-Byt_9e:10:de ARP 42 10.0.0.1 is at 00:10:18:7f:96:a8

  129 4.480108257 Giga-Byt_9e:10:de → Broadcast    ARP 60 Who has 10.0.15.160? Tell 10.0.10.100

  130 4.480184712 ASUSTekC_64:1d:34 → Giga-Byt_9e:10:de ARP 60 10.0.15.160 is at 60:a4:4c:64:1d:34

  131 4.905434857 fe80::210:18ff:fe7f:96a8 → ff02::1:ff38:6402 ICMPv6 86 Neighbor Solicitation for 2607:f2c0:94f9:e313:6988:3d49:1738:6402 from 00:10:18:7f:96:a8

```

[Moderator edit: changed [quote] tags to [code] tags to preserve output layout. -Hu]

----------

## jargon777

This issue is still happening, but I think it is related to hostapd (the desktop acts as a wireless repeater for this room) as it seems to only occur when hostapd is running. The setup with hostapd also has the NIC be part of a bridge. I'm not sure if there's a way to easily debug this further -- neither hostapd nor dmesg throw any info or notice before this happens.

----------

## The Main Man

Maybe it's something in the driver, my onboard model is Realtek RTL8111H - 10/100/1000Mb/s Controller

Driver used is also r8169

I have internet working normally but there's some weird issue when other machine connected to the router gets online then the internet on machine where r8169 is gets really slow and bad, has trouble connecting to sites, as soon as other machines are disconnected then everything goes back to normal.

I suspect the driver because other machines have no trouble (different drivers there)

----------

## jargon777

 *kajzer wrote:*   

> Maybe it's something in the driver, my onboard model is Realtek RTL8111H - 10/100/1000Mb/s Controller
> 
> Driver used is also r8169
> 
> I have internet working normally but there's some weird issue when other machine connected to the router gets online then the internet on machine where r8169 is gets really slow and bad, has trouble connecting to sites, as soon as other machines are disconnected then everything goes back to normal.
> ...

 

Could be, though in my case I don't think the driver has changed too recently and it wasn't occurring before. 

I'm pretty sure it's related to hostapd (or something hostapd does) -- it seems to only happen when hostapd is active and never otherwise. I thought it may be related to ethernet bridging (hostapd adds wlan0 to a bridge with the ethernet card) but it seems not to occur if I bridge the onboard lan with another cheap pci ethernet card and connect a computer to it. It also seems not to occur on any of the other access points with hostapd installed (there are two others). In both cases load was applied with iperf3 but only the hostapd case triggered the failure. Below is the syslog showing messages from hostapd and the kernel when the failure occurs (from start of hostapd to end, took about 20 mins):

```
Apr 21 15:56:32 mw10desk kernel: [79637.702544] br0: port 2(wlp3s0) entered blocking state

Apr 21 15:56:32 mw10desk kernel: [79637.702547] br0: port 2(wlp3s0) entered disabled state

Apr 21 15:56:32 mw10desk kernel: [79637.702608] device wlp3s0 entered promiscuous mode

Apr 21 15:56:33 mw10desk kernel: [79638.977110] IPv6: ADDRCONF(NETDEV_CHANGE): wlp3s0: link becomes ready

Apr 21 15:56:33 mw10desk kernel: [79638.977177] br0: port 2(wlp3s0) entered blocking state

Apr 21 15:56:33 mw10desk kernel: [79638.977179] br0: port 2(wlp3s0) entered forwarding state

Apr 21 15:56:49 mw10desk hostapd: wlp3s0: STA 10:0b:a9:d1:6c:04 IEEE 802.11: authenticated

Apr 21 15:56:49 mw10desk hostapd: wlp3s0: STA 10:0b:a9:d1:6c:04 IEEE 802.11: associated (aid 1)

Apr 21 15:56:49 mw10desk hostapd: wlp3s0: STA 10:0b:a9:d1:6c:04 RADIUS: starting accounting session D4DC54224F528DF7

Apr 21 15:56:49 mw10desk hostapd: wlp3s0: STA 10:0b:a9:d1:6c:04 WPA: pairwise key handshake completed (RSN)

Apr 21 15:56:50 mw10desk hostapd: wlp3s0: STA f8:59:71:18:cf:45 IEEE 802.11: authenticated

Apr 21 15:56:50 mw10desk hostapd: wlp3s0: STA f8:59:71:18:cf:45 IEEE 802.11: associated (aid 2)

Apr 21 15:56:50 mw10desk hostapd: wlp3s0: STA f8:59:71:18:cf:45 RADIUS: starting accounting session F1E9FDDCE5B8156E

Apr 21 15:56:50 mw10desk hostapd: wlp3s0: STA f8:59:71:18:cf:45 WPA: pairwise key handshake completed (RSN)

Apr 21 15:56:57 mw10desk hostapd: wlp3s0: STA 00:08:22:8a:92:fc IEEE 802.11: authenticated

Apr 21 15:56:57 mw10desk hostapd: wlp3s0: STA 00:08:22:8a:92:fc IEEE 802.11: associated (aid 3)

Apr 21 15:56:57 mw10desk hostapd: wlp3s0: STA 00:08:22:8a:92:fc RADIUS: starting accounting session D27AC3E72EF819CE

Apr 21 15:56:57 mw10desk hostapd: wlp3s0: STA 00:08:22:8a:92:fc WPA: pairwise key handshake completed (RSN)

Apr 21 15:57:07 mw10desk hostapd: wlp3s0: STA b0:10:41:5f:82:bf IEEE 802.11: authenticated

Apr 21 15:57:07 mw10desk hostapd: wlp3s0: STA b0:10:41:5f:82:bf IEEE 802.11: associated (aid 4)

Apr 21 15:57:07 mw10desk hostapd: wlp3s0: STA b0:10:41:5f:82:bf RADIUS: starting accounting session 66C8114E55F96B1B

Apr 21 15:57:07 mw10desk hostapd: wlp3s0: STA b0:10:41:5f:82:bf WPA: pairwise key handshake completed (RSN)

Apr 21 15:57:33 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: authenticated

Apr 21 15:57:33 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: associated (aid 5)

Apr 21 15:57:33 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be RADIUS: starting accounting session E03B9A6EDB8E3503

Apr 21 15:57:33 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be WPA: pairwise key handshake completed (RSN)

Apr 21 16:01:01 mw10desk CROND[6867]: (root) CMD (run-parts /etc/cron.hourly)

Apr 21 16:03:33 mw10desk hostapd: wlp3s0: STA 00:22:fb:45:b1:02 IEEE 802.11: authenticated

Apr 21 16:03:33 mw10desk hostapd: wlp3s0: STA 00:22:fb:45:b1:02 IEEE 802.11: associated (aid 6)

Apr 21 16:03:33 mw10desk hostapd: wlp3s0: STA 00:22:fb:45:b1:02 RADIUS: starting accounting session 0770AEB19AD3D08C

Apr 21 16:03:33 mw10desk hostapd: wlp3s0: STA 00:22:fb:45:b1:02 WPA: pairwise key handshake completed (RSN)

Apr 21 16:12:50 mw10desk dhcpcd[3210]: br0: fe80::210:18ff:fe7f:96a8 is unreachable, expiring it

Apr 21 16:13:04 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: authenticated

Apr 21 16:13:04 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: associated (aid 5)

Apr 21 16:13:05 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be RADIUS: starting accounting session 471BD752A405CCBA

Apr 21 16:13:05 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be WPA: pairwise key handshake completed (RSN)

Apr 21 16:13:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: authenticated

Apr 21 16:13:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: associated (aid 5)

Apr 21 16:13:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be RADIUS: starting accounting session D0E2207D3BF93FE1

Apr 21 16:13:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be WPA: pairwise key handshake completed (RSN)

Apr 21 16:14:25 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: authenticated

Apr 21 16:14:25 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: associated (aid 5)

Apr 21 16:14:25 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be RADIUS: starting accounting session ED53B14E1582CE97

Apr 21 16:14:25 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be WPA: pairwise key handshake completed (RSN)

Apr 21 16:15:05 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: authenticated

Apr 21 16:15:05 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: associated (aid 5)

Apr 21 16:15:05 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be RADIUS: starting accounting session B58B4D7D689191C3

Apr 21 16:15:05 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be WPA: pairwise key handshake completed (RSN)

Apr 21 16:15:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: authenticated

Apr 21 16:15:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be IEEE 802.11: associated (aid 5)

Apr 21 16:15:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be RADIUS: starting accounting session F078D41F1DC54B3B

Apr 21 16:15:45 mw10desk hostapd: wlp3s0: STA d4:63:c6:5c:26:be WPA: pairwise key handshake completed (RSN)

Apr 21 16:16:03 mw10desk kernel: [80808.983137] nfs: server mattlin not responding, timed out

Apr 21 16:19:03 mw10desk kernel: [80989.015125] nfs: server mattlin not responding, timed out

Apr 21 16:19:08 mw10desk kernel: [80994.071122] nfs: server mattlin not responding, timed out

```

The only indication of the failure is when a remote nfs server can't be accessed anymore triggering a warning.

I will try two things, bridging hostapd with the cheap pci ethernet card instead (idling the onboard lan) and rolling back the kernel and more rigorously testing if hostapd produces the same effect. I don't understand what in hostapd other than the bridge could trigger this though.

----------

## jargon777

Well rolling back to kernel 4.19.97 didn't fix it, but the issue did not occur when hostapd was run on an installed PCI ethernet card. 

At this point I am just chalking it up to some weird hardware fault. The only thing that has changed on the PC recently is the ram was upgraded from 8gb to 16gb (around when this issue started) so I may've damaged the onboard ethernet or motherboard in some weird way.

----------

