# systemd-resolved switches to wrong DNS

## musv

Hi there.

tldr:

Since my last system update systemd-resolved is doing some crazy stuff. Every few minutes systemd-resolved switches to the wrong DNS server. 

Long story: 

192.168.109.1: My Router, which provides a DNS service to the local network. DHCP is deactivated. Should be only the fallback option.

192.168.109.50: My NAS: provides DNS, DHCP, NFS, Webservice and so on. The DHCP server tells the clients to use 192.168.109.50 as primary DNS server, which works quite well. 

After booting the client:

```

Global

           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

    resolv.conf mode: uplink

  Current DNS Server: 192.168.109.50

         DNS Servers: 192.168.109.50

Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google 2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google

Link 2 (eth0)

    Current Scopes: DNS LLMNR/IPv4

         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Current DNS Server: 192.168.109.50

       DNS Servers: 192.168.109.50 192.168.109.1

        DNS Domain: erz szb

```

Some minutes later:

```

Global

           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

    resolv.conf mode: uplink

  Current DNS Server: 192.168.109.50

         DNS Servers: 192.168.109.50

Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google 2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google

Link 2 (eth0)

    Current Scopes: DNS LLMNR/IPv4

         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Current DNS Server: 192.168.109.1

       DNS Servers: 192.168.109.50 192.168.109.1

        DNS Domain: erz szb

```

Why is systemd-resolved switching to the secondary DNS? This happens always since the last update (Systemd-250.3). With the previous versions I've never had this problem.

To show this daemon the right direction, just for trying I put the main DNS into the config:

```

[Resolve]

DNS=192.168.109.50

```

As the consequence systemd-resolved puts this entry in the first line of the /etc/resolv.conf. So the first 2 entries are: 192.168.109.50, the 3rd line is: 192.168.109.1. But ignorantly after some minutes it switches to 192.168.109.1 as the used DNS server.

----------

## ozzy468

hi,

I can't really answer to your post, basically read it, because of 0 replies.

I use pihole as my dhcp/dns server in my network, and i was stunned how complicated dns can actually be. But from your post, 2 questions derive.

1st. Why do you have 2 dns servers in your network, guessing that both use the same dns servers in the background(8.8.8.8 or so)? and then, from a networking perspective

2nd. I guess on any "elaborated" network protocol setup, within your network, the router will always be 1 hop faster than the other dns server. So it might not be a systemd problem?

Both are more or less questions. 

So why use the router's dns at all?

----------

## fred0

Hi,

Same for me. Any idea ?

----------

## AJM

This is not intended to be an unhelpful reply, but my consistent experience over the years with SystemD is that you can expect things that never, ever gave any problems in the past to suddenly become unreliable / unpredictable.  It's just the way it is I'm afraid - badly thought out, overreaching, buggy software.  At least with Gentoo you have the choice of avoiding it - for now, at least.

----------

## Hu

When systemd-resolved changes the DNS servers, are there any log messages written?  If so, what does it say?  If not, can you enable more verbose logging and repeat the test?

----------

## fred0

Hi,

I have local LXC container with net-dns/dnsmasq that use /etc/dnsmasq-hosts.conf for local network resolution.

On other LXC container (ubuntu, fedora, gentoo + openrc) everything work fine.

One QEMU VM with gentoo + systemd is also OK.

Every systems are up to date.

So long as I ask for a name pointing local network every thing work fine also.

But when I ask for external domain fall-back take place.

Example :

```
systemctl restart systemd-resolved.service
```

```
systemd-resolve --status
```

 *Quote:*   

> Current DNS Server: 192.168.1.104

 

```
ping cloud
```

 *Quote:*   

> 64 bytes from ce68f1cfc269 (192.168.1.111): icmp_seq=1 ttl=64 time=0.127 ms

 

```
systemd-resolve --status
```

 *Quote:*   

> Current DNS Server: 192.168.1.104

 

```
ping gentoo.org
```

 *Quote:*   

> 64 bytes from 151.101.2.137 (151.101.2.137): icmp_seq=1 ttl=55 time=13.2 ms

 

```
systemd-resolve --status
```

 *Quote:*   

> Current DNS Server: 1.1.1.1

 

```
ping cloud
```

 *Quote:*   

> ping: cloud: Name or service not known

 

Actually I have put 

```
systemctl restart systemd-resolved.service
```

 in a cron job every minute but that work not every time.

----------

## fred0

 *fred0 wrote:*   

> Hi,
> 
> I have local LXC container 192.168.1.104 with net-dns/dnsmasq that use /etc/dnsmasq-hosts.conf for local network resolution.
> 
> On other LXC container (ubuntu, fedora, gentoo + openrc) everything work fine.
> ...

 

----------

## ozzy468

hi,

as a workaround, not a fix. Have you tried to set up a local domain? There is an option "Domains=" in systemd.network, and a "routing-only" domain "~."

So something like 

DNS=your.dns.ip

Domains=~yourdomain

might do the trick.

----------

## fred0

It is a Proxmox LXC container. 

Maybe a link between Proxmox kernel and Gentoo systemd ? 

Proxmox host kernel is 5.13.19-6-pve and gentoo QEMU VM with 5.15.26-gentoo kernel works.

I have this in /etc/resolv.conf

```
# --- BEGIN PVE ---

search my_local_domain.local

nameserver 192.168.1.104

nameserver 1.1.1.1

# --- END PVE ---
```

This in /etc/hosts

```
# --- BEGIN PVE ---

127.0.0.1 localhost.localnet localhost

::1 localhost.localnet localhost

127.0.1.1 my_host.my_local_domain.local my_host

# --- END PVE ---
```

And this in /etc/systemd/resolved.conf

```
[Resolve]

# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:

# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com

# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google

# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net

#DNS=

#FallbackDNS=1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google 2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google

#Domains=

#DNSSEC=allow-downgrade

#DNSOverTLS=no

#MulticastDNS=yes

#LLMNR=yes

#Cache=yes

#CacheFromLocalhost=no

#DNSStubListener=yes

#DNSStubListenerExtra=

#ReadEtcHosts=yes

#ResolveUnicastSingleLabel=no
```

Should I change this file ?

Normally no because DNS change are do by Proxmox (between BEGIN PVE and END PVE)

As musv said this happen after last update. I have updated all VMs and Proxmox together last Friday.

----------

## szatox

Do you have a reason not to kill it with fire?

Resolved was a common source of problems on many systems I saw, ended up disabling and masking it in the installation script just to make sure it won't break things down the line.

I think you can change /etc/network/interfaces (since proxmox is based on debian) to make the update to DNS settings permanent. Obviously, update /etc/resolv.conf too, to make it effective immediately.

----------

## ozzy468

hi,

what I meant, was the "client" side. - I really have no xp concerning containers.

the original post, said, systemd falls back to the router/alternative dns server. now, from your post, systemd will also fall back to a dns server, outside your network. on first glance, this could be anything, network optimization, a new feature, that's been missed, or just a bug. we don't know. 

So the idea was just, if you tell the client "there is a local domain"; that it would not switch/or switch and still use this domain for local dns - in a way that it's useless to look up local domains on an internet server. And - that was just a guess - that this behavior is implemented, no matter of buginess or intention.

But from your post, it looks like this has already been done.

----------

## fred0

 :Very Happy:  I am out of trouble thanks to https://wonghoi.humgar.com/blog/2019/09/01/systemd-resolved-dns-resolution-nightmare/

I do :

```
systemctl disable systemd-resolved.service

systemctl stop systemd-resolved.service

reboot
```

please, don't ask me why   :Rolling Eyes: 

----------

## szatox

Mask it too, or it will come back to bite you in the ass later on.

----------

## fred0

 *szatox wrote:*   

> Mask it too, or it will come back to bite you in the ass later on.

 

Do you mean a thing like putting 

```
sys-apps/systemd -resolvconf
```

 in /etc/portage/package.use/sys ???

----------

## szatox

No, I mean 

```
systemctl mask systemd-resolved
```

Systemd keeps unit files in several locations with various priorities, the command above creates a link to /dev/null in the top priority location, which prevents systemd from looking in other locations and the service will not run even if something else attempts to pull it as a dependency.

Systemd-resolved is probably not an issue on non-systemd systems as they don't use it (at least for now). If you do want to have a local DNS cache, you can still use dnsmasq (for simplicity) or bind (for advanced, non-standard setup). Most people don't need it, but if you happened to find yourself an exception, do it yourself so it's done right.

----------

## fred0

Now, without resolved, dns resolution works. I don't know how but there is most likely a dns software on this system.

Is there perhaps a conflict with resolved ?

Could it be possible to remove the other dns software and restore resolved ?

How to find this other piece of code ?

----------

