# [SOLVED] Segfault when trying to unmount davfs

## gr3m1in

Hi all,

I'm having the same problem as posted here half a month before

https://archives.gentoo.org/gentoo-user/message/ba28f9376768d3b0164bbb2a66c0a119

however I haven't noticed the exact moment when it started to happen...

Everything works except unmounting (but it was also working fine earlier).

The share is being mounted by a user inside homedir.

Unmounting does not work and segfaults even by root.

Version in use is net-fs/davfs2-1.5.5

Compiled with GCC 8.2.0-r6 against glibc 2.29-r2

Here is strace output for umount.davfs

https://pastebin.com/7wdNVE34

Can anyone please help with the issue?

Thanks in advance!

Kind regards,

gr3m1inLast edited by gr3m1in on Thu May 09, 2019 2:13 pm; edited 1 time in total

----------

## Hu

Please collect a backtrace from the failed process.  For crashes, a backtrace is often more directly informative than an strace.  You may need to rebuild the affected components with debug symbols first.  If you have the same crash as shown in the mailing list thread, then davfs2 is calling glibc in an invalid way.  The backtrace should tell us more.

----------

## gr3m1in

Thanks for your reply, Hu

I've googled about compiling with debug symbols on gentoo

https://wiki.gentoo.org/wiki/Debugging

However it is still a bit unclear how exactly and which packages should be rebuilt.

Just davfs2? Or glibc also? or everything somehow related to building process (gcc, binutils, ...)?

And which of proposed ways should I do this?

By adding  *Quote:*   

> FEATURES="nostrip"

  or  *Quote:*   

> FEATURES="${FEATURES} splitdebug compressdebug -nostrip"

  or by adding "debugsyms" to package.env or even all of this?

Any advice is deeply appreciated!

Kind regards,

gr3m1in

----------

## Hu

I use nostrip because it is the simplest.  It has the downside that if you want to discard the symbols later to recover space, it's more trouble.  If you care about that, use splitdebug instead.

Rebuild whichever package(s) contribute the faulting executable and any shared libraries shown in the backtrace.  I'd start with davfs2 and glibc.  You may need to add others.  Fortunately, you can reproduce this crash easily, so if you don't get everything on the first try, you can easily rebuild and try again.  If this were a once-a-week crash with no good steps to reproduce, I would put more emphasis on having the perfect environment standing by waiting for the crash.  As is, do those two.  If you get any shared libraries in the backtrace that lack symbols, use equery belongs to find the owning package, rebuild that package with debug symbols, and retry.  If the backtrace is detailed enough to let you check multiple libraries at once, check all that you can and rebuild each affected package, to reduce the number of round trips.

----------

## gr3m1in

So what was done:

1. portage env file for turning on nostrip feature

```
# cat /etc/portage/env/debug

#FEATURES="nostrip"

FEATURES="${FEATURES} nostrip"
```

2. list of packages that should use it

```
# cat /etc/portage/package.env

sys-libs/glibc              debug

net-fs/davfs2               debug
```

3. packages rebuild in order

```
echo glibc davfs2 | xargs -n 1 emerge --ask=n
```

After doing so and mounting davfs, no changes were noticed in umount behavior...

```
$ umount.davfs ~/dav/

Segmentation fault
```

dmesg

```
[ 3057.923833] umount.davfs[1282]: segfault at 44962000 ip 00007f878b5cf836 sp 00007ffc1df5b198 error 4 in libc-2.29.so[7f878b551000+15a000]

[ 3057.923845] Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a <f3> 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83
```

/var/log/messages

```
May  2 12:04:55 book kernel: umount.davfs[1282]: segfault at 44962000 ip 00007f878b5cf836 sp 00007ffc1df5b198 error 4 in libc-2.29.so[7f878b551000+15a000]

May  2 12:04:55 book kernel: Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a <f3> 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83
```

No additional files were created inside /var/log.

Am I doing something wrong?

----------

## Hu

 *gr3m1in wrote:*   

> 3. packages rebuild in order
> 
> ```
> echo glibc davfs2 | xargs -n 1 emerge --ask=n
> ```
> ...

 There is no ordering dependency here.  You could have run emerge --ask --oneshot --verbose sys-libs/glibc net-fs/davfs2. *gr3m1in wrote:*   

> After doing so and mounting davfs, no changes were noticed in umount behavior...
> 
> ```
> $ umount.davfs ~/dav/
> 
> ...

 Yes, this is not intended to make it stable.  It is only intended to let you debug why it is unstable. *gr3m1in wrote:*   

> Am I doing something wrong?

 Yes.  You should be looking at the core file, not log files.  However, before you can do that, you need it to actually create a core file.  Currently, it is not, which is weird.  According to your strace output above, its current working directory was still your home directory, which should be fine.  Is this program suid or otherwise resistant to dumping (see man 5 proc section on /proc/sys/fs/suid_dumpable for why it might not be dumped)?  Do you run it with a ulimit that prevents generating a core file?

Regardless, you could ignore the lack of core file and instead run it under a debugger, so that you can trap when it crashes.  Note that using a debugger on a suid program can be a bit tricky.  If this program is suid, special steps may be required.

----------

## gr3m1in

 *Quote:*   

> Yes, this is not intended to make it stable. It is only intended to let you debug why it is unstable.

 

Yes, I do understand that it shouldn't make is stable.

I mean that the output hasn't changed as well as logs.

However after you mentioned ulimit I found the way it should be used https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces#Core_dumps

and after setting

```
ulimit -c unlimited
```

the core dump has been created

```
$ umount.davfs ~/dav/

Segmentation fault (core dumped)
```

and the core file appeared inside homedir.

As it is described via link above, I declared a helper function

```
$ gdb_get_backtrace() {

>     local exe=$1

>     local core=$2

> 

>     gdb ${exe} \

>         --core ${core} \

>         --batch \

>         --quiet \

>         -ex "thread apply all bt full" \

>         -ex "quit"

> }
```

and run it with corresponding parameters

```
$ gdb_get_backtrace /usr/sbin/umount.davfs ./core

[New LWP 23492]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `umount.davfs /home/gr3m1in/dav/'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x00007f6241be0836 in __strlen_sse2 () from /lib64/libc.so.6

Thread 1 (Thread 0x7f623f706740 (LWP 23492)):

#0  0x00007f6241be0836 in __strlen_sse2 () from /lib64/libc.so.6

No symbol table info available.

#1  0x00007f6241d20740 in ne_concat () from /usr/lib64/libneon.so.27

No symbol table info available.

#2  0x000056481dcd9453 in main ()

No symbol table info available.
```

Is it ok that "No symbol table info available" is shown for /lib64/libc.so.6 since glibc was rebuilt with nostrip feature?

And should I also rebuild net-libs/neon with nostrip?

----------

## Hu

 *gr3m1in wrote:*   

> Is it ok that "No symbol table info available" is shown for /lib64/libc.so.6 since glibc was rebuilt with nostrip feature?

 It's not expected, but the output you have so far is probably good enough for glibc. *gr3m1in wrote:*   

> And should I also rebuild net-libs/neon with nostrip?

 Yes, and davfs2 itself too please.  If I were to guess from the available data, I would say that ne_concat was called with an invalid pointer, which then faulted in glibc when Neon tried to compute the length of the string.  Having reviewed the likely callers of ne_concat in davfs (https://fossies.org/linux/www/davfs2-1.5.5.tar.gz/davfs2-1.5.5/src/umount_davfs.c?m=t), I see several trivially obvious memory leaks, unsafe uses of system, and a buggy way of waiting for the process.  Now that I know how this tool works, I would not run it without a code audit and several patches.

----------

## gr3m1in

I have added -g key to cflags and cxxflags in debug env file so it now looks like this

```
# cat /etc/portage/env/debug 

CFLAGS="${CFLAGS} -g"

CXXFLAGS="${CXXFLAGS} -g"

FEATURES="${FEATURES} nostrip"
```

and rebuilt the following packages again

```
# cat /etc/portage/package.env 

sys-libs/glibc              debug

net-libs/neon               debug

net-fs/davfs2               debug
```

so now after trying to unmount the share the backtrace looks more informative

```
$ gdb_get_backtrace /usr/sbin/umount.davfs ./core 

[New LWP 13150]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `umount.davfs /home/gr3m1in/dav/'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120

120     ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.

Thread 1 (Thread 0x7f757f276740 (LWP 13150)):

#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120

No locals.

#1  0x00007f7581890740 in count_concat (ap=0x7ffd33e19fd0) at ne_string.c:132

        total = 0

        next = 0x45bec000 <error: Cannot access memory at address 0x45bec000>

        total = <optimized out>

        next = <optimized out>

#2  ne_concat (str=0x55cfd85f0095 "ps -p ") at ne_string.c:179

        ap = {{gp_offset = 16, fp_offset = 32629, overflow_arg_area = 0x7ffd33e1a050, reg_save_area = 0x7ffd33e19ff0}}

        total = <optimized out>

        slen = 6

        ret = <optimized out>

#3  0x000055cfd85ef453 in main (argc=<optimized out>, argv=0x7ffd33e1a198) at umount_davfs.c:163

        short_options = 0x55cfd85f001d "Vhflnrt:v"

        options = {{name = 0x55cfd85f00a0 "version", has_arg = 0, flag = 0x0, val = 86}, {name = 0x55cfd85f00a8 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}

        o = <optimized out>

        mpoint = <optimized out>

        umount_command = 0x55cfd8d1c020 "umount -i '/home/gr3m1in/dav'"

        m = <optimized out>

        mp = <optimized out>

        pidfile = 0x55cfd8d1c070 "/var/run/mount.davfs/home-gr3m1in-dav.pid"

        pid = 0x45bec000 <error: Cannot access memory at address 0x45bec000>

        file = 0x55cfd8d1c0b0

        ps_command = <optimized out>

        ps_in = <optimized out>

        found = <optimized out>

        n = 94351176694197

        ps_line = 0x7f758187f408 <__exit_funcs_lock> ""
```

It really looks like an invalid call...

Any advice on how to fix this?

----------

## Hu

 *gr3m1in wrote:*   

> I have added -g key to cflags and cxxflags in debug env file so it now looks like this

 This looks correct.  The backtrace below agrees. *gr3m1in wrote:*   

> 
> 
> ```
>         umount_command = 0x55cfd8d1c020 "umount -i '/home/gr3m1in/dav'"
> ```
> ...

 Although expected from source inspection, this looks wrong.  What if your directory name had a single quote in it? *gr3m1in wrote:*   

> 
> 
> ```
>         pid = 0x45bec000 <error: Cannot access memory at address 0x45bec000>
> ```
> ...

 Agreed, that is invalid.  The Makefile does not set a C standard version, and the implementation appears to rely on being built as pre-C99 due to its use of the %a conversion in fscanf for an allocation, not a floating point.  Recent versions of gcc default to a newer standard.  You could try using the standard-approved %m, or force this package to build in C89 mode.  However, as above, my advice from having read the source is to make rather more invasive edits to clean up other bugs and design flaws.

----------

## gr3m1in

Many thanks, Hu,

After forcing C89 mode it does work!

Corresponding env file was created with the following contents:

```
# cat /etc/portage/env/C89

CFLAGS="${CFLAGS} -std=c89"

CXXFLAGS="${CXXFLAGS} -std=c89"
```

and then used to build davfs2 package

```
cat /etc/portage/package.env

sys-libs/glibc              debug

net-libs/neon               debug

net-fs/davfs2               debug C89
```

While rebuilding I've checked that the variables were set correctly (just in case)

```
x86_64-pc-linux-gnu-gcc -DPROGRAM_NAME=\"mount.davfs\" -DDAV_SYS_CONF_DIR=\"/etc/davfs2\" -DDAV_LOCALSTATE_DIR=\"/var/run\" -DDAV_SYS_RUN=\"/var/run/mount.davfs\" -DDAV_SYS_CACHE=\"/var/cache/davfs2\" -DDAV_SECRETS=\"secrets\" -DDAV_CONFIG=\"davfs2.conf\" -DDAV_CERTS_DIR=\"certs\" -DDAV_CLICERTS_DIR=\"private\" -DDAV_DATA_DIR=\"/usr/share/davfs2\" -DLOCALEDIR=\"/usr/share/locale\" -DDAV_USER=\"nobody\" -DDAV_GROUP=\"davfs2\" -D_FORTIFY_SOURCE=2 -DHAVE_CONFIG_H -I. -I..    -Wall -Werror=format-security -fstack-protector-strong --param=ssp-buffer-size=4 -O2 -pipe -g -std=c89 -I/usr/include/neon -c -o dav_fuse.o dav_fuse.c
```

and then checked mounting and unmounting

```
$ mount dav/

$ umount dav/

/sbin/umount.davfs: waiting while mount.davfs (pid 18095) synchronizes the cache .. OK

$
```

So now I can confirm that forcing C89 mode for davfs2 solves this problem at the moment.

However, it is not a panacea, and without code remastering a sort of similar problem could appear soon.

Anyway, this particular problem is now solved.

Thank you again, Hu!

Kind regards,

gr3m1in

----------

