# kernel vmalloc failed copying files

## Tender

Hi,

I was rsyncing from my root (reisefs) partition to another partition (in turn formatted with ext3/ext4/xfs) to do some evaluation about finding the best file system to use in terms of minimal waste of available space.

The amount of data copied was about 5 GB.

The rsync commands with ext3 and ext4 target ended without problems.

The rsync commands with xfs target caused a repeatable kernel vmalloc error.

Follows a view from dmesg:

```

...

Jan  9 13:57:06 lowpower kernel: [184755.956857] XFS mounting filesystem sda5

Jan  9 13:57:06 lowpower kernel: [184755.988095] Ending clean XFS mount for filesystem: sda5

Jan  9 14:01:46 lowpower kernel: [185035.268771] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.

Jan  9 14:01:46 lowpower kernel: [185035.268779] xfs_buf_get: failed to map pages

Jan  9 14:01:46 lowpower kernel: [185035.268793] BUG: unable to handle kernel NULL pointer dereference at 00000008

Jan  9 14:01:46 lowpower kernel: [185035.271377] IP: [<c11a31bd>] xfs_da_do_buf+0x4d6/0x5e8

Jan  9 14:01:46 lowpower kernel: [185035.272018] *pde = 00000000

Jan  9 14:01:46 lowpower kernel: [185035.272018] Oops: 0000 [#1] SMP

Jan  9 14:01:46 lowpower kernel: [185035.272018] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.3/i2c-2/2-002d/temp3_alarm

Jan  9 14:01:46 lowpower kernel: [185035.272018] Modules linked in: vboxnetflt vboxdrv [last unloaded: vboxdrv]

Jan  9 14:01:46 lowpower kernel: [185035.272018]

Jan  9 14:01:46 lowpower kernel: [185035.272018] Pid: 17440, comm: rsync Tainted: G        W   2.6.36-gentoo-r5 #5 D945GCLF2/

Jan  9 14:01:46 lowpower kernel: [185035.272018] EIP: 0060:[<c11a31bd>] EFLAGS: 00010246 CPU: 1

Jan  9 14:01:46 lowpower kernel: [185035.272018] EIP is at xfs_da_do_buf+0x4d6/0x5e8

Jan  9 14:01:46 lowpower kernel: [185035.272018] EAX: 00000001 EBX: dc5b8c00 ECX: dc75f000 EDX: c32a1cb0

Jan  9 14:01:46 lowpower kernel: [185035.272018] ESI: 00000000 EDI: 00000000 EBP: 00000000 ESP: c32a1c68

Jan  9 14:01:46 lowpower kernel: [185035.272018]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068

Jan  9 14:01:46 lowpower kernel: [185035.272018] Process rsync (pid: 17440, ti=c32a0000 task=f708d310 task.ti=c32a0000)

Jan  9 14:01:46 lowpower kernel: [185035.272018] Stack:

Jan  9 14:01:46 lowpower kernel: [185035.272018]  c11c4351 cb93c720 cf5070e8 00000000 ffffffff 00000001 cb93c738 cf5070e8

Jan  9 14:01:46 lowpower kernel: [185035.272018] <0> f00c9780 cf5070e8 00000000 dc75f000 00000287 c103987d dc75f000 dc75f000

Jan  9 14:01:46 lowpower kernel: [185035.272018] <0> 00000002 00000001 00000000 c32a1ce4 f10e4930 c32a1d94 02000000 c11a332c

Jan  9 14:01:46 lowpower kernel: [185035.272018] Call Trace:

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11c4351>] ? xfs_trans_read_buf+0x229/0x260

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c103987d>] ? up+0x9/0x2a

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11a332c>] ? xfs_da_read_buf+0x19/0x1e

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11a448e>] ? xfs_da_node_lookup_int+0x47/0x200

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11a448e>] ? xfs_da_node_lookup_int+0x47/0x200

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11c7c19>] ? kmem_zone_alloc+0x49/0x87

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11aa0c4>] ? xfs_dir2_node_addname+0x41/0x817

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11c7c70>] ? kmem_zone_zalloc+0x19/0x25

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11c2cb5>] ? xfs_trans_add_item+0x15/0x44

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11c4588>] ? xfs_trans_ijoin+0x1f/0x25

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c1196610>] ? xfs_bmap_last_offset+0xc7/0xda

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11a5232>] ? xfs_dir2_isleaf+0x12/0x3f

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11a5d0d>] ? xfs_dir_createname+0x129/0x131

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11c62bf>] ? xfs_create+0x29b/0x3af

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11ce3af>] ? xfs_vn_mknod+0x93/0xe0

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c11ce408>] ? xfs_vn_create+0x0/0xd

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c107cc06>] ? vfs_create+0x5b/0x83

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c107d7a2>] ? do_last+0x226/0x47d

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c107eea7>] ? do_filp_open+0x179/0x475

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c10756b7>] ? do_sync_write+0x9e/0xd3

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c1096a53>] ? fsnotify+0x1c1/0x1d8

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c1074541>] ? do_sys_open+0x45/0xcd

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c107460d>] ? sys_open+0x1e/0x23

Jan  9 14:01:46 lowpower kernel: [185035.272018]  [<c10024d0>] ? sysenter_do_call+0x12/0x26

Jan  9 14:01:46 lowpower kernel: [185035.272018] Code: 0c eb 16 31 ed 83 7c 24 48 00 74 14 8b 4c 24 70 8d 54 24 48 b8 01 00 00 00 e8 45 fa ff ff 89 c5 83 7c 24 6c 01 0f 85 81 00 00 00 <8b> 4d 08 8b 51 08 8b 01 0f c8 86 f2 0f b7 d2 81 fa ee fb 00 00

Jan  9 14:01:46 lowpower kernel: [185035.272018] EIP: [<c11a31bd>] xfs_da_do_buf+0x4d6/0x5e8 SS:ESP 0068:c32a1c68

Jan  9 14:01:46 lowpower kernel: [185035.272018] CR2: 0000000000000008

Jan  9 14:01:46 lowpower kernel: [185035.429325] ---[ end trace f8fdd162b4122db8 ]---

...

```

After a google search I found a kernel boot parameter using wich I've doubled the initial vmalloc space allocated, after this trimming the problem disappeared.

cat /proc/meminfo now is:

```

...

VmallocTotal:     262144 kB

VmallocUsed:       69288 kB

VmallocChunk:     188052 kB

...

```

Before the trimming the VmallocTotal was 128MB and the VmallocUsed was more or less the same

The kernel is 2.6.36-gentoo-r5, the cpu is an atom n330 board.

Is it a kernel (XFS) bug or XFS requires more resources than the other file systems?

Thanks!

----------

## Hu

When the kernel prints BUG and a backtrace, that is a kernel bug.  :Smile:   The kernel should never dereference a NULL pointer.  I am not familiar enough with XFS to know whether the failure to map pages is also a bug or just an unfortunate event.

----------

## bookwood

I have the same problems under kernel  2.6.36-r5 since kernel 2.6.34:

https://forums.gentoo.org/viewtopic-t-845611-highlight-.html

The xfs vmap errors always come up on high disk activity, for example on an emerge. In this case always my /var/tmp filesystem is crashing. Sometimes after a few hours, sometimes after a few minutes. I have these errors on my Lenovo T61 Notebook and on kvm gentoo guests. I can commit that the errors disappear when I set the vmalloc to 32M or switch back to kernel 2.6.32. The kvm guest run on a core2 Quad system. My T61 and the kvm guest are 32 bit gentoo systems.

----------

## bookwood

Kernel gentoo-sources-2.6.36-r5 chrashed my filesystem again under boot option vmalloc=256M. Increase vmalloc is not a solution for this xfs issue. 

```
skywalker tmp # pwd

/var/tmp

skywalker tmp # LC_ALL="C" ls

ls: cannot open directory .: Input/output error

skywalker tmp # Filesystem "dm-7": xfs_log_force: error 5 returned.

```

 Here is the dmesg output:

```
vmap allocation for size 1048576 failed: use vmalloc=<size> to increase size.

xfs_buf_get: failed to map pages

BUG: unable to handle kernel NULL pointer dereference at 00000008

IP: [<f19234b3>] xfs_da_do_buf+0x4e9/0x672 [xfs]

*pdpt = 0000000019ad2001 *pde = 0000000000000000 

Oops: 0000 [#1] SMP 

last sysfs file: /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq

Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ipt_addrtype xt_DSCP xt_dscp xt_string xt_NFQUEUE xt_multiport xt_mark xt_hashlimit xt_conntrack xt_connmark nf_conntrack ip_tables x_tables bridge stp llc ipv6 rfcomm l2cap bluetooth snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device ext4 jbd2 tun kvm_intel kvm fan cpufreq_powersave cpufreq_conservative cpufreq_userspace cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_analog arc4 ecb i915 iwlagn drm_kms_helper iwlcore snd_hda_intel snd_hda_codec drm sierra i2c_algo_bit r852 sm_common nand nand_ids nand_ecc i2c_i801 usbserial mtd thinkpad_acpi led_class rtc_cmos nvram pcspkr e1000e rtc_core rtc_lib wacom mac80211 i2c_core intel_agp agpgart yenta_socket pcmcia_rsrc video snd_pcm processor tpm_tis backlight cfg80211 tpm tpm_bios snd_timer iTCO_wdt output iTCO_vendor_support joydev rfkill thermal snd snd_page_alloc battery thermal_sys ac button tg3 libphy e1000 fuse xfs exportfs nfs auth_rpcgss nfs_acl lockd sunrpc jfs raid10 dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log dm_mod scsi_wait_scan sbp2 ohci1394 ieee1394 sl811_hcd usbhid ohci_hcd ssb uhci_hcd usb_storage ehci_hcd usbcore aic94xx libsas lpfc qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 cciss 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc scsi_tgt mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx qlogicfas408 gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_pcmcia pcmcia pcmcia_core

Pid: 5585, comm: mv Not tainted 2.6.36-gentoo-r5 #1 6463WNW/6463WNW

EIP: 0060:[<f19234b3>] EFLAGS: 00210246 CPU: 0

EIP is at xfs_da_do_buf+0x4e9/0x672 [xfs]

EAX: 00000001 EBX: edd27400 ECX: 00014005 EDX: cfa3fcb8

ESI: 00000008 EDI: 00000000 EBP: cfa3fcc8 ESP: cfa3fc5c

 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068

Process mv (pid: 5585, ti=cfa3f000 task=e73d4d10 task.ti=cfa3f000)

Stack:

 003807ba 00000000 00000000 c056fe44 dff0345c dd751d98 000200da 00000001

<0> 00000000 00000000 ffffffff dff03440 0000001a cfa3fe54 cfa3fe58 cfa3fe5c

<0> cfa3fe60 cfa3fcbc c0123195 fffff000 00000003 c241e700 00000001 00000000

Call Trace:

 [<c0123195>] ? kmap_atomic_prot+0xc8/0xe7

 [<f1923698>] ? xfs_da_read_buf+0x18/0x1d [xfs]

 [<f1927f94>] ? xfs_dir2_block_addname+0x59/0x689 [xfs]

 [<f1927f94>] ? xfs_dir2_block_addname+0x59/0x689 [xfs]

 [<c028dfdc>] ? random32+0x17/0x19

 [<f1910f17>] ? xfs_bmap_last_offset+0xe8/0xfc [xfs]

 [<f19261f7>] ? xfs_dir2_isblock+0x1a/0x9a [xfs]

 [<f1926da4>] ? xfs_dir_createname+0x106/0x139 [xfs]

 [<f194bd57>] ? xfs_rename+0x249/0x49a [xfs]

 [<c01bf993>] ? do_lookup+0x64/0xf4

 [<f195ce1d>] ? xfs_vn_rename+0x58/0x62 [xfs]

 [<c01c02a8>] ? vfs_rename+0x20e/0x34c

 [<c01bfa56>] ? __lookup_hash+0x33/0x92

 [<c01c1ccd>] ? sys_renameat+0x164/0x1c9

 [<c01a4cd6>] ? remove_vma+0x46/0x4c

 [<c01a5a4f>] ? do_munmap+0x217/0x231

 [<c01c1d44>] ? sys_rename+0x12/0x14

 [<c0102758>] ? sysenter_do_call+0x12/0x28

Code: 55 b8 8b 45 b4 eb 13 31 ff 83 7d f0 00 74 12 8b 4d 18 8d 55 f0 b8 01 00 00 00 e8 ea f8 ff ff 89 c7 83 7d 14 01 0f 85 da 00 00 00 <8b> 57 08 89 55 b4 89 d1 8b 52 08 8b 01 0f c8 86 f2 0f b7 d2 81 

EIP: [<f19234b3>] xfs_da_do_buf+0x4e9/0x672 [xfs] SS:ESP 0068:cfa3fc5c

CR2: 0000000000000008

---[ end trace b88f3c9551a0f88d ]---

vmap allocation for size 1048576 failed: use vmalloc=<size> to increase size.

xfs_buf_get: failed to map pages

Filesystem "dm-7": XFS internal error xfs_trans_cancel at line 1796 of file fs/xfs/xfs_trans.c.  Caller 0xf1952a5f

Pid: 16808, comm: plasma-desktop Tainted: G      D     2.6.36-gentoo-r5 #1

Call Trace:

 [<f19309c8>] xfs_error_report+0x2c/0x2e [xfs]

 [<f194de06>] xfs_trans_cancel+0x54/0x155 [xfs]

 [<f1952a5f>] ? xfs_create+0x4d7/0x553 [xfs]

 [<f1952a5f>] xfs_create+0x4d7/0x553 [xfs]

 [<f195d1e3>] xfs_vn_mknod+0xca/0x156 [xfs]

 [<f195d288>] xfs_vn_create+0xa/0xc [xfs]

 [<c01c07bd>] vfs_create+0x85/0xcf

 [<c01c0e4e>] do_last+0x277/0x4f8

 [<c01c25f1>] do_filp_open+0x197/0x458

 [<c01b6ef9>] do_sys_open+0x48/0xc9

 [<c01b6fbc>] sys_open+0x1e/0x26

 [<c0102758>] sysenter_do_call+0x12/0x28

 [<c03f0000>] ? x86_pmu_notifier+0x33/0x48

xfs_force_shutdown(dm-7,0x8) called from line 1797 of file fs/xfs/xfs_trans.c.  Return address = 0xf194de1d

XFS: Transforming an alert into a BUG.

Filesystem "dm-7": Corruption of in-memory data detected.  Shutting down filesystem: dm-7

------------[ cut here ]------------

kernel BUG at fs/xfs/support/debug.c:101!

invalid opcode: 0000 [#2] SMP 

last sysfs file: /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq

Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ipt_addrtype xt_DSCP xt_dscp xt_string xt_NFQUEUE xt_multiport xt_mark xt_hashlimit xt_conntrack xt_connmark nf_conntrack ip_tables x_tables bridge stp llc ipv6 rfcomm l2cap bluetooth snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device ext4 jbd2 tun kvm_intel kvm fan cpufreq_powersave cpufreq_conservative cpufreq_userspace cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_analog arc4 ecb i915 iwlagn drm_kms_helper iwlcore snd_hda_intel snd_hda_codec drm sierra i2c_algo_bit r852 sm_common nand nand_ids nand_ecc i2c_i801 usbserial mtd thinkpad_acpi led_class rtc_cmos nvram pcspkr e1000e rtc_core rtc_lib wacom mac80211 i2c_core intel_agp agpgart yenta_socket pcmcia_rsrc video snd_pcm processor tpm_tis backlight cfg80211 tpm tpm_bios snd_timer iTCO_wdt output iTCO_vendor_support joydev rfkill thermal snd snd_page_alloc battery thermal_sys ac button tg3 libphy e1000 fuse xfs exportfs nfs auth_rpcgss nfs_acl lockd sunrpc jfs raid10 dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log dm_mod scsi_wait_scan sbp2 ohci1394 ieee1394 sl811_hcd usbhid ohci_hcd ssb uhci_hcd usb_storage ehci_hcd usbcore aic94xx libsas lpfc qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 cciss 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc scsi_tgt mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx qlogicfas408 gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_pcmcia pcmcia pcmcia_core

Pid: 16808, comm: plasma-desktop Tainted: G      D     2.6.36-gentoo-r5 #1 6463WNW/6463WNW

EIP: 0060:[<f1960375>] EFLAGS: 00010246 CPU: 0

EIP is at xfs_fs_vcmn_err+0xa2/0xae [xfs]

EAX: 00000282 EBX: 00000000 ECX: f19834e0 EDX: 00000282

ESI: edd20c00 EDI: f19788a3 EBP: ead22d74 ESP: ead22d64

 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068

Process plasma-desktop (pid: 16808, ti=ead22000 task=ec6e51c0 task.ti=ead22000)

Stack:

 00000282 00000000 00000008 00000000 ead22d84 f1930996 ead22d9c edd20c00

<0> ead22db0 f1954917 00000010 00000001 edd20c00 f19788a3 edac4580 00000000

<0> df5c6c58 00000000 0000000c ead22ddc f194de1d 00000705 f197784c 00000704

Call Trace:

 [<f1930996>] ? xfs_cmn_err+0x40/0x46 [xfs]

 [<f1954917>] ? xfs_do_force_shutdown+0x78/0xf9 [xfs]

 [<f194de1d>] ? xfs_trans_cancel+0x6b/0x155 [xfs]

 [<f1952a5f>] ? xfs_create+0x4d7/0x553 [xfs]

 [<f1952a5f>] ? xfs_create+0x4d7/0x553 [xfs]

 [<f195d1e3>] ? xfs_vn_mknod+0xca/0x156 [xfs]

 [<f195d288>] ? xfs_vn_create+0xa/0xc [xfs]

 [<c01c07bd>] ? vfs_create+0x85/0xcf

 [<c01c0e4e>] ? do_last+0x277/0x4f8

 [<c01c25f1>] ? do_filp_open+0x197/0x458

 [<c01b6ef9>] ? do_sys_open+0x48/0xc9

 [<c01b6fbc>] ? sys_open+0x1e/0x26

 [<c0102758>] ? sysenter_do_call+0x12/0x28

 [<c03f0000>] ? x86_pmu_notifier+0x33/0x48

Code: 00 68 e4 34 98 f1 ff 34 9d c8 5c 96 f1 68 32 9e 97 f1 e8 94 3e a9 ce 83 c4 0c 8b 55 f0 b8 e0 34 98 f1 e8 2d 61 a9 ce 85 db 75 04 <0f> 0b eb fe 8d 65 f4 5b 5e 5f 5d c3 55 b8 e0 34 98 f1 89 e5 57 

EIP: [<f1960375>] xfs_fs_vcmn_err+0xa2/0xae [xfs] SS:ESP 0068:ead22d64

---[ end trace b88f3c9551a0f88e ]---

Filesystem "dm-7": xfs_log_force: error 5 returned.

Filesystem "dm-7": xfs_log_force: error 5 returned.

Filesystem "dm-7": xfs_log_force: error 5 returned.

Filesystem "dm-7": xfs_log_force: error 5 returned.

```

I entered a bug on this issue:

https://bugs.gentoo.org/show_bug.cgi?id=351938

I switched back to gentoo-sources-2.6.32-r24 and everything works fine. Portage compiled 250 packages without any issue.

----------

