# [solved] very slow qcow2 performance

## Elleni

I have rent a hosted rootserver for our company and setup proxmox on top of debian. While it works, the read/write performance seems wrong. Following the specs and setup. 

```
mdadm -D /dev/md0p3

/dev/md0p3:

           Version : 1.2

     Creation Time : Fri Jan 24 12:54:42 2020

        Raid Level : raid1

        Array Size : 1865361408 (1778.95 GiB 1910.13 GB)

     Used Dev Size : 1928159232 (1838.84 GiB 1974.44 GB)

      Raid Devices : 2

     Total Devices : 2

       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Feb  4 20:35:32 2020

             State : clean 

    Active Devices : 2

   Working Devices : 2

    Failed Devices : 0

     Spare Devices : 0

Consistency Policy : bitmap

              Name : 15630:0

              UUID : 0c380f2d:2126b89b:261d71dc:6eb43710

            Events : 23766

    Number   Major   Minor   RaidDevice State

       0       8        2        0      active sync   /dev/sda2

       1       8       18        1      active sync   /dev/sdb2
```

cpu: 

```
cat /proc/cpuinfo 

processor   : 0

vendor_id   : AuthenticAMD

cpu family   : 23

model      : 8

model name   : AMD Ryzen 7 2700 Eight-Core Processor

stepping   : 2

microcode   : 0x800820b

cpu MHz      : 3077.930

cache size   : 512 KB

physical id   : 0

siblings   : 16

core id      : 0

cpu cores   : 8

apicid      : 0

initial apicid   : 0

fpu      : yes

fpu_exception   : yes

cpuid level   : 13

wp      : yes

flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca

bugs      : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass

bogomips   : 6399.21

TLB size   : 2560 4K pages

clflush size   : 64

cache_alignment   : 64

address sizes   : 43 bits physical, 48 bits virtual

power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
```

Raid 1 on two spinning 2TB Disks: 

```
Disk: /dev/sda and sdb

                            Size: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors

                                      Label: dos, identifier: 0xb359d18b

    Device         Boot              Start           End       Sectors     Size    Id Type

>>  /dev/sda1      *                  2048        194559        192512      94M    83 Linux                   

    /dev/sda2                       194560    3856777215    3856582656     1.8T    fd Linux raid autodetect

    /dev/sda3                   3856777216    3907028991      50251776      24G    82 Linux swap / Solaris
```

I have setup a small partition for the os, and the large part is for storage. One big, luksencrypted raid1 partition, formatted with ext4. While it works and a windows 10 vm seems quite responsive on normal use, I see a very bad read/write performance. For example it takes about 20 or even more minutes to copy a 7GB file with reads and writes @ ~2-3 MB/s. So I beleave that either there is a misconfiguration, or maybe a defective component in the server? 

Can anybody please help me find the source/bottleneck, as I can't beleave it is normal or is it ? cpu and memory seem idle while read/write to disk is soo slow. Or can this be because of the encryption? I would have thinked that if that would be the problem, I would see more cpu load? *confused*

I will be happy to give more informations, if you point me to what could be needed to find out whats going on. The raid was built by the debian installer. I just created a new partition, formatted it with ext4 and mounted it. 

```
lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex

00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit

00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)

00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0

00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1

00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2

00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4

00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5

00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6

00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7

03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43d0 (rev 01)

03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)

03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)

16:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

16:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

16:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

16:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

16:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

16:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

18:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

1a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

1c:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller

1d:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710B] (rev a1)

1d:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)

1e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function

1e:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor

1e:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller

1f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function

1f:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

1f:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller
```

Last edited by Elleni on Wed Feb 05, 2020 6:13 pm; edited 2 times in total

----------

## szatox

You mentioned 3 machines in your setup, which one is slow?

If the problem exists exclusively on your gentoo VM: are you using virtio drivers? I think proxmox uses emulated IDE controller by default.

If the hypervisor is slow, what encryption algorithm do you use there? Is it natively supported by your hardware? Have you tested it's performance on your hardware?

Also, do you use some funny setup that amplifies IO on your HDDs?

 *Quote:*   

> So I beleave that either there is a misconfiguration, or maybe a defective component in the server? 

 Anything interesting in dmesg?

----------

## Elleni

We ordered two ryzen 3rd gen machines, each will have 2x2TB nvme but those are not delivered yet. 

This is just one hosted 2nd gen ryzen 8 core machine, which will serve als backup node, where I installed proxmox over debian buster. The vm is a windows 10 vm. Controler is virtio scsi and the virtual disk is qcow2 with writeback cache. This vm will primarly used for a business application which utilizes mssqlexpress thats why, once I have bether read/write performance, I'd like to look further if there is something to optimize for db usage. On the other hand with an average of 2-3 MB/s read/write performance, no need to look into optimizing anyway.   :Wink: 

Here is whats going on while copying a file on the same disk within windows

```
iostat -k 1 2

Linux 5.3.13-2-pve (hrs)    04.02.2020    _x86_64_   (16 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           2.07    0.00    0.56    6.19    0.00   91.18

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

sda             100.28      1093.32      4586.48   43672980  183207555

sdb              91.22       576.12      4586.50   23013376  183208451

md0             131.35      1670.71      4592.52   66736792  183449030

dm-0            102.77      1576.37      4376.91   62968461  174836324

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.07    0.00    0.31    5.26    0.00   93.36

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

sda              66.00       452.00      3504.00        452       3504

sdb              63.00       764.00      2608.00        764       2608

md0              79.00      1216.00      2604.00       1216       2604

dm-0             79.00      1216.00      2620.00       1216       2620
```

I created the encrypted partition following our gentoo wiki. 

```
cryptsetup luksDump /dev/md0p3

LUKS header information

Version:          2

Epoch:            5

Metadata area:    16384 [bytes]

Keyslots area:    16744448 [bytes]

UUID:             9a2bcc12-27a0-475f-9793-1b892b3a785b

Label:            (no label)

Subsystem:        (no subsystem)

Flags:          (no flags)

Data segments:

  0: crypt

   offset: 16777216 [bytes]

   length: (whole device)

   cipher: aes-xts-plain64

   sector: 512 [bytes]

Keyslots:

  1: luks2

   Key:        512 bits

   Priority:   normal

   Cipher:     aes-xts-plain64

   Cipher key: 512 bits

   PBKDF:      argon2i

   Time cost:  4

   Memory:     1048576

   Threads:    4

   Salt:       xx yy zz 

   AF stripes: 4000

   AF hash:    sha256

   Area offset:290816 [bytes]

   Area length:258048 [bytes]

   Digest ID:  0

Tokens:

Digests:

  0: pbkdf2

   Hash:       sha256

   Iterations: 197397

   Salt:       xx yy zz 

   Digest:    xx yy zz
```

dmesg

It can't be normal performance for this setup, can it?

```
cat /proc/mdstat 

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 

md0 : active raid1 sda2[0] sdb2[1]

      1928159232 blocks super 1.2 [2/2] [UU]

      bitmap: 5/15 pages [20KB], 65536KB chunk

unused devices: <none>
```

```
cat /sys/block/sda/queue/scheduler 

[mq-deadline] none
```

```
sysctl -a | grep -i raid

dev.raid.speed_limit_max = 200000

dev.raid.speed_limit_min = 1000
```

```
hdparm -T -t /dev/sda

/dev/sda:

 Timing cached reads:   19782 MB in  2.00 seconds = 9903.88 MB/sec

 Timing buffered disk reads: 210 MB in  3.05 seconds =  68.83 MB/sec
```

```
/dev/sdb:

 Timing cached reads:   22674 MB in  2.00 seconds = 11353.60 MB/sec

 Timing buffered disk reads: 400 MB in  3.01 seconds = 132.92 MB/sec
```

```
/dev/md0:

 Timing cached reads:   19846 MB in  2.00 seconds = 9935.15 MB/sec

 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device

 Timing buffered disk reads: 384 MB in  3.00 seconds = 127.85 MB/sec
```

And finally the encrypted partition:

```
/dev/md0p3:

 Timing cached reads:   19500 MB in  2.00 seconds = 9762.14 MB/sec

 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device

 Timing buffered disk reads: 384 MB in  3.00 seconds = 127.87 MB/sec
```

```

fdisk -l /dev/md0p3

Disk /dev/md0p3: 1.8 TiB, 1910130081792 bytes, 3730722816 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes
```

```
=== START OF READ SMART DATA SECTION ===

SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Short offline       Completed without error       00%      1028
```

```
=== START OF READ SMART DATA SECTION ===

SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Short offline       Completed without error       00%      1028         -
```

I hope those informations can help to find out, whats going on. I mean granted that qcow2 and virtualization in general will decrease performance, but I don't think to the point of 2-3 MB/s is acceptable?

----------

## Elleni

As a comparison I fired up a windows 10 vm on my local gentoo system, which is a 6 core ryzen 1st gen. My gentoo is sitting on an ssd though. But I have 3 x 1 TB raid5 setup for not encrypted userhome. Here the win 10 vm is built on libvirt and it's virtual disk is sitting on the raid5 not the ssd. And I also created a 20 GB scsi virtio disk on my raid as storage and copied it within the same disk. With scsi its even faster copied 8.7GB within a minute. 

It took barely a minute to copy 2 GB on system virtio blk disk, so I think, that there is probably something wrong on my proxmox vm setup? While I must admit, that my gentoo system is sitting on an ssd, the storage pool I created for comparison is said 3x1TB spinning disk setup, and its not encrypted. But the difference in read/write speed is huge. 

Edit to add, that while comparing the device manager of the two vms I see that while on the proxmox vm there is QEMU QEMU HARDDISK SCSI Disk Device, in the much faster performing vm I have Red Hat VirtIO SCSI Disk Device. I will check, if I can change that on proxmox and see how it perform then, and report back. I will also try to scp the vm here and start it in libvirt and the other way round, copy my fast vm to proxmox to finally find out whats going on..

----------

## Elleni

Converting the disk to raw brought back the expected performance. Could copy a 7 GB File in about 3 minutes. I was not aware that performance of qcow2 can decrease so dramatically especially when taking snapshots often. The size of the 120 GB qcow2 image is 222GB already. I will check if the said qcow2 also performs badly on my gentoo libvirt installation, as I would expect it now. 

So I guess snapshotting is not something to do in a production scenario, right? I will try if performance comes back on qcow2 when deleting snapshots. What else to know about qcow2? *still reading and learning* But if snapshotting is not something to do I can directly stick to raw format, as the snapshot ability was the only reason why I had chosen qcow2 in the first place. 

This is the said file, autocreated by ProxMox. Is there anything that one could try to not get such a dramatic decrease of performance? Is it totaly normal, that it gets that slow while the number of snapshots increases?

```

qemu-img info /path_to/name.qcow2 

image: /path_to/name.qcow2

file format: qcow2

virtual size: 120 GiB (128849018880 bytes)

disk size: 108 GiB

cluster_size: 65536

Snapshot list:

ID        TAG                 VM SIZE                DATE       VM CLOCK

1         Snapshotname1         0 B 2020-01-31 13:28:00   00:24:04.579

2         Snapshotname2                 0 B 2020-01-31 19:16:44   00:02:15.000

3         Snapshotname3    0 B 2020-02-01 00:55:16   00:28:19.986

4         Snapshotname4            0 B 2020-02-02 23:49:41   05:47:52.731

5         Snapshotname5    0 B 2020-02-04 10:18:50   00:27:10.907

6         Snapshotname6      0 B 2020-02-04 11:37:53   01:46:09.962

7         Snapshotname7    0 B 2020-02-04 13:43:36   01:52:49.401

8         Snapshotname8    0 B 2020-02-04 15:59:06   03:05:04.159

9         Snapshotname9         0 B 2020-02-04 20:11:53   07:17:49.147

10        Snapshotname10         0 B 2020-02-05 01:11:23   12:16:58.928

Format specific information:

    compat: 1.1

    lazy refcounts: false

    refcount bits: 16

    corrupt: false
```

Will the performance come back if deleting all snapshots but maybe one? Still reading, maybe a problem of too small L2 cache size and/or cluster size/preallocation lazy refcounts and/or other settings?

----------

## AJM

 *Elleni wrote:*   

> So I guess snapshotting is not something to do in a production scenario, right? I will try if performance comes back on qcow2 when deleting snapshots.  Will the performance come back if deleting all snapshots but maybe one? Still reading, maybe a problem of too small L2 cache size and/or cluster size/preallocation lazy refcounts and/or other settings?

 

I'd be interested to know what you find.  I have one production VM running on qcow2 but I just shut it down periodically and copy the image file for backup purposes before doing "risky" updates etc rather than using snapshots.

I believe it's similar on Windows / Hyper-V - I have a few server VMs on Hyper-V elsewhere and have been warned not to use snapshots / checkpoints as they will kill performance.  I haven't risked trying it to confirm!

It's very annoying, since as you say this is one of the major advantages of virtualisation... snapshot running server, perform changes, discover a snag, just revert.

----------

## TigerJr

 *Elleni wrote:*   

> Converting the disk to raw brought back the expected performance. Could copy a 7 GB File in about 3 minutes. I was not aware that performance of qcow2 can decrease so dramatically especially when taking snapshots often. The size of the 120 GB qcow2 image is 222GB already. I will check if the said qcow2 also performs badly on my gentoo libvirt installation, as I would expect it now. 
> 
> So I guess snapshotting is not something to do in a production scenario, right? I will try if performance comes back on qcow2 when deleting snapshots. What else to know about qcow2? *still reading and learning* But if snapshotting is not something to do I can directly stick to raw format, as the snapshot ability was the only reason why I had chosen qcow2 in the first place. 
> 
> This is the said file, autocreated by ProxMox. Is there anything that one could try to not get such a dramatic decrease of performance? Is it totaly normal, that it gets that slow while the number of snapshots increases?
> ...

 

what pveperf shows on proxmox?

----------

## Elleni

CPU BOGOMIPS:      102386.08

REGEX/SECOND:      2771835

HD SIZE:           27.37 GB (/dev/md0p1)

BUFFERED READS:    100.54 MB/sec

AVERAGE SEEK TIME: 12.77 ms

FSYNCS/SECOND:     11.88

DNS EXT:           29.05 ms

----------

