# Poor NVME transfer rates

## Belliash

Hello,

I have bought a new PC and under Gentoo I get a poor NVME disk performance:

```
# hdparm -tT /dev/nvme0n1

/dev/nvme0n1:

 Timing cached reads:   30468 MB in  2.00 seconds = 15253.43 MB/sec

 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device

 Timing buffered disk reads: 1532 MB in  3.00 seconds = 510.37 MB/sec
```

Do you have any ideas what can be wrong with it?

What can I check?

----------

## twalter

What is the drive rated for and what do benchmarks actually show it capable of?  SSDs are quick but marketing is dirty.  You may have a M.2 SATA drive (which tops out at 500MB/s or so, which checks.)

----------

## axl

nvme is no quality assurance. what kind is it? that matters a lot.

----------

## Belliash

Oh, I forgotten to say anything about the disk. It is PLEXTOR PX-512M9PeG 1.07. It is M.2 PCI-E disk that should have around 2300MB/s read speed (about 4x faster).

----------

## Ant P.

Do you get the same speeds testing the whole disk and a single partition?

----------

## axl

https://ssd.userbenchmark.com/SpeedTest/447790/PLEXTOR-PX-512M9PeG

I trust this site, because... it's pretty much impossible to lie. 

500 sounds a bit low. but then again nobody is holding you back to install wine and run some of the windows benchmarks. 

just beware, at cached (memory) benchmarks, linux will outscore offthefreakingcharts numbers because of it's internal cache system... but on the other hand, hdparm is not a reliable benchmark. 

personally I have an intel 750 series. which was like THE nvme drive 3 years ago. I have 3 of them in fact. And waiting for a 905 octane drive. Uhm, these drives really hit 2200 mark in benchmarks easily. and yet:

```
[root@magdalina:~]# hdparm -t /dev/nvme0n1

/dev/nvme0n1:

 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device

 Timing buffered disk reads: 1686 MB in  3.00 seconds = 561.53 MB/sec

```

it's true, this intel 750 series is installed onto a x58 platform, therefor no pciexpress3 4 lanes. so it works in pciexpress 2.0. hdparm says it only does 561, but hdparm is wrong. 

on the other hand, it's hard to test without having kernel jumping into it. 

```
[root@magdalina:~]# dd if=/dev/nvme0n1 of=/dev/null count=8 bs=1G

8+0 records in

8+0 records out

8589934592 bytes (8.6 GB, 8.0 GiB) copied, 7.86102 s, 1.1 GB/s

Fri Aug 30 - 21:00:51

[root@magdalina:~]# dd if=/dev/nvme0n1 of=/dev/null count=8 bs=1G

8+0 records in

8+0 records out

8589934592 bytes (8.6 GB, 8.0 GiB) copied, 2.29293 s, 3.7 GB/s

Fri Aug 30 - 21:00:55

[root@magdalina:~]# dd if=/dev/nvme0n1 of=/dev/null count=8 bs=1G

8+0 records in

8+0 records out

8589934592 bytes (8.6 GB, 8.0 GiB) copied, 1.88403 s, 4.6 GB/s

Fri Aug 30 - 21:01:02

[root@magdalina:~]# dd if=/dev/nvme0n1 of=/dev/null count=8 bs=1G

8+0 records in

8+0 records out

8589934592 bytes (8.6 GB, 8.0 GiB) copied, 1.59048 s, 5.4 GB/s

Fri Aug 30 - 21:01:05

[root@magdalina:~]# dd if=/dev/nvme0n1 of=/dev/null count=8 bs=1G

8+0 records in

8+0 records out

8589934592 bytes (8.6 GB, 8.0 GiB) copied, 1.5589 s, 5.5 GB/s

Fri Aug 30 - 21:01:08

[root@magdalina:~]# dd if=/dev/nvme0n1 of=/dev/null count=8 bs=1G

8+0 records in

8+0 records out

8589934592 bytes (8.6 GB, 8.0 GiB) copied, 1.59524 s, 5.4 GB/s

```

sweet lovely kernel  :Smile: 

PS my drive, as great as it is, can only do 2.2GB/s. 5.4 ... that's twice as much. and that's the kernel stepping in.

----------

## Anon-E-moose

 *Belliash wrote:*   

> Oh, I forgotten to say anything about the disk. It is PLEXTOR PX-512M9PeG 1.07. It is M.2 PCI-E disk that should have around 2300MB/s read speed (about 4x faster).

 

Ok, a few questions.

1. Are you running it on anything else for speed comparison, ie windows?

2. What is the drive connected to, motherboard m2 slot or pcie card? If pcie card, how many lanes, and how many lanes where it's plugged into?

3. If 2 is motherboard, what's the make/model of the mb?

----------

## ct85711

I was hearing, and also reading, the motherboard's m2 slot shares the number of lanes available from the cpu (where the gpu is taking 8 or 16 lanes by it's self and the rest are used by the pci-e slots).  For one person, he found out his m2 slot uses the same lanes as one of his PCI-E slots.

----------

## Belliash

 *Ant P. wrote:*   

> Do you get the same speeds testing the whole disk and a single partition?

 

Exactly.

 *Anon-E-moose wrote:*   

> 1. Are you running it on anything else for speed comparison, ie windows?

 

On Windows it looks much better.

 *Anon-E-moose wrote:*   

> 2. What is the drive connected to, motherboard m2 slot or pcie card? If pcie card, how many lanes, and how many lanes where it's plugged into?

 

Disk is connected to M.2 slot, directly into motherboard.

 *Anon-E-moose wrote:*   

> 3. If 2 is motherboard, what's the make/model of the mb?

 

It is Gigabyte B450 Aorus Elite.

The M.2 slot shares CPU PCI-E lanes with 2 SATA ports. There are no disks connected to these SATA ports.

Any reliable benchmark on Linux?

----------

## axl

 *Belliash wrote:*   

> Any reliable benchmark on Linux?

 

bonnie++ should do the trick.

----------

## Anon-E-moose

 *Belliash wrote:*   

>  *Anon-E-moose wrote:*   1. Are you running it on anything else for speed comparison, ie windows? 
> 
> On Windows it looks much better.

 

What are you using on windows Crystal Disk Mark (sp) or something else? And what's the speed difference between windows and linux?

 *Quote:*   

> Any reliable benchmark on Linux?

 

fio 

gui wrapper for fio at https://github.com/earlephilhower/ezfio

----------

## Belliash

```
# fio --readonly --direct=1 --fsync_on_close=1 --randrepeat=1 --norandommap --time_based --runtime=60 --filename=/dev/nvme0n1 --name=read-phase --rw=randread

read-phase: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1

fio-3.7

Starting 1 process

Jobs: 1 (f=1): [r(1)][100.0%][r=59.0MiB/s,w=0KiB/s][r=15.1k,w=0 IOPS][eta 00m:00s]

read-phase: (groupid=0, jobs=1): err= 0: pid=11218: Sat Aug 31 12:54:32 2019

   read: IOPS=14.7k, BW=57.5MiB/s (60.3MB/s)(3449MiB/60001msec)

    clat (usec): min=10, max=29365, avg=67.37, stdev=108.59

     lat (usec): min=10, max=29365, avg=67.44, stdev=108.59

    clat percentiles (usec):

     |  1.00th=[   13],  5.00th=[   13], 10.00th=[   13], 20.00th=[   15],

     | 30.00th=[   78], 40.00th=[   79], 50.00th=[   80], 60.00th=[   82],

     | 70.00th=[   83], 80.00th=[   94], 90.00th=[   95], 95.00th=[   97],

     | 99.00th=[  100], 99.50th=[  105], 99.90th=[  930], 99.95th=[ 1598],

     | 99.99th=[ 3064]

   bw (  KiB/s): min=33744, max=61904, per=99.98%, avg=58847.46, stdev=4644.33, samples=119

   iops        : min= 8436, max=15476, avg=14711.87, stdev=1161.08, samples=119

  lat (usec)   : 20=28.82%, 50=0.26%, 100=69.93%, 250=0.71%, 500=0.05%

  lat (usec)   : 750=0.01%, 1000=0.16%

  lat (msec)   : 2=0.03%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%

  cpu          : usr=1.67%, sys=2.74%, ctx=882909, majf=0, minf=10

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=882885,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=57.5MiB/s (60.3MB/s), 57.5MiB/s-57.5MiB/s (60.3MB/s-60.3MB/s), io=3449MiB (3616MB), run=60001-60001msec

Disk stats (read/write):

  nvme0n1: ios=881309/0, merge=0/0, ticks=57654/0, in_queue=1214, util=98.19%
```

```
# fio --readonly --direct=1 --fsync_on_close=1 --randrepeat=1 --norandommap --time_based --runtime=60 --filename=/dev/nvme0n1 --name=read-phase --rw=randread  --bs=64k --numjobs=16 --iodepth=4 

read-phase: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=4

...

fio-3.7

Starting 16 processes

Jobs: 16 (f=16): [r(16)][100.0%][r=1269MiB/s,w=0KiB/s][r=20.3k,w=0 IOPS][eta 00m:00s]

read-phase: (groupid=0, jobs=1): err= 0: pid=11272: Sat Aug 31 12:56:20 2019

   read: IOPS=1297, BW=81.1MiB/s (85.0MB/s)(4864MiB/60001msec)

    clat (usec): min=43, max=125903, avg=770.12, stdev=1909.47

     lat (usec): min=43, max=125903, avg=770.22, stdev=1909.47

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   78], 20.00th=[  102],

     | 30.00th=[  506], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[ 1004], 95.00th=[ 1221],

     | 99.00th=[ 6128], 99.50th=[ 9896], 99.90th=[25035], 99.95th=[35914],

     | 99.99th=[74974]

   bw (  KiB/s): min= 1408, max=111488, per=6.25%, avg=83004.28, stdev=27653.93, samples=120

   iops        : min=   22, max= 1742, avg=1296.91, stdev=432.13, samples=120

  lat (usec)   : 50=1.01%, 100=18.37%, 250=7.99%, 500=2.27%, 750=35.27%

  lat (usec)   : 1000=25.02%

  lat (msec)   : 2=7.17%, 4=1.20%, 10=1.21%, 20=0.34%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.28%, sys=0.57%, ctx=77830, majf=0, minf=27

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77825,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11273: Sat Aug 31 12:56:20 2019

   read: IOPS=1299, BW=81.2MiB/s (85.2MB/s)(4873MiB/60001msec)

    clat (usec): min=41, max=151836, avg=768.69, stdev=1948.73

     lat (usec): min=42, max=151836, avg=768.78, stdev=1948.73

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   77], 20.00th=[  102],

     | 30.00th=[  506], 40.00th=[  611], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6194], 99.50th=[ 9765], 99.90th=[23987], 99.95th=[38536],

     | 99.99th=[71828]

   bw (  KiB/s): min= 1280, max=109184, per=6.26%, avg=83160.43, stdev=27708.31, samples=120

   iops        : min=   20, max= 1706, avg=1299.35, stdev=432.97, samples=120

  lat (usec)   : 50=1.14%, 100=18.35%, 250=7.81%, 500=2.31%, 750=35.67%

  lat (usec)   : 1000=24.87%

  lat (msec)   : 2=7.00%, 4=1.13%, 10=1.24%, 20=0.33%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.25%, sys=0.54%, ctx=77984, majf=0, minf=26

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77971,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11274: Sat Aug 31 12:56:20 2019

   read: IOPS=1300, BW=81.3MiB/s (85.2MB/s)(4878MiB/60001msec)

    clat (usec): min=42, max=140422, avg=767.92, stdev=1905.70

     lat (usec): min=42, max=140423, avg=768.01, stdev=1905.71

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   78], 20.00th=[  101],

     | 30.00th=[  506], 40.00th=[  603], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 5997], 99.50th=[10028], 99.90th=[24511], 99.95th=[36963],

     | 99.99th=[67634]

   bw (  KiB/s): min= 1408, max=109696, per=6.27%, avg=83244.57, stdev=27781.72, samples=120

   iops        : min=   22, max= 1714, avg=1300.68, stdev=434.11, samples=120

  lat (usec)   : 50=0.97%, 100=18.76%, 250=7.77%, 500=2.19%, 750=35.49%

  lat (usec)   : 1000=24.95%

  lat (msec)   : 2=7.00%, 4=1.15%, 10=1.21%, 20=0.35%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.29%, sys=0.56%, ctx=78053, majf=0, minf=25

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=78047,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11275: Sat Aug 31 12:56:20 2019

   read: IOPS=1300, BW=81.3MiB/s (85.2MB/s)(4876MiB/60001msec)

    clat (usec): min=42, max=117887, avg=768.21, stdev=1885.35

     lat (usec): min=43, max=117887, avg=768.30, stdev=1885.36

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   77], 20.00th=[  101],

     | 30.00th=[  506], 40.00th=[  603], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[10159], 99.90th=[23987], 99.95th=[38536],

     | 99.99th=[70779]

   bw (  KiB/s): min= 1408, max=110208, per=6.25%, avg=83075.18, stdev=27785.89, samples=119

   iops        : min=   22, max= 1722, avg=1298.04, stdev=434.15, samples=119

  lat (usec)   : 50=1.14%, 100=18.51%, 250=7.76%, 500=2.29%, 750=35.48%

  lat (usec)   : 1000=24.94%

  lat (msec)   : 2=6.99%, 4=1.19%, 10=1.19%, 20=0.36%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.24%, sys=0.55%, ctx=78023, majf=0, minf=24

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=78020,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11276: Sat Aug 31 12:56:20 2019

   read: IOPS=1293, BW=80.8MiB/s (84.8MB/s)(4850MiB/60001msec)

    clat (usec): min=41, max=128476, avg=772.39, stdev=1916.54

     lat (usec): min=41, max=128476, avg=772.48, stdev=1916.54

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   77], 20.00th=[  102],

     | 30.00th=[  515], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6128], 99.50th=[10159], 99.90th=[24249], 99.95th=[39060],

     | 99.99th=[63701]

   bw (  KiB/s): min= 1152, max=111232, per=6.23%, avg=82763.22, stdev=27725.07, samples=120

   iops        : min=   18, max= 1738, avg=1293.13, stdev=433.26, samples=120

  lat (usec)   : 50=1.21%, 100=18.27%, 250=7.42%, 500=2.41%, 750=35.79%

  lat (usec)   : 1000=24.94%

  lat (msec)   : 2=7.06%, 4=1.19%, 10=1.20%, 20=0.35%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.26%, sys=0.49%, ctx=77600, majf=0, minf=26

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77599,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11277: Sat Aug 31 12:56:20 2019

   read: IOPS=1297, BW=81.1MiB/s (85.0MB/s)(4865MiB/60001msec)

    clat (usec): min=42, max=134022, avg=769.94, stdev=1901.19

     lat (usec): min=42, max=134022, avg=770.03, stdev=1901.20

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   78], 20.00th=[  102],

     | 30.00th=[  506], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[10290], 99.90th=[23987], 99.95th=[37487],

     | 99.99th=[63177]

   bw (  KiB/s): min= 1664, max=109952, per=6.25%, avg=83023.91, stdev=27883.30, samples=120

   iops        : min=   26, max= 1718, avg=1297.21, stdev=435.70, samples=120

  lat (usec)   : 50=1.05%, 100=18.37%, 250=7.96%, 500=2.26%, 750=35.29%

  lat (usec)   : 1000=25.10%

  lat (msec)   : 2=7.10%, 4=1.17%, 10=1.17%, 20=0.37%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.25%, sys=0.60%, ctx=77852, majf=0, minf=26

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77843,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11278: Sat Aug 31 12:56:20 2019

   read: IOPS=1295, BW=80.0MiB/s (84.9MB/s)(4860MiB/60001msec)

    clat (usec): min=41, max=116681, avg=770.76, stdev=1906.35

     lat (usec): min=41, max=116681, avg=770.85, stdev=1906.36

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   77], 20.00th=[  102],

     | 30.00th=[  515], 40.00th=[  611], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6128], 99.50th=[10028], 99.90th=[24773], 99.95th=[41157],

     | 99.99th=[71828]

   bw (  KiB/s): min= 1408, max=107904, per=6.24%, avg=82938.52, stdev=27811.22, samples=120

   iops        : min=   22, max= 1686, avg=1295.89, stdev=434.57, samples=120

  lat (usec)   : 50=1.19%, 100=18.33%, 250=7.56%, 500=2.29%, 750=35.82%

  lat (usec)   : 1000=24.90%

  lat (msec)   : 2=7.03%, 4=1.17%, 10=1.21%, 20=0.34%, 50=0.13%

  lat (msec)   : 100=0.03%, 250=0.01%

  cpu          : usr=0.24%, sys=0.52%, ctx=77771, majf=0, minf=27

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77760,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11279: Sat Aug 31 12:56:20 2019

   read: IOPS=1293, BW=80.9MiB/s (84.8MB/s)(4852MiB/60001msec)

    clat (usec): min=43, max=148651, avg=771.99, stdev=1904.92

     lat (usec): min=43, max=148651, avg=772.08, stdev=1904.93

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   77], 20.00th=[  101],

     | 30.00th=[  510], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[ 1004], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[10290], 99.90th=[25297], 99.95th=[36439],

     | 99.99th=[62653]

   bw (  KiB/s): min= 1408, max=107904, per=6.23%, avg=82803.72, stdev=27632.84, samples=120

   iops        : min=   22, max= 1686, avg=1293.77, stdev=431.82, samples=120

  lat (usec)   : 50=1.04%, 100=18.56%, 250=7.60%, 500=2.20%, 750=35.48%

  lat (usec)   : 1000=25.11%

  lat (msec)   : 2=7.13%, 4=1.21%, 10=1.15%, 20=0.36%, 50=0.15%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.23%, sys=0.58%, ctx=77649, majf=0, minf=27

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77637,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11280: Sat Aug 31 12:56:20 2019

   read: IOPS=1296, BW=81.0MiB/s (84.0MB/s)(4863MiB/60001msec)

    clat (usec): min=42, max=118743, avg=770.31, stdev=1914.02

     lat (usec): min=42, max=118743, avg=770.41, stdev=1914.02

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   78], 20.00th=[  102],

     | 30.00th=[  510], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[10159], 99.90th=[25297], 99.95th=[36963],

     | 99.99th=[72877]

   bw (  KiB/s): min= 1280, max=112000, per=6.25%, avg=82984.51, stdev=27702.67, samples=120

   iops        : min=   20, max= 1750, avg=1296.61, stdev=432.89, samples=120

  lat (usec)   : 50=0.97%, 100=18.34%, 250=8.04%, 500=2.28%, 750=35.42%

  lat (usec)   : 1000=25.01%

  lat (msec)   : 2=7.09%, 4=1.16%, 10=1.20%, 20=0.34%, 50=0.14%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.25%, sys=0.60%, ctx=77814, majf=0, minf=26

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77806,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11281: Sat Aug 31 12:56:20 2019

   read: IOPS=1298, BW=81.1MiB/s (85.1MB/s)(4869MiB/60001msec)

    clat (usec): min=39, max=114181, avg=769.35, stdev=1890.77

     lat (usec): min=39, max=114181, avg=769.44, stdev=1890.78

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   77], 20.00th=[  101],

     | 30.00th=[  510], 40.00th=[  611], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[10159], 99.90th=[24249], 99.95th=[40109],

     | 99.99th=[64750]

   bw (  KiB/s): min= 1408, max=114048, per=6.25%, avg=83088.97, stdev=27794.20, samples=120

   iops        : min=   22, max= 1782, avg=1298.23, stdev=434.31, samples=120

  lat (usec)   : 50=1.22%, 100=18.45%, 250=7.57%, 500=2.32%, 750=35.61%

  lat (usec)   : 1000=24.99%

  lat (msec)   : 2=6.97%, 4=1.17%, 10=1.19%, 20=0.36%, 50=0.12%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.26%, sys=0.50%, ctx=77914, majf=0, minf=24

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77904,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11282: Sat Aug 31 12:56:20 2019

   read: IOPS=1298, BW=81.2MiB/s (85.1MB/s)(4869MiB/60001msec)

    clat (usec): min=42, max=118960, avg=769.33, stdev=1896.79

     lat (usec): min=42, max=118960, avg=769.42, stdev=1896.80

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   77], 20.00th=[  102],

     | 30.00th=[  506], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[ 1004], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[ 9896], 99.90th=[25035], 99.95th=[36963],

     | 99.99th=[69731]

   bw (  KiB/s): min= 1408, max=108928, per=6.24%, avg=82959.68, stdev=27726.43, samples=119

   iops        : min=   22, max= 1702, avg=1296.20, stdev=433.28, samples=119

  lat (usec)   : 50=1.08%, 100=18.44%, 250=7.81%, 500=2.35%, 750=35.29%

  lat (usec)   : 1000=25.04%

  lat (msec)   : 2=7.10%, 4=1.18%, 10=1.21%, 20=0.35%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.26%, sys=0.57%, ctx=77914, majf=0, minf=25

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77906,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11283: Sat Aug 31 12:56:20 2019

   read: IOPS=1295, BW=80.9MiB/s (84.9MB/s)(4856MiB/60001msec)

    clat (usec): min=42, max=136348, avg=771.35, stdev=1925.40

     lat (usec): min=42, max=136348, avg=771.44, stdev=1925.41

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   78], 20.00th=[  102],

     | 30.00th=[  515], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[ 1004], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[10028], 99.90th=[24249], 99.95th=[37487],

     | 99.99th=[66323]

   bw (  KiB/s): min= 1408, max=110208, per=6.24%, avg=82874.18, stdev=27661.09, samples=120

   iops        : min=   22, max= 1722, avg=1294.87, stdev=432.26, samples=120

  lat (usec)   : 50=0.98%, 100=18.31%, 250=7.87%, 500=2.20%, 750=35.80%

  lat (usec)   : 1000=24.78%

  lat (msec)   : 2=7.18%, 4=1.15%, 10=1.20%, 20=0.35%, 50=0.13%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.25%, sys=0.58%, ctx=77710, majf=0, minf=28

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77703,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11284: Sat Aug 31 12:56:20 2019

   read: IOPS=1295, BW=80.0MiB/s (84.9MB/s)(4859MiB/60001msec)

    clat (usec): min=43, max=143698, avg=771.01, stdev=1952.22

     lat (usec): min=43, max=143698, avg=771.10, stdev=1952.23

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   78], 20.00th=[  102],

     | 30.00th=[  510], 40.00th=[  611], 50.00th=[  668], 60.00th=[  725],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[ 9896], 99.90th=[23725], 99.95th=[36963],

     | 99.99th=[80217]

   bw (  KiB/s): min= 1664, max=109824, per=6.23%, avg=82788.69, stdev=27824.95, samples=119

   iops        : min=   26, max= 1716, avg=1293.55, stdev=434.80, samples=119

  lat (usec)   : 50=1.04%, 100=18.39%, 250=7.84%, 500=2.22%, 750=35.58%

  lat (usec)   : 1000=25.01%

  lat (msec)   : 2=7.05%, 4=1.17%, 10=1.22%, 20=0.33%, 50=0.14%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.23%, sys=0.60%, ctx=77748, majf=0, minf=27

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77736,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11285: Sat Aug 31 12:56:20 2019

   read: IOPS=1296, BW=81.1MiB/s (84.0MB/s)(4863MiB/60001msec)

    clat (usec): min=43, max=91875, avg=770.23, stdev=1857.36

     lat (usec): min=43, max=91875, avg=770.32, stdev=1857.37

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   78], 20.00th=[  102],

     | 30.00th=[  510], 40.00th=[  611], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[ 1004], 95.00th=[ 1221],

     | 99.00th=[ 6128], 99.50th=[10159], 99.90th=[24249], 99.95th=[39584],

     | 99.99th=[65799]

   bw (  KiB/s): min= 1408, max=110208, per=6.25%, avg=82996.65, stdev=27760.81, samples=120

   iops        : min=   22, max= 1722, avg=1296.79, stdev=433.80, samples=120

  lat (usec)   : 50=1.01%, 100=18.32%, 250=7.92%, 500=2.30%, 750=35.72%

  lat (usec)   : 1000=24.69%

  lat (msec)   : 2=7.20%, 4=1.15%, 10=1.19%, 20=0.34%, 50=0.14%

  lat (msec)   : 100=0.02%

  cpu          : usr=0.22%, sys=0.59%, ctx=77823, majf=0, minf=25

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77815,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11286: Sat Aug 31 12:56:20 2019

   read: IOPS=1302, BW=81.4MiB/s (85.4MB/s)(4884MiB/60001msec)

    clat (usec): min=41, max=113798, avg=766.94, stdev=1871.62

     lat (usec): min=41, max=113798, avg=767.04, stdev=1871.63

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   78], 20.00th=[  101],

     | 30.00th=[  502], 40.00th=[  603], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[ 1004], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[ 9896], 99.90th=[24249], 99.95th=[40109],

     | 99.99th=[67634]

   bw (  KiB/s): min= 1536, max=107776, per=6.27%, avg=83351.33, stdev=27704.08, samples=120

   iops        : min=   24, max= 1684, avg=1302.35, stdev=432.90, samples=120

  lat (usec)   : 50=1.05%, 100=18.60%, 250=7.97%, 500=2.26%, 750=35.44%

  lat (usec)   : 1000=24.70%

  lat (msec)   : 2=7.09%, 4=1.14%, 10=1.25%, 20=0.34%, 50=0.12%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.24%, sys=0.57%, ctx=78154, majf=0, minf=24

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=78147,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

read-phase: (groupid=0, jobs=1): err= 0: pid=11287: Sat Aug 31 12:56:20 2019

   read: IOPS=1298, BW=81.2MiB/s (85.1MB/s)(4871MiB/60001msec)

    clat (usec): min=41, max=120616, avg=769.05, stdev=1900.12

     lat (usec): min=42, max=120616, avg=769.15, stdev=1900.13

    clat percentiles (usec):

     |  1.00th=[   50],  5.00th=[   61], 10.00th=[   78], 20.00th=[  101],

     | 30.00th=[  506], 40.00th=[  611], 50.00th=[  668], 60.00th=[  717],

     | 70.00th=[  783], 80.00th=[  857], 90.00th=[  996], 95.00th=[ 1221],

     | 99.00th=[ 6063], 99.50th=[10421], 99.90th=[23987], 99.95th=[36439],

     | 99.99th=[71828]

   bw (  KiB/s): min= 1280, max=110848, per=6.25%, avg=82975.72, stdev=27742.72, samples=119

   iops        : min=   20, max= 1732, avg=1296.45, stdev=433.53, samples=119

  lat (usec)   : 50=1.11%, 100=18.49%, 250=7.74%, 500=2.27%, 750=35.66%

  lat (usec)   : 1000=24.83%

  lat (msec)   : 2=7.03%, 4=1.17%, 10=1.19%, 20=0.38%, 50=0.12%

  lat (msec)   : 100=0.02%, 250=0.01%

  cpu          : usr=0.30%, sys=0.47%, ctx=77948, majf=0, minf=26

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=77932,0,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):

   READ: bw=1298MiB/s (1361MB/s), 80.8MiB/s-81.4MiB/s (84.8MB/s-85.4MB/s), io=76.0GiB (81.6GB), run=60001-60001msec

Disk stats (read/write):

  nvme0n1: ios=1243163/0, merge=0/0, ticks=951598/0, in_queue=329634, util=83.89%
```

What do you think?

----------

## Anon-E-moose

Interesting statistics, 

What is your queue depth -- "cat /sys/block/<dev>/device/queue_depth "

Edit to add: this is about tuning for intel but many things will still apply for you https://itpeernetwork.intel.com/tuning-performance-intel-optane-ssds-linux-operating-systems/

ETA2: Also how do you have the disk set up? Starting block and sector size.

----------

## Belliash

```
# cat /sys/block/nvme0n1/device/queue_depth

cat: /sys/block/nvme0n1/device/queue_depth: No such file or directory
```

```
# fdisk -l /dev/nvme0n1

Disk /dev/nvme0n1: 476.96 GiB, 512110190592 bytes, 1000215216 sectors

Disk model: PLEXTOR PX-512M9PeG                     

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x94bb23f5

Device         Boot  Start        End   Sectors   Size Id Type

/dev/nvme0n1p1        2048     264191    262144   128M 83 Linux

/dev/nvme0n1p2      264192 1000213167 999948976 476.8G 83 Linux
```

----------

## Anon-E-moose

alignment seems alright, not sure why there's not a queue_depth

does /sys/block/nvme0n1 exist and if so what does "ls /sys/block/nvme0n1/device/" return

----------

## axl

 *Anon-E-moose wrote:*   

> alignment seems alright, not sure why there's not a queue_depth
> 
> does /sys/block/nvme0n1 exist and if so what does "ls /sys/block/nvme0n1/device/" return

 

nvme don't have queue_depth like scsi's. or you can consider it 1024.

----------

## Belliash

 *Anon-E-moose wrote:*   

> alignment seems alright, not sure why there's not a queue_depth
> 
> does /sys/block/nvme0n1 exist and if so what does "ls /sys/block/nvme0n1/device/" return

 

```
# ls /sys/block/nvme0n1/device/

address  cntlid  dev  device  firmware_rev  model  numa_node  nvme0n1  power  rescan_controller  reset_controller  serial  state  subsysnqn  subsystem  transport  uevent
```

----------

## Anon-E-moose

https://www.nvmedeveloperdays.com/English/Collaterals/Proceedings/2018/20181204_PRECON2_Hands.pdf has some hints for tuning nvme drives. (don't know how applicable to your drive, but some things will apply)

and lots of links for performance tuning of nvme under linux https://www.google.com/search?complete=0&q=linux+nvme+performance+tuning&sa=X&ved=2ahUKEwisipnQoq3kAhUJPq0KHTcyB7wQ1QIoAHoECAsQAQ&biw=1400&bih=853

Or just search on "linux nvme performance tuning" google, ddg, etc

Edit to add: There seem to be so many things that can affect the speed under linux, that there's not set formula for speeding it up, lots of trial and error.

But I am interested if you can figure it out, as I will eventually get an nvme drive.

----------

## axl

he got like 1500 MB/s in his test with fio. my intel 750 (in pcix 2.0 instead of 3) got 1600ish. it's about right. 2000 range is more expensive. 

i took the time to lookup his hdd on userbenchmark. 

https://ssd.userbenchmark.com/SpeedTest/447790/PLEXTOR-PX-512M9PeG

once again, try windows benchmarks on linux using wine. INCLUDING but not only limited to: what moose said - crystal disk mark, and more importantly the self testing utility from the page I posted. 

I love userbenchmark.com (not affiliated). It's a auto-self-serving site. People test their rig, stats go up on the site... u get to compare your hardware with other builds and setups. Like in the case of this specific harddisk, just click the link... it's speaking for itself. 

What may not be clear, is that their test client runs just fine on wine. And also it may not be clear that there is a client. Just click on test. And then compare with other builds of your type. 

If you browse userbenchmark long enough, you will figure out at this point in time the top ssd disks are: 1, 2 and 3 intel optane, 4 and 5 some apple devices (which I think are optane dressed as apple) and then intel 750 series. which is old but gold. the 2000 mark is not easy or cheap to beat.

EDIT: also I can't imagine having to tweak my nvme to increase performance. IT IS performance.

----------

## Belliash

 *axl wrote:*   

> he got like 1500 MB/s in his test with fio. my intel 750 (in pcix 2.0 instead of 3) got 1600ish. it's about right. 2000 range is more expensive. 
> 
> i took the time to lookup his hdd on userbenchmark. 
> 
> https://ssd.userbenchmark.com/SpeedTest/447790/PLEXTOR-PX-512M9PeG
> ...

 

This is not my disk  :Smile: 

I own this one: https://www.goplextor.com/Product/Detail/M9Pe(G)#/Features

----------

## axl

don't look at the picture on userbenchmark. i copied and pasted from your fdisk info. PLEXTOR PX-512M9PeG. HEY!!! I did my best; but really if that is not it (not based on picture), what is the right one?

----------

## axl

just run their tiny test app. again, it works just fine on wine. and it will put everything into much deeper perspective. if you can, run it twice. once in windows, once in linux. post links so we can compare. after you run their benchmark, you will get a link. post those. i'm curious as hell now that i'm invested.

----------

## Belliash

```
-----------------------------------------------------------------------

CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo

                          Crystal Dew World : https://crystalmark.info/

-----------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 64,T=12) :  5933.590 MB/s

  Sequential Write (Q= 64,T=12) :  2387.699 MB/s

  Random Read 4KiB (Q=  8,T=12) :   938.439 MB/s [ 229111.1 IOPS]

 Random Write 4KiB (Q=  8,T=12) :   249.863 MB/s [  61001.7 IOPS]

  Random Read 4KiB (Q= 32,T=12) :   616.124 MB/s [ 150420.9 IOPS]

 Random Write 4KiB (Q= 32,T=12) :   286.887 MB/s [  70040.8 IOPS]

  Random Read 4KiB (Q=  1,T=12) :   758.085 MB/s [ 185079.3 IOPS]

 Random Write 4KiB (Q=  1,T=12) :   218.801 MB/s [  53418.2 IOPS]

  Test : 1024 MiB [C: 4.9% (17.7/359.4 GiB)] (x5)  [Interval=3 sec]

  Date : 2019/08/31 18:43:26

    OS : Windows 10 Professional [10.0 Build 18362] (x64)
```

Maybe its not so bad.... launched it on Windows 10 running under KVM.

----------

## Anon-E-moose

Not bad. Yeah, axl, fio showed not too shabby results for single, even a little faster with the 16 processes running (close to the 2000 range)

----------

