Ancestors

Toot

Written by Tomas Vondra on 2025-01-20 at 12:01

A couple weeks ago I upgraded one my home lab machines I use for benchmarking. Aside from getting a new Ryzen, the main change was getting a bunch of Samsung 990 PRO M.2 SSDs for storage. And the performance I'm seeing is rather atrocious :-(

For most workloads (reads/writes, sequential/random) it works fine, e.g. for random writes I can get ~50k IOPS (iodepth=1) quite easily. But as soon as I ask fio to do fsyncs (fsync=1), the performance just absolutely tanks to ~200 IOPS.

=> More informations about this toot | More toots from tomasv@fosstodon.org

Descendants

Written by Tomas Vondra on 2025-01-20 at 12:01

Of course, fsync=1 is expected to have some impact. Every SSD I own shows the same effect, but never this bad. For example:

WD Ultrastar SN640: 50k => 40k IOPS

Intel DC S3700 (6x RAIO0): 24k => 6k IOPS

So the drop 50k => 200 IOPS seems terrible. What am I doing wrong? Any suggestions for a better option (device/model) - still affordable, but handling fsync reasonably?

=> More informations about this toot | More toots from tomasv@fosstodon.org

Written by AndresFreundTec on 2025-01-20 at 14:53

@tomasv FWIW, I would actually have trouble believing that the SN640 is doing correct fsyncs. That's not a big enough drop IME. It's also far less than what I have observed with newer WD NVMes.

=> More informations about this toot | More toots from AndresFreundTec@mastodon.social

Written by Tomas Vondra on 2025-01-20 at 15:01

@AndresFreundTec Not sure, but it's an Ultrastar drive targeted at DC workloads, not a regular consumer drive. Of course, "enterprise" is just a label ...

=> More informations about this toot | More toots from tomasv@fosstodon.org

Written by AndresFreundTec on 2025-01-20 at 16:15

@tomasv Possible. My experience is that either there's basically no drop (because there's a supercap or such that allows writes to be flushed before loosing power, making cache flushes free), or there's a more significant drop than 20%.

=> More informations about this toot | More toots from AndresFreundTec@mastodon.social

Written by AndresFreundTec on 2025-01-20 at 14:38

@tomasv At this point I don't really have a better explanation other than Samsung intentionally slowing down client SSDs to maintain market segmentation. Every generation of Samsung client NVMe is getting slower fsync. And FUA writes are even slower than writes followed by a full-cache-flush.

I've basically given up on using them.

=> More informations about this toot | More toots from AndresFreundTec@mastodon.social

Written by AndresFreundTec on 2025-01-20 at 14:41

@tomasv Just look at the numbers here: https://gist.github.com/anarazel/b527e5317bb7d58483a9858f5f2435ca

Samsung is just atrociously bad.

=> More informations about this toot | More toots from AndresFreundTec@mastodon.social

Written by Tomas Vondra on 2025-01-20 at 14:43

@AndresFreundTec Yeah, seems like that. Any suggestion what other devices to use for a home lab machine?

=> More informations about this toot | More toots from tomasv@fosstodon.org

Written by AndresFreundTec on 2025-01-20 at 14:56

@tomasv I wish I had a good default answer. [Nearly] all the stupid SSD benchmarking sites are not testing durable write performance.

I have a few WD NVMes that I am reasonably happy with - they do heat up a fair bit under load though. I have a few Sabrents that are OK. I have an ADATA, which I am fairly certain is lying about cache flushes, so I'd hesitate.

=> More informations about this toot | More toots from AndresFreundTec@mastodon.social