How to measure cache performance of zfs with cache drive - caching

Question

I am trying to compare different filesystems, most with cache/tiered storage features, but so far it does not seem to work as it should. (btw, I know this might the wrong site, but when I searched for zfs, most SE results were on stackoverflow, so it seemed good to ask here)
When testing zfs, I created a single pool, with a main drive/partition and another drive (ssd) added as a cache. The main drive/partition was around 200 GB, the ssd 120 GB. This showed up correctly in zpool.
Then I ran phoronix test suite with iozone, or iozone separately. After some initial unfamiliarity, I settled on phoronix-test-suite run-default pts/iozone which I than ran on just a hdd, just an ssd and an hdd partition with ssd as a cache. And on two laptops which have ssds for comparisons. In the test with zfs + cache, there was virtually no difference to using just an hdd. It was really really slow. And I made sure to set the working directory to the zpool and verified that the temp file was created there and also checked zpool iostat to make sure that the pool was working.
Now, while I might have suspected lower results, I would hope that the speeds should at least be somewhat slower, especially with an 'easy' test such as this, which just does 3 runs of reading 1 MB records from an 8 GB file, and then 3 runs of writing 1 MB records from an 8 GB file.
Now, maybe because of the way zfs cache and similar ones work - they cannot be captured by such a simple test - but then, which would be a good test to capture the benefit of the cache? However, as the test file fits on the cache ssd easily, why it is not written there first and transferred back to the hdd in the background?
The zpool looks like this:
pool: ztest
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
ztest ONLINE 0 0 0
sdb7 ONLINE 0 0 0
cache
sdc ONLINE 0 0 0
errors: No known data errors

Answer 1

Here are my guesses of what the mismatch in expectation / reality is:
For the read benchmark (3 runs of reading 1 MB records from an 8 GB file)
The ZFS cache device (commonly called the "L2ARC") gets populated when a block is written or read. From your description, I'm guessing that the benchmark writes the file once, then reads it sequentially 3 times. I would expect the L2ARC to make a copy of the blocks on your cache device during the first write, or at the very least when you first read the data. (Although, note that the L2ARC does not yet persist across reboots because the map of what's on disk is only stored in memory -- kind of a silly limitation but probably not what's affecting your test.)
Are you using zfs set secondarycache=all to cache all data blocks, as opposed to just metadata blocks? (Just to disambiguate / explain the naming, the primarycache property has similar settings for the in-RAM cache, aka the "ARC".)
To check if the L2ARC is being used during your benchmark, you can look at arcstat data -- the stats you'll be interested in are:
"l2hits": [6, 1000, "L2ARC hits per second"],
"l2miss": [6, 1000, "L2ARC misses per second"],
With the benchmark you described, I would expect to see a very high hit rate (assuming your SSD is >8GB).
For the write benchmark (3 runs of writing 1 MB records from an 8 GB file)
This will only be helped if you also add an SSD log device (commonly called the "ZIL" like you mentioned in one of the comments). I'd split your SSD into two partitions: one very small to use as the ZIL (only has to store enough data to cache ~10s of writes assuming you haven't tuned the filesystem), and one using the rest of the drive as an L2ARC.
To address the advice you found about not using a ZIL unless you have a big beefy server, I don't think there's any reason not to use a ZIL on a small system. I guess it ties up a little extra SSD that could have been used for a read cache, but it doesn't use extra RAM or a noticeable amount of additional CPU, so effectively it should speed up your write latencies / burst throughput with no adverse side effects.