I have a Z6 HDD pool with 6 x 18T drives and a SLOG, LZ4 compression is enabled by default. Now I need to store large amount of small files and I'm worried about fragmentation.
The files:
- 70K files totaling 70GB
- the sparse part is mostly text files
- they cannot be compressed at application level as they are synced from a remote
- they are sitting on my NVMe pool now with LZ4 enabled, the compression ratio is 0.7, but they need to be moved to my HDD pool.
Here are the options:
- just dump them into the main dataset and hope LZ4 make sure they don't mess up fragmentation level too much.
- create a separate dataset with more aggressive compression.
- create a zvol, this will solve the fragmentation problem but adds overhead and complexity.
Should I worried about fragmentation brought by those small files even with LZ4 enabled?