When a Small Change Writes Too Much – How A Linux Kernel Improvement Can Strike You in PXC/Galera Workloads

We’ve recently been dealing with an interesting customer case where a PXC version upgrade from 5.7 to 8.0 seemed to trigger a completely unexpected problem. Even though the workload and MySQL settings did not change, all PXC members began exhibiting sparse but regular bursts of massive disk writes. Yet the MySQL side workload statistics that we gathered during those bursts did not explain the volume of them at all!

Analyzing these bursts further with pidstat and iostat confirmed that the mysqld process was indeed the sole source of excessive disk writes, and digging further showed the responsible threads to be the Galera receiver thread responsible for subscribing to the replication stream, as well as the main connection threads on the writer node, or Galera applier threads on the secondary members. And what’s most important is that there was a huge disparity between InnoDB engine threads writes (redo, undo, tablespace, etc) and binary log writes versus what was accounted to the Galera threads.

At first, it looked like during the burst, several GB of write-sets were received and applied, yet InnoDB barely changed any pages. All InnoDB-related write activity was completely flat, so they did not participate in these high IO load spikes at all! An example mysqld per-thread breakdown for a few-second burst looked like this:

The quick thought on the small redo log vs potentially huge writeset volume was that maybe transactions changing a single column would bloat the binlog events with other large blob/text/json columns, due to binlog_row_image=full, as the ROW image format applies to both Galera writesets and binary logs. But, analyzing the binary logs (with binlogsum) showed moderate and completely flat volume, nothing related to those bursts.

Looking for clues, I asked myself, so what are those Galera threads exactly writing? The receiver thread copies all replicated writesets into the Galera cache file(s), but also reclaims the space when the ring wraps (the ring-buffer allocator walks forward from the oldest buffers and discards released ones).

Then, the applier or connection threads also update the cache according to the transaction’s state. Still, when there are no spikes with big writesets (MySQL write load stays flat), why would we see such massive IO write bursts only accounted to the Galera threads?

And why would that weird behavior start to happen after the PXC upgrade? We could not find any clues in the source code; there were no changes to the Galera plugin or wsrep extensions that would explain these bursts.

Later, we found out additional information, which turned out to be crucial – not only the PXC version, but also the OS was upgraded! Now, the previous one: Debian 11, uses Linux 5.10 LTS kernel series, while the upgraded version 12 uses the Linux 6.1 LTS kernel series. And as the main suspect at this point was the Gcache store, how would the new kernel change the way this file is handled, though? It turns out that it does indeed, and it is related to the fact that the Galera plugin uses the mmap() path to handle the cache!

Page Folios in Modern Linux Kernels

In pursuit of handling large data on machines with big RAM and disk space more efficiently in Linux, in addition to huge pages, something called page folios was introduced in 5.16. What is relevant to this case is that large folios were implemented to also handle the page cache, initially only for the XFS filesystem in kernel 5.18, and yes, the affected cluster kept Galera cache on the XFS partition. Then, large folios support is what makes the kernel 6.1 different from the previous 5.10.

OK, but why would “folios” cause bursts, and why do they not affect InnoDB writes, but only the Galera threads? Now, unlike any other data or log files in MySQL (except TempTable storage) or MariaDB, Galera cache files do not use the write() or pwrite() path, but mmap rather.

This difference pointed me to some reports complaining about the systemd journal’s excessive IO problems, which turns out to be using mmap() path as well!

Funny enough, the problem is not very well known, and the root cause is still being confused to this day, even in the most affected systemd community. Some reports describe the journal IO pressure being higher than expected, but never define the real culprit. The systemd issue that currently covers this problem best is:
https://github.com/systemd/systemd/issues/40262
Related kernel thread discussions:
https://lwn.net/Articles/1092530/
https://lists.openwall.net/linux-kernel/2026/08/20/1169
And despite many good blog posts about optimizing journal IO, I found only this one that points to the folio impact:
https://www.cnblogs.com/32bin/p/22929557
The problem even reached the general tech press, as still unsolved in 2026:
https://northeasttimes.com/2026/08/14/a-single-log-line-eats-49kb-of-disk-and-linux-admins-are-furious/

In short: on a large folios-enabled kernel and filesystem, page cache entries are grouped in folios. In early kernel implementations (5.18–6.5), write()/pwrite() paths were also affected, but InnoDB typically uses O_DIRECT for table spaces (so not using page cache at all), and redo and binlog use small appends, which create order-0 (4 kB) small folios. Then in Linux 6.6, a patch was implemented that added per-block dirty tracking, as now iomap records the overlapping filesystem blocks (filesystem-block granularity): see RFCv3 for details. With mmap writes, any update only tracks a whole folio. Thus, writeback page cache iteration, in the mmap case, will sync the whole folio (up to 2MB) instead of the exact fs block (4kB). In other words, the mmap() path cannot benefit from the improved I/O tracking loops in 6.6+.

So, yes, the difference may be potentially huge, and small random writes to a mmapped file can induce huge IO bursts when the related page cache gets flushed to disk.

There are important variables here:

• • Readahead size via optimal_io_size → read_ahead_kb, per block device, influences the sizes of large folios allocated through it (from 128kB up to even 2MB)
  • Bursts are specifically periodic due to how writeback of dirty data gets invoked – every dirty_writeback_centisecs (default 5 seconds) the kernel checks if the dirty data (tracked per inode) age is old enough via dirty_expire_centisecs (default 30 seconds) – the observed roughly 33-second bursts on the affected PXC nodes, are consistent with this mechanism (although these settings do not guarantee a fixed flush period) when the object, here the Gcache inode, gets flushed when receiving regular writes.
  • Workload characteristics matter a lot – writes accumulated within a single folio will result in way smaller write amplifications as compared to a file being updated in many random places, hence touching many different large folios.

The whole concept is damn complex, so let me share a more technical deep dive schematic in a separate document. Bear in mind, I am not a Linux kernel expert, so I hope there are not too many errors. The document is available here: https://przemekmalkowski.github.io/folios-mmap/

Galera Cache Impact

Galera’s writeset cache has been historically a mmapped file from the very beginning. And there are a couple of factors that make the large folio-induced write amplifications particularly severe here:

• The cache file is by default 128MB, but very often a much bigger size is used to provide long enough IST coverage.
• In addition to sequential appends of new writesets, there are many small and random updates that change their status to RELEASED when the transaction is already applied and ready to be overwritten.
• Quite often, MySQL data, together with the galera.cache file, is placed on a large storage, like RAID arrays. Specifically for software RAID, the Linux kernel automatically increases the readahead size proportionally to the array geometry and the advertised optimal I/O size. For hardware RAID and dm/LVM stacks, it may be the case too. This leads to even larger folios, and that’s exactly what happened for the said customer case.
• During the PXC member startup, the whole gcache file is scanned sequentially during the recovery process (enabled by default), which can immediately populate the large page cache areas.

Having said that, every PXC/Galera cluster may be affected by extra write amplifications for the galera.cache (and gcache.page.X), if kernel and filesystem versions support large folios. But the impact may be small enough to slip through unnoticed when the default readahead of 128kB is set for the disk block device. The extra IO write pressure may be wrongly assumed to be related to regular (and complex enough) database writes. But when the storage uses bigger readahead, and large folios get to even 2MB in size, the amplification is huge and really difficult to miss.

Possible workarounds

As currently there is no Linux kernel side solution implemented for the mmap() path, there are a couple of workarounds possible, and they apply to any applications using mmapped files, not only PXC/Galera:

• Depending on the kernel version, using the EXT4 filesystem may be an option, as folio support was introduced there much later (in 6.16) than in XFS (5.18) – if applicable, this allows us to avoid the said amplifications altogether. It helped in this customer’s case!
• Lower the readahead size for the disk device – for the price of potentially limited performance, the maximum size of large folios will be impacted by it, making the amplifications smaller.
• Modify the application mmap calls to use MADV_RANDOM via madvise() hints. It will basically disable readahead via enforced random access.
• Stop using mmap() path in favor of pwrite() on Linux 6.6+.
• Finally, use a customized modern kernel, compiled without THP support via the disabled CONFIG_TRANSPARENT_HUGEPAGE option – large folios are not created on recent kernels when THP is not available. (Note that disabling THP in GRUB is not enough to disable large folios!)

For PXC, I reported the problem here: https://perconadev.atlassian.net/browse/PXC-5340

How to check if you’re affected

• Kernel and filesystem: XFS on 5.18+, or ext4 on 6.16+.
• Readahead size: cat /sys/block/<dev>/queue/{read_ahead_kb,optimal_io_size}.
• Large mappings: FilePmdMapped in /proc/meminfo or smaps for the gcache mapping, and thp_file_mapped climbing in /proc/vmstat.
• Bursts: about every 30–35 s (unless different timers in vm.dirty_expire_centisecs and vm.dirty_writeback_centisecs) in iostat, with per-thread pidstat -d charges in up to 2 MB multiples.

More on Large Folios Impact Cases

During research, I found another interesting story. It turns out that mixing active write workloads using both page cache and bypassing it with O_DIRECT may lead to additional unexpected stalls. This time it was visible with RocksDB, as described here:

https://blog.wegenz.in/posts/page-cache-reclaim-foliotune-direct-io-stalemate

The article was created by a human.