Bitcoin-core syncing gets slower and slower

[ThatLinuxDude]

This issue has come up in the forums a couple of times, without an exact pinpoint of the cause, but no real solution was ever provided/cause found. Trying it here with a more extensive analysis. I am experienced in Linux, and am debugging this machine for a friend. I have had it in my home lab now for a couple of days, with ssh access.

His problem description and my remote assistance was the same (slow sync after about 70%). After taking the machine in, I re-installed Start9 on the HP Elitedesk G2 800 with 8 GB RAM, and a TB brand-new 2TB SSD. After enabling Bitcoin (transaction index off, txindex=0) within less than 24h I get to 70% syncing, around block 810000, after that syncing speed massively drops.

Around 25% syncing I had a syncing speed of 1% per 20-30 minutes - now at syncing progress of 70% we are talking less than 0.1% per hour.

I have myself a bitcoin node running in my homelab on an old and weaker laptop from 2014, syncing did not took nearly as long nor did it slow down. The syncing speed is definitely not normal. I added my local bitcoin node manually via the config to the Start9 Bitcoin config to speedup block download and rule out slow internet connection speeds. I verified my local bitcoin node has an incoming (non-routable) connection from the Start9 node.

During the slow sync, the host system shows no sign of oversubscription. htop reports Mem usage of less than 1GB out of 8GB, the 4 CPU cores are all hanging around 50% utilization tops.

I’m tracking syncing progress using “journalctl -f”, here is some example output:

Sep 28 15:26:11 musty-girdle bitcoind.embassy[508100]: 2025-09-28T15:26:11Z UpdateTip: new best=0000000000000000000485993e37cd61641cbba93a61793301bff095728b02ff height=811584 version=0x20000000 log2_work=94.468610 tx=904690973 date='2023-10-10T19:58:24Z' progress=0.726480 cache=578.8MiB(4561877txo)

There is steady progress in the journalctl, without any other outputs (except for some health-check podman spawns).

I even increased the dbcache in the Bitcoin config to 4096M, but no noticeable difference. I navigated to the bitcoin data dir at /embassy-data/package-data/volumes/bitcoind/data/main and check the debug.log file - again nothing suspicious.

Now, I am not a docker/podman expert, but from my research (with the Help of AI and some manual source verification) my suspicion is that this is due to the overlayfs configuration in podman/start9.

From the ouput of “podman info” and “podman inspect ” it becomes apparent that the container uses an overlay backed by btfrs, instead of a volume mount. This is also confirmed by using “mount”, which shows the overlay for the container mounted as type overlay:

overlay on /var/lib/containers/storage/overlay/b160dcd534a8affe1e5b57ca559c7d118f1dc41bc6d4854a57c118e943c68afd/merged type overlay

Upon running:
podman info -f '{{index .Store.GraphStatus "Native Overlay Diff"}}'

I get “false”, which means the overlayfs doesn’t use native diffing, and in combination with btrfs podman falls back to using fuse-overlayfs. Performance issues with fuse-overlayfs are widely known and described, especially for larger filesytems. Given that the bitcoin container grows beyond 500GB+, this is mostlikely the case why the syncing gets slower, and slower over time.

This suspicion is also backed up by the fact that the dashboard monitor shows more than 30% utlization (out of 37% total) comes from kernel space. This is very odd as Bitcoin runs in userspace - another pointer that it is the container/overlay abstraction in kernel-space causing the issue (not just bitcoin block validation being slow, which would happen in user space). The screenshot:

Anybody can confirm this? Any ideas? I wonder why noone has brought this up before, since the very same observation of slow syncing has been reported a couple of times. But if my hunch is correct, this is a design flaw of the bitcoin container in start9.

[Rexter]

Your timing is impeccable. This may be a valuable analysis in the context of a test I just ran:

Endless sync BTC Knots

As I have mini-PC Elite LIVA Z2V ECS N4100, 4 GB, eMMC 64 GB, , 8TB Internal SSD (Samsung 870 QVO SATA III 2.5-Inch), linked to a router by the cable LAN, all available for experiments I installed and tried Umbrel 1.4.2 to run BTC Knots. It synced 100% within 5 days with visible progress with CPU 4 threads load variations between 7%-75% and RAM use up to 3 GB from 3,8 available. However after Knots version updated the web UI was lost and the UMBREL 1.2 system became dead … I decided to try DIY…

[ThatLinuxDude]

I just took a quick glance at your thread, looks like the issues are related. I will take a through look at your thread later tonight. I also reformatted the SSD yesterday of the same box and installed PopOS (a ubuntu derivate) and installed bitcoin-core (flatpak). It is syncing way fast so far, and is at 77% in less than 24h. I kept the debug.log from the bitcoin on start9 and will save the debug.log of the bitcoin-core on PopOS and will then try to analyse the block validation timings.

[ThatLinuxDude]

Update: Syncing on the Pop!_OS installation from 0 to 100% took 55h and was successful.

On the Pop!_OS installation (same machine, HP Elitedesk G2 800 with 8GB Mem and 2TB SSD) the syncing is faster and more consistent. There is a slowdown around blocks 750000 and following, so that area is the most interesting.

I created a script and gnuplot file that parse a debug.log file and plot the timestamp of the block validation (Update Tip message in debug.log) and the blockheight of the Tip into a plot.

These are the results:

Note that the debug.log files start and end of Update Tip messages are different, but the overall trend is obvious: The Start9 scatterplot shows that with increased block height, block validation gets slower and slower. The curve resembles a logarithmic curve, so there is an exponential growth of the slowdown.

The Pop!_OS plot (again, we talk the same machine!) is much more linear. there is a gap for about an hour between 15:00 and 16:00 during which I had to powerdown and relocate the machine for unrelated reasons. After restart I continued where it left off. The linear trend is obvious, much more what is expected.

[morelub]

I confirm your conclusions: 100% correct! My experiments resulted with the same outcomes. Even more: I had prolong experience of running BTC Core syncs on Umbrel 1.3 and 1.4.2 which ususally finished with 5-6 days normally vs 5 -6 weeks for Start9OS v0.3.5.1. It was never the same with Knots on Start9 @ miniPC 4GB or 8 GB/4100/7TB SSD with cable link to router 600/60 mbps.
It is really important to tune the container in the next Start9OS version to improve the performance & remove the pain.

[ThatLinuxDude]

@morelub : Do you happen to still have the debug.log file from the start9 sync run? You would need ssh access and copy it from /embassy-data/package-data/volumes/bitcoind/data/main.

[k0gen]

Hey everyone,
Just jumping in to share that I finally published a little guide to running Bitcoin data on a dedicated SSD under StartOS, by using a direct bind mount for the main data
(link: startos-toolbox/bitcoind-dedicated-drive.md at random · k0gen/startos-toolbox · GitHub).

Based on my own experiments - and everything folks have tested here - it looks like this approach could really help speed up IBD and give some relief to the system, especially for anyone syncing the full chain. It’s not super complicated to try if you’re comfortable with a bit of sysadmin stuff.

Of course, overlayfs is a clever solution for many use cases, but for big datasets like Bitcoin Blockchain there might be some unique quirks worth investigating further. For now, if you’re curious or want to experiment, please feel free to follow my guide and post your results. Would be awesome to see more real-world confirmations!

Big thanks to everyone in this thread for the research and ideas!
Let’s keep digging and sharing what works.

[ThatLinuxDude]

Would be great to see direct comparisson of sync time (you can use my script and the debug.log created in the bitcoin datadir) on identical hardware, using fuse-overlayfs and your solution with a volume bind mount.

Unfortunately, for most users your solution will not be a viable option, as it hardly beats the complexity of installing bitcoin-core on any other commandline linux. A solution that allows Start9 containers to use a bind mount would be the way to go.

[Rexter]

@k0gen works for Start9. We’re not suggesting that this is the “solution” long term. It’s an unsupported hack, and perhaps proof of concept for improvements in future versions of StartOS.

[alvs]

Hello.
I don’t have your technical knowledge, but I have the exact same equipment and the exact same problem.
The syncing is taking forever.
With download rates between 4 and 1 block per minute…
That means that, if that rate is maintained, there would be 917095-888825 = 28270 blocks left.
About 20 days of download time for the remaining 7%…

A fragment of the log
2025-09-30T16:35:45+00:00 2025-09-30T16:35:45Z UpdateTip: new best=000000000000000000009ffdbfdffd11926403fedea970ab4617c7cceaca5de9 height=888823 version=0x26fd6000 log2_work=95.508445 tx=1168899813 date=‘2025-03-21T22:41:55Z’ progress=0.938235 cache=275.0MiB(2097574txo)
2025-09-30T16:36:56+00:00 2025-09-30T16:36:56Z UpdateTip: new best=00000000000000000001fda86fb21314ec19e34fa8abfaf41442c264e62f14f5 height=888824 version=0x2001e000 log2_work=95.508457 tx=1168901033 date=‘2025-03-21T22:46:16Z’ progress=0.938236 cache=276.3MiB(2107738txo)
2025-09-30T16:37:53+00:00 2025-09-30T16:37:53Z UpdateTip: new best=00000000000000000001e7ae2beb3c6cd6b874a2793958e5b732907d2de1bc48 height=888825 version=0x3c000000 log2_work=95.508470 tx=1168902263 date=‘2025-03-21T22:52:32Z’ progress=0.938237 cache=277.3MiB(2117387txo)

[Rexter]

The servers Start9 sells, which is what StartOS is designed for, are a significantly higher spec machine than the G2 800 we are discussing. As we’re discussing here, the IBD is going to take a while on that machine. The best upgrade you can do is increase the RAM. If you can boost that machine to 16GB, it will not take nearly as long. In the Bitcoin config, I’d set the database cache to 75% of the RAM, and not run any other services on the server while the IBD is in progress. Just be sure to remember to clear the field again after it’s done. You don’t want to leave it that way. Once you get past the IBD, that machine should work well as a Bitcoin node on StartOS.

[ThatLinuxDude]

Update on my test: the Pop!_OS (Ubuntu) bitcoin-core sync has finished after 55h from 0% to 100%.

Given the kernelspace utilization on start9 during IBD, the entire bottleneck is probably I/O bound - so not more RAM, but faster RAM or higher memory bandwidth in general would yield better results. Yes more RAM is also always better, but if the bottleneck is the kernel/userspace copying between FUSE, the impact will not be that much. Also please look at the other thread where @Rexter tested a 4GB machine, and it synced decently well on ubuntu linux.

Since the slowdown seems to be exponential (see scatterplot) relative to the number of blocks/diskspace used, and bitcoin having a linear growth of the number of blocks over time, this issue will become a major issue exponentially faster .

Assume number of blocks = n, and we have the old complexity O(n^2) problem, this means:

• Number of people with first-time start9 installation complaining about unfinished IBD will increase here in the forum
• People running Start9 for years that had their IBD also years ago are fine now, but will have issues if they format their drives and start over (on same machine!)
• Recommending faster machines will only work short term, long term beefier/faster machines will too fall behind very fast - hardware performance increases only linearly - and I/O performance is already growing slowly between HW generations
• the discrepancy in syncing times on Start9 vs. other operating systems will become more and more obvious

Lets hope the volume mount becomes a viable solution, or maybe there will be updates in the podman/overlayfs code that fixes the issue.

[Rexter]

I think the next version of StartOS is likely going to solve all this, as it doesn’t use Podman at all.

[morelub]

@ThatLinuxDude - unfortunately I am very busy currently with other things and cannot check. But the same slow down situation takes place with monero node.

[remcoros]

A few things:

• bitcoind data dir is using a bind-mount, so overlayfs is not a factor here
• StartOS version of bitcoind is compiled with -O1 instead of the default -O2, the latter has more compiler optimizations. I’m unsure what the actual impact of this is
• StartOS uses btrfs, but on 0.3.5 Copy on Write (CoW) is NOT disabled for the data dir, which can result in slower writes. In StartOS 0.4, CoW is disabled for the data dir.
• StartOS uses LUKS encryption (aes-stx 512), this can have a significant performance impact on older hardware, especially if the CPU does not support AES. (For example the Pi4 cpu does not have any crypto support, so it will have quite some impact). When using a VM, make sure to choose a CPU type that has AES support.

Given these facts, it’s not surprising to see a difference between StartOS (btrfs+CoW+LUKS) versus another linux distro on a non-encrypted EXT4 volume)

[ThatLinuxDude]

As with the AES, the i5 6000s series in the HP EliteDesks come with AES support, my machine that I used for testing was the i5 6500T. So were the tests on the 3 machines @Rexter did.

CoW, compiler optimization and btrfs should be investigated. As it stands the difference in sync time is orders of magnitude different than other systems (2.5days vs. weeks, if the sync even finishes).

[remcoros]

Here is a build that is compiled with -O2 and disables CoW on the data dir: compile using '-O2' and disable CoW on data dir · Start9Labs/bitcoind-startos@2698322 · GitHub

I don’t have the hardware/disk space to use for benchmarking, but if anyone wants to give it a try, that would be nice.

note: because the CoW flag only applies to new files and folders, make sure to start from scratch
note2: when downloading the build artifact from above link, it’s a .zip file, unzip it first, then you can sideload the .s9pk