Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.@get-ssl-key
I tried installing it a second time. It worked, but does not see my old wallet. Anybody know what I could do?
I do need to rescue my Lightning Node & wallet. I can SSH in if needed (once I figure out how in the new setup), or do I have to abort and reinstall the old version which hopefully the wallet is still there?
It was an upgrade via usb key from 0.3.5.1 so everything should still be there(?) I just verified and I can ssh in, but don’t know where to navigate to verify if the paths are still there, though I didn’t touch anything. Need a bit of guidance here.
When I upgraded I selected to preserve existing, so it should have kept the existing. Other services seem to have migrated fine.
Any ideas? I did take a full backup before upgrading, but I’m hoping not to have to backtrack and deal with force closed channels. Thank-you!
Appreciate you sticking with us on this — since it’s your channels/funds we want to move carefully here rather than fast.
First, can you run these over SSH? Both are read-only, won’t touch or move anything:
sudo du -sh /embassy-data/package-data/volumes/lnd/data/main/
sudo find /embassy-data/package-data/volumes/lnd/data/main/data/chain/bitcoin/mainnet/ -type f
That checks whether your original 0.3.5.1 LND data (wallet.db, channel.backup) is still sitting there untouched — which we think is likely, since the migration failed before it got that far.
Also, so we don’t mix things up: the fresh LND you installed on 0.4.0.1 has its own new, empty wallet living somewhere separate. Can you leave that one alone for now — don’t fund it or touch it — while we sort out the original?
Where this is probably headed: since you already have a full backup from before the upgrade, the safest path back to your existing channels is likely flashing a 0.3.5.1 image and restoring that backup there, rather than trying to hand-migrate the old data into 0.4.0.1. One heads up — restoring a backup like this (instead of a clean Transfer) will most likely cause your channel peers to force-close on you, since your channel state won’t perfectly match what they last saw. That’s expected behavior for this kind of recovery, not a sign something’s gone wrong — your funds aren’t lost, they just land back on-chain once the force-close timelocks clear, rather than staying in open channels. That part just takes some patience.
We’re getting eyes from the rest of the team before recommending you actually run the restore, so hold off there for now — just send us those two command outputs and we’ll follow up with exact next steps.
Ok, for both these commands I get “No such file or directory”.
du: cannot access ‘/embassy…’: No such file or directory
find: ‘/embassy…’: No such file or directory
I tried shortening the FIND to less directories to be broader and it still couldn’t find anything. I’m not sure of the exact setup here so I stopped there, but it seems weird. The interface is up and running so obviously there’s something somewhere. The data drive is separate from the boot drive if that helps.
I have not started any services on the server aside from tor, so everything stayed as-is since posting earlier. Any help would be appreciated.
Should I install 0.3.5.1 again on another drive and use it (instead of the new 0.4.0.1) with the same data drive again? Would that at least allow LND to continue where it left off before the migration? Or should I do a full restore and force close all the channels but at least get my funds back?
Quick correction on the paths I gave you — on 0.4.0.1 the data drive is mounted
somewhere else, so /embassy-data may not resolve even though your old 0.3.5.1
data is still there. If you got “no such file or directory”, ignore it.
Run this instead. Read-only, checks both locations:
If wallet.db and channel.backup come back, your funds are recoverable one way or
another. Keeping your channels open is the harder question. There’s also a database format change between 0.3.5.1 and
0.4.0, so it’s not simply a matter of moving files into place.
There is a way to bring that data across and convert it, and I’m checking with
the team now. If it works, your channels stay open. If not, the fallback
recovers your wallet and closes the channels from your static channel backup —
funds return on-chain after the timelocks, slower and costs you the channels,
but nothing is lost.
In the meantime: don’t uninstall or reinstall LND, and leave the new empty
wallet alone. Just send me that output.
Thanks. Let’s stop pattern-matching on filenames and just look inside the
folder — that answers it directly:
sudo du -sh /media/startos/data/package-data/volumes/lnd
sudo ls -la /media/startos/data/package-data/volumes/lnd/
sudo find /media/startos/data/package-data/volumes/lnd/ -maxdepth 4 | head -60
Two things I’m reading from that. The size: a real 0.3.5.1 LND volume should be
hundreds of MB or more. If instead it’s small with lnd.conf
and store.json sitting at the top level, that’s the fresh install you did on
the 5th, and we’d be looking at a different recovery route.
The timestamps in the ls output will tell us the same story — anything dated
before your upgrade is original.
I don’t know how a Sept 7 date got in there, but everything looks empty. I’ve left the power on but hadn’t touched it so I’m quite confused by the 7th, aside from maybe a reboot. But at any rate at 24k it looks empty unfortunately.
One more check before we move on — can you run this? It searches the whole
disk for the wallet/channel files by name, so it’ll catch them even if they
ended up somewhere other than the paths we’ve already checked:
It’s read-only, just takes a minute or two to run.
If that comes back empty, then at this point it looks like the original
0.3.5.1 wallet data likely didn’t survive the migration. In that case, the
most likely path back to your funds is the full backup you took before the
upgrade — restoring that onto a fresh 0.3.5.1 install should bring back your
wallet and channel state as of that backup. One thing to expect: this will
probably cause your channel peers to force-close, since your channel state
won’t perfectly match what they last saw. That’s normal for this kind of
recovery, not a sign anything else has gone wrong — the funds land back
on-chain once the force-close timelocks clear, rather than staying in open
channels. Just takes some patience.
Before we go further either way — where’s that backup actually stored
(external drive, cloud target, something else), and can you still get to
it? That’s the piece we need confirmed before walking through exact restore
steps, since we want to be sure it’s good before you touch anything else on
the live server.
Thanks for running that, and good that you widened it to the whole system — that’s more conclusive than the scoped version would have been. I won’t sugarcoat it: wallet.db, channel.db and channel.backup aren’t anywhere on the server. The two hits you got are Lightning Terminal’s, unrelated to your LND wallet. There’s also no admin.macaroon for LND itself, which there would be if a wallet existed at that path. So the original LND data did not survive the failed migration, despite what that notice told you. That makes your external backup the recovery route rather than the safety net. It’s 0.3.5.1 format and won’t restore onto 0.4.0.1 — it needs a 0.3.5.1 install to go back onto. It will recover your wallet and your on-chain funds. Your channels will force-close, and those balances return on-chain once the timelocks clear. Slower, and it costs you the channels and the fees to reopen, but the funds are not lost.
I wanted to say Thank-you for the help in trying to rescue the install.
After going back to the backup I’m still in the process of scanning the chain for closed channels which will probably take another week. My years of running a lightning routing node are over I think, time to retire from that.