How do I open an embassy backup to extract data from it? (so I can attempt to manually recover a node without tearing down lightning channels)

StartOS Support
1

TLDR: How do I open an embassy backup to extract data from it? (so I can attempt to manually recover a node without tearing down lightning channels)

I had a hardware failure but I have an embassy backup stored on a NAS. I did a fresh install and then tried to perform a “Recover” from the setup menu, input my NAS creds, unlocked the backup/dive using the password, selected the disk, init started, then the process hung at 64%. Eventually killed the VM, tried a couple times, gave it a few hours, no change, always gets stuck at 64%.

Figured I’d give the “Restore” option a shot, so I proceeded with the “Start Fresh” option. Once that was done and I tried running the Restore process, all services were successfully restored except for CLN. It would just sit there at 100%, but wouldn’t actually complete the restore process on this one service. Rebooting node and the service would be gone, attempt to restore again, gets stuck at 100% again.

SSH into node notice that the c-lightning-embassy.BackupRestore (or something like that) container was running, but didn’t seem to be doing anything, syslogs showed it was installing CLN. After giving it some time and a few different attempts at the process above, I decided to try and do a fresh install of CLN form the marketplace, and it worked without issue.

So at this point bitcoind is in the process of re-synching the chain, I have everything but CLN data restored, but I do have CLN installed. I believe I just need the following files and I should be able to recover the node without having to close channels/build a new instance.

/root/.lightning/bitcoin/hsm_secrect
/root/.lightning/bitcoin/emergency.recover
/root/.lightning/bitcoin/lightningd.sqlite3

But first I’d need to figure out how to open/decrypt the files stored on/in the embassy backup. I started digging into ECRYPTFS, but decided it’s probably best to stop digging deeper and ask for some help.

If anyone has any ideas on how to make use of this backup and/or if you can provide guidance on another way to recover this lightning node (or at least the funds on it), any help wold be greatly appreciated.

2

The supported way of using encrypted backups is via the StartOS interface. To create backups would be System → Create Backup. You’d restore from System → Restore from Backup.

And while you can restore the entirety of the backed up services during the install process of StartOS, when StartOS is installed you can also restore individual services from System → Restore from Backup.

Accessing the encrypted backup, which I won’t go into into detail about here, is technically possible but you’d only retrieve a static channel backup (SCB). There’s no accessible way in Lightning to back up dynamic channel balances, especially when placing them in long term storage. Restoring a SCB will ALWAYS close all the channels because it has no idea what the latest balance of each channel is. Unless you have a channels DB from a milisecond before you node failed, you won’t be keeping your channels intact. To rescue the funds, you have to go through the backup restoration process.

3

Thanks for the response Stu! Unfortunately both of the supported methods [individual services] & [entire server] are both hanging up for whatever reason so I’m looking for alternate solutions.

I took the backup moments before powering down startOS and power cycling the hardware (which is when the failure occurred). So I would think the DB should be usable/current since the node has been offline since this backup was performed.

Unless I can figure out how to get the supported methods functioning, or importing the DB and restoring the node manually, I think my only chance to recover the funds is to strip out the emergency.recover file and do a manual wallet recovery. (like so- Wallet recovery )

4

Bumping this thread in hopes that someone can help me. I’ve been able to restore all aspects/services of my StartOS instance, except for CLN.

When I try restoring the individual CLN service (like I did with all the others) it hangs at “Restoring finalizing”, but never actually finishes. The last attempt I let it sit in the 100% state for 3 days, then had a power outage, and when the node came back up, there was/is no reference to CLN anymore.
ss1

ss2
ss21632×339 15.8 KB

It’s listed in the available apps to restore again, but I’m stuck in this loop and the panic of losing the funds is setting in.

image
image940×637 35.5 KB

I’ve tinkered with eCryptFS a little bit and have it functional on my local system. After looking at the data saved in the StartOS backup I’m unsure how to mount it with the correct options/settings/etc. I dug around in the github repo but that just showed me a little more about how little I know. :sweat_smile:

5

Did we establish what is up with your CLN restore? You don’t seem to mention if you’re aware of what it is. If the restore is failing, we need to work out what is causing that in case there’s some edge case bug that needs to be looked at.

Do you have a Matrix account, and can you join us here: LINK

We can also walk you through the tedious process of picking apart backups too.

6

Not yet, I was looking to get into the backup in hopes to restore CLN without closing channels (since the restore process was failing). I haven’t used matrix before but I’ll reach out there now. Thanks Stu!

1 Like