The OS TOR service issues

CurtisS · February 2, 2024, 3:24am

I have the first round of Start9’s Pro Server, and a Raspi that I had previously built.

Lately the Pro is having serious issues with TOR connections. I double check it against the Raspi to see if its a TOR network issue or something else. The Raspi gets incoming connections quick and fine, almost always. Today, with about a dozen attempts, on the Pro, I’ve gotten the Login screen for the OS one time and it failed to log in giving me ‘unknown error’. The rest of the time it times out.

I cant make sense of the logs but the logs on the Raspi look much different than the logs on the Pro. The pro seems to not be able to access a gateway.

I tried restarting the TOR service and the other option to clear its data and restart. I also hard reboted the Embassy. That worked last night but Im back where I started agsin today.

My first indicator is always when I open Zeus. Either it shows all my channels as inactive or it just fails to connect to the node all together.

I’ll send some logs from the TOR service when I get home tonight.

CurtisS · February 2, 2024, 9:04am

Okay, more details. I get home and the Embassy Pro’s fan is running full tilt boogie. I checked the temp, processer, and memory usage. All less than 35%

I noticed LND was spinning endlessly trying to catch up the sync with the graph. Checked it’s logs. It’s doing it’s thing but can’t get out through TOR (in my opinion). I shut LND down and the fan immediately spun down for a while. I killed ThunderHub, LNDg, Lightning Terminal and RTL. The fan got quiet.

I checked the Embassy’s TOR logs and they looked like this (full file will be included as well).

This is the full log after shutting down the embassy with all lightning services shut down and restarting the Embassy (lightning services still shut down.)

2024-02-02T02:30:09-06:00 Starting tor@default.service - Anonymizing overlay network for TCP…
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.936 [notice] Tor 0.4.8.9 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.0.11, Zlib 1.2.13, Liblzma 5.4.1, Libzstd 1.5.4 and Glibc 2.36 as libc.
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.936 [notice] Tor can’t help you if you use it wrong! Learn how to be safe at Am I totally anonymous if I use Tor? | Tor Project | Support
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.936 [notice] Read configuration file “/usr/share/tor/tor-service-defaults-torrc”.
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.936 [notice] Read configuration file “/etc/tor/torrc”.
2024-02-02T02:30:09-06:00 Configuration was valid
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.957 [notice] Tor 0.4.8.9 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.0.11, Zlib 1.2.13, Liblzma 5.4.1, Libzstd 1.5.4 and Glibc 2.36 as libc.
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.957 [notice] Tor can’t help you if you use it wrong! Learn how to be safe at Am I totally anonymous if I use Tor? | Tor Project | Support
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.957 [notice] Read configuration file “/usr/share/tor/tor-service-defaults-torrc”.
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.957 [notice] Read configuration file “/etc/tor/torrc”.
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.959 [warn] You specified a public address ‘0.0.0.0:9050’ for SocksPort. Other people on the Internet might find your computer and use it as an open proxy. Please don’t allow this unless you have a good reason.
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.959 [notice] Opening Socks listener on 0.0.0.0:9050
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.959 [notice] Opened Socks listener connection (ready) on 0.0.0.0:9050
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.959 [notice] Opening Control listener on 127.0.0.1:9051
2024-02-02T02:30:09-06:00 Feb 02 08:30:09.959 [notice] Opened Control listener connection (ready) on 127.0.0.1:9051
2024-02-02T02:30:09-06:00 We compiled with OpenSSL 300000b0: OpenSSL 3.0.11 19 Sep 2023 and we are running with OpenSSL 300000b0: 3.0.11. These two versions should be binary compatible.
2024-02-02T02:30:09-06:00 Tor 0.4.8.9 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.0.11, Zlib 1.2.13, Liblzma 5.4.1, Libzstd 1.5.4 and Glibc 2.36 as libc.
2024-02-02T02:30:09-06:00 Tor can’t help you if you use it wrong! Learn how to be safe at Am I totally anonymous if I use Tor? | Tor Project | Support
2024-02-02T02:30:09-06:00 Read configuration file “/usr/share/tor/tor-service-defaults-torrc”.
2024-02-02T02:30:09-06:00 Read configuration file “/etc/tor/torrc”.
2024-02-02T02:30:09-06:00 You specified a public address ‘0.0.0.0:9050’ for SocksPort. Other people on the Internet might find your computer and use it as an open proxy. Please don’t allow this unless you have a good reason.
2024-02-02T02:30:09-06:00 Opening Socks listener on 0.0.0.0:9050
2024-02-02T02:30:09-06:00 Opened Socks listener connection (ready) on 0.0.0.0:9050
2024-02-02T02:30:09-06:00 Opening Control listener on 127.0.0.1:9051
2024-02-02T02:30:09-06:00 Opened Control listener connection (ready) on 127.0.0.1:9051
2024-02-02T02:30:09-06:00 Parsing GEOIP IPv4 file /usr/share/tor/geoip.
2024-02-02T02:30:10-06:00 Parsing GEOIP IPv6 file /usr/share/tor/geoip6.
2024-02-02T02:30:10-06:00 Bootstrapped 0% (starting): Starting
2024-02-02T02:30:10-06:00 Starting with guard context “default”
2024-02-02T02:30:10-06:00 Signaled readiness to systemd
2024-02-02T02:30:10-06:00 Started tor@default.service - Anonymizing overlay network for TCP.
2024-02-02T02:30:10-06:00 New control connection opened from 127.0.0.1.
2024-02-02T02:30:11-06:00 Opening Control listener on /run/tor/control
2024-02-02T02:30:11-06:00 Opened Control listener connection (ready) on /run/tor/control
2024-02-02T02:30:11-06:00 Bootstrapped 5% (conn): Connecting to a relay
2024-02-02T02:30:11-06:00 Bootstrapped 10% (conn_done): Connected to a relay
2024-02-02T02:30:11-06:00 Bootstrapped 14% (handshake): Handshaking with a relay
2024-02-02T02:30:11-06:00 Bootstrapped 15% (handshake_done): Handshake with a relay done
2024-02-02T02:30:11-06:00 Bootstrapped 20% (onehop_create): Establishing an encrypted directory connection
2024-02-02T02:30:11-06:00 Bootstrapped 25% (requesting_status): Asking for networkstatus consensus
2024-02-02T02:30:11-06:00 Bootstrapped 30% (loading_status): Loading networkstatus consensus
2024-02-02T02:30:17-06:00 I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
2024-02-02T02:30:17-06:00 Bootstrapped 40% (loading_keys): Loading authority key certs
2024-02-02T02:30:18-06:00 The current consensus has no exit nodes. Tor can only build internal paths, such as paths to onion services.
2024-02-02T02:30:18-06:00 Bootstrapped 45% (requesting_descriptors): Asking for relay descriptors
2024-02-02T02:30:18-06:00 I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/7967, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.)
2024-02-02T02:30:18-06:00 Bootstrapped 50% (loading_descriptors): Loading relay descriptors
2024-02-02T02:30:20-06:00 The current consensus contains exit nodes. Tor can build exit and internal paths.
2024-02-02T02:30:20-06:00 Bootstrapped 55% (loading_descriptors): Loading relay descriptors
2024-02-02T02:30:21-06:00 Bootstrapped 63% (loading_descriptors): Loading relay descriptors
2024-02-02T02:30:21-06:00 Bootstrapped 71% (loading_descriptors): Loading relay descriptors
2024-02-02T02:30:21-06:00 Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits
2024-02-02T02:30:21-06:00 Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits
2024-02-02T02:30:21-06:00 Bootstrapped 95% (circuit_create): Establishing a Tor circuit
2024-02-02T02:30:21-06:00 Bootstrapped 100% (done): Done
2024-02-02T02:30:27-06:00 Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 133 buildtimes.
2024-02-02T02:33:49-06:00 Guard TorRelay3a ($21926E3247C41DC078C1B62678618D7AA4B75A26) is failing more circuits than usual. Most likely this means the Tor network is overloaded. Success counts are 167/239. Use counts are 103/103. 169 circuits completed, 0 were unusable, 2 collapsed, and 29 timed out. For reference, your timeout cutoff is 60 seconds.
2024-02-02T02:37:18-06:00 Guard Unnamed ($9E806C33AEB62CF2893B0F85D5A810D78ADEA51F) is failing more circuits than usual. Most likely this means the Tor network is overloaded. Success counts are 180/258. Use counts are 119/119. 180 circuits completed, 0 were unusable, 0 collapsed, and 29 timed out. For reference, your timeout cutoff is 60 seconds.
2024-02-02T02:38:25-06:00 Closed 1 streams for service [scrubbed].onion for reason resolve failed. Fetch status: No more HSDir available to query.

I checked the ability to access the Embassy on a TOR browser. Success, just like yesterday.

Restarted LND to see what that might do to the TOR logs. TOR logs are starting to look like the picture again but I am still able to connect. Fan is still quiet with only LND running. PS. Amboss shows my node as being online but ALL of my channels have me disabled (probably because they couldn’t reach me over TOR either).

Turned off ‘Use Tor for all traffic’ and ‘Stream Isolation’ in LND config. Firing up ThunderHub (.local) to make sure I’m still pinging Amboss.

More to come…

CurtisS · February 2, 2024, 9:29am

About 20 minutes have gone by. System resource usage on the Embassy Pro are much lower than they used to be. Temp and fan are cooler. Memory is down a bit and CPU is way down.

I have LND, LNDg, ThunderHub and Lightning Terminal running again and things seem fine. Amboss is showing my channels are coming back online. Was able to reconnect and load the Embassy’s interface on my laptop over TOR quite well again. This is how it went yesterday. Rebooted everything and everything worked fine for a few hours.

Tor logs are already ‘closing streams’ about every 30sec to a minute, nonstop.

We’ll see how it goes.

CurtisS · February 2, 2024, 10:32pm

13hrs later. Everything still running smooth.

2 Possibilities I see.

The second round of shutting
erything down and restarting did the trick, or…
If you are trying to run a routing node, even on the ‘system pure’ or Pro, you probably shouldn’t set ‘all traffic over TOR’ in LND’s config.

#2 is the only change I made.

I’ll make this the solution if I have time where I can monitor the box and turn all traffic over TOR back on and see if it happens again. It took several hours for things to get congested (?) the last two times so I cant fix it remotely at that point.

StuPleb · February 3, 2024, 4:56pm

It’s Tor being Tor from the looks of it, and you got a spate of bad luck.

I think your initial step was all you really needed to do (restarting Tor and wiping state), just over and over again until you got a different, better circuit. I think that’s ultimately what you got.