Unable to access server via HTTPS after v0.3.3 upgrade: Potential SSL/TLS config issue?

Hi

…and a Happy New Year to you all!

After following the DIY instructions I have been running EOS on an RPi for many months without problems. Been a very good experience,…thank you!

Yesterday I upgraded to ver 0.3.3 from 0.2.x

The upgrade went smoothly in the sense that the device came back to life, but I can no longer access the embassy Dashboard from an HTTPS connection, (it works OK with HTTP and Tor).

Using a terminal on the client desktop I can see that the embassy server is not providing a “serverhello” as part of the SSL handshake. It is instead providing TLS Alert code 49 which is an “access denied” code and so the connection is closed.


curl -v https://192.168.1.10:443

  • Trying 192.168.1.10:443…
  • Connected to 192.168.1.10 (192.168.1.10) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS alert, access denied (561):
  • error:14094419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied
  • Closing connection 0
    curl: (35) error:14094419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied

My immediate thoughts were a mismatch of supported SSL/TLS protocols negotiated during the handshake, but then I reflected on the fact that there really hasn’t been any negotiation as the server doesn’t send a “serverhello” message,… by using the openssl command line below to try and get more data, I can see that the response lines from the server are reported by the client as “???”, … so does this mean the server is sending back data the client doesn’t understand and it is the client that is reporting “Access Denied”?.


openssl s_client -msg -showcerts -connect 192.168.1.10:443 < /dev/null
CONNECTED(00000003)
>>> ??? [length 0005]
16 03 01 01 16
>>> TLS 1.3, Handshake [length 0116], ClientHello
01 00 01 12 03 03 8c cd ef 4f 7a 07 b2 60 31 03
07 b1 12 9e f3 df a8 9d dc 4f 96 1d 8a 0c 8f 10
7b ba 91 c6 03 04 20 09 75 bc b6 56 9c e2 9a be
ce 18 88 36 e7 de 33 bc 1e b8 eb 32 e1 f1 a8 53
97 a3 e9 77 f8 be 69 00 3e 13 02 13 03 13 01 c0
2c c0 30 00 9f cc a9 cc a8 cc aa c0 2b c0 2f 00
9e c0 24 c0 28 00 6b c0 23 c0 27 00 67 c0 0a c0
14 00 39 c0 09 c0 13 00 33 00 9d 00 9c 00 3d 00
3c 00 35 00 2f 00 ff 01 00 00 8b 00 0b 00 04 03
00 01 02 00 0a 00 0c 00 0a 00 1d 00 17 00 1e 00
19 00 18 00 23 00 00 00 16 00 00 00 17 00 00 00
0d 00 2a 00 28 04 03 05 03 06 03 08 07 08 08 08
09 08 0a 08 0b 08 04 08 05 08 06 04 01 05 01 06
01 03 03 03 01 03 02 04 02 05 02 06 02 00 2b 00
05 04 03 04 03 03 00 2d 00 02 01 01 00 33 00 26
00 24 00 1d 00 20 82 00 cb 7e b2 68 a3 02 97 9b
81 19 7c db 6a 17 fb 05 8e de df 43 73 9c fb f2
9c 8b 21 df 5c 79
<<< ??? [length 0005]
15 03 03 00 02
<<< TLS 1.3, Alert [length 0002], fatal access_denied
02 31
124369879778624:error:14094419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied:…/ssl/record/rec_layer_s3.c:1543:SSL alert number 49
no peer certificate available

No client certificate CA names sent

SSL handshake has read 7 bytes and written 283 bytes
Verification: OK

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)


I then SSH connected to the upgraded server and received a warning that the server was offering a new HOST public key which suggested that something related to the SSL/TLS infrastructure of the server has changed. (My assumption was that in upgrading the server, it would try and retain as much of the legacy configuration as possible and the host identity public key would be something to try and keep if possible).

Rather than push into the embassy server and try to figure out the Docker/nginx/SSLTLS config. I just wanted to post this for the Start9 team to ask the following questions:

Q1: Has there been a significant change to SSL/TLS configuration in ver0.3.3 vs 0.2.x ?

Q2: Is it expected that a new Host key would be offered to an SSH client as a result of the upgrade to 0.3.3?

Q3: Do you have any suggestions of what to check as part of debugging the embassy set up to enable https access again?

Thanks for your support and creating a very useful product.

Hi there,

Yes, there have been massive changes since the legacy 0.2 series. I recommend running through the LAN setup again for all your clients (AND browsers if required). It is unlikely that there is any issue with your Embassy. Please reach out if you need help after going through this process.

Hi start9dave

Am back and reaching out biggly after doing what was suggested above.

Have spent time exploring why I cannot connect to my Embassy over https: Have also replaced the “updated” installation with a downloaded v0.3.3 image installed in order to eliminate the possibility or a borked upgrade. It may well be user error,… but I fail to see where I am going wrong and would appreciate a fresh pair of eyes on what I have done and my expectations for how this should work.

The following is a description of my very simple hardware set up,… what I can do, …what I cannot do,… and whats been done to debug so far. Hopefully there is enough info to help point to what the problem might be and how I can explore further. Thanks in advance.

Physical setup?

  • Unifi UltraMini 5 port Gb switch
  • broadband router (default gateway) @ 10.14.14.1 => LAN connected to port 1 of the switch
  • Embassy @ 10.14.14.2 => connected to port 2 of the switch.
  • ubuntu laptop @ 10.14.14.12 connected to port 3 of the switch

I have followed the initial setup instructions to bring up the embassy.
I have also followed instructions to download the CA certificate, add it to the system trust store for Linux OS, and configure the Firefox Browser to use system trust store. I can see that the system Trust store is used and the Embassy Local Root CA cert for my machine is listed in the CA list. I have “viewed” the CA cert from the browser and can see a Common Name of (Embassy Local Root CA (xxxxxxxx) …number redacted for privacy)

What’s working

  • Can ping embassy IPv4 address and get a reply
  • Can connect to embassy over http (http://10.14.14.2) and log into the management dashboard

What’s not working (for me)

  • CANNOT Connect to embassy at https://10.14.14.2 … I receive message that Secure Connection Failed with ERROR code: SSL_ERROR_ACCESS_DENIED_ALERT
  • CANNOT Connect to embassy at https://embassy-stonky-bonkers.local (assume thats the URL given on the dashboard and in the ssh header greeting for how to connect to my embassy)

… so it looks like the server is denying access and rejecting the connection from the client.

What’s been done to debug?

  • ssh into the embassy @ 10.14.14.2:22

  • Confirm that Embassy has correct time and date (same as laptop)

  • Run netstat to scan ports and confirm if 443 is open and listening => (sudo netstat -an -p TCP) => Can confirm that the embassyd is listening on all interfaces port 443.

  • used curl to get more insights into the TLS process => ( curl -v https://10.14.14.2) . From the response below I can see that an acceptable TLS version is used for the handshake (ver 1.3), but the Embassy appears to reject the client hello and terminates the handshake without responding with a “Server Hello” or wanting to negotiate ciphers etc. I don’t have a lot of experience with debugging TLS but given that the above physical setup is “simple”, and the linux laptop has been configured as per embassy guideline regarding importing the Start9 CA cert and that I can access other https sites on LAN/WAN, then I am struggling to see why the embassy server is terminating the TLS handshake.

Googling potential issues related to early handshake termination identified the following areas to look at:

  • date/time mismatch between server/client? => (confirmed not the case here)
  • Firewalls / Man-In-The-Middle hardware/software blocks? There is really no hardware in the middle as the switch is new and I am not sure I would really be target of a supply chain attack. The linux laptop firewall is enabling access to other https sites on both LAN and WAN where TLS exchanges are happening so I don’t believe its a firewall issue.
  • Browser config?.. I followed the embassy instructions for configuring both the browser and linux-OS system trust store with the Start9 CA cert.
  • Protocol Mismatch? - It looks like TLS v1.3 is being used by server and client so I think I am OK.
  • Cipher Suite Mismatch? - I am not sure it even gets that far in the process if the server rejects the handshake and fails to send a “Server Hello”

$ curl -v https://10.14.14.2

* Trying 10.14.14.2:443...
* Connected to 10.14.14.2 (10.14.14.2 port 443 (#0)
* ALPN, offering h2
* ALPN, offering http:1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, access denied (561):
* error:14094419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied
* Closing connection 0
curl: (35) error:14094419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied

I also used “openssl s_client” to try and get more detail and the response is shown below. Again, I am not an SSL/TLS expert so when the error message says “no peer certificate available”, I am assuming that is the client reporting back that it didn’t get a certificate from the embassy server (presumably because the server terminated the handshake before providing a certificate??).

$ openssl s_client -msg -showcerts -connect 10.14.14.2:443

CONNECTED(00000003)
>>> ??? [length 0005]
    16 03 01 01 16
>>> TLS 1.3, Handshake [length 0116], ClientHello
    01 00 01 12 03 03 59 7e 4f 2e 45 5b 8f c9 25 23
    e2 40 41 a9 d6 f5 e5 1d 01 d9 1f 61 52 15 26 61
    45 2a 45 e1 0c 63 20 a7 43 66 a7 dd 27 14 6a 31
    0c 89 7f b9 fc 91 80 54 d0 65 a6 ef 50 ec 50 e2
    13 1d 17 91 73 29 c8 00 3e 13 02 13 03 13 01 c0
    2c c0 30 00 9f cc a9 cc a8 cc aa c0 2b c0 2f 00
    9e c0 24 c0 28 00 6b c0 23 c0 27 00 67 c0 0a c0
    14 00 39 c0 09 c0 13 00 33 00 9d 00 9c 00 3d 00
    3c 00 35 00 2f 00 ff 01 00 00 8b 00 0b 00 04 03
    00 01 02 00 0a 00 0c 00 0a 00 1d 00 17 00 1e 00
    19 00 18 00 23 00 00 00 16 00 00 00 17 00 00 00
    0d 00 2a 00 28 04 03 05 03 06 03 08 07 08 08 08
    09 08 0a 08 0b 08 04 08 05 08 06 04 01 05 01 06
    01 03 03 03 01 03 02 04 02 05 02 06 02 00 2b 00
    05 04 03 04 03 03 00 2d 00 02 01 01 00 33 00 26
    00 24 00 1d 00 20 a1 9f e3 92 f9 eb 7a d2 e1 fd
    4d da 68 3f 3a b8 44 71 17 36 e9 ad 89 04 c2 79
    3e c0 80 c2 28 76
<<< ??? [length 0005]
    15 03 03 00 02
<<< TLS 1.3, Alert [length 0002], fatal access_denied
    02 31
125421473695040:error:14094419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied:../ssl/record/rec_layer_s3.c:1543:SSL alert number 49
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 283 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Last comment is that this build of the embassy-os is the “latest version downloaded from the github site 2 days ago”. I have also used the instructions to compile from source which successfully created an img file and booted OK, but I also get the same challenge around trying to connect with https with that image.

Thanks for any pointers.

Your cert will not work with the IP address, so https://ip.ad.dr.ess is not expected to work. This will change in an upcoming release. Connecting to https://embassy-unique-id.local is the supported and recommended connection method.

I would begin troubleshooting the LAN setup, which is almost certainly where your problem lies. You might like to test on another system to verify nothing is wrong with mDNS on Embassy, but this is rare, and likely fixed by a simple reboot.

Indigo, were you able to further diagnose your issue? I have done the same with my original embassy with the same outcome plus a couple of problems.

  1. I can connect using TOR and my 192.168.##.## only with HTTP://
  2. I used the approved instructions for adding the .crt certificate to Firefox Mozilla but cannot find a way to add it to my Linux trust store on MX Linux.
  3. Also have an issue with a dependency?? when I run ((sudo apt install -y ca-certificates p11-kit)), and then ((libnssckbiso=/usr/lib/firefox/libnssckbi.so && sudo mv $libnssckbiso $libnssckbiso.bak && sudo ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so $libnssckbiso)) I get this message:-- ((mv: cannot stat ‘/usr/lib/firefox/libnssckbi.so’: No such file or directory))
    ANY ideas would possibly help.
    My LAN is functioning correctly afaik.
    I cannot use the https://embassy-funside-####.local (It dives an ssl error)

Thanks, freedom2b

1 - As stated in my reply, IP address is NOT signed by the certificate (although it will be in future) - so https is not expected to work with IP - Tor is already E2EE and does not require https

2 - You’ll need to sort this out for your distro, but we offer some of the popular distros in our Linux guide

3 - These files may be in a different place, again, you’ll have to dig for your specific distribution on how to trust a cert system-wide

You may also need to add the cert to your browser, using the appropriate browser guides