Unable to Establish SSH Connection Using AWS IoT Secure Tunneling from External Terminal

[zcs-ricciolino]

Describe the bug

I am unable to completely establish an SSH connection using AWS IoT Secure Tunneling from an external system terminal. The connection stalls after being established and does not proceed to the authentication stage.

To Reproduce

Steps to reproduce the behavior:

1. Open a new secure tunnel using the AWS CLI:

aws iotsecuretunneling open-tunnel --region eu-west-1 --destination-config thingName=<my_thing_name>,services=

2. Extract the sourceAccessToken and set it as an environment variable:

export AWSIOT_TUNNEL_ACCESS_TOKEN="<source_access_token>"

3. Start the local proxy in source mode:

./localproxy -v 5 -r eu-west-1 -s 5355

4. Attempt to establish an SSH connection:

ssh -vvv root@localhost -p 5355

Expected behavior

I expect to be prompted for the root user's password and gain SSH access to the remote device, as I can through the AWS Web Console.

Actual behavior

The SSH connection is established but does not proceed to the authentication stage. The connection stalls after showing "Connection established."

Logs

Local Proxy Logs:

[2024-07-04 12:49:11.837514] (0x000074fb29645840) [debug] v2 local proxy starts with v1 local proxy format
[2024-07-04 12:49:11.837543] (0x000074fb29645840) [debug] /home/ricciolino/work/aws-iot-securetunneling-localproxy-example/dependencies/aws-iot-securetunneling-localproxy/build/config does not exist!
[2024-07-04 12:49:11.837570] (0x000074fb29645840) [info] Starting proxy in source mode
[2024-07-04 12:49:11.840027] (0x000074fb29645840) [info] Attempting to establish web socket connection with endpoint wss://data.tunneling.iot.eu-west-1.amazonaws.com:443
[2024-07-04 12:49:11.844590] (0x000074fb29645840) [debug] Resolved proxy server IP: 52.49.63.50
[2024-07-04 12:49:11.846019] (0x000074fb29645840) [debug] Connected successfully with proxy server
[2024-07-04 12:49:11.941616] (0x000074fb29645840) [debug] Successfully completed SSL handshake with proxy server
[2024-07-04 12:49:12.040116] (0x000074fb29645840) [info] Web socket session ID: 0a1747fffeefc5a3-00004f3a-0000b4bd-cf3e3ea8482b0c44-0cfbd306
[2024-07-04 12:49:12.040209] (0x000074fb29645840) [debug] Web socket subprotocol selected: aws.iot.securetunneling-3.0
[2024-07-04 12:49:12.040230] (0x000074fb29645840) [info] Successfully established websocket connection with proxy server: wss://data.tunneling.iot.eu-west-1.amazonaws.com:443
[2024-07-04 12:49:12.040283] (0x000074fb29645840) [debug] Seting up web socket pings for every 20000 milliseconds
[2024-07-04 12:49:12.040379] (0x000074fb29645840) [debug] Scheduled next read:
[2024-07-04 12:49:12.040527] (0x000074fb29645840) [debug] Extracting service Ids from control message 5
[2024-07-04 12:49:12.040568] (0x000074fb29645840) [info] Updated port mapping for v1 format: 
[2024-07-04 12:49:12.040586] (0x000074fb29645840) [info] SSH = 5355
[2024-07-04 12:49:12.040613] (0x000074fb29645840) [info] calling setup from loop
[2024-07-04 12:49:12.040646] (0x000074fb29645840) [debug] Resolving bind address host: localhost
[2024-07-04 12:49:12.040677] (0x000074fb29645840) [debug] Port to connect 5355
[2024-07-04 12:49:12.040707] (0x000074fb29645840) [debug] Starting web socket read loop continue reading...
[2024-07-04 12:49:12.041115] (0x000074fb29645840) [debug] Resolved bind IP: 127.0.0.1
[2024-07-04 12:49:12.041307] (0x000074fb29645840) [info] Listening for new connection on port 5355



[2024-07-04 12:49:44.325861] (0x000074fb29645840) [debug] socket port 5355
[2024-07-04 12:49:44.325921] (0x000074fb29645840) [debug] endpoint mapping:
[2024-07-04 12:49:44.325934] (0x000074fb29645840) [debug] SSH = 5355
[2024-07-04 12:49:44.325945] (0x000074fb29645840) [info] creating tcp connection id 1
[2024-07-04 12:49:44.325968] (0x000074fb29645840) [info] Accepted tcp connection on port 5355 from 127.0.0.1:59136
[2024-07-04 12:49:44.325996] (0x000074fb29645840) [debug] Sending stream start, setting new stream ID to: 1, service id: SSH
[2024-07-04 12:49:44.326128] (0x000074fb29645840) [debug] Starting web socket read loop while web socket is already reading. Ignoring...
[2024-07-04 12:49:44.326197] (0x000074fb29645840) [debug] Prepare to send data message: service id: SSH stream id: 1 connection id: 1
[2024-07-04 12:49:44.326248] (0x000074fb29645840) [debug] Write buffer has enough space, continue tcp read loop for SSH connection id: 1
[2024-07-04 12:49:44.326263] (0x000074fb29645840) [debug] Not starting TCP read loop, socket is already reading
[2024-07-04 12:49:44.326280] (0x000074fb29645840) [debug] not writing, no buffer contents, skip straight to being done draining
[2024-07-04 12:49:48.368035] (0x000074fb29645840) [debug] Handling tcp socket error for service id: SSH connection id: 1. error message: End of file
[2024-07-04 12:49:48.368110] (0x000074fb29645840) [info] Disconnected from: 127.0.0.1:59136
[2024-07-04 12:49:48.368228] (0x000074fb29645840) [debug] not writing, no buffer contents, skip straight to being done draining
[2024-07-04 12:49:48.368270] (0x000074fb29645840) [debug] No connectionId_to_tcp_connection mapping for connection id: 1

SSH Command Logs:

> ssh -vvv root@localhost -p 5355
OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/ricciolino/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/ricciolino/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/ricciolino/.ssh/known_hosts2'
debug2: resolving "localhost" port 5355
debug3: resolve_host: lookup localhost:5355
debug3: ssh_connect_direct: entering
debug1: Connecting to localhost [127.0.0.1] port 5355.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/ricciolino/.ssh/id_rsa type -1
debug1: identity file /home/ricciolino/.ssh/id_rsa-cert type -1
debug1: identity file /home/ricciolino/.ssh/id_ecdsa type -1
debug1: identity file /home/ricciolino/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/ricciolino/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/ricciolino/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/ricciolino/.ssh/id_ed25519 type -1
debug1: identity file /home/ricciolino/.ssh/id_ed25519-cert type -1
debug1: identity file /home/ricciolino/.ssh/id_ed25519_sk type -1
debug1: identity file /home/ricciolino/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/ricciolino/.ssh/id_xmss type -1
debug1: identity file /home/ricciolino/.ssh/id_xmss-cert type -1
debug1: identity file /home/ricciolino/.ssh/id_dsa type -1
debug1: identity file /home/ricciolino/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.10
--> STUCKS_HERE <---

Environment (please complete the following information):

• OS: Ubuntu 22.04.2 LTS
• Version 22.04
• Architecture: x86-64
• Localproxy commit: 2df9b599a8852610310f3f9b9a83c5ee4b10e0f2

Additional context

• The SSH connection works correctly through the AWS Web Console, but the same cannot be replicated using an external system terminal.
• The destination device runs Yocto OS.
• The local proxy logs indicate that the connection is accepted and a TCP connection is created, but the SSH authentication does not proceed.
• The issue persists despite valid access tokens and successful tunnel establishment.

[zcs-ricciolino]

For those facing the same issue, I found the solution reading at this reply.

Running the localproxy on the source side with the --destination-client-type V1 parameter solves the issue, allowing SSH connections to work from an external terminal.