This blog will provide an introduction into the SSH protocol handshake and how it pertains to the identification and detection of SSH tunnels. Then we will introduce a method to fingerprint individual SSH clients and servers and ultimately end with detection of SSH tunneling using both the standalone python version and the zeek implementation of packetStrider.
The Secure SHell protocol is heavily used across most enterprise environments. Observing ssh traffic in your environment likely is commonplace. The most widely used implementation of the secure shell protocol is OpenSSH, which can be leveraged to create l2/l3 tunnels using tun interfaces, dynamic port forwarding (SOCKS) proxy, or local / remote port-forwarding tunnels. SSH is also heavily employed by threat groups during various phases of a campaigns such as data exfiltration, maintaining persistence and tunneling.
Figure 1: high-level depiction
-D dynamic forward
This works by allocating a socket to listen to port on the local side, optionally bound to the specified bind_address. Whenever a connection is made to this port, the connection is forwarded over the secure channel, and the application protocol is then used to determine where to connect to from the remote machine. Currently the SOCKS4 and SOCKS5 protocols are supported, and ssh will act as a SOCKS server.
-R remote forward
Figure 2: Reverse Tunnels
This works by allocating a socket to listen to either a TCP port or to a Unix socket on the remote side. Whenever a connection is made to this port or Unix socket, the connection is forwarded over the secure channel, and a connection is made from the local machine to either an explicit destination specified by host port hostport, or local_socket, or, if no explicit destination was specified, ssh will act as a SOCKS 4/5 proxy and forward connections to the destinations requested by the remote SOCKS client.
-L local forward
Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side of the ssh connection. This works by allocating a socket to listen on that port on the local side, optionally bound to the specified bind_address. Whenever a connection is made to this port, the connection is forwarded thru the secure channel, and a connection is made to host port hostport from the remote machine.
I won’t even touch on agent forwarding (ssh -A) as Skylight Cyber did a great job around that. Now that we have a decent introduction let’s move onto why you are here.
SSH protocol handshake
The SSH protocol relies on the client and server negotiating parameters during an initial handshake before a secure channel can be established. From a high level this looks like:
- Protocol version exchange
- Key exchange
- Elliptic Curve DH Init
- Elliptic DH Curve Reply
- New Key
After protocol version checks the KEX (Key EXchange) process is kicked off by issuing a SSH_MSG_KEX_INIT (server -> client) message with a list of cryptographic primitives supported. The other side will also provide a list of their preference. These lists are ordered by preference as when both sides match a supported primitive, it’ll be selected. These keys will be session based as different clients have unique supported algorithms and order of preference. These exchanges are the initialization for the encrypted channel. Client and Server will use these primitives for key exchange, message authentication and ultimately data encryption. Per the RFC: RFC 4253
When invoking a ssh connection with very verbose output from the cli it will output to stdin each process in the ssh handshake, you have seen it before:
debug1: kex: server->client cipher: email@example.com MAC: <implicit> compression: none debug1: kex: client->server cipher: firstname.lastname@example.org MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: kex: curve25519-sha256 need=32 dh_need=32
In this example, its linux to linux, aes256-gcm and curve25519-sha256 have been agreed upon, with no compression. Once the server receives SSH_MSG_KEX_ECDH_INIT it can then generate it’s own Ephemeral keypair, together with the client pub key, generate the shared secret K.
At this point, the server needs to generate the exchange hash and sign it producing HS. Shown here:
All the required criteria is present for the server to generate SSH_MSG_KEX_ECDH_REPLY which is depicted below.
Note that the keypairs are ephemeral and will only be used during the key exchange and discarded afterwards. These lists of ciphers and hash algorithm can be unique enough to be used as a potential fingerprint towards identification of different client and server implementations.
A recent report by Recorded Future titled; Adversary Infrastructure Report 2020: A Defender’s View shows threat actors are using readily available opensource frameworks. Per the report, the most prolific c2 families:
The list of top offenssive security tools observed is populated by the top open source projects in offenssive security. This can be advantageous for the hunter as lots of research and expertise has gone into detecting these OFST (Opensource Offensive Security Tools). Case in point, we have available to us, Saleforce’s HASSH developed by @benreardon which has been generously released as open-source.
How does HASSH work and what can it help solve/answer? Per the README:
Detect covert exfiltration of data within the components of the Client algorithm sets. In this case, a specially coded SSH Client can send data outbound from a trusted to a less trusted environment within a series of SSH_MSG_KEXINIT packets. In a scenario similar to the more known exfiltration via DNS, data could be sent as a series of attempted, but incomplete and unlogged connections to an SSH server controlled by bad actors who can then record, decode and reconstitute these pieces of data into their original form. Until now such attempts - much less the contents of the clear text packets - are not logged even by mature packet analyzers or on end point systems. Detection of this style of exfiltration can now be performed easily by using anomaly detection or alerting on SSH Clients with multiple different hassh.
Here are some sample hassh and corresponding tools.
|fafc45381bfde997b6305c4e1600f1bf||Ruby/Net::SSH_5.0.2 x86_64-linux||Metasploit exploit module|
|b5752e36ba6c5979a575e43178908adf||Python Paramiko_2.4.1||Metasploit exploit module|
|de30354b88bae4c2810426614e1b6976||Powershell Renci.SshNet.SshClient.0.0.1||Empire exploit|
As we previously discussed, the fact that the list of supported cryptographic primitives (and its order of preference) can be quite unique across different SSH client/Server implementations allows for potential fingerprinting of both client and server. Hassh works by constructing an MD5 hash from the specific set of algorithms that are supported by various SSH Client and Servers alike. If hassh has the ability to see the actual SSH_MSG_KEX_INIT handshake. Hassh is not concerned with higher-level ostensible identifiers such as client / server version exchanges. Let’s use an example straight from the readme:
“Cyberduck” SFTP client (specifically SSH-2.0-Cyberduck/188.8.131.52683 (Mac OS X/10.13.6) (x86_64)
|Function||Algorithms for SSH_MSG_KEXINIT|
|Key Exchange methods||
In an environment which has secure shell tightly defined it would be possible to detect any client as malicious and warrant further investigation that which is outside of your approved clients and their corresponding HASSH MD% signatures. You can either use the standalone Python implementation of HASSH or if you are using Zeek, you can deploy it using the pkg manager:
pkg install hassh
For more information on utilizing hassh using zeek, please refer to hassh.zeek readme
John Althouse III gave a presentation on using Bro to detect ssh tunnels carrying TTY’s.
Through observing packet lengths of keystrokes within attached SSH sessions, insights emerged.
- SSH tunnel + another SSH channel transporting TTY
- packet length = SSH Header + [ previous ssh pckt ] + HMAC = either 76, 84, 98 bytes.
- Exact packet length dependent on block size + HMAC algo impelementations by client/server.
- Each keystroke is echoed back and can be used to inch towards better accuracy.
John released some bro scripts on github implementing his findings. As well as scripts to hunt for metasploit’s default SSL certificate on your network.
Ben Reardon expanded on his research into SSH. Building upon the previous bro releases by teammate John Althouse, he recently released packetstrider as open-source code on his github page. PacketStrider expands on the prior work
As you can see from the output, packet 22 was seen as an indicator for ssh agent forwarding (-A argument).