Quick Look

• Generate SSH key pair. This will create 2 files: ~/.ssh/id_ed25519 and ~/.ssh/id_ed25519.pub

ssh-keygen -C "user@example.com"

• Upload the public key to ssh server.

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host

• Connect to ssh server.

ssh user@host

What is SSH?

An intuitive understanding of SSH is buried somewhere within the Wikipedia definition:

“The Secure Shell Protocol (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network.” wikipedia

If you are a student, this answer will probably be enough to satisfy your professor when he’s marking your exams. But if want to use SSH in practice, you’ll probably need something more than an academic answer. You’ll actually need an understanding. And some understandings are easier to build when you are given analogies. My analogy is thinking of SSH as similar to a student looking for someone interesting to be his roommate. A good place to start understanding something is understanding what problem it’s trying to solve.

Roommates

Matteo is an Italian University student looking forward to the next academic year. His friends can really tell how excited he is by the way he’s passionately narrating his summer vacation in Cebu, Philippines. The people. The language. The region. He wished he could return there some day. So you can imagine his surprise when one of his friends who has been keenly listening tells him, “My classmate…Angelo. He’s mum is from Cebu.” Matteo was practically begging for an introduction.

“My name is Matteo Costa. But my friends call me Matteo”, said Matteo as he shook Angelo’s hand. After a pleasant chat. Matteo thought about asking Angelo to be his roommate for the year. If he’d be interested, he’d have someone to share stories with, in the local language he had learnt to love. All he needed to know is if Angelo spoke the language. He asked Angelo a question, in broken Cebuano, about a certain town in Cebu that only a person who’s been there would know. And to his delight, Angelo responded correctly in native Cebuano. Angelo, spotting the glee on Matteo’s face, asked, “First of all, that sound like the most Google Traslate answer, if there was ever one. Second of all, why did you ask that?” Matteo tells him that he actually lived their for 3 months. “You don’t say,” said Angelo as it was now his turn to ask the questions. He asked Matteo another question about the same town that only a person who’s lived there would know. And to Angelo’s pleasant surprise, Matteo answered correctly, albeit with the same broken Cebuano he asked his question with. “Have you found a roommate yet?” asked Matteo.

Recap

In this story, we have most of the things needed to understand SSH. We have a student (Client) who wants a roommate who reminds them of a good time he had in the Philippines (use case of SSH), and he asks a potential candidate (Server) a question (public key) that only a person from that town in Cebu would know the answer to (private key). The candidate also wants to know if the student is who he says he is (authentication, either key-based or password-based) by asking the student a question that the candidate already knows the answer to. And if all this goes well, they’ll share a secure living space (SSH session) for the academic year (session duration) where they’ll both have the same house key (shared session key).

Check out this article for more information on how the SSH connection process works.

Bringing it all together

So you can say that Matteo wants to be able to walk from school and spend time with Angelo in a private place that only the 2 of them can access. Pause. A Client wants to be able to reach out to a Server and privately chat back and forth with it without letting anyone else know what they are chatting about. Wait…isn’t starting to sound like the definition of SSH?

Technical definition

SSH - Secure Shell - is an open network protocol that allows a client to securely connect to a server over an insecure network. That’s it.

Usage

By being an open protocol, SSH lends itself to multiple implementations with a popular implementation being OpenSSH, available on a Linux, Mac or Windows device near you. And with that being said, all the commands used to show the practical usage of SSH will all be based on the OpenSSH implementation of SSH. OpenSSH is an OpenBSD project that provides a suite of cli tools, but we’ll only delve into 9 of the most common ones here in order to show how you can use them for secure connection, key management, and performing operations on remote machines.

The 9 tools we’ll be using are:

Additionally, some of the tools above make use of certain files depending on whether it’s the client or the server. Clients make use of:

• ~/.ssh/known_hosts
• ~/.ssh/config
• key pairs found in ~/.ssh

Servers make use of:

• ~/.ssh/authorized_keys
• /etc/ssh/sshd_config
• key pairs found in /etc/ssh

Clients store the public keys of remote servers it intends to communicate with (and keep communicating with in future) in the ~/.ssh/known_hosts file, while remote servers store the public ssh keys of clients, allowed to connect to it without a password, in ~/.ssh/authorized_keys. Client-specific configuration is stored in ~/.ssh/config, while server-specific configuration is stored in /etc/ssh/sshd_config.

Background Services

1. sshd - The OpenSSH server

The sshd is a daemon process that acts as the server for the SSH protocol. It listens for connections from clients and provides secure connection between it and the client. sshd is usually started and managed by a service manager such as systemd, and it’s behaviour is configured by editing the /etc/ssh/sshd_config file. The most common configurations are:

1. Setting PermitRootLogin to no. By default, it’s set to yes as root is the only user created during installation. But once you create a non-root user, it’s recommended to set it to no.
2. Setting PasswordAuthentication to no. By default, it’s set to yes as SSH key authentication has not been set up yet. However, once you set it up, you can set it to no.
3. It’s recommended to combine the above two configurations, i.e. set both PermitRootLogin and PasswordAuthentication to no, to further enhance the security of the server. This is done after creating a non-root user, and setting up SSH key authentication between the client you want to connect from and that user on the server.

Note that you can still access root by switching to root once you’ve logged in as a non-root user. For this, use the command su -i. Alternatively, you can create a non-root user with sudo privileges.

2. ssh-agent - The “Login With Google” of OpenSSH

The ssh-agent is a tool that runs as a background process in the current shell session that stores unencrypted private keys and encrypted private keys that have been unencrypted with by a passphrase in memory. The main utility of ssh-agent is that you can add keys to it once (during the shell session) and you login in to several servers configured to use those keys. This eliminates once of the main excuses of not creating private keys encrypted with passphrases - trying in the passphrase each time you log in to a server. Of course this doesn’t eliminate the need to memorize the passphrase. You still have to remember that. Unlike sshd, ssh-agent is not started by default, although some Linux systems do start it for you. Since ssh-agent is not a system-wide service, but a shell-session-specific process, it has to be started for each shell session you open e.g. opening a terminal window. To start ssh-agent on your system, run the following command:

eval $(ssh-agent)
# returns
# Agent pid [ppid]

When the agent is started, it creates a Unix-domain socket and stores the path to it in $SSH_AGENT_SOCK environment variable. It also stores the process id of the agent in $SSH_AGENT_PID environment variable. These environment variables provide an easy way to know if the agent is running or not, by outputting it’s values in the terminal e.g. echo $SSH_AGENT_SOCK.

Managing Keys

1. ssh-keygen - You can skip this, but it’ll be a pain

The ssh-keygen command is used to create a key pair for public key authentication. It accept multiple arguments, but the 2 main ones are -t and -C.

-t specifies the key type, although leaving it blank makes it default to the most secure one (ed25519 as of September 2024). They key types supported by OpenSSH are: RSA, DSA, ECDSA and ED25519.

-C specifies the comment which is a human-readable string that you can use to identify the key. If you don’t specify -C, the default comment is the account user’s name and hostname.

ssh-keygen run’s in an interactive mode, and one of the options you’ll have to choose is whether or not you’ll want a passphrase for your private key. The passphrase is used to encrypt the private key, and this adds a layer to security for your key if it ever leaks or get stolen.

# Simple
ssh-keygen -C "your_email@example.com"

# Specify key type
ssh-keygen -t ed25519 -C "your_email@example.com"

2. ssh-copy-id - If you skipped ssh-keygen, you can skip this too

The ssh-copy-id command is used to install a local public key into a remote server’s authorized keys. It installs it for the user specified in the command.

ssh-copy-id -i ~/.ssh/custom_key.pub user@host

3. ssh-add - If you skipped ssh-keygen, you know the drill

The ssh-add command adds keys to the ssh-agent service running in the current shell session. By default, running the command with no args adds the default identities, i.e. id_rsa, id_dsa, id_ecdsa and id_ed25519, if they exist.

ssh-add ~/.ssh/custom_key

You can also use it to list the keys that are already stored by the agent:

# Lists fingerprints of all identities in the agent
ssh-add -l
# or
# Lists the actual public keys in the agent
ssh-add -L

4. ssh-keyscan - It just has one practical use

The ssh-keyscan command is used to gather the public SSH keys of remote servers. The server stores a key pair for each key type in /etc/ssh/ to give a client flexibility to generate keys with any supported key type they want. However, some hosts only support certain key types, and you can use ssh-keyscan to get a list of those keys in the server. GitHub for example doesn’t support dsa key type (as of September, 2024):

ssh-keyscan github.com

# returns

# github.com:22 SSH-2.0-babeld-fdcea1d49
github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
# github.com:22 SSH-2.0-babeld-fdcea1d49
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
# github.com:22 SSH-2.0-babeld-fdcea1d49
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
# github.com:22 SSH-2.0-babeld-fdcea1d49
# github.com:22 SSH-2.0-babeld-fdcea1d49

Performing Remote Operations

1. ssh - What you think of when you think of SSH

Basics

The ssh command is a client use for logging into a remote server. This is the command that most people think of when they say that they want to learn about SSH. The simplest usage is just specifying the domain or ip address of the remote server. It will assume that you want to log in as the current user.

ssh host

But if the remote user is different from the current user, you can specify the user name as well:

ssh user@host

Either way, once you’ve authenticated yourself (whether by passphrase or ssh key), you’ll be dropped into an interactive shell where you anything you type are sent as commands to the server.

Running Single Commands

But if all you want to do is just run a single command or a single script, it’s easier to just append the name of the command or the script to the end of the ssh command:

# Run single command
ssh user@host ls

# Run single script

ssh user@host bash < script.sh

Using Custom SSH Keys

By default, the ssh command will use the default private keys, i.e. ~/.ssh/id_rsa, ~/.ssh/id_dsa, ~/.ssh/id_ecdsa and ~/.ssh/id_ed25519 to connect to the server, and if none of them matches, it will result in a “permission denied” error. There are cases where you want to create an SSH key pair with a specific (non-default) name for a specific purpose, or are sent an SSH key pair with a specific name or one with a standard name but has a name that’ll clash with your other keys, or the key is in a non-standard location on your file system, etc. Regardless of the reason, there are 2 solutions for this:

1. Use the -i option to specify a path to a specific key

ssh -i ~/.ssh/custom_key user@host

2. Add the custom key to ssh-agent using ssh-add:

eval $(ssh-agent)

ssh-add ~/.ssh/custom_key

ssh user@host

Adhoc-configuration of the ssh command

The behaviour of the ssh command can be modified by changing the options in the ~/.ssh/config file. However this file isn’t generated by default and thus ssh command is wholly controlled by the global /etc/ssh/ssh_config file.

To modify the behaviour of the ssh command at runtime, you’ll need to pass the -o option to ssh specifying the configuration option you want to override. The most common configuration option overrides are:

1. Skipping host verification for hosts you trust by setting SkipHostKeyChecking to no. Note that you only do this if you trust the host you’re connecting to. This is commonly used in CI/CD environments and in building Docker images.

ssh -o StrictHostKeyChecking=no user@host

2. Keeping SSH sessions alive: by default, the ssh command will close the connection after a period of inactivity set by sshd. This can be overridden by setting ServerAliveInterval to a value in seconds or setting TCPKeepAlive to 1. However, ServerAliveInterval might be more appropriate as TCPKeepAlive may not work in cases where the remote sever firewall is configured to drop empty TCP ACK packets.

ssh -o TCPKeepAlive=1 user@host

# or

ssh -o ServerAliveInterval=5 user@host

2. scp - Have you wanted to use the cp command, but across the internet?

The scp command is used to securely copy files between a client and server over unsecure internet in a non-interactive way.

# put file on server
scp /path/to/file user@host:/path/to/file

# get file from server
scp user@host:/path/to/file /path/to/file

# copy entire directories
scp -r user@host:/path/to/dir .

3. sftp - FileZilla, but in the command line

The sftp command is used to securely copy files between a client and server over unsecure internet both in an interactive and non-interactive way.

# get file from server
sftp user@host:/path/to/file /path/to/file

# put file on server
sftp user@host
sftp> put /path/to/file

# copy entire directories
sftp user@host
sftp> put -r /path/to/local/dir
# or
sftp> get -r /path/to/remote/dir