Posts About

Gee...You locked yourself out of your VPS? Try this next time.

Learn how to avoid locking yourself out of your VPS by implementing a simple yet effective strategy.

By Collins Lagat
29/10/2024

Summary

Learn how to avoid locking yourself out of your VPS by implementing a simple yet effective strategy.

Intro

I have lock myself out of several VPS before. The first time was when I created a VPS with an SSH key I no longer had on my machine. Another time was when I tried to deviate from an SSH tutorial to do something custom and ended up disabling root and password login without adding SSH keys for the non-root user. Another time when I was playing with UFW and blocked port 22. These are growing pains of learning to metaphorically getting down and dirty with a VPS. You’ll trip over a configuration setting. Be out of your depth when dealing with something new. Be so anxious that you forget the correct sequence of events. And find yourself pulling up tutorials on how to be safe when doing operations inside the server. You’ll make so many mistakes, such as flipping down switches that should have been slipped up, and many of them were caused by trying to finish the operations too fast. You know, something you are only rewarded when you take your time.

How we shut ourselves out of our VPS

If you collected a bunch of sysadmins and asked them the ways they’ve locked themselves out a server, the list will be something like this:

  • Disabling Root login before creating another user to login as.
  • Disabling password authentication before fully setting up SSH Key authentication.
  • Adding our user(s) to the DenyUsers list.
  • Failing to add our user(s) to the AllowUsers list when we decide to use this configuration setting.
  • When adding users to the AllowUsers list and we’ve also decided to restrict the host the users can access from, but we add the users with the wrong host or host we don’t have access to.
  • Changing the SSH port to something other than port 22, but forgetting which port it was.
  • Blocking port 22 when setting firewall rules.
  • Losing or deleting your SSH keys e.g. accidentally deleting the ~/.ssh folder

This is not an exhaustive list as things such as, expiring certificates and installing security software such as fail2ban, create new scenarios where you can find yourself trying to kick down the door to let yourself in. The aim isn’t necessarily to know all the way you can lose SSH access to your VPS/Server, but have a way to recover once you’ve realised that you can now only see the inside of your house through your window.

Solution

Realising that you can’t log into your server triggers a panic just the same way the panic sets in when you realise that you don’t know where you keys are when you are at your door. And let those keys you’ve lost be the keys to someone else house! But just like locking yourself out of your house, the thing that abates the panic, is the confident knowledge of the existence of a spare key - tucked away in a secure location. Backups. How inconvenient are they when you are making them! But how glad are you when you have them when you actually need them! How glad are you to have 6 months of life expenses saved up when you are laid off? What’s that? Did you say that you’d be relieved? Did you say that it will restore you confidence?

So how about you have a backup SSH session?

Just one extra one.

Just in case.

You know?

When you eventually that thing again.

You don’t want to have to go through that again?

Plan B

Make it a habit of opening up 2 SSH sessions - at least in all sessions when you’ll be touching anything related to configuring the server. Then do all your work on one of the sessions and at the end test if you can still connect to the server. If you find out that you can’t log in, you still have the other session to fix your oopsies. This backup session essentially acts as your spare key that you leave at your parent’s house in case you lost yours. But what if I accidentally close the terminal window where my back up session is running? Well, you’ll have to jump straight to [Plan C](#Plan C). You have accidentally dropped you spare key into the sewer on your way back to your house.

Why didn’t you keep it in your pocket? You just had to carry it in you hand, didn’t you? It can be knocked out of your hand you know? The pocket at least keeps it safe unless something catastrophic happen to you - or at least your pants.

We are still taking about sessions, right?

What if we could put our session in a metaphorical pocket? Like, it can survive most incidences, except catastrophic ones such as switching off the machine. Well, such pockets exist. They are called terminal multiplexers. Whatever you run in them are automatically run in the background independent of the shell session that you are running. Therefore even if you close the terminal, you can still pull up the original session that you open. The only time you lose those sessions is when you manually close them (gracefully or otherwise), when the multiplexer’s session crash due to the multiplexer’s bugs or when the host machine switches off or restarts. Examples of these are tmux, zellij and screen.

Does this work if I lose my keys? Is the SSH session still running? If yes, you are good. Enable password authentication and add new key. If the SSH session isn’t running, got to [Plan C](# Plan C). Note however that losing SSH keys is one of those mistakes that can also be mitigated by backing up your keys, preferably off your machine and on an external storage device. This shouldn’t be an emergency as you’d have a copy of the keys. They are text files at the end of the day.

Plan C

This solution is when you have no backup session or permanently lost your keys and you have no other way to log in. This solution relies on the virtualisation software having a way to log into the virtualise servers running to top the hypervisors. The good news is that most software support this and most VPS providers offer GUI interfaces to access them. They are usually presented as web consoles. Check out this solution for each of your VPS providers. If you are using internal VPS, refer to the documentation of your virtualisation software…or what’s more likely the situation, consult the admin in charge of it and they’ll help you. It’s just the same thing that the public cloud providers provide - only internal.

Conclusion

Fin