Is your Redis instance vulnerable to Kinsing malware

How an exposed port caused downtime in our Slack app service

Hey there, thanks for checking today’s issue. I am gonna narrate an incident that happened to us this week and what we have learned from it.

On 10th January 2022, we were resuming our work on a Monday.

One of our products, Dixiapp, is used for our daily standups. Every day, it asks a set of questions along the lines of what did you do yesterday, what are you planning to do today, etc. to which we answer and our backend records the responses and submits a report to a chosen channel.

Dixiapp (adorably called Dixi) is supposed to trigger a question at exactly 9 AM, but that day it didn’t.

What happened? I am the one responsible for anything happening to this app so I got into debugging it.

Our health check monitoring was working fine which means the Gunicorn server was not down. Neither the Nginx.

The way our app works is, whenever we send a message on the Dixiapp channel, Slack sends an event request to our servers and we process it. This triggers a response from our servers which is displayed on Dixi’s channel. But there is no response that day. Interesting. So has Slack stopped sending messages? I started debugging and found that it’s not a Slack issue.

Finally, I had to SSH into our production server. I checked gunicorn and Nginx systemd processes. They were running as expected. After checking the logs and finding nothing, I ran the most versatile command which shows us everything about the system, i.e. htop

htop shows us all the processes that are running in the system and how much CPU, memory and network bandwidth are they consuming.

And when I ran it, I was shocked to see the result!

All our CPUs were maxed out! They were operating at 100% capacity. No wonder other processes were not running properly. I took a look at the processes and saw a suspicious one consuming all the CPUs.

See that? Hmm, who is this kdevtmpfsi? 🤔

After some exploring, I understood what’s happening. Our Redis port which was exposed to the internet was the culprit. Or rather, it was the victim. Someone tried to exploit a known Redis issue to gain access to our system.

So what exactly is the kdevtmpfsi process?

kdevtmpfsi is a crypto miner. Hackers/script kiddies try to exploit vulnerable ports on a server and install this program to run their mining operations. This is carried out through a malware called Kinsing.

First things first, I gotta get rid of this malware and get the system up and running. After going through a bunch of articles and StackOverflow questions (duh, I am a true developer), I figured to remove it. I am attaching some helpful resources regarding the malware and how to tackle it at the end of this article.

How does the attack occur?

I will explain.

According to my research, it is based on exploiting the CONFIG command of Redis. Here’s a breakdown of the attack without giving the exact details so that script kiddies won’t exploit the loophole

1. The attacker can gain access to the shell using the exposed Redis port via telnet

2. Once he gains access, the attacker can reconfigure Redis during runtime using CONFIG SET command.

3. He first sets the data directory for Redis to something specific and then sets the data db file to a file that contains cron commands to download and run a remote script.

4. This script downloads and runs a crypto miner.

5. Even if you kill the process, thanks to the crontab, it restarts every minute

This consumed 100% of our CPUs resulting in non-availability of the service, while the attacker is enjoying their free CPU 😒

💡 Lesson learned: Be careful when you expose your ports to the internet.

---

---

How to secure your Redis instances?

This incident led me to think deeper about security. I started understanding the architecture and security issues of Redis.

Redis is designed to be accessed by trusted clients inside trusted environments. This means you need to run your Redis instances in a self-contained environment which is not directly connected to the internet. Only the computers running the application should directly access it.

Following are some security measures you need to take while having a Redis instance.

1. Binding Redis to a network

The main configuration file of Redis is something we should look at carefully and understand what each option means. The file is named redis.conf by default and is located in your Redis installation directory.

You can bind Redis to a single interface so that it will only receive requests from that interface. This is done by adding a line in your redis.conf file like this:

bind 127.0.0.1

2. Protected mode

For versions since 3.2.0, Redis has added a special setting called protected mode. Redis configuration doesn’t have password protection by default. If you run a Redis instance with the default configuration, then it enters a special mode called protected mode.

When Redis runs in this mode, it only replies to queries from loopback interfaces, that is localhost (127.0.0.1). If other clients try to connect, they are responded with an error.

3. Authentication via password

As I mentioned above, Redis by default doesn’t have password authentication by default. You can enable it by adding the following line to your redis.conf file

requirepass <password>

Note that this is just a tiny layer of security added since the password is stored in plaintext and anyone with access to this file can read it.

4. Disabling specific commands

One of the best security features that Redis provides is to disable commands which you don’t require. You can also rename them to some unguessable name. Here’s how you rename a command:

rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52

To completely disable a command you can rename it to the empty string as below

rename-command CONFIG ""

---

---

Takeaway

When we start building an application, we tend to take security for granted. We tend to expose unnecessary ports and forget to close them. Every assumption that we make while building an application should be noted down and revisited once we have built the app.

---

What am I reading this week?

“The Making of a Manager” by Julie Zhuo

I would recommend this book to anyone who is transitioning into a management role.

Resources:

1. https://www.trendmicro.com/en_us/research/20/d/exposed-redis-instances-abused-for-remote-code-execution-cryptocurrency-mining.html

2. https://blog.trendmicro.com/trendlabs-security-intelligence/more-than-8000-unsecured-redis-instances-found-in-the-cloud/

3. https://redis.io/topics/security

4. http://antirez.com/news/96

---

You can also follow me on Twitter. Do drop a Hi!