Seeing an increase in MySQL attacks hitting your network and interested in knowing more about them? If so, then these posts are for you. They have all the fun involved from noticing an increase in traffic to extracting malware from a packet capture and analysing it. If you like the thrills and spills of scripting information processing tasks, then read on. If you’d rather I just got to the point and presented information about the attacks, then you can probably skip this first post.
Background
I recently noticed an increase in the amount of traffic hitting my honeynet, and upon checking the Honeywall’s web interface (Walleye), I noticed that it was inbound MySQL connections. Being curious, I used the Walleye interface to download a pcap file of the last 24 hours worth of honeynet traffic, and set about examining the MySQL connections in Wireshark.
What I saw was a number of inbound MySQL connections, authenticating as root, to MySQL’s ‘mysql’ database. All of the login attempts succeeded because Dionaea was emulating a MySQL server and just accepting any credentials that the client cared to give. The client then issued the MySQL command ‘set autocommit=0’, which Dionaea doesn’t handle and instead sends an error response back to the client. Upon receiving the error, the client would disconnect, reconnect, and then attempt to authenticate with a different password (seemingly without, as it turns out, trying the obvious one of ‘password’).
If I wanted to find out more about these attacks, I was going to have to use a real MySQL server. Despite suspecting that it was attacking MySQL on Windows, I quickly built a virtual Linux host (a process which I suspect may have been quicker if my hard disk wasn’t lying about its physical block size) and added it to the honeynet.
Using a Linux MySQL server when the attacks are targeting a Windows MySQL server is not a bad idea, as there is less chance of the honeypot getting infected and then attempting to attack other Internet users (the honeywall box will restrict the number of outbound connections and alert me if this does happen). It also means that the honeypot can keep running and allow for the capture of multiple connections without having to go through the clean up process after each one.
I could have reconfigured Dionaea to not listen on the MySQL port and then installed the MySQL server on the Dionaea honeypot. That would have removed the need to NAT inbound MySQL connections. However, I didn’t know what these attacks were going to do, nor whether they were targeting Windows servers or Linux (or other UNIX) servers. As such, there was a chance that these attacks would successfully compromise the MySQL server and give the attackers access to the system. Consequently, I decided to build a separate virtual machine which I could either do a forensic analysis of (without the artifacts created/left by Dionaea), or throw away if an attack did compromise it. All without affecting Dionaea’s operation.
Virtual Machine Creation
I used virt-manager to create a virtual Linux host with 512 MB of RAM, and a 1 GB hard disk image. I used the raw disk format as that would make it easier to do a forensic analysis, should we need/want to. I was later reminded, when I went to create a snapshot of the ‘clean’ state, that I needed to use a qcow disk format to create snapshots.
Never mind — I got around that snapshot issue by simply making a copy of the disk image file after installing the operating system and applications, but before allowing any network traffic to it. Should we want to find any files that have changed, we can then mount the two images at the same time and diff -r the two mount points, or get md5sum on to it.
I then installed the MySQL server package, connected the virtual host to the virtual network, and set about modifying the iptables (Linux firewall/NAT) rules to send inbound MySQL connections to my new MySQL server and not to the Dionaea honeypot. Obviously, if you have more than one valid Internet IP address then you can just use a different address. I don’t, and so all the honeynet traffic has to use the one Internet IP address which gets NATed by the Internet gateway box to an internal IP address.
Logging
Something that would be handy, would be to send a copy of the system’s log messages to a remote server. One of the things attackers (or their scripts) have a knack of doing, is removing evidence of their attack from the system log files.
Sending the log messages to a remote server would be a way to safe guard the log messages, however, this needs the IP address/hostname of the log server in the syslog configuration file on the honeypot host. This lets the attacker know that there is another target for them to attack, especially if they want to try to destroy log data to cover their tracks.
A nifty trick that you can use in this situation, is to configure syslogd on the honeypot host to send log messages to a remote logging server that doesn’t actually exist. Syslog messages are usually, unless your syslogd allows and you configure it to do so, sent in UDP packets. This means that there is a chance that they could go missing as unlike TCP, UDP doesn’t set up a connection before sending the data. More importantly, in this situation, it also means that log message data gets sent in a single packet without the remote server having to respond. That means we can send the log message data to a server that doesn’t actually exist.
How are we going to log the data if it gets sent to a host that doesn’t exist? The Honeynet Project’s Honeywall installation, by default, will log all packets to a series of pcap files, so as long as the syslog packet gets sent over the network through the honeywall by the honeypot host, the honeywall host will capture and log it.
The best IP address to use would be one that is not a valid Internet address, so one of the reserved 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16 addresses, that isn’t in use (locally). It would also be a good idea to use an IP address that won’t get routed to the Internet, or to configure the router on the other side of the honeywall to simply drop the syslog packets to this address. Note that the chosen IP address needs to (eventually) get routed to a router which not only exists, but which is on the other side of the honeywall. Otherwise the syslog packet won’t traverse the honeywall, in which case the honeywall can’t log it.
Don’t pick an IP address that is on the same IP subnet as any of the honeypot host’s interfaces. The reason being, if the address is on the same subnet, it is reachable (or would be if it existed) directly and won’t get sent to a router. This will cause the operating system to send an ARP request for the remote host’s address which, given it doesn’t exist, is unlikely to get a response. Hence the honeypot will never send the UDP packet containing the log message data.
Modify your syslogd‘s configuration file to instruct it to send a copy of all log messages to a remote server, and use an IP address that isn’t in use. Check the documentation for your version of syslog, but this is usually done by adding a line similar to the following, to the syslog daemon’s configuration file(s):
*.* @555.12.7.14:514
Where *.* is the facility and priority (log level), respectively. These specify what to log. 555.12.7.14 would be the chosen IP address of your imaginary syslog server, and 514 is the syslog port. Note that I have seen some differences between syslog versions. Normal syslogd (I’m using rsyslogd) may not take the :514 on the end but rather just an IP address/hostname. Also, from memory, Solaris’ syslogd won’t allow you to specify * as the priority (the second asterisk). Some syslog implementations need a tab character instead of spaces.
With newer syslog software supporting TCP connections for more reliability, make sure that you configure your syslog server to use UDP, otherwise syslog will fail to set up the TCP connection and won’t send the log message data.
It is also a good idea to turn on the syslog ‘mark’ messages. To do this, add the line $ModLoad immark to the /etc/rsyslog.conf configuration file (or to a file in the /etc/rsyslog.d/ directory if there is a $IncludeConfig line for it in the main configuration file). The ‘mark’ messages are typically produced every twenty minutes and serve as a heartbeat.
The lack of these ‘mark’ messages suggest that the honeypot host may have been demolished to make way for a new cyberspace bypass, or probably more likely that it has died, possibly the result of an attack. It could also indicate a problem with the logging system, for instance an attacker could have killed the syslog daemon to prevent the system from logging their actions. A program like logsurfer can alert about missing ‘mark’ messages, but I won’t go in to that now.
So, now that you’ve gone to all this trouble to set up logging, you’ll want to know how to view the log messages right? You can use tcpdump or wireshark to look at the pcap files created by tcpdump running on the honeywall host.
Alternatively there is a rather nifty command called tshark that is brilliant for extracting data from pcap files, especially in a script. Since I will be using tshark in the next post in the series, I shall keep you in suspense by talking about it then.
Network Address Translation
The next step is to change the destination address of any packet coming in to the MySQL port of 3306/tcp, so that the packet goes to the new MySQL honeypot and not to the Dionaea honeypot. We then also need to change the source address of any packet coming from the MySQL port on the MySQL honeypot, back to the address of the Dionaea honeypot. If we don’t do this latter step, then the reply packets will appear as if they are coming from a different host than that to which the original packets were sent.
When configuring the NAT iptables rules, I noticed a difference in behaviour between the honeywall box (Linux kernel v2.6) and my firewall box (Linux kernel v3.2). The latter seemed to automatically NAT the source address of the reply packets, whereas the former didn’t. Adding an SNAT (source NAT) rule to the honeywall box didn’t work because a source NAT rule (to change the source address) can only be used in the nat table’s POSTROUTING chain. The POSTROUTING chain wasn’t being used because, I suspect, the honeywall was bridging packets and not routing them.
For anyone wanting to do a similar redirection, I used the following iptables command on my firewall box:
iptables -t nat -A PREROUTING -p tcp -d 555.13.7.3/32 --dport 3306 -j DNAT --to-destination 555.13.7.4
Actual addresses may differ from those shown — I’ve used the standard Hollywood trick of using 555, to obfuscate the IP addresses. The .3 address is that of the Dionaea honeypot, and the .4 address is that of the MySQL server/honeypot. 3306 is the MySQL port number.
That command was enough to redirect inbound MySQL connections to the new MySQL honeypot — the kernel seemed to automatically rewrite the source address in the reply packets.
MySQL Configuration
MySQL’s default configuration means that it only listens for network connections on the loopback address (127.0.0.1), which will stop it from accepting connections from the local network/Internet. You will need to change the MySQL configuration file which, on a Debian server at least, is /etc/mysql/my.cnf. Change the bind-address from 127.0.0.1 to 0.0.0.0. The IP address 0.0.0.0 corresponds to INADDR_ANY and when passed to the bind() system call, will cause a later listen() call on the same socket, to listen on all network interfaces.
The next MySQL issue that you will come across, is that it won’t allow you to authenticate from a remote host. The following message informs you of this:
Host '...' is not allowed to connect to this MySQL server
You can see this error message if you attempt to either connect with the mysql -u root -h mysql command, or if you are just testing the connection by attempting to connect with telnet <ipaddr> 3306.
MySQL is denying root access from remote hosts because there isn’t an entry in the mysql.user table to allow such a connection. To allow root to connect from anywhere (this is usually a really bad idea, by the way, and later posts in this series will show you why), use the following MySQL commands:
GRANT ALL ON *.* TO root@'%'; FLUSH PRIVILEGES;
Be very careful running that first command as it will give root user access to all tables in all MySQL databases, without requiring a password!!
I initially went for a bit of security and appended IDENTIFIED BY ‘password’, which would only allow root access from remote hosts if they gave the password of ‘password’. However, the attacks were still failing suggesting that in their brute force attempts to guess root‘s password, they never tried the obvious one of ‘password’.
The FLUSH PRIVILEGES command is there to make sure that the changes to the privileges take effect. MySQL does seem to cache them, but I suspect that that is more a problem when revoking privileges rather than when granting them.
Wait for the next attack
We now have our new MySQL honeypot built, MySQL configured to allow unauthenticated remote root access and cause security administrators to cringe, and the network translation in place to redirect port 3306/tcp to the new MySQL honeypot and not to the Dionaea honeypot. So now we play the waiting game or, if you prefer, Hungry Hungry Hippos.
Hungry Hungry Hippos is one of those old-fashioned games that is played on the floor or on a table, and not on a mobile phone, game console, or a computer. This has the advantage in that you can still play it if your mobile phone battery goes flat or your Internet connection dies (neither of which were much an issue when the game was released), and it doesn’t need the purchasing of expensive hardware. It is, however, susceptible to a disgruntled player nicking off with the marbles, and I dare say that you would probably get a few strange looks if you started playing it on the bus/train on your way to school or in most other situations where mobile phone games are played.
Alternatively, if neither the waiting game nor Hungry Hungry Hippos take your fancy, dive in to the next post in this series where I have another idea for killing time whilst waiting for the next MySQL attack.