Keep getting hacked, want some insight on how they might be getting in
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Rep:
Keep getting hacked, want some insight on how they might be getting in
I have a server at OVH that hosts all my websites as well as other people's. Nothing super serious but it's basically my main online presence. Long story short my web server was running fairly old software (CentOS 6.x fully updated to last point they issued updates for it) and was due to be upgraded and I was procrastinating. Got hacked. So without any choice I order a new server to start migrating stuff to (from offline backups) and it gets hacked again. This time it was running new software, Debian 11 and everything was vanilla since I was still in the process of setting it up. Both attacks appear to be similar in nature.
When I look at the apache logs, I see something like this:
Nov 27 17:56:53 server04 dovecot: pop3-login: Disconnected (no auth attempts in 2 secs): user=<>, rip=x.x.x.x, lip=x.x.x.x, session=<UL3Q7sjRwlorgSFj>
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Followed by many white spaces
I'm guessing these are special characters and vim is just representing them that way. It looks like some kind of injection of sorts.
I figured the first one was simply due to it being a known exploit and fact that I had not updated the OS in a while, but now that it happened to the 2nd one with newer everything, I'm really starting to wonder what in the world is going on. Clearly somebody is attacking me and this is targetted and a rather sophisticated attack.
Anyone ever see anything like this before?
After this attack the server basically becomes inaccessible. I think they did something to SSH so the keys don't work to login to the server, and changed the password. (still had the password auth enabled temporarily). I do not see any signs of brute force attempts. This really seems like some kind of auth bypass attack.
Oh and this bit of log in /var/secure is probably what did the first server in, looks like they were able to load something malicious in /usr/sbin and run it. It also seems to have done somewhere where SSH failed, and so did apache and pretty much everything, except DNS kept working and server was still pinging at that point.
Code:
Nov 24 09:52:02 server03 runuser: pam_unix(runuser:session): session opened for user root by (uid=0)
Nov 24 09:52:02 server03 runuser: pam_unix(runuser:session): session closed for user root
Nov 24 09:52:03 server03 sshd[2009]: error: Bind to port 87 on 192.95.14.97 failed: Permission denied.
Nov 24 09:52:03 server03 sshd[2009]: fatal: Cannot bind any address.
Nov 24 09:52:43 server03 userhelper[2506]: running '/usr/sbin/setup ' with root privileges on behalf of 'root'
Nov 25 21:40:50 server03 runuser: pam_unix(runuser:session): session opened for user root by (uid=0)
Nov 25 21:40:50 server03 runuser: pam_unix(runuser:session): session closed for user root
Nov 25 21:40:52 server03 sshd[2077]: error: Bind to port 87 on 192.95.14.97 failed: Permission denied.
Nov 25 21:40:52 server03 sshd[2077]: fatal: Cannot bind any address.
Nov 25 21:41:01 server03 userhelper[2574]: running '/usr/sbin/setup ' with root privileges on behalf of 'root'
I kept a backup of that file so I can try running it in a VM to see what it does, but have not gotten to it yet.
I had the 2nd server in rescue mode, which is a mode that the data centre offers where it boots into a rescue image and then I can SSH in with a provided long password. That got hacked too. Somebody is clearly attacking me right now with some kind of sophisticated attack. I booted it back up in rescue mode and I'm watching /var/log/syslog now just to see if I spot anything weird. I don't think this is a SSH brute force attack unless that weird stuff I'm seeing in the logs is a way to erase the logs. (maybe issuing a bunch of backspaces as part of the username?)
Live, I would be seeing the attempts though. So I will let this run overnight again to see what happens.
They have not touched the original server which is now in rescue mode though... this whole thing is just so strange. At this point I just want to try to figure out how they got in, I don't think it's just the thing of it being out of date considering the same attack worked on the new server.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
It's hard to tell the damage as they seem very good at hiding their traces. Basically from a user perspective, apache was down hard, SSH was also down, and so was mail (SMTP and POP3). In rescue mode I do have access to the file system so I do have the option to poke around more if there is anything specific I can maybe look at. I have not really found anything new, and even that setup file has an old date, so I wonder if they turned the clock back randomly so that file dates do not stand out.
On the 2nd server, the damage seemed to be minimal as apache still worked, except it looks like they changed the SSH password or at least changed some settings so that I can no longer log in. SSH itself was running but was not able to log in at all. I managed to look via rescue mode and the keys appeared to be there though so it's very strange, but when I tried to login it would not let me login with either the SSH keys or with the password. I had not disabled that yet.
I had fail2ban setup on the original server, SSH was also on a non default port, and I also had key auth setup only, no passwords. I don't see anything in the logs about brute force, though I have had it happen before where people were trying, but it was a really strange attack since they were only trying like every minute and it was random usernames that were not valid.
The attack seems to originate through apache as the time stamp in the log after that big text is basically the time stamp the server stopped responding to my monitoring system.
Edit: The 1st rescue session got hacked now. (basically to get into the 1st server)
At this point I just want to try to get as much forensics data as I can and cancel both and find another hobby.
Last edited by Red Squirrel; 11-27-2021 at 08:21 PM.
It sounds as if you took every reasonable precaution, and it does seem to have been a malicious attack in the light of the damage you report. By "malicious," I do not mean it was directed at you personally, but that they were up to no good. Bad guys love to penetrate web servers because it gives them access to the site's visitors.
A web search for "how to attack apache" might turn up some helpful articles.
I am no pen-testing expert. Hopefully, someone who is will join this thread with some suggestions as to how they got in.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
Managed to get a rescue session up and quickly firewalled out every IP but my home one. Now I can rsync my stuff back to my home server without interruption then cancel this server and get a refund for the prepaid months. This will give me more time to do a post mortem and try to figure out wth happened. I consider all that data compromized of course and will not restore it anywhere.
As a side note this incident is a very good reminder to everyone to check your backups once in a while. Thankfully mine worked, though I was only pulling them once a week, and I perhaps should have been doing it every day.
What a last couple days this has been. I just wish the person that is attacking me would show themselves and tell me WTF they want. But at this point I'm more concerned about HOW they are getting in, because they keep doing it very quickly when I get a fresh server up.
I fail to see how any of this makes sense, if OVH were this poor at default server setup no one would buy from them. Just how do you get a rescue mode on a hacked remote server with no access without help from the data centre where it is located and why the hell would they not be interested in their servers getting hacked in an instant upon the setup of them. So you need to come to some forum to get help with this.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
No idea. Not looking for any arguments just want to get some pointers on how I can further analyze this. This is very targeted and is a sophisticated attack otherwise a lot of their servers would be getting hacked too. The hacker is clearly aware of some kind of unpatched exploit in a default service such as Apache.
No idea. Not looking for any arguments just want to get some pointers on how I can further analyze this. This is very targeted and is a sophisticated attack otherwise a lot of their servers would be getting hacked too. The hacker is clearly aware of some kind of unpatched exploit in a default service such as Apache.
Well then it is time to talk to the people at the Apache Foundation and OVH, you claim to have a 100% record of getting hacked with both of their products used in combination. I would think they will want to investigate this matter to solve the problem. Supposedly you have the perfect setup to catch this in the act. Not asking about it on some forum where you will get no where near the level of analysis they can do on this. That is if you actually want to get the problem solved and not just do further postings here.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
Well in order to do a proper bug report I'm going to want to have concrete evidence and a way to reproduce it otherwise how are they suppose to fix the bug? I do not know 100% yet what the attack vector is and only assuming, this is why I'm hoping someone that knows more than me about Linux/security can help.
The host also won't refund me as I prepaid for 3 months on the new box, so that one will remain available for post mortem. I decided to poke around more on that one and found this in the /var/log/auth.log:
Code:
Nov 27 17:52:20 server04 sshd[40160]: Failed password for root from 185.251.45.83 port 59679 ssh2
Nov 27 17:52:22 server04 sshd[40160]: Connection closed by authenticating user root 185.251.45.83 port 59679 [preauth]
Nov 27 17:52:25 server04 sshd[40162]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.251.45.83 user=root
Nov 27 17:52:27 server04 sshd[40162]: Failed password for root from 185.251.45.83 port 32933 ssh2
Nov 27 17:52:29 server04 sshd[40162]: Connection closed by authenticating user root 185.251.45.83 port 32933 [preauth]
Nov 27 17:52:32 server04 sshd[40164]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.251.45.83 user=root
Nov 27 17:52:34 server04 sshd[40164]: Failed password for root from 185.251.45.83 port 34284 ssh2
Nov 27 17:52:36 server04 sshd[40164]: Connection closed by authenticating user root 185.251.45.83 port 34284 [preauth]
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Nov 27 17:56:12 server04 systemd-logind[882]: New seat seat0.
Nov 27 17:56:12 server04 systemd-logind[882]: Watching system buttons on /dev/input/event3 (Power Button)
Nov 27 17:56:12 server04 systemd-logind[882]: Watching system buttons on /dev/input/event2 (Sleep Button)
Nov 27 17:56:12 server04 systemd-logind[882]: Watching system buttons on /dev/input/event4 (Intel Virtual Button driver)
Nov 27 17:56:12 server04 sshd[940]: Server listening on [server ip] port 22.
Nov 27 17:56:35 server04 sshd[1874]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[my home ip] user=debian
Nov 27 17:56:37 server04 sshd[1874]: Failed password for debian from [my home ip] port 14917 ssh2
Nov 27 17:56:39 server04 sshd[1874]: Connection closed by authenticating user debian [my home ip] port 14917 [preauth]
Someone was trying to brute force their way in, which on it's own would not be an issue once fail2ban is setup, and root logins are not permitted on that server anyway. However, he did something at some point which generated all those ^@ and at that point I lost my connection to the server, and was no longer able to log back in. I would hit the SSH server but it would reject my password. So whatever the attack was it also reset the password and deleted the SSH keys. Not sure what else or how long they were in the box or what data they may have taken. (there was not lot of data yet as I was still restoring backups)
This is on the new server so I believe they may have attacked it through a different attack vector, I was just assuming Apache but maybe it's SSH. But it's weird since lot of other logs have these symbols too so wonder if the attack requires attacking more than one service at once, then they must issue some characters that erase the logs and replace with null characters.
Either way this is all speculation this is why I'm asking for some insight on more specific things to check.
Last edited by Red Squirrel; 11-28-2021 at 11:05 AM.
MacUser2525:~$ whois 185.251.45.83
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object
refer: whois.ripe.net
inetnum: 185.0.0.0 - 185.255.255.255
organisation: RIPE NCC
status: ALLOCATED
whois: whois.ripe.net
changed: 2011-02
source: IANA
# whois.ripe.net
inetnum: 185.251.44.0 - 185.251.45.255
netname: SRVC-1
org: ORG-KGL11-RIPE
country: HK
admin-c: PD358
tech-c: PD358
status: ASSIGNED PA
mnt-by: PawelD-MNT
mnt-by: mnt-us-sammu-1
created: 2018-12-03T19:46:57Z
last-modified: 2020-02-22T09:33:26Z
source: RIPE # Filtered
organisation: ORG-KGL11-RIPE
org-name: KWAIFONG GROUP LIMITED
org-type: OTHER
address: Rm 34, 4/F, Beverley Commercial Centre 87-105 Chatham Road, Tsim Sha Tsui, Kowloon,Hong Kong
abuse-c: KGLA1-RIPE
mnt-ref: PawelD-mnt
mnt-by: PawelD-mnt
created: 2019-01-02T19:31:28Z
last-modified: 2019-01-02T19:31:28Z
source: RIPE # Filtered
person: Pawel Damian
address: Poznan, Poland
phone: +48616419200
nic-hdl: PD358
mnt-by: PawelD-MNT
created: 2006-06-07T10:34:33Z
last-modified: 2019-08-09T13:04:00Z
source: RIPE
% Information related to '185.251.44.0/23AS133115'
route: 185.251.44.0/23
descr: ServiceID-ISER
origin: AS133115
mnt-by: PawelD-mnt
created: 2019-03-27T08:54:40Z
last-modified: 2019-03-27T08:54:40Z
source: RIPE
% This query was served by the RIPE Database Query Service version 1.101 (HEREFORD)
There is no need for bug report you say it happens as soon as the server is setup. This implies OVH are complete and totally incompetent in provisioning a secure by default server instance. Something I find completely hard to believe, they would not still be in business if this was the case. For them not to want to investigate this happening at all is another completely hard to believe scenario. At the very least you have case for a charge back on whatever the means of payments was. According to you they have delivered nothing of value for your purchase.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
This is a highly targetted attack so maybe this is not a known flaw yet and it does not have an update. I don't think this is OVH's fault. I suppose I can just move on from it but I feel like I am in a position where I could discover a new flaw and report it to the appropriate developer. This may be something similar to heart bleed or shell shocker. Those flaws went unnoticed for years before someone finally found them.
On the original server it may very well be shell shocker I got hit with as I'm not sure if the patching level went that far, it was really old (CentOS 6.6) and well overdue to be upgraded. But the new one is what baffles me.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
There's a log I realized I forgot to look at this whole time. This is a client application and not a server application, but it stopped polling when the server was hit with the hack. (this is on the original server) I was curious to see what kind of errors there would be, I assumed it would just be failing to connect to servers, but no, something very different....
Really messed up. I was thinking MAYBE someone managed to find a flaw in my program by acting as a bad client (the nature of my app is for game servers so you can submit a game server to it and the app connects to gather data) and sending bad data to it, but that still would not explain how they hacked the 2nd server (the app was not running yet).
I'm thinking this is more a symptom of the attack, it may have scrambled the memory or something in the process as maybe part of the attack involved loading code into memory by specifying an arbitrary start memory address then executing it. I don't really know that much about hacking at this level so it's a bit beyond my knowledge. This is not just a simple brute force attack it was actual real hacking. There were a lot of brute force attempts in the logs but think that was just noise and "normal" stuff, as I had not changed the default port yet. None of the attempts were trying the proper username either and root login was disabled.
Last edited by Red Squirrel; 11-29-2021 at 09:00 PM.
Not sure the origin of the attack is still debated and if we are actually a source of concern, but just to square up things :
Babbar.tech is not a pen testing site, we just (mainly) operate a web crawler to index the web graph. Our crawler is barkrowler, it shouldn't trigger any such problem on a server.
Either the log line is completely unrelated to your problem, or it's someone faking our user-agent, or, but that seems unlikely, it's a bug in our crawler triggered by some condition that has never occurred elsewhere (we crawl approximately a billion pages each day so I'm quite doubtful).
Anyway, if you think we may help in any way, do not hesitate to contact me.
Guillaume
Quote:
Originally Posted by frankbell
A web search turns up the information that babbar.tech purports be a site for pen testing.
Was any damage done?
What security precautions have you taken (firewall, fail2ban, etc.)?
Hi, I'm the CTO of babbar.tech
Babbar.tech
Guillaume
The site needs work Guillaume. Land the homepage and I click "US" in upper right, yet the bottom 1/4 page notice "Paramètres du cookies" still renders in French?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
