# [SOLVED]SSH attack ongoing - any advice?

## Havin_it

Hi,

I've been under a concerted SSH attack for a few days now. It's slow but methodical and appears to be the work of a botnet: every few minutes a new login attempt, each one from a different IP and using sequential dictionary names for usernames. I'm blocking each IP in iptables using Fail2Ban, and the blocklist is now in the thousands.

I shut the SSH port in my gateway overnight (about 12 hours) hoping the attacker would give up, but the attempts started again as soon as I reopened the port (which I have to do, as I need access while at the office). I'm (supposed to be) on a dynamic IP so I reconnected several times, but just got the same external IP every time. The trained chimp at my ISP can't explain this.

I'm unable to shut the door for more than a few hours at a time, but I really want to shake this bugger off as I'm getting fed up of the logs piling up. Is there anything else I can really do?

Thanks for any advice.Last edited by Havin_it on Thu Jul 01, 2010 8:15 am; edited 1 time in total

----------

## Jaglover

You can use http://denyhosts.sourceforge.net/ if it gives you peace of mind. Every SSH server on this planet is under attack.

----------

## chithanh

If you use fail2ban you have to be careful not to DoS yourself.

You can configure sshd to disable password logins and allow only key-based logins from non-local networks.

----------

## Anon-E-moose

I have limited my ssh port to certain subnets before (iptables rule) you could do that and just not log any ssh hits on the firewall.

----------

## Havin_it

Hi Jaglover =)

Yes I used to use denyhosts, but replaced it with Fail2Ban because (a) iptables blocking is more efficient than hosts.deny, I think and (b) it covers Apache and other servers as well.

Indeed I'm quite used to the usual handful of opportunistic attempts, but this is a massive spike, and it's clearly the work of a single persistent attacker. If they keep it up much longer I might start to get a teensy bit worried about whether my safeguards are enough...

----------

## Jaglover

 *chithanh wrote:*   

> You can configure sshd to disable password logins and allow only key-based logins from non-local networks.

 

++

----------

## ziggysquatch

If it's feasible, you can run ssh on any port you want.  Most of the time when you are logging attacks, they are just dumb bots that are looking specifically at port 22.

I used to log attacks all the time and then I changed sshd to run on a high port number and I haven't seen an attempt in the logs in 3 years.

The other solutions mentioned above are better but if you don't have to have it on port 22 then this is very easy.

----------

## dE_logics

Call the cops... you can take legal action against this chap.

How about attacking HIM instead?

----------

## M

 *ziggysquatch wrote:*   

> If it's feasible, you can run ssh on any port you want.  Most of the time when you are logging attacks, they are just dumb bots that are looking specifically at port 22.
> 
> I used to log attacks all the time and then I changed sshd to run on a high port number and I haven't seen an attempt in the logs in 3 years.
> 
> The other solutions mentioned above are better but if you don't have to have it on port 22 then this is very easy.

 

++

Very effective and simple, although I also use key only auth and I use fail2ban because of other services I use (imap, smtp etc.)

----------

## mokia

 *Havin_it wrote:*   

> I'm (supposed to be) on a dynamic IP so I reconnected several times, but just got the same external IP every time. The trained chimp at my ISP can't explain this.
> 
> 

 

Change your MAC adress in /etc/conf.d/net

mac_eth0="00:11:22:33:44:55"

you will get a new IP

 *ziggysquatch wrote:*   

> If it's feasible, you can run ssh on any port you want.
> 
> .

 

+3Last edited by mokia on Wed Jun 23, 2010 1:44 pm; edited 1 time in total

----------

## krinn

 *dE_logics wrote:*   

> 
> 
> How about attacking HIM instead?

 

Doing illegal thing against an illegal attacker won't make it legal in many (all?) countries...

You can switch your ssh port to use another port, or blacklist all except your office IP

echo "SSHD: ALL except officeIP" >> /etc/hosts.deny

----------

## John R. Graham

I program fail2ban to send polite emails to the attacker's ISP.  I get very few replies from humans but think it reduces the volume of attacks.  I also only allow key-based logins from outside my domain.  Doesn't seem to stop the break-in attempts, though.

- John

----------

## Havin_it

Wow - thanks for so many replies (even while I was replying to the first  :Wink:  )

RE changing ports: I did try this once - assigned an arbitrary high port - but on some of the other public access-points I use I found that port was blocked. Any recommendations on how to avoid such a problem? HTTP and HTTPS ports are already in use.

RE changing MAC address: my router doesn't support that, unfortunately.

RE key-only access: this is probably a good call; I could keep the key on a small USB stick, as I wouldn't want to leave it on an office computer. I'll have a play around with this idea.

----------

## krinn

139 the microsoft one, 25 smtp, 110 pop, 143 imap, 465 smtps, 993 imaps, 995 popssl

pick one you don't use, all are bellow 1024, are standard ports, so your isp shoudn't bug you with them.

----------

## mokia

But if you are behind a router, you acces the internal ssh server width portforward. 

You must only change the forward entry, not the server port.

----------

## Ant P.

 *Havin_it wrote:*   

> RE changing ports: I did try this once - assigned an arbitrary high port - but on some of the other public access-points I use I found that port was blocked.

 

You can have it listen on multiple ports at once. Try using ports for other common stuff like 21, 81, 8080, 5190, 6667 etc.

----------

## ziggysquatch

 *krinn wrote:*   

> 139 the microsoft one, 25 smtp, 110 pop, 143 imap, 465 smtps, 993 imaps, 995 popssl
> 
> pick one you don't use, all are bellow 1024, are standard ports, so your isp shoudn't bug you with them.

 

The above should work for you as far as public networks go.  Otherwise, you may want port knocking http://en.wikipedia.org/wiki/Port_knocking.  I read a really good article in Linux Journal but the one that came up in google is from 2003.  The one I read was from 2009 sometime.

Port knocking let's you set up a combination of port "knocks" that will open access to the desired port only after the combination of "knocks" is made.

----------

## eccerr0r

I've pretty much given up on it.  Just have to be sure you have good passwords or use key-based authentication.  I have the same problem, I can only use port 22 for ssh because the remote site I connect from blocks everything else.

I'm not sure what this botnet of linux/unix machines uses to choose particular machines to attack.  I've found that my linux routers and machines that are usually turned off tend to not get touched for some reason or another - somehow they notice the boxes are not 24/7 or "complete" (as in no disk space like routers and set-top?) boxes, and remove them from the list of candidates...

And yes, these bots appear to be *nix machines, so people who don't monitor their boxes when they get 0wn3d should be ashamed!

----------

## Cyker

I had this a while back; Tried a lot of stuff but gave up and just moved it to a non-standard port.

----------

## ArmorSuit

The solution to this one is so simple. It is pure artistic poetry! A haiku:

Go away from 22.

Your users should be using public key auth.

Don't waste valuable resources on fail2bans, firewall logs or whatnot. The above stanza works perfectly.

----------

## Cyker

I prefer limericks meself...

If you use that port twenty two

Then bot nets will try to hack you

Which is why we say

Just take it away

And use something like port 222  :Mr. Green: 

----------

## kimmie

Some super style this is really showing!

I see this growing to a new sub forum

Where decorum and discipline versed in wit

Strains out the shit

A bit

----------

## Tuna

Have a read here. There are no practical examples I guess but I think that might get you started.

http://en.wikipedia.org/wiki/Port_knocking

----------

## Havin_it

More great contributions from the practical to the lyrical  :Very Happy: 

The attacker quit (after user 'zzz' had tried its luck) a couple of days ago and I got a fw hours' respite, but soon another began. So I've gone ahead and moved to a higher port 222 (now I think about it, maybe 'zzz' inspired that choice). No blocking so far, and not a single attempt. It's les of a nuisance than I thought, as I can put the port number in my clients' ssh_config files just for the one host and not have to invoke it on the commandline or in any of my scripts, which is nice  :Smile: 

As for port-knocking, seems a bit of a faffy solution - and unfortunately neither my router or PuTTy on my phone support it  :Sad: 

Thanks to all.

----------

## kimmie

Just for completeness, another option I've seen mentioned is blocking by country. I don't think it's all that easy (as in emerge something, off you go) but here's a couple of places to start if you're interested: http://www.theillien.com/Sys_Admin_v12/html/v14/i11/a3.htm http://www.debian-administration.org/articles/518

I went through the same thing you did, I guess everybody who starts an ssh server does. I ended up using the VPN in my modem instead for quite a while, although I suspect that was actually less secure than SSH with a public key. I turned on all the logging and didn't see anything, but that doesn't prove much. Then the need went away, but it's coming back soon... 

It's funny, I haven't seen this mentioned before: I know moving off port 22 works really well. But it really shits me, because it means every time I type ssh -p 8421 mycorner.oftheworld.me.org I'm reminded of all the twats and assholes in the world who want to fuck with me. And that's not something I want to think, I'd rather think about how convenient and nice it is I can run a remote X session through an ssh tunnel  and how much bandwidth we've got now etc. etc. I know it's stupid, but it's what my brain does.

Hmm... just had a thought. Of course,  it needs a little more wrapping, but why not

```
alias ssh='\ssh -p 8421'
```

Nothing like a nice wrapper rug to shove the dirt under!

----------

## gerdesj

Have you considered OpenVPN?

Its not too hard to setup and the Gentoo configuration is very good allowing you to create multiple instance of OVPN as both clients and servers.  The docs on the upstream website are excellent and I'll bet there are Gentoo HOWTOs all over the shop.

I look after/manage around 150 separate OVPNs running between various systems and it is seriously reliable.  It runs on Linux, Windows, *BSD (eg pfSense).  

Then block port 22 on your external interface - hack that you buggers.

Another approach I have used is (excerpt from firewall bits embedded in /etc/conf.d/net):      

```
 

einfo "${FW4} SSH chain"

iptables -N SSH

iptables -F SSH

iptables -A SSH -m state --state NEW -m recent --set

iptables -A SSH -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

iptables -A SSH -j LOG --log-prefix "${FW4} SSH ALLOWED "

iptables -A SSH -j ACCEPT

iptables -A SSH -j DROP

```

Note the -m recent lines.

This is not perfect because I probably need to extend the time period it looks at.  I notice that the bot nets are co-ordinated.  One host at a time will try a few names and then bug out for a day or so and let another one have a go.

Just use a VPN - simples!

Cheers

Jon

----------

## Dr.Willy

 *kimmie wrote:*   

> ...
> 
> Hmm... just had a thought. Of course,  it needs a little more wrapping, but why not
> 
> ```
> ...

 

man ssh_config

----------

## kimmie

 *Dr.Willy wrote:*   

> man ssh_config

 

Why did I never think of that?? Thanks!!!

----------

## Havin_it

 *gerdesj wrote:*   

> Have you considered OpenVPN?
> 
> Its not too hard to setup and the Gentoo configuration is very good allowing you to create multiple instance of OVPN as both clients and servers.  The docs on the upstream website are excellent and I'll bet there are Gentoo HOWTOs all over the shop.
> 
> I look after/manage around 150 separate OVPNs running between various systems and it is seriously reliable.  It runs on Linux, Windows, *BSD (eg pfSense).  
> ...

 

Hi Jon,

I agree about OpenVPN - I've brought it in as VPN solution for my workplace, where the server is Windows and the clients are mixed OS, and it's far better than any of the Windows built-in solutions. However this'd be no good for my Nokia phone, as there's no OVPN client for Symbian -- in fact there's *no* VPN client, as the Nokia IPSec one won't work with dynamic IP. I'd certainly love it if such a thing existed, but PuTTY is all I have at present.

However I'm very impressed by your iptables-fu. Can you tell us a bit more about what that listing means, how it works, and would similar tactics work with other servers such as Apache, Postfix (SMTP) or Dovecot (IMAP)? I'd love to see a pure iptables solution handling the tasks Fail2Ban is currently covering, but it's so daunting to configure on its own that I've no idea.

----------

## gerdesj

 *Havin_it wrote:*   

>  *gerdesj wrote:*   Have you considered OpenVPN?
> 
> Its not too hard to setup and the Gentoo configuration is very good allowing you to create multiple instance of OVPN as both clients and servers.  The docs on the upstream website are excellent and I'll bet there are Gentoo HOWTOs all over the shop.
> 
> I look after/manage around 150 separate OVPNs running between various systems and it is seriously reliable.  It runs on Linux, Windows, *BSD (eg pfSense).  
> ...

 

Err, I have to confess I copied it somewhat from elsewhere and rereading, its a bit specific but here goes:

The excerpt is adding a chain that I have called SSH and flushes it (the script is designed to clear everything down that might be configured already and I have gone a bit over the top to try and mitigate errors whilst messing around by always overdoing the flushing etc!)

The SSH chain is branched to by any connection to port 22.

Then the important bit (this is from memory - get the search engine out).  First add any NEW connections (--set). Then the next rule will fire if four or more connections within 60 seconds arrive from the same IP and drop them.

On reflection it probably needs a few changes.  The set of addresses for this rule probably needs a name to differentiate it from other sets and also a much longer period to look at.  However I have to balance that with my fumbled attempts to type my own password. 

The possible good solution to my mind would seem to be setting up RSA keys  - ie passwordless and a long  --update period with a low --hitcount.  That way you avoid your own mistakes and a long lock out but keep the baddies away.  

Based on some firewall logs I've seen, keeping around 7 days or more for the set might be needed. 

Thinking about it I have a customer that was hit by around 14 million separate IPs per month bashing against their mail server (I have Exim passing logs to rsyslog to MySQL to see what the hell was going on - its quite a big DB now!)  I think they would make a good test case for seeing just how many addresses the --set thing can realistically deal with.  OK we are looking at ssh here but smtp gets a hammering as well.  Must get around to having another look to see how things are going.

I'm surprised that the Simbian IPSEC client does not work.  Do you mean that the server is behind a dynamic address or the phone? If you only mean the phone then you need to look into IPSEC road warrior configs.  That's a standard IPSEC setup and I'd be *very* surprised if it wont work.  Get IPSEC working and that is your real solution.  Unless someone ports OVPN to Simbian ...

Cheers

Jon

----------

## Havin_it

 *gerdesj wrote:*   

> 
> 
> I'm surprised that the Simbian IPSEC client does not work.  Do you mean that the server is behind a dynamic address or the phone? If you only mean the phone then you need to look into IPSEC road warrior configs.  That's a standard IPSEC setup and I'd be *very* surprised if it wont work.  Get IPSEC working and that is your real solution.  Unless someone ports OVPN to Simbian ...
> 
> Cheers
> ...

 

Unfortunately, both  :Sad:  The home network my server is on has actually picked up a new external IP since I wrote the first post, so it hasn't "gone static" after all. (I suspect what happened was my previous reconnects were during the day, so no one else connected to my ISP and took that IP while I was disconnected, so I just got it back.) I've yet to try doing it with the current server IP configured in the client, just to prove it works, but that would be of limited value anyway I guess.

Symbian is open-source now, so it's not impossible that someone might port OVPN (as long as the platform just doesn't die soon), or maybe I'll get an Android or Meego phone next time, which should be an easier port.

----------

## gerdesj

 *Havin_it wrote:*   

>  *gerdesj wrote:*   
> 
> I'm surprised that the Simbian IPSEC client does not work.  Do you mean that the server is behind a dynamic address or the phone? If you only mean the phone then you need to look into IPSEC road warrior configs.  That's a standard IPSEC setup and I'd be *very* surprised if it wont work.  Get IPSEC working and that is your real solution.  Unless someone ports OVPN to Simbian ...
> 
> Cheers
> ...

 

You should be able to do IPSEC using DNS names.  Try using a dynamic DNS service to get your self an A record that changes with your ISP assigned IP address and then use that for the phone to connect to.  In theory that is all that is needed.  You may get a connection blip at hand over time but you can run a script or one of the daemons available on your IPSEC "server" which can test for its own external IP address and then update the dynamic DNS server.

On an Android I suspect you can use the full OpenSWAN or StrongSWAN thing ...

Cheers

Jon

----------

## Havin_it

I did try to setup the Nokia VPN using a dyndns hostname, but after much frustration I gave up. I'd just get a "Failed to activate access point, Reason code -5257" (no idea what that means, can't find it) on the phone, and nothing that leaps out as an obvious point of failure in the openswan debug output. Even connecting from within the LAN failed, but for different reasons (xauth password not being accepted for some reason), so I quit.

----------

## coolsnowmen

Noticed this was a recent thread-

   Recently: I had to set up ssh on very public ips, on port 22, w/o port knocking, and with user/password acceptance.

Because I am not that well versed at iptables, I came up with a solution that works pretty well for me.  Looking at the logs, I analyzed the annoying break in attempts and found that 99.9% of them came from china, indea, korea, and japan.  After checking with my boss (we are US based), I simply made a hosts.deny file with ALL of their ip/subnet ranges.  This is my good enough solution, and requires only for me to copy this file when we add a new computer to the network.

PM me with an email if you'ld like a copy of it.

PS it was easy for me to find the ip/cidr of the countries but I had to write a program to convert the ip/n format to ip/aaa.bbb.ccc.ddd format, and then cat'ed them all together.

----------

## Havin_it

Hi coolsnowmen  :Very Happy: 

I've heard of this approach before, and I've always been unhappy with the idea - it's like collective punishment (even though it's hardly a "punishment" being denied access to my home server).

However, it sounds like something that might be of interest as an addition to Fail2Ban: instead of (or on top of) banning single IPs, which as discussed above is not always the answer for a botnet attack, the same program could look for whole netblocks sending a lot of attack requests and temporarily ban them.

I'd be interested to see the script you used to convert the IP/subnets - would you be prepared to post that here?

PS - I'm still not safe on port 222, I have even had about a dozen bans since I switched! Is the attacker reading this thread?   :Shocked: 

----------

## eccerr0r

It looks like port 222 is used for rsh, anotther frequently used login mechanism hackers like to exploit.  However the protocol for rsh and ssh are different so I'm not sure how successful the attacks would be.  Try another port. 

Be glad you have that option to change ports...

(though right now perhaps I can do the same now... now that I have an alternate method of connecting back home: cell phone internet!)

----------

## dmpogo

 *coolsnowmen wrote:*   

> Noticed this was a recent thread-
> 
>    Recently: I had to set up ssh on very public ips, on port 22, w/o port knocking, and with user/password acceptance.
> 
> Because I am not that well versed at iptables, I came up with a solution that works pretty well for me.  Looking at the logs, I analyzed the annoying break in attempts and found that 99.9% of them came from china, indea, korea, and japan.  After checking with my boss (we are US based), I simply made a hosts.deny file with ALL of their ip/subnet ranges.  This is my good enough solution, and requires only for me to copy this file when we add a new computer to the network.
> ...

 

I hope you don't plan to do business with these countries or expect customers from there.

----------

## Havin_it

 *dmpogo wrote:*   

>  *coolsnowmen wrote:*   Noticed this was a recent thread-
> 
>    Recently: I had to set up ssh on very public ips, on port 22, w/o port knocking, and with user/password acceptance.
> 
> Because I am not that well versed at iptables, I came up with a solution that works pretty well for me.  Looking at the logs, I analyzed the annoying break in attempts and found that 99.9% of them came from china, indea, korea, and japan.  After checking with my boss (we are US based), I simply made a hosts.deny file with ALL of their ip/subnet ranges.  This is my good enough solution, and requires only for me to copy this file when we add a new computer to the network.
> ...

 

I assume he meant he was just blocking them from port 22, not the whole server.

----------

## coolsnowmen

 *Havin_it wrote:*   

> Hi coolsnowmen 
> 
> ...
> 
> However, it sounds like something that might be of interest as an addition to Fail2Ban: instead of (or on top of) banning single IPs, which as discussed above is not always the answer for a botnet attack, the same program could look for whole netblocks sending a lot of attack requests and temporarily ban them.
> ...

 

So, its not so much a script as a program, I wrote it myself, so be nice if there is something you don't like.  Released Free to all Gentooers

(compile simply with g++)

```

//Filename: cidr2subnet.cpp

#include <fstream>

#include <iostream>

#include <cstring>

using namespace std;

char ERR_S[32]="Failure";

int bits2mask(const int &bits)

{

   static int val,counter;

   static int mask=128;  //0x80

   val=0;

   for (counter=0; counter<bits; counter++)

   {

      val>>=1;

      val|=mask; 

   }

   return val;

}

void cidr2mask(const char * cidr_s,char * mask_s)

{

  static int one, two, three, four, cidr, masks[4];

  if (sscanf(cidr_s,"%i.%i.%i.%i/%i",&one,&two,&three,&four,&cidr)<5)

  { strcpy(mask_s,ERR_S);  return; //ERROR

  }

  

  for( int idx=0; idx<cidr/8; idx++)

    masks[idx]=255;

  for( int idx=cidr/8 + 1; idx<4; idx++)

    masks[idx]=0;

  masks[cidr/8]=bits2mask(cidr % 8);

  //printf("\n Parsed to: %i.%i.%i.%i/%i", one, two, three, four, cidr);

  sprintf(mask_s,"%i.%i.%i.%i/%i.%i.%i.%i", one, two, three, four, masks[0], masks[1], masks[2], masks[3]);

}

int main (int argc, char ** argv)

{

  if (argc<2)

    cout << "\n No Argument given to cidr2subnet"

         << "\n Written my jon malachowski"

         << "\n   converts ip4/cidr to ip4/sub.net.mask.0\n"

         << "\n    ex:  " << argv[0] << " 1.2.3.4/24"

         << "\n should return: 1.2.3.4/255.255.255.0\n"

         << "\n also in bash try: "

         << "\n for a in `cat testfile.cidr`"

         << "\n   do ./a.out $a;"

         << "\n done" << endl;

  else {

    char mask_s[32];

    for (int arg_idx=1; arg_idx < argc; arg_idx++)

    { cidr2mask(argv[arg_idx],mask_s);

      cout << mask_s << endl;

    }

  }

return 0;

}

```

----------

## coolsnowmen

 *Havin_it wrote:*   

>  *dmpogo wrote:*    *coolsnowmen wrote:*   Noticed this was a recent thread-
> 
>    Recently: I had to set up ssh on very public ips, on port 22, w/o port knocking, and with user/password acceptance.
> 
> Because I am not that well versed at iptables, I came up with a solution that works pretty well for me.  Looking at the logs, I analyzed the annoying break in attempts and found that 99.9% of them came from china, indea, korea, and japan.  After checking with my boss (we are US based), I simply made a hosts.deny file with ALL of their ip/subnet ranges.  This is my good enough solution, and requires only for me to copy this file when we add a new computer to the network.
> ...

 

no and no.  putting ips in hosts.deny does in fact block everything from that ip (ssh/http/cifs/*).  If the incoming ip is in range, the host is directed to drop the packet no matter what it contains, with no response (effectively invisible).  As I said before, I did ask my boss at the time and told him the implications.  I use this on my home server because only me and a few friends need to even see it.  It is a quick and dirty way of reducing your vulnerability in general based on a statistical analysis of my log files.  If you are being paid by a business that need an international web presence in those countries, perhaps this method is not for you.  For a user who simply wants ssh/http connectivity to his home computer, this is the fasted method if you find setting up fail2ban (and other things like it) confusing/consuming.

----------

## Havin_it

Entries in hosts.deny don't have to be a blanket ban though: when I used denyhosts before Fail2ban, it would respond to SSH hammering by adding this type of line:

```
sshd : x.x.x.x
```

So only SSH connections were banned - the banned IP could still access Apache on that server, for example.

Also, isn't it the case that only services with support for tcpd refer to this file? Do you have another component in your setup?

----------

## xibo

ker - be - ros

i can't believe i'm really the first one to name it here.

if you know your clients you can also whitelist port 22 for them and block everything else from it via firewall. using a different port might drive off lesser worms but not a human attacker or smart trojan, for later will have nmap scan for what service is running where.

also, emerging threats maintains a daily updated list of malicious hosts/nets known for botnetting which wouldn't hurt your security when being introduced to your firewall via cron job.

----------

## coolsnowmen

 *xibo wrote:*   

> 
> 
> if you know your clients you can also whitelist port 22 for them and block everything else from it via firewall.

 

That is done by whitelisting their ip right? Well that was always a non starter for me.  Because they they could never just pick up a computer and ssh in to do something.  It also became more of a pain when adding people to a system.  Then when removing people you have to know which ip went with which person, on every firewall.

IMO, if I ever went that far, it would simply be better to do to .ssh/id_dsa files.  That way at least the user gains something for their trouble.

----------

