Monday, July 4, 2011

[HOW TO AVOID] kernel: possible SYN flooding on port 80. Sending cookies.

Sometime back, one of the servers I work on, frequently faced an outage of web services. On investigating, I found that it had something creeping up in it's logs. Something which read -

kernel: possible SYN flooding on port 80. Sending cookies.

It looked like a Denial of service attack. It was evident that I needed to beef up security!

Avoiding a DDOS attack on a web server
iptables comes with a module (limit) using which a DDOS attack can be tackled. Depending on the type of web service running on the server, I decided a limit of 15 HTTP syn packets per second would be enough.

First off, I had a look at the existing rules
# iptables -L -v

This shows you the rules and the default policy that are set in the existing chains - INPUT, FORWARD and OUTPUT.

Then I followed these quick steps -

1. Create a new chain and name it, say, DDOS_SYNFLOOD,
# iptables -N DDOS_SYNFLOOD

2. Add a limit to no.of packets 15 per second with a max burst of about 20, by using the limit module -
# iptables -A DDOS_SYNFLOOD -m limit --limit 15/second --limit-burst 20 -j ACCEPT

Note: Other units - /minute ,  /hour , and /day

3. And of course, we will need to drop packets which exceed the above limitation
# iptables -A DDOS_SYNFLOOD -j DROP

4. Now all that was left was to "jump" to this new chain for incoming tcp syn packets on port 80.
# iptables -A INPUT -p tcp --syn --dport http -j DDOS_SYNFLOOD

And to look at what was set up -

# iptables -L -v
Chain INPUT (policy ACCEPT 95 packets, 4988 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DDOS_SYNFLOOD  tcp  --  any    any     anywhere             anywhere            tcp dpt:http flags:FIN,SYN,RST,ACK/SYN

......
......
......
......

Chain DDOS_SYNFLOOD (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere            limit: avg 15/sec burst 20
    0     0 DROP       all  --  any    any     anywhere             anywhere


And since then, I have had a few peaceful nights.

We should remember, iptables works sequentially and jumps to the target of the first match. Hence, you will need to ensure that there are no conflicting rules ahead of this one to avoid an undesired result.


More info on limit module:

   limit
       This  module  matches at a limited rate using a token bucket filter.  A
       rule using this extension  will  match  until  this  limit  is  reached
       (unless  the `!' flag is used).  It can be used in combination with the
       LOG target to give limited logging, for example.

       [!] --limit rate[/second|/minute|/hour|/day]
              Maximum average matching rate: specified as a  number,  with  an
              optional  `/second',  `/minute',  `/hour', or `/day' suffix; the
              default is 3/hour.

       --limit-burst number
              Maximum initial number of packets to  match:  this  number  gets
              recharged  by  one  every  time the limit specified above is not
              reached, up to this number; the default is 5.


More info on iptables in general:
    http://wiki.centos.org/HowTos/Network/IPTables

Sunday, December 5, 2010

runlevel links at boot time with chkconfig

Generally after every server setup, we need to place a few services on startup.  Depending on the operating system, I have tried quite a few techniques right from manually placing scripts in init.d to the ease of update-rc.d(on ubuntu). However, today I would like to showcase another tool - chkconfig, just for the fact that it has some real clean representation,

A chkconfig --list would show you the list of services and their run level configurations.

Sample output on a CentOS installation:
# chkconfig --list
NetworkManager  0:off   1:off   2:off   3:off   4:off   5:off   6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
anacron         0:off   1:off   2:on    3:on    4:on    5:on    6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off
avahi-dnsconfd  0:off   1:off   2:off   3:off   4:off   5:off   6:off
bluetooth       0:off   1:off   2:on    3:on    4:on    5:on    6:off
capi            0:off   1:off   2:off   3:off   4:off   5:off   6:off
conman          0:off   1:off   2:off   3:off   4:off   5:off   6:off
cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
cups            0:off   1:off   2:on    3:on    4:on    5:on    6:off
dnsmasq         0:off   1:off   2:off   3:off   4:off   5:off   6:off
dund            0:off   1:off   2:off   3:off   4:off   5:off   6:off
firstboot       0:off   1:off   2:off   3:on    4:off   5:on    6:off
gpm             0:off   1:off   2:on    3:on    4:on    5:on    6:off
haldaemon       0:off   1:off   2:off   3:on    4:on    5:on    6:off
hidd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
ip6tables       0:off   1:off   2:on    3:on    4:on    5:on    6:off
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
irda            0:off   1:off   2:off   3:off   4:off   5:off   6:off
irqbalance      0:off   1:off   2:on    3:on    4:on    5:on    6:off
isdn            0:off   1:off   2:on    3:on    4:on    5:on    6:off
kudzu           0:off   1:off   2:off   3:on    4:on    5:on    6:off
lvm2-monitor    0:off   1:on    2:on    3:on    4:on    5:on    6:off
mcstrans        0:off   1:off   2:on    3:on    4:on    5:on    6:off
mdmonitor       0:off   1:off   2:on    3:on    4:on    5:on    6:off
mdmpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
messagebus      0:off   1:off   2:off   3:on    4:on    5:on    6:off
microcode_ctl   0:off   1:off   2:on    3:on    4:on    5:on    6:off
multipathd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
netplugd        0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
nfs             0:off   1:off   2:off   3:off   4:off   5:off   6:off
nfslock         0:off   1:off   2:off   3:on    4:on    5:on    6:off
nscd            0:off   1:off   2:off   3:off   4:off   5:off   6:off
ntpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
oddjobd         0:off   1:off   2:off   3:off   4:off   5:off   6:off
pand            0:off   1:off   2:off   3:off   4:off   5:off   6:off
pcscd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
portmap         0:off   1:off   2:off   3:on    4:on    5:on    6:off
psacct          0:off   1:off   2:off   3:off   4:off   5:off   6:off
rawdevices      0:off   1:off   2:off   3:on    4:on    5:on    6:off
rdisc           0:off   1:off   2:off   3:off   4:off   5:off   6:off
readahead_early 0:off   1:off   2:on    3:on    4:on    5:on    6:off
readahead_later 0:off   1:off   2:off   3:off   4:off   5:on    6:off
restorecond     0:off   1:off   2:on    3:on    4:on    5:on    6:off
rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off
rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off
rpcsvcgssd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
saslauthd       0:off   1:off   2:off   3:off   4:off   5:off   6:off
sendmail        0:off   1:off   2:on    3:on    4:on    5:on    6:off
smartd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
syslog          0:off   1:off   2:on    3:on    4:on    5:on    6:off
tcsd            0:off   1:off   2:off   3:off   4:off   5:off   6:off
wpa_supplicant  0:off   1:off   2:off   3:off   4:off   5:off   6:off
xfs             0:off   1:off   2:on    3:on    4:on    5:on    6:off
ypbind          0:off   1:off   2:off   3:off   4:off   5:off   6:off
yum-updatesd    0:off   1:off   2:on    3:on    4:on    5:on    6:off

To add a new service to this list, an example in this case - httpd,
# chkconfig --add httpd

The list would now include httpd,
# chkconfig --list | grep httpd
httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
As seen above, it will not run automatically on any run level. I wanted it to run on 2,3,4 and 5 and hence,
# chkconfig --level 2345 httpd on
And to check the new configuration,
# chkconfig --list | grep httpd
httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
A reboot can actually verify whether this configuration works.

chkconfig is not available on Ubuntu by default, but can be installed like so,
# sudo apt-get install chkconfig

Friday, October 15, 2010

Playing with SELinux

Recently on a fresh installation of CentOS 5, I was setting up mysql. My troubles started after changing mysql's datadir from the default(/var/lib/mysql) to /db in /etc/my.cnf

All attempts of starting mysqld failed. A quick look in mysql's log showed complaints about Err no. 13.  This was clearly an issue of file permissions. Even chmod / chown could not help. I could not start mysqld.

 However, If I changed the datadir back to the default (/var/lib/mysql), things worked just fine.

My next guess was SELinux. To confirm, I used the following command
# sestatus
And yes, indeed, it was SELinux's doing. SElinux was enabled and enforcing.
Here I stood with 2 choices (and yes, I experimented with both approaches),
  • Disable SELinux and avoid all the headache
  • Change things to work with SELinux


Disable SELinux and avoid all the headache
I just had to edit the config file for SELinux, and change it's value to disabled.
# vi /etc/sysconfig/selinux
However, a reboot was required before the changes were applied.

Change things to work with SELinux
The approach I took was quite straight forward. I looked at what worked and applied the same settings on my new datadir.

When I looked at the SELinux security context on /var/lib/mysql , it showed something like,
# ls -lZ /var/lib/mysql
drwx------  mysql mysql system_u:object_r:mysqld_db_t    mysql
drwx------  mysql mysql system_u:object_r:mysqld_db_t    test
.....
.....

Now I simply applied the same on /db , like so
# chcon -Rv --type=mysqld_db_t -u system_u -r object_r /db
Thats it. SELinux had no more problems with mysqld using the new datadir.


A Complete tutorial on SELinux / CentOS can be found at http://wiki.centos.org/HowTos/SELinux

Monday, September 27, 2010

Mistakes while attempting Passwordless ssh

Frequently people run into problems when they attempt to configure a passwordless ssh between 2 servers.

There are plenty of tutorials available on the internet for passwordless ssh and hence, it is not in scope of this post. However, I would like to highlight mistakes which people make before wondering "why is it not working!!"?


1. People do not follow the tutorial completely and usually miss the last step!
chmod 644 authorized_keys
ssh does not like it when authorized_keys is open to other users. You could also use 600 instead of 644.

2.  A not-so-common mistake is to verify whether or not sshd is running, you could verify it using
ps -ef | grep sshd

3. If you want a bi-directional passwordless ssh, i.e Server A to Server B and Server B to Server A. Then, maybe you should repeat the tutorial again for Server B to Server A.

Sunday, August 1, 2010

Linux and .NET

Sometime ago, I was required to test a few C# .NET sample code snippets. But here lied the problem, I had no access to a windows machine at that time. All I had was a machine with centOS 5.

Fortunately, there is the Mono Project for difficult times like these. Here is what I did,
# yum install mono-core.i386
# yum install xsp.i386
# yum install mono-web.i386
# yum install mod_mono.i386 (if you need it to be configured with apache)
You can configure mono with apache but since I was in a hurry, the fastest and the easiest way to do this was the xsp webserver (which is not usually recommended for production use).

All it needed me to do was to go the folder which had the scripts in it on the command line,
# cd /extra/scripts/
and then type the magic word,
# xsp
Output:
xsp
Listening on port: 8080 (non-secure)
Listening on address: 0.0.0.0
Root directory: /extra/scripts/
Hit Return to stop the server.
I could then test the sample codes by browsing to http://localhost:8080/

However, this does not mean that xsp is not configurable, xsp --help really helps :)

Sunday, March 21, 2010

Clearing saved conversations from Empathy IM

Empathy IM is the current default IM client in GNome.

Every time you sign in and open a chat window, it shows all your previous conversations in the same window above. Doing a Conversation > clear or a ctrl+L will only temporarily clear the screen. When you reopen the window, it reappears. It stores all the conversations as text files (xml with .log extensions).

If you do not want Empathy to log your conversations or rather if you just want to remove all the conversations that it has saved, in the current version(2.28.1.1), there is probably no way to clear the saved conversations permanently from the GUI(at least I could not find it yet).

The Empathy logs are stored in the following folder,

/your_home_folder/.local/share/Empathy/logs/

cd ~/.local/share/Empathy/logs/

you should be able to find a folder named after your IM account in there. And inside it, you should be able to see a folder for each of your contact containing logs of previous conversations.

Deleting my IM account folder from Empathy's log folder cleared the previous conversations permanently for me. However, fresh conversations are logged when you use the client again. Hopefully, future versions will have some mechanism to control this chat log.

Sunday, March 14, 2010

apache2: bad user name ${APACHE_RUN_USER}

I am used to compiling(almost always) the apache 2 server on linux. However, this time when I set up my new ubuntu machine, I decided to try the mighty apt-get/package manager. After installation I tried browsing to http://localhost/ and was quite happy to see the server running.

Sometime later, I felt the need to restart apache. When I tried to restart using the apachectl command, it said command not found. Surely, things were different from how it would have been if I had compiled apache manually.

Doing a "ps -ef | grep apache" showed me the process running as
/usr/sbin/apache2 -k start
Thinking that, probably, thats how we start it in this distro(if installed using apt-get), I tried running the following to restart the webserver,
sudo apache2 -k restart
but I was disappointed to see this message show up,
apache2: bad user name ${APACHE_RUN_USER}
Spending sometime on google and the ubuntu forums and I figured that I was running the wrong command.

Both of these commands(below) worked for me,
sudo apache2ctl -k restart
or
sudo /etc/init.d/apache2 restart
Moreover, apache is not installed where it would be if compiled(it defaults to /usr/local/apache2).

Although the installation was smooth and extremely easy, it still makes me wonder if things(config/folders/etc) would be better if left default?