MTR – Network Diagnostic Tool

Traceroute

You may have used traceroute in a Micrsoft OS or in Linux.  It’s quite simple and can tell you a few things about the connection between you and any other device on the internet.   Just type in traceroute <domain name> and hit enter.  You should see something like the following.

The second column lists the name (or IP) of the next device interface, and then the next column it’s IP address.  This goes on until you hit the destination.  All the while you are given the time it has taken to send the ICMP message to each device and then the return message.  If you notice, the further away the device is, the longer it takes.  But not always, as the last two devices show ~66ms while #11 had ~82ms to ~87ms.

If you were having problems on the internet and it seemed there were a bottleneck somewhere, this nice little tool can help you identify if one exists.  Perhaps if one of those above was returning ~70ms and then the next device gave you ~250ms.   That’s a noticeable difference, and it could indicate the the link between those two devices has maxed it’s bandwidth at that particular time.   Not that you could do anything about it.

MTR

So what if the problem were intermittent, or traceroute did not show any problems because perhaps the chokepoint is experiencing intermittent bursts of data?  Enter MTR, the big brother to traceroute.  MTR (or My Trace Route) can show you a constant display of each link and how it’s holding up.  Take a look.

If you have MTR installed (aptitude install mtr), run it as you would traceroute.  i.e. MTR YAHOO.COM

Click to see full image

The default display looks much like a regular traceroute display, aside from the data is constantly changing and there’s a few more columns of information.  The data changes due to the fact that it’s constantly getting data for you, no once, but constantly until you quit.

After the hostname, you are given columns telling you how much packet loss you get.  How many packets were sent.  Then you get to see how the pings are doing.  You get what the last time it took for the response, the average time, the best it’s been to that particular host and the worst time.  You also get a standard deviation.   Not exactly sure on this one yet, but lower seems better.

As you can see from the example above, once we went from device #4 to  #5, the pings doubled and then from # 5 to #6, the pings doubled again.   The first time it doubled, it’s not too bad because that’s an increase in 30ms whereas the second jump was a good 72+ ms.

Not that I’m saying here’s a bottleneck, by no means.  However, if anywhere between me and nandomedia.com there would be one, that’s my first suspect.   This trace is pretty simple anyway.  If I were to see much larger numbers then I would be concerned.  Again, not that I can do anything about it, but at least I know it’s not my network, unless the problem was on my network, then I’ll point fingers.

Click to see full image

Hit the D key.  Here you can change the display.   You get a nice view of the packets as they are returned.  This first view does not offer much, so hit the D key again.   DO IT!!!  You’ll like it.  At least better than this view.

Click to see full image

Now this is much more informational.  On the bottom of the screen is a scale.  It shows you what each dot, number and letter up above mean.  The period is 7ms and as you can see from my PC to my router, that’s what it should be.  The next hop is to my ISP and there’s 3’s all along there and the other 2 routers at that location.  The 3 is less than 54ms, but since it’s not a 2, it’s greater than 29ms.  You see an ‘a’ and a ‘b’ up there too.  So the return just from my ISP had a ping return over 54ms and one over 120ms just in these 57 pings.

The display is nice because it scrolls nicely to the left and you can see if there’s just a burp as in the ‘c’ at location #9 or if there’s a serious problem.  And if you need to resize your terminal window, MTR will adjust what it displays with ease.  Above it’s got room to show the last 57 pings, but if I widened it, it would quickly change accordingly.

So, as traceroute is handy, it’s not constant, it gives a snapshot and things may be fine at that moment.  Ping is okay, but does not provide very much info.   MTR is the all around tool to replace both.

Hackers!!

Hackers are trying to get in.

Not to alarm anyone, but if you didn’t know already, unscrupulous individuals are trying to get into your system every day.  Most people are protected by their routers and firewall software.   These allow traffic to come into your device ONLY if you have initiated a connection to the remote device from within your own network or computer.

Fair enough.  But for someone who wants to play with their network from remote locations, herein lies the problem.  In order to connect to my Linux box from anywhere outside of my home, I first have to direct port 22 (for SSH) in  my router to my linux.  Now I can have access.  But so can anyone else who attempts a connection to my IP address.  Likely they have a script running that methodically runs down IP addresses, trying to get a response.  Now when they hit my IP address with SSH, instead of the standard no response, they get a LOGIN: prompt.  Now they know it’s something worth trying to access.  The next script starts trying standard login names, then when it hits a upon a valid name, it then goes through the passwords.  Here’s what it looked like last night.

This is a small excerpt from my /var/log/auth.log (grep -i failed /var/log/auth.log)

As you can see, they were using standard names alphabetically.  They did way more than that, using standard usernames like “user, sysadmin, admin, guest, matrix, neo, trinity” and so on.  The lesson learned here is that any usernames on your system should not be dictionary words.  Too easy to guess.  Make it something you can remember, but somewhat resembles a secure password (numbers, big & small letters).

iftop

How did I know someone was hitting my box?  Pure luck.  I figured people were trying before, but did not realize to what extent.  I also wonder how many of these script kiddies had hit my box before?

Every now and then when I’m logged into my linux box, I like to just play around to keep myself in practice of some of the many tools there are, like ipcalc and such.  One of the tools I’ve installed is iftop. It’s much like top, but for network interfaces.  I have a second network card on my box connected to my Cisco router.  The port on that router mirrors all the other ports so that it can see ALL the traffic on my network.  The benefit of this is when I run iftop on this interface, I can see ALL my traffic, not just the data specific to that IP address.  It’s great too for working with WireShark.

(sudo iftop -p -B -i eth1)  (iftop does not come with linux, you must install it)

I can see devices on my network communicating with each other.  Nagios on my linux box polling devices, websites being delivered to another PC on my home network, the router looking for other routers, etc.   What I did see last night though was a URL that did not look right.  It ended with a .gr, not the usual .com, .net, .org and was not on my local network.

I performed a whois on that sucker and found it was from Greece.  The University of Patras in Greece.   Ahhhhh, it figures.  Kid goes to college, takes some computer science cources, learns all about computers, networking and so on.  Now he figures he can make his own little scripts to access networks around the world.  And he is probably successful at times.  Not this time.  I found him.

Now that I see him, what to do?  Well, since I’ve a Cisco router and the know-how.  I create an ACL (access control list) to block him.   I could have blocked just his address, but was not sure if he’d just get another one somehow and try again, so I wanted to block the whole country.  My friend Sean had recently created a perl script that will create an ACL for you to include EVERY ip range for the country of your choice.  Perfect.  I ran this sucker and WOW, Greece had 180 IP ranges.  Weird enough, none of them included my hackers address ( I mean the University of Greece).  We’ll work on that later.  So I applied the new ACL, including a range for the university and applied it to my Interface.

As you can see by the last few lines of my ACL, that after it was applied (BTW, my /var/log/auth.log automatically stopped logging because now he could not get through) I still had 4,677 hits from him.  But he was stopped at the router.

That last line was to make sure anyone else (myself especially) could get through.   Okay, so that does not stop the others.  We’ll get to that soon.  But in the mean time, this particular misfit is denied.

Yes, I need to block out the world and yet still let myself in.   So now I’ll create a smaller ACL that denies everyone except my local region.  This being because I may want to get in from different places, i.e. my parents, work, coffee shops, etc.

So a smaller and complete ACL is shown below.

I changed the real IP addresses for display purposes.  The first 3 lines permit access from 2 major ISPs in my geographical area.  Line 90 allows DNS queries to succeed.  Line 91 allows FTP connections (I’d rather not as it’s not too secure, but I have a couple websites that the hosting companies don’t seem to care too much about the security).  Line 92 allows PING to work.  It’s nice to be able to ping a website and get a response just to know certain things are working while I’m troubleshooting.  Then line 100 allows anything that I’ve initated to come through.  So if I want to access a web page hosted at the University of Greece, I can.  But their students cannot enter my domain.

Take a look at that last line that denies everyone else, there’s 16 matches.  I have these logged, as you can see.  From the log I found hits from China, France and Poland.  All within a few hours.

Log Entry:
Apr 28 23:55:57.064:  list 2100 denied tcp 83.16.20.98 –   Connection attempt from Poland.

More to come

Now this does not totally protect me.  I am still vulnerable from my local area, but for a limited time.  Sean has offered me access to his secure network that will allow me to do one thing and one thing only from there, to connect to my network.  Now he has a static IP address, so my ACL can then be written to allow an un-established connection from the outside from that particular IP address only.

Another safeguard is IPTABLES.  I have briefly toyed with this firewall option built into Linux that is very flexible.  Supposedly one can set it up to allow a set number of failed login attempts before it disables that particular IP address for a set amount of time.  This would seriously slow down script kiddies and others.  However if I were to mistype my password that number of times, I could lock myself out for some time as well.  Oh well, guess I better slow down on my typing.

The Goal

Here is my goal.

Now keep in mind, if you’re working on your own network, mine is likely overkill for most, yet not enough for the more technically astute.  However if your network is simple, there’s still information here that can be of benefit.

I have a Cisco 1811 router which I obtained as a used item, yet it’s still functional.  This I will use to connect to the Internet via DSL and then ALL other devices will connect with it.  This is a good thing because it is capable of doing a lot more than what a basic residential router can do.  i.e. VLANs, GRE tunnelling, etc.

I do have the need for wireless connectivity for a Laptop, Nintendo Wii and a DS.  For that I have a Linksys wireless router.  Of course, be sure to enable the wireless security on your own wireless connections.  A few years back I drove through a neighborhood with NetStumbler loaded on my laptop and within 10-15 minutes time had detected 80 or so wireless signals with 80% of them unsecured.   The last thing you would want is someone doing something illegal on your internet connection because it will be linked back to your connection.

Also attached to my network (where the 8 switch ports on the Cisco1811 will come in handy) are 2 desktop computers, a printer and an XBOX.  I have a spare computer which is going to come in handy with this project in that it will have Linux on it for running various servers.  I have another old PC that will also have Linux on it as well.  Your old computers can still be useful for something.

One of these Linux boxes will be an Apache server for serving up web pages from my network.  I do not intend on running a domain from this, but rather just have a place for accessing files I might need remotely, or for sharing with friends and family.  It will have any easy way to access the information, but will have a degree of separation from my network for security purposes.

The Linux box inside the network (VLAN 1) is going to have other services running.

Nagios is a good service for monitoring the status of the network.  It can monitor the network connections of all the  devices you need to keep your network operational.  It could keep track of whether ALL devices are connected and on, but you would need to keep ALL devices powered on then.  Otherwise it will show alot of downtime when the device is turned off, giving the impression that there’s a problem on your network.  It can even e-mail you or in many instances text message you of any problems.   You can also have it keep track of services running on your Linux servers, again notifying you of any trouble.

Cacti is another service that can monitor SNMP messages from your routers and servers, giving you a graphical display of things such as processor usage, memory, bandwidth of each interface, etc.  (If you’re interested in that.  I will be, so you can visit again at that time that I delve into it.)

Apache will need to be installed on this machine, due to the fact that both Nagios and Cacti provide details via web interface.

Squid Proxy Server is another server that hopefully I can get installed and working properly.  I was successful a while back, but they have since changed things in the configuration, so I’ll be starting anew.  A Proxy Server will cache web data when computers on your network visit sites on the Internet.  So my more regular websites will be served faster since alot of the data won’t need to be downloaded from the Internet again but rather from my local Proxy and it’ll cut down on my bandwidth, due to the same reason.  Since my ISP has a bandwidth cap, this may come in handy.  We’ll see.

One more note on Squid, is the ability to setup a 3rd party service that creates a database which can be updated regularly that blocks web sites that you chose by category.  Once Squid is setup and working, that will be the next task to tackle.

A Mail server is also in order for notifications.

As shown in the planned layout, the network will be divided into 2 VLANs where VLAN1 will have access to the server in VLAN 2, but VLAN 2 will not have access to VLAN 1, keeping out anyone who may try to compromise the server.  This is where having a Cisco router/switch will be an asset.

If you’re curious about the device above the printer, that is a Network Attached Storage.  It’s a great place to store files that you can then access from the other computers on the network.  It stays on, therefore when you want to listen to an mp3 you have, you don’t have to make sure that the PC that has the mp3 file on it is turned on, you just need to access your NAS through your network.  I may eventually move this service to the Linux server with an additional hard drive and create a file server on it instead.

Plans are subject to change as we go along.  Every day it seems I find something else new and useful.