The skill which refuses to die - Shell scripting

Shell scripts are very powerful in text processing, so if you want to find, sort, edit, manipulate and output formatted text you can very easily do it and that too from the command line using a shell script! As many say the philosophy behind writing a shell script is same as from where it originated, which is Unix. You should not write or install a new software to solve your next problem, most probably the solution is already present in form of a shell script.

Now let's see what shell scripts can do for you. Shell scripts can

1. Find text or files for you from thousands of files buried under hundreds of directories.
2. Find and replace text in thousands of files.
3. Read and output delimited files and it's fields.
4. Send mail with attachments.
5. Make backups of your files and also restore them.
6. Compile programs for you.
7. Download files from the internet.
8. Sync your local files with remote files in the server.
9. Scan your files for viruses.
10. FTP your files to and from your remote server.
11. Run batch jobs for you overnight
12. Monitor, report and block IP addresses which attack your server or your PC automatically.
13. Filter your mails for spam.
14. You name it and it can do it for you exactly as you want.

Now let's get into the step by step process of using a shell script to solve your problem.

1. Find out what you need to achieve.
2. Break it into smallest possible steps each of which can be easily done.
3. Sequence your steps identified above one after another.
4. Identify repetitive tasks in your iteration.
5. Identify your failure and success criteria of each of your smaller tasks.

Let me illustrate it with an example with relation to above mentioned steps.

1. Report first 10 IP address from iptables log which attack my server's SSH port most which is blocked by my firewall.
2. Open and read system syslog file, search for text patterns, sort text.
3. Read the server's syslog file, identify logs printed by iptables and separate them out, extract source ip addresses and destination port from those logs, filter out IP addresses where destination port is 22 (SSH), sort the IP addresses, find the count of each distinct IP address, sort again based on count in descending order, report the first 10.
4. No repetitive tasks are present.
5. If no IP addresses attack my SSH port then output a appropriate message or if some error happens then output an error message and quit.

Before going about illustrating the solution to the above problem using a shell script I would like you know how a shell script uses it's capability to bind together the processing abilities of numerous UNIX/Linux utilities to get a desired output. First, most of the output of commands are formatted output which can be directly fed as an input to another command. Like cat command outputs text from a file line by line and grep command can process an input stream line by line. So let's see how they can do it in tandem using the below expression.

 

cat myfile.txt | grep hello;

 

Here the | symbol just takes the output of cat command and feeds it to the grep command line by line, where grep is trying to find the word hello in each line outputted by the cat command. Here grep also outputs line by line the lines which have the word hello. Now let's further process the combined output of both these commands. Let's say now I want to find how many lines have the word hello into it. So my expression now becomes as below.

 

cat myfile.txt | grep hello | wc -l

 

So now you know how output of commands can be further processed by other commands. Now coming back to our original problem and it's solution.

 

cat /var/log/syslog | grep "BLOCKED_PACKETS" | sed -rn 's/^.*SRC=([^ ]*).*DPT=22 .*/\1/p' | sort | uniq -c | sort -nr | head -10

The above one liner will give you your top ten IP addresses who are attacking your SSH port.

Let me explain what every command is doing and what it is passing to the next command.

cat /var/log/syslog - Reads the system log file from location /var/log/syslog and output the contents of the file line by line.

grep "BLOCKED_PACKETS" - Takes the output of the above command and filters lines which have the comment "BLOCKED_PACKETS" which is usually set by the iptables rules when it blocks a packet. The exact comment may vary by your iptables configuration. If your iptables does not append any comments to it's log then you may omit this command altogether.

sed -rn 's/^.*SRC=([^ ]*).*DPT=22 .*/\1/p' - Takes the output of the above command and extracts the IP addresses which tried to connect to the SSH port 22 with the help of regular expressions. Iptables usually writes log in the format SRC=<IP address> and DPT=<port number> to output source IP address and destination port respectively.

sort - Sorts the IP address for next command uniq to report the unique IP addresses correctly.

uniq -c - Takes the output of above command and prints the count and IP address of all unique IP addresses.

sort -nr - Sorts the output of the above command in descending order of the count of IP addresses.

head -10 - Takes the output of the above command and prints only the first 10 lines.

So you can see how a complicated problem can be solved by just passing output of one command to another.

The complete solution to the above stated problem can be solved using the below shell script.

#!/bin/bash   SSH_ATTACK_PRESENT=`cat /var/log/syslog | grep "BLOCKED_PACKETS.* DPT=22" | head -1`;   if [ "$SSH_ATTACK_PRESENT" = "" ] then  echo "No attacks on SSH port.";  exit 0; fi;   cat /var/log/syslog | grep "BLOCKED_PACKETS" | sed -rn 's/^.*SRC=([^ ]*).*DPT=22 .*/\1/p' | sort | uniq -c | sort -nr | head -10;   if [[ $? -ne 0 ]] then  echo "An error occured while processing your request, quitting";  exit 1; fi;

A shell script can be used as program to use Linux/UNIX commands with conditional branching and iteration based on output of the commands and their exit criteria.

A shell script provides the below special variables for the user to program accordingly.

Shell variable	Description
$?	Exit code of last executed command
$*	All command line parameters passed to the shell script separated by $IFS if parameters were passed inside quotes
$@	All command line parameters passed to the shell script as it is.
$1, $2, $3 ....	Command line parameter 1, 2, 3 ... respectively
$0	Invocation command of the shell script itself
$#	Count of command line parameters
$$	Process ID of the running shell script
$!	Process ID of the last command run in the background

Shell uses the below control statements like a programming language to take care of branching and iteration.

Control statement	Description
if, else, elif	Checks for a condition to be true or false, gets into the block if the condition is true
for	Loops through a list separated by $IFS character for each of the values in the list
while	Loops till the condition in the while statement is true
case	Jumps to the statement where the condition matches, behaves similar to switch-case used in various programming languages
break	Jumps out of the current loop block (for, while)
continue	Jumps to the next iteration of the current loop block

Another useful feature of shell scripts is functions. You can write functions in a shell script to execute a block of shell script statements over and over again. A good use case for functions is to perform logging where you need to output the log in a certain format like appending date-time or a suffix. Functions can be invoked similar to invoking another shell script. Also shell script functions utilizes the special variables as described above with respect to it's own invocation like for example I invoke a function with two parameters "param1" and "param2", $1 and $2 variables inside the function will have values param1 and param2 respectively. Below is an example to illustrate this feature.

#!/bin/bash function test_function {     echo $1","$2;     return 0; } test_function "param1" "param2";

The above shell script will print param1,param2 as output.