Home » Linux

Category Archives: Linux

NFS Server

Network File System (NFS)

  •  A Network File System (NFS) allows remote hosts to mount file systems over a network and interact with those file systems as though they are mounted locally. This enables system administrators to consolidate resources onto centralized servers on the network.
  • All versions of NFS can use Transmission Control Protocol (TCP) running over an IP network, with NFSv4 requiring it. NFSv2 and NFSv3 can use the User Datagram Protocol (UDP) running over an IP network to provide a stateless network connection between the client and server.
  • The NFS server sends the client a file handle after the client is authorized to access the shared volume. This file handle is an opaque object stored on the server’s side and is passed along with RPC requests from the client. The NFS server can be restarted without affecting the clients and the cookie remains intact. However, because UDP is stateless, if the server goes down unexpectedly, UDP clients continue to saturate the network with requests for the server. For this reason, TCP is the preferred protocol when connecting to an NFS server.
  • After the client is granted access by TCP wrappers, the NFS server refers to its configuration file,/etc/exports, to determine whether the client is allowed to access any of the exported file systems. Once access is granted, all file and directory operations are available to the user.

Required Services

 Red Hat Enterprise Linux uses a combination of kernel-level support and daemon processes to provide NFS file sharing. All NFS versions rely on Remote Procedure Calls (RPC) between clients and servers. RPC services under Linux are controlled by the portmap service. To share or mount NFS file systems, the following services work together, depending on which version of NFS is implemented:
  • nfs — (/sbin/service nfs start) starts the NFS server and the appropriate RPC processes to service requests for shared NFS file systems.
  • nfslock — (/sbin/service nfslock start) is a mandatory service that starts the appropriate RPC processes to allow NFS clients to lock files on the server.
  • portmap — accepts port reservations from local RPC services. These ports are then made available (or advertised) so the corresponding remote RPC services access them. portmap responds to requests for RPC services and sets up connections to the requested RPC service. This is not used with NFSv4.

The general syntax for the line in /etc/fstab is as follows:

server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr

The mount point /pub must exist on the client machine before this command can be executed. After adding this line to /etc/fstab on the client system, type the command mount /pub at a shell prompt, and the mount point /pub is mounted from the server.

The /etc/fstab file is referenced by the netfs service at boot time, so lines referencing NFS shares have the same effect as manually typing the mount command during the boot process.

A sample /etc/fstab line to mount an NFS export looks like the following example:

<server>:</remote/export> </local/directory> <nfs-type> <options> 0 0

Email Server

The Real E-mail System

For the vast majority of people right now, the real e-mail system consists of two different servers running on a server machine. One is called the SMTP server, where SMTP stands for Simple Mail Transfer Protocol. The SMTP server handles outgoing mail. The other is either a POP3 server or an IMAP server, both of which handle incoming mail. POP stands for Post Office Protocol, and IMAP stands for Internet Mail Access Protocol. A typical e-mail server looks like this:

The SMTP server listens on well-known port number 25, POP3 listens on port 110 and IMAP uses port 143.

The SMTP Server

Whenever you send a piece of e-mail, your e-mail client interacts with the SMTP server to handle the sending. The SMTP server on your host may have conversations with other SMTP servers to deliver the e-mail.

Let’s assume that I want to send a piece of e-mail. My e-mail ID is brain, and I have my account on howstuffworks.com. I want to send e-mail to jsmith@mindspring.com. I am using a stand-alone e-mail client like Outlook Express.

When I set up my account at arpittak.com, I told Outlook Express the name of the mail server — mail.arpittak.com. When I compose a message and press the Send button, here’s what happens:

  1. Outlook Express connects to the SMTP server atmail.arpittak.com using port 25.
  2. Outlook Express has a conversation with the SMTP server, telling the SMTP server the address of the sender and the address of the recipient, as well as the body of the message.
  3. The SMTP server takes the “to” address (jsmith@mindspring.com) and breaks it into two parts: the recipient name (jsmith) and the domain name (mindspring.com). If the “to” address had been another user at  arpittak.com., the SMTP server would simply hand the message to the POP3 server for arpittak.com (using a little program called the delivery agent). Since the recipient is at another domain, SMTP needs to communicate with that domain.
  4. The SMTP server has a conversation with a Domain Name Server, or DNS. It says, “Can you give me the IP address of the SMTP server for mindspring.com?” The DNS replies with the one or more IP addresses for the SMTP server(s) that Mindspring operates.
  5. The SMTP server at arpittak.com connects with the SMTP server at Mindspring using port 25. It has the same simple text conversation that my e-mail client had with the SMTP server for arpittak, and gives the message to the Mindspring server. The Mindspring server recognizes that the domain name for jsmith is at Mindspring, so it hands the message to Mindspring’s POP3 server, which puts the message in jsmith’s mailbox.

If, for some reason, the SMTP server at arpittakcannot connect with the SMTP server at Mindspring, then the message goes into a queue. The SMTP server on most machines uses a program called sendmail to do the actual sending, so this queue is called the sendmail queue. Sendmail will periodically try to resend the messages in its queue. For example, it might retry every 15 minutes. After four hours, it will usually send you a piece of mail that tells you there is some sort of problem. After five days, most sendmail configurations give up and return the mail to you undelivered.

The SMTP server understands very simple text commands like HELO, MAIL, RCPT and DATA. The most common commands are:

  • HELO – introduce yourself
  • EHLO – introduce yourself and request extended mode
  • MAIL FROM: – specify the sender
  • RCPT TO: – specify the recipient

The POP3 and IMAP Servers

When you check your e-mail, your e-mail client connects to the POP3 server using port 110. The POP3 server requires an account name and a password. Once you’ve logged in, the POP3 server opens your text file and allows you to access it. Like the SMTP server, the POP3 server understands a very simple set of text commands. Here are the most common commands:

  • USER – enter your user ID
  • PASS – enter your password
  • QUIT – quit the POP3 server
  • LIST – list the messages and their size

The IMAP Server

As you can see, the POP3 protocol is very simple. It allows you to have a collection of messages stored in a text file on the server. Your e-mail client (e.g. Outlook Express) can connect to your POP3 e-mail server and download the messages from the POP3 text file onto your PC. That is about all that you can do with POP3.

Many users want to do far more than that with their e-mail, and they want their e-mail to remain on the server. The main reason for keeping your e-mail on the server is to allow users to connect from a variety of machines. With POP3, once you download your e-mail it’s stuck on the machine to which you downloaded it. If you want to read your e-mail both on your desktop machine and your laptop (depending on whether you’re working in the office or on the road), POP3 makes life difficult.

IMAP (Internet Mail Access Protocol) is a more advanced protocol that solves these problems. With IMAP, your mail stays on the e-mail server. You can organize your mail into folders, and all the folders live on the server as well. When you search your e-mail, the search occurs on the server machine, rather than on your machine. This approach makes it extremely easy for you to access your e-mail from any machine, and regardless of which machine you use, you have access to all of your mail in all of your folders.

Web Server

The Basic Process

Let’s say that you are sitting at your computer, surfing the Web, and you get a call from a friend who says, “I just read a great article! Type in this URL and check it out. It’s at http://arpittak.com .  So you type that URL into your browser and press return. And magically, no matter where in the world that URL lives, the page pops up on your screen.

Your browser formed a connection to a Web server, requested a page and received it.

The browser broke the URL into three parts:

  • The protocol (“http”)
  • The server name (“www.howstuffworks.com”)
  • The file name (“web-server.htm”)

The browser communicated with a name server to translate the server name “www.arpittak.com” into an IP Address, which it uses to connect to the server machine. The browser then formed a connection to the server at that IP address on port 80.

Following the HTTP protocol, the browser sent a GET request to the server, asking for the file “http://www.arpittak.com/index.htm.” (Note that cookies may be sent from browser to server with the GET request.)

The server then sent the HTML text for the Web page to the browser. (Cookies may also be sent from server to browser in the header for the page.) The browser read the HTML tags and formatted the page onto your screen.

Clients and Servers

In general, all of the machines on the Internet can be categorized as two types: servers and clients. Those machines that provide services (like Web servers or FTP servers) to other machines are servers. And the machines that are used to connect to those services are clients. When you connect to Yahoo! at http://www.yahoo.com to read a page, Yahoo! is providing a machine (probably a cluster of very large machines), for use on the Internet, to service your request. Yahoo! is providing a server. Your machine, on the other hand, is probably providing no services to anyone else on the Internet. Therefore, it is a user machine, also known as a client. It is possible and common for a machine to be both a server and a client, but for our purposes here you can think of most machines as one or the other.

A server machine may provide one or more services on the Internet. For example, a server machine might have software running on it that allows it to act as a Web server, an e-mail server and an FTP server. Clients that come to a server machine do so with a specific intent, so clients direct their requests to a specific software server running on the overall server machine. For example, if you are running a Web browser on your machine, it will most likely want to talk to the Web server on the server machine. Your Telnet application will want to talk to the Telnet server, your e-mail application will talk to the e-mail server, and so on…

IP Addresses

To keep all of these machines straight, each machine on the Internet is assigned a unique address called an IP address. IP stands for Internet protocol, and these addresses are 32-bit numbers, normally expressed as four “octets” in a “dotted decimal number.” A typical IP address looks like this:     216.27.61.137

The four numbers in an IP address are called octets because they can have values between 0 and 255, which is 28 possibilities per octet.

Every machine on the Internet has a unique IP address. A server has a static IP address that does not change very often. A home machine that is dialing up through a modem often has an IP address that is assigned by the ISP when the machine dials in. That IP address is unique for that session — it may be different the next time the machine dials in. This way, an ISP only needs one IP address for each modem it supports, rather than for each customer.

Domain Names

Because most people have trouble remembering the strings of numbers that make up IP addresses, and because IP addresses sometimes need to change, all servers on the Internet also have human-readable names, called domain names. For example, www.arppittak.com is a permanent, human-readable name.

The name http://www.arpittak.com actually has three parts:

  1. The host name (“www”)
  2. The domain name (“arpittak”)
  3. The top-level domain name (“com”)

Name Servers

A set of servers called domain name servers (DNS) maps the human-readable names to the IP addresses. These servers are simple databases that map names to IP addresses, and they are distributed all over the Internet. Most individual companies, ISPs and universities maintain small name servers to map host names to IP addresses. There are also central name servers that use data supplied by VeriSign to map domain names to IP addresses.

Ports

Any server machine makes its services available to the Internet using numbered ports, one for each service that is available on the server. For example, if a server machine is running a Web server and an FTP server, the Web server would typically be available on port 80, and the FTP server would be available on port 21. Clients connect to a service at a specific IP address and on a specific port.

Each of the most well-known services is available at a well-known port number. Here are some common port numbers:

  • echo 7
  • daytime 13
  • qotd 17 (Quote of the Day)
  • ftp 21
  • telnet 23
  • smtp 25 (Simple Mail Transfer, meaning e-mail)
  • time 37
  • nameserver 53

Protocols

Once a client has connected to a service on a particular port, it accesses the service using a specific protocol. The protocol is the pre-defined way that someone who wants to use a service talks with that service. The “someone” could be a person, but more often it is a computer program like a Web browser. Protocols are often text, and simply describe how the client and server will have their conversation.

Perhaps the simplest protocol is the daytime protocol. If you connect to port 13 on a machine that supports a daytime server, the server will send you its impression of the current date and time and then close the connection. The protocol is, “If you connect to me, I will send you the date and time and then disconnect.”

Awk Tutorials

Awk Introduction and Printing Operations

Awk is a programming language which allows easy manipulation of structured data and the generation of formatted reports. Awk stands for the names of its authors “Aho, Weinberger, and Kernighan”

The Awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then perform associated actions.

Some of the key features of Awk are:

  • Awk views a text file as records and fields.
  • Like common programming language, Awk has variables, conditionals and loops
  • Awk has arithmetic and string operators.
  • Awk can generate formatted reports

Awk reads from a file or from its standard input, and outputs to its standard output. Awk does not get along with non-text files.

Syntax:

awk '/search pattern1/ {Actions}
     /search pattern2/ {Actions}' file

In the above awk syntax:

  • search pattern is a regular expression.
  • Actions – statement(s) to be performed.
  • several patterns and actions are possible in Awk.
  • file – Input file.
  • Single quotes around program is to avoid shell not to interpret any of its special characters.

Awk Working Methodology

  1. Awk reads the input files one line at a time.
  2. For each line, it matches with given pattern in the given order, if matches performs the corresponding action.
  3. If no pattern matches, no action will be performed.
  4. In the above syntax, either search pattern or action are optional, But not both.
  5. If the search pattern is not given, then Awk performs the given actions for each line of the input.
  6. If the action is not given, print all that lines that matches with the given patterns which is the default action.
  7. Empty braces with out any action does nothing. It wont perform default printing operation.
  8. Each statement in Actions should be delimited by semicolon.

Let us create employee.txt file which has the following content, which will be used in the
examples mentioned below.

$cat employee.txt
100  Thomas  Manager    Sales       $5,000
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000

Awk Example 1. Default behavior of Awk

By default Awk prints every line from the file.

$ awk '{print;}' employee.txt
100  Thomas  Manager    Sales       $5,000
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000

In the above example pattern is not given. So the actions are applicable to all the lines.
Action print with out any argument prints the whole line by default. So it prints all the
lines of the file with out fail. Actions has to be enclosed with in the braces.

Awk Example 2. Print the lines which matches with the pattern.

$ awk '/Thomas/
> /Nisha/' employee.txt
100  Thomas  Manager    Sales       $5,000
400  Nisha   Manager    Marketing   $9,500

In the above example it prints all the line which matches with the ‘Thomas’ or ‘Nisha’. It has two patterns. Awk accepts any number of patterns, but each set (patterns and its corresponding actions) has to be separated by newline.

Awk Example 3. Print only specific field.

Awk has number of built in variables. For each record i.e line, it splits the record delimited by whitespace character by default and stores it in the $n variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4. $0 represents whole line. NF is a built in variable which represents total number of fields in a record.

$ awk '{print $2,$5;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000

$ awk '{print $2,$NF;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000

In the above example $2 and $5 represents Name and Salary respectively. We can get the Salary using  $NF also, where $NF represents last field. In the print statement ‘,’ is a concatenator.

Awk Example 4. Initialization and Final Action

Awk has two important patterns which are specified by the keyword called BEGIN and END.

Syntax:
BEGIN { Actions}
{ACTION} # Action for everyline in a file
END { Actions }

# is for comments in Awk

Actions specified in the BEGIN section will be executed before starts reading the lines from the input.
END actions will be performed after completing the reading and processing the lines from the input.

$ awk 'BEGIN {print "NametDesignationtDepartmenttSalary";}
> {print $2,"t",$3,"t",$4,"t",$NF;}
> END{print "Report Generatedn--------------";
> }' employee.txt
Name	Designation	Department	Salary
Thomas 	 Manager 	 Sales 	         $5,000
Jason 	 Developer 	 Technology 	 $5,500
Sanjay 	 Sysadmin 	 Technology 	 $7,000
Nisha 	 Manager 	 Marketing 	 $9,500
Randy 	 DBA 	 	 Technology 	 $6,000
Report Generated
--------------

In the above example, it prints headline and last file for the reports.

Awk Example 5. Find the employees who has employee id greater than 200

$ awk '$1 >200' employee.txt
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000

In the above example, first field ($1) is employee id. So if $1 is greater than 200, then just do the default print action to print the whole line.

Awk Example 6. Print the list of employees in Technology department

Now department name is available as a fourth field, so need to check if $4 matches with the string “Technology”, if yes print the line.

$ awk '$4 ~/Technology/' employee.txt
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
500  Randy   DBA        Technology  $6,000

Operator ~ is for comparing with the regular expressions. If it matches the default action i.e print whole line will be  performed.

Awk Example 7. Print number of employees in Technology department

The below example, checks if the department is Technology, if it is yes, in the Action, just increment the count variable, which was initialized with zero in the BEGIN section.

$ awk 'BEGIN { count=0;}
$4 ~ /Technology/ { count++; }
END { print "Number of employees in Technology Dept =",count;}' employee.txt
Number of employees in Tehcnology Dept = 3

Linux Boot Process

6 Stages of Linux Boot Process (Startup Sequence)

Press the power button on your system, and after few moments you see the Linux login prompt.

Have you ever wondered what happens behind the scenes from the time you press the power button until the Linux login prompt appears?

The following are the 6 high level stages of a typical Linux boot process.

1. BIOS

  • BIOS stands for Basic Input/Output System
  • Performs some system integrity checks
  • Searches, loads, and executes the boot loader program.
  • It looks for boot loader in floppy, cd-rom, or hard drive. You can press a key (typically F12 of F2, but it depends on your system) during the BIOS startup to change the boot sequence.
  • Once the boot loader program is detected and loaded into the memory, BIOS gives the control to it.
  • So, in simple terms BIOS loads and executes the MBR boot loader.

2. MBR

  • MBR stands for Master Boot Record.
  • It is located in the 1st sector of the bootable disk. Typically /dev/hda, or /dev/sda
  • MBR is less than 512 bytes in size. This has three components 1) primary boot loader info in 1st 446 bytes 2) partition table info in next 64 bytes 3) mbr validation check in last 2 bytes.
  • It contains information about GRUB (or LILO in old systems).
  • So, in simple terms MBR loads and executes the GRUB boot loader.

3. GRUB

    • GRUB stands for Grand Unified Bootloader.
    • If you have multiple kernel images installed on your system, you can choose which one to be executed.
    • GRUB displays a splash screen, waits for few seconds, if you don’t enter anything, it loads the default kernel image as specified in the grub configuration file.
    • GRUB has the knowledge of the filesystem (the older Linux loader LILO didn’t understand filesystem).
    • Grub configuration file is /boot/grub/grub.conf (/etc/grub.conf is a link to this). The following is sample grub.conf of CentOS.
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-194.el5PAE)
          root (hd0,0)
          kernel /boot/vmlinuz-2.6.18-194.el5PAE ro root=LABEL=/
          initrd /boot/initrd-2.6.18-194.el5PAE.img
  • As you notice from the above info, it contains kernel and initrd image.
  • So, in simple terms GRUB just loads and executes Kernel and initrd images.

4. Kernel

  • Mounts the root file system as specified in the “root=” in grub.conf
  • Kernel executes the /sbin/init program
  • Since init was the 1st program to be executed by Linux Kernel, it has the process id (PID) of 1. Do a ‘ps -ef | grep init’ and check the pid.
  • initrd stands for Initial RAM Disk.
  • initrd is used by kernel as temporary root file system until kernel is booted and the real root file system is mounted. It also contains necessary drivers compiled inside, which helps it to access the hard drive partitions, and other hardware.

5. Init

  • Looks at the /etc/inittab file to decide the Linux run level.
  • Following are the available run levels
    • 0 – halt
    • 1 – Single user mode
    • 2 – Multiuser, without NFS
    • 3 – Full multiuser mode
    • 4 – unused
    • 5 – X11
    • 6 – reboot
  • Init identifies the default initlevel from /etc/inittab and uses that to load all appropriate program.
  • Execute ‘grep initdefault /etc/inittab’ on your system to identify the default run level
  • If you want to get into trouble, you can set the default run level to 0 or 6. Since you know what 0 and 6 means, probably you might not do that.
  • Typically you would set the default run level to either 3 or 5.

6. Runlevel programs

  • When the Linux system is booting up, you might see various services getting started. For example, it might say “starting sendmail …. OK”. Those are the runlevel programs, executed from the run level directory as defined by your run level.
  • Depending on your default init level setting, the system will execute the programs from one of the following directories.
    • Run level 0 – /etc/rc.d/rc0.d/
    • Run level 1 – /etc/rc.d/rc1.d/
    • Run level 2 – /etc/rc.d/rc2.d/
    • Run level 3 – /etc/rc.d/rc3.d/
    • Run level 4 – /etc/rc.d/rc4.d/
    • Run level 5 – /etc/rc.d/rc5.d/
    • Run level 6 – /etc/rc.d/rc6.d/
  • Please note that there are also symbolic links available for these directory under /etc directly. So, /etc/rc0.d is linked to /etc/rc.d/rc0.d.
  • Under the /etc/rc.d/rc*.d/ directories, you would see programs that start with S and K.
  • Programs starts with S are used during startup. S for startup.
  • Programs starts with K are used during shutdown. K for kill.
  • There are numbers right next to S and K in the program names. Those are the sequence number in which the programs should be started or killed.
  • For example, S12syslog is to start the syslog deamon, which has the sequence number of 12. S80sendmail is to start the sendmail daemon, which has the sequence number of 80. So, syslog program will be started before sendmail.

There you have it. That is what happens during the Linux boot process.

Raid Tutorials

RAID stands for Redundant Array of Inexpensive (Independent) Disks.

On most situations you will be using one of the following four levels of RAIDs.

  • RAID 0
  • RAID 1
  • RAID 5
  • RAID 10 (also known as RAID 1+0)

This article explains the main difference between these raid levels along with an easy to understand diagram.

In all the diagrams mentioned below:

  • A, B, C, D, E and F – represents blocks
  • p1, p2, and p3 – represents parity

RAID LEVEL 0

Following are the key points to remember for RAID level 0.

  • Minimum 2 disks.
  • Excellent performance ( as blocks are striped ).
  • No redundancy ( no mirror, no parity ).
  • Don’t use this for any critical system.

RAID LEVEL 1

Following are the key points to remember for RAID level 1.

  • Minimum 2 disks.
  • Good performance ( no striping. no parity ).
  • Excellent redundancy ( as blocks are mirrored ).

RAID LEVEL 5


Following are the key points to remember for RAID level 5.

  • Minimum 3 disks.
  • Good performance ( as blocks are striped ).
  • Good redundancy ( distributed parity ).
  • Best cost effective option providing both performance and redundancy. Use this for DB that is heavily read oriented. Write operations will be slow.

RAID LEVEL 10

Following are the key points to remember for RAID level 10.

  • Minimum 4 disks.
  • This is also called as “stripe of mirrors”
  • Excellent redundancy ( as blocks are mirrored )
  • Excellent performance ( as blocks are striped )
  • If you can afford the dollar, this is the BEST option for any mission critical applications (especially databases).

Linux Commands

1. tar command examples

Create a new tar archive.

$ tar cvf archive_name.tar dirname/

Extract from an existing tar archive.

$ tar xvf archive_name.tar

View an existing tar archive.

$ tar tvf archive_name.tar

2. grep command examples

Search for a given string in a file (case in-sensitive search).

$ grep -i "the" demo_file

Print the matched line, along with the 3 lines after it.

$ grep -A 3 -i "example" demo_text

Search for a given string in all files recursively

$ grep -r "ramesh" *

3. find command examples

Find files using file-name ( case in-sensitve find)

# find -iname "MyCProgram.c"

Execute commands on files found by the find command

$ find -iname "MyCProgram.c" -exec md5sum {} ;

Find all empty files in home directory

# find ~ -empty

4. ssh command examples

Login to remote host

ssh -l jsmith remotehost.example.com

Debug ssh client

ssh -v -l jsmith remotehost.example.com

Display ssh client version

$ ssh -V
OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003

5. sed command examples

When you copy a DOS file to Unix, you could find rn in the end of each line. This example converts the DOS file format to Unix file format using sed command.

$sed 's/.$//' filename

Print file content in reverse order

$ sed -n '1!G;h;$p' thegeekstuff.txt

Add line number for all non-empty-lines in a file

$ sed '/./=' thegeekstuff.txt | sed 'N; s/n/ /'

 6. awk command examples

Remove duplicate lines using awk

$ awk '!($0 in array) { array[$0]; print }' temp

Print all lines from /etc/passwd that has the same uid and gid

$awk -F ':' '$3==$4' passwd.txt

Print only specific field from a file.

$ awk '{print $2,$5;}' employee.txt

 7. vim command examples

Go to the 143rd line of file

$ vim +143 filename.txt

Go to the first match of the specified

$ vim +/search-term filename.txt

Open the file in read only mode.

$ vim -R /etc/passwd

8. diff command examples

Ignore white space while comparing.

# diff -w name_list.txt name_list_new.txt

2c2,3
< John Doe --- > John M Doe
> Jason Bourne

9. sort command examples

Sort a file in ascending order

$ sort names.txt

Sort a file in descending order

$ sort -r names.txt

Sort passwd file by 3rd field.

$ sort -t: -k 3n /etc/passwd | more

10. export command examples

To view oracle related environment variables.

$ export | grep ORACLE
declare -x ORACLE_BASE="/u01/app/oracle"
declare -x ORACLE_HOME="/u01/app/oracle/product/10.2.0"
declare -x ORACLE_SID="med"
declare -x ORACLE_TERM="xterm"

To export an environment variable:

$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0

11. xargs command examples

Copy all images to external hard-drive

# ls *.jpg | xargs -n1 -i cp {} /external-hard-drive/directory

Search all jpg images in the system and archive it.

# find / -name *.jpg -type f -print | xargs tar -cvzf images.tar.gz

Download all the URLs mentioned in the url-list.txt file

# cat url-list.txt | xargs wget –c

12. ls command examples

Display filesize in human readable format (e.g. KB, MB etc.,)

$ ls -lh
-rw-r----- 1 ramesh team-dev 8.9M Jun 12 15:27 arch-linux.txt.gz

Order Files Based on Last Modified Time (In Reverse Order) Using ls -ltr

$ ls -ltr

Visual Classification of Files With Special Characters Using ls -F

$ ls -F

13. pwd command

pwd is Print working directory. What else can be said about the good old pwd who has been printing the current directory name for ages.

14. cd command examples

Use “cd -” to toggle between the last two directories

Use “shopt -s cdspell” to automatically correct mistyped directory names on cd

15. gzip command examples

To create a *.gz compressed file:

$ gzip test.txt

To uncompress a *.gz file:

$ gzip -d test.txt.gz

Display compression ratio of the compressed file using gzip -l

$ gzip -l *.gz
         compressed        uncompressed  ratio uncompressed_name
              23709               97975  75.8% asp-patch-rpms.txt

16. bzip2 command examples

To create a *.bz2 compressed file:

$ bzip2 test.txt

To uncompress a *.bz2 file:

bzip2 -d test.txt.bz2

17. unzip command examples

To extract a *.zip compressed file:

$ unzip test.zip

View the contents of *.zip file (Without unzipping it):

$ unzip -l jasper.zip
Archive:  jasper.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
    40995  11-30-98 23:50   META-INF/MANIFEST.MF
    32169  08-25-98 21:07   classes_
    15964  08-25-98 21:07   classes_names
    10542  08-25-98 21:07   classes_ncomp

18. shutdown command examples

Shutdown the system and turn the power off immediately.

# shutdown -h now

Shutdown the system after 10 minutes.

# shutdown -h +10

Reboot the system using shutdown command.

# shutdown -r now

Force the filesystem check during reboot.

# shutdown -Fr now

19. ftp command examples

Both ftp and secure ftp (sftp) has similar commands. To connect to a remote server and download multiple files, do the following.

$ ftp IP/hostname
ftp> mget *.html

To view the file names located on the remote server before downloading, mls ftp command as shown below.

ftp> mls *.html -
/ftptest/features.html
/ftptest/index.html
/ftptest/othertools.html
/ftptest/samplereport.html
/ftptest/usage.html

20. crontab command examples

View crontab entry for a specific user

# crontab -u john -l

Schedule a cron job every 10 minutes.

*/10 * * * * /home/ramesh/check-disk-space

21. service command examples

Service command is used to run the system V init scripts. i.e Instead of calling the scripts located in the /etc/init.d/ directory with their full path, you can use the service command.

Check the status of a service:

# service ssh status

Check the steatus of all the services.

service --status-all

Restart a service.

# service ssh restart

22. ps command examples

ps command is used to display information about the processes that are running in the system.

While there are lot of arguments that could be passed to a ps command, following are some of the common ones.

To view current running processes.

$ ps -ef | more

To view current running processes in a tree structure. H option stands for process hierarchy.

$ ps -efH | more

23. free command examples

This command is used to display the free, used, swap memory available in the system.

Typical free command output. The output is displayed in bytes.

$ free
             total       used       free     shared    buffers     cached
Mem:       3566408    1580220    1986188          0     203988     902960
-/+ buffers/cache:     473272    3093136
Swap:      4000176          0    4000176

If you want to quickly check how many GB of RAM your system has use the -g option. -b option displays in bytes, -k in kilo bytes, -m in mega bytes.

$ free -g
             total       used       free     shared    buffers     cached
Mem:             3          1          1          0          0          0
-/+ buffers/cache:          0          2
Swap:            3          0          3

If you want to see a total memory ( including the swap), use the -t switch, which will display a total line as shown below.

ramesh@ramesh-laptop:~$ free -t
             total       used       free     shared    buffers     cached
Mem:       3566408    1592148    1974260          0     204260     912556
-/+ buffers/cache:     475332    3091076
Swap:      4000176          0    4000176
Total:     7566584    1592148    5974436

24. top command examples

top command displays the top processes in the system ( by default sorted by cpu usage ). To sort top output by any column, Press O (upper-case O) , which will display all the possible columns that you can sort by as shown below.

Current Sort Field:  P  for window 1:Def
Select sort field via field letter, type any other key to return

  a: PID        = Process Id              v: nDRT       = Dirty Pages count
  d: UID        = User Id                 y: WCHAN      = Sleeping in Function
  e: USER       = User Name               z: Flags      = Task Flags
  ........

To displays only the processes that belong to a particular user use -u option. The following will show only the top processes that belongs to oracle user.

$ top -u oracle

25. df command examples

Displays the file system disk space usage. By default df -k displays output in bytes.

$ df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             29530400   3233104  24797232  12% /
/dev/sda2            120367992  50171596  64082060  44% /home

df -h displays output in human readable form. i.e size will be displayed in GB’s.

ramesh@ramesh-laptop:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              29G  3.1G   24G  12% /
/dev/sda2             115G   48G   62G  44% /home

Use -T option to display what type of file system.

ramesh@ramesh-laptop:~$ df -T
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/sda1     ext4    29530400   3233120  24797216  12% /
/dev/sda2     ext4   120367992  50171596  64082060  44% /home

26. kill command examples

Use kill command to terminate a process. First get the process id using ps -ef command, then use kill -9 to kill the running Linux process as shown below. You can also use killall, pkill, xkill to terminate a unix process.

$ ps -ef | grep vim
ramesh    7243  7222  9 22:43 pts/2    00:00:00 vim

$ kill -9 7243

27. rm command examples

Get confirmation before removing the file.

$ rm -i filename.txt

It is very useful while giving shell metacharacters in the file name argument.

Print the filename and get confirmation before removing the file.

$ rm -i file*

Following example recursively removes all files and directories under the example directory. This also removes the example directory itself.

$ rm -r example

28. cp command examples

Copy file1 to file2 preserving the mode, ownership and timestamp.

$ cp -p file1 file2

Copy file1 to file2. if file2 exists prompt for confirmation before overwritting it.

$ cp -i file1 file2

29. mv command examples

Rename file1 to file2. if file2 exists prompt for confirmation before overwritting it.

$ mv -i file1 file2

Note: mv -f is just the opposite, which will overwrite file2 without prompting.

mv -v will print what is happening during file rename, which is useful while specifying shell metacharacters in the file name argument.

$ mv -v file1 file2

30. cat command examples

You can view multiple files at the same time. Following example prints the content of file1 followed by file2 to stdout.

$ cat file1 file2

While displaying the file, following cat -n command will prepend the line number to each line of the output.

$ cat -n /etc/logrotate.conf
    1	/var/log/btmp {
    2	    missingok
    3	    monthly
    4	    create 0660 root utmp
    5	    rotate 1
    6	}

31. mount command examples

To mount a file system, you should first create a directory and mount it as shown below.

# mkdir /u01

# mount /dev/sdb1 /u01

You can also add this to the fstab for automatic mounting. i.e Anytime system is restarted, the filesystem will be mounted.

/dev/sdb1 /u01 ext2 defaults 0 2

32. chmod command examples

chmod command is used to change the permissions for a file or directory.

Give full access to user and group (i.e read, write and execute ) on a specific file.

$ chmod ug+rwx file.txt

Revoke all access for the group (i.e read, write and execute ) on a specific file.

$ chmod g-rwx file.txt

Apply the file permissions recursively to all the files in the sub-directories.

$ chmod -R ug+rwx file.txt

33. chown command examples

chown command is used to change the owner and group of a file.

To change owner to oracle and group to db on a file. i.e Change both owner and group at the same time.

$ chown oracle:dba dbora.sh

Use -R to change the ownership recursively.

$ chown -R oracle:dba /home/oracle

34. passwd command examples

Change your password from command line using passwd. This will prompt for the old password followed by the new password.

$ passwd

Super user can use passwd command to reset others password. This will not prompt for current password of the user.

# passwd USERNAME

Remove password for a specific user. Root user can disable password for a specific user. Once the password is disabled, the user can login without entering the password.

# passwd -d USERNAME

35. mkdir command examples

Following example creates a directory called temp under your home directory.

$ mkdir ~/temp

Create nested directories using one mkdir command. If any of these directories exist already, it will not display any error. If any of these directories doesn’t exist, it will create them.

$ mkdir -p dir1/dir2/dir3/dir4/

36. ifconfig command examples

Use ifconfig command to view or configure a network interface on the Linux system.

View all the interfaces along with status.

$ ifconfig -a

Start or stop a specific interface using up and down command as shown below.

$ ifconfig eth0 up

$ ifconfig eth0 down

37. uname command examples

Uname command displays important information about the system such as — Kernel name, Host name, Kernel release number,
Processor type, etc.,

Sample uname output from a Ubuntu laptop is shown below.

$ uname -a
Linux john-laptop 2.6.32-24-generic #41-Ubuntu SMP Thu Aug 19 01:12:52 UTC 2010 i686 GNU/Linux

38. whereis command examples

When you want to find out where a specific Unix command exists (for example, where does ls command exists?), you can execute the following command.

$ whereis ls
ls: /bin/ls /usr/share/man/man1/ls.1.gz /usr/share/man/man1p/ls.1p.gz

When you want to search an executable from a path other than the whereis default path, you can use -B option and give path as argument to it. This searches for the executable lsmk in the /tmp directory, and displays it, if it is available.

$ whereis -u -B /tmp -f lsmk
lsmk: /tmp/lsmk

39. whatis command examples

Whatis command displays a single line description about a command.

$ whatis ls
ls		(1)  - list directory contents

$ whatis ifconfig
ifconfig (8)         - configure a network interface

40. locate command examples

Using locate command you can quickly search for the location of a specific file (or group of files). Locate command uses the database created by updatedb.

The example below shows all files in the system that contains the word crontab in it.

$ locate crontab
/etc/anacrontab
/etc/crontab
/usr/bin/crontab
/usr/share/doc/cron/examples/crontab2english.pl.gz
/usr/share/man/man1/crontab.1.gz
/usr/share/man/man5/anacrontab.5.gz
/usr/share/man/man5/crontab.5.gz
/usr/share/vim/vim72/syntax/crontab.vim

41. man command examples

Display the man page of a specific command.

$ man crontab

When a man page for a command is located under more than one section, you can view the man page for that command from a specific section as shown below.

$ man SECTION-NUMBER commandname

Following 8 sections are available in the man page.

  1. General commands
  2. System calls
  3. C library functions
  4. Special files (usually devices, those found in /dev) and drivers
  5. File formats and conventions
  6. Games and screensavers
  7. Miscellaneous
  8. System administration commands and daemons

For example, when you do whatis crontab, you’ll notice that crontab has two man pages (section 1 and section 5). To view section 5 of crontab man page, do the following.

$ whatis crontab
crontab (1)          - maintain crontab files for individual users (V3)
crontab (5)          - tables for driving cron

$ man 5 crontab

42. tail command examples

Print the last 10 lines of a file by default.

$ tail filename.txt

Print N number of lines from the file named filename.txt

$ tail -n N filename.txt

View the content of the file in real time using tail -f. This is useful to view the log files, that keeps growing. The command can be terminated using CTRL-C.

$ tail -f log-file

43. less command examples

less is very efficient while viewing huge log files, as it doesn’t need to load the full file while opening.

$ less huge-log-file.log

One you open a file using less command, following two keys are very helpful.

CTRL+F – forward one window
CTRL+B – backward one window

44. su command examples

Switch to a different user account using su command. Super user can switch to any other user without entering their password.

$ su - USERNAME

Execute a single command from a different account name. In the following example, john can execute the ls command as raj username. Once the command is executed, it will come back to john’s account.

[john@dev-server]$ su - raj -c 'ls'

[john@dev-server]$

Login to a specified user account, and execute the specified shell instead of the default shell.

$ su -s 'SHELLNAME' USERNAME

45. mysql command examples

mysql is probably the most widely used open source database on Linux. Even if you don’t run a mysql database on your server, you might end-up using the mysql command ( client ) to connect to a mysql database running on the remote server.

To connect to a remote mysql database. This will prompt for a password.

$ mysql -u root -p -h 192.168.1.2

To connect to a local mysql database.

$ mysql -u root -p

If you want to specify the mysql root password in the command line itself, enter it immediately after -p (without any space).

46. yum command examples

To install apache using yum.

$ yum install httpd

To upgrade apache using yum.

$ yum update httpd

To uninstall/remove apache using yum.

$ yum remove httpd

47. rpm command examples

To install apache using rpm.

# rpm -ivh httpd-2.2.3-22.0.1.el5.i386.rpm

To upgrade apache using rpm.

# rpm -uvh httpd-2.2.3-22.0.1.el5.i386.rpm

To uninstall/remove apache using rpm.

# rpm -ev httpd

48. ping command examples

Ping a remote host by sending only 5 packets.

$ ping -c 5 gmail.com

49. date command examples

Set the system date:

# date -s "01/31/2010 23:59:53"

Once you’ve changed the system date, you should syncronize the hardware clock with the system date as shown below.

# hwclock –systohc

# hwclock --systohc –utc

50. wget command examples

The quick and effective method to download software, music, video from internet is using wget command.

$ wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.1.tar.gz

Download and store it with a different name.

$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701