Friday, September 21, 2012

awk

why awk?

awk is small, fast, and simple, unlike, say, perl. awk also has a clean comprehensible C-like input language, unlike, say, perl. And while it can’t do everything you can do in perl, it can do most things that are actually text processing, and it’s much easier to work with.

what do you do?

In its simplest usage awk is meant for processing column-oriented text data, such as tables, presented to it on standard input. The variables $1, $2, and so forth are the contents of the first, second, etc. column of the current input line. For example, to print the second column of a file, you might use the following simple awk script:
        awk < file '{ print $2 }'
This means “on every line, print the second field”.
To print the second and third columns, you might use
        awk < file '{ print $2, $3 }'

Input separator

By default awk splits input lines into fields based on whitespace, that is, spaces and tabs. You can change this by using the -F option to awk and supplying another character. For instance, to print the home directories of all users on the system, you might do
        awk < /etc/passwd -F: '{ print $6 }'
since the password file has fields delimited by colons and the home directory is the 6th field.
Print column1, column5 and column7 of a data file or output of any columns list
$awk ‘{print $1, $5, $7}’ data_file
$cat file_name |awk ‘{print $1 $5 $7}’
$ls –al |awk ‘{print $1, $5, $7}’ — Prints file_permissions,size and date
Syntax of running an awk program
Awk ‘program’ input file(s)
List all files names whose file size greater than zero.
$ls –al |awk ‘$5 > 0 {print $9}’
List all files whose file size equal to 512bytes.
$ls –al |awk ‘$5 == 0 {print $9}’
print all lines
$awk ‘{print }’ file_name
$awk ‘{print 0}’ file_name
Number of lines in a file
$awk ‘ END {print NR}’ file_name
Number of columns in each row of a file
$awk ‘ {print NF’} file_name
Sort the output of file and eliminate duplicate rows
$awk ‘{print $1, $5, $7}’ |sort –u
List all file names whose file size is greater than 512bytes and owner is “oracle”
$ls –al |awk ‘$3 == “oracle” && $5 > 512 {print $9}’
List all file names whose owner could be either “oracle” or “root”
$ls –al |awk ‘$3 == “oracle” || $3 == “root” {print $9}’
list all the files whose owner is not “oracle
$ls –al |awk ‘$3 != “oracle” {print $9}’
List all lines which has atlease one or more characters
$awk ‘NF > 0 {print }’ file_name
List all lines longer that 50 characters
$awk ‘length($0) > 50 ‘{print }’ file_name
List first two columns
$awk ‘{print $1, $2}’ file_name
Swap first two columns of a file and print
$awk ‘{temp = $1; $1 = $2; $2 = temp; print }’ file_name
Replace first column as “ORACLE” in a data file
$awk ‘{$1 = “ORACLE”; print }’ data_file
Remove first column values in a data file
$awk ‘{$1 =”"; print }’ data_file
Calculate total size of a directory in Mb
$ls –al |awk ‘{total +=$5};END {print “Total size: ” total/1024/1024 ” Mb”}’
Calculate total size of a directory including sub directories in Mb
$ls –lR |awk ‘{total +=$5};END {print “Total size: ” total/1024/1024 ” Mb”}’
Find largest file in a directory including sub directories
$ls –lR |awk ‘{print $5 “\t” $9}’ |sort –n |tail -1

SWAP PARTITION

SWAP PARTITION :


UNIX / Linux: 2 Ways to Add Swap Space Using dd, mkswap and swapon

by RAMESH NATARAJAN on AUGUST 18, 2010
Question: I would like to add more swap space to my Linux system. Can you explain with clear examples on how to increase the swap space?
Answer: You can either use a dedicated hard drive partition to add new swap space, or create a swap file on an existing filesystem and use it as swap space.

How much swap space is currently used by the system?

Free command displays the swap space. free -k shows the output in KB.
# free -k
             total       used       free     shared    buffers     cached
Mem:       3082356    2043700    1038656          0      50976    1646268
-/+ buffers/cache:     346456    2735900
Swap:      4192956          0    4192956
Swapon command with option -s, displays the current swap space in KB.
# swapon -s
Filename                        Type            Size    Used    Priority
/dev/sda2                       partition       4192956 0       -1
Swapon -s, is same as the following.
# cat /proc/swaps
Filename                        Type            Size    Used    Priority
/dev/sda2                       partition       4192956 0       -1

Method 1: Use a Hard Drive Partition for Additional Swap Space

If you have an additional hard disk, (or space available in an existing disk), create a partition using fdisk command. Let us assume that this partition is called /dev/sdc1
Now setup this newly created partition as swap area using the mkswap command as shown below.
# mkswap /dev/sdc1
Enable the swap partition for usage using swapon command as shown below.
# swapon /dev/sdc1
To make this swap space partition available even after the reboot, add the following line to the /etc/fstab file.
# cat /etc/fstab
/dev/sdc1               swap                    swap    defaults        0 0
Verify whether the newly created swap area is available for your use.
# swapon -s
Filename                        Type            Size    Used    Priority
/dev/sda2                       partition       4192956 0       -1
/dev/sdc1                       partition       1048568 0       -2

# free -k
             total       used       free     shared    buffers     cached
Mem:       3082356    3022364      59992          0      52056    2646472
-/+ buffers/cache:     323836    2758520
Swap:      5241524          0    5241524
Note: In the output of swapon -s command, the Type column will say “partition” if the swap space is created from a disk partition.

Method 2: Use a File for Additional Swap Space

If you don’t have any additional disks, you can create a file somewhere on your filesystem, and use that file for swap space.
The following dd command example creates a swap file with the name “myswapfile” under /root directory with a size of 1024MB (1GB).
# dd if=/dev/zero of=/root/myswapfile bs=1M count=1024
1024+0 records in
1024+0 records out

# ls -l /root/myswapfile
-rw-r--r--    1 root     root     1073741824 Aug 14 23:47 /root/myswapfile
Change the permission of the swap file so that only root can access it.
# chmod 600 /root/myswapfile
Make this file as a swap file using mkswap command.
# mkswap /root/myswapfile
Setting up swapspace version 1, size = 1073737 kB
Enable the newly created swapfile.
# swapon /root/myswapfile
To make this swap file available as a swap area even after the reboot, add the following line to the /etc/fstab file.
# cat /etc/fstab
/root/myswapfile               swap                    swap    defaults        0 0
Verify whether the newly created swap area is available for your use.
# swapon -s
Filename                        Type            Size    Used    Priority
/dev/sda2                       partition       4192956 0       -1
/root/myswapfile                file            1048568 0       -2

# free -k
             total       used       free     shared    buffers     cached
Mem:       3082356    3022364      59992          0      52056    2646472
-/+ buffers/cache:     323836    2758520
Swap:      5241524          0    5241524
Note: In the output of swapon -s command, the Type column will say “file” if the swap space is created from a swap file.
If you don’t want to reboot to verify whether the system takes all the swap space mentioned in the /etc/fstab, you can do the following, which will disable and enable all the swap partition mentioned in the /etc/fstab
# swapoff -a

# swapon -a

Configuring NIS under Red Hat Linux
=========================================


The following describes a procedure to set up NIS network name service under Red Hat Linux. This is geared

toward a small intallation with only one domain. However, it should be fairly evident how to add more NIS

domains. The NIS domain name has nothing to do with any DNS naming convention being used.

-------------------------------------------------------
In these examples, the following conventions are used:
=======================================================

NIS domain: "internal"
Code or configuration file data: colored
Root prompt on NIS master server:      master#
Root prompt on NIS client host:        client#


==================================
Setting up a NIS master server:

Required packages: yp-tools, ypbind, ypserv, portmap
================================================

Set up "time" service to run via inetd/xinetd, or configure xntpd, or otherwise make sure the host's clock is

synchronized.
--------------------
Edit /etc/yp.conf:
domain internal server ip.of.nis.server
=======================
Edit /etc/ypserv.conf:
dns: no
files: 30
xfr_check_port: yes
* : * : shadow.byname : port
* : * : passwd.adjunct.byname : port
=======================

Edit /etc/sysconfig/network:

NISDOMAIN="internal"
Set NIS domain name:
master# domainname internal
master# ypdomainname internal

Create file /var/yp/securenets:

host 127.0.0.1
255.255.255.0   10.0.0.0

Make sure the "portmap" service is running:

master# service portmap start
master# chkconfig portmap on

Portmap will need a rule in /etc/hosts.allow to allow access from localhost and any hosts that need to access

NIS.
Start ypserv service:

master# service ypserv start

Check that it's listening:

master# rpcinfo -u localhost ypserv

You should see:
program 100004 version 1 ready and waiting
program 100004 version 2 ready and waiting
Initialize the NIS maps:

master# /usr/lib/yp/ypinit -m
=================
Specify local hostname, Ctrl-D, y, let finish.
Start up ypbind, yppasswdd, ypxfrd:
====================
master# service ypbind start
master# service yppasswdd start
master# service ypxfrd start

Set YP services to run on boot-up:
=====================
master# chkconfig ypserv on
master# chkconfig ypbind on
master# chkconfig yppasswdd on
master# chkconfig ypxfrd on

NIS client host setup

Required packages: yp-tools ypbind portmap

Edit /etc/sysconfig/network:
NISDOMAIN=internal

Edit /etc/yp.conf:
domain internal server ip.of.master.server

Edit /etc/hosts:
ip.of.master.server    hostname.domain hostname
Set NIS domain-name:
client# domainname internal
client# ypdomainname internal
Edit /etc/nsswitch.conf:
passwd:     files nis
shadow:     files nis
group:      files nis
Make sure the portmap service is running:
client# service portmap start
client# chkconfig portmap on
The /etc/hosts.allow file will need rules allowing access from localhost and the NIS master server.
Start ypbind service:
client# service ypbind start
client# chkconfig ypbind on
Test it out:
client# rpcinfo -u localhost ypbind
client# ypcat passwd

Booting Process In Linux Step By Step

Step by step explanation of Linux boot sequence

In this topic we will discuss indepth of Linux Boot Sequence.How a linux system boots?This will help administrators in
troubleshooting some bootup problem.Before discussing about  I will notedown the major component we need to know
who are responsible for the booting process.
        1.BIOS(Basic Input/Output System)
        2.MBR(Master Boot Record)
        3.LILO or GRUB
             LILO:-LInux LOader
             GRUB:-GRand Unified Bootloader
        4.Kernel
        5.init
        6.Run Levels

1.BIOS:
      i.When we power on BIOS performs a Power-On Self-Test (POST) for all of the different hardware components in the system to make sure everything is working properly
     ii.Also it checks for whether the computer is being started from an off position (cold boot) or from a restart (warm boot) is
stored at this location.
     iii.Retrieves information from CMOS (Complementary Metal-Oxide Semiconductor) a battery operated memory chip on the motherboard that stores time, date, and critical system information.
     iv.Once BIOS sees everything is fine it will begin searching for an operating system Boot Sector on a valid master boot sector
on all available drives like hard disks,CD-ROM drive etc.
     v.Once BIOS finds a valid MBR it will give the instructions to boot and executes the first 512-byte boot sector that is the first
sector (“Sector 0″) of a partitioned data storage device such as hard disk or CD-ROM etc .
2.MBR
     i. Normally we use multi-level boot loader.Here MBR means I am referencing to DOS MBR.
     ii.Afer BIOS executes a valid DOS MBR,the DOS MBR will search for a valid primary partition marked as bootable on the hard disk.
     iii.If MBR finds a valid bootable primary partition then it executes the first 512-bytes of that partition which is second level MBR.
     iv. In linux we have two types of the above mentioned second level MBR known as LILO and GRUB
3.LILO
     i.LILO is a linux boot loader which is too big to fit into single sector of 512-bytes.
     ii.So it is divided into two parts :an installer and a runtime module.
     iii.The installer module places the runtime module on MBR.The runtime module has the info about all operating systems installed.
     iv.When the runtime module is executed it selects the operating system to load and transfers the control to kernel.
     v.LILO does not understand filesystems and boot images to be loaded and treats them as raw disk offsets
GRUB
     i.GRUB MBR consists of 446 bytes of primary bootloader code and 64 bytes of the partition table.
     ii.GRUB locates all the operating systems installed and gives a GUI to select the operating system need to be loaded.
     iii.Once user selects the operating system GRUB will pass control to the karnel of that operating system.
see below what is the difference between LILO and GRUB
4.Kernel
     i.Once GRUB or LILO transfers the control to Kernel,the Kernels does the following tasks
  • Intitialises devices and loads initrd module
  • mounts root filesystem
5.Init
     i.The kernel, once it is loaded, finds init in sbin(/sbin/init) and executes it.
     ii.Hence the first process which is started in linux is init process.
     iii.This init process reads /etc/inittab file and sets the path, starts swapping, checks the file systems, and so on.
     iv.It runs all the boot scripts(/etc/rc.d/*,/etc/rc.boot/*)
     v.starts the system on specified run level in the file /etc/inittab

6.Runlevel
     i.There are 7 run levels in which the linux OS runs and different run levels serves for different purpose.The descriptions are
given below.
  • 0  – halt
  • 1  – Single user mode
  • 2  – Multiuser, without NFS (The same as 3, if you don’t have networking)
  • 3  – Full multiuser mode
  • 4  – unused
  • 5  – X11
  • 6  – Reboot
     ii.We can set in which runlevel we want to run our operating system by defining it on /etc/inittab file.
Now as per our setting in /etc/inittab the Operating System the operating system boots up and finishes the bootup process.
Below are given some few  important differences about LILO and GRUB
LILO
GRUB
LILO has no interactive command interface GRUB has interactive command interface
LILO does not support booting from a network GRUB does support booting from a network
If you change your LILO config file, you have to rewrite the LILO stage one boot loader to the MBR GRUB automatically detects any change in config file and auto loads the OS
LILO supports only linux operating system GRUB supports large number of OS