Monday, 25 May 2015

Cut Command in Linux - Extract Fields and Columns from a file

    In some of my recent articles on text processing, I have explained the use of sed command in Linux/Unix. In case of sed command, we provide an input file to the command, it reads the file line-by-line, processes each line and then prints it on the STDOUT. So, in brief, its a row-wise operation. Similar is the case with cut command - there is an input file, there is processing part and the processed output can be displayed on STDOUT or saved in a file. A minor difference between sed and cut is that, cut command processes the file in vertical manner. So, the outcome of the cut command is a single or multiple columns.


    As of now, just remember that, cut command is just a filter, that processes the file and extracts columns from it. Basically, using cut command, we can process a file in order to extract - either a column of characters or some fields. Thus, to achieve more clarity about cut command, we would study it in two parts.

Here we go!

A. Extracting Column of Characters

To begin with, consider a file cuttest.txt with contents as below:

$ cat cuttest.txt
This is line #1
It is line #2
That is line #3
While, this is line #4
It's line #5
I am line #6
Myself line #7
It's me, line #8
Hello, I am line #9
Last line, line #10
Now, just have a look at the basic syntax of the cut command, to extract column(s) of characters from a file:

cut -c [RANGE] [FILENAME]
To explain this briefly, we are instructing cut command to select on the specific characters specified by RANGE from the file FILENAME.

1. Display a Column of Characters

To begin with, lets display the fourth character from each line of the file cuttest.txt.

Example:

$ cat -c 4 cuttest.txt
s
i
t
l
s
m
e
s
l
t
This does make sense!

2. Display a Group of Columns of Characters

In order to extract a group of columns, we need to specify a range - Start and End, to the cut command. To try with, lets display first five characters of each line of the file.

Example:

$ cut -c 1-5 cuttest.txt
This 
It is 
That 
While
It's 
I am 
Mysel
It's 
Hello
Last 
Conclusion is - a whitespace is also considered as a character.

Another variant of this case is, when you want to start from a particular column and display till the last one. As an example, we will start displaying from the 6th column will the end. So, in this case, we would mention start of the range as '6' and we do not mention any end. Thus, it will print everything after the 6th column.

Example:

$ cut -c 6- cuttest.txt
is line #1
 line #2
is line #3
, this is line #4
line #5
line #6
f line #7
me, line #8
, I am line #9
line, line #10
Similarly, to get first 6 characters from the beginning of each line, we would have an example as follows:

$ cut -c -6 cuttest.txt
This i
It is 
That i
While,
It's l
I am l
Myself 
It's m
Hello,
Last l
Now, there might be a curiosity that, what if I don't mention the start and the end of the range. Let's see what happens-

Example:

$ cut -c - cuttest.txt
cut: invalid range with no endpoint: -
Those who thought that entire columns will be printed, are proved to be wrong. Conclusion is - There has to be a valid range.

B. Extracting Field from a File


In order to understand this usage of cut command, lets consider a csv file as follows:

$ cat employees.txt
Employee ID, Employee Name, Age, Gender, Department, Salary
101, John Davies, 35, M, Finance, $4000
102, Mary Fernandes, 29, F, Human Resources, $3000
103, Jacob Williams, 40, M, Sales, $4700
104, Sean Anderson, 25, M, Production, $2700
105, Nick Jones, 42, M, Finance, $7500
106, Diana Richardson, 29, F, Finance, $3200
Remember, in order to extract a field from a file, we would need a delimiter (i.e. a column separator), based on which the file will be divided into columns and we can extract any of them. In this case, the syntax would be-

cut -d [DELIMITER] -f [RANGE] [FILENAME]
Here, we are instructing cut command to use a particular delimiter with option -d and then extract certain fields using option -f.

1. Display a specific field from a file

In case of a csv file, it is crystal clear that our delimiter will be a comma (,). Now, we need to enlist the names of the employees working in our organization, i.e. field number 2.

Example:

$ cut -d ',' -f 2 employees.txt
Employee Name
John Davies
Mary Fernandes
Jacob Williams
Sean Anderson
Nick Jones
Diana Richardson
Looks good.

2. Displaying Multiple Fields from a File

Moving forward now, lets display more than one field now. Suppose, we need to include 'Age' and 'Gender' fields also. For this, we must specify the range - again, a start and an end.

$ cut -d ',' -f 2-4 employees.txt
Employee Name, Age, Gender
John Davies, 35, M
Mary Fernandes, 29, F
Jacob Williams, 40, M
Sean Anderson, 25, M
Nick Jones, 42, M
Diana Richardson, 29, F
Conclusion, in this case, is that, Input Delimiter = Output Delimiter.

Lets have a look at a variant in this case. Suppose, we need to extract 'Employee ID', 'Employee Name', 'Department' and 'Salary'. In that case, we need to specify two ranges as below:

Example:

$ cut -d ',' -f 1-2,5-6 employees.txt
Employee ID, Employee Name, Department, Salary
101, John Davies, Finance, $4000
102, Mary Fernandes, Human Resources, $3000
103, Jacob Williams, Sales, $4700
104, Sean Anderson, Production, $2700
105, Nick Jones, Finance, $7500
106, Diana Richardson, Finance, $3200
This is just awesome!

3. Change the Delimiter in the Output

As we just saw in one of the examples above, by default, Input Delimiter = Output Delimiter. What if I wish to change the output delimiter? Just have a look at the example below:

Example:

$ cut -d ',' -f 2-4 --output-delimiter='|' employees.txt
Employee Name| Age| Gender
John Davies| 35| M
Mary Fernandes| 29| F
Jacob Williams| 40| M
Sean Anderson| 25| M
Nick Jones| 42| M
Diana Richardson| 29| F

4. Do not Display Certain Columns


Just like above example, if we use --complement as an option, cut command will display all the fields, but the specified field.

Example:

$ cut -d ',' --complement -f 6 employees.txt
Employee ID, Employee Name, Age, Gender, Department
101, John Davies, 35, M, Finance
102, Mary Fernandes, 29, F, Human Resources
103, Jacob Williams, 40, M, Sales
104, Sean Anderson, 25, M, Production
105, Nick Jones, 42, M, Finance
106, Diana Richardson, 29, F, Finance
That's all for this tutorial. Please do let me know about your views about this article in the comment section below, and stay tuned for more awesome articles.

Monday, 18 May 2015

How To: Install/Upgrade to Linux Kernel 4.0.4 in Ubuntu/Linux Mint Systems

    The Linux Kernel 4.0.4 is now available for the users, announced Linus Torvalds. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 4.0.4 in your Ubuntu or Linux Mint system.


Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.4-wily/linux-headers-4.0.4-040004_4.0.4-040004.201505171336_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.4-wily/linux-headers-4.0.4-040004-generic_4.0.4-040004.201505171336_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.4-wily/linux-image-4.0.4-040004-generic_4.0.4-040004.201505171336_i386.deb
Install them.

$ sudo dpkg -i linux-headers-4.0.4*.deb linux-image-4.0.4*.deb
Reboot the system.

sudo reboot

For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.4-wily/linux-headers-4.0.4-040004_4.0.4-040004.201505171336_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.4-wily/linux-headers-4.0.4-040004-generic_4.0.4-040004.201505171336_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.4-wily/linux-image-4.0.4-040004-generic_4.0.4-040004.201505171336_amd64.deb
Install them.

$ sudo dpkg -i linux-headers-4.0.4*.deb linux-image-4.0.4*.deb
Reboot the system.

sudo reboot

To uninstall,

sudo apt-get remove 'linux-headers-4.0.4*' 'linux-image-4.0.4*'

Friday, 15 May 2015

How To: Install/Upgrade to Linux Kernel 3.19.8 in Ubuntu/Linux Mint Systems

    The Linux Kernel 3.19.8 is now available for the users and with this, kernel 3.19 will go end-of-life. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 3.19.8 in your Ubuntu or Linux Mint system.


Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-headers-3.19.8-031908_3.19.8-031908.201505110938_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-headers-3.19.8-031908-generic_3.19.8-031908.201505110938_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-image-3.19.8-031908-generic_3.19.8-031908.201505110938_i386.deb
Install them.

$ sudo dpkg -i linux-headers-3.19.8*.deb linux-image-3.19.8*.deb
Reboot the system.

sudo reboot

For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-headers-3.19.8-031908_3.19.8-031908.201505110938_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-headers-3.19.8-031908-generic_3.19.8-031908.201505110938_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-image-3.19.8-031908-generic_3.19.8-031908.201505110938_amd64.deb
Install them.

$ sudo dpkg -i linux-headers-3.19.8*.deb linux-image-3.19.8*.deb
Reboot the system.

sudo reboot

To uninstall,

sudo apt-get remove 'linux-headers-3.19.8*' 'linux-image-3.19.8*'

Thursday, 14 May 2015

How To: Install/Upgrade to Linux Kernel 4.0.3 in Ubuntu/Linux Mint Systems

    The Linux Kernel 4.0.3 is now available for the users, announced Linus Torvalds. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 4.0.3 in your Ubuntu or Linux Mint system.


Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.3-wily/linux-headers-4.0.3-040003_4.0.3-040003.201505131441_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.3-wily/linux-headers-4.0.3-040003-generic_4.0.3-040003.201505131441_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.3-wily/linux-image-4.0.3-040003-generic_4.0.3-040003.201505131441_i386.deb
Install them.

$ sudo dpkg -i linux-headers-4.0.3*.deb linux-image-4.0.3*.deb
Reboot the system.

sudo reboot

For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.3-wily/linux-headers-4.0.3-040003_4.0.3-040003.201505131441_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.3-wily/linux-headers-4.0.3-040003-generic_4.0.3-040003.201505131441_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.3-wily/linux-image-4.0.3-040003-generic_4.0.3-040003.201505131441_amd64.deb
Install them.

$ sudo dpkg -i linux-headers-4.0.3*.deb linux-image-4.0.3*.deb
Reboot the system.

sudo reboot

To uninstall,

sudo apt-get remove 'linux-headers-4.0.3*' 'linux-image-4.0.3*'

Saturday, 9 May 2015

Introduction to Samba - Share Files and Directories between Linux, Windows and Mac

   Samba is the most popular and efficient way with which you can share your files and directories between Linux, Windows and Mac. You just have to create a Samba user, Decide which file/directory you wish to share and Set the permissions, in order to create a Samba share. Well, initially, this might seem to be difficult- especially for beginners, but when you understand all the configuration options, it will seem to be as simple as anything.


This tutorial will explain how to share files and directories on Linux with Windows and Mac clients.

Step 1: Installing Samba

For Red Hat Linux and derivatives, Samba can be installed using yum as follows:

$ yum install samba smbfs
For Debian and derivatives, you can use apt-get as follows (Know more about apt-get here):

$ sudo apt-get install samba smbfs

Step 2 : Samba Configuration

    The best part in samba is, there is a single configuration file - /etc/samba/smb.conf. So, before making any configurations, it's always a better practice to have this file backed up, so that, if anything goes wrong, we can revert the changes back to original ones pretty easily.

$ cp /etc/samba/smb.conf /etc/samba/smb.conf.bak
Now that, we can move forward to edit this file. Let's do it.

$ vi /etc/samba/smb.conf
    When you open the smb.conf file, you'll typically observe three sections here - [global], [share], [homes] and [printers]. Let's now see the significance of each of them:
  • [global]: As the name denotes, setting defined here will be common to all the subsequent sections.
  • [share]: Settings related to the shares are defined here.
  • [homes]: If this section is present and when a user tries to connect to share_name which is not configured in smb.conf, then it is assumed that share_name is a Linux user and it is trying to connect to his home directory.
  • [printers]: Settings for shared printers appear here.
Here is an example of /etc/samba/smb.conf file:

[global]
    workgroup = MYGROUP
    security = user
    passdb backend = tdbsam
    cups options = raw
    load printers = yes
    log file = /var/log/samba.log.%m

[homes]
    comment = Home Directories
    browseable = no
    writable = yes

[share_name]
    comment = SOME_COMMENT_HERE
    path = /path/to/share
    writable = yes
    browseable = yes
    public = no
    ...
    SOME MORE PERMISSIONS HERE
    ...

[printers]
    comment = All Printers
    path = /var/spool/samba
    guest ok = no
    printable = yes
    browseable = no
Above is just an example of /etc/samba/smb.conf configuration. It might vary from system-to-system, but the syntax would be the same. The values in the colored format can be adjusted to meet your requirements. Let's now understand the configuration part in more detail:

  • security: Here, we will specify the mode of authentication Samba should use to authenticate a user. It can be any of the following:
    • user: It needs to create a Samba user and this samba user must also be the Linux user.
    • domain: A user account present in a centralized account repository shared between domain controllers.
    • ads: Active directory security mode.
  • writable: To set Write permission for the share.
  • read only: To set Read only permission for the share.
  • browsable: To set browsing permissions for the share.
  • read list: Users to be given read only permissions to the share.
  • write list: Users to be given write permissions to the share.
  • valid users: Users allowed to access the share.
  • invalid users: Users to be prevented from accessing the shares.
  • hosts allow: IP addresses allowed to access the share.
  • create mask: Permissions to be given when users create anything inside the share.

Step 3. Creating a Samba User

The most important thing to be remembered is - A Samba user must also be the actual system user. So, in order to create a new Samba user, you must ensure that, that user must be a system user. If it is not, then add that user to the system first.

So, in case the user is the system user and we wish to create a samba user, use the smbpasswd command as follows:

$ smbpasswd -a USERNAME
You will be asked for a Samba Password, when you're done, it will create a Samba user for you.

Example:

Here, I need to create a share with name myshare and it must be accessed by the user mandar only. For that, I must create a system user first.

$ useradd mandar
$ passwd mandar
Then, I must create a Samba user.

$ smbpasswd -a mandar
Time to make some configurations.

$ vi /etc/samba/smb.conf

[global]
    ...
    ...
    ...

[myshare]
    comment = This is Mandar's Share
    path = /smbshare/mandar
    valid users = mandar
    hosts allow = 192.168.100.3
    writable = yes
    public = no

[printers]
    ...
    ...
When we make any configuration changes, we must restart the services.

$ /etc/init.d/smb restart

OR

$ service smb restart
And we're done. If we try to access this share from the system 192.168.100.3 with the user mandar, it should work. In order to access this share from my windows machine, I would execute in the 'RUN' box - \\[samba-server-ip]\myshare.

Friday, 8 May 2015

How To: Install/Upgrade to Linux Kernel 3.19.7 in Ubuntu/Linux Mint Systems

    The Linux Kernel 3.19.7 is now available for the users, announced Linus Torvalds. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 3.19.7 in your Ubuntu or Linux Mint system.



Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.7-vivid/linux-headers-3.19.7-031907_3.19.7-031907.201505062136_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.7-vivid/linux-headers-3.19.7-031907-generic_3.19.7-031907.201505062136_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.7-vivid/linux-image-3.19.7-031907-generic_3.19.7-031907.201505062136_i386.deb
Install them.

$ sudo dpkg -i linux-headers-3.19.7*.deb linux-image-3.19.7*.deb
Reboot the system.

sudo reboot

For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.7-vivid/linux-headers-3.19.7-031907_3.19.7-031907.201505062136_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.7-vivid/linux-headers-3.19.7-031907-generic_3.19.7-031907.201505062136_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.7-vivid/linux-image-3.19.7-031907-generic_3.19.7-031907.201505062136_amd64.deb
Install them.

$ sudo dpkg -i linux-headers-3.19.7*.deb linux-image-3.19.7*.deb
Reboot the system.

sudo reboot

To uninstall,

sudo apt-get remove 'linux-headers-3.19.7*' 'linux-image-3.19.7*'

How To: Install/Upgrade to Linux Kernel 4.0.2 in Ubuntu/Linux Mint Systems

    The Linux Kernel 4.0.2 is now available for the users, announced Linus Torvalds. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 4.0.2 in your Ubuntu or Linux Mint system.


Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.2-wily/linux-headers-4.0.2-040002_4.0.2-040002.201505081529_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.2-wily/linux-headers-4.0.2-040002-generic_4.0.2-040002.201505081529_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.2-wily/linux-image-4.0.2-040002-generic_4.0.2-040002.201505081529_i386.deb
Install them.

$ sudo dpkg -i linux-headers-4.0.2*.deb linux-image-4.0.2*.deb
Reboot the system.

sudo reboot

For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.2-wily/linux-headers-4.0.2-040002_4.0.2-040002.201505081529_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.2-wily/linux-headers-4.0.2-040002-generic_4.0.2-040002.201505081529_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.2-wily/linux-image-4.0.2-040002-generic_4.0.2-040002.201505081529_amd64.deb
Install them.

$ sudo dpkg -i linux-headers-4.0.2*.deb linux-image-4.0.2*.deb
Reboot the system.

sudo reboot

To uninstall,

sudo apt-get remove 'linux-headers-4.0.2*' 'linux-image-4.0.2*'

Wednesday, 6 May 2015

How To : Install/Upgrade to Linux Kernel 4.1-RC2 in Ubuntu/Linux Mint Systems

    The Linux Kernel 4.1-RC2 is now available for the users, announced Linus Torvalds. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 4.1-RC2 in your Ubuntu or Linux Mint system.

Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc2-vivid/linux-headers-4.1.0-040100rc2_4.1.0-040100rc2.201505032335_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc2-vivid/linux-headers-4.1.0-040100rc2-generic_4.1.0-040100rc2.201505032335_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc2-vivid/linux-image-4.1.0-040100rc2-generic_4.1.0-040100rc2.201505032335_i386.deb
Install them.

$ sudo dpkg -i linux-headers-4.1*.deb linux-image-4.1*.deb
Reboot the system.

sudo reboot

For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc2-vivid/linux-headers-4.1.0-040100rc2_4.1.0-040100rc2.201505032335_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc2-vivid/linux-headers-4.1.0-040100rc2-generic_4.1.0-040100rc2.201505032335_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc2-vivid/linux-image-4.1.0-040100rc2-generic_4.1.0-040100rc2.201505032335_amd64.deb
Install them.

$ sudo dpkg -i linux-headers-4.1*.deb linux-image-4.1*.deb
Reboot the system.

sudo reboot

To uninstall,

sudo apt-get remove 'linux-headers-4.1*' 'linux-image-4.1*'

Friday, 1 May 2015

Sed Command in Linux - Search and Replace Text in a File

This is the fourth article of the "Super sed' Series", in which we will learn how to search for a pattern and replace it with other pattern in a file, using regular expressions. In previous articles on sed command, we've learned how to print lines in a file, delete lines from a file and insert/append lines to a file.



Before we directly jump to the main content, every learner should know what sed is. Here is the brief introduction of the Super sed:
  • sed stand for Stream EDitor and it being based on the ed editor, it borrows most of the commands from the ed. It was developed by Lee E. McMahon of Bell Labs.
  • sed offers large range of text transformations that include printing lines, deleting lines, editing line in-place, search and replace, appending and inserting lines, etc.
  • sed is useful whenever you need to perform common editing operations on multiple lines without using 'vi' editor.
  • Whenever sed is executed on an input file or on the contents from stdin, sed reads the file line-by-line and after removing the trailing newline, places it in the "Pattern space", where the commands are executed on them after conditions (as in case of regex matching) are verified, and then printed on the stdout.

Search and Replace a Pattern using sed

Before we start, just remember two points:
  1. sed "s" command searches for a particular pattern in the file and if the pattern is matched, it is replaced with another pattern mentioned in the command. (We will see the syntax a bit later, just remember that, there is a search pattern and a replace pattern).
  2. When ^ means beginning of the line and $ denotes end of the line, ^$ makes an "Empty Line", very useful while deleting empty lines from a file.
  3. sed with option -i will edit the file in place, i.e. unless you use the option -i, the changes will not be written to the file.
  4. sed with option -n will suppress automatic printing of pattern buffer/space. So, we would want to use this option (How and why? Check this article).
For our better understanding, let us have a file sedtest.txt with contents as follows:

$ cat sedtest.txt
This is line #1
It is line #2
That is line #3
While, this is line #4
It's line #5
I am line #6
Myself line #7
It's me, line #8
Hello, I am line #9
Last line, line #10

Syntax for Search and Replace:

The syntax used by sed for 'search and replace' might seem to be a bit confusing initially, but its pretty straight-forward once it's completely understood.

sed 's/SEARCH/REPLACE/OPTIONS' FILE.txt
Let's get deeper in above syntax,
  • s - As mentioned earlier, s is used to search for a pattern.
  • / - It's a delimiter. There are some alternatives for / to be used as a delimiter. A few of them are # % @ :.
  • SEARCH - What are you searching for? You would be mentioning the search pattern here.
  • REPLACE - I've got the SEARCH pattern, what should I replace it with? You would be mentioning the replace pattern here.
  • OPTIONS - In order to make your work more comfortable, s commands comes with following options:
  • i : If you use this option, your search for the SEARCH pattern would be case-insensitive.
    • g : It can be used to replace ALL the SEARCH instances with REPLACE pattern.
    • n : Unlike g, it would replace only the 'n'th occurrence of the pattern.
    • w : Just I've mentioned, in case you wish to edit the original file in place, you will have to use option -i. But, in case you want to write the output to other file, keeping the original file intact, you can use option w. Of course, redirections (> or >>) could have been used instead.
    • p : When this option is used, it would display only those lines wherein replacements are made.
Let us now begin with the practical examples.

1. sed s command with no option specified

When we do not mention any of the options (g, n or w), only the first match of each line is replaced. Have a look at below example, in which we will replace the letter i with X. Here, SEARCH will be letter 'i', REPLACE will be letter 'X' and we do not mention any option therein.

Example:

$ sed 's/i/X/' sedtest.txt
ThXs is line #1
It Xs line #2
That Xs line #3
WhXle, this is line #4
It's lXne #5
I am lXne #6
Myself lXne #7
It's me, lXne #8
Hello, I am lXne #9
Last lXne, line #10
So, from above output, we can conclude that only the first occurrence of letter 'i' is replaced, while letter 'X' along with non-first occurrences of letter 'i' remained unchanged.

2. Replace 'n'th occurrence of a word/string

Consider that, in above file sedtest.txt, we need to replace the 2nd occurrence of the string is with the string XX. So, in this case, SEARCH = is, REPLACE=XX and this time, we would use an option 2 indicating that, we are interested in the 2nd occurrence only.

Example:

$ sed 's/is/XX/2' sedtest.txt
This XX line #1
It is line #2
That is line #3
While, this XX line #4
It's line #5
I am line #6
Myself line #7
It's me, line #8
Hello, I am line #9
Last line, line #10
If you look at line 1 and 4, it has skipped the first occurrence of the pattern 'is' (included in the word This) and replaced the 2nd one with 'XX'.

3. Replace all the occurrences of a word/string

Now, let us replace all the occurrences of the word 'line' with the word 'sentence'. For this purpose, SEARCH = line, REPLACE=sentence and the option would be g indicating that, we are interested in all the occurrences.

Example:

$ sed 's/line/sentence/g' sedtest.txt
This is sentence #1
It is sentence #2
That is sentence #3
While, this is sentence #4
It's sentence #5
I am sentence #6
Myself sentence #7
It's me, sentence #8
Hello, I am sentence #9
Last sentence, sentence #10
So, as expected, all the lines are replaced by sentences.

4. Display only the altered lines using p

As mentioned in the earlier part, sed s command with p option will print only those lines in which substitutions are made. By default, it will also display the pattern buffer, and to suppress it we would use -n.

Example:
Without -n,

$ sed 's/While/Whereas/gp' sedtest.txt
This is sentence #1
It is sentence #2
That is sentence #3
Whereas, this is sentence #4
Whereas, this is sentence #4
It's sentence #5
I am sentence #6
Myself sentence #7
It's me, sentence #8
Hello, I am sentence #9
Last sentence, sentence #10
Could you find 2 'Whereas's?

With -n,

$ sed -n 's/While/Whereas/gp' sedtest.txt
Whereas, this is sentence #4
So, conclusion is, option p prints the line where replacement has been done.

5. Dump the substitutions into a file

Similar to option p, but where p displays the changes on STDOUT, option g writes the changes to a file. In this case, we need to mention a file name, say changes.txt, where changes are supposed to be written to.

Example:

$ sed -n 's/While/Whereas/gpw changes.txt' sedtest.txt
Whereas, this is sentence #4
$
$ cat changes.txt
Whereas, this is sentence #4
Here, with pw together, we can make sure that, the lines displayed by p are written to the output file.

6. Delete words/strings using sed

Just imagine, what would happen if we use nothing as a REPLACE pattern?

$ sed 's/line//g' sedtest.txt
This is  #1
It is  #2
That is  #3
While, this is  #4
It's  #5
I am  #6
Myself  #7
It's me,  #8
Hello, I am  #9
Last ,  #10
Whoops! The 'line's are gone.

Similarly, instead of deleting an entire string, if we wish to delete last 2 characters of every line, our SEARCH pattern will be ..$, where each period [.] (intentionally used square brackets ;)) matches a letter and $ matches end of the line. So, when they combine together make "Last two characters of a line".

Example:

$ sed 's/..$//g' sedtest.txt
This is line 
It is line 
That is line 
While, this is line 
It's line 
I am line 
Myself line 
It's me, line 
Hello, I am line 
Last line, line 
So, this time, line numbers are gone.

7. Replace only if line contains a pattern

Let's consider that, I wish to replace the word line with statement, only if that line contains the word Myself. Had you read previous articles on sed, you would have done this exercise on your own.

$ sed '/Myself/s/line/statement/g' sedtest.txt
This is line #1
It is line #2
That is line #3
While, this is line #4
It's line #5
I am line #6
Myself statement #7
It's me, line #8
Hello, I am line #9
Last line, line #10 
See statement #7, line is a statement now.

8. Multiple actions in sed command

This time, the scenario is - I have a text file and it's heavily commented. I do not want to read those comments and I want those lines to be deleted. So, we need to remove all the characters in a line that starts with a #, which will give us an empty line. And, in order to delete that line, we would need the d command of sed.

In this case, the SEARCH pattern will be ^#.*, indicating whatever that begins with a # in a line, REPLACE pattern will be empty. We have already seen how to delete empty lines using sed with d command.

Example:
$ sed 's/^#.*//g; /^$/d' FILE.txt
Just try this.

That was all about the fourth article on sed command. More articles on sed are coming soon. So, stay tuned. Of course, do not forget to share your feedback in the comment section below.




Thursday, 30 April 2015

How To: Install/Upgrade to Linux Kernel 3.19.6 in Ubuntu/Linux Mint Systems

    The Linux Kernel 3.19.6 is now available for the users, announced Linus Torvalds. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 3.19.6 in your Ubuntu or Linux Mint system.



Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.6-vivid/linux-headers-3.19.6-031906_3.19.6-031906.201504290535_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.6-vivid/linux-headers-3.19.6-031906-generic_3.19.6-031906.201504290535_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.6-vivid/linux-image-3.19.6-031906-generic_3.19.6-031906.201504290535_i386.deb
Install them.

$ sudo dpkg -i linux-headers-3.19.6*.deb linux-image-3.19.6*.deb
Reboot the system.

sudo reboot

For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.6-vivid/linux-headers-3.19.6-031906_3.19.6-031906.201504290535_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.6-vivid/linux-headers-3.19.6-031906-generic_3.19.6-031906.201504290535_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.6-vivid/linux-image-3.19.6-031906-generic_3.19.6-031906.201504290535_amd64.deb
Install them.

$ sudo dpkg -i linux-headers-3.19.6*.deb linux-image-3.19.6*.deb
Reboot the system.

sudo reboot

To uninstall,

sudo apt-get remove 'linux-headers-3.19.6*' 'linux-image-3.19.6*'

    Total Pageviews