Thursday, 2 August 2018

openssl Command in Linux

In the earlier days, when people started to use the Internet, information was being transferred across the globe through HTTP (Hyper Text Transfer Protocol). HTTP was never safe as the information was being transferred in the plain text format and clould easily be available to the hacker who would intercept the communication. This information may consist of sensitive information including credit card details and passwords, which lead to increasing frauds. Hence, it was necessary to pass this information from one corner of the world to another through a secure channel, such that the information is not compromised. Thus, encryption came into the picture and the protocol that uses encryption while using HTTP - Secure Socket Layer (SSL). Since then, it is popularly known as HTTP over SSL or HTTPS.

Why HTTPS?

There are three main reasons:

  • Authenticity - HTTPS ensures that, the data exchange is happening with legitimate user, i.e. it checks the authenticity of the enduser it is connected to. If it is an intruder, it will drop the connection before any data exchange happens.
  • Data privacy - HTTPS encrypts the data such that only the intended enduser can decrypt and read it. Intruders will only be able to capture the encrypted message that is not readable.
  • Data Integrity - HTTPS ensures that the data received at receiver end is the same one which was sent at sender end, i.e. it has not changed/altered in the middle.

How HTTPS works?


  1. Browsers initiates a connection with the server
  2. Servers responds back with a public key (through SSL certificate), while private key is kept with the server itself
  3. Browser generates a random encryption key (session key) and encrypts it with public key
  4. Browser sends the encrypted session key to the server
  5. Server receives encrypted session key and decrypts it using private key, to get original session key
  6. Now, both the browser and the server has the same session key, that they can use to encrypt and decrypt the data
  7. Session key expires when the connection terminates

This article explain tips to generate Private Key, CSR (Certificate signing request) in linux using
OpenSSL command to obtain a Certificate Authority (CA) signed SSL certificate. CSR file used
submit to CA to complete order process of signed SSL certificate. You can get cheap SSL
certificate available at cheapsslshop.com.

Perform following steps to Generate private key, CSR for CA signed certificates in Linux using 'openssl' command

1. Generate a private key


Login to your Linux server and execute the following openssl command

openssl genrsa -des3 -out example.key 2048

Output:

# openssl genrsa -des3 -out example.key 2048
Generating RSA private key, 2048 bit long modulus
...................................................+++
.......+++
e is 65537 (0x10001)
Enter pass phrase for example.key:
Verifying - Enter pass phrase for example.key:

It will create a file example.key which is the private key. Lets look at the contents of this file.

# cat example.key 
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,5B72B5B1A445E6DD

KsIkmOuoagyG/Xs9JhA9qFbqpJjl73CvPWpUyeWUi80e9ZBQbjKJ9gkmKN5IQ/T6
fd/dOf3p5iMlnzjjq+4ST5Zuo/QW2DV1hffvgjTcC3KB0H44oC/Tjox3BfmBS+Km
cLpIKA5n+eysXowrc/uUhbdx1oCNUMUiiP/Cforn+vom+6VgMYbC6674CGPAGdqK
Ffy+NGErM58cUeko1/ax/Cj3ietlkTdCewR2hY4+4uDo3bfZMNMrYklkfy42XlcJ
nI5YzGhDSqB0cE0eblkekw5bbg77cRYdAk9L9onlirWek9ZA+EPZBtZA2S1xeJxA
iIbpg5cHn7oOoefvDkLrv75v3eNrqhP/wg6yFdQoi4edjG94SOpFqBTjueUAdF9p
Q4mA9PheARZ2V3bt4Mi5mgm7tfUwVNqVXwxmU+ErDHyxmpr1GzL2joqOUN0JQMIe
ahrYrcJBY93JK0+i7wlTuHg8ZkGCOCUXdyTbhlDgyfoey4tk9dA37R41N/F5IVPC
/AhZQa2UV7gMWdF18tsAesMpjQgR4OF/5kT03mPxiVNYUEHx4cRQ8JokMC4WlDwV
+ETJyjWToJeWIKAvHD2188MXjh/jMlP5JF/zRpFpEG6xZe2k9Cq8PGn+eM2SCwVH
8tLgQZzdrBL0p6H5ePrnXanVnOIiH09AL07/AuJrpcXNQSTEgUHkVI7wWbb5UbyL
hzG2Ky1X5+G3vgJeKz+UAe5P56qmhfhRHDzQg8FkU60nqCQi5DN1NhN7GeIh5r+n
lzyHEPQZc0lO0vYhXUTT2npCfNgDLvBLOFA8zphGR7hbIhi8t4MvJPM2z7IZuH0k
FPFmkgh15ByF75bLOuLMrOg1C74TbwNkKBihq8nGFtUywDUXcaot2b+xR0/KvULz
qZT/I8HlCUYqK7GfGHpPCGloqFOy/U2+XQ4JS1BR8NCxEun7Kk79KhdJo+d9eean
D62rpAlBdb8cx2j28pUsyhte7sQ9HP3BiujuctxN7XiFDVQsPZ5RpBFwqB+Ou2C1
4nn5AM80JWJGAdAjVnGSLgrPPJkAPOCu9T0sGZFemilismleDG5O7SfL9YXdwuqd
1jOW0W66heIsVrId751L4V+R2LDSztPS4VetwVAulxBvZHgkCJtGlzODZRX2SDob
X/xfwU/uZWoZgfgKM9S2ZIf4iMkIKnMo5tdzBQ3FrezB28BWFUtA5QyPmUWo2Txq
jWLcWuTn/shQl8d8TKGKd3c7TS0Fw9uJXaKW61RWB4clOsbSuwe/o9HDNdVz4uiI
xQkeSo+rGdeVm0ZTxtopOfMMtjWlOfQwp/SCYi947aOIK68wuQYnbViDR/loR2x/
yDb+0dGjpUDYI5m+G/R6XrjjSKlv1DkPgGcjWVoVxBOeTunPXhdab0O+fQIy5J69
LAWjpa5UCTcOY/IYZONy3KLCxy6irvy8xKJQA8LJQYR00s7f8Lz4k0Sgu6aA6E+v
7J9yZ6AfkYv9xzJFlD98nKQxDJImtGLe59/EuEZ/gnJCOHqyagTuyb1DtaI9neoR
FX1N656+PGieUfQvifaBajbNglvbijUqPvhUMs6D5Vpluo8YSOGcyg==
-----END RSA PRIVATE KEY-----

2. Generate a Certificate Signing Request (CSR)


Command:

openssl req -new -out example.csr -key example.key

Output:

# openssl req -new -out example.csr -key example.key 
Enter pass phrase for example.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:MH
Locality Name (eg, city) []:MUMBAI
Organization Name (eg, company) [Internet Widgits Pty Ltd]:EXAMPLE
Organizational Unit Name (eg, section) []:TESTUNIT
Common Name (e.g. server FQDN or YOUR name) []:example.com
Email Address []:user@example.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:      
An optional company name []:EXAMPLE

It will create a file example.csr which is the CSR. Its contents will look like :

-----BEGIN CERTIFICATE REQUEST-----
MIIC5TCCAc0CAQAwgYcxCzAJBgNVBAYTAklOMQswCQYDVQQIDAJNSDEPMA0GA1UE
BwwGTVVNQkFJMRAwDgYDVQQKDAdFWEFNUExFMREwDwYDVQQLDAhURVNUVU5JVDEU
MBIGA1UEAwwLZXhhbXBsZS5jb20xHzAdBgkqhkiG9w0BCQEWEHVzZXJAZXhhbXBs
ZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDlGSFQt0sR4kJC
/39NtTKNNObIRESPpXABZMoQrH0opZROFgsNHMu1Y28/tMIxzENrytPRuOcmsA+R
wMyd9KUjrLZSXoNTwGVJQsyU2gII66fnoXHOIaTHm6YP/a4KuYMDW4jLl5TugNqh
4TjZ/cP4cDuB4dEEuq7X33YBktVH8OMxiel8P8Nh8cU6xBQRZyubluWQwwX6hMiC
lOKkHKMIzx54l1QJ3EiVY1n0N44Q4AIR7xzULY1bghSWdvqi4Tu88NJh/Xy2whWv
ybJiN7o8d4P5qxP9+uHjFWeLcMd1w2G53wdsALvphEUcmd4FwrEiZTjHJoTIIz8h
8EShqq0hAgMBAAGgGDAWBgkqhkiG9w0BCQIxCQwHRVhBTVBMRTANBgkqhkiG9w0B
AQsFAAOCAQEAJlhA5unLl0e4yk5Is0P4LUU2VkZ8nn2osTAo/AYkFblGddxzQFL3
7iy8sfC7lLK9UysG8hmbo8jHSuEaAmH5lkBzlxde2SavBPbafZ8UovjHvdbrZmi9
8MTdVHDPA5KaV6Z7wM1I6Qtm17Q0gx+YppvBv6oyZINs2wcU7KKcixJTE9MyPiFK
VIKvk+OCNXPuDypA6Z90+LNBbrWBvqgRmL4PI7QjLynndhhZoQHTIZao8WFZtx4v
vQF7d0XwHgIUqlyc0bjSvwc6Yx2Q2wWRtFAM3h64XdFt8KtfIOmv0Tshe88bvun0
pkMnhxwY1CyFUKfNNYHsNHZahDenIQP90g==
-----END CERTIFICATE REQUEST-----

Conversions using openssl

1. Convert crt to pem


Command:

openssl x509 -in example.crt -out example.pem -outform PEM

This will generate a file example.pem whose contents are the same as that of example.crt. So, effectively it just copying the exmaple.crt file as example.pem.

2. Convert to p12


This requires example.crt file to be converted to example.pem and you know how to do it :)

Command:

openssl pkcs12 -export -out example.p12 -in example.pem -inkey example.key 

Output:

# openssl pkcs12 -export -out example.p12 -in example.pem -inkey example.key 
Enter pass phrase for example.key:
Enter Export Password:
Verifying - Enter Export Password:

3. Extracting key from p12 file


Command:

openssl pkcs12 -in example.p12 -nocerts -out example2.key

Output:

# openssl pkcs12 -in example.p12 -nocerts -out example2.key
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

This generates a key example2.key from the p12 file.

4. Extracting crt from p12 file


Command:

openssl pkcs12 -in example.p12 -clcerts -nokeys -out example2.crt

Output:

# openssl pkcs12 -in example.p12 -clcerts -nokeys -out example2.crt
Enter Import Password:
MAC verified OK

It generates a crt file example2.crt whose contents can be displayed:

Bag Attributes
    localKeyID: 6C B6 C7 C8 85 56 86 38 46 A8 C9 27 0F 7A 72 8D A8 D5 C7 CF 
subject=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
issuer=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
-----BEGIN CERTIFICATE-----
MIIDjDCCAnQCCQCZctLzit9LpjANBgkqhkiG9w0BAQsFADCBhzELMAkGA1UEBhMC
SU4xCzAJBgNVBAgMAk1IMQ8wDQYDVQQHDAZNVU1CQUkxEDAOBgNVBAoMB0VYQU1Q
TEUxETAPBgNVBAsMCFRFU1RVTklUMRQwEgYDVQQDDAtleGFtcGxlLmNvbTEfMB0G
CSqGSIb3DQEJARYQdXNlckBleGFtcGxlLmNvbTAeFw0xODA4MDIxNTI0MjJaFw0x
OTA4MDIxNTI0MjJaMIGHMQswCQYDVQQGEwJJTjELMAkGA1UECAwCTUgxDzANBgNV
BAcMBk1VTUJBSTEQMA4GA1UECgwHRVhBTVBMRTERMA8GA1UECwwIVEVTVFVOSVQx
FDASBgNVBAMMC2V4YW1wbGUuY29tMR8wHQYJKoZIhvcNAQkBFhB1c2VyQGV4YW1w
bGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5RkhULdLEeJC
Qv9/TbUyjTTmyEREj6VwAWTKEKx9KKWUThYLDRzLtWNvP7TCMcxDa8rT0bjnJrAP
kcDMnfSlI6y2Ul6DU8BlSULMlNoCCOun56FxziGkx5umD/2uCrmDA1uIy5eU7oDa
oeE42f3D+HA7geHRBLqu1992AZLVR/DjMYnpfD/DYfHFOsQUEWcrm5blkMMF+oTI
gpTipByjCM8eeJdUCdxIlWNZ9DeOEOACEe8c1C2NW4IUlnb6ouE7vPDSYf18tsIV
r8myYje6PHeD+asT/frh4xVni3DHdcNhud8HbAC76YRFHJneBcKxImU4xyaEyCM/
IfBEoaqtIQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBvp6MetgOlXC2SbOCz7oxe
XQhAsIzxoxb57uzkGK3fnyE/Bdj+08IscosO2+2CGyS0v/hKAql+aoiCb/TBS3CP
E1zhdRkIruUBmntOy3wbW2lSZAv/bg8rWicrvzuj4QGFL8oCIjBgEH4tmlgmmwzE
rbSqzQXppSZ9YeyCKkVkNOsCldBEFUkDo+hZ5cD3F71oiM/vWjQWJpXWI/ifSRSN
BhnljhPEXFzL+GBMMgVSthKccy5WNj7UBeKK6RhjruSLp2HnEYZWnQauXNz2IDzY
WFEWDdBalp6q+1WhTR7cYFaGdc6ae6jKwvMW0Y5fpPSYl8zwwePTX6xyb1bz0YTx
-----END CERTIFICATE-----

Verification with openssl

Verify a private key


Command:

openssl rsa -in example.key -check

Output:

# openssl rsa -in example.key -check
Enter pass phrase for example.key:
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA5RkhULdLEeJCQv9/TbUyjTTmyEREj6VwAWTKEKx9KKWUThYL
DRzLtWNvP7TCMcxDa8rT0bjnJrAPkcDMnfSlI6y2Ul6DU8BlSULMlNoCCOun56Fx
ziGkx5umD/2uCrmDA1uIy5eU7oDaoeE42f3D+HA7geHRBLqu1992AZLVR/DjMYnp
fD/DYfHFOsQUEWcrm5blkMMF+oTIgpTipByjCM8eeJdUCdxIlWNZ9DeOEOACEe8c
1C2NW4IUlnb6ouE7vPDSYf18tsIVr8myYje6PHeD+asT/frh4xVni3DHdcNhud8H
bAC76YRFHJneBcKxImU4xyaEyCM/IfBEoaqtIQIDAQABAoIBAEggfk2kN109B7GC
MPktF+o/An2AiLlU04uMyNUxCQJ4BtYCuJ37N+M2l/rNlOoKrhtMsx1vAk861NwF
gCYmtKsjoA8UtHH0u66x+ijO7h8S2jzhpivIidFSHpkgO+Aiga3X7pxyb7AbHzoh
5z0yWLVp2EJO7vh3Mb7DGqrqKH3KL5rlRZeQ5+DvF3bFxoCLjNO7uOaPsLNf8vnx
8VGzjuts47LUL4Gtn/92FuOCsiRN1yhhd9yC9fnQ6B7R6fMcu6JSKyMHqp2xMlvO
8/cPUIHgoAOUQm3rLBJywsY7YGViq712qIZJRo2OT0XFCiAgDFQXSuYulg2/akY5
xEqG4sECgYEA+irSjBinHY7iKnAKGq5zbt/GWTyDd5tLa7MEBpVkrWMbTc7kMh3m
LCyYMtXEp0J+F+hbKORKnUY6I9ESTEcvGeCRYYYeghemNdTQK3xZmC5K1Ie1SLxs
BYYAT17O9OwxlG7yElobf689/kbbS6m99lkLAJPIlkkUTXVfJFpCX7kCgYEA6nCN
WPyI9Zr+IVakBbDLfueOVbvBTvty5z2Owm2sWdx+1ZeS9/Y+0mLi+HKWg87qjPfn
Wg/WZoBSAO3EKH7TKeZs2PQnkO5+QFjRdEMRZJsVLMGmHTdwSg+zOQqY1Owrak41
Z1q0OnfHfRD5GSmuC+qL8rnrDKeDRlI6kPiLXKkCgYAX/bVVZCParNI5uabuah59
2o3+DtSYytbPzEoti3QtQJzkuFugBsgFIn2yGlgSpkjLaCgd4s7ZCFwZBTrY/9af
h67JiIAyf6wb633PPcyl0IKyoRUclZ1SZkhvVCtKx7/1eTJT2jpa42ZxlUAAoJWs
I1vvwTxyVS3SQ9hM9y74WQKBgDMNz7pRRII3/p/D/nQ+zBIW2yhQewh4Oc2h2jT9
LUtkeTqKh7b4KTYn0sXILn8F0I5ibj9us4Ie77zECrPG6rV1OL8GbJdjWJsqvHJV
KLSAEVxtz38NU6bNRHpnlGDKDapY3chkOFuDOi5CQ/z21rBBo7h4RHe9AxmNTrPD
bJYBAoGANmFJjJyAkNVEdSGdWCv9iDQhkP3TRsE0hFICEniptHL/hmZzz1GfFdTL
4F53AQ9Prl+zcxjQCL4uenNFieYDFAej/bVlEIP6dI6GYKYCMOaWtoG5sByhq2sZ
oAmtuXsC5dJFHTahxMqDOcxFoHgmhn0lktQ6VRTDEBaa3x6/62w=
-----END RSA PRIVATE KEY-----

2. Verify a certificate


Command:

openssl x509 -in example.crt -noout -text

Output:

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 11057131978596764582 (0x9972d2f38adf4ba6)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=IN, ST=MH, L=MUMBAI, O=EXAMPLE, OU=TESTUNIT, CN=example.com/emailAddress=user@example.com
        Validity
            Not Before: Aug  2 15:24:22 2018 GMT
            Not After : Aug  2 15:24:22 2019 GMT
        Subject: C=IN, ST=MH, L=MUMBAI, O=EXAMPLE, OU=TESTUNIT, CN=example.com/emailAddress=user@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e5:19:21:50:b7:4b:11:e2:42:42:ff:7f:4d:b5:
                    32:8d:34:e6:c8:44:44:8f:a5:70:01:64:ca:10:ac:
                    7d:28:a5:94:4e:16:0b:0d:1c:cb:b5:63:6f:3f:b4:
                    c2:31:cc:43:6b:ca:d3:d1:b8:e7:26:b0:0f:91:c0:
                    cc:9d:f4:a5:23:ac:b6:52:5e:83:53:c0:65:49:42:
                    cc:94:da:02:08:eb:a7:e7:a1:71:ce:21:a4:c7:9b:
                    a6:0f:fd:ae:0a:b9:83:03:5b:88:cb:97:94:ee:80:
                    da:a1:e1:38:d9:fd:c3:f8:70:3b:81:e1:d1:04:ba:
                    ae:d7:df:76:01:92:d5:47:f0:e3:31:89:e9:7c:3f:
                    c3:61:f1:c5:3a:c4:14:11:67:2b:9b:96:e5:90:c3:
                    05:fa:84:c8:82:94:e2:a4:1c:a3:08:cf:1e:78:97:
                    54:09:dc:48:95:63:59:f4:37:8e:10:e0:02:11:ef:
                    1c:d4:2d:8d:5b:82:14:96:76:fa:a2:e1:3b:bc:f0:
                    d2:61:fd:7c:b6:c2:15:af:c9:b2:62:37:ba:3c:77:
                    83:f9:ab:13:fd:fa:e1:e3:15:67:8b:70:c7:75:c3:
                    61:b9:df:07:6c:00:bb:e9:84:45:1c:99:de:05:c2:
                    b1:22:65:38:c7:26:84:c8:23:3f:21:f0:44:a1:aa:
                    ad:21
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         6f:a7:a3:1e:b6:03:a5:5c:2d:92:6c:e0:b3:ee:8c:5e:5d:08:
         40:b0:8c:f1:a3:16:f9:ee:ec:e4:18:ad:df:9f:21:3f:05:d8:
         fe:d3:c2:2c:72:8b:0e:db:ed:82:1b:24:b4:bf:f8:4a:02:a9:
         7e:6a:88:82:6f:f4:c1:4b:70:8f:13:5c:e1:75:19:08:ae:e5:
         01:9a:7b:4e:cb:7c:1b:5b:69:52:64:0b:ff:6e:0f:2b:5a:27:
         2b:bf:3b:a3:e1:01:85:2f:ca:02:22:30:60:10:7e:2d:9a:58:
         26:9b:0c:c4:ad:b4:aa:cd:05:e9:a5:26:7d:61:ec:82:2a:45:
         64:34:eb:02:95:d0:44:15:49:03:a3:e8:59:e5:c0:f7:17:bd:
         68:88:cf:ef:5a:34:16:26:95:d6:23:f8:9f:49:14:8d:06:19:
         e5:8e:13:c4:5c:5c:cb:f8:60:4c:32:05:52:b6:12:9c:73:2e:
         56:36:3e:d4:05:e2:8a:e9:18:63:ae:e4:8b:a7:61:e7:11:86:
         56:9d:06:ae:5c:dc:f6:20:3c:d8:58:51:16:0d:d0:5a:96:9e:
         aa:fb:55:a1:4d:1e:dc:60:56:86:75:ce:9a:7b:a8:ca:c2:f3:
         16:d1:8e:5f:a4:f4:98:97:cc:f0:c1:e3:d3:5f:ac:72:6f:56:
         f3:d1:84:f1

3. Verify a CSR


Command:

openssl req -in example.csr -noout -text -verify

Output:

# openssl req -in example.csr -noout -text -verify
verify OK
Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: C=IN, ST=MH, L=MUMBAI, O=EXAMPLE, OU=TESTUNIT, CN=example.com/emailAddress=user@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e5:19:21:50:b7:4b:11:e2:42:42:ff:7f:4d:b5:
                    32:8d:34:e6:c8:44:44:8f:a5:70:01:64:ca:10:ac:
                    7d:28:a5:94:4e:16:0b:0d:1c:cb:b5:63:6f:3f:b4:
                    c2:31:cc:43:6b:ca:d3:d1:b8:e7:26:b0:0f:91:c0:
                    cc:9d:f4:a5:23:ac:b6:52:5e:83:53:c0:65:49:42:
                    cc:94:da:02:08:eb:a7:e7:a1:71:ce:21:a4:c7:9b:
                    a6:0f:fd:ae:0a:b9:83:03:5b:88:cb:97:94:ee:80:
                    da:a1:e1:38:d9:fd:c3:f8:70:3b:81:e1:d1:04:ba:
                    ae:d7:df:76:01:92:d5:47:f0:e3:31:89:e9:7c:3f:
                    c3:61:f1:c5:3a:c4:14:11:67:2b:9b:96:e5:90:c3:
                    05:fa:84:c8:82:94:e2:a4:1c:a3:08:cf:1e:78:97:
                    54:09:dc:48:95:63:59:f4:37:8e:10:e0:02:11:ef:
                    1c:d4:2d:8d:5b:82:14:96:76:fa:a2:e1:3b:bc:f0:
                    d2:61:fd:7c:b6:c2:15:af:c9:b2:62:37:ba:3c:77:
                    83:f9:ab:13:fd:fa:e1:e3:15:67:8b:70:c7:75:c3:
                    61:b9:df:07:6c:00:bb:e9:84:45:1c:99:de:05:c2:
                    b1:22:65:38:c7:26:84:c8:23:3f:21:f0:44:a1:aa:
                    ad:21
                Exponent: 65537 (0x10001)
...
...

4. Verify a pem file


Command:

openssl pkcs12 -in example.p12 -info

Output:


# openssl pkcs12 -in example.p12 -info
Enter Import Password:
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: 6C B6 C7 C8 85 56 86 38 46 A8 C9 27 0F 7A 72 8D A8 D5 C7 CF 
subject=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
issuer=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
-----BEGIN CERTIFICATE-----
MIIDjDCCAnQCCQCZctLzit9LpjANBgkqhkiG9w0BAQsFADCBhzELMAkGA1UEBhMC
SU4xCzAJBgNVBAgMAk1IMQ8wDQYDVQQHDAZNVU1CQUkxEDAOBgNVBAoMB0VYQU1Q
TEUxETAPBgNVBAsMCFRFU1RVTklUMRQwEgYDVQQDDAtleGFtcGxlLmNvbTEfMB0G
CSqGSIb3DQEJARYQdXNlckBleGFtcGxlLmNvbTAeFw0xODA4MDIxNTI0MjJaFw0x
OTA4MDIxNTI0MjJaMIGHMQswCQYDVQQGEwJJTjELMAkGA1UECAwCTUgxDzANBgNV
BAcMBk1VTUJBSTEQMA4GA1UECgwHRVhBTVBMRTERMA8GA1UECwwIVEVTVFVOSVQx
FDASBgNVBAMMC2V4YW1wbGUuY29tMR8wHQYJKoZIhvcNAQkBFhB1c2VyQGV4YW1w
bGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5RkhULdLEeJC
Qv9/TbUyjTTmyEREj6VwAWTKEKx9KKWUThYLDRzLtWNvP7TCMcxDa8rT0bjnJrAP
kcDMnfSlI6y2Ul6DU8BlSULMlNoCCOun56FxziGkx5umD/2uCrmDA1uIy5eU7oDa
oeE42f3D+HA7geHRBLqu1992AZLVR/DjMYnpfD/DYfHFOsQUEWcrm5blkMMF+oTI
gpTipByjCM8eeJdUCdxIlWNZ9DeOEOACEe8c1C2NW4IUlnb6ouE7vPDSYf18tsIV
r8myYje6PHeD+asT/frh4xVni3DHdcNhud8HbAC76YRFHJneBcKxImU4xyaEyCM/
IfBEoaqtIQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBvp6MetgOlXC2SbOCz7oxe
XQhAsIzxoxb57uzkGK3fnyE/Bdj+08IscosO2+2CGyS0v/hKAql+aoiCb/TBS3CP
E1zhdRkIruUBmntOy3wbW2lSZAv/bg8rWicrvzuj4QGFL8oCIjBgEH4tmlgmmwzE
rbSqzQXppSZ9YeyCKkVkNOsCldBEFUkDo+hZ5cD3F71oiM/vWjQWJpXWI/ifSRSN
BhnljhPEXFzL+GBMMgVSthKccy5WNj7UBeKK6RhjruSLp2HnEYZWnQauXNz2IDzY
WFEWDdBalp6q+1WhTR7cYFaGdc6ae6jKwvMW0Y5fpPSYl8zwwePTX6xyb1bz0YTx
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Bag Attributes
    localKeyID: 6C B6 C7 C8 85 56 86 38 46 A8 C9 27 0F 7A 72 8D A8 D5 C7 CF 
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQISSIn7yGZN+8CAggA
MBQGCCqGSIb3DQMHBAhxEHorIofSVwSCBMjM/SBuC6Qeh9LE6YcLuWleYSC/21Yi
52i/UW81AbSqVcFU7zkgB4ro/1s/WWdrFiohpSXhZUq3/kXoPJlXrU5yQK0zoDaE
Lxz/EHb1KH2TXiLhyIPEB/LL8kVMHIQm7SO94p2Gl3JuP4j1hmxtQwlDpmCc3ewh
7GEK6a5ggazfUhMY6nim9Sx8qe7pHCApFYo72FTl/KlhzexKso1JexsCkc7chD5j
87fcW62+L9dI7SgarGabXtzSlpUQ+TW3Wz16ix2TE0t31DAlDuuK5Qpcv91gIuBB
AL00QGWUQg92WA+4KZJ4WoLYqKgDrgOyWCgrMg2Z+2hhcpjV29hE5D5uWJRdoIz5
tvkrTk8+M/xfs3gfB52ahGO1ttS9zqnffo81xshCPI9d2cU4uFN0sa7hH31rrxtz
Cp4GYWtmxoA7nBg8pyiwk5iQLRwUMDkAN6XtL5uVewhjwmn2Bsv2ouHUJi4mD3D9
zZcYYvbTKKQ7gJ/8gy6/trtuSMcW3tgDGCwpkaRl7YEsXNYkVd2k5jSqAoLLy0tL
M9uGahD76MDq4XVTl3BH7Qt2vVnTqtEaPPue8nBTSnULFhHBy+ZYmmWtZImWsp2C
tYxST09n/MLhm1hkBC7STmJJlaubEdjDy2zPLF0ONwy522xHjqcOQyd/RwoU0hVe
rqR//fmGEHD3Hd/dANMDmvcDwDtsHMJm09WmF2qt23M7MzGI1HGj1/ZPbRG5vxrd
LVP1rd5nFMwJED5xQRlbDw7E8Qxu1pEEeyFDs9uYcE8lmyCzwqmzCPkkT0rwPzNa
e2jLiI2dnBVJbJj9Ti4UuPyCYo3bjqkfksa9sY/EAUrB2Hgs0LA1xCTuHeRfBDfV
yYf5izdIrmtzqDPbNgY0Jel20KRPXqN4JVSaZ9ANQOPQvCFSIywQAZm//jJSXjNY
fTpVYSTV/tvgESYe2lYME9Vfl/10n2pt92NUI+8zIWgG1t2Ad+bSn/M1WIv4oXPb
OEhti+Y0VfhcAnOiP5p1rKTc3ftVlYhnJ0ug6Yw5E0Xj4Nfnewu/JFSumg95Lw/T
eICm/Lzt6f6zldsNlCW9qxBaHosCDr5QZlYQYTSAOQzCn0+K0KZL46jN7NoAYvUD
tjkHLkJDxhP6wXMa5Bdz4XCHnUeS4VSoe2ZGSwVwNYs8c4P00VjutAovmNFL5wQx
rUuadq3L0EOja02/25qmtl/xAtgZd87k7hwCz+Ux3BduclTF2P5Ge2bk4x6uaX4p
jwY5R8Crm20cxhyLXP1QEXUq9rlo0qXi48m/ll7diH4ipRDcryUT8p9odrw1O534
UVJtu8oQNErvZsHLJZTWvyQ97KthKCzWsBePI3F4gI9k8YBQ4TK3vKZ2iPVdMHO6
h9yMoihYKFyYzN02SRdqGHdYkM39dN169bUjykgYpdXuVIoXXuHyIn6tmx7TDtyi
/qWFa2jgK8zJmU1ytv06KPzYyQx3EhqDiNCrmJXyqXa8/QETKs9PnSCcDgtp3aLC
vsXS6s+pH0jlt14aY05aJOj15dlsgBztMQ2h1alKm2P6WkbduN2BXdBwv/CGS46U
fBckbGkbGufmT7vpIUyzDvWR0G7Lj26jqJecJF9v3+gxud+r/U1Sk1XMyl77yryp
mQA=
-----END ENCRYPTED PRIVATE KEY-----


Tuesday, 10 April 2018

AWK Programming Tutorial- Awk built-in variables FS, OFS, RS, ORS, NF, NR

Awk built-in variables: This is the fourth article of this tutorial series on awk and in this one, we will be learning about built-in variables in awk. In case you have missed any of our previous articles, you can find them out here.


Awk comes up with a number of built-in variables. Of these variables, some have a default value associated with them which can be changed e.g. FS ( field separator, with default value of a whitespace ) and RS ( record separator, with default value of \n ). While, some variables are quite useful while doing analysis or creating reports e.g. NF ( number of fields ) and NR ( number of records ). Lets take a look at them one-by-one.

FS (Field Separator) and OFS (Output Field Separator)

  • With FS, we instruct awk that, in a particular input file, fields are separated by some character.
  • Default value if this variable is a whitespace, telling awk that fields are separated by one or more whitespaces (including tabs).
  • This default value can be overwritten with a character or a regular expression. For example, we can use a colon ( : ) to separate fields while working on /etc/passwd file.
  • With OFS, we ask awk to use a particular character to separate the fields in the output.
  • For this variable too, default value is a single whitespace.
Lets take a look at an example now. For this, we will use demo csv file with contents as shown below:

1,"Eldon Base for stackable storage shelf, platinum",Muhammed MacIntyre,3,-213.25,38.94,35,Nunavut,Storage & Organization,0.8
2,"1.7 Cubic Foot Compact ""Cube"" Office Refrigerators",Barry French,293,457.81,208.16,68.02,Nunavut,Appliances,0.58
3,"Cardinal Slant-DÆ Ring Binder, Heavy Gauge Vinyl",Barry French,293,46.71,8.69,2.99,Nunavut,Binders and Binder Accessories,0.39
4,R380,Clay Rozendal,483,1198.97,195.99,3.99,Nunavut,Telephones and Communication,0.58
5,Holmes HEPA Air Purifier,Carlos Soltero,515,30.94,21.78,5.94,Nunavut,Appliances,0.5
6,G.E. Longer-Life Indoor Recessed Floodlight Bulbs,Carlos Soltero,515,4.43,6.64,4.95,Nunavut,Office Furnishings,0.37
7,"Angle-D Binders with Locking Rings, Label Holders",Carl Jackson,613,-54.04,7.3,7.72,Nunavut,Binders and Binder Accessories,0.38
8,"SAFCO Mobile Desk Side File, Wire Frame",Carl Jackson,613,127.70,42.76,6.22,Nunavut,Storage & Organization,
9,"SAFCO Commercial Wire Shelving, Black",Monica Federle,643,-695.26,138.14,35,Nunavut,Storage & Organization,
10,Xerox 198,Dorothy Badders,678,-226.36,4.98,8.33,Nunavut,Paper,0.38

By default FS will use whitespace as a default value. Lets check extracting 1st and 3rd column without default value of FS.

$ awk '{ print $1, $3 }' input.csv 
1,"Eldon for
2,"1.7 Foot
3,"Cardinal Ring
4,R380,Clay and
5,Holmes Air
6,G.E. Indoor
7,"Angle-D with
8,"SAFCO Desk
9,"SAFCO Wire
10,Xerox Badders,678,-226.36,4.98,8.33,Nunavut,Paper,0.38

And now, using comma ( , ) as the field separator value.

$ awk 'BEGIN { FS = ","; } { print $1, $3 }' input.csv
1  platinum"
2 Barry French
3  Heavy Gauge Vinyl"
4 Clay Rozendal
5 Carlos Soltero
6 Carlos Soltero
7  Label Holders"
8  Wire Frame"
9  Black"
10 Dorothy Badders

As we can see in above outputs, awk uses the default value of OFS which is a single whitespace. We can overwrite this value, to say a pipe ( | ) as shown in below example:

$ awk 'BEGIN { FS = ","; OFS = "|" } { print $1, $3 }' input.csv
1| platinum"
2|Barry French
3| Heavy Gauge Vinyl"
4|Clay Rozendal
5|Carlos Soltero
6|Carlos Soltero
7| Label Holders"
8| Wire Frame"
9| Black"
10|Dorothy Badders





RS (Record Separator) and ORS (Output Record Separator)

  • RS and ORS are useful while dealing with multi-line records. In this case, each field is on a new line.
  • Default value of both these variables is a newline character ( \n ).
  • With ORS value overwritten, we can tell awk to separate records with some other character then the newline.
Lets take a look at our demo file wherein each record is separated by dual newlines ( \n\n ) and each field in the record is separated using single newline character ( \n ).

$ cat address.txt 
Cecilia Chapman
711-2880 Nulla St.
Mankato Mississippi 96522
(257) 563-7401

Iris Watson
P.O. Box 283 8562 Fusce Rd.
Frederick Nebraska 20620
(372) 587-2335

Celeste Slater
606-3727 Ullamcorper. Street
Roseville NH 11523
(786) 713-8616

Theodore Lowe
Ap #867-859 Sit Rd.
Azusa New York 39531
(793) 151-6230

Now, to display a person's name ( $1 ) and his/her phone number ( $4 ) on a separate line ( ORS will be \n, while RS is \n\n ), we can use below command:

$ awk ' BEGIN { FS = "\n"; RS = "\n\n"; ORS = "\n" } { print $1, $4 } ' address.txt 
Cecilia Chapman (257) 563-7401
Iris Watson (372) 587-2335
Celeste Slater (786) 713-8616
Theodore Lowe (793) 151-6230

NF (Number of Fields) and NR (Number of Record)

  • Awk variable NF defines the number of fields if the current record ( $0 ).
  • If we try to increase the value of NF, awk adds additional fields separated by the delimiter value in OFS.
  • Whereas, when we decrease the value of NF, all the fields with identifiers greater than the value are ignored.
  • NR is the variable that stores the current record number being processed by awk.
  • There is another variable, FNR, which is useful while dealing with multiple files. It stores the position of the record relative to the current file only.
Lets take a look at below demo file to illustrate this example. If you observe, it has different number of fields on each record.

$ cat cities.txt 
Washington 18 23 21 19
London 10 7 13 5 -1
Moscow 2 0 -3
Mumbai 24 27

Now, we print number of fields a record has before printing the record itself, using below command:

$ awk '{print NF, $0}' cities.txt 
5 Washington 18 23 21 19
6 London 10 7 13 5 -1
4 Moscow 2 0 -3
3 Mumbai 24 27

To illustrate the use of NR, we use the same file again. Its pretty straight forward.

$ awk '{print NR, $0}' cities.txt 
1 Washington 18 23 21 19
2 London 10 7 13 5 -1
3 Moscow 2 0 -3
4 Mumbai 24 27

In case there are multiple files, we can print the record number relative to the current input file being processed using the variable FNR.

$ awk '{print FNR, $0}' cities.txt address.txt 
1 Washington 18 23 21 19
2 London 10 7 13 5 -1
3 Moscow 2 0 -3
4 Mumbai 24 27
1 Cecilia Chapman
2 711-2880 Nulla St.
3 Mankato Mississippi 96522
4 (257) 563-7401
5 
6 Iris Watson
7 P.O. Box 283 8562 Fusce Rd.
8 Frederick Nebraska 20620
9 (372) 587-2335
10 
11 Celeste Slater
12 606-3727 Ullamcorper. Street
13 Roseville NH 11523
14 (786) 713-8616
15 
16 Theodore Lowe
17 Ap #867-859 Sit Rd.
18 Azusa New York 39531
19 (793) 151-6230

Observe the line after line #4. Awk has numbered it #1, just because we have used FNR. Had we used NR here, it would have been numbered #5. You can check this out, I will leave this for you.

That's it for the scope of this article. Please share your feedback and suggestions in the comments section below and stay tuned for more articles. Thanks for reading.

Friday, 6 April 2018

How To: Install or Upgrade to Linux Kernel 4.12 in Ubuntu/Linux Mint

The Linux Kernel 4.12 is available for the users. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 4.12 in your Ubuntu or Linux Mint system.

Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200_4.12.0-041200.201707022031_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200-generic_4.12.0-041200.201707022031_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-image-4.12.0-041200-generic_4.12.0-041200.201707022031_i386.deb

Install them.

$ sudo dpkg -i linux-headers-4.12.0*.deb linux-image-4.12.0*.deb

Reboot the system.

$ sudo reboot





For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200_4.12.0-041200.201707022031_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200-generic_4.12.0-041200.201707022031_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-image-4.12.0-041200-generic_4.12.0-041200.201707022031_amd64.deb

Install them.

$ sudo dpkg -i linux-headers-4.12.0*.deb linux-image-4.12.0*.deb

Reboot the system.

$ sudo reboot

To uninstall,

$ sudo apt-get remove 'linux-headers-4.12.0*' 'linux-image-4.12.0*'

Friday, 30 March 2018

AWK Programming Tutorial - Constants, Variables and Arithmetic Operators

Constants, Variables and Operators in Awk : So far in this tutorial series on awk programming, we have learned to print stuff, we learned about fields, records and delimiters and how to they are referenced. These are very basic operations as we are just extracting data from input lines and printing them. Now, we move a step ahead and manipulate the extracted fields by performing some common arithmetic operations on them. Before we proceed, I recommend you to please go through the third article published on awk - Field separator and Field references.


Constants

There are only two types of constants:
A string constant

  • A string constant is always surrounded within quotes, e.g. "Pineapple", "5_days", etc.
  • String can use escape sequences like \n ( newline ), \t ( horizontal tab ), \b ( backspace ), \v ( vertical tab ), \" ( double quotes ), etc.
A numeric constant

  • A numeric constant is a number without quotes, enough said.
  • A number enclosed within quotes is considered as a string.
Variables

  • A variable is an identifier that references to the memory location that stores a value.
  • We can initialize a variable by assigning a value to it, using = operator, e.g., age = 20, firstName = "Eric", etc. Here, age and firstName are variables.
  • A variable name consist of alphabets, digits and underscores, and it must start with a letter or underscore.
  • Variables are case-sensitive. It means, Age, age and AGE are different variables and they can store different values in them that won't get overwritten.
  • Variable initialization is optional. If we do not initialize a variable, awk defaults the value to numeric 0 or a blank string ( "" ) appropriately.
  • When we assign two or more strings separated by space to a variable, it stores a concatenated value
  • We can assign a field value to a variable using field reference variables $1, $2, etc.
Example:

# Assign a numeric value to variable 'myNum'
myNum = 10

# Assign a string value to variable 'myStr'
myStr = "awk!"

# Space concatenates the strings, so 'myVar' stores the value "AwesomeAwk!"
myVar = "Awesome" "Awk!"

# Assign a field value to a variable using field reference variable
marks = $1





Arithmetic Operators

awk supports basic arithmetic operators to be used in expressions, which are listed as below:

Operator Description
+ Addition
- Subtraction
* Multiplication
/ Division
% Modulus
^ or ** Exponentiation

Below example shows how we can define a variable and perform arithmetic operations on them.

Example:

# Initialize the variable 'salary'
salary = 300000

# Add 25000 to variable 'salary' and store the result in another variable 'newSalary'
newSalary = salary + 25000

# Print the updated salary
print newSalary

Alternately, you can directly print the addition of salary and the number 25000 to further shrink the code as:

salary = 300000
print salary + 25000

This way, we will get the similar result from print statement. But, the value stored in variable salary remains unchanged. If we were to update the variable salary with the added value, we can use assignment operator +=, that combines 2 operations, addition and assignment. So, we have the number 25000 added to the value stored in salary and the result of addition is again stored in the variable salary.

Below is the list of assignment operators:

Operator Description
+= Add and assign
-= Subtract and assign
*= Multiply and assign
/ Divide and assign
% Perform modulo and assign the result
^ or ** Perform exponentiation and assign the result

To demonstrate this, we can use /etc/passwd and count the number of lines in it. For this, we initialize a variable x and increment it after every line is read. After the last line, we print the variable, which gives us total number of records read by awk.

# We can also use { x++ } or { x = x + 1 } instead of { x += 1 } in below command
$ awk ' { x += 1 } END { print x } ' /etc/passwd
31

We can also include a condition here, to print the count of lines those have the string bash inside them.

$ awk '/bash/ { x += 1 } END { print x } ' /etc/passwd
3

Another example. This time, we use a demo file which has 3 fields - Name of the student, Subject name and the Marks. Below is the snippet.

Student Subject Marks
James Biology 31
Velma Biology 43
Kibo Biology 81
Louis Biology 11
Phyllis Biology 18
Zenaida Biology 55
Gillian Biology 38
Constance Biology 16
Giselle Biology 73
...
...

We can calculate the average marks obtained by students in Chemistry as follows:

awk ' /Chemistry/ { total += $3; count += 1 } END { print total/count } ' result.txt 
52.4074

We can also consider a data set of cities and their temperatures as below:

Washington 18 23 21 19 16
London 10 7 13 5 -1
Moscow 2 0 -3 -7 1
Mumbai 24 27 29 29 28

We can find average temperature for every city as shown below:

$ awk ' {total = $2 + $3 + $4 + $5 + $6; print $1 " : " total/5 }' cities.txt 
Washington : 19.4
London : 6.8
Moscow : -1.4
Mumbai : 27.4

That's all for the scope of this article. We did not cover all of the operators here as there are pretty straight forward and most of you may already have an idea about those ones. Let me know about your views and feedback in the comments section below and stay tuned for more articles.

Advance Your Career with Linux Foundation Training

Linux is the largest open-source technology that powers computers, mobiles and various other products and services across the world. It is the OS that runs more than 95% of the top 1 million domains and the top 500 supercomputers while accounting for 80% of smartphones, running Android based on Linux kernel.

Developers and engineers with Linux skills are in high demand in the job market. If you have a qualification or certification in Linux ecosystem, then you can get hired with high salary.
Linux Foundation, which is headed by the creator of Linux, provides several courses and certifications in different Linux technologies. You can easily apply for the course you want and increase your value as a potential employee or independent developer.

Popular Linux Foundation Training Courses

Which Linux Foundation training course should you go for? We will help you decide!

Certification

You can apply for the Linux Foundation Certified Engineer (LFCE) and Linux Foundation Certified System Administrator (LFCS) certification, which are both carried out online.

LFCS imparts the skills and knowledge of a sysadmin, which you need to help prove yourself to employers. LFCE gives you in-depth skills that enable you to design and implement system architecture. You can also provide guidance as a Subject Matter Expert using your newfound expertise.

Both certifications cost $499 and can be completed in 12 months.

Linux Courses

Linux Foundation offers both introductory and advanced Linux courses which can help you land a great job. The courses cover a wide range of aspects and are divided into the following categories-

  • Linux Programming & Development Training
  • Enterprise IT & Linux System Administration Training
  • Open Source Compliance Courses
Some of the courses can be completed in 1 year while others run for four days. You can even take your own time in the case of some courses, as they have no time limit.

The cost of the courses also varies according to their content- you can take some courses for completely free while others may cost anything from $179 to $3,150.

You can also apply for e-Learning courses, which can be completed at your own pace. The cost ranges between $149 and $299 while some are totally free! Use a Linux Foundation coupon and you can receive your certifications at a reduced price!

Who can Benefit from Linux Foundation Training?

The training can either be taken by individuals looking to advance their career or by employees of organizations, depending on Linux solutions.

Top companies like AMD, HP, Intel, and Nokia depend on Linux Foundation training to help their employees get skilled in open-source technologies. This also increases the demand for skilled Linux professionals who can successfully contribute towards the development of better products and services.

There is currently a shortage of talent when it comes to open-source professionals all over the world. If you have a certificate from the creators of Linux guaranteeing your skills, it becomes much easier to increase your demand in the job market.

You can get better and higher paying jobs without settling for anything less. Clients will also be happy to pay higher fees to a Linux freelancer with an approved qualification.

So whether you are a corporate employee or hoping one day to be a Linux pro, the courses at Linux Foundation are ideal to make you skilled and knowledgeable.

AWK Programming Tutorial - Field separator and Field references

Awk Field separator and field references: This is the third article from our tutorial series on awk. In first article, we had an introduction with awk and in second one, we created Hello world program in awk. In this article, we will be learning about separating fields and referencing them using awk.


Referencing Fields and Records

In the first article from this tutorial series, Introduction to awk, we covered following points:
  • awk presumes that the input is a structured type of data
  • It interprets each line from input file(s) as a Record
  • Each line will have strings/words separated (or delimited) by whitespaces or some character. These separators are referred to as delimiters.
  • Each of those strings/words separated by delimiter is called as a Field.

Lets consider a familiar example to know about records, fields and delimiters, /etc/passwd file:

messagebus:x:107:111::/var/run/dbus:/bin/false
uuidd:x:108:112::/run/uuidd:/bin/false
sshd:x:110:65534::/var/run/sshd:/usr/sbin/nologin
foouser:x:1001:1001:,,,:/home/foouser:/bin/bash

In above file, each of the line is interpreted as a record. As each word/string is separated by a colon ( : ), it becomes a delimiter and each word separated by the delimiter i.e. foouser, 1001, /bin/bash, etc. are the fields.





In awk, we reference each field using $ operator, followed by a number or an awk variable. We learn more about awk variables in later articles to keep things simple here. Thus, we can reference first field from the record using $1, second field with $2, third field with $3 and so on. $0 is used to reference the record (or the input line).

Lets take a look at following example. We have an input file result.txt with contents as below [snipped]:

Student Subject Marks
James Biology 31
Velma Biology 43
Kibo Biology 81
Louis Biology 11
Phyllis Biology 18
Zenaida Biology 55
Gillian Biology 38
Constance Biology 16
Giselle Biology 73

We can see that there are 10 records and each record has 3 fields. Now we refer to each record and every field with their respective identifiers.

# Referencing first field
$ awk '{ print $1 }' result.txt
Student
James
Velma
Kibo
...
...

# Referencing second field
$ awk '{ print $2 }' result.txt
Subject
Biology
Biology
Biology
...
...

# Referencing third field
$ awk '{ print $3 }' result.txt 
Marks
31
43
81
...
...

# Referencing all fields
$ awk '{ print $3, $1, $2 }' result.txt
Marks Student Subject
31 James Biology
43 Velma Biology
81 Kibo Biology
...
...

# Referencing a record
$ awk '{ print $0 }' result.txt
Student Subject Marks
James Biology 31
Velma Biology 43
Kibo Biology 81
...
...

Field Separator

In above example, we have not used any field separator or delimiter anywhere in the awk command. So, it can be concluded that, awk considers whitespace as a default field separator. awk allows us to set a field separator of our own choice with -F option followed by the delimiter. Lets check this with /etc/passwd file, that has fields delimited by a colon.

# /etc/passwd file contents (snipped)
$ cat /etc/passwd
...
messagebus:x:107:111::/var/run/dbus:/bin/false
uuidd:x:108:112::/run/uuidd:/bin/false
sshd:x:110:65534::/var/run/sshd:/usr/sbin/nologin
foouser:x:1001:1001:,,,:/home/foouser:/bin/bash
...

$ awk -F ':' '{ print $3, $1, $7 }' /etc/passwd
...
107 messagebus /bin/false
108 uuidd /bin/false
110 sshd /usr/sbin/nologin
1001 foouser /bin/bash
...

While writing an awk script, we can change the field separator by using awk variable FS. We need to instruct awk to consider a custom delimiter before it start reading lines from input file. Here, BEGIN block comes handy. BEGIN block is executed before any input lines are read. Similarly, we have END block which gets executed once all of the lines from input file are read. Both BEGIN and END blocks are optional.

So, we can write an awk script passwd.awk as:

BEGIN { FS = ":" }
{
    print $3, $1, $7
}

As covered in our first tutorial (link), we can use the instructions from this script using option -f as below:

$ awk -f passwd.awk /etc/passwd
...
107 messagebus /bin/false
108 uuidd /bin/false
110 sshd /usr/sbin/nologin
1001 foouser /bin/bash
...

To make the output comprehensible, we can introduce a tab ( \t ) character between two output fields.

$ cat passwd.awk
BEGIN { FS = ":" }
{
    print $3 "\t" $1 "\t" $7
}

$ awk -f passwd.awk /etc/passwd
107	messagebus	/bin/false
108	uuidd	/bin/false
110	sshd	/usr/sbin/nologin
1001	foouser	/bin/bash

By default, all the instructions from the script are executed on every single line from the input file. To execute these instructions on selected lines, we can also introduce pattern matching by enclosing the regular expression within slashes ( /[REGEX]/ ). This will execute the instructions from awk script on only those lines matching the regex.

To verify this, we use our results.txt file again. From the entire list of students and their marks in certain subjects, we can filter only those records of students who got exactly 50 marks, whichever may be the subject. So, we can use 50 as the pattern to match, as shown below:

awk ' /50/ {print $1"\t"$2"\t"$3} ' result.txt 
Ori	Chemistry	50
Hyatt	Mathematics	50

Or we can filter only those records in which students who have their names starting with string Jo. For this, we can use a regex ^Jo with tilde ( ~ ) operator to match against first field ( $1 ) which is name of the student.

$ awk ' $1 ~ /^Jo/ { print $1"\t"$2"\t"$3 }' result.txt 
John	Biology	55
Jonas	Mathematics	40

Or we can negate the same using the bang or logical not operator ( ! ) as shown below (result is be too long, hence now shown):

$ awk ' $1 !~ /^Jo/ { print $1"\t"$2"\t"$3 }' result.txt

That's all for the scope of this article. Please share your feedback and suggestions in the comments section below and stay tuned for more articles on this topic.