Wednesday, 10 October 2018

How to Train a Computer to Bluff

poker game
The world's top Go player, Ke Jie, couldn’t get past Google’s AlphaGo AI. Image credits

In 1996, IBM’s Deep Blue became the first computer to beat a reigning world chess champion. In 2007, after almost 20 years of computation, Checkers was solved. In 2011, IBM’s Watson won in Jeopardy! In 2016, Google’s AlphaGo triumphed over the then world Go champion, only to repeat its success later against a player considered the best at the game.

During the past several decades, the field of artificial intelligence (AI) research had many “firsts,” each of which more impressive than the last one. In 2017, Noam Brown and Tuomas Sandholm from the Carnegie Mellon University (CMU) achieved the latest remarkable feat: they programmed an AI called Libratus who won a No-Limit Texas Hold ‘em heads-up tournament against four poker pros. The scientists’ work was outlined in a series of papers. Here’s a summary.

Libratus: To Err is Human and Makes Computers Smarter


poker game
Noam Brown (L) and one of Libratus’ opponents (R), poker player Daniel McAulay. Image credits

After Libratus’ triumph, the Computer Science Department at CMU published a paper titled “Endgame Solving in Large Imperfect-Information Games.” The paper, co-authored by Sandholm, starts with “The leading approach for computing strong game-theoretic strategies in large imperfect-information games is to first solve an abstracted version of the game offline, then perform a table lookup during gameplay.”

The first algorithm (called “counterfactual regret minimization”) that scientists “fed” Libratus allowed it to create a “blueprint” of possible events a poker player will face during play. But because “a popular variant of two-player no-limit Texas Hold ‘em has about 10,165 nodes [of possible outcomes],” the team behind Libratus bundled similar events, thus downsizing the range of possibilities to 1,012. For example, the initial Libratus blueprint of the game didn’t consider a king flush or a queen flush to be separate events that require an independent strategy. That allowed the AI to get an underlying sense of the game upon which to build through the other two algorithms.

The second-tier algorithm went into more detail. By using “nested subgame solving” (NSS), Libratus treated each round of poker as a subgame, part of the bigger game, fine-tuning its immediate play based on the blueprint and the opponent’s decisions. And while NSS has been successfully applied to AIs that have “conquered” complete information games, such as chess and Go, using this algorithm in poker required some tweaking.

The duo behind Libratus published a paper titled “Safe and Nested Subgame Solving for Imperfect-Information Games,” which detailed how they implemented NSS, building upon existing models and adapting them to the poker game. To get a more flexible gameplay out of Libratus, the researchers kept the trunk of the “game tree” (the initial stages of the game) from the first algorithm and combined it with the second algorithm, which went into more detail to solve the “endgame” (the last stage of the game). This way, the scientists taught Libratus to consider the best way of play for each round but to also keep in mind the long-term strategy. Thus, the AI consisted of both the initially generated blueprint and the developing actual gameplay. A significant implication of Brown and Sandholm’s work is that they “show that subgame solving can be repeated as the game progresses down the line, leading to significantly lower exploitability.”

While each of Libratus’ three algorithms worked in sync, the last one is arguably the most exciting part and one that kept the AI “awake” at night. The first two algorithms allowed Libratus to construct models and continually refine them based on how the game was progressing. But the third algorithm, employing machine learning, instructed Libratus to go over each event from the past day and learn from its mistakes.

The AI did so by analyzing how its human opponents managed to outplay it. “Typically, researchers develop algorithms that try to exploit the opponent’s weaknesses. In contrast, here the daily improvement is about algorithmically fixing holes in our own strategy,” said Sandholm. That makes Libratus invincible — not only is it able to calculate probabilities way beyond the scope of a human brain but also, learns from its mistakes. It was a fact that would be proved once again with Libratus’ successor a couple of months later.

Lengpudashi: The Computer That Bluffs


poker game
Entrepreneur and poker player “Yue” Du, who led the team against Libratus’ successor, was the first Chinese to win a WSOP bracelet. Image credits: PokerNews

Libratus proved that computers are no longer the linear machines that require continuous human overview to do their job. The poker-playing AI managed to perfect its operations after only receiving a limited set of data. And because in the field of AI one breakthrough quickly leads to another, a couple of months after the successful run against poker pros, the team behind Libratus came up with an updated version named Lengpudashi.

Lengpudashi is a refined version of an AI that already proved to be unbeatable by humans, so the players that were supposed to claim a victory for humans also received some perks. This time, the machine was facing a single team of six people — a mix of venture capitalists and computer scientists (including Brown and Sandholm) led by entrepreneur and poker player Alan “Yue” Du.

The approach to the game was also different: instead of trying to beat the AI in poker, the team set out to defeat Lengpudashi on the scientific field by attempting to exploit its vulnerabilities. Despite the combined brainpower of six able minds, Lengpudashi landed a crushing defeat, once again proving that computers are undisputed champions of mind games.

What’s more, Lengpudashi bluffed, prompting Brown to point out that “people have a misunderstanding of what computers and people are each capable of doing well. A computer can learn from experience that if it has a weak hand and it bluffs, it can make more money.” An equally impressive detail is that as per the researchers’ words, Lengpudashi learned to bluff not by studying other players but “comput[ing] from just the rules of the game.” In other words, Lengpudashi figured out that even if it didn’t know the decision to make, acting as if it did could help it win even more money by causing its opponents to fold.

What does it all mean?


poker game
From real-time translation to robotic surgeons, AI is reshaping every industry. Image credits

For starters, it does not mean that computers will take over poker rooms and chess boards, at least for now. However, scientists are using recreational games as a field for testing AI’s capabilities due to their complexity and reliance on various capabilities of the human brain believed to be unachievable by other species let alone a machine.

Making chess, poker, Go or Jeopardy “unfun” for humans is not the focus of research. Instead, computers are now widely used by players to hone their skills in these games. Uncountable are the other areas where AI prove useful. “Many real-world applications can be modeled as imperfect-information games, such as negotiations, business strategy, security interactions and auctions,” pointed out Brown and Sandholm in their paper titled “Libratus: The Superhuman AI for No-Limit Poker.”

Granted, AI is still quite resource-intensive, meaning that only a handful of entities can drive research further. For example, during its 20-day poker streak, Libratus used 19 million core computational hours (CCH) from 600 nodes part of Pittsburgh Supercomputing Center’s “Bridges” supercomputer. That’s a considerable load, especially compared to Libratus’ predecessor, the poker bot Claudicio, which used 2-3 million CCH. But this steep increase in resource demands is a testament to the long way that AI research is coming since its infancy only a couple of decades ago.

Thursday, 2 August 2018

openssl Command in Linux

In the earlier days, when people started to use the Internet, information was being transferred across the globe through HTTP (Hyper Text Transfer Protocol). HTTP was never safe as the information was being transferred in the plain text format and clould easily be available to the hacker who would intercept the communication. This information may consist of sensitive information including credit card details and passwords, which lead to increasing frauds. Hence, it was necessary to pass this information from one corner of the world to another through a secure channel, such that the information is not compromised. Thus, encryption came into the picture and the protocol that uses encryption while using HTTP - Secure Socket Layer (SSL). Since then, it is popularly known as HTTP over SSL or HTTPS.

Why HTTPS?

There are three main reasons:

  • Authenticity - HTTPS ensures that, the data exchange is happening with legitimate user, i.e. it checks the authenticity of the enduser it is connected to. If it is an intruder, it will drop the connection before any data exchange happens.
  • Data privacy - HTTPS encrypts the data such that only the intended enduser can decrypt and read it. Intruders will only be able to capture the encrypted message that is not readable.
  • Data Integrity - HTTPS ensures that the data received at receiver end is the same one which was sent at sender end, i.e. it has not changed/altered in the middle.

How HTTPS works?


  1. Browsers initiates a connection with the server
  2. Servers responds back with a public key (through SSL certificate), while private key is kept with the server itself
  3. Browser generates a random encryption key (session key) and encrypts it with public key
  4. Browser sends the encrypted session key to the server
  5. Server receives encrypted session key and decrypts it using private key, to get original session key
  6. Now, both the browser and the server has the same session key, that they can use to encrypt and decrypt the data
  7. Session key expires when the connection terminates

This article explain tips to generate Private Key, CSR (Certificate signing request) in linux using
OpenSSL command to obtain a Certificate Authority (CA) signed SSL certificate. CSR file used
submit to CA to complete order process of signed SSL certificate. You can get cheap SSL
certificate available at cheapsslshop.com.

Perform following steps to Generate private key, CSR for CA signed certificates in Linux using 'openssl' command

1. Generate a private key


Login to your Linux server and execute the following openssl command

openssl genrsa -des3 -out example.key 2048

Output:

# openssl genrsa -des3 -out example.key 2048
Generating RSA private key, 2048 bit long modulus
...................................................+++
.......+++
e is 65537 (0x10001)
Enter pass phrase for example.key:
Verifying - Enter pass phrase for example.key:

It will create a file example.key which is the private key. Lets look at the contents of this file.

# cat example.key 
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,5B72B5B1A445E6DD

KsIkmOuoagyG/Xs9JhA9qFbqpJjl73CvPWpUyeWUi80e9ZBQbjKJ9gkmKN5IQ/T6
fd/dOf3p5iMlnzjjq+4ST5Zuo/QW2DV1hffvgjTcC3KB0H44oC/Tjox3BfmBS+Km
cLpIKA5n+eysXowrc/uUhbdx1oCNUMUiiP/Cforn+vom+6VgMYbC6674CGPAGdqK
Ffy+NGErM58cUeko1/ax/Cj3ietlkTdCewR2hY4+4uDo3bfZMNMrYklkfy42XlcJ
nI5YzGhDSqB0cE0eblkekw5bbg77cRYdAk9L9onlirWek9ZA+EPZBtZA2S1xeJxA
iIbpg5cHn7oOoefvDkLrv75v3eNrqhP/wg6yFdQoi4edjG94SOpFqBTjueUAdF9p
Q4mA9PheARZ2V3bt4Mi5mgm7tfUwVNqVXwxmU+ErDHyxmpr1GzL2joqOUN0JQMIe
ahrYrcJBY93JK0+i7wlTuHg8ZkGCOCUXdyTbhlDgyfoey4tk9dA37R41N/F5IVPC
/AhZQa2UV7gMWdF18tsAesMpjQgR4OF/5kT03mPxiVNYUEHx4cRQ8JokMC4WlDwV
+ETJyjWToJeWIKAvHD2188MXjh/jMlP5JF/zRpFpEG6xZe2k9Cq8PGn+eM2SCwVH
8tLgQZzdrBL0p6H5ePrnXanVnOIiH09AL07/AuJrpcXNQSTEgUHkVI7wWbb5UbyL
hzG2Ky1X5+G3vgJeKz+UAe5P56qmhfhRHDzQg8FkU60nqCQi5DN1NhN7GeIh5r+n
lzyHEPQZc0lO0vYhXUTT2npCfNgDLvBLOFA8zphGR7hbIhi8t4MvJPM2z7IZuH0k
FPFmkgh15ByF75bLOuLMrOg1C74TbwNkKBihq8nGFtUywDUXcaot2b+xR0/KvULz
qZT/I8HlCUYqK7GfGHpPCGloqFOy/U2+XQ4JS1BR8NCxEun7Kk79KhdJo+d9eean
D62rpAlBdb8cx2j28pUsyhte7sQ9HP3BiujuctxN7XiFDVQsPZ5RpBFwqB+Ou2C1
4nn5AM80JWJGAdAjVnGSLgrPPJkAPOCu9T0sGZFemilismleDG5O7SfL9YXdwuqd
1jOW0W66heIsVrId751L4V+R2LDSztPS4VetwVAulxBvZHgkCJtGlzODZRX2SDob
X/xfwU/uZWoZgfgKM9S2ZIf4iMkIKnMo5tdzBQ3FrezB28BWFUtA5QyPmUWo2Txq
jWLcWuTn/shQl8d8TKGKd3c7TS0Fw9uJXaKW61RWB4clOsbSuwe/o9HDNdVz4uiI
xQkeSo+rGdeVm0ZTxtopOfMMtjWlOfQwp/SCYi947aOIK68wuQYnbViDR/loR2x/
yDb+0dGjpUDYI5m+G/R6XrjjSKlv1DkPgGcjWVoVxBOeTunPXhdab0O+fQIy5J69
LAWjpa5UCTcOY/IYZONy3KLCxy6irvy8xKJQA8LJQYR00s7f8Lz4k0Sgu6aA6E+v
7J9yZ6AfkYv9xzJFlD98nKQxDJImtGLe59/EuEZ/gnJCOHqyagTuyb1DtaI9neoR
FX1N656+PGieUfQvifaBajbNglvbijUqPvhUMs6D5Vpluo8YSOGcyg==
-----END RSA PRIVATE KEY-----

2. Generate a Certificate Signing Request (CSR)


Command:

openssl req -new -out example.csr -key example.key

Output:

# openssl req -new -out example.csr -key example.key 
Enter pass phrase for example.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:MH
Locality Name (eg, city) []:MUMBAI
Organization Name (eg, company) [Internet Widgits Pty Ltd]:EXAMPLE
Organizational Unit Name (eg, section) []:TESTUNIT
Common Name (e.g. server FQDN or YOUR name) []:example.com
Email Address []:user@example.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:      
An optional company name []:EXAMPLE

It will create a file example.csr which is the CSR. Its contents will look like :

-----BEGIN CERTIFICATE REQUEST-----
MIIC5TCCAc0CAQAwgYcxCzAJBgNVBAYTAklOMQswCQYDVQQIDAJNSDEPMA0GA1UE
BwwGTVVNQkFJMRAwDgYDVQQKDAdFWEFNUExFMREwDwYDVQQLDAhURVNUVU5JVDEU
MBIGA1UEAwwLZXhhbXBsZS5jb20xHzAdBgkqhkiG9w0BCQEWEHVzZXJAZXhhbXBs
ZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDlGSFQt0sR4kJC
/39NtTKNNObIRESPpXABZMoQrH0opZROFgsNHMu1Y28/tMIxzENrytPRuOcmsA+R
wMyd9KUjrLZSXoNTwGVJQsyU2gII66fnoXHOIaTHm6YP/a4KuYMDW4jLl5TugNqh
4TjZ/cP4cDuB4dEEuq7X33YBktVH8OMxiel8P8Nh8cU6xBQRZyubluWQwwX6hMiC
lOKkHKMIzx54l1QJ3EiVY1n0N44Q4AIR7xzULY1bghSWdvqi4Tu88NJh/Xy2whWv
ybJiN7o8d4P5qxP9+uHjFWeLcMd1w2G53wdsALvphEUcmd4FwrEiZTjHJoTIIz8h
8EShqq0hAgMBAAGgGDAWBgkqhkiG9w0BCQIxCQwHRVhBTVBMRTANBgkqhkiG9w0B
AQsFAAOCAQEAJlhA5unLl0e4yk5Is0P4LUU2VkZ8nn2osTAo/AYkFblGddxzQFL3
7iy8sfC7lLK9UysG8hmbo8jHSuEaAmH5lkBzlxde2SavBPbafZ8UovjHvdbrZmi9
8MTdVHDPA5KaV6Z7wM1I6Qtm17Q0gx+YppvBv6oyZINs2wcU7KKcixJTE9MyPiFK
VIKvk+OCNXPuDypA6Z90+LNBbrWBvqgRmL4PI7QjLynndhhZoQHTIZao8WFZtx4v
vQF7d0XwHgIUqlyc0bjSvwc6Yx2Q2wWRtFAM3h64XdFt8KtfIOmv0Tshe88bvun0
pkMnhxwY1CyFUKfNNYHsNHZahDenIQP90g==
-----END CERTIFICATE REQUEST-----

Conversions using openssl

1. Convert crt to pem


Command:

openssl x509 -in example.crt -out example.pem -outform PEM

This will generate a file example.pem whose contents are the same as that of example.crt. So, effectively it just copying the exmaple.crt file as example.pem.

2. Convert to p12


This requires example.crt file to be converted to example.pem and you know how to do it :)

Command:

openssl pkcs12 -export -out example.p12 -in example.pem -inkey example.key 

Output:

# openssl pkcs12 -export -out example.p12 -in example.pem -inkey example.key 
Enter pass phrase for example.key:
Enter Export Password:
Verifying - Enter Export Password:

3. Extracting key from p12 file


Command:

openssl pkcs12 -in example.p12 -nocerts -out example2.key

Output:

# openssl pkcs12 -in example.p12 -nocerts -out example2.key
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

This generates a key example2.key from the p12 file.

4. Extracting crt from p12 file


Command:

openssl pkcs12 -in example.p12 -clcerts -nokeys -out example2.crt

Output:

# openssl pkcs12 -in example.p12 -clcerts -nokeys -out example2.crt
Enter Import Password:
MAC verified OK

It generates a crt file example2.crt whose contents can be displayed:

Bag Attributes
    localKeyID: 6C B6 C7 C8 85 56 86 38 46 A8 C9 27 0F 7A 72 8D A8 D5 C7 CF 
subject=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
issuer=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
-----BEGIN CERTIFICATE-----
MIIDjDCCAnQCCQCZctLzit9LpjANBgkqhkiG9w0BAQsFADCBhzELMAkGA1UEBhMC
SU4xCzAJBgNVBAgMAk1IMQ8wDQYDVQQHDAZNVU1CQUkxEDAOBgNVBAoMB0VYQU1Q
TEUxETAPBgNVBAsMCFRFU1RVTklUMRQwEgYDVQQDDAtleGFtcGxlLmNvbTEfMB0G
CSqGSIb3DQEJARYQdXNlckBleGFtcGxlLmNvbTAeFw0xODA4MDIxNTI0MjJaFw0x
OTA4MDIxNTI0MjJaMIGHMQswCQYDVQQGEwJJTjELMAkGA1UECAwCTUgxDzANBgNV
BAcMBk1VTUJBSTEQMA4GA1UECgwHRVhBTVBMRTERMA8GA1UECwwIVEVTVFVOSVQx
FDASBgNVBAMMC2V4YW1wbGUuY29tMR8wHQYJKoZIhvcNAQkBFhB1c2VyQGV4YW1w
bGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5RkhULdLEeJC
Qv9/TbUyjTTmyEREj6VwAWTKEKx9KKWUThYLDRzLtWNvP7TCMcxDa8rT0bjnJrAP
kcDMnfSlI6y2Ul6DU8BlSULMlNoCCOun56FxziGkx5umD/2uCrmDA1uIy5eU7oDa
oeE42f3D+HA7geHRBLqu1992AZLVR/DjMYnpfD/DYfHFOsQUEWcrm5blkMMF+oTI
gpTipByjCM8eeJdUCdxIlWNZ9DeOEOACEe8c1C2NW4IUlnb6ouE7vPDSYf18tsIV
r8myYje6PHeD+asT/frh4xVni3DHdcNhud8HbAC76YRFHJneBcKxImU4xyaEyCM/
IfBEoaqtIQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBvp6MetgOlXC2SbOCz7oxe
XQhAsIzxoxb57uzkGK3fnyE/Bdj+08IscosO2+2CGyS0v/hKAql+aoiCb/TBS3CP
E1zhdRkIruUBmntOy3wbW2lSZAv/bg8rWicrvzuj4QGFL8oCIjBgEH4tmlgmmwzE
rbSqzQXppSZ9YeyCKkVkNOsCldBEFUkDo+hZ5cD3F71oiM/vWjQWJpXWI/ifSRSN
BhnljhPEXFzL+GBMMgVSthKccy5WNj7UBeKK6RhjruSLp2HnEYZWnQauXNz2IDzY
WFEWDdBalp6q+1WhTR7cYFaGdc6ae6jKwvMW0Y5fpPSYl8zwwePTX6xyb1bz0YTx
-----END CERTIFICATE-----

Verification with openssl

Verify a private key


Command:

openssl rsa -in example.key -check

Output:

# openssl rsa -in example.key -check
Enter pass phrase for example.key:
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA5RkhULdLEeJCQv9/TbUyjTTmyEREj6VwAWTKEKx9KKWUThYL
DRzLtWNvP7TCMcxDa8rT0bjnJrAPkcDMnfSlI6y2Ul6DU8BlSULMlNoCCOun56Fx
ziGkx5umD/2uCrmDA1uIy5eU7oDaoeE42f3D+HA7geHRBLqu1992AZLVR/DjMYnp
fD/DYfHFOsQUEWcrm5blkMMF+oTIgpTipByjCM8eeJdUCdxIlWNZ9DeOEOACEe8c
1C2NW4IUlnb6ouE7vPDSYf18tsIVr8myYje6PHeD+asT/frh4xVni3DHdcNhud8H
bAC76YRFHJneBcKxImU4xyaEyCM/IfBEoaqtIQIDAQABAoIBAEggfk2kN109B7GC
MPktF+o/An2AiLlU04uMyNUxCQJ4BtYCuJ37N+M2l/rNlOoKrhtMsx1vAk861NwF
gCYmtKsjoA8UtHH0u66x+ijO7h8S2jzhpivIidFSHpkgO+Aiga3X7pxyb7AbHzoh
5z0yWLVp2EJO7vh3Mb7DGqrqKH3KL5rlRZeQ5+DvF3bFxoCLjNO7uOaPsLNf8vnx
8VGzjuts47LUL4Gtn/92FuOCsiRN1yhhd9yC9fnQ6B7R6fMcu6JSKyMHqp2xMlvO
8/cPUIHgoAOUQm3rLBJywsY7YGViq712qIZJRo2OT0XFCiAgDFQXSuYulg2/akY5
xEqG4sECgYEA+irSjBinHY7iKnAKGq5zbt/GWTyDd5tLa7MEBpVkrWMbTc7kMh3m
LCyYMtXEp0J+F+hbKORKnUY6I9ESTEcvGeCRYYYeghemNdTQK3xZmC5K1Ie1SLxs
BYYAT17O9OwxlG7yElobf689/kbbS6m99lkLAJPIlkkUTXVfJFpCX7kCgYEA6nCN
WPyI9Zr+IVakBbDLfueOVbvBTvty5z2Owm2sWdx+1ZeS9/Y+0mLi+HKWg87qjPfn
Wg/WZoBSAO3EKH7TKeZs2PQnkO5+QFjRdEMRZJsVLMGmHTdwSg+zOQqY1Owrak41
Z1q0OnfHfRD5GSmuC+qL8rnrDKeDRlI6kPiLXKkCgYAX/bVVZCParNI5uabuah59
2o3+DtSYytbPzEoti3QtQJzkuFugBsgFIn2yGlgSpkjLaCgd4s7ZCFwZBTrY/9af
h67JiIAyf6wb633PPcyl0IKyoRUclZ1SZkhvVCtKx7/1eTJT2jpa42ZxlUAAoJWs
I1vvwTxyVS3SQ9hM9y74WQKBgDMNz7pRRII3/p/D/nQ+zBIW2yhQewh4Oc2h2jT9
LUtkeTqKh7b4KTYn0sXILn8F0I5ibj9us4Ie77zECrPG6rV1OL8GbJdjWJsqvHJV
KLSAEVxtz38NU6bNRHpnlGDKDapY3chkOFuDOi5CQ/z21rBBo7h4RHe9AxmNTrPD
bJYBAoGANmFJjJyAkNVEdSGdWCv9iDQhkP3TRsE0hFICEniptHL/hmZzz1GfFdTL
4F53AQ9Prl+zcxjQCL4uenNFieYDFAej/bVlEIP6dI6GYKYCMOaWtoG5sByhq2sZ
oAmtuXsC5dJFHTahxMqDOcxFoHgmhn0lktQ6VRTDEBaa3x6/62w=
-----END RSA PRIVATE KEY-----

2. Verify a certificate


Command:

openssl x509 -in example.crt -noout -text

Output:

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 11057131978596764582 (0x9972d2f38adf4ba6)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=IN, ST=MH, L=MUMBAI, O=EXAMPLE, OU=TESTUNIT, CN=example.com/emailAddress=user@example.com
        Validity
            Not Before: Aug  2 15:24:22 2018 GMT
            Not After : Aug  2 15:24:22 2019 GMT
        Subject: C=IN, ST=MH, L=MUMBAI, O=EXAMPLE, OU=TESTUNIT, CN=example.com/emailAddress=user@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e5:19:21:50:b7:4b:11:e2:42:42:ff:7f:4d:b5:
                    32:8d:34:e6:c8:44:44:8f:a5:70:01:64:ca:10:ac:
                    7d:28:a5:94:4e:16:0b:0d:1c:cb:b5:63:6f:3f:b4:
                    c2:31:cc:43:6b:ca:d3:d1:b8:e7:26:b0:0f:91:c0:
                    cc:9d:f4:a5:23:ac:b6:52:5e:83:53:c0:65:49:42:
                    cc:94:da:02:08:eb:a7:e7:a1:71:ce:21:a4:c7:9b:
                    a6:0f:fd:ae:0a:b9:83:03:5b:88:cb:97:94:ee:80:
                    da:a1:e1:38:d9:fd:c3:f8:70:3b:81:e1:d1:04:ba:
                    ae:d7:df:76:01:92:d5:47:f0:e3:31:89:e9:7c:3f:
                    c3:61:f1:c5:3a:c4:14:11:67:2b:9b:96:e5:90:c3:
                    05:fa:84:c8:82:94:e2:a4:1c:a3:08:cf:1e:78:97:
                    54:09:dc:48:95:63:59:f4:37:8e:10:e0:02:11:ef:
                    1c:d4:2d:8d:5b:82:14:96:76:fa:a2:e1:3b:bc:f0:
                    d2:61:fd:7c:b6:c2:15:af:c9:b2:62:37:ba:3c:77:
                    83:f9:ab:13:fd:fa:e1:e3:15:67:8b:70:c7:75:c3:
                    61:b9:df:07:6c:00:bb:e9:84:45:1c:99:de:05:c2:
                    b1:22:65:38:c7:26:84:c8:23:3f:21:f0:44:a1:aa:
                    ad:21
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         6f:a7:a3:1e:b6:03:a5:5c:2d:92:6c:e0:b3:ee:8c:5e:5d:08:
         40:b0:8c:f1:a3:16:f9:ee:ec:e4:18:ad:df:9f:21:3f:05:d8:
         fe:d3:c2:2c:72:8b:0e:db:ed:82:1b:24:b4:bf:f8:4a:02:a9:
         7e:6a:88:82:6f:f4:c1:4b:70:8f:13:5c:e1:75:19:08:ae:e5:
         01:9a:7b:4e:cb:7c:1b:5b:69:52:64:0b:ff:6e:0f:2b:5a:27:
         2b:bf:3b:a3:e1:01:85:2f:ca:02:22:30:60:10:7e:2d:9a:58:
         26:9b:0c:c4:ad:b4:aa:cd:05:e9:a5:26:7d:61:ec:82:2a:45:
         64:34:eb:02:95:d0:44:15:49:03:a3:e8:59:e5:c0:f7:17:bd:
         68:88:cf:ef:5a:34:16:26:95:d6:23:f8:9f:49:14:8d:06:19:
         e5:8e:13:c4:5c:5c:cb:f8:60:4c:32:05:52:b6:12:9c:73:2e:
         56:36:3e:d4:05:e2:8a:e9:18:63:ae:e4:8b:a7:61:e7:11:86:
         56:9d:06:ae:5c:dc:f6:20:3c:d8:58:51:16:0d:d0:5a:96:9e:
         aa:fb:55:a1:4d:1e:dc:60:56:86:75:ce:9a:7b:a8:ca:c2:f3:
         16:d1:8e:5f:a4:f4:98:97:cc:f0:c1:e3:d3:5f:ac:72:6f:56:
         f3:d1:84:f1

3. Verify a CSR


Command:

openssl req -in example.csr -noout -text -verify

Output:

# openssl req -in example.csr -noout -text -verify
verify OK
Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: C=IN, ST=MH, L=MUMBAI, O=EXAMPLE, OU=TESTUNIT, CN=example.com/emailAddress=user@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e5:19:21:50:b7:4b:11:e2:42:42:ff:7f:4d:b5:
                    32:8d:34:e6:c8:44:44:8f:a5:70:01:64:ca:10:ac:
                    7d:28:a5:94:4e:16:0b:0d:1c:cb:b5:63:6f:3f:b4:
                    c2:31:cc:43:6b:ca:d3:d1:b8:e7:26:b0:0f:91:c0:
                    cc:9d:f4:a5:23:ac:b6:52:5e:83:53:c0:65:49:42:
                    cc:94:da:02:08:eb:a7:e7:a1:71:ce:21:a4:c7:9b:
                    a6:0f:fd:ae:0a:b9:83:03:5b:88:cb:97:94:ee:80:
                    da:a1:e1:38:d9:fd:c3:f8:70:3b:81:e1:d1:04:ba:
                    ae:d7:df:76:01:92:d5:47:f0:e3:31:89:e9:7c:3f:
                    c3:61:f1:c5:3a:c4:14:11:67:2b:9b:96:e5:90:c3:
                    05:fa:84:c8:82:94:e2:a4:1c:a3:08:cf:1e:78:97:
                    54:09:dc:48:95:63:59:f4:37:8e:10:e0:02:11:ef:
                    1c:d4:2d:8d:5b:82:14:96:76:fa:a2:e1:3b:bc:f0:
                    d2:61:fd:7c:b6:c2:15:af:c9:b2:62:37:ba:3c:77:
                    83:f9:ab:13:fd:fa:e1:e3:15:67:8b:70:c7:75:c3:
                    61:b9:df:07:6c:00:bb:e9:84:45:1c:99:de:05:c2:
                    b1:22:65:38:c7:26:84:c8:23:3f:21:f0:44:a1:aa:
                    ad:21
                Exponent: 65537 (0x10001)
...
...

4. Verify a pem file


Command:

openssl pkcs12 -in example.p12 -info

Output:


# openssl pkcs12 -in example.p12 -info
Enter Import Password:
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: 6C B6 C7 C8 85 56 86 38 46 A8 C9 27 0F 7A 72 8D A8 D5 C7 CF 
subject=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
issuer=/C=IN/ST=MH/L=MUMBAI/O=EXAMPLE/OU=TESTUNIT/CN=example.com/emailAddress=user@example.com
-----BEGIN CERTIFICATE-----
MIIDjDCCAnQCCQCZctLzit9LpjANBgkqhkiG9w0BAQsFADCBhzELMAkGA1UEBhMC
SU4xCzAJBgNVBAgMAk1IMQ8wDQYDVQQHDAZNVU1CQUkxEDAOBgNVBAoMB0VYQU1Q
TEUxETAPBgNVBAsMCFRFU1RVTklUMRQwEgYDVQQDDAtleGFtcGxlLmNvbTEfMB0G
CSqGSIb3DQEJARYQdXNlckBleGFtcGxlLmNvbTAeFw0xODA4MDIxNTI0MjJaFw0x
OTA4MDIxNTI0MjJaMIGHMQswCQYDVQQGEwJJTjELMAkGA1UECAwCTUgxDzANBgNV
BAcMBk1VTUJBSTEQMA4GA1UECgwHRVhBTVBMRTERMA8GA1UECwwIVEVTVFVOSVQx
FDASBgNVBAMMC2V4YW1wbGUuY29tMR8wHQYJKoZIhvcNAQkBFhB1c2VyQGV4YW1w
bGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5RkhULdLEeJC
Qv9/TbUyjTTmyEREj6VwAWTKEKx9KKWUThYLDRzLtWNvP7TCMcxDa8rT0bjnJrAP
kcDMnfSlI6y2Ul6DU8BlSULMlNoCCOun56FxziGkx5umD/2uCrmDA1uIy5eU7oDa
oeE42f3D+HA7geHRBLqu1992AZLVR/DjMYnpfD/DYfHFOsQUEWcrm5blkMMF+oTI
gpTipByjCM8eeJdUCdxIlWNZ9DeOEOACEe8c1C2NW4IUlnb6ouE7vPDSYf18tsIV
r8myYje6PHeD+asT/frh4xVni3DHdcNhud8HbAC76YRFHJneBcKxImU4xyaEyCM/
IfBEoaqtIQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBvp6MetgOlXC2SbOCz7oxe
XQhAsIzxoxb57uzkGK3fnyE/Bdj+08IscosO2+2CGyS0v/hKAql+aoiCb/TBS3CP
E1zhdRkIruUBmntOy3wbW2lSZAv/bg8rWicrvzuj4QGFL8oCIjBgEH4tmlgmmwzE
rbSqzQXppSZ9YeyCKkVkNOsCldBEFUkDo+hZ5cD3F71oiM/vWjQWJpXWI/ifSRSN
BhnljhPEXFzL+GBMMgVSthKccy5WNj7UBeKK6RhjruSLp2HnEYZWnQauXNz2IDzY
WFEWDdBalp6q+1WhTR7cYFaGdc6ae6jKwvMW0Y5fpPSYl8zwwePTX6xyb1bz0YTx
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Bag Attributes
    localKeyID: 6C B6 C7 C8 85 56 86 38 46 A8 C9 27 0F 7A 72 8D A8 D5 C7 CF 
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQISSIn7yGZN+8CAggA
MBQGCCqGSIb3DQMHBAhxEHorIofSVwSCBMjM/SBuC6Qeh9LE6YcLuWleYSC/21Yi
52i/UW81AbSqVcFU7zkgB4ro/1s/WWdrFiohpSXhZUq3/kXoPJlXrU5yQK0zoDaE
Lxz/EHb1KH2TXiLhyIPEB/LL8kVMHIQm7SO94p2Gl3JuP4j1hmxtQwlDpmCc3ewh
7GEK6a5ggazfUhMY6nim9Sx8qe7pHCApFYo72FTl/KlhzexKso1JexsCkc7chD5j
87fcW62+L9dI7SgarGabXtzSlpUQ+TW3Wz16ix2TE0t31DAlDuuK5Qpcv91gIuBB
AL00QGWUQg92WA+4KZJ4WoLYqKgDrgOyWCgrMg2Z+2hhcpjV29hE5D5uWJRdoIz5
tvkrTk8+M/xfs3gfB52ahGO1ttS9zqnffo81xshCPI9d2cU4uFN0sa7hH31rrxtz
Cp4GYWtmxoA7nBg8pyiwk5iQLRwUMDkAN6XtL5uVewhjwmn2Bsv2ouHUJi4mD3D9
zZcYYvbTKKQ7gJ/8gy6/trtuSMcW3tgDGCwpkaRl7YEsXNYkVd2k5jSqAoLLy0tL
M9uGahD76MDq4XVTl3BH7Qt2vVnTqtEaPPue8nBTSnULFhHBy+ZYmmWtZImWsp2C
tYxST09n/MLhm1hkBC7STmJJlaubEdjDy2zPLF0ONwy522xHjqcOQyd/RwoU0hVe
rqR//fmGEHD3Hd/dANMDmvcDwDtsHMJm09WmF2qt23M7MzGI1HGj1/ZPbRG5vxrd
LVP1rd5nFMwJED5xQRlbDw7E8Qxu1pEEeyFDs9uYcE8lmyCzwqmzCPkkT0rwPzNa
e2jLiI2dnBVJbJj9Ti4UuPyCYo3bjqkfksa9sY/EAUrB2Hgs0LA1xCTuHeRfBDfV
yYf5izdIrmtzqDPbNgY0Jel20KRPXqN4JVSaZ9ANQOPQvCFSIywQAZm//jJSXjNY
fTpVYSTV/tvgESYe2lYME9Vfl/10n2pt92NUI+8zIWgG1t2Ad+bSn/M1WIv4oXPb
OEhti+Y0VfhcAnOiP5p1rKTc3ftVlYhnJ0ug6Yw5E0Xj4Nfnewu/JFSumg95Lw/T
eICm/Lzt6f6zldsNlCW9qxBaHosCDr5QZlYQYTSAOQzCn0+K0KZL46jN7NoAYvUD
tjkHLkJDxhP6wXMa5Bdz4XCHnUeS4VSoe2ZGSwVwNYs8c4P00VjutAovmNFL5wQx
rUuadq3L0EOja02/25qmtl/xAtgZd87k7hwCz+Ux3BduclTF2P5Ge2bk4x6uaX4p
jwY5R8Crm20cxhyLXP1QEXUq9rlo0qXi48m/ll7diH4ipRDcryUT8p9odrw1O534
UVJtu8oQNErvZsHLJZTWvyQ97KthKCzWsBePI3F4gI9k8YBQ4TK3vKZ2iPVdMHO6
h9yMoihYKFyYzN02SRdqGHdYkM39dN169bUjykgYpdXuVIoXXuHyIn6tmx7TDtyi
/qWFa2jgK8zJmU1ytv06KPzYyQx3EhqDiNCrmJXyqXa8/QETKs9PnSCcDgtp3aLC
vsXS6s+pH0jlt14aY05aJOj15dlsgBztMQ2h1alKm2P6WkbduN2BXdBwv/CGS46U
fBckbGkbGufmT7vpIUyzDvWR0G7Lj26jqJecJF9v3+gxud+r/U1Sk1XMyl77yryp
mQA=
-----END ENCRYPTED PRIVATE KEY-----


Tuesday, 10 April 2018

AWK Programming Tutorial- Awk built-in variables FS, OFS, RS, ORS, NF, NR

Awk built-in variables: This is the fourth article of this tutorial series on awk and in this one, we will be learning about built-in variables in awk. In case you have missed any of our previous articles, you can find them out here.


Awk comes up with a number of built-in variables. Of these variables, some have a default value associated with them which can be changed e.g. FS ( field separator, with default value of a whitespace ) and RS ( record separator, with default value of \n ). While, some variables are quite useful while doing analysis or creating reports e.g. NF ( number of fields ) and NR ( number of records ). Lets take a look at them one-by-one.

FS (Field Separator) and OFS (Output Field Separator)

  • With FS, we instruct awk that, in a particular input file, fields are separated by some character.
  • Default value if this variable is a whitespace, telling awk that fields are separated by one or more whitespaces (including tabs).
  • This default value can be overwritten with a character or a regular expression. For example, we can use a colon ( : ) to separate fields while working on /etc/passwd file.
  • With OFS, we ask awk to use a particular character to separate the fields in the output.
  • For this variable too, default value is a single whitespace.
Lets take a look at an example now. For this, we will use demo csv file with contents as shown below:

1,"Eldon Base for stackable storage shelf, platinum",Muhammed MacIntyre,3,-213.25,38.94,35,Nunavut,Storage & Organization,0.8
2,"1.7 Cubic Foot Compact ""Cube"" Office Refrigerators",Barry French,293,457.81,208.16,68.02,Nunavut,Appliances,0.58
3,"Cardinal Slant-DÆ Ring Binder, Heavy Gauge Vinyl",Barry French,293,46.71,8.69,2.99,Nunavut,Binders and Binder Accessories,0.39
4,R380,Clay Rozendal,483,1198.97,195.99,3.99,Nunavut,Telephones and Communication,0.58
5,Holmes HEPA Air Purifier,Carlos Soltero,515,30.94,21.78,5.94,Nunavut,Appliances,0.5
6,G.E. Longer-Life Indoor Recessed Floodlight Bulbs,Carlos Soltero,515,4.43,6.64,4.95,Nunavut,Office Furnishings,0.37
7,"Angle-D Binders with Locking Rings, Label Holders",Carl Jackson,613,-54.04,7.3,7.72,Nunavut,Binders and Binder Accessories,0.38
8,"SAFCO Mobile Desk Side File, Wire Frame",Carl Jackson,613,127.70,42.76,6.22,Nunavut,Storage & Organization,
9,"SAFCO Commercial Wire Shelving, Black",Monica Federle,643,-695.26,138.14,35,Nunavut,Storage & Organization,
10,Xerox 198,Dorothy Badders,678,-226.36,4.98,8.33,Nunavut,Paper,0.38

By default FS will use whitespace as a default value. Lets check extracting 1st and 3rd column without default value of FS.

$ awk '{ print $1, $3 }' input.csv 
1,"Eldon for
2,"1.7 Foot
3,"Cardinal Ring
4,R380,Clay and
5,Holmes Air
6,G.E. Indoor
7,"Angle-D with
8,"SAFCO Desk
9,"SAFCO Wire
10,Xerox Badders,678,-226.36,4.98,8.33,Nunavut,Paper,0.38

And now, using comma ( , ) as the field separator value.

$ awk 'BEGIN { FS = ","; } { print $1, $3 }' input.csv
1  platinum"
2 Barry French
3  Heavy Gauge Vinyl"
4 Clay Rozendal
5 Carlos Soltero
6 Carlos Soltero
7  Label Holders"
8  Wire Frame"
9  Black"
10 Dorothy Badders

As we can see in above outputs, awk uses the default value of OFS which is a single whitespace. We can overwrite this value, to say a pipe ( | ) as shown in below example:

$ awk 'BEGIN { FS = ","; OFS = "|" } { print $1, $3 }' input.csv
1| platinum"
2|Barry French
3| Heavy Gauge Vinyl"
4|Clay Rozendal
5|Carlos Soltero
6|Carlos Soltero
7| Label Holders"
8| Wire Frame"
9| Black"
10|Dorothy Badders





RS (Record Separator) and ORS (Output Record Separator)

  • RS and ORS are useful while dealing with multi-line records. In this case, each field is on a new line.
  • Default value of both these variables is a newline character ( \n ).
  • With ORS value overwritten, we can tell awk to separate records with some other character then the newline.
Lets take a look at our demo file wherein each record is separated by dual newlines ( \n\n ) and each field in the record is separated using single newline character ( \n ).

$ cat address.txt 
Cecilia Chapman
711-2880 Nulla St.
Mankato Mississippi 96522
(257) 563-7401

Iris Watson
P.O. Box 283 8562 Fusce Rd.
Frederick Nebraska 20620
(372) 587-2335

Celeste Slater
606-3727 Ullamcorper. Street
Roseville NH 11523
(786) 713-8616

Theodore Lowe
Ap #867-859 Sit Rd.
Azusa New York 39531
(793) 151-6230

Now, to display a person's name ( $1 ) and his/her phone number ( $4 ) on a separate line ( ORS will be \n, while RS is \n\n ), we can use below command:

$ awk ' BEGIN { FS = "\n"; RS = "\n\n"; ORS = "\n" } { print $1, $4 } ' address.txt 
Cecilia Chapman (257) 563-7401
Iris Watson (372) 587-2335
Celeste Slater (786) 713-8616
Theodore Lowe (793) 151-6230

NF (Number of Fields) and NR (Number of Record)

  • Awk variable NF defines the number of fields if the current record ( $0 ).
  • If we try to increase the value of NF, awk adds additional fields separated by the delimiter value in OFS.
  • Whereas, when we decrease the value of NF, all the fields with identifiers greater than the value are ignored.
  • NR is the variable that stores the current record number being processed by awk.
  • There is another variable, FNR, which is useful while dealing with multiple files. It stores the position of the record relative to the current file only.
Lets take a look at below demo file to illustrate this example. If you observe, it has different number of fields on each record.

$ cat cities.txt 
Washington 18 23 21 19
London 10 7 13 5 -1
Moscow 2 0 -3
Mumbai 24 27

Now, we print number of fields a record has before printing the record itself, using below command:

$ awk '{print NF, $0}' cities.txt 
5 Washington 18 23 21 19
6 London 10 7 13 5 -1
4 Moscow 2 0 -3
3 Mumbai 24 27

To illustrate the use of NR, we use the same file again. Its pretty straight forward.

$ awk '{print NR, $0}' cities.txt 
1 Washington 18 23 21 19
2 London 10 7 13 5 -1
3 Moscow 2 0 -3
4 Mumbai 24 27

In case there are multiple files, we can print the record number relative to the current input file being processed using the variable FNR.

$ awk '{print FNR, $0}' cities.txt address.txt 
1 Washington 18 23 21 19
2 London 10 7 13 5 -1
3 Moscow 2 0 -3
4 Mumbai 24 27
1 Cecilia Chapman
2 711-2880 Nulla St.
3 Mankato Mississippi 96522
4 (257) 563-7401
5 
6 Iris Watson
7 P.O. Box 283 8562 Fusce Rd.
8 Frederick Nebraska 20620
9 (372) 587-2335
10 
11 Celeste Slater
12 606-3727 Ullamcorper. Street
13 Roseville NH 11523
14 (786) 713-8616
15 
16 Theodore Lowe
17 Ap #867-859 Sit Rd.
18 Azusa New York 39531
19 (793) 151-6230

Observe the line after line #4. Awk has numbered it #1, just because we have used FNR. Had we used NR here, it would have been numbered #5. You can check this out, I will leave this for you.

That's it for the scope of this article. Please share your feedback and suggestions in the comments section below and stay tuned for more articles. Thanks for reading.

Friday, 6 April 2018

How To: Install or Upgrade to Linux Kernel 4.12 in Ubuntu/Linux Mint

The Linux Kernel 4.12 is available for the users. This Linux Kernel version comes with plenty of fixes and improvements. This article will guide you to install or upgrade to Linux Kernel 4.12 in your Ubuntu or Linux Mint system.

Installation

For 32-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200_4.12.0-041200.201707022031_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200-generic_4.12.0-041200.201707022031_i386.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-image-4.12.0-041200-generic_4.12.0-041200.201707022031_i386.deb

Install them.

$ sudo dpkg -i linux-headers-4.12.0*.deb linux-image-4.12.0*.deb

Reboot the system.

$ sudo reboot





For 64-Bit Systems

Download the .deb packages.

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200_4.12.0-041200.201707022031_all.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-headers-4.12.0-041200-generic_4.12.0-041200.201707022031_amd64.deb

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12/linux-image-4.12.0-041200-generic_4.12.0-041200.201707022031_amd64.deb

Install them.

$ sudo dpkg -i linux-headers-4.12.0*.deb linux-image-4.12.0*.deb

Reboot the system.

$ sudo reboot

To uninstall,

$ sudo apt-get remove 'linux-headers-4.12.0*' 'linux-image-4.12.0*'

Friday, 30 March 2018

AWK Programming Tutorial - Constants, Variables and Arithmetic Operators

Constants, Variables and Operators in Awk : So far in this tutorial series on awk programming, we have learned to print stuff, we learned about fields, records and delimiters and how to they are referenced. These are very basic operations as we are just extracting data from input lines and printing them. Now, we move a step ahead and manipulate the extracted fields by performing some common arithmetic operations on them. Before we proceed, I recommend you to please go through the third article published on awk - Field separator and Field references.


Constants

There are only two types of constants:
A string constant

  • A string constant is always surrounded within quotes, e.g. "Pineapple", "5_days", etc.
  • String can use escape sequences like \n ( newline ), \t ( horizontal tab ), \b ( backspace ), \v ( vertical tab ), \" ( double quotes ), etc.
A numeric constant

  • A numeric constant is a number without quotes, enough said.
  • A number enclosed within quotes is considered as a string.
Variables

  • A variable is an identifier that references to the memory location that stores a value.
  • We can initialize a variable by assigning a value to it, using = operator, e.g., age = 20, firstName = "Eric", etc. Here, age and firstName are variables.
  • A variable name consist of alphabets, digits and underscores, and it must start with a letter or underscore.
  • Variables are case-sensitive. It means, Age, age and AGE are different variables and they can store different values in them that won't get overwritten.
  • Variable initialization is optional. If we do not initialize a variable, awk defaults the value to numeric 0 or a blank string ( "" ) appropriately.
  • When we assign two or more strings separated by space to a variable, it stores a concatenated value
  • We can assign a field value to a variable using field reference variables $1, $2, etc.
Example:

# Assign a numeric value to variable 'myNum'
myNum = 10

# Assign a string value to variable 'myStr'
myStr = "awk!"

# Space concatenates the strings, so 'myVar' stores the value "AwesomeAwk!"
myVar = "Awesome" "Awk!"

# Assign a field value to a variable using field reference variable
marks = $1





Arithmetic Operators

awk supports basic arithmetic operators to be used in expressions, which are listed as below:

Operator Description
+ Addition
- Subtraction
* Multiplication
/ Division
% Modulus
^ or ** Exponentiation

Below example shows how we can define a variable and perform arithmetic operations on them.

Example:

# Initialize the variable 'salary'
salary = 300000

# Add 25000 to variable 'salary' and store the result in another variable 'newSalary'
newSalary = salary + 25000

# Print the updated salary
print newSalary

Alternately, you can directly print the addition of salary and the number 25000 to further shrink the code as:

salary = 300000
print salary + 25000

This way, we will get the similar result from print statement. But, the value stored in variable salary remains unchanged. If we were to update the variable salary with the added value, we can use assignment operator +=, that combines 2 operations, addition and assignment. So, we have the number 25000 added to the value stored in salary and the result of addition is again stored in the variable salary.

Below is the list of assignment operators:

Operator Description
+= Add and assign
-= Subtract and assign
*= Multiply and assign
/ Divide and assign
% Perform modulo and assign the result
^ or ** Perform exponentiation and assign the result

To demonstrate this, we can use /etc/passwd and count the number of lines in it. For this, we initialize a variable x and increment it after every line is read. After the last line, we print the variable, which gives us total number of records read by awk.

# We can also use { x++ } or { x = x + 1 } instead of { x += 1 } in below command
$ awk ' { x += 1 } END { print x } ' /etc/passwd
31

We can also include a condition here, to print the count of lines those have the string bash inside them.

$ awk '/bash/ { x += 1 } END { print x } ' /etc/passwd
3

Another example. This time, we use a demo file which has 3 fields - Name of the student, Subject name and the Marks. Below is the snippet.

Student Subject Marks
James Biology 31
Velma Biology 43
Kibo Biology 81
Louis Biology 11
Phyllis Biology 18
Zenaida Biology 55
Gillian Biology 38
Constance Biology 16
Giselle Biology 73
...
...

We can calculate the average marks obtained by students in Chemistry as follows:

awk ' /Chemistry/ { total += $3; count += 1 } END { print total/count } ' result.txt 
52.4074

We can also consider a data set of cities and their temperatures as below:

Washington 18 23 21 19 16
London 10 7 13 5 -1
Moscow 2 0 -3 -7 1
Mumbai 24 27 29 29 28

We can find average temperature for every city as shown below:

$ awk ' {total = $2 + $3 + $4 + $5 + $6; print $1 " : " total/5 }' cities.txt 
Washington : 19.4
London : 6.8
Moscow : -1.4
Mumbai : 27.4

That's all for the scope of this article. We did not cover all of the operators here as there are pretty straight forward and most of you may already have an idea about those ones. Let me know about your views and feedback in the comments section below and stay tuned for more articles.

Advance Your Career with Linux Foundation Training

Linux is the largest open-source technology that powers computers, mobiles and various other products and services across the world. It is the OS that runs more than 95% of the top 1 million domains and the top 500 supercomputers while accounting for 80% of smartphones, running Android based on Linux kernel.

Developers and engineers with Linux skills are in high demand in the job market. If you have a qualification or certification in Linux ecosystem, then you can get hired with high salary.
Linux Foundation, which is headed by the creator of Linux, provides several courses and certifications in different Linux technologies. You can easily apply for the course you want and increase your value as a potential employee or independent developer.

Popular Linux Foundation Training Courses

Which Linux Foundation training course should you go for? We will help you decide!

Certification

You can apply for the Linux Foundation Certified Engineer (LFCE) and Linux Foundation Certified System Administrator (LFCS) certification, which are both carried out online.

LFCS imparts the skills and knowledge of a sysadmin, which you need to help prove yourself to employers. LFCE gives you in-depth skills that enable you to design and implement system architecture. You can also provide guidance as a Subject Matter Expert using your newfound expertise.

Both certifications cost $499 and can be completed in 12 months.

Linux Courses

Linux Foundation offers both introductory and advanced Linux courses which can help you land a great job. The courses cover a wide range of aspects and are divided into the following categories-

  • Linux Programming & Development Training
  • Enterprise IT & Linux System Administration Training
  • Open Source Compliance Courses
Some of the courses can be completed in 1 year while others run for four days. You can even take your own time in the case of some courses, as they have no time limit.

The cost of the courses also varies according to their content- you can take some courses for completely free while others may cost anything from $179 to $3,150.

You can also apply for e-Learning courses, which can be completed at your own pace. The cost ranges between $149 and $299 while some are totally free! Use a Linux Foundation coupon and you can receive your certifications at a reduced price!

Who can Benefit from Linux Foundation Training?

The training can either be taken by individuals looking to advance their career or by employees of organizations, depending on Linux solutions.

Top companies like AMD, HP, Intel, and Nokia depend on Linux Foundation training to help their employees get skilled in open-source technologies. This also increases the demand for skilled Linux professionals who can successfully contribute towards the development of better products and services.

There is currently a shortage of talent when it comes to open-source professionals all over the world. If you have a certificate from the creators of Linux guaranteeing your skills, it becomes much easier to increase your demand in the job market.

You can get better and higher paying jobs without settling for anything less. Clients will also be happy to pay higher fees to a Linux freelancer with an approved qualification.

So whether you are a corporate employee or hoping one day to be a Linux pro, the courses at Linux Foundation are ideal to make you skilled and knowledgeable.