Friday, 29 September 2017

SaltStack Targeting Methods - Grain Matching

We have been learning Salt remote command execution and how minions can be targeted, in particular. We have already covered three methods of targeting minions - Glob matching, PCRE matching and List matching. We move a step further and know about how we can target minions using grain matching.


What are Grains?


Grains are static data stores where we can store data about or describing our minions, in key-value format. We can use these grains to target our minions. We can get information about available grains using below command -

$ salt 'centos-rack1' grains.items
centos-rack1:
    ----------
    SSDs:
    cpu_flags:
        - fpu
        - vme
        - de
 ...
 ...
    num_cpus:
        4
    num_gpus:
        0
    os:
        RedHat
    os_family:
        RedHat
    osarch:
        x86_64
    oscodename:
        Santiago
    osfinger:
        Red Hat Enterprise Linux Server-6
 ...
 ...
 selinux:
        ----------
        enabled:
            False
        enforced:
            Disabled
    server_id:
        538503645
    shell:
        /bin/sh
    uid:
        0
    username:
        root
 ...
 ...

Thus, from above output, we can observe that there several grains, for example os, os_family, osfinger, etc. We can retrieve a specific information by targeting a particular grain using below command -

$ salt 'centos-rack1' grains.item os
centos-rack1:
    ----------
    os:
        RedHat
  
$ salt 'centos-rack1' grains.item os_family
centos-rack1:
    ----------
    os_family:
        RedHat
  
$ salt 'centos-rack1' grains.item osfinger
centos-rack1:
    ----------
    osfinger:
        Red Hat Enterprise Linux Server-6

We can create our own custom grains to store more detailed information, that may be used to target minions more specifically. For example, we may wish to run a remote execution command on the machines situated at a particular location or present in a particular rack. In that case, we can create our custom grain as shown below:

$ salt 'centos-rack1' grains.setval racknum 1
centos-rack1:
    ----------
    racknum:
        1
  
$ salt 'redhat-rack2' grains.setval racknum 2
redhat-rack2:
    ----------
    racknum:
        2

We can confirm this by retrieving the grain information.

$ salt '*' grains.item racknum
redhat-rack2:
    ----------
    racknum:
        2
centos-rack1:
    ----------
    racknum:
        1

We can also set more than one values associated with a grain as shown below :

$ salt 'centos-rack1' grains.setval application '["web", "nginx", "prod"]'
centos-rack1:
    ----------
    application:
        - web
        - nginx
        - prod

$ salt 'redhat-rack2' grains.setval application '["database", "mysql", "dev"]'
redhat-rack2:
    ----------
    application:
        - database
        - mysql
        - dev

In order to delete these values, you can use -

$ salt 'redhat-rack2' grains.delval application
redhat-rack2:
    None

$ salt 'centos-rack1' grains.delval application
centos-rack1:
    None

Please note that, even though we have deleted the custom grains, they do persist with a return value of None. To get rid of them completely, we need to mention destructive=True explicitly.

$ salt 'centos-rack1' grains.delval application destructive=True
centos-rack1:
    None
 
$ salt 'centos-rack1' grains.item application
centos-rack1:
    ----------
    application:

With this available information in the form of grains, we can target our minions using grain matching. In this case, we specify the option -G or --grain with salt command.

Grain Matching


Now that, we know what grains are and how they are retrieved, it's time to target minions using grains such that the remote execution commands are executed based on the values available in the grains. For this, we make use of --grain option as below -

# Targetting the machines with RedHat os_family
$ salt -G 'os_family:RedHat' test.ping
redhat-rack2:
    True
centos-rack1:
    True

We can also target the machines more specifically, e.g. machines with CentOS installed, as -
# Targetting the machines with CentOS os
$ salt -G 'os:CentOS' test.ping
centos-rack1:
    True

With the use of glob -

$ salt -G 'os:cent*' test.ping
centos-rack1:
    True

We can observe that grain matching is case-insensitive and we can combine glob matching with it too. This is very useful in configuration management where we have a large number of machines with different operating systems.


That's all for the scope of this tutorial, stay tuned for more interesting ones.

Thursday, 28 September 2017

SaltStack Targeting Methods - List Matching

This could possibly be the shortest tutorial in this series. We have been dealing with targeting minions and different methods to target minions. We have covered a couple of methods of targeting minions - with glob matching and PCRE matching, in previous tutorials. In this one, we will learn about another methods i.e. using list matching and believe me, it's the simplest one.


This method does not require any fancy stuff to match minion IDs, like we saw in case of previous cases. We just provide a list of minions to the salt command, that we wish to manage. List is a comma separated sequence of minions. In order to process the list on minions, we use the option -L or --list with salt command. Remember, till now, we have created only two minions - centos-rack1 and redhat-rack2.

$ salt -L 'centos-rack1' test.ping
centos-rack1:
    True
 

$ salt -L 'centos-rack1,redhat-rack2,ubuntu-rack1' test.ping
redhat-rack2:
    True
centos-rack1:
    True
 

$ salt -L 'ubuntu-rack1' test.ping
No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received

Wednesday, 27 September 2017

SaltStack Targeting Methods - Perl Compatible Regular Expressions (PCRE) Matching

We have been discussing on targeting minions using minion IDs on Salt master. In the last article, we targetted minions usling glob matching method. In this article, we will learn another method to target minions using minion IDs - Perl compatible regular expression / PCRE matching. This method is very useful when we need to target minions with complex matches, against their minion IDs. PCRE matching makes use of Python re module to parse the regular expressions. It requires a basic understanding of regular expressions to write PCRE strings, so I recommend you to review the documents on regular expressions provided on Python website.


PCRE matching works differently than glob matching, as PCRE match can accept partial matches. In order to indicate Salt that we are using PCRE string to target minions, we need to use an extra option -E or --pcre with salt command.

1. . matches zero or more characters


In our article on glob matching, we observed that * matches anything. Similarly, to match anything (any character appearing zero or more times) using PCRE, we need to use .*.

$ salt -E '.*' test.ping
redhat-rack2:
    True
centos-rack1:
    True


$ salt -E '.*rack1' test.ping
centos-rack1:
    True


$ salt -E 'centos-.*' test.ping
centos-rack1:
    True

2. ^ matches beginning of the line


In order to target minions with IDs starting with centos, we would use ^centos as the target string. This will match centos-rack1 and centossrv, but not some-centos-srv or rack1-centos.

$ salt -E '^centos' test.ping
centos-rack1:
    True 

3. $ matches end of the line


Consider that, all the servers installed in the first rack in our data center are having minion IDs ending with rack1, e.g. centos-rack1,
redhat-rack1, websrv-rack1, etc. When targeting all these servers, we need to use a PCRE which should match minions whose IDs end with the string rack1. In this case, $ comes to the rescue, which matches end of the line, but don't forget about .*.

Note that, .*rack1$ will match all of these minion IDs, but not rack1-web or fedora-rack10.

$ salt -E '.*rack1$' test.ping
centos-rack1:
    True

4. ? matches zero or one occurrences of target string


Questions marks are used for optional matches, to match a string that may or may not be present.

# This matches 'centos-rack1' or 'rack1'
$ salt -E '(centos-)?rack1' test.ping
centos-rack1:
    True

# This matches 'centos-rack' followed by an optional literal
# 'centos-rack1', 'centos-rackA', 'centos-rack' will match, 'centos-rack10' won't
salt -E 'centos-rack.?' test.ping
centos-rack1:
    True

5. | is used for multiple matches


If we were to match centos or redhat followed by -rack1, we can use | operator. Thus, (centos|redhat) will try to match either centos or redhat.

$ salt -E '(centos|redhat)-rack' test.ping
centos-rack1:
    True
redhat-rack2:
    True
 
$ salt -E '(centos|ubuntu)-rack' test.ping
centos-rack1:
    True

That's all for this article. The scope of PCRE being too large, I could not include everything in this tutorial, but the intention was only to explain their use in a salt command. Do remember to learn more about PCRE from Python documentation and stay tuned for more tutorials.

SaltStack Targeting Methods - Glob Matching

So far, we have been targeting only one minion. When it comes to targeting a group of minions or all minions, there are certain ways to do it, which are listed as below:

  • Matching Minion IDs
    • Glob matching
    • PCRE
    • List matching
  • Matching with Grains
  • Matching with Pillars
  • IP address and Subnet Matching
  • Compound Matching


In this article, we will learn to target a subset of minions using Glob matching. For this, we have configured two minions with minion IDs centos-rack1 and redhat-rack2.

1. * matches anything


If you have worked with Linux files and directories, especially while removing all files with rm -f *, you might be familiar with globbing. When we specify *, we simple mean 'anything'. Thus, rm -f * in Linux removes all files in the directory. We can also mention *.pdf to remove files which have names - anything followed by .pdf.

Lets check it in Salt's context while targeting the minions.

$ salt '*' test.ping

centos-rack1:
    True
redhat-rack2:
    True

In above output, we can see the responses from all the minions, currently only two, are received on the master. If we were to target a particular minion, we should mention minion name in place of * as follows:

$ salt 'redhat-rack2' test.ping

redhat-rack2:
    True

Lets see more examples on this with descriptions mentioned in the comment.

2. ? matches one character


$ salt 'centos-rack?' test.ping
centos-rack1:
    True

3. * matches any character


$ salt '*-rack1' test.ping

centos-rack1:
    True

4. We can also mention range within square brackets


# Any letter between 'a' and 'm' followed by anything followed by '-rack' followed by anything
$ salt '[a-m]*-rack*' test.ping

centos-rack1:
    True

# Any letter between 'n' and 'z' followed by anything followed by '-rack' followed by anything 
$ salt '[n-z]*-rack*' test.ping

redhat-rack2:
    True


That's all for this article. These are several different types of targeting minions using glob matching, we have seen the basic ones. Glob matching is not the only option to target minions, there are few others too, as mentioned in earlier portion of this article. We will learn all of them one-by-one, in upcoming tutorials in the series.


Tuesday, 26 September 2017

SaltStack - Introduction to Remote Execution

In the last article, we configured Salt master and Salt minion. We also learned how to accept a minion key in order to instruct the master to trust a particular minion. In this article, we will go a step further and learn about remote execution commands in Salt.


USING REMOTE EXECUTION COMMAND


In the last article, we came across a command in order to test the master-minion connectivity, which is given as below:

$ salt 'salt_minion1' test.ping

This executes ping function from test module on the minion with ID salt_minion1 from the Salt master. This command is an example of Salt remote execution. To know more about salt command, you can simply run -

$ salt --help | less

This tells us that the basic Salt remote execution command consists of five parts and it's syntax is as below :

Usage: salt [options] '<target>' <function> [arguments]

Consider below example to know about it's usage in further details -

salt --verbose 'salt_minion1' test.fib 5

Let's now see each part individually.

SALT COMMAND-LINE OPTIONS


In the example given above, we can categorize various pieces used in salt command as per the syntax as :

  • salt : The Salt remote execution command
  • options (--verbose) : Command line options
  • target (salt_minion1) : Target string
  • function (test.fib) : The salt module function
  • arguments (5) : Arguments to the remote execution function

1. Options

With different output options available, we can display the output returned by salt in various format. The values those can be used for these options may be - nested (the default one), raw, json, yaml, etc. To check this, we introduce a new function here - cmd.run_all. It would return the command output in the form of dictionary that contains process ID or pid of the command, stdout and stderr contents and the return code retcode. Lets check this.

# Output in 'raw' format
$ salt --output=raw 'salt_minion1' cmd.run_all "echo Hello world"
{'salt_minion1': {'pid': 24495, 'retcode': 0, 'stderr': '', 'stdout': 'Hello world'}}

# Output in 'txt' format
$ salt --output=txt 'salt_minion1' cmd.run_all "echo Hello world"
salt_minion1: {'pid': 25668, 'retcode': 0, 'stderr': '', 'stdout': 'Hello world'}

# Output in 'json' format
$ salt --output=json 'salt_minion1' cmd.run_all "echo Hello world"
{
"salt_minion1": {
"pid": 24387,
"retcode": 0,
"stderr": "",
"stdout": "Hello world"
}
}

# Output in 'yaml' format
$ salt --output=yaml 'salt_minion1' cmd.run_all "echo Hello world"
salt_minion1:
pid: 25744
retcode: 0
stderr: ''
stdout: Hello world

We may use other options to change the behavior of Salt command, e.g. --timeout, --verbose, --summary.

2. Target strings

So far, we have executed the Salt command on a single minion with minion ID salt_minion1. However, with Salt, we can manage hundreds or thousands of minion using a single master, that too without writing complex scripts or using a SSH loop. Sometimes, we may need to execute a command on a particular group of machines rather than all of them. In that case, we need to write an appropriate target string. Ho to do it, we will learn in the upcoming article.

3. Execution modules and functions

Earlier, we have learned that, the Salt remote execution commands are actually the Python functions which run on target hosts. When these functions are grouped together, they make execution module. We have came across a couple of execution modules before - test and cmd. When we executed test.ping, we asked Salt to execute ping function from test module. In short, all remote execution commands must be specified in module_name.function_name format.

You can list all the available execution modules with the use of list_modules function within sys module as below:

$ salt 'salt_minion1' sys.list_modules
salt_minion1:
- aliases
- alternatives
- apache
...
...
- cmd
- composer
...
...
- test
- timezone
...
...

All these execution modules are useful in performing variety of operations on the minion. We will learn some of the basic execution modules in upcoming articles.

4. Arguments

Argument is the extra piece of information that we provide to the execution command (hence, the functions), e.g. what user is to be added, which package needs to be installed, etc. In the below example, we have provided '4' as the argument to instruct the test.fib function to display 4th element in Fibonacci sequence.

$ salt 'salt_minion1' test.fib 4
salt_minion1:
- 3
- 6.91413879395e-06

That's all for the introductory part of remote execution in Salt. I hope we will learn a lot and in much depth, as we go further in this series of articles. Thanks.

Monday, 25 September 2017

SaltStack - Introduction, Installation and Configuration

INTRODUCTION


What is Salt?
  • A configuration management tool for your infrastructure
  • A lightening-fast remote execution tool
  • It based on Python and uses function calls to execute tasks on the servers, from a central hub
Components of salt
  1. Salt Master
    • It is the central server or the control server, who provides instructions to it's clients (or servers in your infrastructure)
    • you can execute commands and manage configurations over thousands of servers in seconds
  2. Salt Minion
    • They connect to master and receive instructions from Salt Master
    • They can work without master, but configurations have to be made on each and every Salt Minion separately - a time consuming option


INSTALLATION


On RHEL/CentOS

1. Create and configure the repository.
2. Create and edit the /etc/yum.repos.d/saltstack.repo with contents as below:

[saltstack-repo]
name=SaltStack repo for Red Hat Enterprise Linux $releasever
baseurl=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest
enabled=1
gpgcheck=0
gpgkey=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest/SALTSTACK-GPG-KEY.pub
       https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest/base/RPM-GPG-KEY-CentOS-7

3. Install the package.

yum install salt-master salt-minion

On Ubuntu

1. Add the PPA.

sudo add-apt-repository ppa:saltstack/salt

2. Update the database.

sudo apt-get update

3. Install the package.

sudo apt-get install salt-master salt-minion

CONFIGURATION


  • For our first exercise, we will configure Salt master and Salt minion on the same server
  • The master and minion configuration files are located in /etc/salt directory
  • /etc/salt/master holds the configuration for Salt master
  • /etc/salt/minion contains the configuration for Salt minion

Salt Minion Configuration

1. Open the Salt minion configuration file - /etc/salt/minion
2. Search for the line which looks like-

#master: salt

3. In above line, salt should be replaced with the DNS hostname or IP address of the Salt master, to connect with the Salt master
4. In our case, we should replace salt with localhost, as we have our Salt master on the same server, and uncomment the line

master: localhost

5. Next, we need to set the minion ID, a unique identity for the minion, which may usually be a hostname.
6. For this, search for the line in the configuration file which looks like -

#id

7. Lets set the minion ID for this server to be salt_minion1

id: salt_minion1

8. Once you're done, save and close the file.
9. For the changes to take effect, restart salt-minion service with below command -

$ service salt-minion restart

Service salt-minion:root:salt_minion1 is not running
Starting salt-minion:root:salt_minion1 daemon: OK

Salt Master Configuration

1. For the introductory article, we do not need to make any changes in configuration file for Salt master
2. Just to make sure that Salt master is up, restart the salt-master service as below -

$ service salt-master restart

Stopping salt-master daemon:                               [FAILED]
Starting salt-master daemon:                               [  OK  ]

Accept the Salt minion key on Salt master

1. Having done the configuration for Salt master and minion, we need to accept the minion key on the master.
2. This instructs the master to trust the minion.
3. Before we accept the key, let's check whether our minion has contacted the master with 'salt-key' command as below -

$ salt-key
Accepted Keys:
Denied Keys:
Unaccepted Keys:
salt_minion1
Rejected Keys:

3. You can observe that the minion we just configured - salt_minion1 is listed under Unaccepted Keys section.
4. This indicates that, the minion with ID salt_minion1 has contacted the server and the master has stored its public key.
5. But, the minion's key is not accepted yet by the master.
6. We can check the minion's key as stored by the master with below command -

$ salt-key -f salt_minion1
Unaccepted Keys:
salt_minion1:  26:18:fc:10:4e:b3:c8:73:fd:53:95:8b:6a:f2:30:5a:0c:3d:7d:04:69:59:a0:7b:91:30:54:bc:5f:18:7c:9e

7. We can also verify the minion's key by running salt-call command on the minion (same server, for our case) as below :

$ salt-call --local key.finger
local:
26:18:fc:10:4e:b3:c8:73:fd:53:95:8b:6a:f2:30:5a:0c:3d:7d:04:69:59:a0:7b:91:30:54:bc:5f:18:7c:9e

8. Both keys match, thus we can accept the key on master, as below :

$ salt-key -a salt_minion1

The following keys are going to be accepted:
Unaccepted Keys:
salt_minion1
Proceed? [n/Y] Y
Key for minion salt_minion1 accepted.

9. To ensure that the key has been accepted, we can run salt-key command again -

$ salt-key

Accepted Keys:
salt_minion1
Denied Keys:
Unaccepted Keys:
Rejected Keys:

10. And the key is accepted!

TESTING THE SETUP


1. Now that the minion's key is received on the master, it's time to check the setup if it working.
2. For this, we ask the minion 'salt_minion1' to execute a function 'ping' from module 'test' from the master, as shown below :

$ salt 'salt_minion1' test.ping

salt_minion1:
	True

3. Above command is used to check if the minion is alive and we've received a response as 'True'.

That's all for the scope of this article. In the next one, we will check some of the basic commands useful while understanding SaltStack further. Thank you!.