CodeName Blogs

SaltStack (SALT): A Comprehensive Overview

SaltStack, commonly referred to as SALT, is a powerful open-source infrastructure management platform designed for scalability. Leveraging event-driven workflows, SALT provides an adaptable solution for automating configuration management, remote execution, and orchestration across diverse infrastructures.

This document offers an in-depth guide to SALT for both technical teams and business stakeholders, demystifying its features and applications.

What is SALT?

SALT is a versatile tool that serves multiple purposes in infrastructure management:

• Configuration Management Tool (like Ansible, Puppet, Chef): Automates the setup and maintenance of servers and applications.

• Remote Execution Engine (similar to Fabric or SSH): Executes commands on systems remotely, whether targeting a single node or thousands.

• State Enforcement System: Ensures systems maintain desired configurations over time.

• Event-Driven Automation Platform: Detects system changes and triggers actions in real-time.

Key Technologies:

• YAML: Used for defining states and configurations in a human-readable format.

• Jinja: Enables dynamic templating for YAML files.

• Python: Provides extensibility through custom modules and scripts.

Supported Architectures

SALT accommodates various architectures to suit organizational needs:

• Master/Minion: A centralized control model where a Salt Master manages Salt Minions to send commands and execute tasks.

• Masterless: A decentralized approach using salt-ssh to execute tasks locally without requiring a master node.

Core Components of SALT

Key Features of SALT

Common Use Cases

SALT is widely used across industries for tasks like:

• Provisioning new systems and applying base configurations.

• Enforcing security policies and managing firewall rules.

• Installing and enabling software packages (e.g., HTTPD, Nginx).

• Scheduling and automating patching across multiple environments.

• Monitoring logs and system states with automatic remediation for issues.

• Managing VM and container lifecycles (e.g., Docker, LXC).

Real-World Examples

1. Remote Command Execution:

   • salt '*' test.ping (Pings all connected systems).

   • salt 'web*' cmd.run 'systemctl restart nginx' (Restarts Nginx service on all web servers).

2. State File Example (YAML):

   nginx:
     pkg.installed: []
     service.running:
       - enable: True
       - require:
         - pkg: nginx
   

Comparing SALT to Other Tools

Security Considerations

SALT ensures secure communication and authentication:

• Authentication: Uses public/private key pairs to authenticate minions.

• Encryption: Communicates via ZeroMQ encrypted with AES.

• Access Control: Defines granular controls using Access Control Lists (ACLs) in the Salt Master configuration.

Additional Information

For organizations seeking enhanced usability, SaltStack Config offers a graphical interface to streamline workflow management. Additionally, SALT's integration with VMware Tanzu provides advanced automation for enterprise systems.

Installation Example

On a master node (e.g., RedHat):

sudo yum install salt-master

On minion nodes:

sudo yum install salt-minion

Configure /etc/salt/minion with:

master: your-master-hostname

Then start the minion:

sudo systemctl enable --now salt-minion

Accept the minion on the master:

sudo salt-key -L         # list all keys
sudo salt-key -A         # accept all pending minion keys

Where to Go Next

• Salt Project Docs

• Git-based states with gitfs

• Masterless setups for container deployments

• Custom modules in Python

• Event-driven orchestration with beacons + reactors

Large 600+ Server Patching in 3 Regions with 3 different Environments Example

Let give an example of have 3 different environments DEV (Development), PREP (Preproduction), and PROD (Production), now let's dig a little deeper and say we have 3 different regions EUS (East US), WUS (West US), and EUR (European) and we would like these patches to be applied on changing dates, such as DEV will be patched on 3 days after the second Tuesday, PREP will be patched on 5 days after the second Tuesday, and PROD will be 5 days after the 3rd Tuesday. The final clause to this mass configuration is, we would like the patches to be applied on the Client Local Time.

In many configurations such as AUM, or JetPatch, you would need several different Maintenace Schedules or plans to create this setup. With SALT, the configuration lies inside the minion, so configuration is much more defined, and simple to manage.

Use Case Recap

You want to patch three environment groups based on local time and specific schedules:

Each server knows its environment via Salt grains, and the local timezone via OS or timedatectl.

Step-by-Step Plan

1. Set Custom Grains for Environment & Region

2. Create a Python script (run daily) that:

   • Checks if today matches the schedule per group

   • If yes, uses Salt to target minions with the correct grain and run patching

3. Schedule this script via cron or Salt scheduler

4. Use Salt States to define patching

Step 1: Define Custom Grains

On each minion, configure /etc/salt/minion.d/env_grains.conf:

grains:
  environment: dev   # or prep, prod
  region: us-east    # or us-west, eu-central, etc.

Then restart the minion:

sudo systemctl restart salt-minion

Verify:

salt '*' grains.items

Step 2: Salt State for Patching

Create patching/init.sls:

update-packages:
  pkg.uptodate:
    - refresh: True
    - retry:
        attempts: 3
        interval: 15

reboot-if-needed:
  module.run:
    - name: system.reboot
    - onlyif: 'test -f /var/run/reboot-required'

Step 3: Python Script to Orchestrate Patching

Let’s build run_patching.py. It:

• Figures out the correct date for patching

• Uses salt CLI to run patching for each group

• Handles each group in its region and timezone

#!/usr/bin/env python3
import subprocess
import datetime
import pytz
from dateutil.relativedelta import relativedelta, TU

# Define your environments and their rules
envs = {
    "dev": {"offset": 3, "week": 2},
    "prep": {"offset": 5, "week": 2},
    "prod": {"offset": 5, "week": 3}
}

# Map environments to regions (optional)
regions = {
    "dev": ["us-east", "us-west"],
    "prep": ["us-east", "eu-central"],
    "prod": ["us-east", "us-west", "eu-central"]
}

# Timezones per region
region_tz = {
    "us-east": "America/New_York",
    "us-west": "America/Los_Angeles",
    "eu-central": "Europe/Berlin"
}

def calculate_patch_date(year, month, week, offset):
    second_tuesday = datetime.date(year, month, 1) + relativedelta(weekday=TU(week))
    return second_tuesday + datetime.timedelta(days=offset)

def is_today_patch_day(env, region):
    now = datetime.datetime.now(pytz.timezone(region_tz[region]))
    target_day = calculate_patch_date(now.year, now.month, envs[env]["week"], envs[env]["offset"])
    return now.date() == target_day and now.hour >= desired_hour

def run_salt_target(environment, region):
    target = f"environment:{environment} and region:{region}"
    print(f"Patching {target}...")
    subprocess.run([
        "salt", "-C", target, "state.apply", "patching"
    ])

def main():
    for env in envs:
        for region in regions[env]:
            if is_today_patch_day(env, region):
                run_salt_target(env, region)

if __name__ == "__main__":
    main()

Make it executable:

chmod +x /srv/scripts/run_patching.py

Test it:

./run_patching.py

Step 4: Schedule via Cron (on Master)

Edit crontab:

crontab -e

Add daily job:

# Run daily at 6 AM UTC
0 6 * * * /srv/scripts/run_patching.py >> /var/log/salt/patching.log 2>&1

This assumes the local time logic is handled in the script using each region’s timezone.

Security & Safety Tips

• Test patching states on a few dev nodes first (salt -G 'environment:dev' -l debug state.apply patching)

• Add Slack/email notifications (Salt Reactor or Python smtplib)

• Consider dry-run support with test=True (in pkg.uptodate)

• Use salt-run jobs.list_jobs to track job execution

Optional Enhancements

• Use Salt Beacons + Reactors to monitor and patch in real-time

• Integrate with JetPatch or Ansible for hybrid control

• Add patch deferral logic for critical services

• Write to a central patching log DB with job status per host

Overall Architecture

Minions:

• Monitor the date/time via beacons

• On patch day (based on local logic), send a custom event to the master

Master:

• Reacts to that event via a reactor

• Targets the sending minion and applies the patching state

Step-by-Step: Salt Beacon + Reactor Model

1. Define a Beacon on Each Minion

File: /etc/salt/minion.d/patchday_beacon.conf

beacons:
  patchday:
    interval: 3600  # check every hour

This refers to a custom beacon we will define.

2. Create the Custom Beacon (on all minions)

File: /srv/salt/_beacons/patchday.py

import datetime
from dateutil.relativedelta import relativedelta, TU
import pytz

__virtualname__ = 'patchday'

def beacon(config):
    ret = []

    grains = __grains__
    env = grains.get('environment', 'unknown')
    region = grains.get('region', 'unknown')

    # Define rules
    rules = {
        "dev": {"offset": 3, "week": 2},
        "prep": {"offset": 5, "week": 2},
        "prod": {"offset": 5, "week": 3}
    }

    region_tz = {
        "us-east": "America/New_York",
        "us-west": "America/Los_Angeles",
        "eu-central": "Europe/Berlin"
    }

    if env not in rules or region not in region_tz:
        return ret  # invalid or missing config

    tz = pytz.timezone(region_tz[region])
    now = datetime.datetime.now(tz)
    rule = rules[env]

    patch_day = (datetime.date(now.year, now.month, 1)
                 + relativedelta(weekday=TU(rule["week"]))
                 + datetime.timedelta(days=rule["offset"]))

    if now.date() == patch_day:
        ret.append({
            "tag": "patch/ready",
            "env": env,
            "region": region,
            "datetime": now.isoformat()
        })

    return ret

3. Sync Custom Beacon to Minions

On the master:

salt '*' saltutil.sync_beacons

Enable it:

salt '*' beacons.add patchday '{"interval": 3600}'

4. Define Reactor on the Master

File: /etc/salt/master.d/reactor.conf

reactor:
  - 'patch/ready':
    - /srv/reactor/start_patch.sls

5. Create Reactor SLS File

File: /srv/reactor/start_patch.sls

{% set minion_id = data['id'] %}

run_patching:
  local.state.apply:
    - tgt: {{ minion_id }}
    - arg:
      - patching

This reacts to patch/ready event and applies the patching state to the calling minion.

6. Testing the Full Flow

1. Restart the minion: systemctl restart salt-minion

2. Confirm the beacon is registered: salt '*' beacons.list

3. Trigger a manual test (simulate patch day by modifying date logic)

4. Watch events on master:

salt-run state.event pretty=True

5. Confirm patching applied:

salt '*' saltutil.running

7. Example: patching/init.sls

Already shared, but here it is again for completeness:

update-packages:
  pkg.uptodate:
    - refresh: True
    - retry:
        attempts: 3
        interval: 15

reboot-if-needed:
  module.run:
    - name: system.reboot
    - onlyif: 'test -f /var/run/reboot-required'

Benefits of This Model

• Real-time and event-driven – no need for polling or external scripts

• Timezone-aware, thanks to local beacon logic

• Self-healing – minions signal readiness independently

• Audit trail – each event is logged in Salt’s event bus

• Extensible – you can easily add Slack/email alerts via additional reactors

Goal

1. Track patching event completions per minion

2. Store patch event metadata: who patched, when, result, OS, IP, environment, region, etc.

3. Generate readable reports in:

   • CSV/Excel

   • HTML dashboard

   • JSON for API or SIEM ingestion

Step 1: Customize Reactor to Log Completion

Let’s log each successful patch into a central log file or database (like SQLite or MariaDB).

Update Reactor: /srv/reactor/start_patch.sls

Add a returner to store job status.

{% set minion_id = data['id'] %}

run_patching:
  local.state.apply:
    - tgt: {{ minion_id }}
    - arg:
      - patching
    - kwarg:
        returner: local_json  # You can also use 'mysql', 'elasticsearch', etc.

Configure Returner (e.g., local_json)

In /etc/salt/master:

returner_dirs:
  - /srv/salt/returners

ext_returners:
  local_json:
    file: /var/log/salt/patch_report.json

Or use a MySQL returner:

mysql.host: 'localhost'
mysql.user: 'salt'
mysql.pass: 'yourpassword'
mysql.db: 'salt'
mysql.port: 3306

Enable returners:

salt-run saltutil.sync_returners

Step 2: Normalize Patch Data (Optional Post-Processor)

If using JSON log, create a post-processing script to build reports:

process_patch_log.py

import json
import csv
from datetime import datetime

def load_events(log_file):
    with open(log_file, 'r') as f:
        return [json.loads(line) for line in f if line.strip()]

def export_csv(events, out_file):
    with open(out_file, 'w', newline='') as f:
        writer = csv.DictWriter(f, fieldnames=[
            'minion', 'date', 'environment', 'region', 'result'
        ])
        writer.writeheader()
        for e in events:
            writer.writerow({
                'minion': e['id'],
                'date': datetime.fromtimestamp(e['_stamp']).isoformat(),
                'environment': e['return'].get('grains', {}).get('environment', 'unknown'),
                'region': e['return'].get('grains', {}).get('region', 'unknown'),
                'result': 'success' if e['success'] else 'failure'
            })

events = load_events('/var/log/salt/patch_report.json')
export_csv(events, '/srv/reports/patching_report.csv')

Step 3: Build a Simple Web Dashboard

If you want to display reports via a browser:

🛠 Tools:

• Flask or FastAPI

• Bootstrap or Chart.js

• Reads JSON/CSV and renders:

Example Chart Dashboard Features:

• ✅ Last patch date per server

• 📍 Patching success rate per region/env

• 🔴 Highlight failed patching

• 📆 Monthly compliance timeline

Would you like a working example of that Flask dashboard? I can include the full codebase if so.

Step 4: Send Reports via Email (Optional)

🐍 Python: send_report_email.py

import smtplib
from email.message import EmailMessage

msg = EmailMessage()
msg["Subject"] = "Monthly Patch Report"
msg["From"] = "patchbot@example.com"
msg["To"] = "it-lead@example.com"
msg.set_content("Attached is the patch compliance report.")

with open("/srv/reports/patching_report.csv", "rb") as f:
    msg.add_attachment(f.read(), maintype="text", subtype="csv", filename="patching_report.csv")

with smtplib.SMTP("localhost") as s:
    s.send_message(msg)

Schedule that weekly or monthly with cron.

Flask Dashboard (Patch Reporting)

app.py

from flask import Flask, render_template
import csv
from collections import defaultdict

app = Flask(__name__)

@app.route('/')
def index():
    results = []
    success_count = defaultdict(int)
    fail_count = defaultdict(int)

    with open('/srv/reports/patching_report.csv', 'r') as f:
        reader = csv.DictReader(f)
        for row in reader:
            results.append(row)
            key = f"{row['environment']} - {row['region']}"
            if row['result'] == 'success':
                success_count[key] += 1
            else:
                fail_count[key] += 1

    summary = [
        {"group": k, "success": success_count[k], "fail": fail_count[k]}
        for k in sorted(set(success_count) | set(fail_count))
    ]

    return render_template('dashboard.html', results=results, summary=summary)

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5000)

templates/dashboard.html

<!DOCTYPE html>
<html>
<head>
  <title>Patch Compliance Dashboard</title>
  <style>
    body { font-family: Arial; padding: 20px; }
    table { border-collapse: collapse; width: 100%; margin-bottom: 30px; }
    th, td { border: 1px solid #ccc; padding: 8px; text-align: left; }
    th { background-color: #f4f4f4; }
    .fail { background-color: #fdd; }
    .success { background-color: #dfd; }
  </style>
</head>
<body>
  <h1>Patch Compliance Dashboard</h1>

  <h2>Summary</h2>
  <table>
    <tr><th>Group</th><th>Success</th><th>Failure</th></tr>
    {% for row in summary %}
    <tr>
      <td>{{ row.group }}</td>
      <td>{{ row.success }}</td>
      <td>{{ row.fail }}</td>
    </tr>
    {% endfor %}
  </table>

  <h2>Detailed Results</h2>
  <table>
    <tr><th>Minion</th><th>Date</th><th>Environment</th><th>Region</th><th>Result</th></tr>
    {% for row in results %}
    <tr class="{{ row.result }}">
      <td>{{ row.minion }}</td>
      <td>{{ row.date }}</td>
      <td>{{ row.environment }}</td>
      <td>{{ row.region }}</td>
      <td>{{ row.result }}</td>
    </tr>
    {% endfor %}
  </table>
</body>
</html>

How to Use

pip install flask
python app.py

Then visit http://localhost:5000 or your server’s IP at port 5000.

Optional: SIEM/Event Forwarding

If you use Elasticsearch, Splunk, or Mezmo:

• Use a returner like es_return, splunk_return, or send via custom script using REST API.

• Normalize fields: hostname, env, os, patch time, result

• Filter dashboards by compliance groupings

TL;DR: Reporting Components Checklist