.jpg.7633371fa53fa19028f71f2e3a72fc4e.jpg)
Introduction to SALT: A Unified Platform for Infrastructure Automation
Linux · Jessica Brown · 04/11/25 09:04 AM
SaltStack, commonly referred to as SALT, is a powerful open-source infrastructure management platform designed for scalability. Leveraging event-driven workflows, SALT provides an adaptable solution for automating configuration management, remote execution, and orchestration across diverse infrastructures.
This document offers an in-depth guide to SALT for both technical teams and business stakeholders, demystifying its features and applications.
What is SALT?
SALT is a versatile tool that serves multiple purposes in infrastructure management:
Configuration Management Tool (like Ansible, Puppet, Chef): Automates the setup and maintenance of servers and applications.
Remote Execution Engine (similar to Fabric or SSH): Executes commands on systems remotely, whether targeting a single node or thousands.
State Enforcement System: Ensures systems maintain desired configurations over time.
Event-Driven Automation Platform: Detects system changes and triggers actions in real-time.
Key Technologies:
YAML: Used for defining states and configurations in a human-readable format.
Jinja: Enables dynamic templating for YAML files.
Python: Provides extensibility through custom modules and scripts.
Supported Architectures
SALT accommodates various architectures to suit organizational needs:
Master/Minion: A centralized control model where a Salt Master manages Salt Minions to send commands and execute tasks.
Masterless: A decentralized approach using salt-ssh to execute tasks locally without requiring a master node.
Core Components of SALT
Component
Description
Salt Master
Central control node that manages minions, sends commands, and orchestrates infrastructure tasks.
Salt Minion
Agent installed on managed nodes that executes commands from the master.
Salt States
Declarative YAML configuration files that define desired system states (e.g., package installations).
Grains
Static metadata about a system (e.g., OS version, IP address), useful for targeting specific nodes.
Pillars
Secure, per-minion data storage for secrets and configuration details.
Runners
Python modules executed on the master to perform complex orchestration tasks.
Reactors
Event listeners that trigger actions in response to system events.
Beacons
Minion-side watchers that emit events based on system changes (e.g., file changes or CPU spikes).
Key Features of SALT
Feature
Description
Agent or Agentless
SALT can operate in agent (minion-based) or agentless (masterless) mode.
Scalability
Capable of managing tens of thousands of nodes efficiently.
Event-Driven
Reacts to real-time system changes via beacons and reactors, enabling automation at scale.
Python Extensibility
Developers can extend modules or create custom ones using Python.
Secure
Employs ZeroMQ for communication and AES encryption for data security.
Role-Based Config
Dynamically applies configurations based on server roles using grains metadata.
Granular Targeting
Targets systems using name, grains, regex, or compound filters for precise management.
Common Use Cases
SALT is widely used across industries for tasks like:
Provisioning new systems and applying base configurations.
Enforcing security policies and managing firewall rules.
Installing and enabling software packages (e.g., HTTPD, Nginx).
Scheduling and automating patching across multiple environments.
Monitoring logs and system states with automatic remediation for issues.
Managing VM and container lifecycles (e.g., Docker, LXC).
Real-World Examples
Remote Command Execution:
salt '*' test.ping (Pings all connected systems).
salt 'web*' cmd.run 'systemctl restart nginx' (Restarts Nginx service on all web servers).
State File Example (YAML):
nginx: pkg.installed: [] service.running: - enable: True - require: - pkg: nginx
Comparing SALT to Other Tools
Feature
Salt
Ansible
Puppet
Chef
Language
YAML + Python
YAML + Jinja
Puppet DSL
Ruby DSL
Agent Required
Optional
No
Yes
Yes
Push/Pull
Both
Push
Pull
Pull
Speed
Very Fast
Medium
Medium
Medium
Scalability
High
Medium-High
Medium
Medium
Event-Driven
Yes
No
No
Limited
Security Considerations
SALT ensures secure communication and authentication:
Authentication: Uses public/private key pairs to authenticate minions.
Encryption: Communicates via ZeroMQ encrypted with AES.
Access Control: Defines granular controls using Access Control Lists (ACLs) in the Salt Master configuration.
Additional Information
For organizations seeking enhanced usability, SaltStack Config offers a graphical interface to streamline workflow management. Additionally, SALT's integration with VMware Tanzu provides advanced automation for enterprise systems.
Installation Example
On a master node (e.g., RedHat):
sudo yum install salt-master On minion nodes:
sudo yum install salt-minion Configure /etc/salt/minion with:
master: your-master-hostname Then start the minion:
sudo systemctl enable --now salt-minion Accept the minion on the master:
sudo salt-key -L # list all keys sudo salt-key -A # accept all pending minion keys Where to Go Next
Salt Project Docs
Git-based states with gitfs
Masterless setups for container deployments
Custom modules in Python
Event-driven orchestration with beacons + reactors
Large 600+ Server Patching in 3 Regions with 3 different Environments Example
Let give an example of have 3 different environments DEV (Development), PREP (Preproduction), and PROD (Production), now let's dig a little deeper and say we have 3 different regions EUS (East US), WUS (West US), and EUR (European) and we would like these patches to be applied on changing dates, such as DEV will be patched on 3 days after the second Tuesday, PREP will be patched on 5 days after the second Tuesday, and PROD will be 5 days after the 3rd Tuesday. The final clause to this mass configuration is, we would like the patches to be applied on the Client Local Time.
In many configurations such as AUM, or JetPatch, you would need several different Maintenace Schedules or plans to create this setup. With SALT, the configuration lies inside the minion, so configuration is much more defined, and simple to manage.
Use Case Recap
You want to patch three environment groups based on local time and specific schedules:
Environment
Schedule Rule
Timezone
Dev
3rd day after 2nd Tuesday of the month
Local
PREP
5th day after 2nd Tuesday of the month
Local
Prod
5th day after 3rd Tuesday of the month
Local
Each server knows its environment via Salt grains, and the local timezone via OS or timedatectl.
Step-by-Step Plan
Set Custom Grains for Environment & Region
Create a Python script (run daily) that:
Checks if today matches the schedule per group
If yes, uses Salt to target minions with the correct grain and run patching
Schedule this script via cron or Salt scheduler
Use Salt States to define patching
Step 1: Define Custom Grains
On each minion, configure /etc/salt/minion.d/env_grains.conf:
grains: environment: dev # or prep, prod region: us-east # or us-west, eu-central, etc. Then restart the minion:
sudo systemctl restart salt-minion Verify:
salt '*' grains.items
Step 2: Salt State for Patching
Create patching/init.sls:
update-packages: pkg.uptodate: - refresh: True - retry: attempts: 3 interval: 15 reboot-if-needed: module.run: - name: system.reboot - onlyif: 'test -f /var/run/reboot-required'
Step 3: Python Script to Orchestrate Patching
Let’s build run_patching.py. It:
Figures out the correct date for patching
Uses salt CLI to run patching for each group
Handles each group in its region and timezone
#!/usr/bin/env python3 import subprocess import datetime import pytz from dateutil.relativedelta import relativedelta, TU # Define your environments and their rules envs = { "dev": {"offset": 3, "week": 2}, "prep": {"offset": 5, "week": 2}, "prod": {"offset": 5, "week": 3} } # Map environments to regions (optional) regions = { "dev": ["us-east", "us-west"], "prep": ["us-east", "eu-central"], "prod": ["us-east", "us-west", "eu-central"] } # Timezones per region region_tz = { "us-east": "America/New_York", "us-west": "America/Los_Angeles", "eu-central": "Europe/Berlin" } def calculate_patch_date(year, month, week, offset): second_tuesday = datetime.date(year, month, 1) + relativedelta(weekday=TU(week)) return second_tuesday + datetime.timedelta(days=offset) def is_today_patch_day(env, region): now = datetime.datetime.now(pytz.timezone(region_tz[region])) target_day = calculate_patch_date(now.year, now.month, envs[env]["week"], envs[env]["offset"]) return now.date() == target_day and now.hour >= desired_hour def run_salt_target(environment, region): target = f"environment:{environment} and region:{region}" print(f"Patching {target}...") subprocess.run([ "salt", "-C", target, "state.apply", "patching" ]) def main(): for env in envs: for region in regions[env]: if is_today_patch_day(env, region): run_salt_target(env, region) if __name__ == "__main__": main() Make it executable:
chmod +x /srv/scripts/run_patching.py Test it:
./run_patching.py
Step 4: Schedule via Cron (on Master)
Edit crontab:
crontab -e Add daily job:
# Run daily at 6 AM UTC 0 6 * * * /srv/scripts/run_patching.py >> /var/log/salt/patching.log 2>&1 This assumes the local time logic is handled in the script using each region’s timezone.
Security & Safety Tips
Test patching states on a few dev nodes first (salt -G 'environment:dev' -l debug state.apply patching)
Add Slack/email notifications (Salt Reactor or Python smtplib)
Consider dry-run support with test=True (in pkg.uptodate)
Use salt-run jobs.list_jobs to track job execution
Optional Enhancements
Use Salt Beacons + Reactors to monitor and patch in real-time
Integrate with JetPatch or Ansible for hybrid control
Add patch deferral logic for critical services
Write to a central patching log DB with job status per host
Overall Architecture
Minions:
Monitor the date/time via beacons
On patch day (based on local logic), send a custom event to the master
Master:
Reacts to that event via a reactor
Targets the sending minion and applies the patching state
Step-by-Step: Salt Beacon + Reactor Model
1. Define a Beacon on Each Minion
File: /etc/salt/minion.d/patchday_beacon.conf
beacons: patchday: interval: 3600 # check every hour This refers to a custom beacon we will define.
2. Create the Custom Beacon (on all minions)
File: /srv/salt/_beacons/patchday.py
import datetime from dateutil.relativedelta import relativedelta, TU import pytz __virtualname__ = 'patchday' def beacon(config): ret = [] grains = __grains__ env = grains.get('environment', 'unknown') region = grains.get('region', 'unknown') # Define rules rules = { "dev": {"offset": 3, "week": 2}, "prep": {"offset": 5, "week": 2}, "prod": {"offset": 5, "week": 3} } region_tz = { "us-east": "America/New_York", "us-west": "America/Los_Angeles", "eu-central": "Europe/Berlin" } if env not in rules or region not in region_tz: return ret # invalid or missing config tz = pytz.timezone(region_tz[region]) now = datetime.datetime.now(tz) rule = rules[env] patch_day = (datetime.date(now.year, now.month, 1) + relativedelta(weekday=TU(rule["week"])) + datetime.timedelta(days=rule["offset"])) if now.date() == patch_day: ret.append({ "tag": "patch/ready", "env": env, "region": region, "datetime": now.isoformat() }) return ret 3. Sync Custom Beacon to Minions
On the master:
salt '*' saltutil.sync_beacons Enable it:
salt '*' beacons.add patchday '{"interval": 3600}' 4. Define Reactor on the Master
File: /etc/salt/master.d/reactor.conf
reactor: - 'patch/ready': - /srv/reactor/start_patch.sls 5. Create Reactor SLS File
File: /srv/reactor/start_patch.sls
{% set minion_id = data['id'] %} run_patching: local.state.apply: - tgt: {{ minion_id }} - arg: - patching This reacts to patch/ready event and applies the patching state to the calling minion.
6. Testing the Full Flow
Restart the minion: systemctl restart salt-minion
Confirm the beacon is registered: salt '*' beacons.list
Trigger a manual test (simulate patch day by modifying date logic)
Watch events on master:
salt-run state.event pretty=True Confirm patching applied:
salt '*' saltutil.running 7. Example: patching/init.sls
Already shared, but here it is again for completeness:
update-packages: pkg.uptodate: - refresh: True - retry: attempts: 3 interval: 15 reboot-if-needed: module.run: - name: system.reboot - onlyif: 'test -f /var/run/reboot-required' Benefits of This Model
Real-time and event-driven – no need for polling or external scripts
Timezone-aware, thanks to local beacon logic
Self-healing – minions signal readiness independently
Audit trail – each event is logged in Salt’s event bus
Extensible – you can easily add Slack/email alerts via additional reactors
Goal
Track patching event completions per minion
Store patch event metadata: who patched, when, result, OS, IP, environment, region, etc.
Generate readable reports in:
CSV/Excel
HTML dashboard
JSON for API or SIEM ingestion
Step 1: Customize Reactor to Log Completion
Let’s log each successful patch into a central log file or database (like SQLite or MariaDB).
Update Reactor: /srv/reactor/start_patch.sls
Add a returner to store job status.
{% set minion_id = data['id'] %} run_patching: local.state.apply: - tgt: {{ minion_id }} - arg: - patching - kwarg: returner: local_json # You can also use 'mysql', 'elasticsearch', etc. Configure Returner (e.g., local_json)
In /etc/salt/master:
returner_dirs: - /srv/salt/returners ext_returners: local_json: file: /var/log/salt/patch_report.json Or use a MySQL returner:
mysql.host: 'localhost' mysql.user: 'salt' mysql.pass: 'yourpassword' mysql.db: 'salt' mysql.port: 3306 Enable returners:
salt-run saltutil.sync_returners Step 2: Normalize Patch Data (Optional Post-Processor)
If using JSON log, create a post-processing script to build reports:
process_patch_log.py
import json import csv from datetime import datetime def load_events(log_file): with open(log_file, 'r') as f: return [json.loads(line) for line in f if line.strip()] def export_csv(events, out_file): with open(out_file, 'w', newline='') as f: writer = csv.DictWriter(f, fieldnames=[ 'minion', 'date', 'environment', 'region', 'result' ]) writer.writeheader() for e in events: writer.writerow({ 'minion': e['id'], 'date': datetime.fromtimestamp(e['_stamp']).isoformat(), 'environment': e['return'].get('grains', {}).get('environment', 'unknown'), 'region': e['return'].get('grains', {}).get('region', 'unknown'), 'result': 'success' if e['success'] else 'failure' }) events = load_events('/var/log/salt/patch_report.json') export_csv(events, '/srv/reports/patching_report.csv') Step 3: Build a Simple Web Dashboard
If you want to display reports via a browser:
🛠 Tools:
Flask or FastAPI
Bootstrap or Chart.js
Reads JSON/CSV and renders:
Example Chart Dashboard Features:
✅ Last patch date per server
📍 Patching success rate per region/env
🔴 Highlight failed patching
📆 Monthly compliance timeline
Would you like a working example of that Flask dashboard? I can include the full codebase if so.
Step 4: Send Reports via Email (Optional)
🐍 Python: send_report_email.py
import smtplib from email.message import EmailMessage msg = EmailMessage() msg["Subject"] = "Monthly Patch Report" msg["From"] = "patchbot@example.com" msg["To"] = "it-lead@example.com" msg.set_content("Attached is the patch compliance report.") with open("/srv/reports/patching_report.csv", "rb") as f: msg.add_attachment(f.read(), maintype="text", subtype="csv", filename="patching_report.csv") with smtplib.SMTP("localhost") as s: s.send_message(msg) Schedule that weekly or monthly with cron.
Flask Dashboard (Patch Reporting)
app.py
from flask import Flask, render_template import csv from collections import defaultdict app = Flask(__name__) @app.route('/') def index(): results = [] success_count = defaultdict(int) fail_count = defaultdict(int) with open('/srv/reports/patching_report.csv', 'r') as f: reader = csv.DictReader(f) for row in reader: results.append(row) key = f"{row['environment']} - {row['region']}" if row['result'] == 'success': success_count[key] += 1 else: fail_count[key] += 1 summary = [ {"group": k, "success": success_count[k], "fail": fail_count[k]} for k in sorted(set(success_count) | set(fail_count)) ] return render_template('dashboard.html', results=results, summary=summary) if __name__ == '__main__': app.run(debug=True, host='0.0.0.0', port=5000)
templates/dashboard.html
<!DOCTYPE html> <html> <head> <title>Patch Compliance Dashboard</title> <style> body { font-family: Arial; padding: 20px; } table { border-collapse: collapse; width: 100%; margin-bottom: 30px; } th, td { border: 1px solid #ccc; padding: 8px; text-align: left; } th { background-color: #f4f4f4; } .fail { background-color: #fdd; } .success { background-color: #dfd; } </style> </head> <body> <h1>Patch Compliance Dashboard</h1> <h2>Summary</h2> <table> <tr><th>Group</th><th>Success</th><th>Failure</th></tr> {% for row in summary %} <tr> <td>{{ row.group }}</td> <td>{{ row.success }}</td> <td>{{ row.fail }}</td> </tr> {% endfor %} </table> <h2>Detailed Results</h2> <table> <tr><th>Minion</th><th>Date</th><th>Environment</th><th>Region</th><th>Result</th></tr> {% for row in results %} <tr class="{{ row.result }}"> <td>{{ row.minion }}</td> <td>{{ row.date }}</td> <td>{{ row.environment }}</td> <td>{{ row.region }}</td> <td>{{ row.result }}</td> </tr> {% endfor %} </table> </body> </html>
How to Use
pip install flask python app.py Then visit http://localhost:5000 or your server’s IP at port 5000.
Optional: SIEM/Event Forwarding
If you use Elasticsearch, Splunk, or Mezmo:
Use a returner like es_return, splunk_return, or send via custom script using REST API.
Normalize fields: hostname, env, os, patch time, result
Filter dashboards by compliance groupings
TL;DR: Reporting Components Checklist
Component
Purpose
Tool
JSON/DB logging
Track patch status
Returners
Post-processing script
Normalize data for business
Python
CSV/Excel export
Shareable report format
Python csv module
HTML dashboard
Visualize trends/compliance
Flask, Chart.js, Bootstrap
Email automation
Notify stakeholders
smtplib, cron
SIEM/Splunk integration
Enterprise log ingestion
REST API or native returners
- Read more...
- 0 comments
- 57 views
-