Alerting With SpectX

SpectX Server does not have built-in alert management. However, it can be easily implemented using SpectX API.

Alerts are based on queries that are executed on a periodical basis after some interval. The results are then passed to the desired channel(s) for the notification (email, instant messaging, SMS, etc.) In some cases, you may also want to initiate automated remediation actions based on the results, for instance, block an IP-address accessing your server in case of attack suspicion. This is a widely used approach by many operations and security teams.

The management of notification channels and event-based workflows is provided by automation platforms. As a rule, they offer RESTful API’s for integration (often called webhooks). In this example, we are going to use a simple Python script which uses integration API’s to execute a query in SpectX and pass the results to StackStorm. The Python script serves as an integration application that can easily be adapted for any other automation tool. Also, you can use either crontab utility or the automation platform of your choice to manage the scheduling of execution.

Although queries used for alerting are just like any other, there are some things to keep in mind.

Alert query must complete within the bounds of execution interval. I.e when you want your query to be run every 5 minutes, then you must make sure it gets finished in less than 5 minutes. Therefore always limit the files for query scope. Use a particular file where live updates are posted. In case you must read from already rotated files then limit the number of files by using Time Patterns.

Detecting and Remediating SSH User Enumeration Attempts With StackStorm

In this example we want to detect OpenSSH Username Enumeration vulnerability. When such an event occurs we want to be notified in Slack and block the offender’s IP-address for some period.

The vulnerability manifests itself in the authentication log (/var/log/auth.log):

Aug 16 21:42:10 victim sshd[10680]: fatal: ssh_packet_get_string: incomplete message [preauth]

Note that the IP address of the attacker is not logged. The only way to remediate is to temporarily block the offender’s IP address. To get it we need to correlate the event with firewall logs.

Step1: Prepare Target Host Environment

In this example we use iptables firewall which needs following rule inserted to enable logging incoming ssh connections:

# create chain ssh
sudo iptables -N ssh

# add rule for logging all ssh connection attempts
sudo iptables -I ssh 1 -j LOG --log-prefix='[netfilter] '

#route all new incoming tcp connects to port 22 to ssh chain
sudo iptables -A INPUT   -p TCP --dport 22   -m state --state NEW -j ssh

We’re going to use ipset utility for keeping the offender’s IP-addresses. To check the list we need to add one more rule to ssh chain:

# create a list with timeout support
sudo ipset create ssh_ban hash:ip timeout 600

# add rule for dropping connection if src ip is in the list
sudo iptables -I ssh 2 -m set --match-set ssh_ban src -j DROP

Normally iptables writes to /var/log/kern.log, where the records will be mixed with other kernel logged events. To minimize the amount of data scanned for alert it would be good to have them in separate file:

printf ':msg,contains,"[netfilter]" /var/log/iptables.log\n& stop' | sudo tee -a /etc/rsyslog.d/20-my_iptables.conf > /dev/null
sudo service rsyslog restart

Step2: Configure StackStorm

Install stackstorm-spectx pack:

st2 pack install

The pack creates a custom webhook rule, mapping the webhook trigger to the remediation workflow. The rule spectx.ssh_enum_alert defines the following:

  • webhook trigger URL: ssh_enum_alert
  • action reference: spectx.block_ipaddr the name of a workflow executed when the webhook is triggered
  • action parameters: IP address and hostname taken from SpectX query result passed via webhook
    name: "ssh_enum_alert"
    pack: "spectx"
    description: "Rule for blocking ip attempting to enumerate users using ssh"
    enabled: true

        type: "core.st2.webhook"
            url: "ssh_enum_alert"

    criteria: {}

        ref: "spectx.block_ipaddr"
            ip: "{{trigger.body.src}}"
            host: "{{}}"

The spectx.block_ipaddr is ActionChain type workflow, which does the following:

  1. Posts a notification message on Slack channel ‘ops-test’ with the hostname and offender IP-address
  2. Adds offenders IP address to the blocklist at the target host, effectively blocking the IP from further ssh connections for the next 60 seconds.
  3. Posts message on Slack notifying of IP-address blocking success at the target host.

This example requires ChatOps package installed and configured for Slack access. Create ‘ops-test’ channel and add Hubot app to it. Alternatively you can change the channel by modifying channel parameter in /opt/stackstorm/packs/spectx/actions/chains/block_ipaddr.yaml and /opt/stackstorm/packs/spectx/rules/sx_failure_alert.yaml.

The workflow uses account ubuntu for executing iplist add commands on the target host. You must configure SSH and SUDO for ubuntu. In case you’d like to use another account you must also adjust username and private_key values in /opt/stackstorm/packs/spectx/actions/chains/block_ipaddr.yaml accordingly. Add your ssh public key to ~/.ssh/authorized_keys if you’re testing on the localhost.

Step3: Configure Execution Script

The execution Python script is located in /opt/stackstorm/packs/spectx/etc/ It takes two command line arguments:

-qp SX_QUERY     path and name of SpectX query script
-wh WEBHOOK_URL  StackStorm API endpoint (webhook url)

The credentials for accessing SpectX and Stackstorm API’s must be exported as environment variables:

  • export SX_API_KEY="xxxxxxxxxxxx" : obtain from SpectX user properties API Access key.
  • export ST2_API_KEY="xxxxxxxxxxxx" : generate as shown here

Also, the API root URLs must be configured according to your setup. These are defined as global variables in the script:

# API url endpoints for SpectX and StackStorm:
sx_api_root = "http://localhost:8388/API/v1.0/"
ss_api_root = ""

Step4: Test

To test if our setup works create the following query:

// Emulate ssh username enumeration detection event:
          host:'localhost',       //target host. Replace this according to your needs.
          message:'testing ssh_enum_detection')

Save it as /shared/alerts/ Now execute Python script manually:

/usr/bin/python /opt/stackstorm/packs/spectx/etc/ -qp='/shared/alerts/' -wh=ssh_enum_alert

Upon successful execution the script exits with 0. The workflow should have all steps successfully executed (use st2 execution list cmd to get execution id):

$ st2 execution get 5bc861de8a70f5462bca5f92
id: 5bc861de8a70f5462bca5f92
action.ref: spectx.block_ipaddr
  host: localhost
status: succeeded (4s elapsed)
result_task: report_success
  channel: ops-test
  extra: null
  message: Blocked on localhost for 60 seconds
    channel: ops-test
    extra: null
    message: Blocked on localhost for 60 seconds
    user: null
    whisper: false
  user: null
  whisper: false
start_timestamp: Thu, 18 Oct 2018 10:35:10 UTC
end_timestamp: Thu, 18 Oct 2018 10:35:14 UTC
| id                       | status                 | task            | action               | start_timestamp               |
| 5bc861df8a70f5460cb03521 | succeeded (0s elapsed) | notify_on_slack | chatops.post_message | Thu, 18 Oct 2018 10:35:11 UTC |
| 5bc861e08a70f5460cb03523 | succeeded (1s elapsed) | block_ip        | core.remote          | Thu, 18 Oct 2018 10:35:12 UTC |
| 5bc861e18a70f5460cb03525 | succeeded (0s elapsed) | report_success  | chatops.post_message | Thu, 18 Oct 2018 10:35:13 UTC |

You should also see a notification in Slack:


Step 5: Prepare Query

First, you need to define a datastore for accessing logs at the target host. If you’re experimenting with localhost then define a file:// datastore named ‘varlog’ in the /system/datastores (set root to /var/log).

Otherwise, once you have access to the host you can browse to the log files, copy the URI to clipboard and replace the src: argument values respectively in the query script below:

$syslogPattern = <<<EOP
  TIMESTAMP('MMM d HH:mm:ss', tz='GMT'):timestamp
  ' ' LD:host ' ' [!:\[]+:process ('[' INT:pid ']')? ':'?
  ' ' DATA:message EOL

$iptablesPattern = <<<EOP
  TIMESTAMP('MMM d HH:mm:ss'):timestamp  ' ' LD:host ' kernel: ' ('[' DOUBLE:num ']') ' ' '[' LD:action '] ';
  'IN=' LD:in ' ' ('PHYSIN=' LD:physin ' ')?
  'OUT=' LD*:out ' ' ('PHYSOUT=' LD:physout ' ')?
  ('MAC=' MACADDR:srcMac ':'MACADDR:dstMac ':' LD:frameType ' ')?
  'SRC=' IPV4:src ' '
  'DST=' IPV4:dst ' '
  'LEN=' INT:len ' '
  'TOS=' HEXINT:typeOfService ' '
  'PREC=' HEXINT:precedence ' '
  'TTL=' INT:ttl ' '
  'ID=' INT:id ' '
  ('DF' ' ')?
  'PROTO=' LD:proto ' '
  'SPT=' INT:sport ' '
  'DPT=' INT:dport ' '
  LD //ignore state

// auth.log is where the ssh username enumeration attempts are appearing:
@authlog = PARSE(
    src:'file://varlog/auth.log')            //replace the uri to auth.log according to your needs
| filter(timestamp > now()[-5 min])          //we only need last 5 min data as we're going to execute
;                                            //the script every 5 minutes

// log messages from iptables provide us the ip-address of offender:
@iptableslog = PARSE(
    src:'file://varlog/iptables.log')        //replace the uri to iptables log according to your needs
| filter(timestamp > now()[-5 min])          //we only need last 5 min data and
| filter(proto = 'TCP' and dport = 22)       //ssh connection attempts

/* To put the two together we need to join the enum attempts in auth.log with iptables.log using timestamps
   (as this is the only available common element between the two). */
| filter(message contains 'ssh_packet_get_string:')      //filter out ssh user enum attempt records from @authlog
| join(@iptableslog on left.timestamp = right.timestamp) //join the records using timestamp
| select(timestamp, host, src, message)                  //select only necessary fields as alert data

Now save the script as /shared/alerts/ and run it to make sure there’s no exceptions.

Step6: Schedule Execution

Let’s use crontab utility to execute our Python script at every 5 minutes:

*/5 * * * * export $SX_API_KEY="xxxx"; export $ST2_API_KEY="xxxx"; /usr/bin/python /opt/stackstorm/packs/spectx/etc/ -qp='shared/alerts/' -wh=ssh_enum_alert

And we’re done! Sit back, relax and enjoy bad guys getting rejected.