Sunday, February 24, 2013

Recovering from Suricata Gone Wild

Recently I tried interacting with one of my lab Security Onion sensors running the Suricata IDS. I found the Sguil server was taking a really long time to offer services on port 7734 TCP. Since I hadn't worked with this lab system in a while, I guessed that there might be too many uncategorized events in the Sguil database. I dusted off an old blog post titled More Snort and Sguil Tuning from 2006 and took a look at the system.

First I stopped the NSM applications on the server.

sudo service nsm stop
Stopping: securityonion
  * stopping: sguil server                                [  OK  ]
Stopping: HIDS
  * stopping: ossec_agent (sguil)                         [  OK  ]
Stopping: Bro
stopping ds61so-eth1-1 ...
stopping proxy ...
stopping manager ...
Stopping: ds61so-eth1
  * stopping: netsniff-ng (full packet data)              [  OK  ]
  * stopping: pcap_agent (sguil)                          [  OK  ]
  * stopping: snort_agent (sguil)                         [  OK  ]
  * stopping: suricata (alert data)                       [  OK  ]
  * stopping: barnyard2 (spooler, unified2 format)        [  OK  ]
  * stopping: prads (sessions/assets)                     [  OK  ]
  * stopping: sancp_agent (sguil)                         [  OK  ]
  * stopping: pads_agent (sguil)                          [  OK  ]
  * stopping: argus                                       [  OK  ]
  * stopping: http_agent (sguil)                      
Next I ran a query to look for the top uncategorized events.
$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1639
Server version: 5.5.29-0ubuntu0.12.04.1 (Ubuntu)

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use securityonion_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
+---------+----------------------------------------------------------------------------------+
| count   | signature                                                                        |
+---------+----------------------------------------------------------------------------------+
| 2299160 | SURICATA STREAM Packet with invalid ack                                          |
| 2298505 | SURICATA STREAM ESTABLISHED invalid ack                                          |
| 1777530 | SURICATA STREAM ESTABLISHED packet out of window                                 |
|   38700 | SURICATA STREAM ESTABLISHED retransmission packet before last ack                |
|   24181 | SURICATA STREAM TIMEWAIT ACK with wrong seq                                      |
|    5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management |
|    3160 | SURICATA STREAM Last ACK with wrong seq                                          |
|     753 | ET POLICY Dropbox.com Offsite File Backup in Use                                 |
|     637 | SURICATA HTTP unknown error                                                      |
|     626 | SURICATA STREAM SHUTDOWN RST invalid ack                                         |
|     505 | SURICATA STREAM FIN1 FIN with wrong seq                                          |
|     494 | SURICATA HTTP request field too long                                             |
|     448 | ET POLICY PE EXE or DLL Windows file download                                    |
|     315 | ET RBN Known Malvertiser IP (22)                                                 |
|     270 | ET POLICY iTunes User Agent                                                      |
|     266 | SURICATA STREAM CLOSEWAIT ACK out of window                                      |
|     237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                   |
|     219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                      |
|     217 | SURICATA STREAM 3way handshake with ack in wrong dir                             |
|     151 | SURICATA STREAM FIN2 FIN with wrong seq                                          |
+---------+----------------------------------------------------------------------------------+
20 rows in set (15.24 sec)
Wow, that's a lot of SURICATA STREAM events. I need to categorize them as non-issues to recover the Sguil server.

mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%';
Query OK, 6443375 rows affected, 65535 warnings (3 min 4.89 sec)
Rows matched: 6443375  Changed: 6443375  Warnings: 6443375
Let's see what the database thinks now.
mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
+------+-----------------------------------------------------------------------------------------+
| cnt  | signature                                                                               |
+------+-----------------------------------------------------------------------------------------+
| 5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management        |
|  753 | ET POLICY Dropbox.com Offsite File Backup in Use                                        |
|  637 | SURICATA HTTP unknown error                                                             |
|  494 | SURICATA HTTP request field too long                                                    |
|  448 | ET POLICY PE EXE or DLL Windows file download                                           |
|  315 | ET RBN Known Malvertiser IP (22)                                                        |
|  270 | ET POLICY iTunes User Agent                                                             |
|  237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                          |
|  219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                             |
|  133 | ET INFO PDF Using CCITTFax Filter                                                       |
|  106 | ET POLICY Pandora Usage                                                                 |
|   97 | ET CHAT Facebook Chat (buddy list)                                                      |
|   93 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET                     |
|   58 | ET POLICY Internal Host Retrieving External IP via whatismyip.com - Possible Infection  |
|   41 | PADS New Asset - ssl TLS 1.0 Client Hello                                               |
|   39 | SURICATA HTTP response header invalid                                                   |
|   39 | ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client             |
|   36 | ET POLICY Python-urllib/ Suspicious User Agent                                          |
|   36 | ET MALWARE Possible Windows executable sent when remote host claims to send a Text File |
|   28 | ET POLICY Http Client Body contains pw= in cleartext                                    |
+------+-----------------------------------------------------------------------------------------+
20 rows in set (0.03 sec)
That's much better.

Before restarting the NSM services, I edit the autocat.conf file to add the following.

none||ANY||ANY||ANY||ANY||ANY||ANY||%%REGEXP%%^SURICATA STREAM||1
This will auto-categorize any SURICATA STREAM alerts as non-issues. I want to keep adding events to the database for testing purposes, but I don't want to see them in the console.

Now I restart the NSM services.

sudo service nsm start
Starting: securityonion
  * starting: sguil server                                                                [  OK  ]
Starting: HIDS
  * starting: ossec_agent (sguil)                                                         [  OK  ]
Starting: Bro
starting manager ...
starting proxy ...
starting ds61so-eth1-1 ...
Starting: ds61so-eth1
  * starting: netsniff-ng (full packet data)                                              [  OK  ]
  * starting: pcap_agent (sguil)                                                          [  OK  ]
  * starting: snort_agent (sguil)                                                         [  OK  ]
  * starting: suricata (alert data)                                                       [  OK  ]
  * starting: barnyard2 (spooler, unified2 format)                                        [  OK  ]
  * starting: prads (sessions/assets)                                                     [  OK  ]
  * starting: pads_agent (sguil)                                                          [  OK  ]
  * starting: sancp_agent (sguil)                                                         [  OK  ]
  * starting: argus                                                                       [  OK  ]
  * starting: http_agent (sguil)                                                          [  OK  ]
  * disk space currently at 22%
I check to see if port 7734 TCP is listening.
sudo netstat -natup | grep 7734
tcp        0      0 0.0.0.0:7734            0.0.0.0:*               LISTEN      10729/tclsh
Now the Sguil server is listening. I can connect with a Sguil client, even the 64 bit Windows .exe that I just found this morning. Check it out at sourceforge.net/projects/sguil/

Friday, February 22, 2013

Using Bro to Log SSL Certificates

I remember using an older version of Bro to log SSL certificates extracted from the wire. The version shipped with Security Onion is new and that functionality doesn't appear to be enabled by default. I asked Seth Hall about this capability, and he told me how to get Bro to log all SSL certs that it sees.

Edit /opt/bro/share/bro/site/local.bro to contain the changes as shown below.

diff -u /opt/bro/share/bro/site/local.bro.orig /opt/bro/share/bro/site/local.bro
--- /opt/bro/share/bro/site/local.bro.orig      2013-02-23 01:54:53.291457193 +0000
+++ /opt/bro/share/bro/site/local.bro   2013-02-23 01:55:16.151996423 +0000
@@ -56,6 +56,10 @@
 # This script enables SSL/TLS certificate validation.
 @load protocols/ssl/validate-certs

+# Log certs per Seth
+@load protocols/ssl/extract-certs-pem
+redef SSL::extract_certs_pem = ALL_HOSTS;
+
 # If you have libGeoIP support built in, do some geographic detections and
 # logging for SSH traffic.
 @load protocols/ssh/geo-data
Restart Bro.
~# broctl

Welcome to BroControl 1.1

Type "help" for help.

[BroControl] > install
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating standalone-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.
[BroControl] > status
Name       Type       Host       Status        Pid    Peers  Started            
bro        standalone localhost  running       3042   0      17 Feb 13:22:42
[BroControl] > restart
stopping ...
stopping bro ...
starting ...
starting bro ...
[BroControl] > exit

After restarting you will have a new log for all SSL certs:

ls -al certs-remote.pem
-rw-r--r-- 1 root root 31907 Feb 23 02:05 certs-remote.pem

New certs are appended to the file as Bro sees them. A cert looks like this:

-----BEGIN CERTIFICATE-----
MIIGYjCCBUqgAwIBAgIQdyRQbU+ah51Lxm5niPJgyTANBgkqhkiG9w0BAQUFADCB
ujELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNjE0MDIGA1UEAxMr
VmVyaVNpZ24gQ2xhc3MgMyBFeHRlbmRlZCBWYWxpZGF0aW9uIFNTTCBDQTAeFw0x
MjAyMjkwMDAwMDBaFw0xMzAyMjgyMzU5NTlaMIIBJjETMBEGCysGAQQBgjc8AgED
EwJVUzEZMBcGCysGAQQBgjc8AgECEwhEZWxhd2FyZTEdMBsGA1UEDxMUUHJpdmF0
ZSBPcmdhbml6YXRpb24xEDAOBgNVBAUTBzI5Mjc0NDIxCzAJBgNVBAYTAlVTMQ4w
DAYDVQQRFAU2MDYwMzERMA8GA1UECBMISWxsaW5vaXMxEDAOBgNVBAcUB0NoaWNh
Z28xGjAYBgNVBAkUETEzNSBTIExhIFNhbGxlIFN0MSQwIgYDVQQKFBtCYW5rIG9m
IEFtZXJpY2EgQ29ycG9yYXRpb24xHzAdBgNVBAsUFk5ldHdvcmsgSW5mcmFzdHJ1
Y3R1cmUxHjAcBgNVBAMUFXd3dy5iYW5rb2ZhbWVyaWNhLmNvbTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAL3mUutqncWzNlwQNaM6IJdaadkQtUBvVnyp
obSS69GgKykAiQlx8QZQGbPCpJmHxmd7gz1JRnDntjp7N6Pg/cC47RvH2GOEgBdP
oGjaqMIprDXWSOgsBg7sBG0Qu9jPdAwHKhl0pv+wbkIBY2hn2XAxM2EWmqakjbp7
ArUkrYV1/qI1LIUPoO5oGsGXYBLTafAy4fO8auz/gqYxfciUj9mWi09PAqhnB5eU
jPYqu4yF6SA1V46AhC4cmaSZdH18ZmO6onp344tvjyJOn86Erb0VPmFfc8EgbLfK
paheO7GropabCr/TKV6fhSuwcp7sDs1SC2PJhV+w6/0ZUqpp9B8CAwEAAaOCAfMw
ggHvMAkGA1UdEwQCMAAwHQYDVR0OBBYEFK333BMwfBgnezSDatzj3Y2KbimNMAsG
A1UdDwQEAwIFoDBCBgNVHR8EOzA5MDegNaAzhjFodHRwOi8vRVZTZWN1cmUtY3Js
LnZlcmlzaWduLmNvbS9FVlNlY3VyZTIwMDYuY3JsMEQGA1UdIAQ9MDswOQYLYIZI
AYb4RQEHFwYwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29t
L3JwYTAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHwYDVR0jBBgwFoAU
/IpQup65JVp7VYVPlQBjj+lYa0MwfAYIKwYBBQUHAQEEcDBuMC0GCCsGAQUFBzAB
hiFodHRwOi8vRVZTZWN1cmUtb2NzcC52ZXJpc2lnbi5jb20wPQYIKwYBBQUHMAKG
MWh0dHA6Ly9FVlNlY3VyZS1haWEudmVyaXNpZ24uY29tL0VWU2VjdXJlMjAwNi5j
ZXIwbgYIKwYBBQUHAQwEYjBgoV6gXDBaMFgwVhYJaW1hZ2UvZ2lmMCEwHzAHBgUr
DgMCGgQUS2u5KJYGDLvQUjibKaxLB4shBRgwJhYkaHR0cDovL2xvZ28udmVyaXNp
Z24uY29tL3ZzbG9nbzEuZ2lmMA0GCSqGSIb3DQEBBQUAA4IBAQB31shk3CQ/jMfz
O1h6qCm+OeWUqgCvmAf26JoBx9hiHx+sWj1/z11rLp3oEt7fiqFsj76zWXAdhyH0
bp/sPGxAD7VQJEiAvtUR7015OUyNo+qnwJk2rZNlvwZydtsEmnYywVEgLQuFm962
csbbjmAqE+ODT9wk6jbIplfqhnSj2AL4xTNS2Rj3+jKsXlZvzCBdXs8Ewq9IwocL
UpaWV6ObhXsxkgFon/KX0fS9TAams4RaPwIJzvr5ExE+NSyaufs1utdKoEwUaoS1
2Z1QVtxiueNgdFKoTATfODowb1C+IDEPJmY0urBzEhdrsMECtYxJVYBDAhbhocG6
yYpg3ayS
-----END CERTIFICATE-----
OpenSSL can read them one at a time, e.g.:
openssl x509 -in certs-remote.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            77:24:50:6d:4f:9a:87:9d:4b:c6:6e:67:88:f2:60:c9
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=Terms of use at https://www.verisign.com/rpa (c)06, CN=VeriSign Class 3 Extended Validation SSL CA
        Validity
            Not Before: Feb 29 00:00:00 2012 GMT
            Not After : Feb 28 23:59:59 2013 GMT
        Subject: 1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/businessCategory=Private Organization/serialNumber=2927442, C=US/postalCode=60603, ST=Illinois, L=Chicago/street=135 S La Salle St, O=Bank of America Corporation, OU=Network Infrastructure, CN=www.bankofamerica.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bd:e6:52:eb:6a:9d:c5:b3:36:5c:10:35:a3:3a:
                    20:97:5a:69:d9:10:b5:40:6f:56:7c:a9:a1:b4:92:
                    eb:d1:a0:2b:29:00:89:09:71:f1:06:50:19:b3:c2:
                    a4:99:87:c6:67:7b:83:3d:49:46:70:e7:b6:3a:7b:
                    37:a3:e0:fd:c0:b8:ed:1b:c7:d8:63:84:80:17:4f:
                    a0:68:da:a8:c2:29:ac:35:d6:48:e8:2c:06:0e:ec:
                    04:6d:10:bb:d8:cf:74:0c:07:2a:19:74:a6:ff:b0:
                    6e:42:01:63:68:67:d9:70:31:33:61:16:9a:a6:a4:
                    8d:ba:7b:02:b5:24:ad:85:75:fe:a2:35:2c:85:0f:
                    a0:ee:68:1a:c1:97:60:12:d3:69:f0:32:e1:f3:bc:
                    6a:ec:ff:82:a6:31:7d:c8:94:8f:d9:96:8b:4f:4f:
                    02:a8:67:07:97:94:8c:f6:2a:bb:8c:85:e9:20:35:
                    57:8e:80:84:2e:1c:99:a4:99:74:7d:7c:66:63:ba:
                    a2:7a:77:e3:8b:6f:8f:22:4e:9f:ce:84:ad:bd:15:
                    3e:61:5f:73:c1:20:6c:b7:ca:a5:a8:5e:3b:b1:ab:
                    a2:96:9b:0a:bf:d3:29:5e:9f:85:2b:b0:72:9e:ec:
                    0e:cd:52:0b:63:c9:85:5f:b0:eb:fd:19:52:aa:69:
                    f4:1f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Subject Key Identifier:
                AD:F7:DC:13:30:7C:18:27:7B:34:83:6A:DC:E3:DD:8D:8A:6E:29:8D
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://EVSecure-crl.verisign.com/EVSecure2006.crl

            X509v3 Certificate Policies:
                Policy: 2.16.840.1.113733.1.7.23.6
                  CPS: https://www.verisign.com/rpa

            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:
                keyid:FC:8A:50:BA:9E:B9:25:5A:7B:55:85:4F:95:00:63:8F:E9:58:6B:43

            Authority Information Access:
                OCSP - URI:http://EVSecure-ocsp.verisign.com
                CA Issuers - URI:http://EVSecure-aia.verisign.com/EVSecure2006.cer

            1.3.6.1.5.5.7.1.12:
                0`.^.\0Z0X0V..image/gif0!0.0...+......Kk.(.....R8.).K..!..0&.$http://logo.verisign.com/vslogo1.gif
    Signature Algorithm: sha1WithRSAEncryption
         77:d6:c8:64:dc:24:3f:8c:c7:f3:3b:58:7a:a8:29:be:39:e5:
         94:aa:00:af:98:07:f6:e8:9a:01:c7:d8:62:1f:1f:ac:5a:3d:
         7f:cf:5d:6b:2e:9d:e8:12:de:df:8a:a1:6c:8f:be:b3:59:70:
         1d:87:21:f4:6e:9f:ec:3c:6c:40:0f:b5:50:24:48:80:be:d5:
         11:ef:4d:79:39:4c:8d:a3:ea:a7:c0:99:36:ad:93:65:bf:06:
         72:76:db:04:9a:76:32:c1:51:20:2d:0b:85:9b:de:b6:72:c6:
         db:8e:60:2a:13:e3:83:4f:dc:24:ea:36:c8:a6:57:ea:86:74:
         a3:d8:02:f8:c5:33:52:d9:18:f7:fa:32:ac:5e:56:6f:cc:20:
         5d:5e:cf:04:c2:af:48:c2:87:0b:52:96:96:57:a3:9b:85:7b:
         31:92:01:68:9f:f2:97:d1:f4:bd:4c:06:a6:b3:84:5a:3f:02:
         09:ce:fa:f9:13:11:3e:35:2c:9a:b9:fb:35:ba:d7:4a:a0:4c:
         14:6a:84:b5:d9:9d:50:56:dc:62:b9:e3:60:74:52:a8:4c:04:
         df:38:3a:30:6f:50:be:20:31:0f:26:66:34:ba:b0:73:12:17:
         6b:b0:c1:02:b5:8c:49:55:80:43:02:16:e1:a1:c1:ba:c9:8a:
         60:dd:ac:92
Since each cert has a standard header and footer, I bet someone could write a parser to extract each cert from the certs-remote.pem file to separate files. Thanks a lot Seth!

Monday, February 11, 2013

Practical Network Security Monitoring Book on Schedule

First the good news: my new book Practical Network Security Monitoring is on track, and you can pre-order with a 30% discount using code NSM101.

I'm about 1/3 of the way through writing the book. Since I announced the project last month, I've submitted chapters 1, 2, and 3. They are in various stages of review by No Starch editors and my technical editors. I seem to be writing more than I expected, despite trying to keep the book at an introductory level. I find that I want to communicate the topic sufficiently to make my point, but I try to avoid going too deeply into related areas.

I'm also encountering situations where I have to promise to explain some concepts later, rather than explain everything immediately. I believe once I get the first chapter ironed out with the editor, the rest will be easier to digest. I'm taking a fairly methodical approach (imagine that), so once the foundation in chapter 1 is done the rest is more straightforward.

I'm keeping a fairly aggressive schedule. Basically I have to write a chapter each week, get it to my technical editors, and then spend additional time working with No Starch to get the text legible and ready for print. All of this is happening in parallel in order to have the books in print by Black Hat. That means the text must done by the first week in April. My family is helping me stay on track by giving me time and space to write, especially on the weekends. Thank you!

When working on the examples, I've been very pleased with the performance of VMWare Workstation 9. I have one copy installed on Windows 7, where I write with Word. I have a second copy installed on Ubuntu Server, where it acts like a "VMWare Server." I used to run a real ESXi server on server-class hardware. Now, to save electricity and to more tailor my computer power to my requirements, I run a Shuttle DS61 with a Core i5-3450S 2.80GHz CPU, 16 GB RAM, 750 GB HDD, and two onboard NICs. The two NICs are really awesome in a device this small -- 190(L) x 165(W) x 43(H) mm. With two NICs, I can devote one for management and one for network traffic collection and interpretation. I use a Net Optics Dual Port Aggregator Tap for access to the wire.

I use VMWare Workstation this way. I run a Linux VM on Workstation on my Windows 7 laptop. I connect via Workstation to the Workstation instance on Ubuntu on the DS61. Then I create whatever VMs I need on the DS61. For example, I created a Security Onion server and sensor to test that setup. With 16 GB RAM, I have plenty of RAM for both, plus another VM that I'm running as my "production" Security Onion sensor for the lab network.

Writing is going well, despite the fact that I last wrote a book in 2005. I promised my youngest daughter, who wasn't born until 2006, that this new book is for her. If you have any questions on the writing process, please post them here or ask me on Twitter.