Cyber Threat Monitoring System with Ossec + ZeroMQ + Logstash + ElasticSearch and Kibana

Hi

I’ve been using Ossec as Intrusion Detection System for year. Ossec is awesome service for detection and notification. Thus I’ve decided to build a cyber threat monitoring system with open source technologies. In order to do that, I decided to get logs from Ossec and send them to the Elasticsearch engine. This write-up going to be about installation of Ossec and Logstash/ElastichSearch and integration between of these services.

Technologies

Operating System

I used Centos 6.x x64 bit minimal iso on  4gb ram, 150gb SSD disk and 4 CPU VPS server. Everything going to be installed this server.

ZeroMQ as Messaging Service

We have to collect logs from Ossec and send them to Logstash. There is a few different way to do that but I decided to use ZeroMQ because Ossec and Logstash has a support for ZeroMQ. Ossec with ZeroMQ is able to parsing fields from log data.Also tracing file and try to parse it with logstash is not easy, for me at least.

Our plan is Ossec going to start ZeroMQ service at localhost with specified port. Logstash going to be subscriber on that server and get log data directly from Ossec.

ElasticSearch

Nothing to say about it.. We will be using latest version of ELS.

Kibana ( with secure Nginx Installation )

I will explain Nginx configuration for kiban. We need to enable some extra security features in order to keep kibana in secure!

Let start.

Installation of Required Libs

Before do anything, lets install developments tools and update our Centos server.

Lets start with installation of libzmq. Please execute following commands in order to install latest version of libzmq

That’s it. Now please execute following commands for stable ZeroMQ version ( zeromq4-x ) installation.

Optional Step:

You may getting following error while trying to run bash autogen.sh .

You gotta instal higher version of autoconf by following below steps.

If you want to use ZeroMQ feature of Ossec, you should install czmq library otherwise you wont be able to compile ossec. Let’s install czmq with following commands.

 Ossec Installation

First of all, I won’t describe details of install.sh process of Ossec because of it’s pretty good documented at online. (

Most important thing here is enabling zeromq. Please do not forget execute 4th line.

I will be using ossec with english language.

 

We are installing ossec as a server. Please choose “server”.

/var/ossec is pretty good location for ossec. I wont change it. Press enter and move another step.

I will enable email notification.

Enable integrity check.

And ofcourse enable rootkit detection.

I will not enable active response it.

Ossec ZeroMQ Configuration

Please add following line into <global> xml tag. (/var/ossec/etc/ossec.conf)

Ossec going to take 5556 tcp port for publish mode. We will configure  logstash as a subscriber mode. We are ready to start ossec service.

In order to be sure about zeromq services that started by ossec, please execute following command and you should see one single as output

Elasticsearch

First thing first. We need java.

Now we are free to go.  Let’s add elasticsearch’s repo and install it.

Create elasticsearch.repo file under the /etc/yum.repos.d/ folder and add following lines into that file.

Install ELS and add it into startup services list.

Now open /etc/elasticsearch/elasticsearch.yml and add following lines end of yml file.

and restart elasticsearch service.

Optional :  If you see “permission denied on key ‘vm.max_map_count’” error remove/disable following from /etc/init.d/elasticsearch file.

Installation Kibana + Nginx with Security

We need epel repo for nginx and install it with httpd-tools package.

Kibana does not have authentication mechanism. That means anyone can access Kibana directly from outside. Thus, enabling HTTP Auth is good solution.

PS: Don’t forget that max lenght is 8 for htpasswd.

Now it’s time to install Kibana

Nginx will be located as reverse proxy mode. Following configuration can be found here also gist ( https://gist.github.com/mmetince/b0d44cc14e4c4c10cd64 )

Also before move forward, you should  generate certificate for kibana and locate them as described nginx configuration.

Please execute following lines for be sure cert’s owner and permissions.

We need to do small changes on Kibana config.js file. Please open /usr/share/nginx/html/yourdomain/kibana/config.js file and set elasticsearch: to following one. So Kibana GUI can access ELS without having any problem.

Installation of LogStash

UPDATE : Since the Logstash made changes on their ffi-rzmq gem package, this section is NOT necessary to follow. ( https://github.com/logstash-plugins/logstash-input-zeromq/pull/1 ). I haven’t test lastest version of Logstash but it seems this problem is not exist anymore. 

Everything should be okay until now. But logstash have a lot of bug about using zeromq.. I’ve spend a day for figure it out. Let me explain what I’ve tried

Goal

We want to connect Ossec’s zeromq service and get log stream from it.

Issues

First of all, I’ve installed logstash from repo which was 1.3.x version. When I’ve run service it couldn’t connect ossec’s service. Because Logstash using ffi-rzmq 1.0.0 version gem package and it’s incompatible with libzmq 4.x library. I found a issue ticket about it ( over here ) . They said that they solved this issue on 2.0.1 version.

I thought that ffi-rzmq issue must be solved on logstash stable version ( 1.4.x right now ) . I’ve download logstash tarball and start service but I saw that latest version is using ffi-rzmq with 1.0.0 again.

After few hours I’ve found this issue ticket ( https://github.com/elasticsearch/logstash/issues/1870 ) . Last comment is “you are right about the gemspec role, if you update the version in the gemspec and runbin/logstash deps the library will be updated. If you manage to test it, please add your result to #1366this would help getting it merged.” which means they know that issue and try to fix it.  I decided to change version to 2.0.1 from 1.0.0 on gemspec file and build my own tarball.

Let’s build our own logstash tarball. PS : 1.4 is stable version right now.

Now open logstash.gemspec  file and do following changes.

Lets fetch dependencies which includes ffi-rzmq with 2.0.1 version.

This will take few minutes. You probably see outputs like that…

It’s finished. We are free to make our own tarball with following command.

Output will be like this.

Let’s move tarball to another directory and excrat files

Sample Logstash Config for Ossec

We are ready. Let’s do some tests.. Create sample file /etc/sample.conf

And start logstash with following command.

YAY! Logstash working like a charm.. Following screenshot grabbed from output of logstash

ossec-logstash-zeromq-sample-output

Let’s check out Kibana. Please go https://yourdomain.com/kibana and enter your username & password that we generated at previous section.

ossec-logstash-kibana

 

Init.d Script For Logstash

Following commands will move logstash folder to under /opt folder and create init.d script for logstash.

Benefits of Using ZeroMQ instead of File as Logstash Input

As you can see, important field such as rule_comment, rule_level, srcip, dstuser are already coming as parsed from Ossec ZeroMQ. That means we do not need to work on parse file data. We have attacker ip address named as srcip so we can easily enable logstash geo_ip feature. Also if you want to strip out more information. You can find everything else on full_log variable. Also index can be changes depends on attack type.

Thanks

You would like to say a “big” thank you to Jeremy Rossi ( @jrossi ) who developed zeromq feature of Ossec!

References

http://pragmasec.wordpress.com/2014/07/14/set-up-ossec-with-kibana-and-logstash/

http://vichargrave.com/ossec-log-management-with-elasticsearch/

http://vichargrave.com/improved-ossec-log-parsing-with-logstash/

 

 

 

  • Michael Woode

    I am unable to see logs in kibana even though “bin/logstash -f /tmp/sample.conf” works fine. What am I doing wrong ? Thanks

    • mehmet ince

      Hey
      Are you sure ossec is working ? What is output of netstat -tnlp|grep 5556 ? if you see output that means ossec is working. Please add user to system so ossec will generate log. Because another case could be ossec didnt generate log because there wasn’t a cyber attack.

  • Julio Cesar

    While compiling (czmq) in ./configure I’m getting this error:

    checking for zmq… configure: error: Package requirements (libzmq) were not met:

    No package ‘libzmq’ found

    Consider adjusting the PKG_CONFIG_PATH environment variable if you
    installed software in a non-standard prefix.

    Alternatively, you may set the environment variables zmq_CFLAGS
    and zmq_LIBS to avoid the need to call pkg-config.
    See the pkg-config man page for more details.

    • mehmet ince

      Did you install libzmq ? In order to install czmq you need to install libzmq before. Also you may need load your library path to ldconfig with following commands (echo /usr/local/lib > /etc/ld.so.conf.d/local.conf AND ldconfig) so you can access to libzmq shared libraries without write full path. I will try to fresh install on centos 6.x when I’ve a time. I will let you know and append to post if any extra configuration/command needed.

  • Murat

    Bilgi çok güzel fakat türkçe yayınlarmısın

  • Bobby

    Do you think it would work under CentOS 7?
    If yes, I will try and output results.

    • mehmet ince

      I believe it will work with little modifications. Please let us know the results.

  • CraigL

    While running the configure command for libzmq I am constantly coming up against this issue:

    No package ‘libsodium’ found

    I have installed libsodium, even tried to manually adjust this via the configure switch –with-libsodium=/root/to/libsodium/

    I am installing on CentOS 6.5 x64, can anyone make any recommendations? Assist in any way?

    • I will check it out as soon as possible but I need a few days ‘cuz of busy schedule.

      • CraigL

        Much appreciated :) No probs, we all have them! Will report back if I manage to fix it

      • CraigL

        I managed to fix this by adding this environment variable:

        export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

        After this everything installs perfectly, there are some other pre-requisites which were required for libzmq, these are libsodium, which in turn had its own pre-requisite of autoconf 2.65 or higher.

        I’m going to get a clean box and run through this again, will post my findings somewhere

        • It seems that I’ve forgot to mention these parts at article :) Thank you. I will go through from beginning and update article with missing sub-steps.

          • CraigL

            lol – I’m getting trolled by OSSEC now…!

            Starting OSSEC HIDS v2.8 (by Trend Micro Inc.)…
            OSSEC analysisd: Testing rules failed. Configuration error. Exiting.

            The only edits I have made to ossec.conf is adding the 2 zeromq lines in the section…

            Also, while I remember about ossec, the commands for accessing the stable branch were different from what the ones you supplied, I used:

            git clone https://github.com/ossec/ossec-hids.git
            git checkout remotes/origin/stable
            git checkout -b stable remotes/origin/stable

            Then verified the active branch with:

            [root@ossec ossec-hids]# git branch
            master
            * stable

          • CraigL

            No logs in ossec.log, posted help to ossec-list on google groups and dan recommended ossec-logtest -t which gave me this error:

            /var/ossec/bin/ossec-logtest: error while loading shared libraries: libczmq.so.1: cannot open shared object file: No such file or directory

            Fixed this issue with this command:

            export LD_LIBRARY_PATH=”/usr/local/lib”

          • Glad you’ve solved it!! Which steps should I append export LD_LIBRARY_PATH=”/usr/local/lib” step ? I can update article in order to prevent communities from same issues :)

          • CraigL

            You would just need it to be before you try to start ossec, lol, also having library issues starting nginx (libperl.so) now, I’m thinking of doing it a different way as per digitalocean instructions for the nginx bit on CentOS 6, will let you know how I get on

          • Thanks so much for posting this fix @disqus_Evs2QEcO6R:disqus . I was going crazy trying to figure it out.

  • On instruction #4 above in the OSSEC section make setzeromq I’m getting [root@logserver src]# make setzeromq

    make: *** No rule to make target setzeromq’. Stop.`

    Any ideas?

  • @mmetince:disqus in the OSSEC config section above, consider changing this “Please add following line into xml tag.” to this “Please add following line into xml tag in /var/ossec/etc/ossec.conf”
    Just to help out the newbies who might not know where that info goes.

    • Hey @rickchatham:disqus I’ve made these changes on document. Thank you for your feedback.

  • Also typo here:
    vi /etc/yum.repo.d/elasticsearch.repo
    Should be:
    vi /etc/yum.repos.d/elasticsearch.repo

  • ahmed

    no output when iam using netstat -tnlp |grep 5556 !
    also iam getting error when doing command ( make setzeromq ) which is :
    make: *** No rule to make target `setzeromq’. Stop.

  • Hello,

    I had some problem when I was following this article. I want share solutions with you.

    1. When I compiled ossec with zeromq. I got this error.

    make: *** No rule to make target `setzeromq’. Stop.

    After I compiled ossec following command. This problem has been resolved.

    make USE_ZEROMQ=1

    2. When I started ossec with following command.

    /var/ossec/bin/ossec-control start

    I got this error.

    OSSEC analysisd: Testing rules failed. Configuration error. Exiting.

    After I executed following command.

    export LD_LIBRARY_PATH=/usr/local/lib

    Problem has been resolved.