1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Tutorial: Create a Logstash GeoIP dashboard in Kibana 3.

Create a Logstash GeoIP dashboard in Kibana 3.

  1. sbagmeijer
    (THIS GUIDE IS FOR KIBANA 3)

    In this small example I will show you how to make a GeoIP based Dashboard for Logstash that for example can show by analyzing your access logs where your users come from.

    This example was based on Nginx but nothing stops you from doing the exact same on Linux and Windows all you have to do is make a correct pattern for or Apache logs or for IIS logs and then change the configs above accordingly.

    Also a more easy way is to make your Nginx log as json then you can avoid needing to make a pattern however since Nginx by default does not do so I will assume in this guide you not have Nginx logs in json format.

    I will start by giving some steps to install it and then further down the post I will try go a little more into the details of the configuration file. (see below picture for this).

    First make sure you have a Logstash installation up and running, you can follow as example the following guide: https://www.ulyaoth.net/threads/how-to-install-logstash-kibana-on-fedora-using-rsyslog-as-shipper.11/

    You have to skip starting Logstash if you followed the guide above.

    I assume you have done so then you can go ahead and follow the following steps.

    Step 1: Download a new logstash configuration file and delete the old one
    Code:
    $ cd /etc/logstash/conf.d/
    $ rm -rf logstash.conf
    $ wget https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/geoip/logstash.conf
    
    Step 2: Download the Nginx pattern.
    Code:
    $ cd /opt/logstash/patterns/
    $ wget https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/patterns/nginx
    
    Step 3: Download the dashboard to your pc. (not server)
    Code:
    https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/dashboards/nginxaccesslocations
    
    Now this is everything you need, you might want to change the name of the nginxaccesslog dashboard if you open the file just change the bit "Ulyaoth: Access Locations" to whatever you wish to call it.

    Now simply go to your logstash website and press the "Load file" button (little folder icon) in the right top corner and then go with your mouse over "Advanced" and click on "Choose file" now simply select the Dashboard file you downloaded at "Step 3" called "nginxaccesslocations".

    I would suggest once loaded you press the save button (next to the folder icon" in the right top corner, once you do this you can next time directly choose this Dashboard from your "Load File" button without having to upload it.

    You should now see your dashboard based on your nginx access file as below:
    [​IMG]

    Now lets explain the configuration files a little bit in more details, lets start with the smallest one that is the "nginx" pattern file.

    nginx pattern file:
    Code:
    NGINXACCESSLOG %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}
    
    All this file does is just splitting the Nginx access.log file in different variables that you then can use in Logstash, I think it is self explanatory if you have a look at a Nginx access.log example:
    Code:
    8.8.4.4 - - [24/May/2014:08:25:46 -0400] "GET /styles/default/xenforo/xenforo-smilies-sprite.png HTTP/1.1" 200 8205 "http://community.ulyaoth.net/threads/how-to-install-logstash-kibana-on-fedora-using-rsyslog-as-shipper.11/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36"
    
    some examples:
    NGINXACCESSLOG = this is the name of the pattern that you use in logstash.conf.
    %{IPORHOST:remote_addr} = this matches the 8.8.4.4
    %{INT:status}= this matches the 200 okay we got in this entry
    %{QS:request} = this matches the actual requested page or file so in this case "/styles/default/xenforo/xenforo-smilies-sprite.png"

    Lets move to the logstash.conf file now, the whole file looks as following:
    Code:
    input {
      file {
        type => nginx_access
        path => [ "/var/log/nginx/kibana/access.log" ]
        start_position => "beginning"
      }
    }
    
    filter {
      mutate {
        add_field => [ "hostip", "%{host}" ]
      }
    
    if [type] == "nginx_access" {
      grok {
        type => "nginx_access"
        patterns_dir => "./patterns"
        pattern => "%{NGINXACCESSLOG}"
      }
    
      geoip {
        source => "remote_addr"
        target => "geoip"
        database =>"/opt/logstash/vendor/geoip/GeoLiteCity.dat"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
      }
    
      mutate {
        convert => [ "[geoip][coordinates]", "float" ]
      }
    }
    }
    
    output {
      elasticsearch {
        host => localhost
      }
    }
    
    Now as you can see I have as input my nginx access.log, the important part here is that i called the type "nginx_access" because if you look further you see now the geoip part and pattern part lets split that bit up a little bit.

    Code:
    if [type] == "nginx_access" {
      grok {
        type => "nginx_access"
        patterns_dir => "./patterns"
        pattern => "%{NGINXACCESSLOG}"
      }
    
    In this bit above I am basically saying if Logstash catches anything that has the type "nginx_access" then it should apply a pattern to all of this and the pattern it should apply is the "NGINXACCESSLOG" you remember? this was the file I explained above.

    Code:
      geoip {
        source => "remote_addr"
        target => "geoip"
        database =>"/opt/logstash/vendor/geoip/GeoLiteCity.dat"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
      }
    
      mutate {
        convert => [ "[geoip][coordinates]", "float" ]
      }
    
    This bit above is the code required to make GeoIP work important here is the "source" input, the input "remote_addr" is the ip addresses that it will check, if you call your field of remote ip addresses different you have to specify this here. Also if you have the GeoIP database file on a different location for example if you have the paid version then specify this at the "database" bit. Logstash provides by default the free lite version.

    I will update this guide in the near future to explain the dashboard file in more details, once I am more confident with it myself.

    I hope this helped you since this is also for me the first time I start looking into this there might be errors and mistakes I did make, feel free to comment if you know how to do things more easy.

    Many thanks go to Espen for his help and providing examples!: blog.mrfjo.org

    [​IMG]