1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

logstash-forwarder and grok examples

Discussion in 'Random' started by sbagmeijer, Jul 11, 2015.

  1. sbagmeijer

    sbagmeijer Machine

    Just adding some logstash-forwarder examples so I have them as a reminder for the future and perhaps it will help someone else also.

    You can test your groks at this website:
    This is very useful to see if your grok does actually work correctly.

    I will provide one full example for a Nginx "access.log" and then afterwards I provide simply the configs for additional programs that I find interesting.
    Example: Nginx access.log
    Step 1:
    Create the patterns directory
    $ sudo mkdir -p /opt/logstash/patterns
    Step 2: Set the correct user and group on patterns directory
    $ sudo chown logstash:logstash /opt/logstash/patterns
    Step 3: Create a log_format for nginx
    $ sudo vi /etc/nginx/nginx.conf
    You add the following piece of code inside the "http" area this code below is default nginx so should be there already.

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '  '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
    Step 4: Add a filter inside your logstash config
    $ sudo vi /etc/logstash/logstash.conf
    Then add the following filter:
    filter {
    if [type] == "nginx-access" {
      grok {
        match => { "message" => "%{NGINXACCESS}" }
    Step 5: Add the log file and field in logstash-forwarder config
    $ sudo vi /opt/logstash-forwarder/conf/logstash-forwarder.conf
    Then inside your "files" bracket add the following:
          "paths": [
          "fields": { "type": "nginx-access" }
    Step 6: Create the nginx pattern for logstash
    $ sudo vi /opt/logstash/patterns/nginx
    And add the following:
    NGINXACCESS %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}
    Save the file and give the correct permissions to the file:
    $ sudo chown logstash:logstash /opt/logstash/patterns/nginx
    If you did it correctly then it should look like this:

    As you can see the log file is now fully split up and you can use the different fields for better graphing.

    I will now below provide some example for other logs but I just provide the grok and config, the full steps you have to follow as above.
    Last edited: Aug 5, 2015

Share This Page