1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Output to Elasticsearch with rsyslog in Logstash format

Discussion in 'Random' started by sbagmeijer, Oct 10, 2014.

  1. sbagmeijer

    sbagmeijer Machine

    This post is just a reminder for myself of how to ship logs directly from rsyslog to Elasticsearch so I not have to go trough Logstash.

    I got allot of help from the people on the rsyslog mailing list and for me the following is working pretty good so far.

    Rsyslog mailing list: http://lists.adiscon.net/pipermail/rsyslog/2014-October/038567.html

    My config:

    Code:
    module(load="imfile")
    module(load="mmjsonparse")
    module(load="omelasticsearch")
    
    template(name="logstash-index"
      type="list") {
        constant(value="logstash-")
        property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
        constant(value=".")
        property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
        constant(value=".")
        property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
    }
    
    input(type="imfile" File="/var/log/nginx/access.json"
          Tag="nginxulyaoth"
          PersistStateInterval="10000"
          StateFile="nginxulyaoth"
          Severity="info"
          MaxSubmitAtOnce="20000"
          Facility="user"
          Ruleset="nginxrule")
    
    template(name="uly-nginx" type="list") {
      constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")         property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\"}")
    }
    
    ruleset(name="nginxrule") {
      action(type="mmjsonparse" name="jsonparse")
      action(type="omelasticsearch"
      server="loghost.ulyaoth.net"
      serverport="9200"
      template="uly-nginx"
      searchIndex="logstash-index"
      dynSearchIndex="on"
      errorFile="/var/log/rsyslog/ES-error.log")
    }
    

    Still working on getting the json log split into variables so it will show more nice in Kibana.

    [​IMG]
     
    Last edited: May 10, 2015

Share This Page