1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Tutorial: How to Install Logstash and Kibana 4 on Fedora with rsyslog.

How to Install Logstash and Kibana 4 on Fedora with rsyslog.

  1. sbagmeijer
    In this guide I will provide an example of how to set up a Logstash server with a Kibana interface that does get the logs from rsyslog. While there are multiple other ways to get logs into Logstash I will focus in this guide on rsyslog only.

    I am aware that in the new Logstash rpm everything such as Kibana is merged into one package, But I feel personally it is better to install things separate as this gives you the possibility to update certain parts when you want without having to wait for a new rpms.

    If you are going to use this in a production environment then please make sure to check the security implications of going the rsyslog way as you would need to open a port. So unless you are in an internal network everyone will be able to ship logs to your Logstash server.

    So what is Logstash!?:
    "Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, Logstash comes with a web interface for searching and drilling into all of your logs."

    There are a lot of examples on the official Logstash so I definitely recommend having a look there!
    Their website: http://www.logstash.net

    For the people that just want a quick Logstash server and do not care about the full guide simply run this command as root and it does all steps from the guide automatically:
    Code:
    $ wget https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/install-logstash-rsyslog.sh ; chmod +x install-logstash-rsyslog.sh ; ./install-logstash-rsyslog.sh -o install
    
    *small reminder*:
    Fedora 23 uses "dnf" instead of "yum", so if you insist on using this guide or the above script on another rpm based distro such as rhel, scientific linux, oracle linux or an older fedora then please replace all dnf commands with "yum".

    Now let's start, for this guide I will be using the following programs:
    Fedora (I am using Fedora 23 for this guide)
    Logstash
    rsyslog
    ElasticSearch
    Nginx
    Kibana 4

    Step 1: Import the Logstash and Elasticsearch GPG key.
    Code:
    $ sudo rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
    Step 2: Go to your yum repository directory.
    Code:
    $ cd /etc/yum.repos.d/
    Step 3: Download the Logstash and Elasticsearch repository files.
    Code:
    $ sudo wget https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/repository/logstash.repo
    $ sudo wget https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/repository/elasticsearch.repo
    Step 4: Install the Ulyaoth repository to your server.
    Code:
     $
    dnf install https://downloads.ulyaoth.net/rpm/Fedora/x86_64/ulyaoth-1.0.10-1.fc23.x86_64.rpm
    
    If you are using another Fedora or RHEL version please have a look here if your repository is supported: https://www.ulyaoth.net/resources/ulyaoth-repository.6/

    Step 5: Install all required packages
    Code:
    $ sudo dnf install -y ulyaoth-nginx ulyaoth-kibana java elasticsearch logstash rsyslog tar wget policycoreutils-python zip
    Step 5: Reload the systemd daemon.
    Code:
    $ sudo systemctl daemon-reload
    Step 6: Go to the Logstash config directory
    Code:
    $ cd /etc/logstash/conf.d
    Step 7: Download the following Logstash config file
    Code:
    $ sudo wget https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/rsyslog/logstash.conf
    Step 8: Change the ownership of the Logstash config file
    Code:
    $ sudo chown logstash:logstash logstash.conf
    Step 9: Create the following directories:
    Code:
    $ sudo mkdir -p /var/log/nginx/kibana
    
    Step 10: Change the owner ship on the kibana nginx log folder.
    Code:
    $ sudo chown nginx:adm /var/log/nginx/kibana
    
    Step 11: wget the kibana vhost file
    Code:
    $ sudo wget https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/nginx/vhost/kibana4.conf -O /etc/nginx/sites-available/kibana.conf
    Step 12: Open the kibana vhost file
    Code:
    $ sudo vi /etc/nginx/sites-available/kibana.conf
    Step 13: Change the site name
    Simply change the "logstash.ulyaoth.net" to whatever your logstash url will be and save the file.

    Step 14: Symbolic link the vhost file so nginx will load it
    Code:
    $ sudo ln -s /etc/nginx/sites-available/kibana.conf /etc/nginx/sites-enabled/kibana.conf

    Step 15:
    Fix selinux
    Code:
    $ sudo semanage port -a -t http_port_t -p tcp 9200
    $ sudo semanage port -a -t http_port_t -p tcp 5601
    
    Step 16: Fix firewalld
    Code:
    $ firewall-cmd --permanent --zone=FedoraServer --add-service=http
    $ firewall-cmd --permanent --zone=FedoraServer --add-service=https
    $ firewall-cmd --permanent --zone=FedoraServer --add-port=5544/udp
    
    Please be aware that the zone can depend on your setup or os version.

    Step 17: Restart firewalld.
    Code:
    $ sudo systemctl restart firewalld.service
    
    Step 18: Put Logstash, ElasticSearch, Nginx and Kibana on autostart.
    Code:
    $ sudo systemctl enable elasticsearch.service
    $ sudo systemctl enable logstash.service
    $ sudo systemctl enable nginx.service
    $ sudo systemctl enable kibana.service
    
    Step 19: Start the services in the order below ie "elasticsearch -> logstash -> nginx -> kibana".
    Code:
    $ sudo systemctl start elasticsearch.service
    $ sudo systemctl start logstash.service
    $ sudo systemctl start nginx.service
    $ sudo systemctl start kibana.service
    
    If you now go to your website for example for me "http://logstash.ulyaoth.net" you will see something like this:
    [​IMG]
    Make sure to choose the same options as I did above and then press on "Create" this will finish the Kibana configuration and you can start using it afterwards.

    Logstash is a product that is always in development so the screenshot above is outdated probably by now as they keep changing the interface.

    This is it everything should be working now :) you should now be seeing something like this if you go to your Logstash website:
    [​IMG]

    You probably question how come there are already logs, well this is because I added the following config already to the logstash.conf.
    Code:
    file {
    type => syslog
    path => [ "/var/log/nginx/kibana/*.log", "/var/log/nginx/*.log" ]
    }
    
    You can simply remove it or keep it.

    You are now ready to ship your logs to port "5544" and they should show up in Kibana! Congratulations!

    *problems that could occur*
    There is a bug in Logstash currently that it can only handle utf8 if your log is different then this it will crash Logstash a workaround is as you can see above to add the following:

    Code:
    codec => plain { charset => "ISO-8859-1" }
    I hope this guide has helped you if you see any mistakes or have improvements please give me a reply and I will update the guide accordingly I am always happy to hear improvements.

    [​IMG]

Recent Updates

  1. Updated Logstash & Elasticsearch.