Adding Logstash Filters To Improve Centralized Logging

Hi. I am trying to troubleshoot logstash not collecting nginx logs. I followed the tutorial on setting up elasticsearch, kibana and logstash now this one, but the nginx logs don’t seem to be flowing through.

Is this logstash-forwarder config correct?

  "network": {
    "servers": [ "localhost:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },

  "files": [
    {
      "paths": [
        "/var/log/syslog",
        "/var/log/auth.log"
       ],
      "fields": { "type": "syslog" }
    }


    {
      "paths": [
        "/var/log/nginx/ElasticSearch01.access.log"
       ],
      "fields": { "type": "nginx-access" }
    }
           ]

Thanks Jock

@jock.forrester: There’s a missing comma between the two “paths” objects. Try using this instead:

   "network": {  
      "servers": [  
         "localhost:5000"
      ],
      "timeout": 15,
      "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
   },
   "files": [  
      {  
         "paths": [  
            "/var/log/syslog",
            "/var/log/auth.log"
         ],
         "fields": {  
            "type": "syslog"
         }
      },
      {  
         "paths": [  
            "/var/log/nginx/ElasticSearch01.access.log"
         ],
         "fields": {  
            "type": "nginx-access"
         }
      }
   ]

Hi, I’m using cisco ASA 5505. When i enter to /opt/logstash/patterns/firewalls i dont find the ASA 5505. Also i cant change the option … All I want is to have ip source ; ip destination ; port source ; port destination as field in kabana. Thanks

@sammdoun post a sample log

{“message”:“<166>Aug 20 2014 05:51:34: %ASA-6-302014: Teardown TCP connection 8440 for inside:192.168.2.209/51483 to outside:104.16.13.8/80 duration 0:00:53 bytes 13984 TCP FINs\n”,“@version”:“1”,“@timestamp”:“2014-08-20T14:17:58.452Z”,“host”:“192.168.2.1”,“tags”:[“_grokparsefailure”],“priority”:13,…

@sammdoun: Assuming your message is (and the rest of the relevant logs are similar):

<166>Aug 20 2014 05:51:34: %ASA-6-302014: Teardown TCP connection 8440 for inside:192.168.2.209/51483 to outside:104.16.13.8/80 duration 0:00:53 bytes 13984 TCP FINs\n

The following pattern should match and name the fields you specified:

<%{INT}>%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}: %{GREEDYDATA}inside:%{IP:source_ip}/%{INT:source_port} to outside:%{IP:destination_ip}/%{INT:destination_port}

I’m assuming the first IP/port is source and the second is destination.

ok :) but how i can add them as field in my kibana

Can you see anything wrong with my logstash-forwarder config? Logs are not sent when configured as follows;

{
  "network": {
    "servers": [ "host:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "/var/log/apache2/*.log" ],
      "fields": { "type": "apache-all" }
    },
   "files": [  
      {  
         "paths": [  
            "/var/log/syslog",
            "/var/log/auth.log"
         ],
         "fields": {  
            "type": "syslog"
         }
      }
   ]
}

Logs are sent ok when it is configured as follows;

{
  "network": {
    "servers": [ "host:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "/var/log/apache2/*.log" ],
      "fields": { "type": "apache" }
    }
   ]
}

@manicas ! thank you verry much ! i’m working on SIEM project and you really help me. Actually the system used to work two weeks ago, but now i have an error message which is " Oops! SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]" i think it’s due to server private ip adress, because it’s dhcp, i’ve generated another ssl certificate, with the new adress, but i still have this error ! can you help me please !

How to install plugins? bin/plugin not inside elasticsearch directory.

if we want to add apache error and modsecurity so it would like for apache error {

if [type] == “apache-error” { grok { match => { “message” => “%{APACHEERRORLOG}” } }

is that the right configuration. Or need some change that’s the filter my access log is working fine so do i need to make another input file for error log kindly help

That is my forwarder { “network”: { “servers”: [ “ip:5000” ], “timeout”: 15, “ssl ca”: “/etc/pki/tls/certs/logstash-forwarder.crt” }, “files”: [

{ “paths”: [ “/var/log/syslog”, “/var/log/auth.log” ], “fields”: { “type”: “syslog” } } , { “paths”: [ “/var/log/apache2/access.log” ], “fields”: { “type”: “apache-access” } } , { “paths”: [ “/var/log/apache2/error.log” ], “fields”: { “type”: “apache-error” } }

]

Here is my 12-apacheerror.conf

filter { if [type] == “apache-error” { grok { match => { “message” => “%{APACHEERRORLOG}” } } } }

But i am unable to see error logs in Kibana where as access logs are coming

Hi,

i am trying to collect the apache logs, i follow your config. after editing the logstash-forwader config i do, sudo service logstash-forwarder force-reload shows no process is running. i check the status: sudo service logstash-forwarder status shows:

• logstash-forwarder is not running

logstash log:

{:timestamp=>“2014-12-05T18:35:21.509000+0800”, :message=>“±--------------------------------------------------------+\n| An unexpected error occurred. This is probably a bug. |\n| You can find help with this problem in a few places: |\n| |\n| * chat: #logstash IRC channel on freenode irc. |\n| IRC via the web: http://goo.gl/TI4Ro |\n| * email: logstash-users@googlegroups.com |\n| * bug system: https://logstash.jira.com/ |\n| |\n±--------------------------------------------------------+\nThe error reported is: \n Address already in use - bind - Address already in use”}

but i can able to see syslog & auth.log on kibana dashboard.

can you help me …

Thanks

Thank you for sharing your article series , I have done successfully without encountering problems. I 'm doing one LAB “logstash with mysql” , but no success . you have articles about this issue . thank you

Hi, I’m trying to parse a log file that comes with JavaStackTrace events I use multiline codec but i can’t get the right output. I need to combine the stack trace with its event. Anyone can help please? ** - 2014-01-14 11:09:38,623 [main] ERROR (support.context.ContextFactory) Error getting connection to database jdbc:oracle:thin at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70) ** *input { file { path => “/root/test2.log” start_position => “beginning” codec => multiline { pattern => "^ - %{TIMESTAMP_ISO8601} " negate => true what => “previous” } } }

filter { grok { match => [ “message”, " -%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:time} [%{WORD:main}] %{LOGLEVEL:loglevel}%{SPACE}%{SPACE}(%{JAVACLASS:class}) %{GREEDYDATA:mydata} %{JAVASTACKTRACEPART}"] } date { match => [ “timestamp” , “dd/MMM/yyyy:HH:mm:ss Z” ] } }

output { elasticsearch { host => “194.3.227.23” }} *

You should change sudo vi /etc/logstash-forwarder to sudo vi /etc/logstash-forwarder.conf to synchronize between part 1 and 2 :-)

You should update this command: sudo service logstash-forwarder force-reload Which gives this output: Usage: {start|force-start|stop|force-start|force-stop|status|restart}

So it should say: sudo service logstash-forwarder restart

Does this logstash pattern work for the Nginx error log, too?

the logstash pattern defined in logstash server, NGUSERNAME [a-zA-Z.@-+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} [%{HTTPDATE:timestamp}] “%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}” %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:“(?:%{URI:referrer}|-)”|%{QS:referrer}) %{QS:agent}

those names, for example, NGUSERNAME, NGUSER is a, whatever name defined by user ? What I mean is I could use NGINXUSERNAME, NGINXUSER

HI I have the same question, when I added Nginx stuff as tutorial says, I got following error and logstash stopped:

{:timestamp=>“2015-04-19T14:49:27.140000+0430”, :message=>“SIGTERM received. Shutting down the pipeline.”, :level=>:warn} {:timestamp=>“2015-04-19T14:49:27.144000+0430”, :message=>“Exception in lumberjack input”, :exception=>#<LogStash::ShutdownSignal: LogStash::ShutdownSignal>, :level=>:error} {:timestamp=>“2015-04-19T14:49:39.038000+0430”, :message=>“The error reported is: \n pattern %{NGUSERNAME} not defined”} {:timestamp=>“2015-04-19T15:15:40.602000+0430”, :message=>“The error reported is: \n pattern %{NGUSER:ident} not defined”} {:timestamp=>“2015-04-19T15:20:08.404000+0430”, :message=>“SIGTERM received. Shutting down the pipeline.”, :level=>:warn} {:timestamp=>“2015-04-19T15:20:08.408000+0430”, :message=>“Exception in lumberjack input”, :exception=>#<LogStash::ShutdownSignal: LogStash::ShutdownSignal>, :level=>:error}

what is the soloution? thanks

Hi Mitchell,

i have a question about how logstash works. What i can’t figure out is when logstash gather a log, and if i am able to gather logs from “simple” files. If i have a simple file to write logs in it, can i gather these logs? For example :

1. mkdir /var/log/mytest.log
2. “This is a test log” > mytest.log

The question is can logstash gather that log?

Thanx.

I have below multiline logs and forwarding them using Logstash .

Here is the filter which I have written ,

filter { if [path] =~ “xx-xx-xx-xx” { multiline { pattern => “^%{TIMESTAMP_ISO8601:timestamp}” negate => true what => “previous” }

However , it is only forwarding the first line and ignoring the rest . Please suggest what kind of filter should I use.

2015-03-26 00:01:06.961 INFO [[ACTIVE] ExecuteThread: ‘34’ for queue: ‘weblogic.kernel.Default (self-tuning)’] [com.xx.xx.xx.xx.xx.xxManager] (xxManager.java:50) - Error Number = 0 Error Category = null Exception = null Transaction Id = xx Timing => Start = 2015-03-26 00:01:06.959 1427299266959 Stop = 2015-03-26 00:01:06.959 1427299266959 Delta = 0ms (0:0:0.000) Request Type = com.xx.xx.xx.xx.xx.xxManager.xxRequestInfo Request => Ipay88RequestInfo = com.xx.xx.xx.xx.xx.xx.mio.value.PaymentRequest=>{MerchantCode=xx&PaymentId=2&RefNo=xx7&Amount=xx&Currency=xx&ProdDesc=xx&UserName=xx&UserEmail=xx&UserContact=xx&Remark=&Lang=UTF-8&Signature=xx&ResponseURL=https://xx/storefront/xx&BackendURL=https://xx:443/xx/xx} Response = null Tracking Id = null Tracking From = null Admin Login Id = null User Login Id = null Module = null Sub Module = null Function = null Sub Results = null Attributes = null Messages = empty

Regards , Anuj

Manicas - any thought to doing a tutorial on how to add Nginx LDAP to our fancy new ELK stacks we’ve all built? There seems to be a lot of demand for this, but most people on the internet seem to be intentionally vague when writing how to’s… you seem to be the exception. Help a brother out! :)

On your Logstash server, create a new filter configuration file called 11-nginx.conf:

What is the reasoning behind the naming of these files? Can you pick a random number?

hi,

I just want to send standard cisco router syslog messages to logstash. How do I do that?

Hi,

You are using Logstash Forwarder on port 5000 to receive your logs -> Isn’t it just Logstash listening on port 5000?

Oh, I would forget to share with you this amazing apps:

• grokdebug
• grokconstructor

Hello, I have used your article to configure ELK, It was really nice,

I’m having a issue in configuring magento exception log which has multi-line exception (with stack trace) , What I did was, I created new .conf file in logstash server with following, after that it does not shipping the exceptions to elastic search, when I commenting the Input section It start to send with _grokparsefailure.

input {
    file {
        type => "staging-all-lincraft-logs"
        path => "/home/deploy/lindcraft/current/codepool/var/log/exception.log"
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601}"
            negate => true
            what => previous
        }
    }
}


filter {
    if [type] == "staging-all-lincraft-logs" {
        mutate {
          gsub => ["message", "\n", " "]
        }

        grok {
            match => [ "message", "%{TIMESTAMP_ISO8601:date} %{GREEDYDATA:exception} Stack trace: %{GREEDYDATA:stack_trace}" ]
        }
    }
}

And files 01-lumberjack-input.conf and 30-lumberjack-output.conf remain without change.

in addition to that I have added following to the forwarder config in the app server as follows

  "files": [

    {
      "paths": [
        "/home/deploy/lindcraft/current/codepool/var/log/*.log"
       ],
      "fields": { "type": "staging-all-lincraft-logs"


	  }
    }

(in forwarder I’m sending all and in the logstach i’m catching only the exception.log file)

HI, i am trying to get openvpn.log, anyone could teach me for the filter and pattern?