How To Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana)

Things to note, if you’re using ELK 5.0 and encounter this error:

[ERROR][logstash.filters.geoip ] The GeoLite2 MMDB database provided is invalid or corrupted. {:exception=>com.maxmind.db.InvalidDatabaseException: Could not find a MaxMind DB metadata marker in this file (GeoLiteCity.dat). Is this a valid MaxMind DB file?

It’s because you downloaded the legacy version, download the GeoLite2 City at: curl -O “http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz”

Followed every part of the tutorial ,

Getting the Error ,

No Compatible Fields: The “[filebeat-]YYYY.MM.DD” index pattern does not contain any of the following field types: geo_point

have searched Google and unable to find the root cause of this error , can you help

Any pointers on how to do this for fluentd (using logstash format) instead of logstash?

Hi,

Thanks for the nice tutorial.

Just to mention, you may need to change the mappings for geoip.location field and set it to ‘geo_point’ or else you won’t be able to get the Field ‘geiop.location’ displayed under Geo Co-ordinates > Field when you try to create a MAP.

I was using an index called ‘apache’ so initially the filed was set to double

$ curl http://localhost:9200/apache*/_mapping/apache-access/field/geoippretty
{
  "apache" : {
    "mappings" : {
      "apache-access" : {
        "geoip.location" : {
          "full_name" : "geoip.location",
          "mapping":{"location":{"type":"double"}}
        }
      }
    }
  }
}

I had to change it to ‘geo-point’ to get the filed on the map.

$curl http://localhost:9200/apache*/_mapping/apache-access/field/geoip.location?pretty
{
  "apache" : {
    "mappings" : {
      "apache-access" : {
        "geoip.location" : {
          "full_name" : "geoip.location",
          "mapping":{"location":{"type":"geo_point"}}
        }
      }
    }
  }
}

To change the mappings:

1. copy the default lib/logstash/outputs/elasticsearch/elasticsearch-template.json to something like lib/logstash/outputs/elasticsearch/elasticsearch-apache-template.json
2. Edit the elasticsearch-apache-template.json and give it a name “template” : “apache”,
3. You will require to delete your existing index (THIS WILL LOOSE DATA). curl -XDELETE http://localhost:9200/apache*
4. Edit your logstash.conf file to include the template in the output

output {
  elasticsearch {
     host => localhost
     cluster => es_24
     index   => "apache"
     template => "path_to_elasticsearch-apache-template.json"
     template_name => "apache"
  }
  stdout { codec => rubydebug }
}

5. Restart ES and LS and it should work.

This is 2017. ELK stack is known as Elastic Stack now.

Other than GeoIP, we can also use the IP2Location filter in Logstash.

https://www.ip2location.com/tutorials/how-to-use-ip2location-filter-plugin-with-elastic-stack

Great article. Any examples of doing this same thing, but instead coming from an IIS weblog instead of Nginx?

Is there a reason to use this config?

geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }

I believe that the below config can do the same, the “location” field contains longitude and latitude and can be used in Kibana tile map.

geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      fields => ["country_name", "real_region_name", "city_name", "location"]   # this is to output only certain fields
    }

Conflict on geoip.location and the floowing error:

No Compatible Fields: The "filebeat-*" index pattern does not contain any of the following field types: geo_point

Strange how I have 20 tabs open here to try and figure this one out. It looks like a common problem with no clear solution for the ELk stack beginner. I came here cause I wanted solutions not more problems…

I followed the tutorial and was getting this error:

No Compatible Fields: The "filebeat-*" index pattern does not contain any of the following field types: geo_point

This is how I solved it.

My server is CentOS 7 with these versions of relevant packages: elasticsearch-2.3.5-1.noarch filebeat-1.2.3-1.x86_64 kibana-4.5.4-1.x86_64 logstash-2.3.4-1.noarch

I use filebeat as the collector on all my nodes and my index pattern is ‘*filebeat-**’

Contents of: 30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

My Apache filter file is like the nginx one in the tutorial above.

I continued attempting to resolve the error by reading the instructions here: (Note: I’m not at all certain that the steps at these two links are required. I implemented them while trying to resolve the issue and did not try the solution without these steps. I have a suspicion that they are nor necessary) https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html Which pointed me to apply the changes here: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html#geo-percolate and then back to here where I applied the first option: https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

But I still got the same error.

Eventually, I saw ckyconsultinguk’s comment above from April 22, 2015 and adapted his solution a bit.

Based on his suggestions I edited the file: elasticsearch-template.json (Mine was located here: ‘/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch’ ) and changed the value of “template” from “logstash-" to "filebeat-”:

2c2
<   "template" : "logstash-*",
---
>   "template" : "filebeat-*",

(This saved me the step of having to add a template entry in the output file and since I do not use the ‘logstash-*’ index I do not need the original template.)

I then: Stopped logstash Deleted my old index DELETE /filebeat-* (THIS WILL DELETE ALL OF YOUR ‘filebeat-*’ DATA) Restarted ES Started logstash and went to sleep happy. :-)

I also found a similar implementation of this idea here: https://michael.lustfield.net/misc/geo-point-with-elasticsearch-2x

What a neat tutorial … hats off to author and team … keep up good work !! SF From SW England ! P.S. Is there any place i can see a example of application log which use log4j logs … btway that’s what i m going to do next …