RSS Feed

Making Peace with Logstash Part 1 – Input and Output


February 21, 2018 by Mike Hillwig

Logstash is an incredibly powerful tool. If you can put data into a text file, Logstash can parse it. It works well with a lot of data, but I’m finding myself using it more to use it for event data. When I say event data, if it triggers a log event and it writes to a log, it’s an event. For the purposes of my demos, I’m using data from the Bureau of Transportation Statistics. They track flight performance data, which works perfectly for my uses. It’s a great example dataset without using anything related to my real job.

Logstash configuration files typically have three sections, INPUT, FILTER, and OUTPUT. However, FILTER is optional.

In this case, my configuration looks like this:

[code]input {

file {

path => [“/Users/mikehillwig/elastic/flights/*.csv”]

sincedb_path => “/dev/null”

start_position => “beginning”



output {

stdout {}


This is pretty bare bones for a Logstash config. I did put in a wildcard to the file path. The one thing I added that’s unusual is the sincedb_path option. Logstash is smart enough to avoid reprocessing a file twice and it knows where it processed a file during an exicution. In this case, I don’t want Logstash to remember this, so I forced the sincedb_path to /dev/null.
Note that I’m sending the output to stdout. That allows me to see the data passing through my pipeline without having to look for it in Elasticsearch. We’ll put this into Elasticsearch in a future post.
The output of this config looks something like this:


Some of this data makes sense, but some of it is just noise, Next time, I’ll show you how to parse this as a CSV file, then we’ll eliminate some noise.


  1. […] Mike Hillwig gets us started on Logstash: […]

  2. […] my last post, I went over the basics of importing data from the US government about flight performance data […]

  3. I am often to blogging and i really appreciate your content. The article has really peaks my interest. I am going to bookmark your site and keep checking for new information.

  4. You are my breathing in, I own few web logs and often run out from to brand : (.

  5. I love what you guys are up too. This kind of clever work and exposure! Keep up the wonderful works guys I’ve you guys to our blogroll.

  6. so much great info on here, : D.

  7. Like!! Really appreciate you sharing this blog post.Really thank you! Keep writing.

  8. I’m still learning from you, but I’m trying to achieve my goals. I absolutely love reading all that is posted on your website.Keep the information coming. I enjoyed it!

  9. Oh my goodness! an amazing article dude. Thanks However I’m experiencing challenge with ur rss . Don’t know why Unable to subscribe to it. Is there anyone getting similar rss drawback? Anyone who is aware of kindly respond. Thnkx

  10. wow, awesome blog article.Much thanks again. Want more.

Leave a Reply

Your email address will not be published. Required fields are marked *