RSS Feed

Making Peace with Logstash Part 4 – Manipulating Dates


April 9, 2018 by Mike Hillwig

I’ve been slacking on my blogging again. The last few weeks have been busy, including a visit from Elastic Consulting. But it’s back to the flight data.

So far, I’ve done a decent job getting the data into shape. My biggest challenge, though, was the dates and times. Dates are in one field, and the times are in another. Dates look like 2014-02-26 and times look like 0852 Using a traditional datetime datatype would be nice to have, so I’ll have to do it myself. In order to turn a date and time into a datetime, I need to abut the two fields and then convert it.

I accomplished this by using a mutate filter, employing by several add_field commands. Notice how I simply abut the two times.

[code] mutate{
add_field => {“ts” => “%{FlightDate}” }
add_field => {“[@metadata][indexname]” => “flights-%{+YYYY}” }
add_field => {“[CRSDepTimeDT]” => “%{FlightDate} %{CRSDepTime}”}
add_field => {“[CRSArrTimeDT]” => “%{FlightDate} %{CRSArrTime}”}
add_field => {“[DepTimeDT]” => “%{FlightDate} %{DepTime}”}
add_field => {“[ArrTimeDT]” => “%{FlightDate} %{ArrTime}”}
add_field => {“[WheelsoffDT]” => “%{FlightDate} %{WheelsOff}”}
add_field => {“[WheelsOnDT]” => “%{FlightDate} %{WheelsOn}”}[/code]

After that, it was as easy as running this through a date filter.

[code] date {
match => [ “CRSArrTimeDT” , “yyyy-MM-dd HHmm” ]
target => “CRSArrTimeDT”
locale => “en-US”
date {
match => [ “DepTimeDT” , “yyyy-MM-dd HHmm” ]
target => “DepTimeDT”
locale => “en-US”

When executed, it looks like this:

As I looked at the dates, something was off, and I just couldn’t figure out what it was. When the date filter runs, it converts the timezome to UTC, and I’m in EST. Surely there has to be a way to get this right, too. Since I have the city name and airport code, there should be a way to convert that to a timezone. That’s a project for another day. The other thing I need to do is increment the date when a flight lands after midnight.

If you’re following along at home, this configuration file is in my GitHub repository.

Next, we’re going to start enriching the data so we can look at aircraft types.


  1. […] Mike Hillwig continues his Logstash series: […]

  2. LeonardLib says:


  3. Patricktoona says:

    Selling Aged Twitter Accounts for Cheap 2008-2015

    Hello! I will sell Aged Twitter Accounts for Cheap 2008-2015

    Accounts Prices:
    2008 – 10$ Per Account
    2009 – 9$ Per Account
    2010 – 8$ Per Account
    2011 – 7$ Per Account
    2012 – 6$ Per Account
    2013 – 5$ Per Account
    2014 – 4$ Per Account
    2015 – 3$ Per Account

    Buy for testing accept!
    Discounts available if you buy in bulk

    with Format
    Username:Password Twitter:Live Emai:Password Email,
    joindate,tweets,following,followers and likes.

    Accounts Info:
    Gender can either be female or male.
    Accounts can be either empty with photos or without photos, or with already added entries, for example: tweets, retweets, likes, comments and followers.

    Where you can contact me,PM me here!

    Skype & Telegram : congmmo
    ICQ : @652720497
    Email : congmmo@gmail . com

    Payments Accepted:
    BTC,ETH,ETC,LTC or Payoneer or Paypal
    Paypal fees will be paid by the buyer
    No refunds allowed, if you’re having issues with an account you’ve bought I will gladly replace within the first hour of purchase!
    Thank you!

  4. maniiyao says:

    Bestellen Sie SEO-Suchmaschinenoptimierung der Website, Bestellen Sie Website-Promotion-Services Bei allen Fragen wenden Sie sich an den Benutzernamen Skype pokras7777
    Wir sammeln auch Basen

  5. DennisRarge says: смотреть кино онлайн, смотреть онлайн фильмы, онлайн кинотеатр

  6. Utterly written subject material, regards for selective information. “In the fight between you and the world, back the world.” by Frank Zappa.

  7. I and my friends have been analyzing the best secrets and techniques located on your web blog and then I had a horrible suspicion I had not thanked the site owner for those techniques. Those young boys are actually for that reason excited to see all of them and have now really been taking pleasure in these things. We appreciate you turning out to be simply helpful as well as for getting this sort of fine tips most people are really needing to be informed on. Our honest regret for not expressing gratitude to sooner.

  8. maniiyao says:

    Bestellen Sie SEO-Suchmaschinenoptimierung der Website, Bestellen Sie Website-Promotion-Services Bei allen Fragen wenden Sie sich an den Benutzernamen Skype pokras7777
    Wir sammeln auch Basen

  9. LarryNah says:

    Writers For each Hour – trustable Personalized crafting support which offers college students a group for the pursuing: essay writers, IB prolonged and ToK papers, examine paper writers.

  10. Shannonexone says:

    Вы можете зайти страничку специалистов в журналистике.

    Команда Всевышний PR – это информационная сила проверенная временем.

    Посетите сами как все работает, вот ютюб ролик

    Вы можете заказать пиар поддержку. Должное качество сервиса. Адекватная ценовая политика. Конкретная отчетность о каждом действии и шаге.

    Бог спасет – пиар поддержит!

  11. JerryAgink says:

    Как дела? Вот есть ссылка про русского бизнесмена Игоря Сосина

    Заходи все здесь

    Игорь Владимирович Сосин открыл сети уникальных VR клубов Unreal place. Ребят это вондерфуль знание!

    Российский бизнес самый мощный

  12. JamesFob says:

    Доброе утро коллеги!

    Тема есть позитивная типография в Москве. Печатаем легко на всем даже на стекле и сувенирах. Доставка бесплатная по Москве. Работаем без праздников все четко!

    Вы можете заказать самоклейки и с бесплатной доставкой!

    Для заказа Дмитровская типография

  13. Domenicbop says:

    Don’t miss the opportunity to win the best prizes!

  14. Most of whatever you claim is astonishingly appropriate and it makes me wonder the reason why I had not looked at this in this light before. Your article really did turn the light on for me as far as this particular subject matter goes. Nonetheless there is actually 1 position I am not necessarily too comfortable with so whilst I attempt to reconcile that with the actual main idea of your position, let me observe what the rest of the visitors have to point out.Well done.

  15. WendellFub says:

    Improve your time on site

  16. JamesFob says:

    Здаровеньки булы!

    Тема в наличии экстрасовременная типография в Москве. Печатаем на всем. Доставка есть по Москве. Работаем без выходных все четко!

    Вы можете заказать срочную типографскую продукцию и с бесплатной доставкой!

    Для заказа типография партизанская

  17. Like!! I blog frequently and I really thank you for your content. The article has truly peaked my interest.

  18. Thank you! This article really helped. Think of any further advice I might need? Website Design

  19. my sources says:

    Greetings! Very helpful advice on this article! It is the little changes that make the biggest changes. Thanks a lot for sharing!

  20. find more says:

    I am really loving the theme/design of your weblog. Do you ever run into any internet browser compatibility problems? A few of my blog visitors have complained about my site not working correctly in Explorer but looks great in Safari. Do you have any recommendations to help fix this problem?

  21. Muchos Gracias for your article post. Much obliged.

Leave a Reply

Your email address will not be published. Required fields are marked *