elastic/logstash

Logstash 1.5.0, Unable to remove field with nil value if it comes from redis input

Open

#3,310 opened on May 25, 2015

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Ruby (3,496 forks)batch import
bughelp wanted

Repository metrics

Stars
 (14,197 stars)
PR merge metrics
 (Avg merge 4d 15h) (51 merged PRs in 30d)

Description

Running logstash 1.5.0, I have a file with csv data that I read into logstash and parse using the csv plugin. Some fields will always be empty and gets a value of null. I then do some checking and remove these fields before shipping to elasticsearch. This all works fine, unless I first ship the csv data to redis, and then parse it, then I'm unable to remove the fields with value null.

Here are two small config files that I can use to recreate this issue:

redis-send.conf

input {
    stdin {}
}

output {
    redis {
        data_type => "list"
        key => "logs"
    }
}

redis-receive.conf

input {
    stdin {}
    redis {
        data_type => "list"
        key => "logs"
    }
}

filter {
    csv {
        columns => ["a", "b", "c", "d", "e"]
    }
    mutate {
        remove_field => "a"
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}

On the sender, I'm typing the string "1,2,3,4,5" to stdin. On the receiver agent, I get this output:


{
       "message" => [
        [0] "1,2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:12:44.173Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

Just like I expect.

However, leaving the first field empty:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:17:54.859Z",
          "host" => "starscream",
             "a" => nil,
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

It does not remove the field "a".

But if I type the same string in stdin on the receiver agent, it removes the field:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:18:24.907Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

I've tried the above in logstash 1.4.2 as well but didn't encounter the issue in that version.

Contributor guide