Listener dies and fails to restart causing massive loss of logs

[c10l]

After the GELF input crashes it enters a loop where it restarts and dies again. It looks to me like the thread that died first failed to release the socket so any subsequent attempts to bind to that same socket cause a new crash:

{:timestamp=>"2015-12-07T10:10:53.466000+0000", :message=>"gelf listener died", :exception=>#<LogStash::Json::ParserError: Invalid UTF-
8 start byte 0xa3
 at [Source: [B@5b6b2d63; line: 1, column: 1088]>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/li
b/logstash/json.rb:41:in `jruby_load'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/json.rb:38:in `jruby_load'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:99:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:63:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:180:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:174:in `start_input'"], :level=>:warn}
{:timestamp=>"2015-12-07T10:10:58.467000+0000", :message=>"Starting gelf listener", :address=>"0.0.0.0:12201", :level=>:info}
{:timestamp=>"2015-12-07T10:10:58.468000+0000", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:84:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:63:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:180:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:174:in `start_input'"], :level=>:warn}
{:timestamp=>"2015-12-07T10:11:03.469000+0000", :message=>"Starting gelf listener", :address=>"0.0.0.0:12201", :level=>:info}
{:timestamp=>"2015-12-07T10:11:03.470000+0000", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:84:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:63:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:180:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:174:in `start_input'"], :level=>:warn}
...

It just gets stuck on this start-crash-restart loop.

[msimons]

Same issue here. Is someone already into the issue?

{:timestamp=>"2015-12-24T14:17:53.175000+0100", :message=>"gelf listener died", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Exception': was expecting ('true', 'false' or 'null')
 at [Source: [B@2c565816; line: 1, column: 11]>, :backtrace=>["/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/json.rb:41:in `jruby_load'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-$
{:timestamp=>"2015-12-24T14:17:58.179000+0100", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.$
{:timestamp=>"2015-12-24T14:18:03.179000+0100", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.$
{:timestamp=>"2015-12-24T14:18:08.180000+0100", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.$
{

[c10l]

#27 fixes the crash. It does not fix the fact that the listener does not restart after crashing but it will avoid the crash on the JSON parser.

[alex88]

Is the restart something that has to be implemented on logstash side?

[Cactusbone]

PR #51 should avoid listener dying