LS2 support + safe stop? (shutdown refactor)

[teebu]

Thanks to @mantika I was able to install dynamodb plugin to LS2, but when running a test, I can't seem to get it to stop (ctrl+c) it just keeps going and throws {"inflight_count"=>0 ... } warnings. It does stop in LS1.5.

Also, sudo service logstash stop it never wants to stop. I have to kill -9 the pid.

I belive this maybe related to new plugin API, some ref to a request here logstash-plugins/logstash-input-dynamodb#3

elastic/logstash#3963

Looking in the logs:

{:timestamp=>"2016-08-18T22:54:55.141000+0000", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2016-08-18T22:59:29.088000+0000", :message=>#<LogStash::PipelineReporter::Snapshot:0x3cc4fe9f @data={:events_filtered=>6604, :events_consumed=>6604, :worker_count=>1, :inflight_count=>75, :worker_states=>[{:status=>"run", :alive=>true, :index=>0, :inflight_count=>75}],

{:timestamp=>"2016-08-18T23:01:37.979000+0000", :message=>"The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.", :level=>:error}

And then it just continues. It may never end because it's always trying to fill the buffer?