-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Fine Tuning PubSub Receive Performance
The PubSub client is a gRPC-based client with knobs for fine-tuning performance. We set reasonable defaults to these knobs, but we encourage users to adjust per their needs. This document describes some of these knobs and gives tips on setting them. This document is limited in scope to reception of messages and does not address publication.
A few implementation details must be described to fully understand the settings:
As the PubSub client receives messages from the PubSub server, it puts them into a local buffer. The client hands messages from the buffer to the user by way of the callback function passed in to Receive. The user must Ack or Nack a message in this function. Each invocation by the client of the passed-in callback occurs in a goroutine; that is, messages are processed concurrently.
The buffer holds a maximum of MaxOutstandingMessages messages or MaxOutstandingBytes bytes, and the client stops requesting more messages from the server whenever the buffer is full. Messages in the buffer have an ack deadline; that is, the server keeps a deadline for each outstanding messages. When that deadline expires, the server considers the message lost and re-sends the message. Each message in the buffer automatically has their deadline periodically extended. If a message is held beyond its deadline - for example, if your program hangs - the message will be redelivered.
This is the maximum amount of time that the client will extend a message's deadline. This value should be set as high as messages are expected to be processed, plus some buffer. It's fairly safe to set it quite high; the only downside is that it will take longer to recover from hanging programs. The higher the extension allowed, the longer it takes before the server considers messages lost and re-sends them to some other, healthy instance of your application.
This is the maximum amount of messages that are allowed to be processed by the callback function at a time. Once this limit is reached, the client waits for messages to be acked or nacked by the callback before requesting more messages from the server.
Note that there may be more messages stored locally than MaxOutstandingMessages, which we'll describe as "message overflow". This is due to the fact that the client side does not have fine-grained control over how many messages the server sends in a single RPC. For example, if the client has room for 3 messages, but calls server.receive and gets 50 messages, there's not much it can do but hold on to the extra 47 until other messages get acked/nacked.
This value is set by default to a fairly conservatively low number. We strongly encourage setting this number as high as memory allows, since a low setting will artificially rate limit reception. Setting this value to -1 causes it to be unbounded.
This is the maximum amount of bytes (message size) that are allowed to be processed by the callback function at a time. Once this limit is reached, the client waits for messages to be acked or nacked by the callback before requesting more messages from the server.
Note that the same message overflow can occur with MaxOutstandingBytes as with MaxOutstandingMessages.
The same advice as with MaxOutstandingMessages applies. Setting this value to -1 causes it to be unbounded.
This is the number of goroutines that the client will spin out to receive messages from the Pubsub server. This setting affects the rate of message intake from server to local buffer.
We generally encourage higher NumGoroutines values as CPU availability allows, since a low NumGoroutines value may cause a bottleneck to reception. However, an overly high value may negatively impact systems. Each goroutine has a chance to cause message overflow - more messages to be received than can be processed by the caller. These overflown messages sit idle until a message-being-processed is acked, freeing up space for an idle/overflown message to be processed. This is unfortunate: less overflown messages is better for the system. Had those messages not been pulled by the busy client, and instead been pulled by another (less busy) client, they might have been processed faster.
Generally, each application should use a single PubSub client instead of creating many.