Huge Performance Drop observed with large payloads of around 300KB or more #702
Replies: 2 comments 8 replies
-
|
Hey! On what machines the cluster is running? |
Beta Was this translation helpful? Give feedback.
-
|
@Jarema pls find details in original edited thread. |
Beta Was this translation helpful? Give feedback.
-
|
this might be an issue. what do you see if you remove - _ = Task.Run(async () => await ProcessMessageAsync(natsConsumerOptions,consumerOptions,onMessageReceived,jsMsg));;
+ await ProcessMessageAsync(natsConsumerOptions,consumerOptions,onMessageReceived,jsMsg); |
Beta Was this translation helpful? Give feedback.
-
|
@mtmk rps is requests per second, commonly also tps i believe😅. Could you pls elaborate "Using multiple connections with multiple consumers might help without going crazy about it". I am currently using Connection Singleton instance. |
Beta Was this translation helpful? Give feedback.
-
|
edit: as for multiple consumers if processing of each message is taking too long, you can 'load balance' them using multiple consumers rather than a single consumer and spin a task. edit2: I mean multiple subscriptions to a single consumer from multiple application instances e.g. if you're doing autoscaling thanks @Jarema for checking this. please feel free to add any other comments you might have |
Beta Was this translation helpful? Give feedback.
-
|
(@YashasAnand just made some edits above, in case you missed them) |
Beta Was this translation helpful? Give feedback.
-
|
@mtmk as per your edit2, that is exactly what ive done, if you missed it from my edited oringinal post "We are using 75 Consumers totally of JobTestTopic type per pod & we have 3 pods of consumer service (totally 225 consumers) , 40 of these topics are unique durable consumers example (JobTestTopic1, JobTestTopic2 .. JobTestTopic39) . other 35 consumers are clients bounded to same consumers & jetstream is distributing it in round-robin fashion to different consumer threads." beyond 3 nodes we dont want to go as with 3 nodes & 1 producer consumer service pod on each node, the consumerCPU's are on higher side (70+ percent on all 3 nodes). Also one thing im not understanding is "[7] 2025/01/09 06:13:54.806113 [INF] 10.35.0.1:6222 - rid:10 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 2264 chunks of 117746424 total bytes." why does this not tell me which consumer is slow? Also is slow consumer applicable to jetstream as far as i read from blogs its a coreNATS concept only, How can i pinpoint which consumer is slow? |
Beta Was this translation helpful? Give feedback.
-
|
you need to find what works best for you. my main comment would be don't just spin tasks without tracking them in the consume loop. btw, in case you didn't know we also have a dedicated JetStream channel: https://slack.nats.io #jetstream |
Beta Was this translation helpful? Give feedback.
-
I am doing a benchmark at around 300rps with a large payload of around 300KB in k8s nats using helm. My client code is written in c#.
This is my producer code:
NatsMessageModel.cs
Consumer Code:
The payload i am sending is around 300KB, same if i send small payloads around 1KB or 10KB we see very fast consumption & produc rates. can the above code be optimized or any thing specific we can do regarding this? Also ive observed stream size grows exponentially in this case,
The CPU & memory on nodes are going very high around 90% on each nats node given a 3 node nats jetstream cluster with current Replica set to 1 ( temporarily for testsing ,later will increase replicas).
Nats was installed using helm chart as suggested in the cmminity github https://github.com/nats-io/k8s
Edit 1:
@Jarema pls find metrics below:
K8s cluster consisting of 3 node nats deployed through official helm chart.
-----------------------------------------------------TEST Case 1---------------------------------------------------------------------------------
Test Parameters:
PayloadSize: 1KB
RPS: 1.5k
Test Duration : 10m
Stream Replicas: 3 (file storage)
Node Config: (2 Core 4GB Ram with 1:1 cpu to vcpu ratio & 20GB Storage)
Consumer Info:
We are using 75 Consumers totally of JobTestTopic type per pod & we have 3 pods of consumer service (totally 225 consumers) , 40 of these topics are unique durable consumers example (JobTestTopic1, JobTestTopic2 .. JobTestTopic39) . other 35 consumers are clients bounded to same consumers & jetstream is distributing it in round-robin fashion to different consumer threads.
Messags produced are round-robined to each consumer using custom algorithm to distribute load to all consumers.
Observations
CPU of 3 NATS Nodes: are around 90%
Service (Producer & consumer) CPUs are around 60 to 70%
NOTE: we are able to rate of produce & rate of consume is almost equal, due to the fact that processing & acking is happening on a different thread in consumer as shown above.
Here is the stream & consumer info
These are consumer info of 2 consumers
-----------------------------------------------------TEST Case 2---------------------------------------------------------------------------------
Exact same test as above but freshly created stream, with payload size of 270KB at around 300rps
Test Parameters:
PayloadSize: 270KB
RPS: 300rps
Test Duration : 10m
Stream Replicas: 3 (file storage)
Node Config: (2 Core 4GB Ram with 1:1 cpu to vcpu ratio & 20GB Storage)
Consumer Info:
We are using 75 Consumers totally of JobTestTopic type per pod & we have 3 pods of consumer service (totally 225 consumers) , 40 of these topics are unique durable consumers example (JobTestTopic1, JobTestTopic2 .. JobTestTopic39) . other 35 consumers are clients bounded to same consumers & jetstream is distributing it in round-robin fashion to different consumer threads.
Messags produced are round-robined to each consumer using custom algorithm to distribute load to all consumers.
Observations
CPU of 3 NATS Nodes: are around 85 to 100%
Service (Producer & consumer) CPUs are around 35 to 50%
Observed pod Restarts of NATS, Also produce errors in producer service (logs attached later section)
Able to reach only 50RPS
Stream config
Consumer Info:
Before Pod Restart These Logs are observed:
Producer Service Exceptions: Also i feel for some reason this No response received from the server is happeing due to overwhelming the nats server because i am able to see consumers still consuming slowly. Need your advice on this...
Beta Was this translation helpful? Give feedback.
All reactions