Use with Coral TPU Accelerator?

[midzelis]

Hi - I recently purchased a Coral TPU for the Frigate NVR (https://docs.frigate.video/configuration/detectors/) - but now I'm thinking maybe I could use this for WIS too. It seems like it should have the same performance as a 1080 CUDA (https://blog.raccoons.be/coral-tpu-jetson-nano-performance) and it was only $60 (if you can find it)

I recently bought the ESP s3 box, and I'm planning to get started with using willow sometime soon. Has anyone tried using the Coral TPU with WIS? Is there anything special that needs to be done to set it up? Just thought I'd ask before diving in myself.

Thanks.

[kristiankielhofner]

The Coral TPU supports a maximum model size of 6MB. This works well for very, very small TFlite models where you're doing things such as object detection/classification across objects measured in the hundreds (like with Frigate).

Models to understand the full spectrum of human speech are significantly more complex and thus significantly larger. The smallest speech recognition model we use is 75MB (int8), and thus well beyond what a Coral is capable of.

The models that start producing what we consider to be acceptable quality are measured in the hundreds of megabytes, or in the case of large-v2 nearly a gigabyte (all numbers int8).

In short, it is fundamentally impossible to do broad range free flow speech recognition on the Coral and thus it will not be supported.

[midzelis]

Thanks for the reply - I didn't know that about the Coral - good to know. I do have an old GTX1070 which I'll pop into the server rack for this.

I'm also curious. You don't have any benchmarks showing CPU. Do you know how much performance you can get out of a late-model Ryzen or Intel CPU instead of the GPU? My CPU isn't doing much most of the time, and I'd rather not have to pay the power cost of idling a GPU if the CPU could work. I'll have to measure the watts consumed by a idling GPU to see if it matters.

[kristiankielhofner]

A GTX 1070 is currently the best "bang for the buck" card we suggest people buy on the used market if they want to utilize CUDA at low cost - I use one daily in my home.

We don't have any CPU benchmarks yet but you can look at Whisper benchmarks for ctranslate2/faster-whisper which should be very close to what you would see with WIS. Or you can of course benchmark on your own hardware. You will be amazed at how much better a six year old GPU does vs CPU.

A GTX 1070 idles at around 8 watts (in my case). When performing inference on typical speech segments you will see it "spike" to roughly 70 watts or so for roughly ~500ms. It will then scale back down.