How to run Semantic Kernel in Production

[timlaverty]

@dluc @craigomatic

Lots of discussion about this amongst the team so far but we'd like community input on it. My mental model is folks would run SK inside an Azure service (either ACI, AKS, or other) and put keys in keyvault. The client app (typescript, wpf, etc.) would talk to the cloud service when making its AI calls. If everything has to be in a thick client app, I still picture pulling keys from a cloud vault and never storing them on client.

I think we need a write a paper on, “How to run Semantic Kernel in Production” in our repo. I'd love input from folks as they build apps and deploy them - that practical experience would be good influence.

-Tim

[dluc]

While we brainstorm the topic and find the time to share it, here are some high level notes:

• Bootstrapping a kernel instance is cheap, and we assume that the typical use case will be creating a new instance whenever there's a need to: create a plan, execute a plan, access memory, create and execute a semantic function on the fly. Even during the execution of a plan, is reasonable to create a new instance step by step.
• The kernel is thread safe, so it can support the execution of multiple plans in parallel and isolation. This should be properly verified, but that's the underlying intention.
• The kernel does not support concurrent users though. The kernel state (e.g. the Memory instance), currently, is user specific. This can be addressed with custom memory implementations, or limiting the memory interaction to native skills, but it's not recommended, and error-prone. For user specific Memory, we should expand the context object first, e.g. allow contextual memory. This is not in the current roadmap, hence the recommendation to not use singleton kernel instances, and to discard kernel instances in multi-user environments/services.
• Native skills should be designed to be thread safe and support multiple users. The kernel cannot verify how skills behave though, and the current kernel holds skill instances active, for performance reasons. Instantiating skills on demand, e.g using a factory, could help, but this is not in the roadmap, and potentially not a responsibility of the kernel. SK Core skills are designed to support concurrent users, but third party skills might not, so the right approach depends on which skills are used and their internal design.

[timlaverty]

Running SK in Azure hosted by an Azure Function: https://devblogs.microsoft.com/semantic-kernel/how-to-deploy-semantic-kernel-to-azure-in-minutes/. Note this means any SK Skills you use need to be stateless. All the Skills "inbox" are, but any you add would need to be stateless too.

An option for stateful hosting would be ASP.NET.

[AceHack]

Can you expand on what it means that the memory instance is user specifics? What is the memory instance? What is it used for? Thanks.