README-Manage-Model-Versions.md

Manage Model Versions

Assuming we have an idea of what intents (and potentially entities) we'd like to detect, we can look into training a LUIS model using the portal.

We can utilize this Getting Starting guide for creating a LUIS model in the portal.

The main concept here from the LUIS Best Practices Guidance:

Each authoring cycle should be within a new version, cloned from an existing version.

Helpful Links

1. Getting Started With LUIS
2. Tutorial for LUIS with C# Bot Framework
3. LUIS Best Practices Guidance
4. Intents
5. Utterances
6. Entities
7. LUIS Development Lifecycle
8. Active Learning
9. Review Endpoint Utterances
10. LUIS Docker Containers
11. Bot Framework Emulator
12. LUIS App Iteratiion
13. Creating a new version for each cycle
14. Importing and Exporting the LUIS Model
15. Using Publish Slots
16. Bot Framework CLI

Setup LUIS Model

We have included a sample LUIS model, but we'll assume that we're going to start with a brand new one.

Reminder on the Sample

For example, suppose we want to detect some intents like location, get invoice, talk to an agent, or get pay policy.

It would also be helpful to think about representative utterances (e.g. this is what a user will send to a LUIS model through a chatbot) to populate the LUIS model. We can use these utterances when we create the LUIS model

Some Concepts and Design

First, determine some example Utterances, Intents, and even Entities. In this scenario, we'll start simple with some utterances and intents we'd like LUIS to detect.

We should understand a typical LUIS authoring life cycle.

Notice that we can have this cycle of creating a version of an app, adding training examples, training and publishing, and then making sure to review the https endpoint utterances. As we start modifying the LUIS app, it is possible to change these in the portal, but it is also helpful to keep track of prior versions.

From this section on Creating a new version for each cycle:

Each version is a snapshot in time of the LUIS app. Before you make changes to the app, create a new version. It is easier to go back to an older version than to try to remove intents and utterances to a previous state. The version ID consists of characters, digits or '.' and cannot be longer than 10 characters. The initial version (0.1) is the default active version.

Creating a new version

We can either clone the existing model in the portal, or we can use a file representation of the model.

Clone an existing version to use as a starting point for each new version. After you clone a version, the new version becomes the active version.

File versions can be handled through Importing and Exporting the LUIS Model.

Let's try to export our existing LUIS model as a JSON file.

Suppose we have already Reviewed Endpoint Utterances and noticed some utterances from end users that we can now map to some intents.

While we can add the intent directly in the portal (and then export the model), we can also modify a copy of the local file which represents the snapshot of the LUIS model from our export earlier.

We can increment our version:

{
  "luis_schema_version": "4.0.0",
  "versionId": "0.1.1",
  ...
}

For example, we can add an intent to the intents array:

    ...
     "intents": [
    ...
    {
      "text": "hey I want some pizza",
      "intent": "None",
      "entities": []
    },
    ...
    ]
    ...

Or we can even add in entities (importing a LUIS Model version with an Entities Sample).

In this case, we can capture not only intents, but also entities which will help provide additional guidance for the intent.

Now that we have the changes, we can take a few different approaches.

Publish Manually

The simple path is is to just upload the modified file as a new version of the model. This is the import step.

Notice that if we select a new version, then the model will be shown as active.

However, we would still need to publish the LUIS model to a Publish Slot. From the doc:

You can publish to either the stage and/or production slots. Each slot can have a different version or the same version. This is useful for verifying changes before publishing to production, which is available to bots or other LUIS calling apps. Trained versions aren't automatically available at your LUIS app's endpoint. You must publish or republish a version in order for it to be available at your LUIS app endpoint.

Suppose we're using the new version of the model that we have imported.

It's also worth revisiting the test console in the LUIS portal to compare the imported version results with the one that is already published.

Click on compare with published.

Publish Automatically (An Approach)

Going back to the idea that we have files, we could look into CI/CD as well assuming that we have appropriate agent connectivity, testing, and source control.

This would be a much more involved approach, but would be more representative of a production scenario. We'll look at a possible key step that will help this scenario work. This approach could also involve other SDKs or APIs, but of course this depends on familiarity and validating that the underlying calls will work.

As a test, we could use the Bot Framework CLI and test this first locally with the file, but ultimately this could live in a script task that runs on an agent that will deploy built artifacts (e.g. Azure DevOps build / release pipelines)

For example, suppose we update the VersionId in the JSON file as part a change in the model.

Next, we can use a Bot Framework CLI to import the model from a file.

bf luis:version:import --endpoint $location --subscriptionKey $subKey --appId $appId --in $modelFilePath --versionId $versionId

We can run this command to import the model.

We can now check to see the model was imported in the portal as well by looking at the versions for the model.

After importing the model, we'll need to make sure to train the model.

We can see this in two steps - run and show.

bf luis:train:run --endpoint $location --subscriptionKey $subKey --appId $appId --versionId $versionId
bf luis:train:show --endpoint $location --subscriptionKey $subKey --appId $appId --versionId $versionId

We can run training.

We can also show how the training is proceeding.

We can check in the LUIS Portal to see that the model was imported and now trained.

Once satisfied with training, we can also test the model.

This is will allow us to test against the model before publishing. We'll borrow the batch testing file as input.

For convenience, this is wrapped in the sample batch file test script.

Be sure to supply values in the script as appropiate for testing the version of the model that we had just imported.

$location = "westus" #set to LUIS endpoint host
$subscriptionKey = "" #set to LUIS App Key
$appId = "" #set to LUIS App Id
$versionId = "" #set to LUIS Version Id
$batchFilePath = "" #set to file path for the Batch Test File

If we run the script in the location where the host can reach Get-LUISResult, the input batch test file (see this sample batch test file), and we have supplied the appropriate credentials and have network access, we should be able to see results that reflect the queries against the LUIS model.

From an automation standpoint, we can also look into including additional tests in a test suite that we can run on the model in our CI/CD pipelines. This can be something that is revisited by the team and also later be updated based on results seen in live usage for LUIS.

We can then look into publishing the model. We can publish to staging slot as a starting point.

bf luis:application:publish --endpoint $location --subscriptionKey $subKey --versionId $versionId --appId $appId --staging

We can also send some sample queries to LUIS to validate that we're getting expected results.

While we can use luis:application:query, it looks like as of now, batch testing with LUIS is not yet available.

bf luis:application:query --endpoint $location --subscriptionKey $subKey --appId $appId --query "Can you share with my last pay slip" --staging

If needed, we could script various tests to run against the staging slot, and this could roll up as part of the pipeline.

If we're satisified with the performance of the LUIS model, we can also publish the model to the production slot.

bf luis:application:publish --endpoint $location --subscriptionKey $subKey --versionId $versionId --appId $appId

Again, we could also run the queries to validate that the production slot works.

Again, batch testing could also be used (and isn't yet available right now with bf cli), but this will could be used as part of an end to end test as well.

bf luis:application:query --endpoint $location --subscriptionKey $subKey --appId $appId --query "Can you share with my last pay slip"