You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Related to #152 but slightly different. This issue is focused to auditing capabilities for Classifiers, in particular on the input, output, and execution context of Classifier::process_request().
In highly regulated environments, customers are required to inspect punctually all the interactions among agents, but most improtantly at the interface between human users and the orchestrator.
Classification and dispatching of intents are critical for the correct operation and evolution of an agentic system, especially if this is to be integrated with existing legacy systems (CRMs, CMSs, Payment...).
Our main challenge with MAO's current implementation is the lack of access to session_id and user_id at the classifier scope. Ideally we'd like to be able to log input, output and some contextual information related to the invocation of the LLM (prompt version id, classification confidence label, inference parameters, link to chat history and messages used in the inference context window, input and outputs of encoders/decoders). We'd like to index this information based on session_id and user_id.
In #152 an approach is proposed to audit agent behaviour, but this won't provide the same capabilites for classifiers.
Ideally these two trace streams (agents and user/encoders/decoders/classifiers) would then be combined to produce a full audit trace of every interaction with the system.
Solution/User Experience
## Proposed Solution
provide user_id and session_id in the scope of downstream components of the Orchestrator, like a classifier
wrappers/decorators or global static logging channel for auditing purposes
so that whenever this method is executed relevant information is stored in the auditing channel of choice (in this example, a DynamoDB table, similar to what's done for Chat History)
alternatively
defprocess_request(self,input_text: str, chat_history: List[ConversationMessage], user_id, session_id) ->ClassifierResult:
AuditStreamProvider.log(user_id, session_id, ClassifierResult, ClassifierInput) # this could also be a logger, but we'd still require access to user_id and session_id
Alternative solutions
The text was updated successfully, but these errors were encountered:
Use case
Related to #152 but slightly different. This issue is focused to auditing capabilities for Classifiers, in particular on the input, output, and execution context of Classifier::process_request().
In highly regulated environments, customers are required to inspect punctually all the interactions among agents, but most improtantly at the interface between human users and the orchestrator.
Classification and dispatching of intents are critical for the correct operation and evolution of an agentic system, especially if this is to be integrated with existing legacy systems (CRMs, CMSs, Payment...).
Our main challenge with MAO's current implementation is the lack of access to session_id and user_id at the classifier scope. Ideally we'd like to be able to log input, output and some contextual information related to the invocation of the LLM (prompt version id, classification confidence label, inference parameters, link to chat history and messages used in the inference context window, input and outputs of encoders/decoders). We'd like to index this information based on session_id and user_id.
In #152 an approach is proposed to audit agent behaviour, but this won't provide the same capabilites for classifiers.
Ideally these two trace streams (agents and user/encoders/decoders/classifiers) would then be combined to produce a full audit trace of every interaction with the system.
Solution/User Experience
## Proposed Solution
User Experience
in the implementation of a classifier
so that whenever this method is executed relevant information is stored in the auditing channel of choice (in this example, a DynamoDB table, similar to what's done for Chat History)
alternatively
Alternative solutions
The text was updated successfully, but these errors were encountered: