Auditing
Agents are audited by default. The audit can be configured and limited as part of the "Audit" tab of an agent.
Flowable is separating similar as it's done for BPMN and CMMN between runtime and historic to keep the execution efficient. In the runtime the current running agents are handled. While in the history tables all information about audited agents are saved.
The audit capabilities depend on the type of the agent. There are currently three different levels.
Log basic agent interactions
This will enable the auditing of the agent instances. To the basic information logs the following information:
- Start and end time of an agent
- State of an agent
- Agent definition invoked
- User starting the agent instance
- Reference scope id and type (could be a case or an external agent)
- Agent task which invoked the agent
Log low-level exchange
With logging the low-level exchange, also the information which is sent to the LLM is logged. As well as messages received from the user in case there is a communication through chat. Furthermore, the following information is audited:
- Invoked operation of an agent
- The time when the item was created
- Content which was sent to the LLM
- Content displayed to the user
- The amount of tokens used for the exchange
The token amount splits into two parts, there is the request and the response. For the response the amount matches with exactly that message content. However, for the request the token count is the aggregated count which was consumed from all exchange items sent to the LLM. This means, that it's often not just the single exchange item, it's all exchange items sent during that invocation when this message was sent the first time.
In total, the sum of all token counts are the token counts used during the invocation of this agent instance.
Log knowledge base usage
The knowledge base usage only applies for knowledge agents. When activated, it will store a reference to the specific knowledge base items which have been used during this request. This allows to analyze the sources used for the content item. Furthermore, also the "score" is saved of how good those items are a match. With that it's possible to refine the knowledge base and optimize the search results.