Disambiguation: Dynamic context for effective RAG question suggestions

This approach reminds of a technique initially implemented by IBM Watson called disambiguation. Where ambiguous input from the user is responded to with about five or less options to choose from. Hence allowing the user to perform disambiguation themselves, and allowing the system to learn from it.

This approach also solves for the blank canvas problem, where the user does not know what the ambit of the interface is, and ends up asking ambiguous exploratory questions.

So instead of forcing users to continuously refine their queries while being furnished with inadequate responses, the Conversational UI takes the initiative to disambiguate the conversation.

The generator is designed to generate suggestion questions that the agent can answer, guided by the user’s initial query. ~ Source

Proposed framework

1. The framework identifies appropriate passages (dynamically retrieved contexts) based on the user’s initial input.
2. The process is aided by questions, answer and suggestion questions (generated via dynamic few-shot).
3. Suggestion questions are generated by constituting a dynamic context prompt consisting of the dynamically retrieved contexts, and dynamic few- shot examples.
4. By providing suggestion disambiguation questions, users are relieved of the cognitive load of question formulation.
5. The fundamental aim of this research is to establish the user’s true intent during interaction.
6. What is appealing of this solution, is that it is a low resource approach.

Task orientated dialogs

The example below is one of task-oriented information seeking. A user needs to search for information related to a specific task. Due to the ambiguous user input and user profile, the task- oriented dialogue system should ask clarification/disambiguation questions to clarify the user intent and the user profile based on the task knowledge to provide the answer.

Below, the task-oriented System Ask Paradigm.

Conclusion

This study aims to reduce apologetic responses from the Conversational UI and enhance user experience by providing informed system suggestions.

Dynamic Contexts generate suggestion questions answerable by the agent through:

1. Dynamic few-shot examples and
2. Dynamically retrieved contexts.

Unlike traditional few-shot prompting, dynamic few-shot examples dynamically select contextually relevant triplets based on user queries, accommodating varied question formats and structures.

Links to the studies:

Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialogues