Over the past year and a half, the rapid improvement of generative AI has created a wealth of new opportunities to deliver impact in the digital space. One of the main areas that has seen renewed interest due to enhanced capabilities is chatbots and conversational AI. The introduction of Retrieval-Augmented Generation, or RAG, now allows AI applications to enhance the capabilities of machine learning models by integrating them with a retrieval component. This in turn enables conversational AI and advanced chatbot applications that produce highly contextual and informed responses for users based on custom data models and sources.

However, developing these conversational AI applications in a robust, scalable, and efficient way is no easy task. The following is a comprehensive guide to help organizations looking to build a conversational AI that is enhanced by the use of RAG, including considerations for planning, design, and technology stack.

Choose the Right Approach

There are many options for designing and developing a RAG-enhanced conversational AI solution. High-level options include custom GPTs on top of general purpose Large Language Models (LLMs) such as ChatGPT, vertical-specific out-of-the-box solutions, semi-custom vendor solutions such as Azure OpenAI, and fully custom models and applications. 

Each type of solution has benefits and drawbacks, so it’s important to understand the landscape and map the right solution to the use case at hand. A robust discovery phase is recommended to understand both user and organizational requirements prior to selecting a particular approach, solution, vendor, or model. 

Design Responsibly

Responsible AI refers to the practice of designing, developing, and deploying artificial intelligence systems in a manner that is ethical, transparent, and accountable. It is an important factor to consider when looking at not only designing a RAG-based conversational AI, but also when designing organizational AI policy in general. Designing responsible AI involves ensuring that AI technologies are being used to benefit people and society while minimizing harm and respecting human rights. 

Key principles and considerations in responsible AI include, but aren’t limited to:

• Ethical Consideration: AI should be developed and used according to ethical guidelines and values. This includes respecting human dignity, privacy, and rights, and ensuring fairness and justice in AI outcomes.
• Privacy and Security: The development and use of AI must respect user privacy and ensure data security. This includes protecting personal and sensitive data from unauthorized access and ensuring that data collection and use follow relevant laws and regulations.
• Safety and Reliability: AI systems should be safe and reliable. They should function as intended, and there should be measures in place to prevent and mitigate harmful malfunctions or misuse.
• Human in the Loop: AI should augment, not replace, human decision-making. Humans should remain in control of critical decisions, and AI should be used as a tool to enhance human capabilities.
• Inclusivity: AI development should include diverse perspectives and stakeholders to ensure that AI systems are inclusive and address the needs of a wide range of users.

Plan for Scalability & Change

• Plan for Scalability: RAG applications can grow quickly in terms of data and user load. Utilizing cloud services like AWS, Azure or Google Cloud is important in providing scalability and flexibility. In addition, consider container-based or serverless solutions that can provide unlimited scalability while limiting overall cost.
• Avoid Vendor and Model Lock-in: The AI space is still evolving rapidly. It’s important to be able to test various pre-trained models to determine which delivers the best results. In addition, a worst-case scenario is being highly coupled to an outdated model as advancements continue. To mitigate this, it’s important to follow a microservice-based approach and to implement a tech stack that allows for easy swapping between models. This can be built as a custom solution or through utilizing tools such as LangChain.
• Continuously Learn and Adapt: It’s important to design processes and implement systems that allow for the continual training of models with new data to improve the accuracy and relevance of responses over time. This includes setting up feedback loops from user interactions and vetting and testing models with internal stakeholders before wider release or adoption by the public.

Consider the User Experience

As with any digital product, the success of a conversational AI relies heavily on the user experience. The interface should be intuitive and responsive, providing quick, relevant, and accurate answers to user questions. There are several user experience considerations unique to RAG-enhanced conversational AIs that are vital to ensuring the best experience for users, including:

• Context Awareness: The system should maintain context over the course of a conversation, understanding previous interactions and adjusting responses accordingly.
• Consistency: Responses should be consistent in voice, tone, style, and factual accuracy across different sessions and contexts.
• Latency: Users expect quick responses in conversational interfaces. The latency in fetching documents and generating responses should be minimized to maintain a smooth and natural conversation flow. Techniques such as streaming, back-pressure, and response caching can be used to mitigate latency.
• Feedback Loops: Allow users to correct misunderstandings by the AI, which can help in refining the context or the user’s intent. Additionally, implement features alongside the conversational AI for users to provide feedback, which can then be used to improve model performance and accuracy.
• Response Quality and Relevance: The most important factor is working to ensure that the responses generated are not only relevant but also accurate, providing trustworthy and verifiable information. Techniques such as pre-grounding the model, built-in prompt engineering, and post-grounding in the UI can all be used to improve quality and accuracy.

Choose the Right Technology Stack

When developing a fully custom conversational AI, it’s important to choose a technology stack that is flexible, scalable, and future-friendly. There are several layers of the stack that are important to consider when developing a RAG-enhanced conversational AI solution:

• Data Layer: The data layer is the base layer of the stack and consists of connections to external and internal data sources, a way to store data points, and a means to transform data into a usable state through data cleaning, processing, and transformation. From a database perspective, options include relational databases such as PostgreSQL and vector databases such as Neo4j. Vector or graph-based databases can often offer more efficiency when querying relationships between data points. 
• Retrieval Layer: The retrieval layer is an integral part of the stack where search and querying of the data take place to support the generation of responses. Tools such as Pgvector can be used for efficient vector search and to perform advanced search operations vital for handling large datasets and complex queries. Selecting the right technology or combination of technologies for the retrieval layer depends on specific project needs such as the expected query load, the type of queries, latency requirements, and the nature of the data involved. 
• Machine Learning Model: The machine learning layer is where model training and management take place. This layer typically includes a framework such as TensorFlow or PyTorch to handle advanced model training and deployment. As part of this layer, leveraging pre-trained models such as GPT for generating human-like text or BERT for understanding context can significantly reduce development time and improve overall quality and efficiency. It is also important to structure this layer in a way that allows for migrating between pre-trained models and model versions or using multiple pre-trained models for different needs, either through a custom orchestration layer or off-the-shelf solutions such as LangChain.
• Application Layer: The application layer consists of both the APIs used to handle interaction between the user interface and the back-end AI services as well as the front-end application that users interact with. It is important to construct this layer in a way that is flexible and scalable and in a decoupled fashion. Conversational AI frameworks such as the Vercel AI SDK can be leveraged to build a high-quality user experience. These frameworks come with pre-built functionality for things like streaming, back-pressure, and response caching to mitigate latency as well as helpers for orchestrating the interaction between the user interface, model APIs, and pre-trained models. 

Final Thoughts

Building a RAG-enhanced conversational AI involves careful planning, design, and consideration of the technology stack. It requires a blend of robust data handling, efficient retrieval mechanisms, powerful machine-learning models, and an excellent user experience. By thoughtfully integrating these components alongside a responsible approach to AI, a solution can be created that exceeds user expectations, provides insightful and context-aware interactions, and continues to scale and improve alongside the rapidly changing landscape of generative AI.