This guide will help you configure Large Language Model (LLM) integration with VisibleThread 7+, allowing you to enable the Visi Chat content generation features.

Important Note: LLM integration requires additional infrastructure and costs beyond the base VT7 deployment. Before proceeding, please consider:

• Hardware requirements for self-hosted LLMs (typically requiring GPU resources)
• API / Token usage costs for cloud-based LLM providers
• Network connectivity and security considerations
• Ongoing maintenance responsibilities

See: Frequently Asked Questions: VT Writer, VTRAG, and LLM Integration
 

Overview

VT7 can integrate with LLMs to provide content generation capabilities. The system is designed to be flexible, working with various LLM providers through a standard configuration interface; That being the OpenAI API.

Understanding LLM Components

VT7 works with two separate LLM components:

1. VT7 LLM (chat completion model): Used for generating content, rephrasing, and other generative AI tasks. This is the "Visi Chat" module.
2. VTRAG Embedding LLM (embeddings model): Used specifically for creating vector embeddings when using document retrieval features. Requires "Visi Chat" to be enabled.
   
   See: How-to-Deploy-VisibleThread-RAG-VTRAG-for-VisibleThread-7
   See: VTRAG-Configuration-Guide

These components must be configured separately, even if you're using the same provider for both.

Prerequisites

Before configuring LLM features, ensure you have:

• VT7 installed
• Access to a compatible LLM provider (e.g. - OpenAI, Azure OpenAI, AWS Bedrock, or Ollama)
  • See the Ollama configuration guide for information on self-hosted LLM
• API keys and endpoint information for your chosen LLM provider
• Administrator access to the VT7 System Admin portal

Configuration Steps

1. Access the System Admin Portal

1. Log in to VT7 with administrator credentials
2. Click the user menu and select "System Admin"
3. Navigate to "Generative AI" in the left sidebar

2. Locate the Generative AI Section

In the System Admin panel, locate the "Generative AI" section, which contains all LLM-related settings.

3. Enable Generative AI Features

Toggle the "Enable generative AI features" switch to the "On" position.

4. Configure LLM Settings

Complete the following fields:

• Framework: Select your LLM provider from the dropdown (Ollama, OpenAI, Azure OpenAI, or AWS Bedrock)
• Endpoint: Enter the API endpoint URL for your LLM provider (typical examples below)
  • For Ollama: http://your-ollama-server:11434
  • For OpenAI: https://api.openai.com
  • For Azure OpenAI: Your deployment-specific endpoint
  • For AWS Bedrock: Your region-specific endpoint
• Model: Enter the model name you wish to use
  • For Ollama: mistral-nemo (recommended) or another installed model
  • For OpenAI: gpt-4-turbo or similar
  • For Azure OpenAI: The deployment name of the model (not model name).
  • For AWS Bedrock: The model ID
• API Key: Enter your API key or access token
  • For Ollama, this may be left blank if no authentication is configured

Example configuration for Azure OpenAI;

5. Optional: Enable Streaming

If your LLM provider supports streaming responses, you can enable the "Enable Streaming" option to see responses appear in real-time.

6. Test, then Save Changes

Click the "Test Connection" button. If information is entered correctly, you should get a "connection successful" message. Then click the "Save Changes" button to apply your configuration. 

Important Considerations

• VT7 does not include any LLM models; you must provide your own
• The customer is entirely responsible for deploying, configuring, maintaining, and governing their LLM
• Performance may vary depending on your chosen LLM provider and model
• Content generation quality depends on the capabilities of your selected LLM
• Costs will vary based on your chosen LLM solution:
  • Self-hosted solutions (Ollama) require hardware investment but have no per-token costs
  • Cloud API services (OpenAI, Azure, AWS) have ongoing token usage fees

Troubleshooting

If you encounter issues with your LLM integration:

1. Verify network connectivity between VT7 and your LLM provider
2. Check that your API key and endpoint URL are correct
3. Check for extra whitespace copied over in your API key and endpoint
4. Ensure your selected model is available through your provider
5. Verify firewall rules allow communication on the required ports
6. Check system logs for any error messages

For additional assistance, contact VisibleThread Support at support@visiblethread.com.