General Overview
What is VT Writer?
VT Writer is a browser-based application (with an optional MS Word Add-in) for creating and editing textual content. It scores content, suggests improvements for clarity, and can generate content when connected to an LLM.
What is VTRAG?
VTRAG is an optional component service that enables Retrieval Augmented Generation (RAG) functionality for VT Writer. It allows your proprietary documents to be used as context when generating content in VT Writer, improving relevance and accuracy. VTRAG requires its own LLM model specific to Embeddings.
See: VTRAG Configuration Guide
How do VT Writer and VTRAG work together?
VT Writer can connect to VTRAG to provide document-based context for generative AI features. VTRAG handles the embedding of document chunks, which can then be used as context when VT Writer communicates with an LLM. This is an optional portion of the LLM functionality and is accessible via the Use Files feature in the LLM prompt window.
LLM Integration
What is an LLM?
A Large Language Model (LLM) is a type of artificial intelligence that has been trained on vast amounts of text data to understand and generate human-like text.
In the context of VT Writer, the capabilities available after integrating an LLM are primarily content creation and rephrasing.
Does VT Writer come with an LLM built in?
No, VT Writer does not come packaged with any LLM. When deployed in a customer-hosted environment, there is no Large Language Model component included. Customers who wish to use generative features must provide and maintain their own LLM, and that is wholly the responsibility of the customer.
What types of LLMs can I use with VT Writer?
VT Writer is LLM-agnostic, meaning it can work with most standard LLMs that satisfy basic integration requirements. Many options offered by OpenAI, Azure OpenAI (including GCC High), and AWS Sagemaker will work. The primary dependency is that the API interface be OpenAI API-compatible.
The recommended option for customers who do not want to use an external service is a self-hosted Ollama server with the LLM model Mistral NeMo, which has a permissive license and is optimized for text creation and summarization. See How to configure a customer-hosted LLM server with Ollama
How do I enable LLM features in VT Writer?
LLM features can be enabled through the VT Writer system admin portal. You'll need to:
- Ensure you have access to a compatible, accessible LLM
- Enable generative AI features in the admin settings
- Configure the Framework, Endpoint, Model, and API Key for your LLM
See: VT Writer LLM Configuration Guide
Who is responsible for LLM governance and maintenance?
The customer is entirely responsible for deploying, configuring, maintaining, and governing their LLM. Many customers deploying VT Writer will already have an available Customer Controlled LLM they can
integrate into. VisibleThread makes no warranties regarding the fitness, performance, or bias of any customer-controlled LLM.
Is my data secure when using VT Writer?
Yes, your data is secure.
No customer data ever goes outside of your firewall, and all data is 100% secure and private to your
environment. VisibleThread staff have no ability to see your data, since it is 100% controlled by you behind the
firewall.
Does the LLM train on my data?
No, no data is used to train any aspect of the LLM as part of the VT Writer application.
Understanding the Two Separate LLM Components
VT Writer requires access to an LLM for 2 specific scenarios (both are optional)
1. Generative AI capabilities that enabled the AI to rephrase or simplify content and generate new draft content.
2. RAG capabilities that retrieve documents in your domain and use snippets of those to influence the AI content generation.
Does VT Writer use the same LLM for content generation and document retrieval?
No. VT Writer and VTRAG use two separate LLM components for different purposes:
- VT Writer LLM: Used for generating content, rephrasing, and other generative AI tasks. This is typically a general-purpose LLM like GPT-4, Mistral NeMo, or similar models that excel at text generation.
- VTRAG Embedding LLM: Used specifically for creating vector embeddings of document chunks. This is an embedding-specialized model like mxbai-embed-large (for Ollama), text-embedding-3-small (for Azure OpenAI), or similar models optimized for text embedding rather than generation.
Do I need to configure both LLMs separately?
Yes. You need to:
- Configure the main LLM in VT Writer through the System Admin portal
- Configure the embedding LLM in VTRAG through its .env file
Each requires its own separate API keys, endpoints, and model selections.
Can I use the same service provider for both LLMs?
Yes, you can use the same provider (e.g., Azure OpenAI) for both, but you would still need to configure separate models - one for text generation in VT Writer and another for embeddings in VTRAG. They serve different functions and typically use different model types.
What happens if one LLM service is unavailable?
If the main VT Writer LLM is unavailable, generative features won't work, but you can still use the scoring and readability features of VT Writer.
If the VTRAG embedding LLM is unavailable, document retrieval features won't work, but VT Writer's general content generation capabilities (without document context) will still function.
Do the two LLMs need to be from the same vendor?
No. You can mix and match providers according to your preferences and requirements. For example, you could use Ollama for embeddings in VTRAG while using Azure OpenAI for content generation in VT Writer.
VTRAG Configuration
How do I configure VTRAG?
See: VTRAG Configuration Guide
VTRAG is configured using an .env file with various parameters. The key settings include:
- Database configuration (PostgreSQL with pgvector extension)
- LLM service selection (via SPRING_PROFILE)
- Service-specific authentication details
- Document handling parameters
What embedding models can VTRAG use?
VTRAG supports multiple embedding services:
- Ollama (e.g., mxbai-embed-large)
- OpenAI embedding models
- Azure OpenAI (text-embedding-3-small, text-embedding-3-large)
- AWS Bedrock (Titan or Cohere embeddings)
Can I use multiple embedding services simultaneously?
No, you can only configure one embedding service at a time. You should set the appropriate SPRING_PROFILE (e.g., "ollama", "openai", "azure-openai", or "bedrock") and remove or comment out the configuration sections for unused profiles.
Does VTRAG need to be on the same server as VT Writer?
No, VTRAG can be deployed standalone or on the same server as VT Writer. It just needs a PostgreSQL database with the pgvector extension enabled and network connectivity to VT Writer.
How does VT Writer connect to VTRAG?
VT Writer connects to VTRAG via a REST API, typically on port 8010. In the VT Writer web UI, you'll need to configure the VTRAG endpoint (default: http://localhost:8010).
Document Management
What file types can VTRAG process?
VTRAG processes Word (.docx) and PDF (.pdf) files.
How are documents stored in VTRAG?
VTRAG doesn't store entire documents; instead, it breaks them into chunks of text and stores these chunks in a vector database (PostgreSQL with pgvector). The chunks are automatically generated and indexed.
How long are documents kept in VTRAG?
Documents are automatically deleted after a configurable period (default: 90 days). This can be adjusted in the VTRAG configuration.
Is there a size limit for documents?
Yes, by default the maximum file size is approximately 19MB per file, but this is configurable via the vtrag.env file.
SharePoint Integration
See: VT Writer SharePoint Integration Guide
Can VTRAG access documents from SharePoint?
No, VTRAG itself doesn't connect directly to SharePoint. VT Writer can be configured to access SharePoint, and users can select SharePoint documents through the VT Writer interface. These documents are then processed by VTRAG.
What SharePoint environments are supported?
For both customer-hosted and VisibleThread cloud-hosted deployments, SharePoint Online (standard Microsoft cloud) is supported. For customer-hosted deployments, SharePoint in GCC and GCC-High (Microsoft Government Community Cloud) is also supported. SharePoint Server editions are NOT supported.
How do I configure SharePoint integration?
SharePoint integration is configured in the VT Writer web interface. The process involves:
- Creating an App Registration in Microsoft Entra
- Configuring VT Writer's SharePoint Connection with the required app IDs and secrets
What permissions are needed for SharePoint integration?
VT Writer uses delegated permissions through Microsoft Graph. Users can only access documents they already have permission to view in SharePoint.
Performance and Security
Does using VTRAG with VT Writer affect performance?
Performance may degrade with large collections of documents. For optimal results, users should be selective about which documents they include in a collection when prompting.
What are the deployment options for VTRAG?
VTRAG can be deployed:
- Using Ollama (completely offline, no external API services required). Note that an Ollama server requires a GPU with enough VRAM to run your chosen model.
- Using cloud API services (OpenAI, Azure OpenAI, AWS Bedrock)
Are there cost implications to different LLM services?
Yes. Ollama requires a server with GPU, typically costing $300/month or more on cloud providers. External API services (OpenAI, Azure, AWS) may be more cost-effective for smaller deployments but have ongoing token usage fees.
Troubleshooting
What are common issues when setting up VT Writer with LLM/VTRAG?
Common issues include:
- Network connectivity problems
- Incorrect endpoint URLs
- PostgreSQL configuration issues, particularly with the pgvector extension
- Improper configuration of the SPRING_PROFILE
How can I verify VTRAG is working correctly?
After configuring VTRAG and connecting it to VT Writer, you can test the connection by uploading a test document and creating a prompt that references information in that document.
Where can I get technical support?
You can find technical support by contacting the VisibleThread Support Team at support@visiblethread.com.