Broker guide
Local LLM: Run AI on Your Own Computer
A local LLM runs on your computer and can keep client data on the device. See what hardware you need, which models to try and what they cannot do.
- Published
- Updated
A local large language model (LLM) runs on your own computer and can process prompts and documents without sending them to a hosted AI service. You need a compatible model, enough memory and an app that runs it. For a broker, local AI is useful for document summaries and draft writing when the whole workflow stays on the device.
Start with a small model and a fictional document. A laptop that handles short text comfortably can still struggle with a large client file. Keep lender-policy decisions separate from the model’s general knowledge.
What a Local LLM Does
A local LLM generates answers using model files stored on your computer. The app loads those files into memory and processes your prompt there. Once the files and required software are downloaded, this can work without an internet connection.
A model can rewrite an email or summarise information you supply. For example, it can turn a fictional employment history into a short file note. It doesn’t automatically have access to your documents, email or lender portals.
The model and the app do different jobs. The model produces language, while the app handles tasks such as reading an attachment and saving the conversation. A local model inside an app that calls external services doesn’t make the entire workflow private.
LM Studio’s offline documentation says downloaded models and its document-chat processing work locally. That makes it a useful desktop starting point. Your brokerage still needs to manage any saved chats and copies in synced folders.
“Self hosted LLM” can also describe a model running on a rented server. A local LLM runs on your own device. The broader private AI options include managed services and private cloud deployments with different responsibilities.
What You Need to Run One
To run a local LLM, you need a supported computer with enough free memory and disk space for the chosen model. System memory, or RAM, holds the running model alongside your other applications. Graphics memory can help run it, but it is separate from RAM on many Windows computers.
LM Studio’s system requirements recommend at least 16GB of RAM. Its Mac app requires Apple Silicon and macOS 14 or newer. For Windows, its recommendations include at least 4GB of dedicated graphics memory.
An 8GB Mac can run smaller models with modest context sizes, according to those requirements. For document work, 16GB is a more practical starting point. More memory gives you room for larger models and longer inputs, but doesn’t establish how quickly a particular computer will answer.
Choose a compressed model download that leaves memory available for the operating system and your documents. Compression is called quantisation. LM Studio’s model download guidance recommends a 4-bit option or higher when the computer can run it.
The download size is only part of the memory requirement. Longer conversations need more working space. Keep spare disk space for the app and any additional model files you download.
Run Your First Model
Use fictional information until your brokerage has approved the setup for client work. These steps follow LM Studio’s getting started instructions.
- Install LM Studio on a computer that meets its requirements. Open the app and confirm it starts before downloading a model.
- Open Discover and search for
Qwen3.5-9B. Choose a supported download that fits your computer, using a 4-bit version if suitable. Wait until the download completes. - Open Chat, then open the model loader and select the downloaded model. Wait until it is loaded before sending a prompt.
- Paste this fictional practice note: “The applicant has worked for Employer A for 18 months. The latest payslip is missing.” Ask for a one-sentence summary using only those facts.
- Confirm the answer preserves the 18-month employment history and identifies the missing payslip. Disconnect the computer from the network and repeat the prompt in a fresh chat.
An answer while disconnected confirms that local generation works. It doesn’t prove every connected feature stays local when you reconnect. Keep external integrations outside the approved client workflow unless your brokerage has assessed their processing.
LM Studio supports attaching PDF, DOCX and text files to a chat. Its document-chat guide explains that short documents can enter the conversation in full. For longer documents, it retrieves selected passages, so an answer can miss material elsewhere in the file.
Fix Common Setup Problems
| Symptom | Check | Next action |
|---|---|---|
| The model won’t load | Available memory and the selected model’s size | Close other memory-heavy apps. Choose a smaller model or reduce the context setting. |
| Answers take too long | Model size and whether the app uses available graphics hardware | Start with a smaller model and a short prompt before adding documents. |
| A document answer omits information | Whether the relevant text is readable and reaches the model | Ask about a short extracted passage. Compare the answer with the original document. |
Best Local LLMs for Document Work
Qwen3.5-9B is a practical first choice for short document summaries and draft writing when your computer supports it. Gemma 4 12B is another option for a machine with more memory available. Choose by the documents you need to process and the model format your app supports.
| Model version | Document and writing use | What changes the choice |
|---|---|---|
| Qwen3.5-9B | General text work with image input support. Its official evaluations include document understanding. | The smaller parameter count makes it a sensible first download before trying larger models. |
| Gemma 4 12B | Text generation with native visual and audio inputs. Useful when your workflow includes more than extracted text. | Google describes local operation with 16GB of graphics or unified memory. Leave room for the app and the document. |
The Qwen3.5-9B model card lists its supported inputs and evaluations. Its advertised context window is larger than a modest laptop can necessarily use comfortably. A model’s maximum input length doesn’t guarantee that your chosen download and hardware can sustain it.
Google’s Gemma 4 12B announcement describes its native multimodal inputs and memory target. The app must support those inputs too. A model that understands images cannot inspect a scanned page if the app supplies only extracted text.
For writing, ask the model to draft from supplied facts and preserve figures exactly. For document reading, ask it to identify the passage supporting each answer. Keep the original open when checking names or amounts before using the result in a client file.
What Local Models Cannot Do
Local models cannot supply live lender information while they are offline. Their downloaded knowledge doesn’t update when a lender changes policy. An old policy PDF remains old when a model summarises it.
Laptop-sized models can also give weaker answers than leading hosted models on complex tasks. Reducing precision to fit a model into memory can reduce answer quality further. Privacy and reasoning quality need separate decisions.
A fluent summary can omit a condition or invent a fact. Check consequential statements against the original document. A local model’s explanation cannot establish that a borrower meets a lender’s current assessment rules.
Your brokerage also owns the device-security work. Restrict access to saved conversations and client documents, protect the computer with encryption and keep its software updated. Include backups and synced folders when deciding where the data can go.
Start with a repeatable task such as drafting an internal note from approved source text. Give the task an owner who checks the output and controls storage. Expand to another document type only after the brokerage has approved that workflow too.
