Apply for AI Grants India

What Ollama does—and what this tutorial fixes

Ollama packages the model runtime, model files, and a local HTTP API into a simple developer workflow. It is useful when you want to prototype without sending prompts to a cloud provider, build an internal assistant, or test an AI feature before paying for hosted inference.

This Ollama local LLM setup tutorial uses Ollama’s native installation rather than an unnecessary Docker build. The commands below are suited to a workstation, lab machine, or small server in India. For broader deployment decisions, compare this walkthrough with how to deploy large language models locally.

Check hardware and choose a model

Ollama can run on CPU, but a supported GPU makes generation substantially faster. Before installing, check:

• Operating system: macOS, Linux, or Windows with Ollama’s current desktop or Linux support.
• Memory: 8 GB RAM is a practical starting point for small models; larger models need considerably more.
• Disk space: model files can occupy several gigabytes each. Keep models on an SSD and leave room for updates.
• GPU support: Apple Silicon benefits from unified memory; Linux users should verify NVIDIA drivers and CUDA compatibility. Windows users should use the current Ollama desktop release and supported GPU drivers.
• Network access: required for the first model download, but not for subsequent local inference.

Start small. A 3B–8B instruct model is generally easier to run than a 30B-plus model. Quantised models reduce memory needs, though they can trade away some quality. If your machine is modest, see how to deploy lightweight LLMs locally in 2026. If you need Indian-language or regional-language workflows, test the model with your actual Hindi, Tamil, Bengali, or mixed-language prompts rather than relying only on English benchmarks.

Install Ollama

Use the official installer for your operating system. Avoid cloning the Ollama source repository or building a Docker image unless you are contributing to Ollama itself or have a specific container deployment requirement.

macOS and Windows

Download and install the current desktop application from Ollama’s official website. After installation, launch the application. It runs the local service in the background and provides the ollama command in a terminal.

Open Terminal on macOS or PowerShell on Windows and verify the installation:

ollama --version

Linux

On a supported Linux machine, the official installation command is:

curl -fsSL https://ollama.com/install.sh | sh

Then verify the binary:

ollama --version

If the service is not already running, start it in the foreground for a quick test:

ollama serve

Keep that terminal open, or configure Ollama as a system service for a persistent workstation or server. Do not expose the service directly to the public internet without authentication, network controls, and a clear threat model.

Download and run your first model

Pull a model from the Ollama library using its model name. For example:

ollama pull llama3.2

Run an interactive chat:

ollama run llama3.2

Ask a few representative questions, then exit with Ctrl+D or /bye, depending on the client behaviour. List locally available models with:

ollama list

Remove an unused model to recover disk space:

ollama rm llama3.2

Model tags matter. A larger tag may improve reasoning but increase RAM, VRAM, and latency requirements. Record the exact model and tag in your project documentation so another developer can reproduce your results.

Call the local API

Ollama normally serves its API at http://localhost:11434. Test generation with curl:

curl http://localhost:11434/api/generate \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "llama3.2",
    "prompt": "Give three practical uses for a private local AI assistant in an Indian small business.",
    "stream": false
  }'

For a chat-style request, use /api/chat:

curl http://localhost:11434/api/chat \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "llama3.2",
    "messages": [
      {"role": "user", "content": "Summarise this note in five bullets."}
    ],
    "stream": false
  }'

The default API is intended for local access. A browser visit to the root URL may not show a useful interface; successful API responses are the better health check. For Python applications, install a compatible client or call the endpoint with requests. Keep prompts, model names, timeout values, and response parsing in configuration rather than scattering them through application code.

Configure storage, context, and performance

Ollama stores models in a local directory. If your system disk is small, move the model location using the environment configuration supported by your platform, then restart Ollama. Confirm the new location before deleting the old files.

Useful performance practices include:

• Close memory-heavy applications before loading a large model.
• Prefer an SSD and avoid running models from a slow external drive.
• Use a smaller quantised model when latency matters more than maximum quality.
• Test context length carefully; long documents increase memory use and response time.
• Measure first-token latency, tokens per second, RAM, VRAM, and power draw.
• Keep one stable model for production experiments and separate it from models under evaluation.

On shared office or lab machines, treat prompts and model files as sensitive assets. Local inference reduces third-party exposure, but it does not automatically provide encryption, access control, audit logs, or safe output handling. Teams building privacy-sensitive systems should also review secure local-first operating systems for privacy.

Create a reusable model configuration

For a consistent persona or task, create a Modelfile:

FROM llama3.2
PARAMETER temperature 0.2
SYSTEM You are a concise assistant. State uncertainty and never invent citations.

Build and run it:

ollama create india-helpdesk -f Modelfile
ollama run india-helpdesk

Use low temperature for extraction, classification, and structured business workflows. Validate generated JSON before sending it to another system. If your goal is domain adaptation rather than prompting, read fine-tuning large language models on local hardware before committing to a training pipeline.

Troubleshoot common failures

• `ollama: command not found`: restart the terminal, check installation paths, and reinstall the official package if necessary.
• Connection refused: start Ollama or ollama serve, then confirm that port 11434 is available.
• Model download fails: check connectivity, disk space, proxy settings, and whether a firewall blocks the request.
• Out-of-memory errors: use a smaller model, reduce context length, close other applications, or move inference to a machine with more memory.
• Slow responses: check whether inference is falling back to CPU, update GPU drivers, and compare a smaller quantised model.
• Unexpected answers: improve the system prompt, reduce ambiguity, add examples, and test against a fixed evaluation set.

A practical validation checklist

Before integrating Ollama into an application, verify that you can:

• Install and version the runtime reproducibly.
• Pull the exact model tag required by the project.
• Generate both interactive and API responses.
• Restart the machine and recover the service cleanly.
• Handle timeouts, malformed output, and unavailable models.
• Keep sensitive prompts off logs and backups.
• Document hardware, model, quantisation, latency, and quality results.

Ollama is a strong starting point for local prototypes and private internal tools, but it is not a complete production platform by itself. Add authentication, monitoring, queues, model evaluation, and resource controls as usage grows. For workflow automation, consider automating personal workflows with local AI agents, while keeping human review for consequential decisions.

FAQ

Is Ollama free?

The Ollama software is available at no licence cost for local use, but you still pay for hardware, electricity, storage, and engineering time. Model licences vary, so review the terms before commercial deployment.

Can I run Ollama without a GPU?

Yes. CPU inference works for smaller models, although responses may be slower. Start with a compact quantised model and measure performance on your actual workload.

Should I use Docker?

Not for a first installation. Native installation is simpler and usually gives better access to local GPU acceleration. Use Docker only when your deployment architecture requires container isolation or orchestration.

Can other devices on my network use the API?

Ollama is commonly used through localhost. Network binding can be configured for controlled environments, but expose it only behind suitable access controls and a private network. Never assume a local API is safe to publish openly.

Can I use Ollama for Indian-language applications?

Yes, provided the selected model supports the language and script well enough for your task. Evaluate transliteration, code-mixed prompts, names, local place references, and domain terminology with a representative dataset.