Codex TrueNAS Helper

This project is a collection of scripts, configurations, and applications to manage and enhance a TrueNAS SCALE server, with a special focus on running and interacting with large language models (LLMs) like those powered by llama.cpp and Ollama.

Features

• llama.cpp Wrapper: A sophisticated wrapper for the llama.cpp TrueNAS application that provides:
  • An OpenAI-compatible API for chat completions and embeddings.
  • A web-based UI for managing models (listing, downloading).
  • The ability to hot-swap models without restarting the llama.cpp container by interacting with the TrueNAS API.
• TrueNAS Inventory: A snapshot of the TrueNAS server's configuration, including hardware, storage, networking, and running applications.
• Automation Scripts: A set of PowerShell and Python scripts for tasks like deploying the wrapper and testing remote endpoints.
• LLM Integration: Tools and configurations for working with various LLMs.

Directory Structure

• AGENTS.md & AGENTS.full.md: These files contain detailed information and a complete inventory of the TrueNAS server's configuration.
• llamaCpp.Wrapper.app/: A Python-based application that wraps the llama.cpp TrueNAS app with an OpenAI-compatible API and a model management UI.
• scripts/: Contains various scripts for deployment, testing, and other tasks.
• inventory_raw/: Raw data dumps from the TrueNAS server, used to generate the inventory in AGENTS.full.md.
• reports/: Contains generated reports, test results, and other artifacts.
• llamacpp_runs_remote/ & ollama_runs_remote/: Logs and results from running LLMs.
• modelfiles/: Modelfiles for different language models.
• tests/: Python tests for the llamaCpp.Wrapper.app.

llamaCpp.Wrapper.app

This is the core component of the project. It's a Python application that acts as a proxy to the llama.cpp server running on TrueNAS, but with added features.

Running Locally

1. Install the required Python packages:

   pip install -r llamaCpp.Wrapper.app/requirements.txt
   

2. Run the application:

   python -m llamaCpp.Wrapper.app.run
   

   This will start two web servers: one for the API (default port 9093) and one for the UI (default port 9094).

Docker (TrueNAS)

The wrapper can be run as a Docker container on TrueNAS. See the llamaCpp.Wrapper.app/README.md file for a detailed example of the docker run command. The wrapper needs to be configured with the appropriate environment variables to connect to the TrueNAS API and the llama.cpp container.

Model Hot-Swapping

The wrapper can switch models in the llama.cpp server by updating the application's command via the TrueNAS API. This is a powerful feature that allows for dynamic model management without manual intervention.

Scripts

• deploy_truenas_wrapper.py: A Python script to deploy the llamaCpp.Wrapper.app to TrueNAS.
• remote_wrapper_test.py: A Python script for testing the remote wrapper.
• update_llamacpp_flags.ps1: A PowerShell script to update the llama.cpp flags.
• llamacpp_remote_test.ps1 & ollama_remote_test.ps1: PowerShell scripts for testing llama.cpp and Ollama remote endpoints.

Getting Started

1. Explore the Inventory: Start by reading AGENTS.md and AGENTS.full.md to understand the TrueNAS server's configuration.
2. Set up the Wrapper: If you want to use the llama.cpp wrapper, follow the instructions in llamaCpp.Wrapper.app/README.md to run it either locally or as a Docker container on TrueNAS.
3. Use the Scripts: The scripts in the scripts directory can be used to automate various tasks.

Development

The llamaCpp.Wrapper.app has a suite of tests located in the tests/ directory. To run the tests, use pytest:

pytest