70 lines
3.8 KiB
Markdown
70 lines
3.8 KiB
Markdown
# Codex TrueNAS Helper
|
|
|
|
This project is a collection of scripts, configurations, and applications to manage and enhance a TrueNAS SCALE server, with a special focus on running and interacting with large language models (LLMs) like those powered by `llama.cpp` and `Ollama`.
|
|
|
|
## Features
|
|
|
|
* **`llama.cpp` Wrapper:** A sophisticated wrapper for the `llama.cpp` TrueNAS application that provides:
|
|
* An OpenAI-compatible API for chat completions and embeddings.
|
|
* A web-based UI for managing models (listing, downloading).
|
|
* The ability to hot-swap models without restarting the `llama.cpp` container by interacting with the TrueNAS API.
|
|
* **TrueNAS Inventory:** A snapshot of the TrueNAS server's configuration, including hardware, storage, networking, and running applications.
|
|
* **Automation Scripts:** A set of PowerShell and Python scripts for tasks like deploying the wrapper and testing remote endpoints.
|
|
* **LLM Integration:** Tools and configurations for working with various LLMs.
|
|
|
|
## Directory Structure
|
|
|
|
* `AGENTS.md` & `AGENTS.full.md`: These files contain detailed information and a complete inventory of the TrueNAS server's configuration.
|
|
* `llamaCpp.Wrapper.app/`: A Python-based application that wraps the `llama.cpp` TrueNAS app with an OpenAI-compatible API and a model management UI.
|
|
* `scripts/`: Contains various scripts for deployment, testing, and other tasks.
|
|
* `inventory_raw/`: Raw data dumps from the TrueNAS server, used to generate the inventory in `AGENTS.full.md`.
|
|
* `reports/`: Contains generated reports, test results, and other artifacts.
|
|
* `llamacpp_runs_remote/` & `ollama_runs_remote/`: Logs and results from running LLMs.
|
|
* `modelfiles/`: Modelfiles for different language models.
|
|
* `tests/`: Python tests for the `llamaCpp.Wrapper.app`.
|
|
|
|
## `llamaCpp.Wrapper.app`
|
|
|
|
This is the core component of the project. It's a Python application that acts as a proxy to the `llama.cpp` server running on TrueNAS, but with added features.
|
|
|
|
### Running Locally
|
|
|
|
1. Install the required Python packages:
|
|
```bash
|
|
pip install -r llamaCpp.Wrapper.app/requirements.txt
|
|
```
|
|
2. Run the application:
|
|
```bash
|
|
python -m llamaCpp.Wrapper.app.run
|
|
```
|
|
This will start two web servers: one for the API (default port 9093) and one for the UI (default port 9094).
|
|
|
|
### Docker (TrueNAS)
|
|
|
|
The wrapper can be run as a Docker container on TrueNAS. See the `llamaCpp.Wrapper.app/README.md` file for a detailed example of the `docker run` command. The wrapper needs to be configured with the appropriate environment variables to connect to the TrueNAS API and the `llama.cpp` container.
|
|
|
|
### Model Hot-Swapping
|
|
|
|
The wrapper can switch models in the `llama.cpp` server by updating the application's command via the TrueNAS API. This is a powerful feature that allows for dynamic model management without manual intervention.
|
|
|
|
## Scripts
|
|
|
|
* `deploy_truenas_wrapper.py`: A Python script to deploy the `llamaCpp.Wrapper.app` to TrueNAS.
|
|
* `remote_wrapper_test.py`: A Python script for testing the remote wrapper.
|
|
* `update_llamacpp_flags.ps1`: A PowerShell script to update the `llama.cpp` flags.
|
|
* `llamacpp_remote_test.ps1` & `ollama_remote_test.ps1`: PowerShell scripts for testing `llama.cpp` and `Ollama` remote endpoints.
|
|
|
|
## Getting Started
|
|
|
|
1. **Explore the Inventory:** Start by reading `AGENTS.md` and `AGENTS.full.md` to understand the TrueNAS server's configuration.
|
|
2. **Set up the Wrapper:** If you want to use the `llama.cpp` wrapper, follow the instructions in `llamaCpp.Wrapper.app/README.md` to run it either locally or as a Docker container on TrueNAS.
|
|
3. **Use the Scripts:** The scripts in the `scripts` directory can be used to automate various tasks.
|
|
|
|
## Development
|
|
|
|
The `llamaCpp.Wrapper.app` has a suite of tests located in the `tests/` directory. To run the tests, use `pytest`:
|
|
|
|
```bash
|
|
pytest
|
|
```
|