# Plugin Framework Data Access and Persistence

Date: 2026-07-03
Repository: `C:\Users\Tee\projects\llmloader2`
Related references:

- `docs/chat_js_plugin_framework_hook_inventory.md`
- `docs/chat_js_plugin_framework_developer_api.md`

## Purpose

This document explains how plugins in `llmloader2` access framework-owned data and how they persist plugin data safely.

It is based on the current codebase, not a hypothetical design.

The main goals are:

- show how plugins read framework state and database-backed information
- show the real persistence patterns already used in the repo
- explain when to use each pattern
- reduce regressions by keeping persistence logic in the right layer

## Core Rule

A plugin should use the highest-level framework surface that already exists.

In practice that means:

1. Prefer `app.state` objects and helper functions if the framework already exposes them.
2. Prefer plugin-owned JSON/file storage under `data_dir` for simple plugin state.
3. Prefer the collaboration DB object `app.state.collab_db` over raw SQLite when the data belongs to projects, sessions, users, or chat messages.
4. Use direct SQLite only when the plugin is intentionally creating and managing its own schema or when the framework DB object does not expose the needed operation.

## 1. Where Framework Data Comes From

The main framework-owned server data surfaces are attached in `app.py` through `app.state`.

Representative examples currently exposed:

- `app.state.settings`
- `app.state.sess_meta`
- `app.state.user_rag`
- `app.state.lib_rag`
- `app.state.jobs`
- `app.state.ai_jobs`
- `app.state.gen_scheduler`
- `app.state.model`
- `app.state.set_model`
- `app.state.repo_ingest`

These are installed centrally in [app.py](C:\Users\Tee\projects\llmloader2\app.py).

## Why this matters

If a plugin needs framework information that already lives on `app.state`, it should usually read it from there rather than opening files or DB tables directly.

That keeps the plugin aligned with the app's actual runtime state and reduces duplicated persistence logic.

## 2. The Main Ways Plugins Access Framework Data

## 2.1 Reading `app.state` directly

This is the most common pattern for helper routes and workflow code.

Example:

```python
settings = app.state.settings() if callable(app.state.settings) else app.state.settings
jobs = app.state.ai_jobs
user_rag = app.state.user_rag
```

Use this when:

- the app already exposes a runtime object
- the plugin is reading active in-memory state
- the plugin should not care where that state originally came from

Typical use cases:

- active model access
- settings lookup
- job registry lookup
- scheduler lookup
- RAG manager access

## 2.2 Using `require_state(...)` from the helper framework

This is the safer pattern when a dependency might be registered under different names.

Utility:

- `plugins.gui_helpers._framework.utils.require_state`

Example from repo pattern:

```python
from plugins.gui_helpers._framework.utils import require_state

def _api(app):
    return require_state(app, "repo_api", "repo_ingest", "repo_panel_api")
```

Use this when:

- a plugin depends on framework state that may vary by setup
- you want a clear failure if the dependency is missing

This is better than hardcoding a single attribute name when the framework already supports aliases.

## 2.3 Using framework helper objects instead of raw storage

Some framework data is already wrapped in service objects.

Examples:

- `app.state.user_rag`
- `app.state.ai_jobs`
- `app.state.collab_db`

Use the object API first when available.

Example:

```python
reg = getattr(app.state, "ai_jobs", None)
if reg is not None:
    jobs = reg.snapshot()
```

Another example:

```python
db = getattr(app.state, "collab_db", None)
if db is not None:
    rows = db.list_messages(pid=pid, sid=sid, limit=20)
```

This is preferable to manually opening the same storage backend yourself.

## 3. Accessing the Collaboration Database

The main framework database for users, projects, sessions, messages, and related chat metadata is the collaboration database used by `collab_chat`.

The implementation lives in:

- [plugins/gui_helpers/collab_chat/routes.py](C:\Users\Tee\projects\llmloader2\plugins\gui_helpers\collab_chat\routes.py)

Its DB wrapper class is `_DB`, and it manages:

- users
- tokens
- projects
- project members
- sessions
- session members
- messages
- join requests
- GUI preferences and router preferences
- prompt/config tables

The class also exposes a `path` property pointing at the SQLite file.

## 3.1 Preferred way: use `app.state.collab_db`

If the plugin needs project/session/chat data, it should prefer the framework DB object rather than opening SQLite manually.

Typical pattern:

```python
db = getattr(app.state, "collab_db", None)
if db is None:
    raise HTTPException(status_code=503, detail="collab_db unavailable")

rows = db.list_messages(pid=pid, sid=sid, after_msg_id=None, since_ts=None, limit=20)
```

Use this when:

- the plugin needs chat/session/project/user data
- the needed method already exists on `collab_db`
- the plugin should respect framework behavior and schema ownership

Examples in repo:

- workflow sandbox helpers read messages through `db.list_messages(...)`
- workflow sandbox helpers issue tokens through `db.issue_token(...)`
- workflow sandbox helpers create hidden sessions through `db.ensure_session(...)`

## 3.2 Direct SQLite access to the framework DB

Some code in the repo opens SQLite directly against the collaboration DB path.

Representative example:

- [plugins/gui_helpers/agent_flow/skills/workflow/_workflow_store.py](C:\Users\Tee\projects\llmloader2\plugins\gui_helpers\agent_flow\skills\workflow\_workflow_store.py)

That module:

- discovers the framework DB path from `app.state.collab_db.path` if available
- falls back to `data_dir/collab_chat.db`
- opens a raw SQLite connection
- creates its own workflow-specific tables in the same DB

Example pattern:

```python
collab_db = getattr(getattr(app, "state", None), "collab_db", None)
db_path = getattr(collab_db, "path", None)
if not db_path:
    db_path = str((data_dir / "collab_chat.db").resolve())

con = sqlite3.connect(db_path, check_same_thread=False)
con.row_factory = sqlite3.Row
```

Use direct SQLite only when:

- the plugin owns its own schema
- the plugin needs SQL-level control
- there is no existing framework API for the required operation

## Caution

If you use raw SQLite against the framework DB:

- do not modify framework-owned tables casually
- keep your plugin’s schema isolated with plugin-specific table names
- initialize your schema explicitly
- avoid bypassing framework permission checks for user-facing operations

This pattern is justified in `agent_flow` because it owns workflow-specific storage with dedicated tables.

## 4. Persistent Data Storage Patterns Already Used in the Repo

There are several real persistence patterns in `llmloader2`.

## 4.1 Plugin-owned JSON files under `data_dir`

This is the most common simple persistence pattern.

Examples:

- `theme_demo/default_theme.json`
- `pin_messages/sessions/<sid>/pins.json`
- `skills_settings/settings.json`

Typical path resolution pattern:

```python
def _data_root(app):
    cand = getattr(app.state, "data_dir", None) or getattr(app.state, "workdir", None) or None
    if isinstance(cand, str) and cand.strip():
        root = cand
    else:
        root = os.path.abspath("./data")
    os.makedirs(root, exist_ok=True)
    return root
```

Use this when:

- state is plugin-specific
- data shape is naturally document-style
- plugin does not need relational queries
- human-readable storage is useful

Good fit for:

- plugin settings
- saved UI state
- per-session plugin metadata
- small structured records

## Example: simple JSON settings file

```python
import json
import os

def _settings_path(app):
    base = os.path.join(_data_root(app), "gui_helpers", "example_plugin")
    os.makedirs(base, exist_ok=True)
    return os.path.join(base, "settings.json")

def load_settings(app):
    path = _settings_path(app)
    if not os.path.isfile(path):
        return {"version": 1, "settings": {}}
    with open(path, "r", encoding="utf-8") as fh:
        return json.load(fh)

def save_settings(app, payload):
    path = _settings_path(app)
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(payload, fh, indent=2, ensure_ascii=True)
```

## 4.2 Atomic JSON writes

Some plugins use atomic writes for safer persistence.

Representative example:

- [plugins/gui_helpers/skills_settings/store.py](C:\Users\Tee\projects\llmloader2\plugins\gui_helpers\skills_settings\store.py)

Pattern:

- write to a temporary file in the same directory
- replace the target file with `os.replace(...)`

Example:

```python
import json
import os
import tempfile

def atomic_save_json(path, data):
    fd, tmp = tempfile.mkstemp(prefix=path.name + ".", suffix=".tmp", dir=str(path.parent))
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            json.dump(data, fh, ensure_ascii=True, indent=2, sort_keys=True)
        os.replace(tmp, path)
    finally:
        try:
            if os.path.exists(tmp):
                os.unlink(tmp)
        except Exception:
            pass
```

Use this when:

- the file is important configuration
- a partially written file would be harmful
- the plugin may be updated while the service is running

Good fit for:

- settings
- indexes
- workflow descriptors
- shared defaults

## 4.3 Binary asset storage on disk

Some plugins need to persist file assets, not just JSON.

Representative example:

- [plugins/gui_helpers/theme_demo/routes.py](C:\Users\Tee\projects\llmloader2\plugins\gui_helpers\theme_demo\routes.py)

Pattern:

- derive an assets directory under `data_dir`
- convert incoming data URLs to files
- store them under a content-based filename
- expose them through a helper route

Use this when:

- the plugin stores uploaded images or generated files
- the plugin needs stable file URLs
- the data is too large or unsuitable for JSON

Good fit for:

- theme background images
- generated documents
- cached plugin assets
- uploads owned by the plugin

## 4.4 Per-session JSON documents

Some plugin state is naturally scoped to a chat session but does not need to live in the framework DB.

Representative example:

- [plugins/gui_helpers/pin_messages/routes.py](C:\Users\Tee\projects\llmloader2\plugins\gui_helpers\pin_messages\routes.py)

Pattern:

- store under `data/sessions/<sid>/...`
- validate the session id
- keep one JSON file per session

Use this when:

- the data is plugin-owned
- it is scoped to a session
- you do not need SQL querying across sessions

Good fit for:

- pinned message layouts
- session-local plugin boards
- per-session notes or annotations

## 4.5 Plugin-owned tables inside SQLite

This is the heavy-weight persistence pattern.

Representative example:

- [plugins/gui_helpers/agent_flow/skills/workflow/_workflow_store.py](C:\Users\Tee\projects\llmloader2\plugins\gui_helpers\agent_flow\skills\workflow\_workflow_store.py)

Pattern:

- open SQLite
- initialize plugin tables with `CREATE TABLE IF NOT EXISTS`
- store structured records with indexes

Use this when:

- the plugin needs filtering, indexing, history, or relational access
- records need to be queried by multiple dimensions
- JSON files would become too awkward

Good fit for:

- workflow registries
- run history
- validation/update logs
- searchable plugin records

## Example: plugin-owned SQLite table

```python
import sqlite3

def init_schema(con):
    con.execute(
        """
        CREATE TABLE IF NOT EXISTS example_records (
            record_id TEXT PRIMARY KEY,
            plugin_scope TEXT NOT NULL,
            title TEXT NOT NULL,
            payload_json TEXT NOT NULL,
            created_ts INTEGER NOT NULL,
            updated_ts INTEGER NOT NULL
        )
        """
    )
    con.execute("CREATE INDEX IF NOT EXISTS idx_example_records_scope ON example_records(plugin_scope)")
    con.commit()
```

## 4.6 Using framework-managed persistence services

Not all persistence should be implemented by the plugin itself.

Examples:

- `user_rag` manages its own document stores
- `collab_db` manages projects, sessions, and messages
- `ai_jobs` manages job lifecycle state

Use this when:

- the framework already owns the data domain
- the plugin only needs to consume or augment it

Avoid duplicating this data into parallel plugin storage unless there is a clear reason.

## 5. Common Data Domains and the Right Storage Choice

## Project/session/user/chat data

Preferred storage/access:

- `app.state.collab_db`

Why:

- framework-owned
- relational
- already permission-sensitive
- used by collaboration, auth, and session flows

## Plugin configuration

Preferred storage/access:

- plugin-owned JSON under `data_dir`
- atomic writes preferred

Why:

- small document data
- easy to inspect
- low complexity

## Plugin UI defaults and media assets

Preferred storage/access:

- JSON plus files under plugin-specific asset directories

Why:

- mixes structured config and binary assets

## Searchable plugin records and history

Preferred storage/access:

- SQLite with plugin-owned tables

Why:

- indexing and query needs

## RAG or document knowledge

Preferred storage/access:

- `app.state.user_rag` or related framework service

Why:

- the framework already owns ingestion, indexing, and retrieval

## Ephemeral runtime-only state

Preferred storage/access:

- `app.state`

Why:

- no need to persist
- should reflect current service state only

## 6. Recommended Access Patterns

## Pattern A: read framework settings

```python
def _get_settings(app):
    st = getattr(app.state, "settings", None)
    if callable(st):
        return dict(st() or {})
    return dict(st or {})
```

Use this when the plugin needs service settings but does not own them.

## Pattern B: read framework DB through `collab_db`

```python
db = getattr(app.state, "collab_db", None)
if db is None:
    raise HTTPException(status_code=503, detail="collab_db unavailable")

rows = db.list_messages(pid=pid, sid=sid, limit=20, order_desc=True)
```

Use this when reading chat/session/project data.

## Pattern C: plugin-owned settings document

```python
from pathlib import Path
import json
import time

def settings_path(app) -> Path:
    root = Path(str(getattr(app.state, "data_dir", None) or "./data")).resolve()
    path = root / "gui_helpers" / "example_plugin"
    path.mkdir(parents=True, exist_ok=True)
    return path / "settings.json"

def save_doc(app, data):
    payload = dict(data or {})
    payload["updated_ts"] = int(time.time())
    settings_path(app).write_text(json.dumps(payload, indent=2), encoding="utf-8")
```

Use this when the state belongs entirely to the plugin.

## Pattern D: plugin-owned SQLite schema

```python
import sqlite3
from pathlib import Path

def open_plugin_db(app):
    root = Path(str(getattr(app.state, "data_dir", None) or "./data")).resolve()
    path = root / "gui_helpers" / "example_plugin" / "plugin.db"
    path.parent.mkdir(parents=True, exist_ok=True)
    con = sqlite3.connect(str(path), check_same_thread=False)
    con.row_factory = sqlite3.Row
    return con
```

Use this when plugin data should not be mixed into framework DB tables and the plugin needs SQL features.

## Pattern E: plugin tables inside framework DB

```python
collab_db = getattr(app.state, "collab_db", None)
db_path = getattr(collab_db, "path", None)
con = sqlite3.connect(db_path, check_same_thread=False)
```

Use this only when:

- there is a strong integration reason
- the plugin owns isolated tables
- the plugin is intentionally co-locating with framework DB storage

## 7. Security and Stability Notes

## Validate identifiers used in file paths

If you store per-session or per-project files:

- sanitize or validate `sid`, `pid`, or plugin keys
- do not directly trust path fragments from request payloads

The `pin_messages` plugin is a good example of validating `sid`.

## Prefer framework permission checks on user-facing routes

If a plugin exposes HTTP routes:

- use `require_gui_plugin_enabled(...)`
- use permissions manager checks when needed

Do not assume that reading or writing plugin data is automatically safe just because it is plugin-owned.

## Avoid writing directly into framework-owned tables unless necessary

Project/session/message/user tables are framework territory.

If your plugin needs them:

- read through `collab_db`
- write through `collab_db` methods where possible

If you must add plugin-specific SQLite tables, namespace them clearly.

## Prefer atomic writes for important config

If the file matters to startup, behavior, or auth-like configuration:

- use temp-file plus replace

This is safer than direct overwrite.

## 8. Recommended Decision Guide

If you are deciding how a plugin should persist or access data, use this order:

1. Does the framework already expose the data on `app.state`?
2. Does a framework service object already own the domain, such as `collab_db`, `user_rag`, or `ai_jobs`?
3. Is the plugin data small and document-shaped? Use JSON under `data_dir`.
4. Is the plugin storing assets or uploads? Use plugin-specific directories under `data_dir`.
5. Does the plugin need indexed/queryable history? Use SQLite with plugin-owned tables.
6. Only then consider direct access to the framework SQLite file.

## 9. Practical Recommendations for New Plugins

- For simple plugin settings, copy the `skills_settings` persistence style.
- For session-local plugin state, copy the `pin_messages` style.
- For shared theme-like config plus assets, copy the `theme_demo` style.
- For framework chat/project/session data, use `app.state.collab_db`.
- For workflow/history registries, use plugin-owned SQLite tables like `agent_flow` does.
- For knowledge/document retrieval, use `app.state.user_rag` instead of inventing a second store.

## Conclusion

The framework already supports several persistence tiers:

- in-memory runtime state through `app.state`
- framework-owned service objects like `collab_db`, `user_rag`, and `ai_jobs`
- plugin-owned JSON documents
- plugin-owned asset directories
- plugin-owned SQLite schemas
- limited direct access to the framework SQLite database when justified

The safest default is to stay as high-level as possible. Read from framework services when the framework owns the data, and only introduce lower-level storage when the plugin truly owns that data domain.