feat: LiteLLM multi-backends + jean-proxy avec WoL
Architecture (voir docs/ARCHITECTURE-LITELLM.md):
open-webui -> LiteLLM:4000
|- qwen3.6-27b -> jean-proxy:3012 -> Jean (.11:8080)
|- llama3.1-8b -> wol-proxy:3011 -> Ollama (.11:11434)
Nouveaux fichiers:
- wol-proxy/ (versionne l'app.py de prod deja en place)
- jean-proxy/ (clone du wol-proxy avec HEALTH_PATH=/health + retry post-WoL
pour services qui mettent du temps a demarrer apres WoL, ex: Jean qui
charge 14 Go en VRAM)
- litellm/config.yaml (routing par nom de modele, drop_params=true)
- docker-compose.litellm.yml (override ADDITIF: ajoute uniquement jean-proxy
et litellm, ne redemarre PAS open-webui/wol-proxy/hermes)
Teste en prod sur .27: Jean + Ollama repondent via LiteLLM:4000 avec
path-through de chat_template_kwargs.enable_thinking (raisonnement Qwen3.6
desactivable par requete). Zero impact hermes-webui/wol-proxy.
Basculer open-webui reste une etape manuelle (step 3 de la doc).
This commit is contained in:
@@ -0,0 +1,57 @@
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# LiteLLM Proxy — Routing multi-backends pour Effi
|
||||
# Expose un endpoint OpenAI-compatible sur :4000
|
||||
#
|
||||
# Backends :
|
||||
# 1. Jean (llama.cpp/ROCm sur PC gaming .11:8080) → modele "qwen3.6-27b"
|
||||
# 2. Ollama (PC gaming .11:11434 via wol-proxy :3011) → modele "llama3.1-8b" (+ autres)
|
||||
#
|
||||
# Les deux backends sont sur le meme PC gaming (192.168.1.11)
|
||||
# qui s eteint la nuit. Les proxies (wol-proxy + jean-proxy) declenchent
|
||||
# le WOL si besoin avec la meme adresse MAC cible.
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
model_list:
|
||||
# ═════════════════════════════════════════════════════════
|
||||
# Backend 1 — Jean (llama.cpp/v1 OpenAI-compatible)
|
||||
# ═════════════════════════════════════════════════════════
|
||||
- model_name: qwen3.6-27b
|
||||
litellm_params:
|
||||
model: openai/qwen3.6-27b
|
||||
api_base: http://localhost:3012/v1
|
||||
api_key: "null"
|
||||
timeout: 600
|
||||
# Alias court pour open-webui (sans version)
|
||||
- model_name: qwen3.6
|
||||
litellm_params:
|
||||
model: openai/qwen3.6-27b
|
||||
api_base: http://localhost:3012/v1
|
||||
api_key: "null"
|
||||
timeout: 600
|
||||
|
||||
# ═════════════════════════════════════════════════════════
|
||||
# Backend 2 — Ollama (via wol-proxy :3011 + WOL auto)
|
||||
# ═════════════════════════════════════════════════════════
|
||||
- model_name: llama3.1-8b
|
||||
litellm_params:
|
||||
model: ollama/llama3.1:8b
|
||||
api_base: http://localhost:3011
|
||||
timeout: 600
|
||||
|
||||
- model_name: llama3.1
|
||||
litellm_params:
|
||||
model: ollama/llama3.1:8b
|
||||
api_base: http://localhost:3011
|
||||
timeout: 600
|
||||
|
||||
# ═════════════════════════════════════════════════════════
|
||||
# Ollama "fallback" auto : tout modele ollama/* est route vers wol-proxy
|
||||
# (laisse open-webui lister les autres modeles ollama si besoin)
|
||||
# ═════════════════════════════════════════════════════════
|
||||
|
||||
litellm_settings:
|
||||
drop_params: true
|
||||
telemetry: false
|
||||
|
||||
general_settings:
|
||||
master_key: sk-effi-litellm-master
|
||||
Reference in New Issue
Block a user