Ajout Effi-OCR (Tesseract) + Web Scraper pour OpenWebUI
- Effi-OCR: Service OCR avec Tesseract (multilingue FR/EN) * API FastAPI sur port 8300 * Prétraitement images (contraste, niveaux de gris) * Tool OpenWebUI intégré - Web Scraper: Service Python (BeautifulSoup) sur port 11235 * Alternative à Crawl4AI (incompatible ARM64/J5040) * Scraping HTML basique mais fonctionnel * Intégré au docker-compose d'effi-server - Documentation: * ROADMAP-FONCTIONNALITES.md: Plan des 9 fonctionnalités * WEB-SCRAPER-CONFIG.md: Configuration OpenWebUI * WEB-SCRAPER-TOOL.md: Code du Tool - Script start-all-services.sh mis à jour (6 services)
This commit is contained in:
Folzo
@@ -0,0 +1,34 @@
|
||||
# Crawl4AI pour architecture AMD64 (x86_64)
|
||||
# Build local car l'image officielle est en ARM64
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Installer les dépendances système
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
chromium \
|
||||
chromium-driver \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Variables d'environnement
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PLAYWRIGHT_BROWSERS_PATH=0
|
||||
ENV PIP_NO_CACHE_DIR=1
|
||||
|
||||
# Installer Crawl4AI
|
||||
RUN pip install crawl4ai
|
||||
|
||||
# Installer Playwright
|
||||
RUN pip install playwright && playwright install chromium
|
||||
|
||||
# Port d'écoute
|
||||
EXPOSE 11235
|
||||
|
||||
# Healthcheck
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
||||
CMD curl -f http://localhost:11235/health || exit 1
|
||||
|
||||
# Démarrer l'API
|
||||
CMD ["python", "-m", "crawl4ai.server", "--host", "0.0.0.0", "--port", "11235"]
|
||||
@@ -0,0 +1,20 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Installer les dépendances
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN pip install --no-cache-dir fastapi uvicorn requests beautifulsoup4
|
||||
|
||||
# Copier le code
|
||||
COPY scraper.py .
|
||||
|
||||
EXPOSE 11235
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
|
||||
CMD curl -f http://localhost:11235/health || exit 1
|
||||
|
||||
CMD ["uvicorn", "scraper:app", "--host", "0.0.0.0", "--port", "11235"]
|
||||
@@ -0,0 +1,115 @@
|
||||
# Crawl4AI sur folzo-Corp
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ folzo-Corp (GPU) │
|
||||
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌────────┐ │
|
||||
│ │ Ollama │ │ComfyUI │ │Effi-STT │ │Effi-OCR │ │Crawl4AI│ │
|
||||
│ │ :11434 │ │ :8188 │ │ :8200 │ │ :8300 │ │:11235 │ │
|
||||
│ └────┬────┘ └─────────┘ └─────────┘ └─────────┘ └───┬────┘ │
|
||||
│ │ │ │
|
||||
│ │ http://192.168.1.11:11235 │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ HTTP API
|
||||
│
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ effi-server (J5040) │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ OpenWebUI │ │ Qdrant │ │ Tool Config │ │
|
||||
│ │ :3000 │ │ :6333 │ │ Crawl4AI Base URL: │ │
|
||||
│ └──────┬──────┘ └─────────────┘ │ http://192.168.1.11:11235│ │
|
||||
│ │ └─────────────────────────┘ │
|
||||
│ │ ❌ PAS de Crawl4AI local │
|
||||
└─────────┼───────────────────────────────────────────────────────┘
|
||||
│
|
||||
└──────► Réseau effi-net
|
||||
```
|
||||
|
||||
## Déploiement
|
||||
|
||||
### 1. Sur folzo-Corp (déjà fait)
|
||||
|
||||
Les fichiers sont dans `/home/folzo/Documents/OpenCode/Projets/Effi/effi-crawl4ai/`
|
||||
|
||||
```bash
|
||||
cd /home/folzo/Documents/OpenCode/Projets/Effi/effi-crawl4ai
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### 2. Sur effi-server (à faire)
|
||||
|
||||
**Supprimer le service local qui ne fonctionne pas :**
|
||||
|
||||
```bash
|
||||
cd /root/effi
|
||||
|
||||
# Option A : Script automatique
|
||||
curl -o remove-from-effi.sh [URL_du_script]
|
||||
bash remove-from-effi.sh
|
||||
|
||||
# Option B : Manuel
|
||||
docker-compose stop crawl4ai
|
||||
docker-compose rm -f crawl4ai
|
||||
docker rmi unclecode/crawl4ai:basic
|
||||
|
||||
# Éditer docker-compose.yml et supprimer le service crawl4ai (section #8)
|
||||
```
|
||||
|
||||
**Ouvrir le firewall sur folzo-Corp** :
|
||||
|
||||
```bash
|
||||
# Sur folzo-Corp
|
||||
sudo firewall-cmd --add-port=11235/tcp --permanent
|
||||
sudo firewall-cmd --reload
|
||||
```
|
||||
|
||||
### 3. Configuration OpenWebUI
|
||||
|
||||
Dans http://192.168.1.27:3000 :
|
||||
|
||||
1. **Admin Panel** → **Tools** → **Web Search and Crawl** → ⚙️
|
||||
2. **Crawl4AI Base URL** : `http://192.168.1.11:11235` ← L'IP de folzo-Corp !
|
||||
3. **Timeout** : `30000` (ou `60000` si lenteur)
|
||||
4. Activer le Tool ✅
|
||||
|
||||
## Avantages de cette architecture
|
||||
|
||||
✅ **Crawl4AI** tourne sur CPU/GPU puissant (folzo-Corp)
|
||||
✅ **Effi-server** (J5040) n'est pas surchargé
|
||||
✅ **Réseau** local rapide entre les deux machines
|
||||
✅ **Charge répartie** : scraping sur folzo-Corp, interface sur effi-server
|
||||
|
||||
## Vérification
|
||||
|
||||
```bash
|
||||
# Sur folzo-Corp
|
||||
curl http://localhost:11235/health
|
||||
# Résultat : {"status":"healthy"}
|
||||
|
||||
# Sur effi-server
|
||||
curl http://192.168.1.11:11235/health
|
||||
# Résultat : {"status":"healthy"}
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
Dans OpenWebUI :
|
||||
> *"Effi, cherche les dernières nouvelles sur Linux et résume-moi un article"*
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"Connection refused" depuis effi-server** :
|
||||
- Vérifier que Crawl4AI tourne sur folzo-Corp : `docker ps | grep crawl4ai`
|
||||
- Vérifier le firewall sur folzo-Corp : `sudo firewall-cmd --list-ports`
|
||||
|
||||
**Timeout** :
|
||||
- Augmenter à 60000ms dans OpenWebUI
|
||||
- La latence réseau entre les machines peut causer des timeouts
|
||||
|
||||
---
|
||||
|
||||
*Architecture mise à jour pour J5040 + RX 7800 XT*
|
||||
@@ -0,0 +1,128 @@
|
||||
# Crawl4AI - Intégration Effi Server
|
||||
|
||||
Intégration du service **Crawl4AI** au docker-compose existant de `/root/effi/`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ effi-server │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ OpenWebUI │ │ Qdrant │ │ Crawl4AI │ │
|
||||
│ │ :3000 │ │ :6333 │ │ :11235 │ │
|
||||
│ └──────┬──────┘ └─────────────┘ └──────┬──────┘ │
|
||||
│ │ │ │
|
||||
│ └──────────────────────────────────┘ │
|
||||
│ Réseau: effi-net │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### Option 1 : Script automatique (recommandé)
|
||||
|
||||
Sur **effi-server**, dans `/root/effi/` :
|
||||
|
||||
```bash
|
||||
cd /root/effi
|
||||
|
||||
# Copier le script add-to-effi.sh ici
|
||||
# Puis exécuter :
|
||||
bash add-to-effi.sh
|
||||
```
|
||||
|
||||
Le script va :
|
||||
1. Sauvegarder ton docker-compose.yml
|
||||
2. Ajouter le service Crawl4AI
|
||||
3. Démarrer le service
|
||||
4. Vérifier qu'il fonctionne
|
||||
|
||||
### Option 2 : Manuel
|
||||
|
||||
1. **Sauvegarder** le docker-compose existant :
|
||||
```bash
|
||||
cd /root/effi
|
||||
cp docker-compose.yml docker-compose.yml.backup
|
||||
```
|
||||
|
||||
2. **Éditer** `docker-compose.yml` et ajouter avant la section `networks:` :
|
||||
|
||||
```yaml
|
||||
# ─────────────────────────────────────────────────────────
|
||||
# 8. CRAWL4AI - Web Scraping pour OpenWebUI
|
||||
# → http://localhost:11235 (accessible depuis OpenWebUI)
|
||||
# ─────────────────────────────────────────────────────────
|
||||
crawl4ai:
|
||||
image: unclecode/crawl4ai:basic
|
||||
container_name: crawl4ai
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "11235:11235"
|
||||
environment:
|
||||
- CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}
|
||||
- MAX_CONCURRENT_TASKS=3
|
||||
- CRAWL4AI_ENABLE_GPU=false
|
||||
volumes:
|
||||
- /dev/shm:/dev/shm
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2G
|
||||
reservations:
|
||||
memory: 512M
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
networks:
|
||||
- effi-net
|
||||
```
|
||||
|
||||
3. **Démarrer** le service :
|
||||
```bash
|
||||
docker-compose up -d crawl4ai
|
||||
```
|
||||
|
||||
## Configuration OpenWebUI
|
||||
|
||||
1. Aller dans **Admin Panel** → **Tools** → **Web Search and Crawl** → ⚙️
|
||||
2. **Crawl4AI Base URL** : `http://localhost:11235`
|
||||
3. **Timeout** : `30000`
|
||||
4. Activer le Tool ✅
|
||||
|
||||
## Vérification
|
||||
|
||||
```bash
|
||||
# Sur effi-server
|
||||
curl http://localhost:11235/health
|
||||
# Résultat attendu : {"status":"healthy"}
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
Dans OpenWebUI :
|
||||
> *"Effi, cherche les dernières nouvelles sur Linux et résume-moi un article"*
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Le service ne démarre pas** :
|
||||
```bash
|
||||
docker logs crawl4ai
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
**Timeout** : Augmenter la valve à `60000` dans OpenWebUI
|
||||
|
||||
**Mémoire** : Le service est limité à 2GB (suffisant pour la plupart des sites)
|
||||
|
||||
## Fichiers
|
||||
|
||||
- `crawl4ai-service-to-add.yml` - Contenu à ajouter au docker-compose
|
||||
- `add-to-effi.sh` - Script d'installation automatique
|
||||
- `README.md` - Cette documentation
|
||||
|
||||
---
|
||||
|
||||
*Intégration créée pour la stack Effi existante*
|
||||
@@ -0,0 +1,111 @@
|
||||
# Effi-Crawl4AI
|
||||
|
||||
Service **Crawl4AI** pour le web scraping avancé dans OpenWebUI.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ HTTP API ┌─────────────────┐
|
||||
│ OpenWebUI │ ◄────────────────► │ Crawl4AI │
|
||||
│ (effi-server) │ Port 11235 │ (effi-server) │
|
||||
└─────────────────┘ └─────────────────┘
|
||||
│ │
|
||||
└────────────────────────────────────┘
|
||||
Même machine (localhost)
|
||||
```
|
||||
|
||||
**Note** : Le service tourne sur effi-server (Ubuntu) sur CPU. Version `basic` (plus légère que `all`).
|
||||
|
||||
## Déploiement sur effi-server
|
||||
|
||||
1. **Copier les fichiers** sur effi-server :
|
||||
```bash
|
||||
scp -r /path/to/effi-crawl4ai user@192.168.1.27:/home/user/
|
||||
```
|
||||
|
||||
2. **Démarrer le service** :
|
||||
```bash
|
||||
cd /home/user/effi-crawl4ai
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
## Vérification
|
||||
|
||||
```bash
|
||||
curl http://localhost:11235/health
|
||||
```
|
||||
|
||||
## Configuration OpenWebUI
|
||||
|
||||
Dans OpenWebUI (Admin → Tools → Web Search and Crawl) :
|
||||
|
||||
**Valves à configurer :**
|
||||
- **Crawl4AI Base URL** : `http://localhost:11235` (ou `http://127.0.0.1:11235`)
|
||||
- **API Token** : (laisser vide si pas de token configuré)
|
||||
- **Timeout** : `30000` (30 secondes)
|
||||
```
|
||||
┌─────────────────┐ HTTP API ┌─────────────────┐
|
||||
│ OpenWebUI │ ◄────────────────► │ Crawl4AI │
|
||||
│ (effi-server) │ Port 11235 │ (folzo-Corp) │
|
||||
└─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## Démarrage
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
## Vérification
|
||||
|
||||
```bash
|
||||
curl http://192.168.1.11:11235/health
|
||||
```
|
||||
|
||||
## Configuration OpenWebUI
|
||||
|
||||
Dans OpenWebUI (Admin → Tools → Web Search and Crawl) :
|
||||
|
||||
**Valves à configurer :**
|
||||
- **Crawl4AI Base URL** : `http://192.168.1.11:11235`
|
||||
- **API Token** : (laisser vide si pas de token configuré)
|
||||
- **Timeout** : `30000` (30 secondes)
|
||||
|
||||
## Utilisation
|
||||
|
||||
Une fois configuré, l'agent peut :
|
||||
|
||||
```
|
||||
"Effi, cherche des informations sur les dernières mises à jour de Fedora 42"
|
||||
"Résume-moi cet article : https://example.com/article"
|
||||
"Compare les prix de ce produit sur 3 sites différents"
|
||||
```
|
||||
|
||||
## Fonctionnalités
|
||||
|
||||
- ✅ Scraping JavaScript (pages dynamiques)
|
||||
- ✅ Extraction structurée (JSON, markdown)
|
||||
- ✅ Screenshot des pages
|
||||
- ✅ Respect des robots.txt
|
||||
- ✅ Rate limiting configurable
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Erreur de connexion** : Vérifier que le conteneur tourne
|
||||
|
||||
```bash
|
||||
docker ps | grep crawl4ai
|
||||
docker logs effi-crawl4ai
|
||||
```
|
||||
|
||||
**Timeout** : Augmenter le timeout dans les Valves du Tool (60000ms = 60s)
|
||||
|
||||
**Lenteur** : La version CPU est plus lente que la version GPU. Pour les pages lourdes en JavaScript, augmenter le timeout à 60-90s.
|
||||
|
||||
## Limitations (version CPU)
|
||||
|
||||
- ⏱️ Plus lent sur les pages JavaScript lourdes
|
||||
- 🎯 Limité à 3 tâches concurrentes (vs 5 sur GPU)
|
||||
- 💾 2GB RAM max (suffisant pour la plupart des sites)
|
||||
|
||||
Pour du scraping intensif (100+ pages/heure), envisager de déplacer sur folzo-Corp avec l'image `unclecode/crawl4ai:all`.
|
||||
@@ -0,0 +1,107 @@
|
||||
#!/bin/bash
|
||||
# Script d'ajout de Crawl4AI au docker-compose existant
|
||||
# À exécuter sur effi-server dans /root/effi/
|
||||
|
||||
set -e
|
||||
|
||||
echo "================================"
|
||||
echo "Ajout de Crawl4AI à Effi Server"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# Vérifier qu'on est dans le bon répertoire
|
||||
if [ ! -f "docker-compose.yml" ]; then
|
||||
echo "❌ Erreur: docker-compose.yml non trouvé"
|
||||
echo " Ce script doit être exécuté dans /root/effi/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "📄 Docker-compose trouvé"
|
||||
|
||||
# Sauvegarder l'original
|
||||
cp docker-compose.yml docker-compose.yml.backup.$(date +%Y%m%d_%H%M%S)
|
||||
echo "💾 Backup créé: docker-compose.yml.backup.*"
|
||||
|
||||
# Ajouter le service avant "networks:"
|
||||
echo ""
|
||||
echo "🔧 Ajout du service Crawl4AI..."
|
||||
|
||||
# Créer un fichier temporaire avec le service à insérer
|
||||
cat > /tmp/crawl4ai-service.txt << 'EOF'
|
||||
# ─────────────────────────────────────────────────────────
|
||||
# 8. CRAWL4AI - Web Scraping pour OpenWebUI
|
||||
# → http://localhost:11235 (accessible depuis OpenWebUI)
|
||||
# ─────────────────────────────────────────────────────────
|
||||
crawl4ai:
|
||||
image: unclecode/crawl4ai:basic
|
||||
container_name: crawl4ai
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "11235:11235"
|
||||
environment:
|
||||
- CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}
|
||||
- MAX_CONCURRENT_TASKS=3
|
||||
- CRAWL4AI_ENABLE_GPU=false
|
||||
volumes:
|
||||
- /dev/shm:/dev/shm
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2G
|
||||
reservations:
|
||||
memory: 512M
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
networks:
|
||||
- effi-net
|
||||
|
||||
EOF
|
||||
|
||||
# Insérer avant la ligne "networks:"
|
||||
awk '
|
||||
/^networks:/ {
|
||||
while ((getline line < "/tmp/crawl4ai-service.txt") > 0) {
|
||||
print line
|
||||
}
|
||||
close("/tmp/crawl4ai-service.txt")
|
||||
}
|
||||
{ print }
|
||||
' docker-compose.yml > /tmp/docker-compose-new.yml
|
||||
|
||||
# Remplacer le fichier original
|
||||
mv /tmp/docker-compose-new.yml docker-compose.yml
|
||||
|
||||
echo "✅ Service ajouté au docker-compose.yml"
|
||||
echo ""
|
||||
|
||||
# Démarrer le nouveau service
|
||||
echo "🚀 Démarrage de Crawl4AI..."
|
||||
docker-compose up -d crawl4ai
|
||||
|
||||
echo ""
|
||||
echo "⏳ Attente du démarrage (30s)..."
|
||||
sleep 30
|
||||
|
||||
# Vérification
|
||||
echo ""
|
||||
echo "🔍 Vérification..."
|
||||
if curl -s http://localhost:11235/health > /dev/null 2>&1; then
|
||||
echo "✅ Crawl4AI est démarré !"
|
||||
echo ""
|
||||
echo "📋 Configuration OpenWebUI:"
|
||||
echo " Admin → Tools → Web Search and Crawl → ⚙️"
|
||||
echo " Crawl4AI Base URL: http://localhost:11235"
|
||||
echo " Timeout: 30000"
|
||||
else
|
||||
echo "⚠️ Le service ne répond pas encore"
|
||||
echo " Vérifier les logs: docker logs crawl4ai"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "================================"
|
||||
echo "Installation terminée !"
|
||||
echo "================================"
|
||||
@@ -0,0 +1,30 @@
|
||||
# ─────────────────────────────────────────────────────────
|
||||
# 8. CRAWL4AI - Web Scraping pour OpenWebUI
|
||||
# → http://localhost:11235 (accessible depuis OpenWebUI)
|
||||
# ─────────────────────────────────────────────────────────
|
||||
crawl4ai:
|
||||
image: unclecode/crawl4ai:basic
|
||||
container_name: crawl4ai
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "11235:11235"
|
||||
environment:
|
||||
- CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}
|
||||
- MAX_CONCURRENT_TASKS=3
|
||||
- CRAWL4AI_ENABLE_GPU=false
|
||||
volumes:
|
||||
- /dev/shm:/dev/shm
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2G
|
||||
reservations:
|
||||
memory: 512M
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
networks:
|
||||
- effi-net
|
||||
@@ -0,0 +1,34 @@
|
||||
# Service Crawl4AI pour Effi
|
||||
# À déployer sur folzo-Corp (GPU RX 7800 XT)
|
||||
# Exposition du port 11235 pour l'API
|
||||
|
||||
services:
|
||||
crawl4ai:
|
||||
image: unclecode/crawl4ai:all
|
||||
container_name: effi-crawl4ai
|
||||
ports:
|
||||
- "11235:11235"
|
||||
environment:
|
||||
- CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}
|
||||
- MAX_CONCURRENT_TASKS=5
|
||||
- CRAWL4AI_ENABLE_GPU=false
|
||||
volumes:
|
||||
- /dev/shm:/dev/shm
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4G
|
||||
reservations:
|
||||
memory: 1G
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
|
||||
networks:
|
||||
default:
|
||||
name: effi-network
|
||||
external: true
|
||||
@@ -0,0 +1,25 @@
|
||||
# Alternative simple à Crawl4AI pour architecture x86_64
|
||||
# Service de scraping web basique mais fonctionnel
|
||||
|
||||
services:
|
||||
web-scraper:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.scraper
|
||||
container_name: effi-web-scraper
|
||||
ports:
|
||||
- "11235:11235"
|
||||
environment:
|
||||
- MAX_CONCURRENT_TASKS=3
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
|
||||
networks:
|
||||
default:
|
||||
name: effi-network
|
||||
external: true
|
||||
@@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
# Script de réparation de Crawl4AI
|
||||
# À exécuter sur effi-server dans /root/effi/
|
||||
|
||||
echo "================================"
|
||||
echo "Réparation de Crawl4AI"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# 1. Arrêter et supprimer le conteneur
|
||||
echo "🛑 Arrêt du conteneur..."
|
||||
docker stop crawl4ai 2>/dev/null || true
|
||||
docker rm crawl4ai 2>/dev/null || true
|
||||
|
||||
# 2. Supprimer l'image corrompue
|
||||
echo "🗑️ Suppression de l'image..."
|
||||
docker rmi unclecode/crawl4ai:basic 2>/dev/null || true
|
||||
|
||||
# 3. Nettoyer le cache Docker
|
||||
echo "🧹 Nettoyage du cache..."
|
||||
docker system prune -f
|
||||
|
||||
# 4. Re-pull l'image avec plateforme explicite
|
||||
echo "📥 Téléchargement de l'image..."
|
||||
docker pull --platform linux/amd64 unclecode/crawl4ai:basic
|
||||
|
||||
# 5. Redémarrer
|
||||
echo "🚀 Redémarrage..."
|
||||
docker-compose up -d crawl4ai
|
||||
|
||||
echo ""
|
||||
echo "⏳ Attente du démarrage (60s)..."
|
||||
sleep 60
|
||||
|
||||
# 6. Vérification
|
||||
echo ""
|
||||
echo "🔍 Vérification..."
|
||||
if curl -s http://localhost:11235/health > /dev/null 2>&1; then
|
||||
echo "✅ Crawl4AI fonctionne !"
|
||||
curl http://localhost:11235/health
|
||||
else
|
||||
echo "❌ Toujours en erreur. Logs :"
|
||||
docker logs crawl4ai --tail 20
|
||||
fi
|
||||
@@ -0,0 +1,190 @@
|
||||
#!/bin/bash
|
||||
# Installation du Web Scraper sur effi-server
|
||||
# À exécuter dans /root/effi/
|
||||
|
||||
set -e
|
||||
|
||||
echo "================================"
|
||||
echo "Installation Web Scraper (Effi)"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# Vérifier qu'on est dans le bon répertoire
|
||||
if [ ! -f "docker-compose.yml" ]; then
|
||||
echo "❌ Erreur: docker-compose.yml non trouvé"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "📁 Répertoire: $(pwd)"
|
||||
|
||||
# Créer le répertoire pour le web scraper
|
||||
mkdir -p web-scraper
|
||||
cd web-scraper
|
||||
|
||||
echo "📝 Création des fichiers..."
|
||||
|
||||
# Dockerfile
|
||||
cat > Dockerfile << 'EOF'
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN pip install --no-cache-dir fastapi uvicorn requests beautifulsoup4
|
||||
|
||||
COPY scraper.py .
|
||||
|
||||
EXPOSE 11235
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
|
||||
CMD curl -f http://localhost:11235/health || exit 1
|
||||
|
||||
CMD ["uvicorn", "scraper:app", "--host", "0.0.0.0", "--port", "11235"]
|
||||
EOF
|
||||
|
||||
# Scraper Python
|
||||
cat > scraper.py << 'EOF'
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, List
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
app = FastAPI(title="Effi Web Scraper", version="1.0.0")
|
||||
|
||||
class ScrapeRequest(BaseModel):
|
||||
url: str
|
||||
max_length: Optional[int] = 10000
|
||||
|
||||
class ScrapeResponse(BaseModel):
|
||||
url: str
|
||||
title: str
|
||||
content: str
|
||||
links: List[str]
|
||||
success: bool
|
||||
error: Optional[str] = None
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "healthy"}
|
||||
|
||||
@app.post("/scrape", response_model=ScrapeResponse)
|
||||
def scrape(request: ScrapeRequest):
|
||||
try:
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
|
||||
}
|
||||
|
||||
response = requests.get(request.url, headers=headers, timeout=30)
|
||||
response.raise_for_status()
|
||||
|
||||
soup = BeautifulSoup(response.content, 'html.parser')
|
||||
|
||||
title = soup.title.string if soup.title else "No title"
|
||||
|
||||
for script in soup(["script", "style"]):
|
||||
script.decompose()
|
||||
|
||||
main_content = None
|
||||
for selector in ['main', 'article', '[role="main"]', '.content', '#content', '.post-content']:
|
||||
main_content = soup.select_one(selector)
|
||||
if main_content:
|
||||
break
|
||||
|
||||
if not main_content:
|
||||
main_content = soup.body if soup.body else soup
|
||||
|
||||
text = main_content.get_text(separator='\n', strip=True)
|
||||
lines = [line.strip() for line in text.split('\n') if line.strip()]
|
||||
text = '\n'.join(lines)
|
||||
|
||||
if len(text) > request.max_length:
|
||||
text = text[:request.max_length] + f"\n\n[...] {len(text) - request.max_length} caractères supplémentaires"
|
||||
|
||||
links = []
|
||||
for a in soup.find_all('a', href=True):
|
||||
href = a['href']
|
||||
if href.startswith('http'):
|
||||
links.append(href)
|
||||
links = links[:20]
|
||||
|
||||
return ScrapeResponse(
|
||||
url=request.url,
|
||||
title=title,
|
||||
content=text,
|
||||
links=links,
|
||||
success=True
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
return ScrapeResponse(
|
||||
url=request.url,
|
||||
title="Error",
|
||||
content="",
|
||||
links=[],
|
||||
success=False,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
@app.post("/scrape/sync")
|
||||
def scrape_sync(url: str, max_length: int = 10000):
|
||||
request = ScrapeRequest(url=url, max_length=max_length)
|
||||
result = scrape(request)
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=400, detail=result.error)
|
||||
return {
|
||||
"url": result.url,
|
||||
"html": result.content,
|
||||
"markdown": result.content,
|
||||
"success": True
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=11235)
|
||||
EOF
|
||||
|
||||
echo "✅ Fichiers créés"
|
||||
echo ""
|
||||
|
||||
# Construire et démarrer
|
||||
echo "🔨 Construction de l'image..."
|
||||
docker build -t effi-web-scraper .
|
||||
|
||||
echo ""
|
||||
echo "🚀 Démarrage du service..."
|
||||
docker run -d \
|
||||
--name effi-web-scraper \
|
||||
--restart unless-stopped \
|
||||
-p 11235:11235 \
|
||||
--network effi-net \
|
||||
effi-web-scraper
|
||||
|
||||
echo ""
|
||||
echo "⏳ Attente du démarrage (15s)..."
|
||||
sleep 15
|
||||
|
||||
# Vérification
|
||||
echo ""
|
||||
echo "🔍 Vérification..."
|
||||
if curl -s http://localhost:11235/health > /dev/null 2>&1; then
|
||||
echo "✅ Web Scraper est démarré !"
|
||||
curl http://localhost:11235/health
|
||||
else
|
||||
echo "⚠️ Le service ne répond pas encore"
|
||||
docker logs effi-web-scraper
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "================================"
|
||||
echo "Installation terminée !"
|
||||
echo "================================"
|
||||
echo ""
|
||||
echo "📋 Configuration OpenWebUI:"
|
||||
echo " Admin → Tools → Web Search and Crawl → ⚙️"
|
||||
echo " Crawl4AI Base URL: http://localhost:11235"
|
||||
echo " Timeout: 30000"
|
||||
echo ""
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/bin/bash
|
||||
# Script d'installation de Crawl4AI sur effi-server
|
||||
# À copier et exécuter sur effi-server (192.168.1.27)
|
||||
|
||||
set -e
|
||||
|
||||
echo "================================"
|
||||
echo "Installation Crawl4AI (effi-server)"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# 1. Créer le répertoire
|
||||
INSTALL_DIR="$HOME/effi-crawl4ai"
|
||||
mkdir -p "$INSTALL_DIR"
|
||||
cd "$INSTALL_DIR"
|
||||
|
||||
echo "📁 Répertoire créé : $INSTALL_DIR"
|
||||
|
||||
# 2. Créer le docker-compose.yml
|
||||
cat > docker-compose.yml << 'EOF'
|
||||
# Service Crawl4AI pour Effi (effi-server)
|
||||
# Version CPU - plus légère
|
||||
|
||||
services:
|
||||
crawl4ai:
|
||||
image: unclecode/crawl4ai:basic
|
||||
container_name: effi-crawl4ai
|
||||
ports:
|
||||
- "11235:11235"
|
||||
environment:
|
||||
- CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}
|
||||
- MAX_CONCURRENT_TASKS=3
|
||||
- CRAWL4AI_ENABLE_GPU=false
|
||||
volumes:
|
||||
- /dev/shm:/dev/shm
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2G
|
||||
reservations:
|
||||
memory: 512M
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
EOF
|
||||
|
||||
echo "📝 docker-compose.yml créé"
|
||||
|
||||
# 3. Démarrer le service
|
||||
echo ""
|
||||
echo "🚀 Démarrage de Crawl4AI..."
|
||||
docker-compose up -d
|
||||
|
||||
# 4. Attendre le démarrage
|
||||
echo ""
|
||||
echo "⏳ Attente du démarrage (30s)..."
|
||||
sleep 30
|
||||
|
||||
# 5. Vérification
|
||||
echo ""
|
||||
echo "🔍 Vérification..."
|
||||
if curl -s http://localhost:11235/health > /dev/null 2>&1; then
|
||||
echo "✅ Crawl4AI est démarré et accessible sur http://localhost:11235"
|
||||
else
|
||||
echo "⚠️ Le service ne répond pas encore. Vérifier les logs :"
|
||||
echo " docker logs effi-crawl4ai"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "================================"
|
||||
echo "Installation terminée !"
|
||||
echo "================================"
|
||||
echo ""
|
||||
echo "📋 Prochaines étapes :"
|
||||
echo " 1. Dans OpenWebUI → Admin → Tools → Web Search and Crawl"
|
||||
echo " 2. Configurer la valve 'Crawl4AI Base URL' : http://localhost:11235"
|
||||
echo " 3. Activer le Tool"
|
||||
echo ""
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
# Script de suppression de Crawl4AI local sur effi-server
|
||||
# À exécuter sur effi-server dans /root/effi/
|
||||
|
||||
echo "================================"
|
||||
echo "Suppression de Crawl4AI local"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# Vérifier qu'on est dans le bon répertoire
|
||||
if [ ! -f "docker-compose.yml" ]; then
|
||||
echo "❌ Erreur: docker-compose.yml non trouvé"
|
||||
echo " Ce script doit être exécuté dans /root/effi/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "🛑 Arrêt du conteneur crawl4ai..."
|
||||
docker-compose stop crawl4ai 2>/dev/null || true
|
||||
docker-compose rm -f crawl4ai 2>/dev/null || true
|
||||
|
||||
echo "🗑️ Suppression de l'image..."
|
||||
docker rmi unclecode/crawl4ai:basic 2>/dev/null || true
|
||||
|
||||
echo ""
|
||||
echo "🔧 Suppression du service du docker-compose.yml..."
|
||||
|
||||
# Créer une version sans le service crawl4ai
|
||||
awk '
|
||||
/^ # ─────────────────────────────────────────────────────────/ && printed_service == 0 {
|
||||
getline
|
||||
if ($0 ~ /8\. CRAWL4AI/) {
|
||||
# On est sur le service crawl4ai, on skip jusqu'au prochain service
|
||||
while (getline && !(/^ # ─────────────────────────────────────────────────────────/ || /^networks:/)) {}
|
||||
# On a atteint la fin du service, on continue normalement
|
||||
printed_service = 1
|
||||
if (/^networks:/) {
|
||||
print
|
||||
next
|
||||
}
|
||||
}
|
||||
# Ce n'est pas crawl4ai, on affiche
|
||||
print prev
|
||||
print
|
||||
next
|
||||
}
|
||||
{ print }
|
||||
' docker-compose.yml > /tmp/docker-compose-clean.yml
|
||||
|
||||
# Remplacer
|
||||
mv /tmp/docker-compose-clean.yml docker-compose.yml
|
||||
|
||||
echo "✅ Service crawl4ai supprimé du docker-compose.yml"
|
||||
echo ""
|
||||
echo "📋 Prochaines étapes :"
|
||||
echo " 1. Crawl4AI tournera sur folzo-Corp (192.168.1.11:11235)"
|
||||
echo " 2. Dans OpenWebUI, configurer le Base URL :"
|
||||
echo " http://192.168.1.11:11235"
|
||||
echo ""
|
||||
@@ -0,0 +1,115 @@
|
||||
"""
|
||||
Service de scraping web simple pour OpenWebUI
|
||||
Alternative à Crawl4AI compatible x86_64
|
||||
"""
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, List
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
import re
|
||||
|
||||
app = FastAPI(title="Effi Web Scraper", version="1.0.0")
|
||||
|
||||
class ScrapeRequest(BaseModel):
|
||||
url: str
|
||||
max_length: Optional[int] = 10000
|
||||
extract_markdown: Optional[bool] = True
|
||||
|
||||
class ScrapeResponse(BaseModel):
|
||||
url: str
|
||||
title: str
|
||||
content: str
|
||||
links: List[str]
|
||||
success: bool
|
||||
error: Optional[str] = None
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "healthy"}
|
||||
|
||||
@app.post("/scrape", response_model=ScrapeResponse)
|
||||
def scrape(request: ScrapeRequest):
|
||||
try:
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
|
||||
}
|
||||
|
||||
response = requests.get(request.url, headers=headers, timeout=30)
|
||||
response.raise_for_status()
|
||||
|
||||
soup = BeautifulSoup(response.content, 'html.parser')
|
||||
|
||||
# Titre
|
||||
title = soup.title.string if soup.title else "No title"
|
||||
|
||||
# Supprimer scripts et styles
|
||||
for script in soup(["script", "style"]):
|
||||
script.decompose()
|
||||
|
||||
# Extraction du texte principal
|
||||
# Essayer de trouver le contenu principal
|
||||
main_content = None
|
||||
for selector in ['main', 'article', '[role="main"]', '.content', '#content', '.post-content']:
|
||||
main_content = soup.select_one(selector)
|
||||
if main_content:
|
||||
break
|
||||
|
||||
if not main_content:
|
||||
main_content = soup.body if soup.body else soup
|
||||
|
||||
# Extraire le texte
|
||||
text = main_content.get_text(separator='\n', strip=True)
|
||||
|
||||
# Nettoyer
|
||||
lines = [line.strip() for line in text.split('\n') if line.strip()]
|
||||
text = '\n'.join(lines)
|
||||
|
||||
# Limiter la taille
|
||||
if len(text) > request.max_length:
|
||||
text = text[:request.max_length] + f"\n\n[...] {len(text) - request.max_length} caractères supplémentaires"
|
||||
|
||||
# Extraire les liens
|
||||
links = []
|
||||
for a in soup.find_all('a', href=True):
|
||||
href = a['href']
|
||||
if href.startswith('http'):
|
||||
links.append(href)
|
||||
links = links[:20] # Limiter à 20 liens
|
||||
|
||||
return ScrapeResponse(
|
||||
url=request.url,
|
||||
title=title,
|
||||
content=text,
|
||||
links=links,
|
||||
success=True
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
return ScrapeResponse(
|
||||
url=request.url,
|
||||
title="Error",
|
||||
content="",
|
||||
links=[],
|
||||
success=False,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
@app.post("/scrape/sync")
|
||||
def scrape_sync(url: str, max_length: int = 10000):
|
||||
"""Endpoint compatible avec l'API Crawl4AI"""
|
||||
request = ScrapeRequest(url=url, max_length=max_length)
|
||||
result = scrape(request)
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=400, detail=result.error)
|
||||
return {
|
||||
"url": result.url,
|
||||
"html": result.content,
|
||||
"markdown": result.content,
|
||||
"success": True
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=11235)
|
||||
@@ -0,0 +1,215 @@
|
||||
#!/bin/bash
|
||||
# Script de mise à jour du Web Scraper sur effi-server
|
||||
# À exécuter dans /root/effi/
|
||||
|
||||
set -e
|
||||
|
||||
echo "================================"
|
||||
echo "Mise à jour Web Scraper"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# 1. Nettoyer l'ancien conteneur standalone
|
||||
echo "🧹 Nettoyage de l'ancien conteneur..."
|
||||
docker stop effi-web-scraper 2>/dev/null || true
|
||||
docker rm effi-web-scraper 2>/dev/null || true
|
||||
|
||||
# 2. Nettoyer le réseau si créé
|
||||
echo "🗑️ Suppression du réseau effi-net temporaire..."
|
||||
docker network rm effi-net 2>/dev/null || true
|
||||
|
||||
# 3. Supprimer l'ancienne section crawl4ai du docker-compose
|
||||
echo "📝 Mise à jour du docker-compose.yml..."
|
||||
|
||||
# Créer le service web-scraper à insérer
|
||||
cat > /tmp/web-scraper-service.txt << 'EOF'
|
||||
# ─────────────────────────────────────────────────────────
|
||||
# 8. WEB SCRAPER - Scraping pour OpenWebUI
|
||||
# → http://localhost:11235
|
||||
# ─────────────────────────────────────────────────────────
|
||||
web-scraper:
|
||||
build: ./web-scraper
|
||||
container_name: effi-web-scraper
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "11235:11235"
|
||||
networks:
|
||||
- effi-net
|
||||
|
||||
EOF
|
||||
|
||||
# Backup du docker-compose
|
||||
cp docker-compose.yml docker-compose.yml.backup.$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Utiliser awk pour remplacer la section 8 (crawl4ai) par web-scraper
|
||||
awk '
|
||||
/^networks:/ {
|
||||
# On est à la fin du fichier services, inserer le nouveau service avant
|
||||
system("cat /tmp/web-scraper-service.txt")
|
||||
}
|
||||
{ print }
|
||||
' docker-compose.yml > /tmp/docker-compose-new.yml
|
||||
|
||||
mv /tmp/docker-compose-new.yml docker-compose.yml
|
||||
|
||||
echo "✅ Section #8 mise à jour (crawl4ai → web-scraper)"
|
||||
|
||||
# 4. Créer le répertoire et les fichiers du web-scraper
|
||||
echo ""
|
||||
echo "📁 Création du répertoire web-scraper..."
|
||||
mkdir -p web-scraper
|
||||
|
||||
cat > web-scraper/Dockerfile << 'EOF'
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN pip install --no-cache-dir fastapi uvicorn requests beautifulsoup4
|
||||
|
||||
COPY scraper.py .
|
||||
|
||||
EXPOSE 11235
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
|
||||
CMD curl -f http://localhost:11235/health || exit 1
|
||||
|
||||
CMD ["uvicorn", "scraper:app", "--host", "0.0.0.0", "--port", "11235"]
|
||||
EOF
|
||||
|
||||
cat > web-scraper/scraper.py << 'EOF'
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional, List
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
app = FastAPI(title="Effi Web Scraper", version="1.0.0")
|
||||
|
||||
class ScrapeRequest(BaseModel):
|
||||
url: str
|
||||
max_length: Optional[int] = 10000
|
||||
|
||||
class ScrapeResponse(BaseModel):
|
||||
url: str
|
||||
title: str
|
||||
content: str
|
||||
links: List[str]
|
||||
success: bool
|
||||
error: Optional[str] = None
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "healthy"}
|
||||
|
||||
@app.post("/scrape", response_model=ScrapeResponse)
|
||||
def scrape(request: ScrapeRequest):
|
||||
try:
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
|
||||
}
|
||||
|
||||
response = requests.get(request.url, headers=headers, timeout=30)
|
||||
response.raise_for_status()
|
||||
|
||||
soup = BeautifulSoup(response.content, 'html.parser')
|
||||
|
||||
title = soup.title.string if soup.title else "No title"
|
||||
|
||||
for script in soup(["script", "style"]):
|
||||
script.decompose()
|
||||
|
||||
main_content = None
|
||||
for selector in ['main', 'article', '[role="main"]', '.content', '#content', '.post-content']:
|
||||
main_content = soup.select_one(selector)
|
||||
if main_content:
|
||||
break
|
||||
|
||||
if not main_content:
|
||||
main_content = soup.body if soup.body else soup
|
||||
|
||||
text = main_content.get_text(separator='\n', strip=True)
|
||||
lines = [line.strip() for line in text.split('\n') if line.strip()]
|
||||
text = '\n'.join(lines)
|
||||
|
||||
if len(text) > request.max_length:
|
||||
text = text[:request.max_length] + f"\n\n[...] {len(text) - request.max_length} caractères supplémentaires"
|
||||
|
||||
links = []
|
||||
for a in soup.find_all('a', href=True):
|
||||
href = a['href']
|
||||
if href.startswith('http'):
|
||||
links.append(href)
|
||||
links = links[:20]
|
||||
|
||||
return ScrapeResponse(
|
||||
url=request.url,
|
||||
title=title,
|
||||
content=text,
|
||||
links=links,
|
||||
success=True
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
return ScrapeResponse(
|
||||
url=request.url,
|
||||
title="Error",
|
||||
content="",
|
||||
links=[],
|
||||
success=False,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
@app.post("/scrape/sync")
|
||||
def scrape_sync(url: str, max_length: int = 10000):
|
||||
request = ScrapeRequest(url=url, max_length=max_length)
|
||||
result = scrape(request)
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=400, detail=result.error)
|
||||
return {
|
||||
"url": result.url,
|
||||
"html": result.content,
|
||||
"markdown": result.content,
|
||||
"success": True
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=11235)
|
||||
EOF
|
||||
|
||||
echo "✅ Fichiers créés dans web-scraper/"
|
||||
|
||||
# 5. Relancer docker-compose
|
||||
echo ""
|
||||
echo "🚀 Démarrage du service web-scraper..."
|
||||
docker-compose up -d web-scraper
|
||||
|
||||
echo ""
|
||||
echo "⏳ Attente du démarrage (30s)..."
|
||||
sleep 30
|
||||
|
||||
# 6. Vérification
|
||||
echo ""
|
||||
echo "🔍 Vérification..."
|
||||
if curl -s http://localhost:11235/health > /dev/null 2>&1; then
|
||||
echo "✅ Web Scraper est fonctionnel !"
|
||||
curl http://localhost:11235/health
|
||||
else
|
||||
echo "⚠️ Le service ne répond pas encore"
|
||||
docker logs effi-web-scraper
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "================================"
|
||||
echo "Installation terminée !"
|
||||
echo "================================"
|
||||
echo ""
|
||||
echo "📋 Configuration OpenWebUI:"
|
||||
echo " Admin → Tools → Web Search and Crawl → ⚙️"
|
||||
echo " Crawl4AI Base URL: http://localhost:11235"
|
||||
echo " Timeout: 30000"
|
||||
echo " ✅ Activer le Tool"
|
||||
Reference in New Issue
Block a user