mirror of
https://github.com/LearningCircuit/local-deep-research.git
synced 2026-06-16 20:10:39 +03:00
feat: enhance HTTP API examples with retry logic and automatic user creation
Major improvements to HTTP API examples: - Add intelligent retry logic for fetching research results (up to 2 minutes) - Implement automatic user creation for out-of-the-box functionality - Fix API endpoint usage (/api/start_research instead of /research/api/start) - Add proper CSRF token handling and authentication flow - Create comprehensive documentation with environment variable configuration - Add progress monitoring and detailed status reporting - Include remote Ollama and SearXNG configuration examples - Provide multiple example scripts for different use cases - Use pathlib.Path instead of os.path for modern Python practice Examples now work completely out of the box without manual user setup and include proper error handling and user guidance throughout the process.
This commit is contained in:
LearningCircuit
@@ -17,7 +17,7 @@ Since LDR v2.0, all API access requires authentication due to per-user encrypted
|
||||
- `retriever_usage_example.py` - Using LangChain retrievers with LDR
|
||||
|
||||
- **`http/`** - HTTP REST API usage (requires running server)
|
||||
- `working_api_example.py` - ✅ **WORKING EXAMPLE** - Tested and ready to use
|
||||
- `simple_working_example.py` - ✅ **BEST WORKING EXAMPLE** - Clean, tested, and ready to use
|
||||
- `simple_http_example.py` - Quick start example (needs updating for auth)
|
||||
- `http_api_examples.py` - Comprehensive examples including batch processing
|
||||
|
||||
@@ -45,21 +45,40 @@ with get_user_db_session(username="your_username", password="your_password") as
|
||||
|
||||
### HTTP API (REST)
|
||||
|
||||
First, start the server:
|
||||
**🎯 Quick Start - Works Completely Out of the Box!**
|
||||
|
||||
Our tested working example requires zero manual setup:
|
||||
|
||||
```bash
|
||||
# 1. Start the server
|
||||
python -m local_deep_research.web.app
|
||||
|
||||
# 2. Run the working example (creates user automatically!)
|
||||
python examples/api_usage/http/simple_working_example.py
|
||||
|
||||
# 3. Done! ✅ No other steps required
|
||||
```
|
||||
|
||||
Then authenticate and use the API:
|
||||
The example will:
|
||||
- ✅ Create a unique test user automatically
|
||||
- ✅ Test authentication with proper CSRF handling
|
||||
- ✅ Execute a research query using the correct API endpoint
|
||||
- ✅ Provide credentials for manual testing (if desired)
|
||||
- ✅ Show results with direct links to view them
|
||||
|
||||
**📋 Manual API Usage:**
|
||||
|
||||
If you want to integrate the API into your own code:
|
||||
|
||||
```python
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
# Create session for cookie persistence
|
||||
session = requests.Session()
|
||||
|
||||
# Login - get CSRF token first
|
||||
login_page = session.get("http://localhost:5000/auth/login")
|
||||
from bs4 import BeautifulSoup
|
||||
soup = BeautifulSoup(login_page.text, 'html.parser')
|
||||
csrf_input = soup.find('input', {'name': 'csrf_token'})
|
||||
login_csrf = csrf_input.get('value')
|
||||
@@ -86,6 +105,12 @@ response = session.post(
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
**⚠️ Important Notes:**
|
||||
- Use the correct endpoint: `/api/start_research` (not `/research/api/start`)
|
||||
- Login with form data (not JSON)
|
||||
- Handle CSRF tokens properly
|
||||
- User must be created through web interface first
|
||||
|
||||
## Which API Should I Use?
|
||||
|
||||
- **Programmatic API**: Use when integrating LDR into your Python application
|
||||
|
||||
209
examples/api_usage/http/README.md
Normal file
209
examples/api_usage/http/README.md
Normal file
@@ -0,0 +1,209 @@
|
||||
# HTTP API Examples
|
||||
|
||||
This directory contains working examples for using the LDR HTTP API with authentication.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 1. Start the LDR Server
|
||||
|
||||
```bash
|
||||
# Option 1: Direct startup
|
||||
python -m local_deep_research.web.app
|
||||
|
||||
# Option 2: Use the restart script (recommended)
|
||||
bash scripts/dev/restart_server.sh
|
||||
|
||||
# Option 3: Docker compose
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### 2. Run the Simple Working Example
|
||||
|
||||
```bash
|
||||
# This example works completely out of the box!
|
||||
python simple_working_example.py
|
||||
```
|
||||
|
||||
## 📁 Available Examples
|
||||
|
||||
### 🎯 `simple_working_example.py` - BEST FOR STARTING
|
||||
- ✅ **Works completely out of the box**
|
||||
- ✅ **Automatic user creation** (no manual setup needed)
|
||||
- ✅ **Correct API endpoints** and authentication
|
||||
- ✅ **Tested and verified** to work
|
||||
- ⏱️ **Runtime:** 2-10 minutes (research processing time)
|
||||
|
||||
**Perfect for:** First-time users, testing if API works, quick demos
|
||||
|
||||
### 📚 `simple_http_example.py` - LEARNING GUIDE
|
||||
- ✅ **Automatic user creation**
|
||||
- 📊 **Multiple API examples** (research, settings, history)
|
||||
- 🔍 **Progress monitoring** with status updates
|
||||
- ⏱️ **Runtime:** 3-15 minutes (more comprehensive testing)
|
||||
|
||||
**Perfect for:** Learning different API endpoints, understanding the full API surface
|
||||
|
||||
### 🚀 `http_api_examples.py` - ADVANCED CLIENT
|
||||
- 🔧 **Reusable client class** for integration
|
||||
- 📈 **Advanced features** (batch processing, polling)
|
||||
- 🎛️ **Comprehensive patterns** for production use
|
||||
- ⏱️ **Runtime:** 5-30 minutes (extensive testing)
|
||||
|
||||
**Perfect for:** Building applications, production integration, advanced use cases
|
||||
|
||||
### 🛠️ `curl_examples.sh` - SHELL/CURL EXAMPLES
|
||||
- 🔧 **Command-line examples** using curl
|
||||
- 📝 **Shell script format** for automation
|
||||
- 🔤 **Pure HTTP requests** (no Python needed)
|
||||
|
||||
**Perfect for:** Testing from command line, integration with other tools
|
||||
|
||||
## ⚙️ Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
You can configure the LDR service endpoints using environment variables:
|
||||
|
||||
```bash
|
||||
# For local Ollama (default)
|
||||
export LDR_LLM_OLLAMA_URL=http://localhost:11434
|
||||
|
||||
# For remote Ollama server
|
||||
export LDR_LLM_OLLAMA_URL=http://192.168.178.66:11434
|
||||
|
||||
# For Docker compose service names
|
||||
export LDR_LLM_OLLAMA_URL=http://ollama:11434
|
||||
|
||||
# For Docker with host networking
|
||||
export LDR_LLM_OLLAMA_URL=http://host.docker.internal:11434
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
|
||||
In your `docker-compose.yml`, you can set the Ollama URL:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
ldr:
|
||||
environment:
|
||||
# For service name (recommended for docker-compose)
|
||||
- LDR_LLM_OLLAMA_URL=http://ollama:11434
|
||||
|
||||
# For remote Ollama instance
|
||||
# - LDR_LLM_OLLAMA_URL=http://192.168.178.66:11434
|
||||
|
||||
# For host machine Ollama
|
||||
# - LDR_LLM_OLLAMA_URL=http://host.docker.internal:11434
|
||||
```
|
||||
|
||||
### Common Network Scenarios
|
||||
|
||||
| Scenario | Environment Variable | When to Use |
|
||||
|----------|---------------------|-------------|
|
||||
| **Local Ollama** | `http://localhost:11434` | Running Ollama on same machine |
|
||||
| **Remote Ollama** | `http://IP:11434` | Ollama on different server |
|
||||
| **Docker Compose** | `http://ollama:11434` | Using docker-compose service names |
|
||||
| **Docker Host** | `http://host.docker.internal:11434` | Docker container accessing host Ollama |
|
||||
|
||||
## 🔍 Monitoring Progress
|
||||
|
||||
### Server Logs
|
||||
```bash
|
||||
# Monitor real-time progress
|
||||
tail -f /tmp/ldr_server.log
|
||||
|
||||
# Check recent logs
|
||||
tail -20 /tmp/ldr_server.log
|
||||
```
|
||||
|
||||
### Web Interface
|
||||
- **Research Results:** http://localhost:5000/results/{research_id}
|
||||
- **Settings:** http://localhost:5000/settings
|
||||
- **History:** http://localhost:5000/history
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**❌ "Cannot connect to server"**
|
||||
```bash
|
||||
# Start the server first
|
||||
python -m local_deep_research.web.app
|
||||
# or
|
||||
bash scripts/dev/restart_server.sh
|
||||
```
|
||||
|
||||
**❌ "Authentication failed"**
|
||||
- The examples create users automatically, so this shouldn't happen
|
||||
- If it does, check that the server is running correctly
|
||||
|
||||
**❌ "Research failed"**
|
||||
```bash
|
||||
# Check server logs for details
|
||||
tail -f /tmp/ldr_server.log
|
||||
|
||||
# Common issues:
|
||||
# - Ollama not running or wrong URL
|
||||
# - Model not available in Ollama
|
||||
# - Network connectivity issues
|
||||
```
|
||||
|
||||
**❌ "No output from script"**
|
||||
- Scripts may take 2-10 minutes to complete research
|
||||
- Monitor progress in server logs
|
||||
- Check if research started successfully
|
||||
|
||||
### Model Configuration
|
||||
|
||||
Make sure your Ollama has the required models:
|
||||
|
||||
```bash
|
||||
# List available models
|
||||
ollama list
|
||||
|
||||
# Pull a model if needed
|
||||
ollama pull gemma3:12b
|
||||
ollama pull llama3
|
||||
ollama pull mistral
|
||||
```
|
||||
|
||||
## 📚 What Each Example Demonstrates
|
||||
|
||||
### simple_working_example.py
|
||||
- ✅ User creation and authentication
|
||||
- ✅ Basic research request
|
||||
- ✅ Proper CSRF token handling
|
||||
- ✅ Result URL generation
|
||||
|
||||
### simple_http_example.py
|
||||
- ✅ All of the above PLUS:
|
||||
- ✅ Settings management
|
||||
- ✅ Research history
|
||||
- ✅ Progress polling
|
||||
- ✅ Multiple research examples
|
||||
|
||||
### http_api_examples.py
|
||||
- ✅ All of the above PLUS:
|
||||
- ✅ Batch processing
|
||||
- ✅ Advanced polling strategies
|
||||
- ✅ Error handling patterns
|
||||
- ✅ Production-ready client class
|
||||
|
||||
### curl_examples.sh
|
||||
- ✅ Pure HTTP requests
|
||||
- ✅ Command-line integration
|
||||
- ✅ No dependencies needed
|
||||
|
||||
## 🎯 Recommended Usage Path
|
||||
|
||||
1. **Start with `simple_working_example.py`** - Verify everything works
|
||||
2. **Try `simple_http_example.py`** - Learn the API surface
|
||||
3. **Use `http_api_examples.py`** - Build your application
|
||||
4. **Reference `curl_examples.sh`** - For command-line integration
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- [Main API Documentation](../README.md)
|
||||
- [API Quick Start](../../docs/api-quickstart.md)
|
||||
- [Docker Configuration](../../../docker-compose.yml)
|
||||
- [Troubleshooting Guide](../../docs/TROUBLESHOOTING.md)
|
||||
@@ -1,28 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple HTTP API Example for Local Deep Research v2.0+
|
||||
Simple HTTP API Example for Local Deep Research v1.0+
|
||||
|
||||
This example shows how to use the LDR API with authentication.
|
||||
Requires LDR v2.0+ with authentication features.
|
||||
Works completely out of the box with automatic user creation.
|
||||
"""
|
||||
|
||||
import requests
|
||||
import time
|
||||
import sys
|
||||
from bs4 import BeautifulSoup
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to Python path for programmatic user creation
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "src"))
|
||||
|
||||
from local_deep_research.database.encrypted_db import DatabaseManager
|
||||
from local_deep_research.database.models import User
|
||||
from local_deep_research.database.auth_db import get_auth_db_session
|
||||
|
||||
# Configuration
|
||||
API_URL = "http://localhost:5000"
|
||||
USERNAME = "your_username" # Change this!
|
||||
PASSWORD = "your_password" # Change this!
|
||||
|
||||
|
||||
def create_test_user():
|
||||
"""Create a test user programmatically."""
|
||||
username = f"testuser_{int(time.time())}"
|
||||
password = "testpassword123"
|
||||
|
||||
print(f"Creating test user: {username}")
|
||||
|
||||
try:
|
||||
# Create user in auth database
|
||||
auth_db = get_auth_db_session()
|
||||
new_user = User(username=username)
|
||||
auth_db.add(new_user)
|
||||
auth_db.commit()
|
||||
auth_db.close()
|
||||
|
||||
# Create encrypted database for user
|
||||
db_manager = DatabaseManager()
|
||||
db_manager.create_user_database(username, password)
|
||||
|
||||
print(f"✅ User created successfully: {username}")
|
||||
return username, password
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to create user: {e}")
|
||||
return None, None
|
||||
|
||||
|
||||
def main():
|
||||
print("=== LDR HTTP API Example ===")
|
||||
print("🎯 This example works completely out of the box!\n")
|
||||
|
||||
print("⚠️ IMPORTANT NOTES:")
|
||||
print(" • This script may take several minutes to complete")
|
||||
print(" • Research progress can be monitored in the server logs")
|
||||
print(" • Server logs are available at: /tmp/ldr_server.log")
|
||||
print(
|
||||
" • Use 'tail -f /tmp/ldr_server.log' to monitor progress in real-time"
|
||||
)
|
||||
print(" • Results will be available at the URL shown when complete\n")
|
||||
|
||||
# Check if server is running
|
||||
try:
|
||||
response = requests.get(f"{API_URL}/", timeout=5)
|
||||
if response.status_code != 200:
|
||||
print("❌ Server is not responding correctly")
|
||||
print("\n📋 HOW TO START THE SERVER:")
|
||||
print(" • Option 1: python -m local_deep_research.web.app")
|
||||
print(
|
||||
" • Option 2: bash scripts/dev/restart_server.sh (recommended)"
|
||||
)
|
||||
print(
|
||||
" • Note: restart_server.sh will kill existing server process"
|
||||
)
|
||||
sys.exit(1)
|
||||
print("✅ Server is running")
|
||||
except Exception:
|
||||
print(
|
||||
"❌ Cannot connect to server. Please make sure it's running on http://localhost:5000"
|
||||
)
|
||||
print("\n📋 HOW TO START THE SERVER:")
|
||||
print(" • Option 1: python -m local_deep_research.web.app")
|
||||
print(" • Option 2: bash scripts/dev/restart_server.sh (recommended)")
|
||||
print(" • Note: restart_server.sh will kill existing server process")
|
||||
sys.exit(1)
|
||||
|
||||
# Create test user automatically
|
||||
username, password = create_test_user()
|
||||
if not username:
|
||||
print("❌ Failed to create test user")
|
||||
sys.exit(1)
|
||||
|
||||
# Create a session to persist cookies
|
||||
session = requests.Session()
|
||||
|
||||
print("=== LDR HTTP API Example ===")
|
||||
print(f"Connecting to: {API_URL}")
|
||||
print(f"\nTesting with user: {username}")
|
||||
|
||||
# Step 1: Login
|
||||
print("\n1. Authenticating...")
|
||||
@@ -41,8 +115,8 @@ def main():
|
||||
login_response = session.post(
|
||||
f"{API_URL}/auth/login",
|
||||
data={
|
||||
"username": USERNAME,
|
||||
"password": PASSWORD,
|
||||
"username": username,
|
||||
"password": password,
|
||||
"csrf_token": login_csrf,
|
||||
},
|
||||
allow_redirects=False,
|
||||
@@ -52,8 +126,6 @@ def main():
|
||||
print(f"❌ Login failed: {login_response.text}")
|
||||
print("\nPlease ensure:")
|
||||
print("- The server is running: python -m local_deep_research.web.app")
|
||||
print("- You have created an account through the web interface")
|
||||
print("- You have updated USERNAME and PASSWORD in this script")
|
||||
sys.exit(1)
|
||||
|
||||
print("✅ Login successful")
|
||||
@@ -65,8 +137,16 @@ def main():
|
||||
headers = {"X-CSRF-Token": csrf_token}
|
||||
print("✅ CSRF token obtained")
|
||||
|
||||
# Initialize research_id to None
|
||||
research_id = None
|
||||
|
||||
# Example 1: Quick Summary (using the start endpoint)
|
||||
print("\n=== Example 1: Quick Summary ===")
|
||||
print(
|
||||
"📝 This example demonstrates starting a research query and polling for results"
|
||||
)
|
||||
print("⏱️ This typically takes 1-3 minutes to complete\n")
|
||||
|
||||
research_request = {
|
||||
"query": "What is machine learning?",
|
||||
"model": None, # Will use default from settings
|
||||
@@ -76,6 +156,7 @@ def main():
|
||||
}
|
||||
|
||||
# Start research - CORRECT ENDPOINT
|
||||
print("🚀 Starting research...")
|
||||
start_response = session.post(
|
||||
f"{API_URL}/api/start_research", json=research_request, headers=headers
|
||||
)
|
||||
@@ -86,28 +167,71 @@ def main():
|
||||
|
||||
research_data = start_response.json()
|
||||
research_id = research_data["research_id"]
|
||||
print(f"✅ Research started with ID: {research_id}")
|
||||
print("✅ Research started successfully!")
|
||||
print(f"🆔 Research ID: {research_id}")
|
||||
print("📊 Monitor progress in server logs: tail -f /tmp/ldr_server.log")
|
||||
print(f"🌐 Results will be available at: {API_URL}/results/{research_id}\n")
|
||||
|
||||
# Poll for results
|
||||
print("\nWaiting for results...")
|
||||
while True:
|
||||
print("⏳ Waiting for research to complete...")
|
||||
print(
|
||||
"⚠️ NOTE: This will poll for up to 3 minutes to ensure research completes"
|
||||
)
|
||||
print(
|
||||
" If it fails, the research may still be running - check the results URL\n"
|
||||
)
|
||||
|
||||
poll_count = 0
|
||||
max_polls = 18 # Maximum 3 minutes (18 * 10 seconds)
|
||||
|
||||
while poll_count < max_polls:
|
||||
status_response = session.get(
|
||||
f"{API_URL}/research/api/research/{research_id}/status"
|
||||
f"{API_URL}/api/research/{research_id}/status"
|
||||
)
|
||||
|
||||
if status_response.status_code == 200:
|
||||
status = status_response.json()
|
||||
print(f" Status: {status.get('status', 'unknown')}")
|
||||
current_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
if status.get("status") == "completed":
|
||||
poll_count += 1
|
||||
elapsed_time = poll_count * 10 # 10 seconds per poll
|
||||
print(
|
||||
f" Check {poll_count} ({elapsed_time}s): Status = {current_status} (Progress: {progress}%)"
|
||||
)
|
||||
|
||||
if current_status == "completed":
|
||||
print("🎉 Research completed successfully!")
|
||||
break
|
||||
elif status.get("status") == "failed":
|
||||
elif current_status == "failed":
|
||||
print(
|
||||
f"❌ Research failed: {status.get('error', 'Unknown error')}"
|
||||
)
|
||||
print(
|
||||
"📋 Check server logs for details: tail -f /tmp/ldr_server.log"
|
||||
)
|
||||
sys.exit(1)
|
||||
elif current_status in ["queued", "in_progress"]:
|
||||
# Continue polling
|
||||
pass
|
||||
else:
|
||||
print(f"⚠️ Unexpected status: {current_status}")
|
||||
|
||||
time.sleep(2)
|
||||
else:
|
||||
print(
|
||||
f"⚠️ Status check failed with code: {status_response.status_code}"
|
||||
)
|
||||
|
||||
time.sleep(10) # Wait 10 seconds between polls
|
||||
|
||||
if poll_count >= max_polls:
|
||||
print("⏰ 3-minute timeout reached - research is still running")
|
||||
print("💡 This is normal for complex research queries!")
|
||||
print(f"📊 Check results later at: {API_URL}/results/{research_id}")
|
||||
print("📋 Monitor progress with: tail -f /tmp/ldr_server.log")
|
||||
print(
|
||||
"🔍 The script will still try to fetch results (may be incomplete)"
|
||||
)
|
||||
|
||||
# Get results
|
||||
results_response = session.get(
|
||||
@@ -148,18 +272,147 @@ def main():
|
||||
f"- {item.get('query', 'Unknown query')} ({item.get('created_at', 'Unknown date')})"
|
||||
)
|
||||
|
||||
# Example 4: Get and Display Research Results (with retry logic)
|
||||
print("\n=== Example 4: Research Results ===")
|
||||
if research_id:
|
||||
print(f"📄 Fetching research results for ID: {research_id}")
|
||||
print(
|
||||
"🔄 Will retry until results are available (up to 2 additional minutes)\n"
|
||||
)
|
||||
|
||||
# Retry fetching results until available
|
||||
results_retries = 0
|
||||
max_results_retries = 12 # 2 minutes (12 * 10 seconds)
|
||||
|
||||
while results_retries < max_results_retries:
|
||||
results_response = session.get(
|
||||
f"{API_URL}/api/research/{research_id}/report"
|
||||
)
|
||||
|
||||
if results_response.status_code == 200:
|
||||
# Results are available, parse and display them
|
||||
results = results_response.json()
|
||||
|
||||
content = results.get("content", "")
|
||||
sources = results.get("sources", [])
|
||||
findings = results.get("findings", [])
|
||||
|
||||
print(
|
||||
f"✅ Results retrieved successfully after {(results_retries + 1) * 10} seconds!"
|
||||
)
|
||||
print("\n📝 RESEARCH SUMMARY:")
|
||||
print("=" * 50)
|
||||
if content:
|
||||
# Show first 500 characters of the summary
|
||||
summary_preview = (
|
||||
content[:500] + "..." if len(content) > 500 else content
|
||||
)
|
||||
print(summary_preview)
|
||||
else:
|
||||
print("No summary content available")
|
||||
|
||||
print(f"\n📚 SOURCES FOUND: {len(sources)}")
|
||||
for i, source in enumerate(
|
||||
sources[:3], 1
|
||||
): # Show first 3 sources
|
||||
title = source.get("title", "Unknown Title")
|
||||
url = source.get("url", "No URL")
|
||||
print(f" {i}. {title}")
|
||||
print(f" {url}")
|
||||
|
||||
if len(sources) > 3:
|
||||
print(f" ... and {len(sources) - 3} more sources")
|
||||
|
||||
print(f"\n🔍 KEY FINDINGS: {len(findings)}")
|
||||
for i, finding in enumerate(
|
||||
findings[:3], 1
|
||||
): # Show first 3 findings
|
||||
finding_text = finding.get("text", "No finding text")
|
||||
finding_preview = (
|
||||
finding_text[:150] + "..."
|
||||
if len(finding_text) > 150
|
||||
else finding_text
|
||||
)
|
||||
print(f" {i}. {finding_preview}")
|
||||
|
||||
if len(findings) > 3:
|
||||
print(f" ... and {len(findings) - 3} more findings")
|
||||
|
||||
print(
|
||||
f"\n🌐 View full results at: {API_URL}/results/{research_id}"
|
||||
)
|
||||
print("=" * 50)
|
||||
print("🎉 Results displayed successfully!")
|
||||
break # Exit retry loop - success!
|
||||
|
||||
elif results_response.status_code == 404:
|
||||
results_retries += 1
|
||||
elapsed_time = results_retries * 10
|
||||
print(
|
||||
f" Retry {results_retries}/{max_results_retries} ({elapsed_time}s): Results not ready yet, waiting..."
|
||||
)
|
||||
time.sleep(10) # Wait 10 seconds before retrying
|
||||
|
||||
else:
|
||||
print(
|
||||
f"❌ Failed to fetch results: {results_response.status_code}"
|
||||
)
|
||||
print(f"Response: {results_response.text[:200]}")
|
||||
break # Exit retry loop - error
|
||||
|
||||
# Handle case where max retries reached
|
||||
if results_retries >= max_results_retries:
|
||||
print(
|
||||
f"\n⏰ Maximum retry time reached ({max_results_retries * 10} seconds)"
|
||||
)
|
||||
print("💡 This is normal for complex research queries!")
|
||||
print(f"📊 Check results later at: {API_URL}/results/{research_id}")
|
||||
print("📋 Monitor progress with: tail -f /tmp/ldr_server.log")
|
||||
print(
|
||||
"🔍 The research is still running - results will be available when complete"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
"⚠️ No research ID available - research may not have started properly"
|
||||
)
|
||||
|
||||
# Logout
|
||||
print("\n4. Logging out...")
|
||||
print("\n5. Logging out...")
|
||||
session.post(f"{API_URL}/auth/logout", headers=headers)
|
||||
print("✅ Logged out successfully")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("Make sure the LDR server is running:")
|
||||
print(" python -m local_deep_research.web.app\n")
|
||||
print("🎯 Simple LDR HTTP API Example - Works out of the box!")
|
||||
print("⚡ This script creates a user automatically and tests the API")
|
||||
print(
|
||||
"⏱️ Total runtime: Up to 3 minutes polling + 2 minutes results retry + research time"
|
||||
)
|
||||
print(
|
||||
"🔄 Automatically retries fetching results until available (up to 2 minutes)\n"
|
||||
)
|
||||
|
||||
if USERNAME == "your_username":
|
||||
print("⚠️ WARNING: Please update USERNAME and PASSWORD in this script!")
|
||||
print(" Create an account through the web interface first.\n")
|
||||
print("📋 REQUIREMENTS:")
|
||||
print(" • LDR server running")
|
||||
print(" • Beautiful Soup: pip install beautifulsoup4\n")
|
||||
|
||||
print("🚀 START THE SERVER:")
|
||||
print(" • Option 1: python -m local_deep_research.web.app")
|
||||
print(" • Option 2: bash scripts/dev/restart_server.sh (recommended)")
|
||||
print(" • Note: restart_server.sh will kill existing server process\n")
|
||||
|
||||
print("📊 MONITORING:")
|
||||
print(" • Server logs: tail -f /tmp/ldr_server.log")
|
||||
print(" • This script polls for up to 3 minutes")
|
||||
print(" • If research takes longer, script shows where to check results\n")
|
||||
|
||||
print("⏰ TIMING INFO:")
|
||||
print(" • Script polls for 3 minutes to let research complete")
|
||||
print(" • Then retries fetching results for up to 2 additional minutes")
|
||||
print(" • Research typically completes in 2-10 minutes")
|
||||
print(" • Script displays results automatically when available")
|
||||
print(
|
||||
" • If timeout reached, results URL provided for checking completion\n"
|
||||
)
|
||||
|
||||
main()
|
||||
|
||||
242
examples/api_usage/http/simple_working_example.py
Normal file
242
examples/api_usage/http/simple_working_example.py
Normal file
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple Working HTTP API Example for Local Deep Research v1.0+
|
||||
|
||||
This is a clean, working example that demonstrates the correct way to use the LDR API.
|
||||
It creates a user automatically and handles authentication properly.
|
||||
|
||||
Requirements:
|
||||
- LDR v1.0+ server running: python -m local_deep_research.web.app
|
||||
- Beautiful Soup: pip install beautifulsoup4
|
||||
|
||||
This example works COMPLETELY out of the box - no manual setup required!
|
||||
"""
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to Python path for programmatic user creation
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "src"))
|
||||
|
||||
from local_deep_research.database.encrypted_db import DatabaseManager
|
||||
from local_deep_research.database.models import User
|
||||
from local_deep_research.database.auth_db import get_auth_db_session
|
||||
|
||||
|
||||
def create_test_user():
|
||||
"""Create a test user programmatically - works out of the box!"""
|
||||
username = f"testuser_{int(time.time())}"
|
||||
password = "testpassword123"
|
||||
|
||||
print(f"Creating test user: {username}")
|
||||
|
||||
try:
|
||||
# Create user in auth database
|
||||
auth_db = get_auth_db_session()
|
||||
new_user = User(username=username)
|
||||
auth_db.add(new_user)
|
||||
auth_db.commit()
|
||||
auth_db.close()
|
||||
|
||||
# Create encrypted database for user
|
||||
db_manager = DatabaseManager()
|
||||
db_manager.create_user_database(username, password)
|
||||
|
||||
print(f"✅ User created successfully: {username}")
|
||||
return username, password
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to create user: {e}")
|
||||
return None, None
|
||||
|
||||
|
||||
def test_api_with_user(username, password):
|
||||
"""Test the API with the created user."""
|
||||
print(f"\n=== Testing API with user: {username} ===")
|
||||
|
||||
base_url = "http://localhost:5000"
|
||||
session = requests.Session()
|
||||
|
||||
# 1. Test login
|
||||
print("1. Testing login...")
|
||||
try:
|
||||
login_page = session.get(f"{base_url}/auth/login")
|
||||
if login_page.status_code != 200:
|
||||
print(f" ❌ Failed to get login page: {login_page.status_code}")
|
||||
return False
|
||||
|
||||
soup = BeautifulSoup(login_page.text, "html.parser")
|
||||
csrf_input = soup.find("input", {"name": "csrf_token"})
|
||||
login_csrf = csrf_input.get("value")
|
||||
|
||||
if not login_csrf:
|
||||
print(" ❌ No CSRF token found")
|
||||
return False
|
||||
|
||||
login_response = session.post(
|
||||
f"{base_url}/auth/login",
|
||||
data={
|
||||
"username": username,
|
||||
"password": password,
|
||||
"csrf_token": login_csrf,
|
||||
},
|
||||
allow_redirects=False,
|
||||
)
|
||||
|
||||
print(f" Login status: {login_response.status_code}")
|
||||
|
||||
if login_response.status_code not in [200, 302]:
|
||||
print(" ❌ Login failed")
|
||||
return False
|
||||
|
||||
print(" ✅ Login successful")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ Login error: {e}")
|
||||
return False
|
||||
|
||||
# 2. Get CSRF token for API
|
||||
print("\n2. Getting API CSRF token...")
|
||||
try:
|
||||
csrf_response = session.get(f"{base_url}/auth/csrf-token")
|
||||
if csrf_response.status_code != 200:
|
||||
print(
|
||||
f" ❌ Failed to get CSRF token: {csrf_response.status_code}"
|
||||
)
|
||||
return False
|
||||
|
||||
csrf_data = csrf_response.json()
|
||||
csrf_token = csrf_data.get("csrf_token")
|
||||
|
||||
if not csrf_token:
|
||||
print(" ❌ No CSRF token in response")
|
||||
return False
|
||||
|
||||
print(f" ✅ API CSRF token: {csrf_token[:20]}...")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ CSRF token error: {e}")
|
||||
return False
|
||||
|
||||
# 3. Test research API
|
||||
print("\n3. Testing research API...")
|
||||
|
||||
research_request = {
|
||||
"query": "What is machine learning?",
|
||||
"model": "gpt-4o-mini",
|
||||
"search_engines": ["searxng"],
|
||||
}
|
||||
|
||||
headers = {"X-CSRF-Token": csrf_token, "Content-Type": "application/json"}
|
||||
|
||||
# Test the correct endpoint
|
||||
print("\n 3.1 Testing /api/start_research...")
|
||||
try:
|
||||
url = f"{base_url}/api/start_research"
|
||||
response = session.post(url, json=research_request, headers=headers)
|
||||
|
||||
print(f" Status: {response.status_code}")
|
||||
print(f" Response: {response.text[:300]}")
|
||||
|
||||
if response.status_code == 200:
|
||||
try:
|
||||
data = response.json()
|
||||
if data.get("status") == "success":
|
||||
print(" ✅ Research started successfully!")
|
||||
research_id = data.get("research_id")
|
||||
if research_id:
|
||||
print(f" Research ID: {research_id}")
|
||||
print("\n🎉 SUCCESS! API is working correctly.")
|
||||
print(
|
||||
f"📊 View results at: {base_url}/results/{research_id}"
|
||||
)
|
||||
return True
|
||||
elif data.get("status") == "queued":
|
||||
print(" ✅ Research queued successfully!")
|
||||
return True
|
||||
else:
|
||||
print(
|
||||
f" ⚠️ Research returned: {data.get('status', 'unknown')}"
|
||||
)
|
||||
except Exception:
|
||||
print(" ⚠️ Response is not valid JSON")
|
||||
elif response.status_code == 401:
|
||||
print(" ❌ Authentication failed")
|
||||
elif response.status_code == 403:
|
||||
print(" ❌ Forbidden - CSRF token issue")
|
||||
elif response.status_code == 404:
|
||||
print(" ❌ Endpoint not found")
|
||||
elif response.status_code == 500:
|
||||
print(" ❌ Server error")
|
||||
print(" Check server logs: tail -f /tmp/ldr_server.log")
|
||||
else:
|
||||
print(" ⚠️ Unexpected status code")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ Error testing endpoint: {e}")
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function that works completely out of the box!"""
|
||||
print("=== Simple LDR API Working Example ===")
|
||||
print(
|
||||
"🎯 This example works completely out of the box - no manual setup required!\n"
|
||||
)
|
||||
|
||||
# Check if server is running
|
||||
try:
|
||||
response = requests.get("http://localhost:5000/", timeout=5)
|
||||
if response.status_code != 200:
|
||||
print("❌ Server is not responding correctly")
|
||||
print("\nPlease start the server:")
|
||||
print(" python -m local_deep_research.web.app")
|
||||
sys.exit(1)
|
||||
print("✅ Server is running")
|
||||
except Exception:
|
||||
print(
|
||||
"❌ Cannot connect to server. Please make sure it's running on http://localhost:5000"
|
||||
)
|
||||
print("\nStart the server with:")
|
||||
print(" python -m local_deep_research.web.app")
|
||||
sys.exit(1)
|
||||
|
||||
# Create test user automatically
|
||||
username, password = create_test_user()
|
||||
if not username:
|
||||
print("❌ Failed to create test user")
|
||||
sys.exit(1)
|
||||
|
||||
# Test API
|
||||
success = test_api_with_user(username, password)
|
||||
|
||||
if success:
|
||||
print("\n✅ API test completed successfully")
|
||||
print(f"\n🔑 Created user: {username}")
|
||||
print("📝 You can now use this user for manual testing:")
|
||||
print(f" Username: {username}")
|
||||
print(f" Password: {password}")
|
||||
print(" Login URL: http://localhost:5000/auth/login")
|
||||
print("\nNext steps:")
|
||||
print("- Try different research queries")
|
||||
print("- Explore other API endpoints")
|
||||
print("- Check out the web interface at http://localhost:5000")
|
||||
print("- Use the credentials above to log in manually")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print("\n❌ API test failed")
|
||||
print("\nTroubleshooting:")
|
||||
print(
|
||||
"- Make sure the server is running: python -m local_deep_research.web.app"
|
||||
)
|
||||
print("- Check server logs for errors: tail -f /tmp/ldr_server.log")
|
||||
print("- Ensure all dependencies are installed")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,305 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Working HTTP API Example for Local Deep Research v2.0+
|
||||
|
||||
This is a tested, working example that demonstrates proper API usage.
|
||||
It handles authentication correctly and uses the right endpoints.
|
||||
|
||||
Requirements:
|
||||
- LDR v2.0+ (with authentication features)
|
||||
- User account created through web interface at http://localhost:5000/auth/register
|
||||
- LDR server running: python -m local_deep_research.web.app
|
||||
- Beautiful Soup: pip install beautifulsoup4
|
||||
"""
|
||||
|
||||
import time
|
||||
import sys
|
||||
from typing import Optional, Dict, Any
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
|
||||
class LDRClient:
|
||||
"""Working client for interacting with LDR API v2.0+ with proper authentication"""
|
||||
|
||||
def __init__(self, base_url: str = "http://localhost:5000"):
|
||||
self.base_url = base_url
|
||||
self.session = requests.Session()
|
||||
self.csrf_token = None
|
||||
self.username = None
|
||||
|
||||
def login(self, username: str, password: str) -> bool:
|
||||
"""
|
||||
Authenticate with the LDR server.
|
||||
Returns True if successful, False otherwise.
|
||||
"""
|
||||
try:
|
||||
# Step 1: Get login page and extract CSRF token
|
||||
print("📄 Getting login page...")
|
||||
login_page = self.session.get(f"{self.base_url}/auth/login")
|
||||
|
||||
if login_page.status_code != 200:
|
||||
print(f"❌ Failed to get login page: {login_page.status_code}")
|
||||
return False
|
||||
|
||||
# Extract CSRF token from the login form
|
||||
soup = BeautifulSoup(login_page.text, "html.parser")
|
||||
csrf_input = soup.find("input", {"name": "csrf_token"})
|
||||
login_csrf = csrf_input.get("value") if csrf_input else None
|
||||
|
||||
if not login_csrf:
|
||||
print("❌ No CSRF token found in login page")
|
||||
return False
|
||||
|
||||
# Step 2: Submit login form with CSRF token
|
||||
print(f"🔐 Logging in as {username}...")
|
||||
login_response = self.session.post(
|
||||
f"{self.base_url}/auth/login",
|
||||
data={
|
||||
"username": username,
|
||||
"password": password,
|
||||
"csrf_token": login_csrf,
|
||||
},
|
||||
allow_redirects=False, # Don't follow redirects to see response clearly
|
||||
)
|
||||
|
||||
if login_response.status_code not in [200, 302]:
|
||||
print(f"❌ Login failed: {login_response.status_code}")
|
||||
print(f"Response: {login_response.text[:200]}")
|
||||
return False
|
||||
|
||||
self.username = username
|
||||
print("✅ Login successful")
|
||||
|
||||
# Step 3: Get API CSRF token
|
||||
print("🔑 Getting API CSRF token...")
|
||||
csrf_response = self.session.get(f"{self.base_url}/auth/csrf-token")
|
||||
|
||||
if csrf_response.status_code != 200:
|
||||
print(
|
||||
f"❌ Failed to get CSRF token: {csrf_response.status_code}"
|
||||
)
|
||||
return False
|
||||
|
||||
csrf_data = csrf_response.json()
|
||||
self.csrf_token = csrf_data.get("csrf_token")
|
||||
|
||||
if not self.csrf_token:
|
||||
print("❌ No CSRF token in API response")
|
||||
return False
|
||||
|
||||
print("✅ Authentication complete")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Login error: {e}")
|
||||
return False
|
||||
|
||||
def start_research(self, query: str, **kwargs) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Start a new research query.
|
||||
|
||||
Args:
|
||||
query: The research question
|
||||
**kwargs: Additional parameters (model, search_engines, etc.)
|
||||
|
||||
Returns:
|
||||
Dict with research_id and status, or None if failed
|
||||
"""
|
||||
if not self.csrf_token:
|
||||
print("❌ Not authenticated - call login() first")
|
||||
return None
|
||||
|
||||
# Default parameters
|
||||
research_request = {
|
||||
"query": query,
|
||||
"model": kwargs.get("model", "gpt-4o-mini"),
|
||||
"search_engines": kwargs.get("search_engines", ["searxng"]),
|
||||
"iterations": kwargs.get("iterations", 3),
|
||||
"questions_per_iteration": kwargs.get("questions_per_iteration", 3),
|
||||
}
|
||||
|
||||
# Add any additional parameters
|
||||
for key, value in kwargs.items():
|
||||
if key not in research_request:
|
||||
research_request[key] = value
|
||||
|
||||
headers = {
|
||||
"X-CSRF-Token": self.csrf_token,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
try:
|
||||
print(f"🔍 Starting research: {query[:50]}...")
|
||||
|
||||
# Use the CORRECT endpoint!
|
||||
response = self.session.post(
|
||||
f"{self.base_url}/api/start_research", # ✅ CORRECT ENDPOINT
|
||||
json=research_request,
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
print(f"❌ Failed to start research: {response.status_code}")
|
||||
print(f"Response: {response.text[:300]}")
|
||||
return None
|
||||
|
||||
data = response.json()
|
||||
status = data.get("status")
|
||||
|
||||
if status == "success":
|
||||
print("✅ Research started successfully!")
|
||||
return data
|
||||
elif status == "queued":
|
||||
print(
|
||||
f"⏳ Research queued (position: {data.get('queue_position', 'unknown')})"
|
||||
)
|
||||
return data
|
||||
else:
|
||||
print(f"❌ Research returned status: {status}")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error starting research: {e}")
|
||||
return None
|
||||
|
||||
def get_research_status(self, research_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get the status of a research query."""
|
||||
try:
|
||||
response = self.session.get(
|
||||
f"{self.base_url}/api/research/{research_id}/status"
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
print(f"❌ Failed to get status: {response.status_code}")
|
||||
return None
|
||||
|
||||
return response.json()
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error getting status: {e}")
|
||||
return None
|
||||
|
||||
def wait_for_completion(
|
||||
self, research_id: str, timeout: int = 300
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Wait for research to complete.
|
||||
|
||||
Args:
|
||||
research_id: The research ID to wait for
|
||||
timeout: Maximum time to wait in seconds
|
||||
|
||||
Returns:
|
||||
Final status data or None if timed out
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
while time.time() - start_time < timeout:
|
||||
status = self.get_research_status(research_id)
|
||||
if not status:
|
||||
time.sleep(2)
|
||||
continue
|
||||
|
||||
status_code = status.get("status")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
if status_code == "completed":
|
||||
print("✅ Research completed!")
|
||||
return status
|
||||
elif status_code == "failed":
|
||||
print("❌ Research failed")
|
||||
return status
|
||||
elif status_code in ["in_progress", "queued"]:
|
||||
print(f"⏳ Status: {status_code} (Progress: {progress}%)")
|
||||
time.sleep(3)
|
||||
else:
|
||||
print(f"❓ Unknown status: {status_code}")
|
||||
time.sleep(3)
|
||||
|
||||
print(f"⏰ Timeout after {timeout} seconds")
|
||||
return None
|
||||
|
||||
def get_research_report(self, research_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get the final research report."""
|
||||
try:
|
||||
response = self.session.get(
|
||||
f"{self.base_url}/api/research/{research_id}/report"
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
print(f"❌ Failed to get report: {response.status_code}")
|
||||
return None
|
||||
|
||||
return response.json()
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error getting report: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
"""Demonstration of the LDR API usage."""
|
||||
print("=== Local Deep Research API Example ===\n")
|
||||
|
||||
# Configuration - UPDATE THESE
|
||||
USERNAME = "your_username" # ⚠️ Change this!
|
||||
PASSWORD = "your_password" # ⚠️ Change this!
|
||||
|
||||
if USERNAME == "your_username" or PASSWORD == "your_password":
|
||||
print("⚠️ Please update USERNAME and PASSWORD in this script!")
|
||||
print(" 1. Start the server: python -m local_deep_research.web.app")
|
||||
print(" 2. Visit http://localhost:5000/auth/register")
|
||||
print(" 3. Create an account and update the credentials below\n")
|
||||
|
||||
# Create client and login
|
||||
client = LDRClient()
|
||||
|
||||
if not client.login(USERNAME, PASSWORD):
|
||||
print(
|
||||
"\n❌ Login failed. Please check your credentials and that the server is running."
|
||||
)
|
||||
print("💡 Create an account at: http://localhost:5000/auth/register")
|
||||
sys.exit(1)
|
||||
|
||||
# Example 1: Quick research
|
||||
print("\n=== Example 1: Quick Research ===")
|
||||
result = client.start_research(
|
||||
"What is quantum computing?",
|
||||
search_engines=["wikipedia"], # Fast for demo
|
||||
iterations=1,
|
||||
questions_per_iteration=2,
|
||||
)
|
||||
|
||||
if result:
|
||||
research_id = result.get("research_id")
|
||||
print(f"📋 Research ID: {research_id}")
|
||||
|
||||
# Wait for completion
|
||||
final_status = client.wait_for_completion(research_id, timeout=120)
|
||||
|
||||
if final_status and final_status.get("status") == "completed":
|
||||
# Get the report
|
||||
report = client.get_research_report(research_id)
|
||||
if report:
|
||||
content = report.get("content", "")
|
||||
print(f"\n📝 Summary: {content[:300]}...")
|
||||
|
||||
# Example 2: Another query
|
||||
print("\n=== Example 2: Different Query ===")
|
||||
result2 = client.start_research(
|
||||
"What are the benefits of renewable energy?",
|
||||
model="gpt-4o-mini",
|
||||
search_engines=["searxng"],
|
||||
)
|
||||
|
||||
if result2:
|
||||
print(f"📋 Research ID: {result2.get('research_id')}")
|
||||
print("📝 Check the web interface at http://localhost:5000 for results")
|
||||
|
||||
print("\n✅ Examples completed!")
|
||||
print("🌐 View all results at: http://localhost:5000/history")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user