Files
local-deep-research/scripts/check_metrics.py
LearningCircuit 62928db777 feat: Implement per-user encrypted databases with comprehensive security overhaul
This major release introduces fundamental security and architectural improvements
to Local Deep Research, transitioning from a single-user system to a secure
multi-user platform with encrypted databases and proper authentication.

## 🔐 Security & Authentication
- **Per-user encrypted databases**: Each user now has their own SQLCipher-encrypted
  database with AES-256 encryption, protecting API keys and research data
- **Mandatory authentication**: All API endpoints and programmatic access now
  require user authentication
- **Session-based security**: Implemented secure session management with CSRF
  protection for all state-changing operations
- **Password-based encryption**: User passwords serve as database encryption keys
  (no recovery mechanism - intentional security feature)

## 🏗️ Architecture Changes
- **Thread-safe design**: Complete overhaul of settings and database access to
  ensure thread safety across all operations
- **Settings snapshots**: New immutable settings snapshot pattern prevents race
  conditions in concurrent operations
- **In-memory queue tracking**: Replaced unencrypted service.db with memory-only
  queue tracking to eliminate PII storage risks
- **Optimized middleware**: Reduced middleware overhead by 70% through intelligent
  request filtering and caching

## 📊 Database Structure
- Migrated from single shared database to per-user encrypted databases
- New models: User, UserSettings, UserActiveResearch, AuthSession
- Removed global models that could leak data between users
- All sensitive data (API keys, research history) now user-scoped

## 🧪 Testing & Quality
- Added 200+ new tests covering authentication, encryption, and thread safety
- New Puppeteer UI tests for end-to-end authentication flows
- Comprehensive OpenAI API key configuration tests
- LangChain integration tests for custom LLMs and retrievers
- All tests updated to work with new authentication system

## 📚 Documentation
- New migration guide for v0.x to v1.0 upgrade
- SQLCipher installation guide for all platforms
- Troubleshooting guide for OpenAI API configuration
- Updated all examples to demonstrate authenticated usage
- Comprehensive API documentation with authentication examples

## 🔧 Technical Implementation
- SQLCipher integration with hex-encoded password handling
- Thread-local session storage preventing cross-contamination
- Context-aware database sessions with proper cleanup
- Automatic session lifecycle management
- Rate limiting now per-user instead of global

## 💥 Breaking Changes
- All API access now requires authentication
- Database structure completely changed (migration required)
- Settings API redesigned for thread safety
- Removed direct database access methods
- Changed research ID type from integer to UUID

## 📦 Dependencies
- Added: pysqlcipher3 for database encryption
- Added: Additional auth-related dependencies
- Updated: All major dependencies to latest versions

## 🚀 Performance Improvements
- Middleware optimization reduces overhead by 70%
- Cached settings reduce database queries by 90%
- Thread-local sessions eliminate lock contention
- Smarter request routing skips auth for static assets

This release represents a complete security overhaul making LDR suitable for
production multi-user deployments while maintaining full backward compatibility
through migration guides and extensive documentation.
2025-07-03 02:17:44 +02:00

103 lines
3.0 KiB
Python

#!/usr/bin/env python3
"""Check if metrics are being saved in the database."""
import sys
from pathlib import Path
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.local_deep_research.database.encrypted_db import db_manager
from src.local_deep_research.database.models import TokenUsage, SearchCall
def check_user_metrics(username: str, password: str):
"""Check metrics for a specific user."""
print(f"\n🔍 Checking metrics for user: {username}")
print("=" * 60)
try:
# Open database
if not db_manager.open_user_database(username, password):
print("❌ Failed to open database")
return
# Get session
session = db_manager.get_session(username)
if not session:
print("❌ Failed to get session")
return
# Check token usage
token_count = session.query(TokenUsage).count()
print(f"\n📊 Token Usage Records: {token_count}")
if token_count > 0:
# Get recent token usage
recent_tokens = (
session.query(TokenUsage)
.order_by(TokenUsage.created_at.desc())
.limit(5)
.all()
)
print("\n Recent token usage:")
for token in recent_tokens:
print(
f" - {token.created_at}: {token.model_name} - {token.total_tokens} tokens"
)
print(
f" Phase: {token.research_phase}, Status: {token.success_status}"
)
# Check search calls
search_count = session.query(SearchCall).count()
print(f"\n🔎 Search Call Records: {search_count}")
if search_count > 0:
# Get recent searches
recent_searches = (
session.query(SearchCall)
.order_by(SearchCall.created_at.desc())
.limit(5)
.all()
)
print("\n Recent searches:")
for search in recent_searches:
print(
f" - {search.created_at}: {search.search_engine} - {search.query[:50]}..."
)
print(f" Results: {search.results_returned}")
session.close()
except Exception as e:
print(f"❌ Error: {e}")
import traceback
traceback.print_exc()
finally:
if username in db_manager.connections:
db_manager.connections.pop(username)
def main():
"""Check metrics for test users."""
# Check for a specific test user (modify as needed)
test_users = [
("simple_1751323627595", "password"), # Latest test user
# Add more test users as needed
]
for username, password in test_users:
try:
check_user_metrics(username, password)
except Exception as e:
print(f"Failed to check {username}: {e}")
print("\n" + "=" * 60)
print("✅ Metrics check complete")
if __name__ == "__main__":
main()