* feat(library): raise PDF storage cap default to 3 GB to match upload validator
Follow-up to #4196. The upload validator's per-file cap was raised to
3 GB, but `PDFStorageManager.max_pdf_size_bytes` still defaulted to
100 MB — so the validator would accept a 3 GB upload only for the
downstream storage step to silently skip it ("PDF size exceeds limit,
skipping storage"). Aligns the two limits.
* `defaults/research_library/library_settings.json`: setting default
`100 -> 3072` MB; UI ceiling `max_value: 500 -> 10240` so admins can
actually configure values above the new default.
* `pdf_storage_manager.py`: constructor fallback default
`max_pdf_size_mb: int = 100 -> 3072`.
* `download_service.py`: runtime fallback `settings.get_setting(
"research_library.max_pdf_size_mb", 100) -> 3072` for the case where
the setting row is missing.
* Tests: `test_initializes_with_defaults` and `test_default_max_size`
now expect the 3 GB default. Other tests that construct with explicit
`max_pdf_size_mb=` values are unaffected.
* `tests/settings/golden_master_settings.json` regenerated via
`scripts/dev/regenerate_golden_master.py` (the `value` and
`max_value` deltas only).
Memory bound is unchanged: large PDFs still spool to disk via
`DiskSpoolingRequest.max_form_memory_size`. The library-storage step
either writes to a filesystem path or to a `DocumentBlob` row, both of
which already handle multi-GB inputs.
* review: consolidate 3 GB fallback into one constant + cross-ref the two limits
Addresses the AI Code Review on #4198:
* Single source of truth: new `DEFAULT_MAX_PDF_SIZE_MB = 3072` module
constant in `pdf_storage_manager.py`, imported and reused by
`download_service.py` for its `settings.get_setting(...)` fallback.
Drops two duplicate literals so a future bump only touches the constant
(and the JSON, which is unavoidable since it is the runtime source of
truth and not Python-importable).
* Cross-reference the relationship between the two limits so an admin
who bumps one knows it interacts with the other:
- `library_settings.json` setting description now notes that uploads
above `LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB` (default 3 GB) are
rejected before reaching storage, so raising the library cap above
that value has no effect.
- `PDFStorageManager.__init__` docstring mirrors the note.
- `file_upload_validator.py` module-level comment points back at
the library-side cap.
* `docs/env_configuration.md` gains an Upload Size Limit section
documenting `LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB` (the env var was
previously discoverable only by reading the code).
* `docs/CONFIGURATION.md` and `tests/settings/golden_master_settings.json`
regenerated via their respective scripts (only the setting description
changed; defaults and constraints are unchanged from the prior commit).
Two small follow-ups from the #3976 investigation.
connection_cleanup.py: bump dispose-failure log from debug to warning.
The 30-min periodic pool dispose at web/auth/connection_cleanup.py:154-171
is the workaround for ADR-0004's SQLCipher + WAL handle leak. Pre-fix,
_checkpoint_wal/engine.dispose() failures were swallowed at logger.debug,
hiding silent drift. Now surfaces at WARNING with the exception TYPE NAME
only (matches the _report_silent_exception pattern in utilities/log_utils.py:146-194,
which deliberately drops the exception value to avoid leaking sensitive locals
through the sensitive-logging hook).
New test test_dispose_failures_surface_as_warnings locks in:
- the warning fires and names the user + exception type
- the exception's message text does NOT leak
docs/CONFIGURATION.md: document that LDR_APP_DEBUG=true also enables
Loguru diagnose=True on every sink, which materialises local-variable
values into exception traces. Those traces can include credentials,
decrypted user content, and other sensitive locals. Documentation-only.
Refs: #3976
Upload endpoints (rag.upload_to_collection, etc.) had a hardcoded
"10 per minute;100 per hour" limit that could not be overridden by
any environment variable, only by disabling rate limiting globally.
This blocked legitimate bulk RAG library workflows.
- Add LDR_SECURITY_RATE_LIMIT_UPLOAD_USER and LDR_SECURITY_RATE_LIMIT_UPLOAD_IP
- Raise default to "60 per minute;1000 per hour" (10x the prior cap, sufficient
for typical bulk-upload workflows like a few hundred small files)
- Per-user and per-IP buckets are now independently configurable
- Fix existing TestUploadRateLimitFunctional which hardcoded "10 per minute"
and would silently drift with the new default
- Regenerate docs/CONFIGURATION.md and golden master settings
Note: the auto-index ThreadPoolExecutor at rag_routes.py:76 has an
unbounded queue; users enabling research_library.auto_index_enabled
should set conservative LDR_SECURITY_RATE_LIMIT_UPLOAD_USER values
or wait for a follow-up PR adding queue-depth backpressure.
Lower the default context window size for local LLM providers (Ollama,
LlamaCpp, LM Studio) from 30208 to 20480 to fit on systems with more
limited VRAM while still leaving room for the typical synthesis prompt.
Users whose synthesis prompts are getting truncated can increase this;
users low on VRAM can reduce it further and switch to the source-based
strategy as the description notes.
* fix(llm): remove silent gemma3:12b fallback for Ollama model
The settings UI's Language Model field silently pre-filled "gemma3:12b"
whenever llm.model was unset, and the same fallback was injected at five
backend layers (default seed, JS pre-fill, get_llm(), corruption recovery,
provider class). This caused Ollama to download a 7-12 GB binary the user
never asked for and masked unconfigured-model bugs as runtime failures
deep in langchain.
Drop the fallback at every layer and fail loudly when llm.model is unset:
- default_settings.json: seed value -> ""
- get_llm(): raise ValueError with actionable message naming the setting
and example values for each provider
- OllamaProvider: default_model -> "", create_llm() raises if no model
- Corruption-recovery in settings_routes.py: use "" instead of gemma
- /api/check/ollama_model: HTTP 400 (error_type "model_not_configured")
when no model in query or config, instead of silently checking gemma
- followup_research/routes.py: drop snapshot fallback, let get_llm() raise
- News recommender scheduled job: catch ValueError from get_llm() and log
one concise warning per topic instead of spamming stack traces every tick
- benchmarks/cli.py: catch ValueError, print friendly Hint, exit code 2
Frontend:
- settings.js: drop || 'gemma3:12b' pre-fill at line 3416
- settings_form.html: inline yellow warning under the model dropdown
shown when value is empty, with role="status" + aria-live="polite"
- settings.css: new .ldr-input-warning class mirroring .ldr-input-help
- settings.js: updateModelEmptyWarning() helper wired to both the hidden
input change event and the visible input event for live keystroke feedback
Migration safety verified: settings/manager.py:1129-1133 preserves any
non-None DB value when re-importing defaults (overwrite=False), so existing
users who had gemma3:12b saved keep it. Only fresh DBs and missing rows
pick up the new empty seed.
Mirrors the existing API-key-not-configured pattern in openai_base.py and
the URL-not-configured pattern in providers/implementations/ollama.py.
* fix(llm): address review feedback on PR #3670
- benchmarks/cli.py: extend ValueError handler to run_optimization and
run_comparison, matching the run_profiling pattern. (Note: cli.py is
shadowed by benchmarks/cli/ package and currently unreachable; kept
for internal consistency.)
- settings.js: filterModelOptionsForProvider now calls
updateModelEmptyWarning when it auto-sets the hidden input, so the
warning state stays in sync after a provider switch (the .value=
assignments don't fire change events).
- Add test for topic_based ValueError swallow path (returns None,
does not construct AdvancedSearchSystem).
- Add tests for /api/check/ollama_model HTTP 400 + error_type
"model_not_configured" response.
* fix(ui): warn when provider has zero filtered models in primary dropdown path
The primary updateDropdownOptions branch of filterModelOptionsForProvider
returned early without handling the filteredModels.length === 0 case.
A user switching to a provider with no models would keep a stale warning
state. Mirrors the backup setupCustomDropdown path's existing handling:
clear the hidden + visible inputs and surface the warning.
* refactor(llm): replace in-process llama-cpp-python with HTTP llama-server
LDR's llamacpp provider used langchain_community.llms.LlamaCpp (the
llama-cpp-python Python binding) which loads model files in-process. That
approach has real costs: heavy optional dep (C compilation, CUDA/Metal
toolchains), couples model loading into the LDR process, can't separate
compute machine from LDR, and adds path-validation security code,
.gguf/.bin extension checks, four GPU/batch/f16/path settings, and lots
of test surface.
llama.cpp's own llama-server binary already exposes an OpenAI-compatible
HTTP API (defaults to http://localhost:8080/v1). Most modern users already
run that. Rewrite the llamacpp provider to mirror LMStudioProvider — a thin
ChatOpenAI wrapper pointed at the configured URL.
Backend:
- New file: llm/providers/implementations/llamacpp.py mirrors lmstudio.py
- constants.py: add DEFAULT_LLAMACPP_URL = http://localhost:8080/v1
- llm_config.py: is_llamacpp_available() now delegates to the provider's
HTTP probe (no langchain_community.llms import). The inline llamacpp
branch in get_llm() drops PathValidator, .gguf/.bin extension check,
GPU/batch/f16 params, and the directory-listing logic — replaced by a
22-line ChatOpenAI dispatch identical in shape to lmstudio.
Settings:
- default_settings.json: drop llm.llamacpp_f16_kv, llm.llamacpp_model_path,
llm.llamacpp_n_batch, llm.llamacpp_n_gpu_layers; add llm.llamacpp.url
- settings.js: drop the same keys from tabSpecificSettings/prioritySettings,
add llamacpp_url
- Existing users with llm.provider=llamacpp keep that setting; they need
to start `llama-server -m <model.gguf>` (port 8080 by default) to use it.
Old llamacpp_* settings become harmless orphans (delete_extra=True on
initialize cleans them up).
Tests:
- New HTTP-based tests verify ChatOpenAI is called with the configured URL
(test_llm_config_providers.py, test_llm_config_extended.py).
- is_llamacpp_available coverage rewritten to mock LlamaCppProvider.is_available.
- TestGetLlmLlamaCppEdgeCases (path/extension/directory edge cases) deleted
from test_llm_config_missing_coverage.py with a comment pointing to the
new HTTP tests.
- Updated test_llm_config_missing_coverage.py fallback-skip test to mirror
the lmstudio sibling exactly.
- Updated test_llm_provider_integration.py to use the new url setting.
- Regenerated golden_master_settings.json (497 settings; 4 dropped, 1 added).
* feat(llm): add optional api_key setting for lmstudio + llamacpp
Both providers previously hardcoded a placeholder API key on the assumption
that LM Studio and llama-server don't need auth. That assumption is wrong
when either is deployed behind an auth proxy (or when LM Studio's own auth
feature is enabled).
Add an optional API key setting per provider:
- llm.lmstudio.api_key
- llm.llamacpp.api_key
Both default to an empty string. When set, the provider passes it to
ChatOpenAI; when empty, falls back to the existing placeholder so no-auth
setups keep working unchanged. Wired into both the provider classes
(LMStudioProvider, LlamaCppProvider) and the inline dispatch branches in
llm_config.get_llm().
UI element is "password" so the value is masked in the settings page.
* refactor(llm): drop silent default_model fallback across all providers
Apply the same no-silent-default principle to every LLM provider that we
already applied to Ollama in this PR. Previously each provider class had a
hardcoded default_model (e.g. "gpt-3.5-turbo", "claude-3-sonnet-20240229",
"gemini-1.5-flash", "grok-beta") that kicked in when create_llm() was called
without a model_name. This is the same antipattern as Ollama's "gemma3:12b"
silent default — just less harmful (no surprise multi-GB download). For
direct callers of provider classes (programmatic API users, registered
custom LLMs), this could silently route to an old model the user didn't pick.
Changes:
- OpenAICompatibleProvider (parent): default_model = ""; both create_llm
and _create_llm_instance raise ValueError when model_name is empty
- OpenAIProvider, AnthropicProvider: default_model = ""; their create_llm
overrides also raise
- GoogleProvider, OpenRouterProvider, IONOSProvider, XAIProvider,
CustomOpenAIEndpointProvider: default_model = "" (inherit parent's raise)
- LMStudioProvider: default_model = "" (was "local-model")
- LlamaCppProvider: already "" from earlier in this PR
Also document `api_key_setting = None` on lmstudio + llamacpp: it tells the
parent's create_llm "no key required", but the override below reads
`llm.{lmstudio,llamacpp}.api_key` directly with placeholder fallback for
the optional auth-enabled case. The previous comment was outdated after we
added the optional setting earlier in this PR.
Tests: ~73 tests across 12 files updated to either pass an explicit
model_name="test-model" (when the test was exercising temperature/api_key/
URL/max_tokens behavior unrelated to model selection) or to assert the new
ValueError raise (when the test was specifically verifying the now-removed
silent-default behavior). No tests deleted.
Docs: regenerated docs/CONFIGURATION.md to drop the four old llamacpp_*
keys and document the new llm.llamacpp.{url,api_key} + llm.lmstudio.api_key
settings.
* test(llm): update missed default_model assertion in test_llm/test_providers.py
Subagent that fixed the bulk of the test failures missed this file (it was
in tests/test_llm/ rather than tests/llm_providers/). Same one-line flip:
`OpenAICompatibleProvider.default_model` is now "" instead of
"gpt-3.5-turbo".
* test(llm): update missed default_model assertions in tests/llm/
Fixes 10 remaining failures in tests/llm/ that were missed when the
silent default_model fallback was removed. Tests calling create_llm()
without model_name now either supply a model or assert the new
ValueError("model not configured").
* docs(readme): document llamacpp HTTP migration + add LM Studio/llama.cpp to local models
PR #3670 changed two user-facing behaviors that need a heads-up for
upgraders:
1. llm.model is no longer auto-filled with gemma3:12b. The field starts
empty — users have to consciously pick a model. Previously a fresh
install + research kicked off a silent multi-GB Ollama download.
2. The llamacpp provider switched from in-process llama-cpp-python (with
llm.llamacpp_model_path pointing at a local .gguf) to talking HTTP to
llama-server. The old model_path setting is no longer read.
While here, the "Local Models" section only mentioned Ollama. Now lists
all three supported local options (Ollama / LM Studio / llama.cpp) with
their default endpoint URLs, since LM Studio + llama.cpp have been
supported but invisible in the README.
* test(llm): add llamacpp tests, LM Studio API-key test, and 1.7.0 release notes
- Add test_llamacpp_provider.py (19 tests): metadata, create_llm,
is_available, and requires_auth coverage for the new HTTP-based
llama.cpp provider.
- Add test_create_llm_uses_configured_api_key to LM Studio tests,
verifying that a user-configured llm.lmstudio.api_key is passed
through to ChatOpenAI instead of the "not-required" placeholder.
- Add docs/release_notes/1.7.0.md documenting BREAKING changes
(no default model, llamacpp HTTP migration) and new features
(LM Studio optional API key, UI model-empty warning).
* fix(llm): reject whitespace-only model names in validation
The model-name guard used `if not model_name:` which treats
whitespace-only strings like " " as truthy, allowing them to leak
into downstream LLM libraries. Changed all 6 validation sites to
`if not model_name or not model_name.strip():` so whitespace-only
values now correctly raise ValueError.
Files changed:
- llm_config.py (central get_llm() gate)
- openai_base.py (create_llm + _create_llm_instance)
- anthropic.py, openai.py, ollama.py (provider-specific checks)
* docs(security): document DNS-rebinding TOCTOU window in notification SSRF
The notification URL validator (PR #3092 / #3311) resolves hostnames once
at validation time and checks resolved IPs against private ranges, but
Apprise re-resolves at send time -- a DNS-rebinding attacker can serve a
public IP at validation and a private IP at send. Apprise exposes no
DNS/Session hook to close this in code without fragile monkey-patching of
its plugin internals.
Given LDR's threat model (single-tenant local app, @login_required on
settings routes, per-user encrypted SQLCipher DBs), the residual risk is
acceptable as long as it's visible. This change makes it visible:
- Updated the inline comment in NotificationURLValidator._is_private_ip
to describe the TOCTOU window and recommend plugin schemes
(discord://, slack://, ntfy://, etc.) over raw http(s):// webhooks.
- Added a parallel comment in ssrf_validator.validate_url, since
safe_requests has the same pattern.
- Added a "Notification Webhook SSRF" subsection to SECURITY.md with
the rebinding window, the rationale for not closing it in code, the
threat-model factors that make it acceptable, and operator-side
mitigations (prefer plugin schemes, restrict egress).
No behavior change.
* feat(notifications): default-off + env-only master switch (LDR_NOTIFICATIONS_ENABLED)
Outbound notifications via Apprise carry a known DNS-rebinding TOCTOU
window: the URL validator resolves once at config time, but Apprise
re-resolves at send time, and a logged-in user with a controllable domain
can rebind to internal services on the LDR server (e.g.
127.0.0.1:<internal-port>) or the local network. The window cannot be
closed in code without fragile monkey-patching of Apprise's plugin
internals (HTTPS-only, doesn't follow redirects).
Since LDR is multi-user (per-user SQLCipher DBs behind @login_required),
the right default is to keep outbound notifications off until the
operator explicitly opts in -- a server-level decision, not something a
logged-in user can flip via the settings API.
Changes:
- Add notifications.enabled env-only setting (default False), registered
alongside notifications.allow_private_ips in env_definitions/security.py.
Auto-mapped to LDR_NOTIFICATIONS_ENABLED.
- NotificationManager reads the env at __init__ and gates send_notification
before any other check; force=True bypasses per-user toggles only, never
the operator switch.
- NotificationService now takes enabled=False; test_service refuses with
a clear error pointing at LDR_NOTIFICATIONS_ENABLED. The settings route
/api/notifications/test-url passes the env-read value through.
- Refresh inline TOCTOU comment in NotificationURLValidator._is_private_ip
to reflect the new gate, and add a parallel comment near getaddrinfo in
ssrf_validator.py for cross-cutting consistency (same TOCTOU pattern).
- Rewrite the SECURITY.md "Notification Webhook SSRF" subsection: lead
with "disabled by default", explain how to enable, document the residual
risk operators are accepting when they flip the switch.
- Tests:
- tests/notifications/conftest.py autouse-enables the gate so existing
tests exercising the inner logic still work.
- TestMasterSwitchEnvGate covers the gate behavior explicitly: env unset
=> send_notification returns False (even with force=True), test_service
returns a disabled error.
- TestNotificationManager in test_notification_coverage.py gets a
class-scoped autouse fixture for the same reason.
- Existing NotificationService(...) calls in tests pass enabled=True so
their inner-logic assertions keep working.
This is a behavior change. Existing users with notifications working will
need to set LDR_NOTIFICATIONS_ENABLED=true on upgrade.
* fix(notifications): rename env gate to allow_outbound + clearer logs + docs
Two issues with the previous commit:
1. Key collision. The env gate was named notifications.enabled, which is
already a (currently dormant) per-user DB setting in
default_settings.json. Renaming the env-only setting to
notifications.allow_outbound (env: LDR_NOTIFICATIONS_ALLOW_OUTBOUND)
keeps the two layers distinct. Symmetric with the existing
notifications.allow_private_ips env-only setting.
2. Log levels. The gate-closed paths logged at DEBUG, which is invisible
under default log configuration. An operator wondering why
notifications aren't firing wouldn't see the actionable signal.
Upgrade to WARNING with messages that explicitly name the env var and
point at SECURITY.md.
Also:
- Regenerate docs/CONFIGURATION.md (auto-generated from env definitions
+ default_settings.json) so LDR_NOTIFICATIONS_ALLOW_OUTBOUND appears in
the env-only table at line 52.
- Add a "Server-Side Opt-In Required" section at the top of
docs/NOTIFICATIONS.md, including the symptoms an operator would see
when the gate is closed (so debugging "why isn't this working?" is a
one-step lookup).
- Rename NotificationService kwarg enabled -> outbound_allowed and the
manager's self._notifications_enabled -> self._outbound_allowed for
internal consistency with the new setting name.
- Update tests + conftest accordingly. 507 tests pass, pre-commit clean.
* fix(notifications): defense-in-depth gate in service.send() + module-scope test fixture
Two non-blocker recommendations from code review on PR #3675:
1. service.send() did not enforce the outbound_allowed gate itself --
the manager always wraps it, but a future direct caller could bypass.
Add the same WARNING-level guard at the top of send() that
test_service already has, so the security boundary lives at the
service layer (one place) instead of relying on call-chain
discipline.
2. Promote tests/web/services/test_notification_coverage.py's autouse
gate-opening fixture from class-scope (TestNotificationManager only)
to module-scope, so any future test class added to the file picks it
up automatically. Drop the now-redundant class-scoped duplicate.
Tests:
- TestSendOutboundGate in tests/notifications/test_service.py covers the
new gate: outbound_allowed=False => send() returns False without
touching Apprise (.notify and .add must not be called); gate open =>
the existing send path runs.
- _make_service helper in test_service_extra_coverage.py now sets
outbound_allowed=True so the SSRF/Apprise-failure tests exercise the
inner logic, not the gate.
509 passed, 1 skipped, pre-commit clean.
* config: disable general.enable_fact_checking by default
Flip the default for general.enable_fact_checking from true to false.
When enabled, the citation handler does an extra LLM call per
analyze_followup that re-analyzes sources for consistency and injects
the result into the synthesis prompt. For agentic strategies (LangGraph
in particular) this is largely duplicative — the agent already
cross-references sources during its tool loop — and adds cost,
latency, and prompt bloat without clear quality gains.
Users who rely on the extra validation pass can re-enable it via the
setting. See #3671 for the discussion and trade-offs.
Refs #3671
* config: sync docs and edge-case test to new fact-check default
- Regenerate docs/CONFIGURATION.md so the documented default for
general.enable_fact_checking reflects the new value (false).
- Update test_fact_checking_enabled_by_default in
test_citation_handler_edge_cases.py: with no settings_snapshot, the
handler now sees fact-checking as disabled, so analyze_followup
invokes the LLM only once. Renamed to
test_fact_checking_disabled_by_default and flipped the call-count
assertion accordingly.
Refs #3671
Two sites still document / read the legacy default of `4` even though
the authoritative default in `src/local_deep_research/defaults/
default_settings.json` has been `2` since the journal-quality
redesign (PR #3081 family) lowered it.
- `docs/CONFIGURATION.md:534`: table cell documented default `4`;
corrected to `2` and added the "drops predatory (score 1) only"
note already used in `docs/journal-quality.md` and the JSON
description.
- `advanced_search_system/filters/journal_reputation_filter.py:72`:
`get_setting_from_snapshot("search.journal_reputation.threshold",
4, ...)` — the fallback is effectively unreachable in production
(settings are seeded from `default_settings.json` on first-run),
but the mismatch was misleading to readers and would silently
change filter behavior for any caller that bypasses the snapshot.
* feat: add needs_reranking flag to auto-enable LLM relevance filtering for keyword-based engines
Engines with poor native relevance ranking (arXiv, PubMed, Wikipedia,
GitHub, Mojeek, etc.) now auto-enable LLM-based result filtering via
a new `needs_reranking` class attribute. This fixes the priority bug
where the global `skip_relevance_filter=True` incorrectly overrode
auto-detection for engines that genuinely need filtering.
Priority is now: per-engine setting > needs_reranking > global skip.
The global skip only affects unclassified engines.
Closes#2297
* fix: address 7 code-review issues on needs_reranking branch
1. Rename needs_reranking → needs_llm_relevance_filter for consistency
with enable_llm_relevance_filter and skip_relevance_filter naming
2. Fix Paperless dead code: replace non-existent _apply_content_filters
with proper _filter_for_relevance() call in custom run() override
3. Fix misleading skip_relevance_filter description to accurately
reflect checkbox behavior and keyword engine exceptions
4. Delete 4 vacuously-true inline tests that duplicated factory logic
instead of calling the real factory (coverage tests already exist)
5. Add needs_llm_relevance_filter to EXTENDING.md and OVERVIEW.md
6. Clarify is_generic comment: generic does not imply good ranking
7. Upgrade no-LLM log from debug to warning when filtering was
requested but no LLM is available (with should_filter guard)
* fix: remove Paperless fallback that overrode valid empty LLM filter results
Replace the fallback that restored all previews when the LLM filter
returned empty with an info log. The base class _filter_for_relevance()
already handles errors internally (returns previews[:5] on exception
or JSON parse failure). An empty result means the LLM legitimately
found nothing relevant — trust it, don't override it.
* refactor: rename needs_llm_relevance_filter → is_lexical
The flag describes what the engine IS (lexical/keyword-based search)
rather than what it needs. This is a general classification that can
drive multiple behaviors beyond just the relevance filter — e.g.
query optimization strategies, result deduplication, or UI hints.
Matches the existing is_* naming pattern (is_scientific, is_generic).
* Revert "refactor: rename needs_llm_relevance_filter → is_lexical"
This reverts commit c322d478a1.
* Reapply "refactor: rename needs_llm_relevance_filter → is_lexical"
This reverts commit 853dfe90bd.
* feat: add is_lexical classification flag alongside needs_llm_relevance_filter
Separates classification from behavior:
- is_lexical: informational flag indicating the engine uses keyword/lexical
search. Reusable for query optimization, UI hints, deduplication, etc.
- needs_llm_relevance_filter: behavioral flag that the factory reads to
auto-enable LLM relevance filtering on the engine instance.
Both flags are set on all 15 keyword-based engines. The factory only
checks needs_llm_relevance_filter for filtering decisions.
* fix: improve relevance filter error handling and logging
- Return [] on all error paths instead of hiding failures behind
previews[:5] fallback — failures should be visible, not masked
- Log errors at error level (not warning) for LLM parse failures
- Add engine name prefix to all log messages for traceability
- Add token estimate debug log to help diagnose context overflow
- Reduce log noise: routine operations are debug, only summary is info
- Consolidate validation into single check
* fix: address PR review findings for relevance filter
- Fix literal \n in EXTENDING.md code block
- Remove 'Maximum results to return' from LLM prompt (LLM decides)
- Add INPUT/KEPT/REMOVED debug logging for filter quality analysis
- Add is_lexical + needs_llm_relevance_filter to ElasticsearchSearchEngine
- Delete vacuously-true test_missing_llm_returns_none test
- Downgrade no-op skip_relevance_filter log from info to debug
* refactor: extract relevance filter into dedicated module
Pull the inline _filter_for_relevance() logic out of BaseSearchEngine
into a new web_search_engines/relevance_filter.py module.
- Use with_structured_output() with Pydantic schema; let LangChain
pick the per-provider default method (JSON schema on Ollama,
tool-calling on Anthropic, responseSchema on Gemini).
- Trim prompt: drop URLs, cap snippets at 200 chars.
- Suppress reasoning on Ollama thinking-by-default models via
reasoning=False — saves 30-60s per call on qwen3 dense variants.
- Treat empty LLM responses as valid judgments; log a warning on
batches >2 so users notice a misbehaving model.
- On exception or parse failure, return first N previews (cap=5 or
max_filtered_results) to avoid overwhelming downstream.
* refactor(relevance_filter): cleanup + add direct tests
* feat(relevance_filter): batch previews in parallel for speed and reliability
Adds two tunable parameters to the LLM relevance filter:
- batch_size: split previews into chunks before sending to the LLM.
Each batch uses local indices [0..batch_size-1] mapped back to
global. Default 10. Smaller batches are faster per call AND more
reliable on weaker models that struggle with many indices in one
context.
- max_parallel_batches: dispatch batches concurrently via a
ThreadPoolExecutor. Default 4. Result order is preserved across
parallel batches.
Both exposed as BaseSearchEngine class attributes
(relevance_filter_batch_size, relevance_filter_max_parallel_batches)
so individual engines can override.
Failure semantics:
- Hard exception on any batch -> capped slice fallback (unchanged).
- Parse failure on a single batch -> skip that batch only, keep
results from successful batches.
Adds 4 direct unit tests covering chunk/index mapping, batch_size=None
single-call mode, failed-batch-skip-keeps-others, and parallel dispatch
order preservation. All 120 tests pass.
* refactor(relevance_filter): drop structured output, parse plain text
The Pydantic with_structured_output() path had several issues:
- qwen3 dense models returned prose instead of JSON, raising
OutputParserException and disabling the filter for that call
- grammar-constrained output on Ollama was 6-10x slower than plain
text generation (~24s vs ~4s for 50 previews)
- per-provider quirks (function_calling latency, schema bikeshedding)
Switch to plain llm.invoke() and parse integers from the response with
a tightened regex (word-boundary, no decimal fractions). The prompt
now instructs the model to output ONLY the indices, which combined
with the regex is robust against prose-injection of small numbers.
Removes RelevanceResult Pydantic class, _invoke_structured, the
_BATCH_FAILED_PARSE sentinel, and the "all batches failed" branch
(all dead under the new contract). Updates tests to mock llm.invoke
directly. Tightens default batch_size to 5 and parallel batches to 10
based on benchmark runs against Ollama.
* docs: fix stale _filter_for_relevance docstring after text-parsing rewrite
* refactor: move engine module paths from settings DB to hardcoded registry
Engine implementation details (module_path, class_name, full_search_module,
full_search_class) are internal wiring, not user configuration. Storing them
in the settings DB created a security attack surface requiring blocklist
validation and route blocking.
Changes:
- New engine_registry.py with frozen dataclass entries for all 24 engines
- search_engines_config.py injects registry data after loading DB settings
- search_engine_factory.py passes engine_config to full search wrapper
- Remove ~52 module/class entries from 9 JSON defaults files
- Remove BLOCKED_SETTING_PATTERNS, is_blocked_setting(), and 4 call sites
- Remove absolute→relative normalization from module_whitelist.py
- Update docs, tests, and golden master
* fix: remove TestGetBlockedSettingsError that references removed function
The get_blocked_settings_error() function was removed as part of the
engine registry refactor. This test class was added on main after the
PR was created and wasn't caught by conflict resolution.
* fix: remove TestSaveSettingsPostBlockedSetting that tests removed blocking logic
BLOCKED_SETTING_PATTERNS and is_blocked_setting() were removed as part of
the engine registry refactor. This test was added on main and references
the now-removed blocking behavior.
* fix: inject ENGINE_REGISTRY into parallel/meta engine _get_search_config()
Both ParallelSearchEngine and MetaSearchEngine manually extract config
from settings_snapshot without going through search_config(). Since
module_path/class_name are no longer in the settings DB (they live in
the hardcoded registry), these engines would silently fail to discover
sub-engines on fresh installations.
Fix: inject ENGINE_REGISTRY values after extraction, matching the
pattern used in search_config().
Also fixes MetaSearchEngine's stale check for
"search.engine.auto.class_name" in settings_snapshot — this key no
longer exists in settings DB, so auto engine config would be skipped.
* fix: update tests for engine registry refactor
- test_whitelist_config_consistency: check ENGINE_REGISTRY instead of
JSON defaults (module_path/class_name no longer in defaults)
- test_meta_search_engine_high_value: expect registry-injected
module_path/class_name in _get_search_config() output
- test_meta_search_engine_extended: registry overwrites snapshot values
- test_settings_routes_coverage: remove blocked setting tests (blocking
logic removed — registry is now the security mechanism)
- test_settings_routes_deep_coverage2: same as above
* fix: add 5 missing engines to registry, strip module_path from their settings
Add gutenberg, openlibrary, pubchem, stackexchange, and zenodo to
ENGINE_REGISTRY (were added to main in #1540 after this branch diverged).
Remove module_path/class_name from their settings JSON files and golden
master, matching the pattern established for all other engines.
Expand test_engine_registry.py to scan per-engine settings_*.json files
and verify no settings files still contain module_path/class_name.
* fix: inject full_search_module/class in meta/parallel engine _get_search_config()
The registry injection in MetaSearchEngine and ParallelSearchEngine was
missing full_search_module and full_search_class fields, making it
inconsistent with the main search_config() injection. This would cause
full-search wrappers to fail when created through meta/parallel engines.
* fix: resolve pre-commit formatting issues and sync pdm.lock after merge with main
The in-process vLLM provider (requiring torch+transformers+vllm ~10GB) is
obsolete — vLLM is universally run as a server and accessed via its
OpenAI-compatible API, which the openai_endpoint provider already handles.
Removes vllm from: config, pricing, rate limiting, hardware warnings,
frontend dropdowns, pyproject.toml optional deps, docs, default_settings.json,
golden master, benchmark template, and all related tests (37 files, -436 lines).
Keeps vLLM mentions in openai_endpoint context (labels, docs) since that's
the correct usage path.
* cleanup: remove @pytest.mark.requires_llm decorators and fallback LLM doc references
Remove the `@pytest.mark.requires_llm` decorator from all test files since
the fallback LLM infrastructure is being removed. Update docs to remove
references to `LDR_TESTING_USE_FALLBACK_LLM` and `LDR_USE_FALLBACK_LLM`
environment variables from troubleshooting and CI configuration tables.
* test: remove fallback LLM references from test files
Remove all fallback-related test code: TestGetFallbackModel classes,
FakeListChatModel assertions, check_fallback_llm parameters, and
LDR_USE_FALLBACK_LLM skipif markers. Replace fallback-returning tests
with ValueError-expecting tests for missing API keys and unavailable
providers.
* cleanup: remove remaining use_fallback_llm references from source and tests
Remove use_fallback_llm() imports and calls from db_utils.py and
rate_limiting/tracker.py. Clean up test files that referenced
check_fallback_llm, get_llm_setting_from_snapshot, and
LDR_USE_FALLBACK_LLM env var.
* cleanup: remove remaining fallback LLM references from test files
Remove all use_fallback_llm mocks, LDR_USE_FALLBACK_LLM env var checks,
and related skip logic from test files since the fallback LLM feature
has been removed from source code.
- test_db_utils.py: Remove use_fallback_llm mock patches from 4 tests
- test_rate_limiter.py: Replace use_fallback_llm mock with is_ci_environment
- test_tracker.py: Replace fallback mode test with CI mode test
- test_tracker_quality_stats.py: Remove 8 use_fallback_llm decorators
- test_openai_api_key_usage.py: Remove LDR_USE_FALLBACK_LLM skipif
- test_llm_provider_integration.py: Remove LDR_USE_FALLBACK_LLM skipif
- test_ci_config.py: Remove LDR_USE_FALLBACK_LLM env var setting
- test_search_system.py: Remove LDR_USE_FALLBACK_LLM skipif
- run_all_tests.py: Remove LDR_USE_FALLBACK_LLM log line
- test_env_auto_generation.py: Remove testing.use_fallback_llm mapping
- test_lmstudio_provider.py: Fix docstring referencing removed function
* refactor: remove fallback LLM from providers, settings, CI, and tests
- Remove FakeListChatModel import and get_llm_setting_from_snapshot wrapper
- Update all provider imports to use get_setting_from_snapshot directly
- Remove LDR_USE_FALLBACK_LLM env var from CI workflows
- Remove use_fallback_llm setting and registry function
- Remove skip_if_using_fallback_llm fixture from conftest.py
- Update tests to expect ValueError instead of fallback model
* refactor: remove fallback model from llm_config and thread_settings
- Remove get_fallback_model() and all call sites in get_llm()
- Replace fallback returns with descriptive ValueError raises
- Remove LDR_USE_FALLBACK_LLM env check block from get_llm()
- Remove check_fallback_llm parameter from get_setting_from_snapshot
- Remove get_llm_setting_from_snapshot convenience wrapper
- Add ValueError re-raise in Ollama model-not-found path
- Regenerate golden master with ensure_ascii=False for proper Unicode
* fix: restore requires_llm skip mechanism and fix CI test failures
Three fixes for CI regressions from fallback LLM removal:
1. Restore @pytest.mark.requires_llm decorator and skip fixture
(skip_if_no_real_llm) that checks LDR_TESTING_WITH_MOCKS env var.
Re-add decorators to 17+ tests across 9 files that need real LLMs.
2. Fix type coercion in test_openai_api_key_usage.py by converting
fixture from dict format to simplified raw-value format, bypassing
get_typed_setting_value string coercion.
3. Fix golden master format mismatch by adding ensure_ascii=False to
test serialization to match regeneration script. Narrow pre-commit
hook trigger to only defaults/*.json files.
* fix: remove remaining fallback LLM references from coverage tests
- Delete TestGetFallbackModel class from test_llm_config_coverage.py
(5 tests that imported removed get_fallback_model)
- Update test_llm_config_missing_coverage.py: 6 tests that expected
FakeListChatModel fallback now expect ValueError/exception raises
- Remove use_fallback_llm mocks from test_rate_limiting_tracker_coverage.py
(delete 4 fallback-specific tests, fix 9 tests)
- Remove use_fallback_llm mocks from rate_limiting/test_tracker_coverage.py
(fix _make_tracker helper and 25 tests)
- Add @pytest.mark.requires_llm to test_analyze_documents_minimal
- Merge upstream main to pick up new coverage test files
* fix: remove dead LDR_USE_FALLBACK_LLM env var from accessibility tests CI
This env var was added to the accessibility test server but has no
effect since the fallback LLM code was removed.
* fix: align pre-commit hook description and error listing with defaults-only trigger
The hook file pattern was narrowed to defaults/ only, but the description
and error-listing code still referenced config/. Remove dead config/ path
from the file listing and update messaging to match.
* fix: update test_llm_config_deep_coverage.py for fallback LLM removal
File was added on main after branch diverged. Remove TestGetLlmFallbackEnvVar
class (tests removed functionality) and update test_provider_lowercased to
expect ValueError instead of fallback model.
* fix: improve "none" provider error message and fix stale CI-mode test
- Add explicit handler for provider="none" with user-friendly message
instead of misleading "this is a bug" error
- Fix test_load_estimates_skipped_in_ci_mode: _load_estimates no longer
checks is_ci_environment, test now correctly verifies deferred loading
behavior in non-programmatic mode
- Update 4 test assertions to match new "none" provider error message
* fix: remove dead serpapi full_search_module/class references
The serpapi engine pointed to `.engines.full_serp_search_results_old` with
class `FullSerpAPISearchResults`, but neither the module file nor the class
exist. All engines (including serpapi) use `.engines.full_search` /
`FullSearchResults`. Update defaults, golden master, docs, and remove the
stale whitelist entry.
* fix: normalize full_search_module in search_config()
search_config() normalized legacy absolute module_path values but skipped
full_search_module. Extend the normalization loop to cover both keys for
consistency with the defense-in-depth normalization in get_safe_module_class().
* fix: check full_search_module key in pre-commit hook
The pre-commit hook only validated module_path keys in JSON files. Extend it
to also check full_search_module, and add regression tests for both cases.
* fix: add debug logging for absolute module path normalization
When get_safe_module_class() normalizes an absolute path to relative form,
log the conversion at debug level for easier debugging of Docker user issues.
* refactor: eliminate server_config.json, make server settings env-var-only
Remove the JSON file-based server configuration and sync mechanism.
All 8 server settings (host, port, debug, HTTPS, allow_registrations,
and 3 rate limits) are now read exclusively from environment variables
via get_typed_setting_value() with the existing LDR_* naming convention.
- Rewrite server_config.py: remove get_server_config_path(),
save_server_config(), sync_from_settings(); simplify load_server_config()
to use get_typed_setting_value(key, None, ...) for all settings
- Add rate_limit_settings to the config dict (was only via .get() fallback)
- Remove sync_from_settings calls from 3 sites in settings_routes.py
- Hide server settings from UI (visible: false, editable: false) in
default_settings.json and settings_security.json
- Add security.rate_limit_settings entry to settings_security.json
- Fix swapped min/max on web.port (was min:65535, max:0)
- Update descriptions to reference env var names
- Rewrite test_server_config.py: remove 21 JSON-file tests, keep 13
defaults/fail-closed tests, add 8 env var override tests (35 total)
- Regenerate golden master settings
- Remove server_config.py from check-file-writes.sh exemption list
- Update docstrings in rate_limiter.py and app.py
* fix: address review findings for server_config.json elimination
- Fix save_all_settings response: return dict (keyed by setting key)
instead of list, matching GET /settings/api shape; include missing
visible, min_value, max_value, step fields so visibility filter works
- Fix JS consumer: use dict key access instead of .find() on response
- Fix docs: LDR_WEB_PORT is the correct env var for server bind port,
not LDR_APP_PORT; add clarifying note
- Remove stale KNOWN_NUMERIC_ISSUES entry for web.port (now fixed)
- Add tests: empty-string and whitespace env var edge cases for
allow_registrations fail-closed, and env-var override coverage
* feat: add deprecation migration path for server_config.json (#2549)
Users who set `allow_registrations: false` via the UI (persisted in
server_config.json) would silently lose that setting on upgrade,
re-enabling open registration. Docker users are especially at risk
since named volumes persist the file across container upgrades.
Add read-only migration: if server_config.json exists, honor its
values as fallbacks (env var > legacy file > default) and log
deprecation warnings guiding users to migrate to env vars.
No write-back logic is re-added — save_server_config() and
sync_from_settings() remain removed per the PR's intent.
* feat: show web UI warning when legacy server_config.json is detected
Adds a dismissible warning banner in the web interface when the
deprecated server_config.json file exists, using the existing
warning_checks system. Addresses reviewer feedback from PR #2505.
* fix: address review findings for server_config.json elimination
- Change web.host, web.port, web.use_https type from SEARCH to APP
in both default_settings.json and golden_master_settings.json
- Add 3 tests for check_legacy_server_config() covering dismissed,
missing file, and file-exists branches
- Add autouse fixture to clear LDR_APP_ALLOW_REGISTRATIONS env var
in test_server_config.py to prevent test pollution from dev shell
* fix: round 2 review findings for server_config.json elimination
- Fix flaky test_all_four_warnings_simultaneously by mocking
get_server_config_path to prevent real server_config.json on disk
from breaking exact set equality assertion
- Add dismiss_legacy_config to _make_settings_manager defaults and
rename test_all_six_settings_read → test_all_seven_settings_read
- Add orchestrator-level tests for legacy_server_config warning
(exists/absent/dismissed scenarios)
- Add fail-closed guard for legacy JSON allow_registrations string
values (e.g. "disabled" → False) to match env var guard
- Log warning for unrecognized keys in legacy server_config.json
to surface typos like "Port" instead of "port"
- Regenerate CONFIGURATION.md to remove stale server_config.json
reference in app.debug description
* fix: round 3 review findings — test quality and migration docs
- Replace vacuous `is not None` assertion with meaningful env-var-vs-legacy
guard priority test using unrecognized values on both paths
- Add positive test for DEPRECATED banner when recognized keys present
- Rename misleading test name to reflect actual scope (hardware + context)
- Add migration section to env_configuration.md for server_config.json users
* security: gate global scheduler control behind setting
The news scheduler is a global singleton — starting, stopping, or
triggering it affects all users. Add a setting to control whether
these operations are accessible via API.
- Add `news.scheduler.allow_api_control` setting (default: true)
- Env var: LDR_NEWS_SCHEDULER_ALLOW_API_CONTROL
- Also configurable via settings UI
- Add `@scheduler_control_required` decorator that checks the setting
- Apply to destructive endpoints: start, stop, check-now, cleanup-now
- Read-only endpoints (status, users, stats) remain accessible to
any authenticated user
Multi-user deployments can set `LDR_NEWS_SCHEDULER_ALLOW_API_CONTROL=false`
to prevent any user from starting/stopping the global scheduler.
* test: add tests for scheduler_control_required decorator
Tests cover:
- Decorator allows execution when setting is enabled
- Decorator returns 403 when setting is disabled
- Error response includes informative message
- Correct setting key is checked
- Function name is preserved (wraps)
* fix: make scheduler API control setting non-editable for security
The news.scheduler.allow_api_control setting controls a global security
boundary (the scheduler singleton affects all users). Following the
precedent set by app.allow_registrations, this setting should not be
editable from the UI — it must be configured via environment variable
LDR_NEWS_SCHEDULER_ALLOW_API_CONTROL only.
Also adds integration tests verifying that mutating scheduler endpoints
(start, stop, check-now, cleanup-now) return 403 when disabled, while
read-only endpoints (status, users, stats) remain accessible.
* fix: change allow_api_control default from true to false
Make scheduler API control secure-by-default. Since the setting is
editable: false (env-var-only), no existing UI state is affected.
Users who want API control must now explicitly opt in via
LDR_NEWS_SCHEDULER_ALLOW_API_CONTROL=true.
* fix: regenerate config docs and add audit logging for scheduler gate
Regenerate CONFIGURATION.md to include the new
news.scheduler.allow_api_control setting (fixes check-config-docs CI).
Add logger.warning when scheduler API control is blocked, for audit
trail in multi-user deployments.
* fix: improve 5 setting descriptions and widen tooltip for readability
- knowledge_accumulation: list correct enum values (ITERATION, QUESTION,
NO_KNOWLEDGE), omit dead MAX_NR_OF_CHARACTERS option
- knowledge_accumulation_context_limit: note that value is loaded but
not actively used at runtime
- max_filtered_results: clarify filtering pipeline without jargon
- enable_web: replace implementation detail with user-facing language
- dismiss_model_mismatch: clarify 70b name-based detection
- Widen tooltip from fixed 200px to max-width 300px with 200px mobile cap
- Regenerate golden master and CONFIGURATION.md
* fix: address review — revert model mismatch description, hide unused context limit
Revert dismiss_model_mismatch description to original short form since
'70b' substring detection is unreliable. Hide knowledge_accumulation_context_limit
from UI (visible: false) as the setting is loaded but never enforced at runtime.
