geckon01/local-deep-research
1
0
Fork 0
mirror of https://github.com/LearningCircuit/local-deep-research.git synced 2026-06-15 19:46:56 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
ba0912056c5b78bf2c8a1fb15eef202d72fec401
local-deep-research/.github/workflows/playwright-webkit-tests.yml
LearningCircuit e6d72faa14 test(embedding-settings): regression spec for model dropdown reset (#3863) (#3949) 
* test(embedding-settings): regression spec for model dropdown reset (#3863)

Adds a Playwright spec that mocks the embedding-settings backend
(`/library/api/rag/models`, `/library/api/rag/settings`,
`/settings/api/local_search_embedding_model`,
`/settings/api/embeddings.ollama.url`) so the page renders without a real
Ollama or Sentence-Transformers backend. Three tests cover the bug from
#3863 and the surrounding contract:

1. selecting a non-top model auto-saves it and persists across reload —
   exercises the change-listener path that the per-field auto-save relies
   on.
2. Ollama URL change does not reset the selected model — the
   load-bearing regression test. Reverting the preserve+restore patch in
   `updateModelOptions()` was confirmed to make this test fail with the
   same symptom the issue reporter saw (dropdown snaps to the index-0
   model).
3. "Save Default Settings" button is gone — guards against an accidental
   re-introduction of the redundant button that originally triggered the
   bug.

Mock route registration order matters here: the catch-all
`**/settings/api/**` is registered first so the specific PUT mocks for
`local_search_embedding_model` etc. (registered later) win Playwright's
last-registered-wins precedence.

* test(embedding-settings): address review findings on regression spec

Round-2 review of PR #3949 surfaced that the spec was not actually wired
into any CI workflow and had a couple of correctness/flakiness issues.

- Add `embedding-settings-dropdown` to the curated Safari filename filter
  in `.github/workflows/playwright-webkit-tests.yml` (both Desktop Safari
  and Mobile Safari runs) so the daily release-gate Playwright run picks
  up this regression spec.
- Replace `waitForTimeout(500)` with a deterministic poll on a new
  `state.modelsFetches` counter that ticks on each `/library/api/rag/models`
  GET. Once the count rises past the pre-action baseline, the post-save
  `loadAvailableModels()` call has completed and any spurious save the
  bug would trigger has already fired.
- Gate the `local_search_embedding_model` and `local_search_embedding_provider`
  PUT mocks on request method, mirroring the `embeddings.ollama.url`
  handler. A stray GET hitting these handlers would otherwise push
  `undefined` into `state.modelSaves`/`providerSaves`.
- Skip the entire describe block on mobile projects (`test.skip(({ isMobile })
  => isMobile, ...)`). This is a desktop form-state regression test, not
  a layout test — it doesn't need to run across 12 device profiles.
- Reframe test #1's docstring: it's a happy-path smoke test for the
  per-field auto-save contract, not a regression test for #3863. The old
  comment claimed it would catch the bug, which I confirmed empirically
  it does not (the dropdown-rebuild path that #3863 exploited isn't on
  the model-pick-and-reload flow).
- Add a new test for the provider-change path (`updateModelOptions` is
  also reached from the provider-change handler at line 325 of
  `embedding_settings.js`). The model-shared-across-providers fixture
  forces the preserve+restore branch in `updateModelOptions` to fire and
  asserts the selection survives. Verified locally: this test fails
  alongside the Ollama-URL test when the patch is reverted.

* test(embedding-settings): extract mocks helper + cover text_separators reset

Round-2 review left two advisory items: extract the mock infrastructure
to a shared helper, and cover the text_separators reset behavior added in
the follow-up commit on PR #3940 (40678b2). Both are addressed here.

- Move BASE_MODELS_PAYLOAD, defaultSettings, and mockEmbeddingApis to
  `tests/ui_tests/playwright/tests/helpers/embedding-settings-mocks.js`,
  matching the CommonJS pattern used by `mobile-utils.js` so additional
  embedding-page specs can reuse the same backend mocks. Also export
  DEFAULT_TEXT_SEPARATORS so callers can assert against it without
  duplicating the literal.
- Add `state.textSeparatorsSaves` and a method-gated PUT route mock for
  `/settings/api/local_search_text_separators` to capture saves.
- New test: clearing the text_separators textarea and blurring persists
  the default array. Sanity-checked locally by reverting the
  empty-textarea branch in `embedding_settings.js:395-415` to its
  pre-#3940 `if (!rawValue) return;` state — the new test fails (no PUT
  fired); restoring the fix makes it pass again.

* ci: drop embedding-settings-dropdown from Mobile Safari filter

The spec uses `test.skip(({ isMobile }) => isMobile, ...)` at the
describe level, so on Mobile Safari Playwright would load the file and
skip every test — harmless but noisy. Keep it in the Desktop Safari
filter only, where it actually runs.
2026-05-10 08:11:19 +02:00

292 lines
9.8 KiB
YAML
Raw Blame History

name: Playwright WebKit Tests
on:
workflow_call: # Called by release pipeline
workflow_dispatch: # Manual trigger
schedule:
- cron: '0 2 * * *' # Daily at 2 AM UTC
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
# Shared setup: start server, register user, then run Desktop Safari tests
desktop-safari:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@a5ad31d6a139d249332a2605b85202e8c0b78450 # v2.19.1
with:
egress-policy: audit
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
version: 2.26.2
cache: true
- name: Install Python dependencies
run: |
pdm install -d
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
cache: 'npm'
cache-dependency-path: |
package-lock.json
tests/ui_tests/playwright/package-lock.json
- name: Build Vite frontend assets
run: |
npm ci
npm run build
- name: Install Playwright dependencies
working-directory: tests/ui_tests/playwright
run: |
npm ci
# Install chromium for auth setup step + webkit for Safari tests
npx playwright install chromium webkit --with-deps
- name: Setup test directories
run: |
mkdir -p data/encrypted_databases
echo "Created data directories for tests"
- name: Start LDR server
env:
CI: true
TEST_ENV: true
FLASK_ENV: testing
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DISABLE_RATE_LIMITING: true
SECRET_KEY: test-secret-key-for-ci
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
pdm run python -m local_deep_research.web.app &
SERVER_PID=$!
echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
# Wait for server to be ready.
# --connect-timeout/--max-time bound TCP and total request time so a
# hung connection fails fast instead of eating the job timeout.
for i in {1..30}; do
if curl -s --connect-timeout 2 --max-time 5 http://127.0.0.1:5000 > /dev/null; then
echo "Server is ready"
break
fi
echo "Waiting for server... ($i/30)"
sleep 2
done
# Verify server is running
if ! curl -s --connect-timeout 2 --max-time 5 http://127.0.0.1:5000 > /dev/null; then
echo "Server failed to start"
exit 1
fi
- name: Initialize test database
env:
TEST_ENV: true
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DB_KDF_ITERATIONS: "1000"
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
pdm run python scripts/ci/init_test_database.py
- name: Register CI test user
working-directory: tests/ui_tests
run: |
npm ci
node register_ci_user.js http://127.0.0.1:5000
- name: Run Desktop Safari Tests
working-directory: tests/ui_tests/playwright
env:
CI: true
TEST_BASE_URL: http://127.0.0.1:5000
TEST_USERNAME: test_admin
TEST_PASSWORD: testpass123
# Safari CI runs a critical subset for fast release gate:
# - all-pages-mobile: overflow, touch targets, nav clearance across all pages
# - auth-pages-mobile: login, register, change-password mobile layouts
# - desktop-layout: sidebar positioning and breakpoint testing
# - interactive-states: advanced options, search, help panels, mobile menus
# - settings-subpages-mobile: settings tab navigation and form layouts
# - mobile-ui-audit: comprehensive mobile UI quality checks
# - embedding-settings-dropdown: regression for #3863 (model dropdown reset)
# Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
# When adding Safari-critical tests, update this filter pattern.
run: |
npx playwright test \
--project="Desktop Safari" \
"all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit|embedding-settings-dropdown"
- name: Upload Playwright Report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: playwright-report-desktop-safari
path: |
tests/ui_tests/playwright/playwright-report/
tests/ui_tests/playwright/test-results/
- name: Stop server
if: always()
run: |
if [ -n "$SERVER_PID" ]; then
kill "$SERVER_PID" || true
fi
pkill -f "python -m local_deep_research.web.app" || true
# Mobile Safari tests run in parallel on a separate runner
mobile-safari:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@a5ad31d6a139d249332a2605b85202e8c0b78450 # v2.19.1
with:
egress-policy: audit
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
version: 2.26.2
cache: true
- name: Install Python dependencies
run: |
pdm install -d
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
cache: 'npm'
cache-dependency-path: |
package-lock.json
tests/ui_tests/playwright/package-lock.json
- name: Build Vite frontend assets
run: |
npm ci
npm run build
- name: Install Playwright dependencies
working-directory: tests/ui_tests/playwright
run: |
npm ci
# Install chromium for auth setup step + webkit for Safari tests
npx playwright install chromium webkit --with-deps
- name: Setup test directories
run: |
mkdir -p data/encrypted_databases
echo "Created data directories for tests"
- name: Start LDR server
env:
CI: true
TEST_ENV: true
FLASK_ENV: testing
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DISABLE_RATE_LIMITING: true
SECRET_KEY: test-secret-key-for-ci
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
pdm run python -m local_deep_research.web.app &
SERVER_PID=$!
echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
# Wait for server to be ready.
# --connect-timeout/--max-time bound TCP and total request time so a
# hung connection fails fast instead of eating the job timeout.
for i in {1..30}; do
if curl -s --connect-timeout 2 --max-time 5 http://127.0.0.1:5000 > /dev/null; then
echo "Server is ready"
break
fi
echo "Waiting for server... ($i/30)"
sleep 2
done
# Verify server is running
if ! curl -s --connect-timeout 2 --max-time 5 http://127.0.0.1:5000 > /dev/null; then
echo "Server failed to start"
exit 1
fi
- name: Initialize test database
env:
TEST_ENV: true
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DB_KDF_ITERATIONS: "1000"
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
pdm run python scripts/ci/init_test_database.py
- name: Register CI test user
working-directory: tests/ui_tests
run: |
npm ci
node register_ci_user.js http://127.0.0.1:5000
- name: Run Mobile Safari Tests
working-directory: tests/ui_tests/playwright
env:
CI: true
TEST_BASE_URL: http://127.0.0.1:5000
TEST_USERNAME: test_admin
TEST_PASSWORD: testpass123
# Safari CI runs a critical subset for fast release gate:
# - all-pages-mobile: overflow, touch targets, nav clearance across all pages
# - auth-pages-mobile: login, register, change-password mobile layouts
# - desktop-layout: sidebar positioning and breakpoint testing
# - interactive-states: advanced options, search, help panels, mobile menus
# - settings-subpages-mobile: settings tab navigation and form layouts
# - mobile-ui-audit: comprehensive mobile UI quality checks
# Note: embedding-settings-dropdown is desktop-only (skips on isMobile),
# so it intentionally only appears in the Desktop Safari filter above.
# Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
# When adding Safari-critical tests, update this filter pattern.
run: |
npx playwright test \
--project="Mobile Safari" \
"all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit"
- name: Upload Playwright Report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: playwright-report-mobile-safari
path: |
tests/ui_tests/playwright/playwright-report/
tests/ui_tests/playwright/test-results/
- name: Stop server
if: always()
run: |
if [ -n "$SERVER_PID" ]; then
kill "$SERVER_PID" || true
fi
pkill -f "python -m local_deep_research.web.app" || true
Reference in New Issue View Git Blame Copy Permalink