rushabh/SimpleScraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add profile endpoint and validation

Browse Source
This commit is contained in:
Rushabh Gosar
2025-12-29 00:45:13 -08:00
parent 711d87a998
commit 68805ed80a
3 changed files with 600 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
AGENTS.md
View File

@@ -16,6 +16,14 @@
- If `strikeLimit` is greater than available strikes, all available rows are returned.
- `pruned_calls_count` and `pruned_puts_count` report how many rows were removed beyond the limit.
- `selected_expiration` reports the resolved expiry (epoch + label), and mismatches return an error.
- Route: `GET /profile`
- Query params:
- `stock`: symbol (default `MSFT`).
- Behavior:
- Loads `https://finance.yahoo.com/quote/<SYMBOL>/` with Playwright.
- Pulls the embedded SvelteKit payloads (quoteSummary, quote, quoteType, ratings, recommendations).
- Parses rendered DOM for recent news and the ticker news summary.
- Returns company profile, key stats, earnings, analyst data, performance, and news in JSON.
## Guard Rails
- Run local 10-cycle validation (4 stocks x 4 expiries) before any deploy or push.
@@ -28,9 +36,26 @@
- Local server:
- Start: `.\venv\Scripts\python.exe scraper_service.py`
- Validate: `python scripts/test_cycles.py --base-url http://127.0.0.1:9777/scrape_sync`
- Profile validation (local server):
- Validate: `python scripts/test_profile_cycles.py --base-url http://127.0.0.1:9777/profile --runs 8`
- Docker server:
- Start: `docker run --rm -p 9777:9777 rushabhtechie/yahoo-scraper:latest`
- Validate: `python scripts/test_cycles.py --base-url http://127.0.0.1:9777/scrape_sync`
- Profile validation (docker server):
- Validate: `python scripts/test_profile_cycles.py --base-url http://127.0.0.1:9777/profile --runs 8`
## Update Log (2025-12-28)
- Added `/profile` endpoint backed by SvelteKit payload parsing (quoteSummary, quote, quoteType, ratings, recommendations) plus rendered news extraction.
- Response now includes company profile, key stats, earnings, analyst trends, performance overview, ticker news summary, and recent news items.
- Validation added to ensure quote data matches the requested symbol, with issues reported in `validation`.
- Issue encountered: existing server instance bound to port 9777 without `/profile`, resolved by restarting the service with the updated script.
- Tests executed (local):
- `.\venv\Scripts\python.exe scripts\test_profile_cycles.py --runs 8 --timeout 180`
- `.\venv\Scripts\python.exe scripts\test_cycles.py --base-url http://127.0.0.1:9777/scrape_sync`
- Tests executed (docker):
- `docker build -t rushabhtechie/yahoo-scraper:latest .`
- `.\venv\Scripts\python.exe scripts\test_cycles.py --base-url http://127.0.0.1:9777/scrape_sync`
- `.\venv\Scripts\python.exe scripts\test_profile_cycles.py --base-url http://127.0.0.1:9777/profile --runs 8 --timeout 180`
- The test harness verifies:
- Requested expiration matches `selected_expiration.value`.
- Contract symbols include the expected YYMMDD code.