joseph/internal-joseph
1
0
Fork 0
Code Issues 21 Pull requests Projects Releases Packages Wiki Activity Actions

[OneDev #4] feat(tooling): refine X (Twitter) bookmarks scanning + scraping #28

New issue
Closed
opened 2026-05-24 04:46:39 +00:00 by joseph · 1 comment
joseph commented 2026-05-24 04:46:39 +00:00
Owner
Copy link

Imported from OneDev issue #4 (id 35180)
Original project: internal-joseph
Original state: Closed
Original submitterId: 5
Original submitDate: 2026-04-08T14:37:18.497+00:00
Original lastActivity: {'date': '2026-04-18T12:22:44.866+00:00', 'description': "changed state to 'Closed'", 'userId': 5}

Original fields:

  • Type: New Feature
  • Priority: Normal
  • Assignees: None

Idea

Improve the X bookmarks ingestion pipeline — make it more reliable, more useful, and more queryable.

Likely improvements (refine after a current-state pass)

  • Auth flow stability (X breaks scrapers regularly — expect cookie/session refresh logic)
  • Incremental scraping (only new bookmarks since last run, not full re-pull)
  • Normalize into a structured store (SQLite? JSONL?)
  • Tag / topic extraction via local LLM (Lemonade Coder-30B)
  • Search interface — CLI or web UI tied into existing dashboards
  • Backup/export so a banned account doesn't lose history

Next session handoff notes

  • TBD by Joseph: where the current scraper lives (repo + last-run state)
  • TBD by Joseph: which of the improvements above are highest leverage right now
  • Likely uses Lightpanda or Playwright (both already deployed on VMID 128)

Why

Bookmarks are a high-signal personal knowledge stream that's currently locked inside X's UI. Every refinement compounds.

Imported from OneDev issue #4 (id 35180) Original project: internal-joseph Original state: Closed Original submitterId: 5 Original submitDate: 2026-04-08T14:37:18.497+00:00 Original lastActivity: {'date': '2026-04-18T12:22:44.866+00:00', 'description': "changed state to 'Closed'", 'userId': 5} Original fields: - Type: New Feature - Priority: Normal - Assignees: None ## Idea Improve the X bookmarks ingestion pipeline — make it more reliable, more useful, and more queryable. ## Likely improvements (refine after a current-state pass) - Auth flow stability (X breaks scrapers regularly — expect cookie/session refresh logic) - Incremental scraping (only new bookmarks since last run, not full re-pull) - Normalize into a structured store (SQLite? JSONL?) - Tag / topic extraction via local LLM (Lemonade Coder-30B) - Search interface — CLI or web UI tied into existing dashboards - Backup/export so a banned account doesn't lose history ## Next session handoff notes - **TBD by Joseph**: where the current scraper lives (repo + last-run state) - **TBD by Joseph**: which of the improvements above are highest leverage *right now* - Likely uses Lightpanda or Playwright (both already deployed on VMID 128) ## Why Bookmarks are a high-signal personal knowledge stream that's currently locked inside X's UI. Every refinement compounds.
joseph commented 2026-05-24 04:46:48 +00:00
Author
Owner
Copy link

Imported from OneDev comment id 372 on issue #4
Original userId: 5
Original date: 2026-04-18T12:22:43.866+00:00

Closing as out of scope for this tracker — X bookmarks is a personal project, not PCT infrastructure. Can be reopened in a personal workspace if pursued.

Imported from OneDev comment id 372 on issue #4 Original userId: 5 Original date: 2026-04-18T12:22:43.866+00:00 Closing as out of scope for this tracker — X bookmarks is a personal project, not PCT infrastructure. Can be reopened in a personal workspace if pursued.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
joseph/internal-joseph#28
No description provided.