8 Commits

Author SHA1 Message Date
devin ivy
af7d3912a3
Starter packs (#2553)
* lexicon: initial starter pack record lexicons

* lexicon: first pass on views for starter packs

* lexicon: add starter pack query endpoints, misc fixes

* lexicon: add starter pack related fields to profile views

* lexicon: codegen

* api prerelease

* bsky: protos for starter packs

* appview: implement starter pack endpoints in dataplane, protos update

* appview: starter pack hydration and view, mock dataplane indexing

* appview: wire-up starter pack graph endpoints

* appview: test basic starter packs functionality, update dev-env

* fix snaps for profile createdAt

* appview: fix list item count for starter packs in dataplane

* fix ozone snaps, tidy

* appview: reorg list aggs for starter packs

* appview: starter pack lexicon and proto tweaks, move around aggregations

* appview: fix views and snaps

* api prerelease

* fix snaps

* appview: misc proto updates for starter packs, snaps

* appview: add starter pack counts to profile detail

* build

* api prerelease

* pds: starter pack slur detection

* lexicon: starter pack notifications and batch fetch

* appview: starter pack notification and batch fetch

* appview: add starterpack-joined notif to dataplane and test

* api prerelease

* ozone: fix snaps

* pds build

*  Proxy starterpack requests through ozone and include takedowns (#2594)

*  Proxy starterpack requests through ozone and include takedowns

* 🧹 Cleanup

*  Commit snapshots

*  Update snapshot

* lexicon: move starterpacks back to TIDs

* changeset

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-06-24 21:04:33 -04:00
devin ivy
499fabe7b6
Appview: add chat details to profile views, chronological likes (#2471)
* update bsky protos

* appview: add associated chat deets to profile views, switch to sorted likes

* appview: test associated chat info on profile views

* tidy
2024-05-09 12:45:06 -04:00
devin ivy
203e72f1c1
Appview preset image URLs (#1248)
* update image signing logic to presets on bsky

* update bsky config and views for preset img urls

* update bsky image tests for preset urls

* update bsky snapshots for preset image urls

* update pds proxy tests for bsky preset image urls

* tweak image paths
2023-08-09 10:21:43 -04:00
Daniel Holmgren
08fbb33fdf
Appview blocks (#1187)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* hook up indexing

* Setup custom feeds on bsky

* graph service + getList routes

* migration

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* Test custom algos on bsky

* add mute state to snapshots

* table & indexing plugin

* getBlocks

* bail out of mute subquery when no viewer

* fleshing out services

* Test pds getFeed proxy

* tidy, fix build

* tidy

* Fix minor race in bsky mutelist tests

* blocks in feed views

* db migration, impl block logic & tests

* Update pds proxy snaps based on appview mutes

* missed some files on last commit

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

* apply blocks in getFeed

* update snaps

* upadate one last snap

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 11:05:22 -05:00
devin ivy
7981c5948f
Misc. syncing-up bsky appview with on-pds appview (#1189)
* Misc notification and optimization in bsky, tidy

* Handle notifs on message queue in bsky appview

* Sync-up profile labeling behavior in bsky

* No async work during actor unindexing in bsky, include additional indexes

* Prevent deeply nested notifs w/ hellthread exception in bsky appview

* Add indexes to bsky appview for more efficient actor deletion/unindexing

* Update pds proxy snaps for updated bsky profile labeling

* Return custom suggestions in bsky appview
2023-06-11 14:55:43 -04:00
Daniel Holmgren
4f2f3d8d17
Appview mutes (#1184)
* hook up indexing

* graph service + getList routes

* migration

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* add mute state to snapshots

* bail out of mute subquery when no viewer

* Fix minor race in bsky mutelist tests

* Update pds proxy snaps based on appview mutes

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-10 17:12:15 -04:00
devin ivy
47890d65b9
Add labeler functionality to the appview (#851)
* Add moderation and labeling model to bsky appview

* Carry over moderation service from pds

* Pass over bsky moderation service to get it working, missing blob support

* Carry over label service from pds to bsky

* Reorg bsky http auth, implement admin auth

* Carry over relevant admin endpoints pds to bsky

* Retrofit bsky admin routes from pds impls

* Implement resolve handle on appview

* Avoid loop in bsky handle resolution

* Add bsky appview to dev-env, opt-in

* Fix bsky searchRepos with empty term, tidy

* Include blobs on bsky admin views

* Stop resolving taken-down blobs on appview

* Tidy

* Carry over pds moderation tests to bsky

* Support image cache invalidation on bsky appview

* Add missing changes for bsky tests

* Test takedowns in bsky views

* Test takedowns on bsky notifs

* Separate blob resolution logic into standalone helper

* Carry over labeler from pds to bsky

* Update labelers for bsky usage

* Configure labeler on appview, tidy blob-resolver

* Add labels to views for posts and actors

* Update snapshots on bsky appview to include labels

* Carry over labeler tests from pds to bsky

* Tidy-up labeler tests for bsky appview

* Update bsky seeds and snapshots for keyword labeling

* Tidy

* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.

* Include labels on appview listNotifications, fix blob resolver content-type, tidy
2023-04-23 23:58:10 -04:00
devin ivy
4e83748545
Bsky app view (#716)
* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect

* Cull bsky config, services, auth, etc.

* Sweep app view xrpc methods, tidy deps, add storage back for img server

* Run repo subscription on bsky app view

* Collapse db migrations down for bsky app view

* Tidy app view bin

* Remove mute functionality from app view, delegate to pds

* Initial tidy/culling of bsky app view tests

* Passing bsky app view db, server, and repo subscription tests

* Passing bsky app view duplicate-records tests

* Bsky app view test tidy/cull

* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.

* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.

* Setup actor handles by did in bsky app view

* Passing indexing tests on bsky app view

* Passing image tests on bsky app view

* Fix bsky actor reindexing, support custom lock id for testing repo subs

* Sweep bsky view tests, misc tests, passing

* Tidy bsky deps

* Include did in resized image uris

* Update bsky image process server to use getBlob

* Update image server tests, misc fixes

* Implement bsky blob resolver

* Wire local image processing server to local blob resolver, test blob resolver

* Tidy

* Tidy

* Tidy

* Tidy app view init

* Fix handle resolution, tidy

* Add utils for partitioning indexing by did

* Update repo sub to parallelize work per repo

* Dep tidy

* Tidy bsky tests for updated repo sub destroy()

* Update thead indexing to handle out-of-order posts

* Sketch out strategy in bsky for handling too-big commits

* Set content-type on sync.getBlob

* Add logging for failed transmissions in bsky blob resolver

* Tidy

* Tidy bsky repo indexing and supporting repo interfaces

* Sort in app view based on combo of creation and indexing times

* Fix types

* Add retry utils to bsky

* Add retries to http requests made by bsky

* Test repo indexing

* Update bsky db/model for lex refactor

* Update bsky lexicons for lex refactor

* Update bsky actor service for lex refactor

* Update bsky feed service for lex refactor

* Update bsky indexing service for lex refactor

* Update bsky repo subscription for lex refactor

* Tidy bsky repo sub

* Add unspecced endpoints to bsky app view, update entrypoint

* Update bsky xrpc utils for lex refactor

* Update bsky xrpc methods for lex refactor

* Update bsky test seeds for lex refactor, tidy api entrypoint

* Update bsky non-view tests for lex refactor

* Update bsky likes view test for lex refactor, minor fix

* Update bsky author feed tests for lex refactor, minor test util fix

* Update bsky follow, profile, repost, search view tests for lex refactor

* Update bsky timeline view tests for lex refactor

* Replace bsky out-of-order thread indexing logic

* Update bsky thread view tests for lex refactor, general test tidying

* Handle rebases and too-big commits in repo subscription, tracking commit data cid

* Tidy

* Ensure did resolver reports "not found" only when positively not found

* Handle tombstones and handle updates in bsky

* Test indexing handle updates and did tombstones

* Support cors on bsky

* Allow app view to serve most routes unauthed

* Tests for bsky unauthed views

* Tidy bsky service entrypoint and dockerfile

* Remove unused storage interfaces from bsky

* Bsky entrypoint and dockerfile fixes, tidy

* Add workflow for bsky build to aws

* Use more standard db env variables, make migration creds optional

* Make bsky repo subscription optional

* Fix lex->json serialization in bsky

* Split bsky actor sync state into its own table

* Skip invalid records on indexing full repo, tidy

* Tidy

* Leader test timing

* Tidy/lint

* Fix bsky config overrides
2023-04-06 18:47:46 -04:00