* mv appview
* copy
* finalize copy
* package names
* big WIP
* first pass at mod servce
* some tidy
* tidy & fix compiler errors
* rename to ozone, db migrations, add to dev-env & pds cfg
* getRecord & getRepo mostly working
* fix open handle
* get record tests all working
* moderation events working
* statuses working
* tidy test suite
* search repos
* server & db tests
* moderation tests
* wip daemon + push events
* pds fanout working
* fix db test
* fanning takedowns out to appview
* rm try/catch
* bsky moderation test
* introduce mod subject wrappers
* more tidy
* refactor event reversal
* tidy some db stuff
* tidy
* rename service to mod-service
* fix test
* tidy config
* refactor auth in bsky
* wip patching up auto-mod
* add label ingester in appview
* fix a couple build issues
* fix some timing bugs
* tidy polling logic
* fix up tests
* fix some pds tests
* eslint ignore
* fix ozone tests
* move seeds to dev-env
* move images around
* fix db schemas
* use service auth admin reqs
* fix remaining tests
* auth tests bsky
* another test
* random tidy
* fix up search
* clean up bsky mod service
* more tidy
* default attempts to 0
* tidy old test
* random tidy
* tidy package.json
* tidy logger
* takedownId -> takedownRef
* misc pr feedback
* split daemon out from ozone application
* fix blob takedown mgiration
* refactor ozone config
* do push event fanout on write instead of on read
* make suspend error work again
* add attempts check & add supporting index
* fix takedown test ref
* get tests working
* rm old test
* fix timing bug in event pusher tests
* attempt another fix for timing bug
* await req
* service files
* remove labelerDid cfg
* update snaps for labeler did + some cfg changes
* fix more snaps
* pnpm i
* build ozone images
* build
* make label provider optional
* fix build issues
* fix build
* fix build
* build pds
* build on ghcr
* fix syntax in entry
* another fix
* use correct import
* export logger
* remove event reverser
* adjust push event fanout
* push out multiple
* remove builds
* Include own replies to own post roots in post_no_replies
* Updates
* Simplify
* snaps
* snaps
* Remove unused import
* Add new threads filter to getAuthorFeed
* Implement new filter
* Typo
* rm tables
* rm event-stream & proxied
* Remove appview services, move label service to pds
* only proxy appview stuff
* delete more tables
* Start removing message dispatched from pds
* more syncing-up removal of message dispatcher in pds
* merged
* remove feedgens from pds, remove getPopular
* remove unused image helper from pds
* fixing compiler errors
* clean up sharp
* rm label service
* first pass on cleaning up tests
* fix up a bunch of tests
* moderation view tests
* last admin tests
* got a lil overzealous in deletes
* clean up unused cfg
* clean up label table
* simplify admin repo search query/logic
* tidy pds entrypoint
* in-progress pds config changes
* cfg fiddling
* finish cleaning up cfg/ctx
* comments
* building
* pds prefix on env
* test env
* collapse pds migrations down into a single migration
* fix up dev-env
* tidy
* cleanup
* fix pds admin tests
* fix handle test
* fix pds proxy tests
* fix subscribe repos test
* fix sqlite config in pds tests
* add sqlite clause in sequencer-leader
* fix actor search w/ sqlite on pds
* fixes
* fix dev env build
* update pds service entrypoint
* simple env example
* make takedown ids opaque identifiers in the pds
* use pds routes for api tests
* update pds dockerfile with volume and correct port env var
* add a couple env vars to example
* add comments to env example
* @atproto/pds 0.2.0-beta.0
* @atproto/aws 0.0.1-beta.0
* appview did
* @atproto/aws 0.0.1
* enable logs by default
* update env example
* bugfixing sandbox issues
* consistency in pds env var name for appview url
* log on pds start and stop, configure version at runtime
* @atproto/pds 0.2.0-beta.1
* fix semver matching for pds beta version
* v0.2.0-beta.2
* default invites to being not required
* fix flaky test
* limit db connections in tests
* publish 0.2.0-beta.d3
* fix invite required parsing
* @atproto/pds 0.2.0-beta.5
* Proxy getPopularFeedGenerators on simplified pds (#1222)
proxy getPopularFeedGenerators on pds
Co-authored-by: dholms <dtholmgren@gmail.com>
* tidy migrations
* fix service entry
* bump version
* change auth order
* bump version
* bump version
* add upgradeRepoVersion & fallback url for cdn
* bump version
* merging
* merge pds
* building dev-env
* merging tests
* merge service entry
* test fixing
* tidy
* fix admin search
* tidy
* tidy
* add snap for getListFeed
* add backup nameserver cfg
* tidy + pr feedback
* tidy
* tidy env
* bit more
* re-add dotenv to root package.json
* fix dep
* build branch
* fix tests
* Refactor tests to make better use of dev-env (#1690)
* refactor pds tests to use dev env
* refactor bsky tests
* fix pds test
* tidy bsky tests
* build pds correctly
* fix entry point
* default logging to false (for now)
* format service entry
* Switch takedown ids back to ints on pds distribution (#1694)
* switch takedown ids back to ints, consistent with live pds
* tidy/fix migration
* update migration for sqlite
* export moderation action reversal
* takedown tests
* dont build branch
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* initial lexicons for interaction-gating
* add interactions view to post views
* codegen
* model bad reply/interaction check state on posts
* initial impl for checking bad reply or interaction on write
* omit invalid interactions from post thread
* support not-found list in interaction view
* hydrate can-reply state on threads
* present interaction views on posts
* misc fixes, update snaps
* tidy/reorg
* tidy
* split interaction gating into separate record in lexicon
* switch interaction-gating impl to use separate record type
* allow checking reply gate w/ root post deletion
* fix
* initial gating tests
* tighten gated reply views, tests
* reply-gating list rule tests
* allow custom post rkeys within window
* hoist actors out of composeThread()
* tidy
* update thread gate lexicons, codegen
* lex fix
* rename gate to threadgate in bsky, update views
* lex fix
* improve terminology around reply validation
* fix down migration
* remove thread gates on actor unindexing
* add back .prettierignore
* tidy
* run ci on all prs
* syntax
* run ci on all prs
* format
* fix snap
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* lex
* list purpose
* lex gen
* add route
* add proxy route
* seed client helpers
* tests
* mutes and blocks
* proxy test
* snapshot
* hoist actors out of composeThread()
* tidy
* tidy
* run ci on all prs
* format
* format
* fix snap name
* fix snapsh
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* hoist actors out of composeThread()
* tidy
* run ci on all prs
* format
* build
* proxy graph.getListBlocks
* remove unneeded index
* build pds
* setup noop listblock indexer on pds
* remove build
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* Add self-label schemas
* switch around array and union for self-labels
* apply self-labels to post and profile views
* test for self labels in pds
* apply self-labels to post and profile views in bsky appview
* test for self labels in bsky appview
* update pds proxied test snapshots
* Add support for self-labels to the mdoeration sdk
* Disable unknown-labeler filtering until 3P support is added
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* run with-test-db in monorepo test
* improve pg script
* tidy
* namespace bsky pg schemas
* differentiate schemas
* clean up script
* first pass at invalid handles in appview
* tests for handle invalidation
* move mock to instance instead of prototype
* change network mocks in general
* fixing pagination on actor-search
* fix snap & normalize handles on index
* handling did pagination + update tests
* one last update
* add cid to ListView and ListViewBasic
* add test for mute list embed
* add support for list embeds views
* test
* port to appview
* update missing snap
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* Add languages field to post record
* helper for parsing bcp47
* add language format to lexicon
* codegen for post record langs field
* re-export language parsing in api package
* tests and tidy for lexicon language format
* index post langs, in-progress
* update snapshots, fixes record-with-media embed issue
* index post langs on bsky appview
* don't bother indexing post langs in pds appview, tidy
* pds proxy to appview based on headers
* reorg appview routes forwarding mutes to appview
* tidy
* clean up ctx
* track notif state
* use notif state for notif routes
* add back in missing file
* pr feedback
* wip
* more views
* more views
* update snaps & suggestions test
* better choices for like & reposted
* note
* better block test
* Test pds-proxied mute and notif state
* Update bsky snaps, remove unnecessary #postView
* Update bsky notification tests to use updateSeen
* add getList getFeedGen tests
* Re-disconnect pds from in-process appview indexing for tests
* clean up forSnapshot
* tidy
* feed gen view tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Model post and profile aggs in bsky
* Add background, onCommit, db and background stats
* Add aggregations for post and profile
* Use background queue in bsky labeler, misc plumbing
* Process background throughout bsky tests
* bsky tidy and fixes
* Build views using agg tables in bsky
* Views of feedgens on bsky
* hook up indexing
* Setup custom feeds on bsky
* graph service + getList routes
* migration
* Feed views refactor for bsky, implement getFeed w/ auth passthrough
* mute lists
* add getListMutes
* mute list tests
* add mutes table & mute/unmuteActor routes
* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests
* add getMutes
* implement mute functionality
* remove unspecced
* mute tests
* mute impl tests
* remove proxy tests
* Test custom algos on bsky
* add mute state to snapshots
* table & indexing plugin
* getBlocks
* bail out of mute subquery when no viewer
* fleshing out services
* Test pds getFeed proxy
* tidy, fix build
* tidy
* Fix minor race in bsky mutelist tests
* blocks in feed views
* db migration, impl block logic & tests
* Update pds proxy snaps based on appview mutes
* missed some files on last commit
* ensure process after unmute
* update snap
* move adminAuth to seedclient param
* apply blocks in getFeed
* update snaps
* upadate one last snap
* pr feedback
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Misc notification and optimization in bsky, tidy
* Handle notifs on message queue in bsky appview
* Sync-up profile labeling behavior in bsky
* No async work during actor unindexing in bsky, include additional indexes
* Prevent deeply nested notifs w/ hellthread exception in bsky appview
* Add indexes to bsky appview for more efficient actor deletion/unindexing
* Update pds proxy snaps for updated bsky profile labeling
* Return custom suggestions in bsky appview
* actor type in profiles
* skeleton lex
* more schemas
* first pass at feed composition
* cleanup services
* more tidying
* fix up tests
* bookmarking schemas
* make feeds records instead of actors
* some cleanup
* fleshing out feeds as records
* wip
* working into cfg & dev-env
* fix appview test util
* bookmarking + views
* tidy
* add displayName
* db migration
* wip test
* blob processing
* first test working!
* Sync-up snapshots
* wip
* Fix pds ctx type
* Sync lexicons on pds and api
* wip
* fully refactor
* Prep for open union in reply data in bsky appview, sync-up snapshots
* msgs & ports
* Make pds did cache sqlite-friendly
* Start scaffolding more feedgen tests
* revamp appview tests
* revamp pds tests
* merge in dev env utils
* ensure plc close
* add feed gen to dev-env
* second feed gen
* named feeds in mock
* fix issue with dev-env pds appview
* update bsky snaps
* Flesh-out some feed gen tests
* Fix feed gen reply test
* Test feed bookmarking
* update skeleton defs
* merge
* Sync-up pds with updated feed skeleton contract
* Fix feed bookmark viewer state
* add feed embeds
* Test inaccurate feed gen repost
* Tighten some error behaviors around feed gens
* generator view in record embed
* test
* bookmarked/subscribed -> saved
* updated snaps
* name & descript on alices feed
* add like counts
* proposed lex
* tweak
* fetch feed generator details
* add describeFeedGen to dev-env
* adds tests for getFeedGenerator
* Update lexicons/app/bsky/feed/getFeedGenerator.json
Co-authored-by: devin ivy <devinivy@gmail.com>
* update describe
* add cid to feed gen view
* add alice feed gen avatar
* generator displayName required
* update snaps
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* cleanup lex
* mount route
* Starter bsky algos (#1046)
* implement some starter bsky algos
* add team dids
* tidy
* add quotepost check to whats hot classic
* tests
* carve out feedgen section of pds
* apply limits
* add describeFeedGenerator route
* Whats hot new (#1060)
* map out new whats hot
* only available in postgres
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* build algos branch
* remove relative imports to @atproto/api
* update hot classic name
* rm build
---------
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* v0.3.6
* v0.3.7
* v0.1.10
* update name of skyline
* build branch
* aws build
* getFeedGenerators test
* For now only use new replyRef union for custom feeds, for compat
* Don't use replyRef union on new feeds yet
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* Normalize record datetimes used for sort to simplified iso
* Update tests and snapshots for alternate valid datetime formats
* Fix notifications race during record update
* fix up a couple of tsc errors in app view merge
* wip
* simple proxy
* use dev-env for appview tests
* process all in blob resolver
* another test fix
* wip
* copy proxied tests & add mutes to getFollows/getFollowers
* tidy & add mutes to likes
* more routes + getAuthorFeed tests
* more testing
* tests for feed views
* thread testing
* finished tests for threads
* temporarily skip some tests
* cleaning up & updating test names
* separate db schema for appview
* rearrange
* typo
* add notifications
* re-enable notifs on pds appview
* update schemas
* updated some bsky snaps
* wip
* refactor did-resolver
* clean up deps
* some fixups + caching utilities
* fix up & move to appCtx
* neat its working
* update bsky tests to new auth
* rm unused pds config var
* tidy
* check exp in seconds
* cache dids in postgres
* add migration & did-cache
* start tests
* couple helpers around cache invalidation
* fix expired check
* wip
* change cache semantics
* did cache testing
* do some cache revalidation in indexing
* fix config
* fix issue w did-resolver test-env prototype
* use map instead of record
* stale + expired
* tests
* clear entry method
* fix up build
* expired dids
* clear missing dids
* better verify payload
* bump test timeout
* fix notifs test
* fix up proxied actor search tests
* update snaps to include labels
* fix dev env
* fix up moderation route auth
* fix more auth headers
* fix auth on getPosts
* increase jest timeouts
* fix snaps
* Add moderation and labeling model to bsky appview
* Carry over moderation service from pds
* Pass over bsky moderation service to get it working, missing blob support
* Carry over label service from pds to bsky
* Reorg bsky http auth, implement admin auth
* Carry over relevant admin endpoints pds to bsky
* Retrofit bsky admin routes from pds impls
* Implement resolve handle on appview
* Avoid loop in bsky handle resolution
* Add bsky appview to dev-env, opt-in
* Fix bsky searchRepos with empty term, tidy
* Include blobs on bsky admin views
* Stop resolving taken-down blobs on appview
* Tidy
* Carry over pds moderation tests to bsky
* Support image cache invalidation on bsky appview
* Add missing changes for bsky tests
* Test takedowns in bsky views
* Test takedowns on bsky notifs
* Separate blob resolution logic into standalone helper
* Carry over labeler from pds to bsky
* Update labelers for bsky usage
* Configure labeler on appview, tidy blob-resolver
* Add labels to views for posts and actors
* Update snapshots on bsky appview to include labels
* Carry over labeler tests from pds to bsky
* Tidy-up labeler tests for bsky appview
* Update bsky seeds and snapshots for keyword labeling
* Tidy
* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.
* Include labels on appview listNotifications, fix blob resolver content-type, tidy
* Add moderation and labeling model to bsky appview
* Carry over moderation service from pds
* Pass over bsky moderation service to get it working, missing blob support
* Carry over label service from pds to bsky
* Reorg bsky http auth, implement admin auth
* Carry over relevant admin endpoints pds to bsky
* Retrofit bsky admin routes from pds impls
* Implement resolve handle on appview
* Avoid loop in bsky handle resolution
* Add bsky appview to dev-env, opt-in
* Fix bsky searchRepos with empty term, tidy
* Include blobs on bsky admin views
* Stop resolving taken-down blobs on appview
* Tidy
* Carry over pds moderation tests to bsky
* Support image cache invalidation on bsky appview
* Add missing changes for bsky tests
* Test takedowns in bsky views
* Test takedowns on bsky notifs
* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.
* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect
* Cull bsky config, services, auth, etc.
* Sweep app view xrpc methods, tidy deps, add storage back for img server
* Run repo subscription on bsky app view
* Collapse db migrations down for bsky app view
* Tidy app view bin
* Remove mute functionality from app view, delegate to pds
* Initial tidy/culling of bsky app view tests
* Passing bsky app view db, server, and repo subscription tests
* Passing bsky app view duplicate-records tests
* Bsky app view test tidy/cull
* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.
* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.
* Setup actor handles by did in bsky app view
* Passing indexing tests on bsky app view
* Passing image tests on bsky app view
* Fix bsky actor reindexing, support custom lock id for testing repo subs
* Sweep bsky view tests, misc tests, passing
* Tidy bsky deps
* Include did in resized image uris
* Update bsky image process server to use getBlob
* Update image server tests, misc fixes
* Implement bsky blob resolver
* Wire local image processing server to local blob resolver, test blob resolver
* Tidy
* Tidy
* Tidy
* Tidy app view init
* Fix handle resolution, tidy
* Add utils for partitioning indexing by did
* Update repo sub to parallelize work per repo
* Dep tidy
* Tidy bsky tests for updated repo sub destroy()
* Update thead indexing to handle out-of-order posts
* Sketch out strategy in bsky for handling too-big commits
* Set content-type on sync.getBlob
* Add logging for failed transmissions in bsky blob resolver
* Tidy
* Tidy bsky repo indexing and supporting repo interfaces
* Sort in app view based on combo of creation and indexing times
* Fix types
* Add retry utils to bsky
* Add retries to http requests made by bsky
* Test repo indexing
* Update bsky db/model for lex refactor
* Update bsky lexicons for lex refactor
* Update bsky actor service for lex refactor
* Update bsky feed service for lex refactor
* Update bsky indexing service for lex refactor
* Update bsky repo subscription for lex refactor
* Tidy bsky repo sub
* Add unspecced endpoints to bsky app view, update entrypoint
* Update bsky xrpc utils for lex refactor
* Update bsky xrpc methods for lex refactor
* Update bsky test seeds for lex refactor, tidy api entrypoint
* Update bsky non-view tests for lex refactor
* Update bsky likes view test for lex refactor, minor fix
* Update bsky author feed tests for lex refactor, minor test util fix
* Update bsky follow, profile, repost, search view tests for lex refactor
* Update bsky timeline view tests for lex refactor
* Replace bsky out-of-order thread indexing logic
* Update bsky thread view tests for lex refactor, general test tidying
* Handle rebases and too-big commits in repo subscription, tracking commit data cid
* Tidy
* Ensure did resolver reports "not found" only when positively not found
* Handle tombstones and handle updates in bsky
* Test indexing handle updates and did tombstones
* Support cors on bsky
* Allow app view to serve most routes unauthed
* Tests for bsky unauthed views
* Tidy bsky service entrypoint and dockerfile
* Remove unused storage interfaces from bsky
* Bsky entrypoint and dockerfile fixes, tidy
* Add workflow for bsky build to aws
* Use more standard db env variables, make migration creds optional
* Make bsky repo subscription optional
* Fix lex->json serialization in bsky
* Split bsky actor sync state into its own table
* Skip invalid records on indexing full repo, tidy
* Tidy
* Leader test timing
* Tidy/lint
* Fix bsky config overrides
Daniel Holmgren
Eric Bailey
intrnl
Paul Frazee
devin ivy
Ansh