* rm tables
* rm event-stream & proxied
* Remove appview services, move label service to pds
* only proxy appview stuff
* delete more tables
* Start removing message dispatched from pds
* more syncing-up removal of message dispatcher in pds
* merged
* remove feedgens from pds, remove getPopular
* remove unused image helper from pds
* fixing compiler errors
* clean up sharp
* rm label service
* first pass on cleaning up tests
* fix up a bunch of tests
* moderation view tests
* last admin tests
* got a lil overzealous in deletes
* clean up unused cfg
* clean up label table
* simplify admin repo search query/logic
* tidy pds entrypoint
* in-progress pds config changes
* cfg fiddling
* finish cleaning up cfg/ctx
* comments
* building
* pds prefix on env
* test env
* collapse pds migrations down into a single migration
* fix up dev-env
* tidy
* cleanup
* fix pds admin tests
* fix handle test
* fix pds proxy tests
* fix subscribe repos test
* fix sqlite config in pds tests
* add sqlite clause in sequencer-leader
* fix actor search w/ sqlite on pds
* fixes
* fix dev env build
* update pds service entrypoint
* simple env example
* make takedown ids opaque identifiers in the pds
* use pds routes for api tests
* update pds dockerfile with volume and correct port env var
* add a couple env vars to example
* add comments to env example
* @atproto/pds 0.2.0-beta.0
* @atproto/aws 0.0.1-beta.0
* appview did
* @atproto/aws 0.0.1
* enable logs by default
* update env example
* bugfixing sandbox issues
* consistency in pds env var name for appview url
* log on pds start and stop, configure version at runtime
* @atproto/pds 0.2.0-beta.1
* fix semver matching for pds beta version
* v0.2.0-beta.2
* default invites to being not required
* fix flaky test
* limit db connections in tests
* publish 0.2.0-beta.d3
* fix invite required parsing
* @atproto/pds 0.2.0-beta.5
* Proxy getPopularFeedGenerators on simplified pds (#1222)
proxy getPopularFeedGenerators on pds
Co-authored-by: dholms <dtholmgren@gmail.com>
* tidy migrations
* fix service entry
* bump version
* change auth order
* bump version
* bump version
* add upgradeRepoVersion & fallback url for cdn
* bump version
* merging
* merge pds
* building dev-env
* merging tests
* merge service entry
* test fixing
* tidy
* fix admin search
* tidy
* tidy
* add snap for getListFeed
* add backup nameserver cfg
* tidy + pr feedback
* tidy
* tidy env
* bit more
* re-add dotenv to root package.json
* fix dep
* build branch
* fix tests
* Refactor tests to make better use of dev-env (#1690)
* refactor pds tests to use dev env
* refactor bsky tests
* fix pds test
* tidy bsky tests
* build pds correctly
* fix entry point
* default logging to false (for now)
* format service entry
* Switch takedown ids back to ints on pds distribution (#1694)
* switch takedown ids back to ints, consistent with live pds
* tidy/fix migration
* update migration for sqlite
* export moderation action reversal
* takedown tests
* dont build branch
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* lex
* list purpose
* lex gen
* add route
* add proxy route
* seed client helpers
* tests
* mutes and blocks
* proxy test
* snapshot
* hoist actors out of composeThread()
* tidy
* tidy
* run ci on all prs
* format
* format
* fix snap name
* fix snapsh
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* hoist actors out of composeThread()
* tidy
* run ci on all prs
* format
* build
* proxy graph.getListBlocks
* remove unneeded index
* build pds
* setup noop listblock indexer on pds
* remove build
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* pseudocode for sending notification to user
* add notification push token table
* lexicon codegen
* `pds` and `api` codegen
* update lexicon
* add simple function to `putNotificationPushToken` into `notification_push_token` table
* reorgnaize imports
* add unspecced `putNotificationPushToken` to pds
* add `notification-push-tokens` table to PDS
* check if token exists before adding it to db
* add endpoint and appId to PDS table
* setup notification server
* fix logic for inserting token into db
* fix NotificationServer methods by making them static
* fix merge conflicts
* add comments on how sending ntoifications work
* remove dead code
* move notifServer to AppContext
* refactor code to handle notif sending in `Indexer` service
* add additional data when sending notifs
* clean up code
* move notif logic to indexer
* add `appId` and `endpoint` optional params to putNotificationPushToken
* clone notification code to AppView
* add endpoint to register push token with app view
* disable pds and enable app view notification server
* clean up code
* simply logic to check if a token already exists
* remove NotificationServer from PDS
* remove notification-push-token table from PDS
* remove `putNotificationPushToken` endpoint
* clean up code
* let `axios` throw error if `gorush` has an error
* let `kysely` throw error if notif cannot be registered by client
* rename `registerPushNotificationEndpoint` to `registerPushNotifications`
* delete `putNotificationPushToken` from AppView
* rename putPushNotificationToken to registerPushNotification
* remove dead notification code from pds
* remove sanitizeDisplayName from NotificationSever
* move `pushNotificationEndpoint` to config
* temp add `pushNotificationEndpoint` to dev-env setup
* remove example test from feed-generation.test.ts
* add test for registerPushNotification and clean up error handling
* move notifications test to its own file
* add test for NotificationServer to check if tokens are retrieved correctly
* remove unused functions from NotificationServer
* add additional tests for NotificationServer
* add return type to getNotificationDisplayAttributes function
* remove unnecessary comments
* remove dead NotificationServer code from PDS
* clean up code to prepareNotifsToSend
* put sending notifications as part of the backgroundQueue
* log instead of throwing error if notification attributes don't exist
* remove logs
* add more tests to `notification-server.test.ts`
* show replied with text for reply notifs
* better error handling when sending notifications via backgroundQueue
* add rate limit and batching to sending notifications
* add comments to NotificationServer
* merge with main
* use redis for rate limiting instead of normal rate limits
* move `notificationBatchSize` into config
* usePrimary db in test
* hoist push notif migration to present, update model to remove endpoint
* update push notif lexicon
* pare down lex for unregistering push
* helpers for working with service endpoints from did doc
* service-authed register/unregister push notifs
* add well-known endpoint to appview
* update bsky notif service tests
* fix to batching logic, misc tidy
* batch display notifications
* colocate all notification sending logic
* tidy tests
* remove unregister token for now
* fix registerPush lexicon, make a procedure
* fix registerPush impl, test pds proxy to notif service
* fix tests, make notif server optional when not configured
* fix notif server config for bsky app service
* move notif server rate limiting in-mem for now, add sending retry
* codegen tidy
* only push notifs on commit
* build
* fix notif rate limiter check
* send notifs from users w/o a profile
* remove build
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* setup bsky so that writes always go to a db primary
* test bsky primary db functionality
* sprinkle db primary over moderation and view maintainer usage
* make bsky tests more strict on primary/secondary db, fixes
* maintain prev db config on appview, require primary db, make primary/replica explicit in config
* tidy
* sketch out db replicas
* wip
* support multiple tags per bsky replica, "any" tag, fallbacks and warns
* clarify tags
* use new coordinator api across routes & services
* wire-up replica tagging config
* tidy
* cleanup entroypoints and tests for db coordinator
* fix pds test
* re-hookup migrate db
* standardize env vars for db on bsky services
* tidy
* build
* fix bav api entrypoint
* support DB_POSTGRES_URL on ingester/indexer for backwards compat
* remove build
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* first pass on profiles
* quick test
* wip
* wip
* test post thread
* record embeds
* get author feed profiles
* wip timeline
* fix get timeline
* switch from counter to tid
* tidy into a service
* quick tid test
* pr feedback
* clock -> rev
* update image formatting
* disable migration & build branch
* add recent posts to getAuthorFeed & handle post thread not found errors
* refactor for lag header
* tidy
* rm collections check
* tidy test
* pr feedback
* fix small bug
* build branch
* get migrations into system
* enable migrations
* Add self-label schemas
* switch around array and union for self-labels
* apply self-labels to post and profile views
* test for self labels in pds
* apply self-labels to post and profile views in bsky appview
* test for self labels in bsky appview
* update pds proxied test snapshots
* Add support for self-labels to the mdoeration sdk
* Disable unknown-labeler filtering until 3P support is added
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* proxy timeline skeleton construction to appview
* add getFeedSkeleton to appview
* mount route
* smart proxy feed skeletons
* tests
* proper error code
* only proxy specific feeds
* build branch
* update proxyable feed logic, should use feed publisher rather than generator
* fix feed proxy tests, configure feed publisher (in addition to generator)
* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)
* prevent user-supplied rkey on posts with createRecord
* allow empty-string rkey parameter
Co-authored-by: devin ivy <devinivy@gmail.com>
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
* add slurs to reserved words (#1318)
* add slurs to reserved words (#1314)
* Update reserved.ts
Add slurs to reserved words
* Update reserved.ts
fix typo
* Update reserved.ts to clean up the slur list
* linting
* pluralise
---------
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
* identifier: tweaks and additions to slur list (#1319)
* Refactor appview repo subscription for memleak (#1308)
* refactor to remove closure in loop
* move consecutive item out of p-queue
* Handle validation improvements (#1336)
* Handle matches and false positives for unacceptable words in handles
* move handle validation logic to pds
* missed merge
* add cfg flag
* encode lists
* fix build issues
* move words to cfg
* tidy
---------
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
* Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
* ✨ Allow searching reports by moderator did (#1283)
* ✨ Allow searching reports by moderator did
* ✅ Remove .only flags on tests
* ✅ Update snapshot
* ✅ Add checks for did match in actions
* v0.4.1
* Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds
* tidy
* use 127.0.0.1 in with-test-db.sh for colima (#1297)
So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).
Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts. See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)
To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.
(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)
* Subscription util tests (#1295)
* consecutive list tests
* flesh out subscription util tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Content reporting on record fields (#1351)
* content reporting on record fields
* fix test
* tests
* tidy
* Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
* ✨ Added new procedure for sending admin email (#1312)
* 🚧 Added new lexicon for sending admin email
* ✨ Add moderation mailer
* ✨ Switch to text email content from html
* 🧹 Cleanup some early implementation code and reflect PR reivew
* ✨ Use smtp host instead of gmail service config
* ✨ Move to using single smtp url
* v0.4.2
* Patch up a couple sqlite tests (#1355)
patch up a couple sqlite tests
* enable feeds & build branch
* disable branch building & enable without proxy header
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
* limit backsearch to 1 day instead of 3
* lower like count threshold
* bump to 6
* disable like count check
* disable with friends
* preemptively cache last commit
* inline list mutes
* actor service
* label cache
* placehodler on popular with friends
* bulk sequence
* no limit but chunk
* bump chunk to 5k
* try 10k
* fix notify
* tweaking
* syntax
* one more fix
* increase backfill allowance
* full refresh label cache
* limit 1 on mute list
* reserve aclu handle
* clean up testing with label cache
* note on with-friends
* rm defer from label cache
* label cache error handling
* rm branch build
* run with-test-db in monorepo test
* improve pg script
* tidy
* namespace bsky pg schemas
* differentiate schemas
* clean up script
* first pass at invalid handles in appview
* tests for handle invalidation
* move mock to instance instead of prototype
* change network mocks in general
* fixing pagination on actor-search
* fix snap & normalize handles on index
* handling did pagination + update tests
* one last update
* add cid to ListView and ListViewBasic
* add test for mute list embed
* add support for list embeds views
* test
* port to appview
* update missing snap
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* Add languages field to post record
* helper for parsing bcp47
* add language format to lexicon
* codegen for post record langs field
* re-export language parsing in api package
* tests and tidy for lexicon language format
* index post langs, in-progress
* update snapshots, fixes record-with-media embed issue
* index post langs on bsky appview
* don't bother indexing post langs in pds appview, tidy
* pds proxy to appview based on headers
* reorg appview routes forwarding mutes to appview
* tidy
* clean up ctx
* track notif state
* use notif state for notif routes
* add back in missing file
* pr feedback
* wip
* more views
* more views
* update snaps & suggestions test
* better choices for like & reposted
* note
* better block test
* Test pds-proxied mute and notif state
* Update bsky snaps, remove unnecessary #postView
* Update bsky notification tests to use updateSeen
* add getList getFeedGen tests
* Re-disconnect pds from in-process appview indexing for tests
* clean up forSnapshot
* tidy
* feed gen view tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Model post and profile aggs in bsky
* Add background, onCommit, db and background stats
* Add aggregations for post and profile
* Use background queue in bsky labeler, misc plumbing
* Process background throughout bsky tests
* bsky tidy and fixes
* Build views using agg tables in bsky
* Views of feedgens on bsky
* hook up indexing
* Setup custom feeds on bsky
* graph service + getList routes
* migration
* Feed views refactor for bsky, implement getFeed w/ auth passthrough
* mute lists
* add getListMutes
* mute list tests
* add mutes table & mute/unmuteActor routes
* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests
* add getMutes
* implement mute functionality
* remove unspecced
* mute tests
* mute impl tests
* remove proxy tests
* Test custom algos on bsky
* add mute state to snapshots
* table & indexing plugin
* getBlocks
* bail out of mute subquery when no viewer
* fleshing out services
* Test pds getFeed proxy
* tidy, fix build
* tidy
* Fix minor race in bsky mutelist tests
* blocks in feed views
* db migration, impl block logic & tests
* Update pds proxy snaps based on appview mutes
* missed some files on last commit
* ensure process after unmute
* update snap
* move adminAuth to seedclient param
* apply blocks in getFeed
* update snaps
* upadate one last snap
* pr feedback
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Misc notification and optimization in bsky, tidy
* Handle notifs on message queue in bsky appview
* Sync-up profile labeling behavior in bsky
* No async work during actor unindexing in bsky, include additional indexes
* Prevent deeply nested notifs w/ hellthread exception in bsky appview
* Add indexes to bsky appview for more efficient actor deletion/unindexing
* Update pds proxy snaps for updated bsky profile labeling
* Return custom suggestions in bsky appview
* Model post and profile aggs in bsky
* Add background, onCommit, db and background stats
* Add aggregations for post and profile
* Use background queue in bsky labeler, misc plumbing
* Process background throughout bsky tests
* bsky tidy and fixes
* Build views using agg tables in bsky
* Views of feedgens on bsky
* Setup custom feeds on bsky
* Feed views refactor for bsky, implement getFeed w/ auth passthrough
* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests
* Test custom algos on bsky
* Test pds getFeed proxy
* tidy, fix build
* tidy
* actor type in profiles
* skeleton lex
* more schemas
* first pass at feed composition
* cleanup services
* more tidying
* fix up tests
* bookmarking schemas
* make feeds records instead of actors
* some cleanup
* fleshing out feeds as records
* wip
* working into cfg & dev-env
* fix appview test util
* bookmarking + views
* tidy
* add displayName
* db migration
* wip test
* blob processing
* first test working!
* Sync-up snapshots
* wip
* Fix pds ctx type
* Sync lexicons on pds and api
* wip
* fully refactor
* Prep for open union in reply data in bsky appview, sync-up snapshots
* msgs & ports
* Make pds did cache sqlite-friendly
* Start scaffolding more feedgen tests
* revamp appview tests
* revamp pds tests
* merge in dev env utils
* ensure plc close
* add feed gen to dev-env
* second feed gen
* named feeds in mock
* fix issue with dev-env pds appview
* update bsky snaps
* Flesh-out some feed gen tests
* Fix feed gen reply test
* Test feed bookmarking
* update skeleton defs
* merge
* Sync-up pds with updated feed skeleton contract
* Fix feed bookmark viewer state
* add feed embeds
* Test inaccurate feed gen repost
* Tighten some error behaviors around feed gens
* generator view in record embed
* test
* bookmarked/subscribed -> saved
* updated snaps
* name & descript on alices feed
* add like counts
* proposed lex
* tweak
* fetch feed generator details
* add describeFeedGen to dev-env
* adds tests for getFeedGenerator
* Update lexicons/app/bsky/feed/getFeedGenerator.json
Co-authored-by: devin ivy <devinivy@gmail.com>
* update describe
* add cid to feed gen view
* add alice feed gen avatar
* generator displayName required
* update snaps
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* cleanup lex
* mount route
* Starter bsky algos (#1046)
* implement some starter bsky algos
* add team dids
* tidy
* add quotepost check to whats hot classic
* tests
* carve out feedgen section of pds
* apply limits
* add describeFeedGenerator route
* Whats hot new (#1060)
* map out new whats hot
* only available in postgres
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* build algos branch
* remove relative imports to @atproto/api
* update hot classic name
* rm build
---------
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* v0.3.6
* v0.3.7
* v0.1.10
* update name of skyline
* build branch
* aws build
* getFeedGenerators test
* For now only use new replyRef union for custom feeds, for compat
* Don't use replyRef union on new feeds yet
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* Normalize record datetimes used for sort to simplified iso
* Update tests and snapshots for alternate valid datetime formats
* Fix notifications race during record update
* fix up a couple of tsc errors in app view merge
* wip
* simple proxy
* use dev-env for appview tests
* process all in blob resolver
* another test fix
* wip
* copy proxied tests & add mutes to getFollows/getFollowers
* tidy & add mutes to likes
* more routes + getAuthorFeed tests
* more testing
* tests for feed views
* thread testing
* finished tests for threads
* temporarily skip some tests
* cleaning up & updating test names
* separate db schema for appview
* rearrange
* typo
* add notifications
* re-enable notifs on pds appview
* update schemas
* updated some bsky snaps
* wip
* refactor did-resolver
* clean up deps
* some fixups + caching utilities
* fix up & move to appCtx
* neat its working
* update bsky tests to new auth
* rm unused pds config var
* tidy
* check exp in seconds
* cache dids in postgres
* add migration & did-cache
* start tests
* couple helpers around cache invalidation
* fix expired check
* wip
* change cache semantics
* did cache testing
* do some cache revalidation in indexing
* fix config
* fix issue w did-resolver test-env prototype
* use map instead of record
* stale + expired
* tests
* clear entry method
* fix up build
* expired dids
* clear missing dids
* better verify payload
* bump test timeout
* fix notifs test
* fix up proxied actor search tests
* update snaps to include labels
* fix dev env
* fix up moderation route auth
* fix more auth headers
* fix auth on getPosts
* increase jest timeouts
* fix snaps
