* pinned posts lexicon
* codegen
* change lexicon, different approach
* codegen 2
* dataplane db migration
* move pinned post lexicon to right place
* add pinned posts optionally to getAuthorFeed
* remove type modification
* Clarify naming, add viewer state, add tests
* return pinnedPost with profileViewDetailed
* allow pinned replies in `posts_and_author_threads`
* clearer variable naming
* annotate type of `items`
* boolean --> varchar
* reuse authorDid in viewerPinned
* simplify test
* make pinned post not top post in test
* update snapshot
* changeset
---------
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>
* lexicon: initial lexicons for video embeds in bsky app
* lexicon: fix video caption file size limit
* codegen
* appview: stub out video embed view logic
* api prerelease
* api prerelease
* lexicon: video upload/processing lexicons
* tidy
* lexicon: app.bsky.video lexicons for uploads
* codegen
* api prerelease
* appview: present video embeds on posts
* appview: snaps
* changeset
* appview: fix wiring of video url config
* Add new postgate lex, hiddeReplies to threadgate, codegen
* Add protobufs
* Add to mock dataplane
* Add matching postgate method to feed hydration methods
* Add to getRecord
* Add to HydrationState
* Fix typo
* Add to mergeStates, fetch embeds in threads
* Integrate into embed views
* Add test for QPs in threads
* Add feed test
* Fix naming convention in protos
* Add #viewRemoved record view, rename postgate.json
* Integrate new view
* Filter hidden replies from feeds
* Filter out replies at the handler level, do not filter for author feeds
* Fix lint
* Move hidden reply check to view layer
* Reduce, reuse, recycle
* Rename to lowercase
* Rename layer vars
* Add quote gate props to postgate (#2693)
* Add quote gate props to postgate
* Consistent naming
* Fix record structure
* Codegen
* Show hidden replies in author feed
* Allow reposts of hidden replies
* Lex and codegen
* Add violates_quote_gate to proto
* Consistent naming, codegen
* Integrate violatesQuotegate and canQuotepost
* Remove rules, codegen
* Hydrate all postgates for all requested posts
* Match other impl
* Add test, need to split these out
* Format
* Hydrate first nested embeds too
* Add postgate test suite
* Add violatesQuoteGate to dataplane
* Ingest and set violatesQuoteGate, return on meta
* Return removed embed for quotes that violate gate
* Add test
* Dedupe URIs before fetching postgates
* Update snaps
* Snap
* Format
* Updating naming conventions for postgate-related attributes
* Correct naming
* Consistency
* Proto too
* Rename to viewDetached
* Codegen
* Rename everything
* Codegen
* Quotes that violate a quote gate can still be quoted themselves
* Couple more renames
* Snaps
* Ensure reply ref is tombstoned for hidden replies
* Split out hidden replies tests and create fresh fixture
* Hydrate threadgates for reply notifications, filter hidden replies
* Remove snap
* Add flaky test
* Rename violatesEmbeddingRules
* Fix flaky test
* Only write to db if violatesEmbeddingRules is true
* DRY up post uri -> gate uri logic
* isThreadgateListRule
* Don't share users object between tests
* No pascal
* Remove default params
* Find -> some
* canQuotepost -> canEmbed, remove optional
* Fix quoteee typo
* await follows
* Throw in post uri -> gate utils
* Ensure fetch threadgates for reply roots
* Don't hydrate threadgates twice
* DRY up uri -> did parsing
* Clean up parsePostgate logic
* Format
* Revert change
* Revert change
* Replace a couple more uri->did conversions
* Only filter replies from feeds if viewer hid them
* Revert, filter out replies that are hidden from feeds
* Remove old test
* Replace uri->did util
* Revert change to unused file
* Only validatePostEmbed and check postgates for post records
* Ensure notifications aren't generated down a hidden reply chain
* Changeset
* Cleanup
* Fix notification filtering logic
* Simplify
* Don't notify for invalid embeds
* Use new APIs
* Add hasPostGate and hasThreadGate flags from dataplane
* Only fetch postgates if post has one
* Only fetch threadgates if post has one or was deleted
* Remove notification filtering
* Don't hydrate threadgates for notifications
* Move hidden replies in feeds to match block handling
* Do no filtering of hidden replies in feeds
* Revert "Don't hydrate threadgates for notifications"
This reverts commit 1dcec0b239a7b9d6800427b26b8ba3e6a54210f9.
* Revert "Remove notification filtering"
This reverts commit 1e7069dfd809d1f18e9f05fd1d422e7399aa1bb0.
* Filter notifications for OP only
* Add additional check to hidden replies test
* Move noty filter logic into method handler
* Update .changeset/perfect-parrots-appear.md
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/bsky/tests/seed/postgates.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Another structuredClone
* Update packages/bsky/src/hydration/hydrator.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Better comment
* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Regen protos to match dataplane
* Update quotes snap to include embeddingDisabled
* Clarify usage of post uri -> gate utils
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
* add quote count to post_agg, add getPostQuotes
rework schema
rework schema
add getPostQuotes to api
use posts
use posts
codegen
use items instead of quotes
codegen
add getPostQuotes
add quoteCount to response
update lexicon for postview
increment post ags
add quote to post aggs
add quote interface
oops
add quote table migration
* update
* bufgen
* update params
* update to use v2
* logs
* rm comment
* pass cursor
* add index
* Update packages/bsky/src/data-plane/server/db/migrations/20240723T220703655Z-quotes.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* only if its a post
* tests
* Discard changes to packages/bsky/tests/views/posts.test.ts
* fix client call
* Include new quotes agg in test expectation
* Use new API for headers
* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* revert rm
* rm timeout
* cursor test
* Changeset
* Remove pds specific bump
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: Eric Bailey <git@esb.lol>
* schemas
* protos
* implementation
* add filtering on ingestion
* test
* add thread viewer state to post views
* update snaps
* changeset
* fix threadgating test
* lex: add account to actor search, for following boost
* lex: addition post search query params
* lex: add tag to post search; set default for sort
* lex: fix formats and skeleton consistency
* lex: searchPosts 'until' should be exclusive
* lex: relax post time range syntax
* lex: allow multiple tags ('AND') as search params
* make fmt
* lex: use 'viewer' for account DID making profile search
* lex: no viewer param in searchActor (only skeleton); do include in post skeleton
* codegen search param additions
* lexicon: add labels to feedgen and list views
* appview: hydrate labels onto lists, feedgens
* test hydration of labels on lists and feedgens
* update pds snap
* Include own replies to own post roots in post_no_replies
* Updates
* Simplify
* snaps
* snaps
* Remove unused import
* Add new threads filter to getAuthorFeed
* Implement new filter
* Typo
* add tags to post lex
* kiss
* add richtext facet and validation attrs
* add tag validation attrs to post
* codegen
* add maxLength for tags, add description
* validate post tags on write
* add test
* handle tags in indexer
* add tags to postView, codegen
* return tags on post thread view
* format
* revert formatting change to docs
* use establish validation pattern
* add changeset
(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)
* remove tags from postView, codegen
* remove tags from thread view
* revert unused changes
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* initial lexicons for interaction-gating
* add interactions view to post views
* codegen
* model bad reply/interaction check state on posts
* initial impl for checking bad reply or interaction on write
* omit invalid interactions from post thread
* support not-found list in interaction view
* hydrate can-reply state on threads
* present interaction views on posts
* misc fixes, update snaps
* tidy/reorg
* tidy
* split interaction gating into separate record in lexicon
* switch interaction-gating impl to use separate record type
* allow checking reply gate w/ root post deletion
* fix
* initial gating tests
* tighten gated reply views, tests
* reply-gating list rule tests
* allow custom post rkeys within window
* hoist actors out of composeThread()
* tidy
* update thread gate lexicons, codegen
* lex fix
* rename gate to threadgate in bsky, update views
* lex fix
* improve terminology around reply validation
* fix down migration
* remove thread gates on actor unindexing
* add back .prettierignore
* tidy
* run ci on all prs
* syntax
* run ci on all prs
* format
* fix snap
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* lex
* list purpose
* lex gen
* add route
* add proxy route
* seed client helpers
* tests
* mutes and blocks
* proxy test
* snapshot
* hoist actors out of composeThread()
* tidy
* tidy
* run ci on all prs
* format
* format
* fix snap name
* fix snapsh
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Add self-label schemas
* switch around array and union for self-labels
* apply self-labels to post and profile views
* test for self labels in pds
* apply self-labels to post and profile views in bsky appview
* test for self labels in bsky appview
* update pds proxied test snapshots
* Add support for self-labels to the mdoeration sdk
* Disable unknown-labeler filtering until 3P support is added
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Add languages field to post record
* helper for parsing bcp47
* add language format to lexicon
* codegen for post record langs field
* re-export language parsing in api package
* tests and tidy for lexicon language format
* index post langs, in-progress
* update snapshots, fixes record-with-media embed issue
* index post langs on bsky appview
* don't bother indexing post langs in pds appview, tidy
* Misc notification and optimization in bsky, tidy
* Handle notifs on message queue in bsky appview
* Sync-up profile labeling behavior in bsky
* No async work during actor unindexing in bsky, include additional indexes
* Prevent deeply nested notifs w/ hellthread exception in bsky appview
* Add indexes to bsky appview for more efficient actor deletion/unindexing
* Update pds proxy snaps for updated bsky profile labeling
* Return custom suggestions in bsky appview
* Test out-of-order indexing of aggregations on bsky appview
* Proxy pds getPopular to hot-classic feed on appview. Proxy getPopularFeedGenerators to appview.
* Make did field required on feedgen view
* actor type in profiles
* skeleton lex
* more schemas
* first pass at feed composition
* cleanup services
* more tidying
* fix up tests
* bookmarking schemas
* make feeds records instead of actors
* some cleanup
* fleshing out feeds as records
* wip
* working into cfg & dev-env
* fix appview test util
* bookmarking + views
* tidy
* add displayName
* db migration
* wip test
* blob processing
* first test working!
* Sync-up snapshots
* wip
* Fix pds ctx type
* Sync lexicons on pds and api
* wip
* fully refactor
* Prep for open union in reply data in bsky appview, sync-up snapshots
* msgs & ports
* Make pds did cache sqlite-friendly
* Start scaffolding more feedgen tests
* revamp appview tests
* revamp pds tests
* merge in dev env utils
* ensure plc close
* add feed gen to dev-env
* second feed gen
* named feeds in mock
* fix issue with dev-env pds appview
* update bsky snaps
* Flesh-out some feed gen tests
* Fix feed gen reply test
* Test feed bookmarking
* update skeleton defs
* merge
* Sync-up pds with updated feed skeleton contract
* Fix feed bookmark viewer state
* add feed embeds
* Test inaccurate feed gen repost
* Tighten some error behaviors around feed gens
* generator view in record embed
* test
* bookmarked/subscribed -> saved
* updated snaps
* name & descript on alices feed
* add like counts
* proposed lex
* tweak
* fetch feed generator details
* add describeFeedGen to dev-env
* adds tests for getFeedGenerator
* Update lexicons/app/bsky/feed/getFeedGenerator.json
Co-authored-by: devin ivy <devinivy@gmail.com>
* update describe
* add cid to feed gen view
* add alice feed gen avatar
* generator displayName required
* update snaps
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* cleanup lex
* mount route
* Starter bsky algos (#1046)
* implement some starter bsky algos
* add team dids
* tidy
* add quotepost check to whats hot classic
* tests
* carve out feedgen section of pds
* apply limits
* add describeFeedGenerator route
* Whats hot new (#1060)
* map out new whats hot
* only available in postgres
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* build algos branch
* remove relative imports to @atproto/api
* update hot classic name
* rm build
---------
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* v0.3.6
* v0.3.7
* v0.1.10
* update name of skyline
* build branch
* aws build
* getFeedGenerators test
* For now only use new replyRef union for custom feeds, for compat
* Don't use replyRef union on new feeds yet
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* block lexicons
* table schema
* block processing
* Actor-block migration
* Add blocked post lexicon to getPostThread
* getBlocks routes
* Add missing lexicons to bsky
* Add block state to actor views
* throw on blocked accounts in getAuthorFeed
* blcok state in profile views
* Update getPostThread to support block views, update existing test
* strip out blocks out oof timeline & popular
* more routes
* Test blocked posts in threads
* Support blocks on record embeds
* notifs
* Update pds test snapshots w/ block state
* fix search
* Update pds actor search snapshots w/ blocks
* test
* tests
* update lex
* fix up from returning blocking as record
* ci for branch
* fix build issue in bsky
* more blocks tests
* fix actor search test
* hook getBlocks up to router
* bugfix & actually test lol
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* com.atproto.label.label lexicon
For persisting un-signed label objects in repositories.
There will likely be a handful of XRPC query endpoints in the
com.atproto.label namespace in the near future.
* lexicons: com.atproto.label.subscribeLabels
* lexicons: com.atproto.label.query
* lexicons: label updates post lex-refactor
* lexicons: rename label.query to label.queryLabels
* lexicons: queryLabels larger batch sizes; improve description
* lexicons: add labels to profiles, posts, notifications
* lexicons: add negation ('neg') flag to labels
* lexicon: label vals on moderation actions
This is a mechanism to create (or negate) labels as part of a moderation
action. The fields are optional, so this is backwards compatible, and if
we wanted to embed entire labels in the future (instead of just "val"
strings), that would be possible using a different field name.
* lexicons: update label.label to label.defs#label
* codegen
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* remove return in test
* couple of fixups in other pacakges
* Add dummy checks to declaration and follow app migrations, remove paranoid join
* update db nsid migration
* Ensure there are writes in follow app migration
* Add dumy check to votes-to-likes app migration, tidy
* Ensure there are writes in vote-to-like app migration
* update migration name
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* Update api package to use updated richtext facets
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* Fix notification reason for change from vote to like
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* Add lexicon for post embeds
* Model post embeds
* Handle and present post embeds
* Test and tidy post embeds
* Generalize post embeds to record embeds
* Tidy
* Add lexicon for getProfiles
* Split out presentation of profiles, implement getProfiles, test
* Cull actor/profile view lexicons down to three variants
* Map viewer in profile view, mark todos, lex tweak
* Apply basic profile view to methods, lex fix
* Switch searchTypeahead to actor-with-info view
* Switch getVotes to actor-with-info view
* Switch getMutes to actor-with-info view
* Switch getFollows and getFollowers to actor-with-info view
* Switch getRepostedBy to actor-with-info view
* Switch feed and notification views to use updated actor-with-info view
