* fix up a couple of tsc errors in app view merge
* wip
* simple proxy
* use dev-env for appview tests
* process all in blob resolver
* another test fix
* wip
* copy proxied tests & add mutes to getFollows/getFollowers
* tidy & add mutes to likes
* more routes + getAuthorFeed tests
* more testing
* tests for feed views
* thread testing
* finished tests for threads
* temporarily skip some tests
* cleaning up & updating test names
* separate db schema for appview
* rearrange
* typo
* add notifications
* re-enable notifs on pds appview
* update schemas
* updated some bsky snaps
* wip
* refactor did-resolver
* clean up deps
* some fixups + caching utilities
* fix up & move to appCtx
* neat its working
* update bsky tests to new auth
* rm unused pds config var
* tidy
* check exp in seconds
* cache dids in postgres
* add migration & did-cache
* start tests
* couple helpers around cache invalidation
* fix expired check
* wip
* change cache semantics
* did cache testing
* do some cache revalidation in indexing
* fix config
* fix issue w did-resolver test-env prototype
* use map instead of record
* stale + expired
* tests
* clear entry method
* fix up build
* expired dids
* clear missing dids
* better verify payload
* bump test timeout
* fix notifs test
* fix up proxied actor search tests
* update snaps to include labels
* fix dev env
* fix up moderation route auth
* fix more auth headers
* fix auth on getPosts
* increase jest timeouts
* fix snaps
* Add moderation and labeling model to bsky appview
* Carry over moderation service from pds
* Pass over bsky moderation service to get it working, missing blob support
* Carry over label service from pds to bsky
* Reorg bsky http auth, implement admin auth
* Carry over relevant admin endpoints pds to bsky
* Retrofit bsky admin routes from pds impls
* Implement resolve handle on appview
* Avoid loop in bsky handle resolution
* Add bsky appview to dev-env, opt-in
* Fix bsky searchRepos with empty term, tidy
* Include blobs on bsky admin views
* Stop resolving taken-down blobs on appview
* Tidy
* Carry over pds moderation tests to bsky
* Support image cache invalidation on bsky appview
* Add missing changes for bsky tests
* Test takedowns in bsky views
* Test takedowns on bsky notifs
* Separate blob resolution logic into standalone helper
* Carry over labeler from pds to bsky
* Update labelers for bsky usage
* Configure labeler on appview, tidy blob-resolver
* Add labels to views for posts and actors
* Update snapshots on bsky appview to include labels
* Carry over labeler tests from pds to bsky
* Tidy-up labeler tests for bsky appview
* Update bsky seeds and snapshots for keyword labeling
* Tidy
* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.
* Include labels on appview listNotifications, fix blob resolver content-type, tidy
* Add moderation and labeling model to bsky appview
* Carry over moderation service from pds
* Pass over bsky moderation service to get it working, missing blob support
* Carry over label service from pds to bsky
* Reorg bsky http auth, implement admin auth
* Carry over relevant admin endpoints pds to bsky
* Retrofit bsky admin routes from pds impls
* Implement resolve handle on appview
* Avoid loop in bsky handle resolution
* Add bsky appview to dev-env, opt-in
* Fix bsky searchRepos with empty term, tidy
* Include blobs on bsky admin views
* Stop resolving taken-down blobs on appview
* Tidy
* Carry over pds moderation tests to bsky
* Support image cache invalidation on bsky appview
* Add missing changes for bsky tests
* Test takedowns in bsky views
* Test takedowns on bsky notifs
* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.
* fix up a couple of tsc errors in app view merge
* wip
* use dev-env for appview tests
* process all in blob resolver
* another test fix
* fixed missed merge conflict
* fix one more merge conflict
* fix popular test
* adding to moderation flows
* some bugfixing in labels
* hardcode profile nsid
* make labels off moderation action
* db updates
* wip
* report labels in admin views & reverse moderation actions
* Test admin get record and repo w/ labels
* update db
* fix
* exclude negs from labels
* exclude neg on moderation views as well
* Check-in missing lex
* Check-in missing lex
* In-progress admin label tests
* Test label creation/reversal via actions
* Admin label test snapshots (#808)
* new snapshots for label on user
* fix get moderation action snap
* fix dev-env
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect
* Cull bsky config, services, auth, etc.
* Sweep app view xrpc methods, tidy deps, add storage back for img server
* Run repo subscription on bsky app view
* Collapse db migrations down for bsky app view
* Tidy app view bin
* Remove mute functionality from app view, delegate to pds
* Initial tidy/culling of bsky app view tests
* Passing bsky app view db, server, and repo subscription tests
* Passing bsky app view duplicate-records tests
* Bsky app view test tidy/cull
* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.
* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.
* Setup actor handles by did in bsky app view
* Passing indexing tests on bsky app view
* Passing image tests on bsky app view
* Fix bsky actor reindexing, support custom lock id for testing repo subs
* Sweep bsky view tests, misc tests, passing
* Tidy bsky deps
* Include did in resized image uris
* Update bsky image process server to use getBlob
* Update image server tests, misc fixes
* Implement bsky blob resolver
* Wire local image processing server to local blob resolver, test blob resolver
* Tidy
* Tidy
* Tidy
* Tidy app view init
* Fix handle resolution, tidy
* Add utils for partitioning indexing by did
* Update repo sub to parallelize work per repo
* Dep tidy
* Tidy bsky tests for updated repo sub destroy()
* Update thead indexing to handle out-of-order posts
* Sketch out strategy in bsky for handling too-big commits
* Set content-type on sync.getBlob
* Add logging for failed transmissions in bsky blob resolver
* Tidy
* Tidy bsky repo indexing and supporting repo interfaces
* Sort in app view based on combo of creation and indexing times
* Fix types
* Add retry utils to bsky
* Add retries to http requests made by bsky
* Test repo indexing
* Update bsky db/model for lex refactor
* Update bsky lexicons for lex refactor
* Update bsky actor service for lex refactor
* Update bsky feed service for lex refactor
* Update bsky indexing service for lex refactor
* Update bsky repo subscription for lex refactor
* Tidy bsky repo sub
* Add unspecced endpoints to bsky app view, update entrypoint
* Update bsky xrpc utils for lex refactor
* Update bsky xrpc methods for lex refactor
* Update bsky test seeds for lex refactor, tidy api entrypoint
* Update bsky non-view tests for lex refactor
* Update bsky likes view test for lex refactor, minor fix
* Update bsky author feed tests for lex refactor, minor test util fix
* Update bsky follow, profile, repost, search view tests for lex refactor
* Update bsky timeline view tests for lex refactor
* Replace bsky out-of-order thread indexing logic
* Update bsky thread view tests for lex refactor, general test tidying
* Handle rebases and too-big commits in repo subscription, tracking commit data cid
* Tidy
* Ensure did resolver reports "not found" only when positively not found
* Handle tombstones and handle updates in bsky
* Test indexing handle updates and did tombstones
* Support cors on bsky
* Allow app view to serve most routes unauthed
* Tests for bsky unauthed views
* Tidy bsky service entrypoint and dockerfile
* Remove unused storage interfaces from bsky
* Bsky entrypoint and dockerfile fixes, tidy
* Add workflow for bsky build to aws
* Use more standard db env variables, make migration creds optional
* Make bsky repo subscription optional
* Fix lex->json serialization in bsky
* Split bsky actor sync state into its own table
* Skip invalid records on indexing full repo, tidy
* Tidy
* Leader test timing
* Tidy/lint
* Fix bsky config overrides
* remove return in test
* couple of fixups in other pacakges
* Add dummy checks to declaration and follow app migrations, remove paranoid join
* update db nsid migration
* Ensure there are writes in follow app migration
* Add dumy check to votes-to-likes app migration, tidy
* Ensure there are writes in vote-to-like app migration
* update migration name
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* Update api package to use updated richtext facets
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* Fix notification reason for change from vote to like
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* hardening: permissive tests for handles
* identifers: permissive tests for NSIDs
* identifiers: add DID validation
* identifiers: wrong about domain syntax (duh, 4chan.com)
In short, labels can start with digits except for the final TLD part.
I think that is all for "DNS domains as hostnames which are actually
used in the real world", though there might be other modern things.
Underscores are obviously allowed in DNS for things like SRV records,
but I don't think as "regular hostnames".
Not sure we want this for NSIDs, so not updating that code or tests yet.
* identifiers: ATURI test corner cases
* identifiers: remove TODOs (moved to formalism doc)
* identifiers: small comment typos and corrections
* move around & integrate indentifier hardneing
* fixed up some tests
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* Add helper to consume subscription message frames
* Implement xrpc subscription consumer
* Simplify subscription consumer params
* Remove duped test
* Reconnect to subscription on initial setup
* Add reconnect hook on subscription, for logging
* Make subscription consumption cancelable
* Include error information in sub reconnect hook
* Setup table for tracking subscription state on app view
* Support ending app-view leader, clean disconnects from xrpc subscription consumer
* Frame out stateful app-view repo subscription
* Setup app-view lifecycle, run repo subscription
* Add logging to repo sub. Add jitter to leader polling.
* Support configuring app-view repo host
* Allow running pds/app-view dev env with an external repo host
* Fix subscription abort test
* Add test for app-view repo subscription
* Skip invalid repo sub messages w/ log, tidy
* Ensure app-view is permissive of reindexing
* Tidy app-view, renaming sync config, improve error logging
* Fix sync cursor todo, order migrations
* Map new ops field in repo subscription
* In-progress, support indexing updates
* Revert "In-progress, support indexing updates"
This reverts commit 4e082daf535c11c5d4cf69474e5d48a1a3fbe8aa.
* Support indexing record updates
* Prep notifications for post updates. Basic tests for indexing.
* Update indexing tests to better exercise indexes
* integrating new plc lib
* patching up did-resolver
* buffing up pds tests
* didResolver on ctx & plc in postgres
* bring dev env up to date
* re-add extension for linting
* tidy
* use current env vars
* pr feedback
* Scaffold xrpc-stream
* Write and test stream frames
* Write and test stream server
* Test bad stream frame parsing cases
* Proof of concept streaming on xrpc-server
* Test bad streaming endpoint
* Make frame stream to fix buffering frame bytes together
* reorg
* some lex-cli things
* better codegen for subscriptions
* scaffolding repo subscriptions
* wip
* remove repo ops
* setup notify/listen for db
* end pool to fix hanging test
* small comment in test
* basic sequencer
* some refactoring
* switch to event emitter
* reconnect on listener error
* rename notifyClient
* remove payload on channels
* pr feedback
* subscribeRepo outbox
* some cleanup
* wip
* wip
* bugfixin
* only send msgs after tx is committed
* better handle event-emitter -> generator
* max buffer size
* cleanup
* Sync-up xrpc subscriptions with current iteration of the spec
* Allow missing encoding for subscriptions
* track sequencedAt & eventType as well
* Resolve streaming type codes, streaming tests
* Fix interface for hooking into http server for streaming, fix stream routing
* Minor reorg how streaming is enabled in xrpc-server
* Server codegen for xrpc subscriptions, supporting types in xrpc-server
* fix up buffer overloading
* Rename data frame to message frame, rename subscription lex output to message
* Move body param first for streaming frames
* Tidy
* clean up evt types
* buff up tests
* missed merge conflict
* new schema
* blobs on subscriptions
* rm genned client subscription methods
* backfill limits
* testing subscription route & quick outbox bugfix
* fix up migration
* cascade on delete
* comments & naming
* fix dev env
* delete seqs on account deletion
* tidy
* fixing things up with db notify system for schemas
* fix duplicates in outbox
* tidy
* Break out of stream handler when socket closes
* fixing up some timing issues
* tidy
* terminate ws after using
* bump up timer on async reader
* fixing up NOTIFY in txns
* pr feedback
* pr bugfixes
* make order asc explicit
* bringing tests up to speed w atpagent
* bump up max listeners on sequencer
* increase timeouts for reading from generators
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Refactor @atproto/api to use the simplified AtpAgent API
* xrpc package: Export the defaultFetchHandler to reuse in api
* api package: Use the defaultFetchHandler defined in xrpc
* Update all usages of the api for the new AtpAgent
* Clear promise on thrown codepath
* Avoid updating the atpagent session until ready to return
* introducing appcontext & pds class
* revamp rest of server + auth
* revamp tests
* add to dev-env
* fix api test
* add start method to pds
* hook up terminator before
* PLC AppContext (#425)
* convert plc server to app context
* add start method to plc
* hooke up terminator on server setup
* fix test
* cleanup & pr feedback
* fix up one more test
* Enable all modules to be built shallowly, externalizing their deps
* Setup shallow builds to modify package.json main
* Hoist update-main-to-dist to separate call, fix lex-cli and plc shallow builds
* Generate sourcemaps during build
* Tidy
* Hoist new build deps up to root
* Include scene trends in feeds
* Tidy
* Tidy
* Sweep app.bsky schemas for standard actor declarations, remove reference from com.atproto
* Replace declarationCid on follow, assertion, confirmation records
* Sweep pds for standard app.bsky actor declarations
* Fix dev-env
* Tidy after merge
* Add a scene to the dev-env mock data
* Add 'member' to myState output of getProfile
* Await db writes in dev env mock data
Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
* Model votes in lexicon, remove likes
* Model votes in db, setup notifs, fix schema
* Update pds logic, move likes to votes
* Codegen
* Update tests for votes
* Test vote filter by direction
* Fix dev-env
* Add plc healthcheck, unify plc server interface with pds
* Add pds healthcheck
* Make plc test util file consistent with other packages
* Fix types in dev-env
* Type fixes
* Tidy
* Rename healthcheck routes, place pds's under /xrpc
* Segment lexicons into more specific namespaces and tighten up some naming conventions
* Update API to use new lexicons
* Update pds and dev-env to use new lexicons
* Update lexicons of xprc procedures to avoid using query params when possible
* Update code generated by lex-cli to reduce param counts
* Update all API usages to new signatures
* Fix readme
* Remove params from postNotifiactionsSeen lexicon schema
* schemas
* database & buffing up schemas
* declaration on createAccount, + fixing up test client
* fix up dev-env
* schema comments
* nsid for declaration actorType
* declaration description
* oops token schema slipped in
* declaration -> declarationCid
* missed a couple of db things
* user-did table
* yay fixed it
* resolve available domains from db
* serverDid + tests
* check for invalid domains
* mv available domain check to constructor
* oops reverse check
* clean up old todos
* recovery key in server cfg & allow user provided key
* tests & rename
* testing did creation
* recoveryKey in dev-env
* require recovery key
* Setup PDS with kysely migrations
* Nix KyselyWithDialect from PLC, found it didn't work as intended in PDS
* Fix down migration in PDS
* Fix update profile on postgres
* Support postgres and migrations in PDS bin
* Lint
