Improve BskyAgent performance
By using z.discriminatedUnion instead of z.union
we can hint to zod at which field it should look when
validating/parsing schemas
This improves BskyAgent instantiation time around 4 times,
when creating 10 agents using stuff from #1015,
the time went down from ~1200ms to 330ms
* remove return in test
* couple of fixups in other pacakges
* Add dummy checks to declaration and follow app migrations, remove paranoid join
* update db nsid migration
* Ensure there are writes in follow app migration
* Add dumy check to votes-to-likes app migration, tidy
* Ensure there are writes in vote-to-like app migration
* update migration name
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* Update api package to use updated richtext facets
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* Fix notification reason for change from vote to like
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* Add helper to consume subscription message frames
* Implement xrpc subscription consumer
* Simplify subscription consumer params
* Remove duped test
* Reconnect to subscription on initial setup
* Add reconnect hook on subscription, for logging
* Make subscription consumption cancelable
* Include error information in sub reconnect hook
* Yield info frames in byMessage()
* Scaffold xrpc-stream
* Write and test stream frames
* Write and test stream server
* Test bad stream frame parsing cases
* Proof of concept streaming on xrpc-server
* Test bad streaming endpoint
* Make frame stream to fix buffering frame bytes together
* reorg
* some lex-cli things
* better codegen for subscriptions
* scaffolding repo subscriptions
* wip
* remove repo ops
* setup notify/listen for db
* end pool to fix hanging test
* small comment in test
* basic sequencer
* some refactoring
* switch to event emitter
* reconnect on listener error
* rename notifyClient
* remove payload on channels
* pr feedback
* subscribeRepo outbox
* some cleanup
* wip
* wip
* bugfixin
* only send msgs after tx is committed
* better handle event-emitter -> generator
* max buffer size
* cleanup
* Sync-up xrpc subscriptions with current iteration of the spec
* Allow missing encoding for subscriptions
* track sequencedAt & eventType as well
* Resolve streaming type codes, streaming tests
* Fix interface for hooking into http server for streaming, fix stream routing
* Minor reorg how streaming is enabled in xrpc-server
* Server codegen for xrpc subscriptions, supporting types in xrpc-server
* fix up buffer overloading
* Rename data frame to message frame, rename subscription lex output to message
* Move body param first for streaming frames
* Tidy
* clean up evt types
* buff up tests
* missed merge conflict
* new schema
* blobs on subscriptions
* rm genned client subscription methods
* backfill limits
* testing subscription route & quick outbox bugfix
* fix up migration
* cascade on delete
* comments & naming
* fix dev env
* delete seqs on account deletion
* tidy
* fixing things up with db notify system for schemas
* fix duplicates in outbox
* tidy
* Break out of stream handler when socket closes
* fixing up some timing issues
* tidy
* terminate ws after using
* bump up timer on async reader
* fixing up NOTIFY in txns
* pr feedback
* pr bugfixes
* make order asc explicit
* bringing tests up to speed w atpagent
* bump up max listeners on sequencer
* increase timeouts for reading from generators
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Update LexXrpcBody encoding to always be a single string with comma-separated values
* Update lexicon union-type to use as a discriminator, and add closed flag to establish if new union members may be added
* Fix to generated code for open unions
* Implement types and validation for new NIH lexicon DSL
* Update xprc and xrpc-server to use new NIH lexicon DSL
* Simplification: dont allow lexicon definitions to be refs or ref arrays
* Add 'unknown' primitive type to NIH lexicon
* Implement client codegen for NIH Lexicon
* Implement server codegen for NIH Lexicon
* Add the 'datetime' primitive type to the NIH Lexicon
* Rework lexicons to use new NIH Lexicon DSL
* Readd tokens to code generation
* Update all code to use new NIH Lexicon (all tests passing!)
* Add imports from external files to lex-cli codegen
* Do not allow records, queries, or procedures outside of the main def
* Update packages/lex-cli/src/util.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Linter pass
* Add knownValue output to codegen
* Add comments to codegen
* Use enum and const in codegen
* Rework lexicon 'properties' structure
* Treat falsy query params as undefined
Co-authored-by: devin ivy <devinivy@gmail.com>
* Remove now-redundant param presence validation
Co-authored-by: devin ivy <devinivy@gmail.com>
* Enable all modules to be built shallowly, externalizing their deps
* Setup shallow builds to modify package.json main
* Hoist update-main-to-dist to separate call, fix lex-cli and plc shallow builds
* Generate sourcemaps during build
* Tidy
* Hoist new build deps up to root
* Model procedure for setting a vote on a post
* Implement app.bsky.feed.setVote() method
* Support json schema subset for parameters in lexicon and lex-cli
* Update lexicons w/ json schema parameters
* Use enum in getVotes parameters
* Support json schema subset for parameters in xrpc-server, ajv validation
* Include parameter enums in generated markdown
* Tidy
* Unify xrpc parameter md docs with inputs and outputs: show ts interfaces
* Tidy
* Add token schema types to lexicon package
* Add token-schema support to lex-cli package generation of markdown, client code, and server code
* Add two token lexicons: actorUser and actorScene
* Remove dead code
* Rename ADX to ATP
* Remove old docs (now in atproto website repo)
* Update readme
* Update README.md
Co-authored-by: devin ivy <devinivy@gmail.com>
* A couple of @ -> AT
Co-authored-by: devin ivy <devinivy@gmail.com>
* update schemas for keyed
* codegen dict
* codgen record ids
* determine key type from schema
* allow non-tid keys
* rm tids from batch write schema
* use schema ids record in db layouts
* standardize on rkey
* doc update
* crud test
* repo tests for custom rkeys
* Update lexicon doc format to support .defs field and .descriptions within method bodies
* Add correct resolution to lexicon, including to lex: URIs
* Update lexicons and generated code to use new resolution
* Add lexicon doc
* Update error-handling spec
* Implement new error-behaviors in xrpc and xrpc-server packages
* Update lexicon and lex-cli packages to add xrpc error behaviors
* Generate new API and test an error behavior
