* Retain type of `schemas` using definition type instead of obscuring into a `LexiconDoc[]`
* Improve validation performances by using discriminated unions where possible
* Export the generated lexicons `schemas` definitions
* optimization
* changeset
* tidy
* feat(api): support creation of oauth based AtpAgents
* oauth: misc fixes for confidential clients
* fix(xprc): remove ReadableStream.from polyfill
* OAuth docs tweaks (#2679)
* OAuth: clarification about client_name being shown
* OAuth: re-write handle resolution privacy concern
* avoid relying on ReadableStream.from in xrpc-server tests
* feat(oauth-types): expose "ALLOW_UNSECURE_ORIGINS" constant
* feat(handle-resolver): expose "AtprotoIdentityDidMethods" type
* fix(oauth-client): ensure that the oauth metadata document contains client_id_metadata_document_supported
* fix(oauth-types): prevent unknown query string in loopback client id
* fix(identity-resolver): check that handle is in did doc's "alsoKnownAs"
* feat(oauth-client:oauth-resolver): allow logging in using either the PDS URL or Entryway URL
* fix(oauth-client): return better error in case of invalid "oauth-protected-resource" status code
* refactor(did): group atproto specific checks in own
* feat(api): relax typing of "appLabelers" and "labelers" AtpClient properties
* allow any did as labeller (for tests mainly)
* fix(api): allow to override "atproto-proxy" on a per-request basis
* remove release candidate versions from changelog
* update changeset for api and xrpc packages
* Add missing changeset
* revert RC versions
* Proper wording in OAUTH.md api example
* remove "pre" changeset file
* xrpc: restore original behavior of setHEader and unsetHeader
* docs: add comment for XrpcClient 's constructor arg
* feat(api): expose "schemas" publicly
* feat(api): allow customizing the whatwg fetch function of the AtpAgent
* docs(api): improve migration docs
* docs: change reference to BskyAgent to AtpAgent
* docs: mention the breaking change regarding setSessionPersistHandler
* fix(api): better split AtpClient concerns
* fix(xrpc): remove unused import
* refactor(api): simplify class hierarchu by removeing AtpClient
* fix(api): mock proper method for facets detection
* restore ability to restore session asynchronously
* feat(api): allow instantiating Agent with same argument as super class
* docs(api): properly extend Agent class
* style(xrpc): var name
* docs(api): remove "async" to header getter
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Hailey <me@haileyok.com>
* chore(deps): update zod
* chore(deps): update pino to match entryway version
* chore(tsconfig): remove truncation of types through noErrorTruncation
* add support for DPoP token type when logging
* fix(bsky): JSON.parse does not return value of type JSON
* fix(pds): add res property to ReqCtx
* fix(pds): properly type getPreferences return value
* chore(tsconfig): disable noFallthroughCasesInSwitch
* refactor(pds): move tracer config in own file
* feat(dev-env): start with "pnpm dev"
* feat(oauth): add oauth provider & client libs
* feat(pds): add oauth provider
* chore: changeset
* feat: various fixes and improvements
* chore(deps): update better-sqlite3 to version 10.0.0 for node 22 compatibility
* chore(deps): drop unused tslib
* fix(did): normalize service IDs before looking for duplicates
* fix(did): avoid minor type casting
* fix(did): improve argument validation
* fix(fetch): explicit use of negation around number comparison
* fix(oauth-provider): improve argument validation
* feat(did): add ATPROTO specific "isAtprotoDidWeb" method
* feat(rollup-plugin-bundle-manifest): add readme
* feat(lint): add eqeqeq rule (only allow == and != with null)
* fix(oauth-client-browser): typo in gitignore
* fix(oauth-provider): properly name error class file
* fix(oauth-provider): remove un-necessary useMemo
* fix(did-resolver): properly build did:web document url
* fix(did-resolver): remove unused types
* fix(fetch): remove unused utils
* fix(pds): remove unused script and dependency
* fix(oauth-provider): simplify isSubPath util
* fix(oauth-provider): add InvalidRedirectUriError static constructor
* fix(jwk): improve JWT validation to provide better error messages and distinguish between signed and unsigned tokens
* fix(pds): use "debug" log level for fetch method
* fix(pds): allow access tokens to contain an unknown "typ" claim (with the exception of "dpop+jwt")
* fix(jwk): remove un-necessary code
* fix(pds): account for whitespace chars when checking JSON
* fix(pds): remove oauth specific config
* fix(pds): run all write queries through transaction or executeWithRetry
fix(pds): remove outdated comments
fix(pds): rename used_refresh_token columns & added primary key
fix(pds): run cleanup task through backgroundQueue
fix(pds): add device.id foreign key to device_account
fix(pds): add comment on cleanup of used_refresh_token
fix(pds): add primary key on device_account
* fix(oauth-provider:time): simplify constantTime util
* fix(pds): rename disableSsrf into disableSsrfProtection
* fix(oauth-client-react-native): remove incomplete package
* refactor(pds): remove status & active from ActorAccount
* fix(pds): invalidate all oauth tokens on takedown
* fix(oauth-provider): enforce token expiry
* fix(pds): properly support deactivated accounts
* perf(pds:db): allow transaction function to be sync
* refactor(psq:account-manager): expose only query builders & data transformations utils from helpers
* fix(oauth-provider): imports from self
* fix(ci): add nested packages to build artifacts
* style(fetch): rename TODO into @TODO
* style(rollup-plugin-bundle-manifest): remove "TODO" from comment
* style(oauth-client): rename TODO into @TODO
* style(oauth-provider): rename TODO into @TODO
* refactor(oauth-client): remove "OAuth" prefix from types
* fix(oauth-client-browser): better type SessionListener
* style(oauth): rename TODO into @TODO
* fix(oauth-provider): enforce provider max session age
* fix(oauth-provider): check authentication parameters against all client metadata
* fix(api): tests
* fix(pds): remove .js from imports for tests
* fix(pds): change account status to match tests
* chore(deps): make all packages depend on the same zod version
* fix(common-web): remove un-necessary binding of Checkable to "zod"
* refactor(jwk): infer jwt schema from refinement definition
* fix(handle-resolver): allow resolution errors to propagate
docs(handle-resolver): better handling of DNS resolution errors
fix(handle-resolver): properly handle DOH responses
* fix(did): service endpoint arrays must contain "one or more" element
* refactor(pipe): simplify implementation
* fix(pds): add missing DB indexes
* feat(oauth): Resolve Authorization Server URI through Protected Resource Metadata
* style:(oauth-client): import order
* docs(oauth-provider:redirect-uri): add reference url
* feat(oauth): implement "OAuth Client ID Metadata Document" from draft-parecki-oauth-client-id-metadata-document-latest internet draft
* feat(oauth-client): backport changes from feat-oauth-client
* docs(simple-store): improve comments
* feat(lexicons): add iterable capabilities
* fix(pds): type error in dev mode
* feat(oauth-provider): improved error reporting
* fix(oauth-types): allow insecure issuer during tests
* fix(xrpc-server): allow upload of empty files
* fix: lint
* feat(fetch): keep request reference in errors
feat(fetch): utilities improvements
* fix(pds): allow more than one session token per user
* feat(ozone): improve env validation error messages
* fix(oauth-client): account for DPoP when checking for invalid_token errors
* fixup! feat(fetch): keep request reference in errors feat(fetch): utilities improvements
* fixup! feat(fetch): keep request reference in errors feat(fetch): utilities improvements
* fix(oauth): various validation fixes
feat(oauth): share client_id validation and parsing utilities between client & provider
* feat(dev-env): fix ozone port number
* fix(fetch-node): prevent fetch against invalid domain names
* fix(oauth-provider): add typings for psl dep
* feat(jwk): make type def compatible with TS 4.x
* fix(oauth): fixed various spec compliance
fix(oauth): return "sub" in refresh token response
fix(oauth): limit token validity for third party clients
fix(oauth): hide client image when not trusted
* fix(oauth): lint
* pds: switch changeset to patch, no breaking changes
* changeset and config for new oauth deps
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* move nsid package in to identifier
* move uri package in to identifier
* update packages to pull from identifier, not nsid or uri
* rename @atproto/identifier to @atproto/syntax
* update all refs from @atproto/identifer to @atproto/syntax
* Add languages field to post record
* helper for parsing bcp47
* add language format to lexicon
* codegen for post record langs field
* re-export language parsing in api package
* tests and tidy for lexicon language format
* index post langs, in-progress
* update snapshots, fixes record-with-media embed issue
* index post langs on bsky appview
* don't bother indexing post langs in pds appview, tidy
lexicon: remove 'infos' field from subscriptions
This is unused in any current lexicons, and not actually implemented
anywhere in code. We refactored "info" messages in to regular messages
within a union.
This makes the whole type of bugs like #1080 impossible,
Also fixes `app.bsky.feed.defs` in the bsky package as it was now throwing errors
Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
* Add lexicon LexXrpcParameters, Object required properties check
Now schemas like the one below won't validate
```
...
{
"type": "object",
"required": ["foo"],
"properties": {}
}
...
````
* Fix lexicons with missing required fields
* Improve lexicon user type error messages and document why it needs to be `z.custom`
Improve BskyAgent performance
By using z.discriminatedUnion instead of z.union
we can hint to zod at which field it should look when
validating/parsing schemas
This improves BskyAgent instantiation time around 4 times,
when creating 10 agents using stuff from #1015,
the time went down from ~1200ms to 330ms
* remove return in test
* couple of fixups in other pacakges
* Add dummy checks to declaration and follow app migrations, remove paranoid join
* update db nsid migration
* Ensure there are writes in follow app migration
* Add dumy check to votes-to-likes app migration, tidy
* Ensure there are writes in vote-to-like app migration
* update migration name
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* Update api package to use updated richtext facets
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* Fix notification reason for change from vote to like
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* Add helper to consume subscription message frames
* Implement xrpc subscription consumer
* Simplify subscription consumer params
* Remove duped test
* Reconnect to subscription on initial setup
* Add reconnect hook on subscription, for logging
* Make subscription consumption cancelable
* Include error information in sub reconnect hook
* Yield info frames in byMessage()
* Scaffold xrpc-stream
* Write and test stream frames
* Write and test stream server
* Test bad stream frame parsing cases
* Proof of concept streaming on xrpc-server
* Test bad streaming endpoint
* Make frame stream to fix buffering frame bytes together
* reorg
* some lex-cli things
* better codegen for subscriptions
* scaffolding repo subscriptions
* wip
* remove repo ops
* setup notify/listen for db
* end pool to fix hanging test
* small comment in test
* basic sequencer
* some refactoring
* switch to event emitter
* reconnect on listener error
* rename notifyClient
* remove payload on channels
* pr feedback
* subscribeRepo outbox
* some cleanup
* wip
* wip
* bugfixin
* only send msgs after tx is committed
* better handle event-emitter -> generator
* max buffer size
* cleanup
* Sync-up xrpc subscriptions with current iteration of the spec
* Allow missing encoding for subscriptions
* track sequencedAt & eventType as well
* Resolve streaming type codes, streaming tests
* Fix interface for hooking into http server for streaming, fix stream routing
* Minor reorg how streaming is enabled in xrpc-server
* Server codegen for xrpc subscriptions, supporting types in xrpc-server
* fix up buffer overloading
* Rename data frame to message frame, rename subscription lex output to message
* Move body param first for streaming frames
* Tidy
* clean up evt types
* buff up tests
* missed merge conflict
* new schema
* blobs on subscriptions
* rm genned client subscription methods
* backfill limits
* testing subscription route & quick outbox bugfix
* fix up migration
* cascade on delete
* comments & naming
* fix dev env
* delete seqs on account deletion
* tidy
* fixing things up with db notify system for schemas
* fix duplicates in outbox
* tidy
* Break out of stream handler when socket closes
* fixing up some timing issues
* tidy
* terminate ws after using
* bump up timer on async reader
* fixing up NOTIFY in txns
* pr feedback
* pr bugfixes
* make order asc explicit
* bringing tests up to speed w atpagent
* bump up max listeners on sequencer
* increase timeouts for reading from generators
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Update LexXrpcBody encoding to always be a single string with comma-separated values
* Update lexicon union-type to use as a discriminator, and add closed flag to establish if new union members may be added
* Fix to generated code for open unions
* Implement types and validation for new NIH lexicon DSL
* Update xprc and xrpc-server to use new NIH lexicon DSL
* Simplification: dont allow lexicon definitions to be refs or ref arrays
* Add 'unknown' primitive type to NIH lexicon
* Implement client codegen for NIH Lexicon
* Implement server codegen for NIH Lexicon
* Add the 'datetime' primitive type to the NIH Lexicon
* Rework lexicons to use new NIH Lexicon DSL
* Readd tokens to code generation
* Update all code to use new NIH Lexicon (all tests passing!)
* Add imports from external files to lex-cli codegen
* Do not allow records, queries, or procedures outside of the main def
* Update packages/lex-cli/src/util.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Linter pass
* Add knownValue output to codegen
* Add comments to codegen
* Use enum and const in codegen
* Rework lexicon 'properties' structure
* Treat falsy query params as undefined
Co-authored-by: devin ivy <devinivy@gmail.com>
* Remove now-redundant param presence validation
Co-authored-by: devin ivy <devinivy@gmail.com>
* Model procedure for setting a vote on a post
* Implement app.bsky.feed.setVote() method
* Support json schema subset for parameters in lexicon and lex-cli
* Update lexicons w/ json schema parameters
* Use enum in getVotes parameters
* Support json schema subset for parameters in xrpc-server, ajv validation
* Include parameter enums in generated markdown
* Tidy
* Unify xrpc parameter md docs with inputs and outputs: show ts interfaces
* Tidy
* Add token schema types to lexicon package
* Add token-schema support to lex-cli package generation of markdown, client code, and server code
* Add two token lexicons: actorUser and actorScene
* Remove dead code
* Rename ADX to ATP
* Remove old docs (now in atproto website repo)
* Update readme
* Update README.md
Co-authored-by: devin ivy <devinivy@gmail.com>
* A couple of @ -> AT
Co-authored-by: devin ivy <devinivy@gmail.com>
* update schemas for keyed
* codegen dict
* codgen record ids
* determine key type from schema
* allow non-tid keys
* rm tids from batch write schema
* use schema ids record in db layouts
* standardize on rkey
* doc update
* crud test
* repo tests for custom rkeys
* Update lexicon doc format to support .defs field and .descriptions within method bodies
* Add correct resolution to lexicon, including to lex: URIs
* Update lexicons and generated code to use new resolution
* Add lexicon doc
* Update error-handling spec
* Implement new error-behaviors in xrpc and xrpc-server packages
* Update lexicon and lex-cli packages to add xrpc error behaviors
* Generate new API and test an error behavior
