* chore(deps): update zod
* chore(deps): update pino to match entryway version
* chore(tsconfig): remove truncation of types through noErrorTruncation
* add support for DPoP token type when logging
* fix(bsky): JSON.parse does not return value of type JSON
* fix(pds): add res property to ReqCtx
* fix(pds): properly type getPreferences return value
* chore(tsconfig): disable noFallthroughCasesInSwitch
* refactor(pds): move tracer config in own file
* feat(dev-env): start with "pnpm dev"
* feat(oauth): add oauth provider & client libs
* feat(pds): add oauth provider
* chore: changeset
* feat: various fixes and improvements
* chore(deps): update better-sqlite3 to version 10.0.0 for node 22 compatibility
* chore(deps): drop unused tslib
* fix(did): normalize service IDs before looking for duplicates
* fix(did): avoid minor type casting
* fix(did): improve argument validation
* fix(fetch): explicit use of negation around number comparison
* fix(oauth-provider): improve argument validation
* feat(did): add ATPROTO specific "isAtprotoDidWeb" method
* feat(rollup-plugin-bundle-manifest): add readme
* feat(lint): add eqeqeq rule (only allow == and != with null)
* fix(oauth-client-browser): typo in gitignore
* fix(oauth-provider): properly name error class file
* fix(oauth-provider): remove un-necessary useMemo
* fix(did-resolver): properly build did:web document url
* fix(did-resolver): remove unused types
* fix(fetch): remove unused utils
* fix(pds): remove unused script and dependency
* fix(oauth-provider): simplify isSubPath util
* fix(oauth-provider): add InvalidRedirectUriError static constructor
* fix(jwk): improve JWT validation to provide better error messages and distinguish between signed and unsigned tokens
* fix(pds): use "debug" log level for fetch method
* fix(pds): allow access tokens to contain an unknown "typ" claim (with the exception of "dpop+jwt")
* fix(jwk): remove un-necessary code
* fix(pds): account for whitespace chars when checking JSON
* fix(pds): remove oauth specific config
* fix(pds): run all write queries through transaction or executeWithRetry
fix(pds): remove outdated comments
fix(pds): rename used_refresh_token columns & added primary key
fix(pds): run cleanup task through backgroundQueue
fix(pds): add device.id foreign key to device_account
fix(pds): add comment on cleanup of used_refresh_token
fix(pds): add primary key on device_account
* fix(oauth-provider:time): simplify constantTime util
* fix(pds): rename disableSsrf into disableSsrfProtection
* fix(oauth-client-react-native): remove incomplete package
* refactor(pds): remove status & active from ActorAccount
* fix(pds): invalidate all oauth tokens on takedown
* fix(oauth-provider): enforce token expiry
* fix(pds): properly support deactivated accounts
* perf(pds:db): allow transaction function to be sync
* refactor(psq:account-manager): expose only query builders & data transformations utils from helpers
* fix(oauth-provider): imports from self
* fix(ci): add nested packages to build artifacts
* style(fetch): rename TODO into @TODO
* style(rollup-plugin-bundle-manifest): remove "TODO" from comment
* style(oauth-client): rename TODO into @TODO
* style(oauth-provider): rename TODO into @TODO
* refactor(oauth-client): remove "OAuth" prefix from types
* fix(oauth-client-browser): better type SessionListener
* style(oauth): rename TODO into @TODO
* fix(oauth-provider): enforce provider max session age
* fix(oauth-provider): check authentication parameters against all client metadata
* fix(api): tests
* fix(pds): remove .js from imports for tests
* fix(pds): change account status to match tests
* chore(deps): make all packages depend on the same zod version
* fix(common-web): remove un-necessary binding of Checkable to "zod"
* refactor(jwk): infer jwt schema from refinement definition
* fix(handle-resolver): allow resolution errors to propagate
docs(handle-resolver): better handling of DNS resolution errors
fix(handle-resolver): properly handle DOH responses
* fix(did): service endpoint arrays must contain "one or more" element
* refactor(pipe): simplify implementation
* fix(pds): add missing DB indexes
* feat(oauth): Resolve Authorization Server URI through Protected Resource Metadata
* style:(oauth-client): import order
* docs(oauth-provider:redirect-uri): add reference url
* feat(oauth): implement "OAuth Client ID Metadata Document" from draft-parecki-oauth-client-id-metadata-document-latest internet draft
* feat(oauth-client): backport changes from feat-oauth-client
* docs(simple-store): improve comments
* feat(lexicons): add iterable capabilities
* fix(pds): type error in dev mode
* feat(oauth-provider): improved error reporting
* fix(oauth-types): allow insecure issuer during tests
* fix(xrpc-server): allow upload of empty files
* fix: lint
* feat(fetch): keep request reference in errors
feat(fetch): utilities improvements
* fix(pds): allow more than one session token per user
* feat(ozone): improve env validation error messages
* fix(oauth-client): account for DPoP when checking for invalid_token errors
* fixup! feat(fetch): keep request reference in errors feat(fetch): utilities improvements
* fixup! feat(fetch): keep request reference in errors feat(fetch): utilities improvements
* fix(oauth): various validation fixes
feat(oauth): share client_id validation and parsing utilities between client & provider
* feat(dev-env): fix ozone port number
* fix(fetch-node): prevent fetch against invalid domain names
* fix(oauth-provider): add typings for psl dep
* feat(jwk): make type def compatible with TS 4.x
* fix(oauth): fixed various spec compliance
fix(oauth): return "sub" in refresh token response
fix(oauth): limit token validity for third party clients
fix(oauth): hide client image when not trusted
* fix(oauth): lint
* pds: switch changeset to patch, no breaking changes
* changeset and config for new oauth deps
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* tidy bsky auth
* hook up new auth verifier
* update auth throughout ozone
* handle mod signing keys
* add client proxy heads to pds
* hook up rest of routes
* simplify pipethrough & add some SSRF protection
* tests
* fix bad var
* fix key parsing in pds
* fix admin auth test
* rename test
* add pipethrough to write routes
* update did doc id values
* null creds string -> `none`
* fix fetchLabels auth check
* ✨ Add a couple more proxied requests that we use in ozone ui
* Add runit to the services/bsky Dockerfile (#2254)
add runit to the services/bsky Dockerfile
* Improve tag detection (#2260)
* Allow tags to lead with and contain only numbers
* Break tags on other whitespace characters
* Export regexes from rich text detection
* Add test
* Add test
* Disallow number-only tags
* Avoid combining enclosing screen chars
* Allow full-width number sign
* Clarify tests
* Fix punctuation edge case
* Reorder
* Simplify, add another test
* Another test, comment
* Version packages (#2261)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* 🐛 Increment attempt count after each attempt to push ozone event (#2239)
* Ozone delegates email sending to actor's pds (#2272)
* ozone delegates email sending to user's pds
* lexicon: add content field to mod email event
* test email sending via mod event
* fix auth verifier method
* build branch
* fix url check
* better error handling for get account infos
* fix labeler service id
* fix iss on auth headers
* fix dev-env ozone did
* fix tests & another jwt issuer
* ozone: fix ip check
* fix aud check on pds mod service auth
* tidy
* fix pipethrough of headers
---------
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
* tidy bsky auth
* hook up new auth verifier
* update auth throughout ozone
* handle mod signing keys
* fix bad var
* fix key parsing in pds
* fix admin auth test
* rename test
* update did doc id values
* null creds string -> `none`
* fix fetchLabels auth check
* ✨ Add a couple more proxied requests that we use in ozone ui
* Add runit to the services/bsky Dockerfile (#2254)
add runit to the services/bsky Dockerfile
* Improve tag detection (#2260)
* Allow tags to lead with and contain only numbers
* Break tags on other whitespace characters
* Export regexes from rich text detection
* Add test
* Add test
* Disallow number-only tags
* Avoid combining enclosing screen chars
* Allow full-width number sign
* Clarify tests
* Fix punctuation edge case
* Reorder
* Simplify, add another test
* Another test, comment
* Version packages (#2261)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* 🐛 Increment attempt count after each attempt to push ozone event (#2239)
* Ozone delegates email sending to actor's pds (#2272)
* ozone delegates email sending to user's pds
* lexicon: add content field to mod email event
* test email sending via mod event
* fix auth verifier method
* better error handling for get account infos
* fix labeler service id
* fix iss on auth headers
* fix dev-env ozone did
* fix tests & another jwt issuer
* ozone: fix ip check
* fix aud check on pds mod service auth
* tidy
---------
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
* draft of account migration lexicons
* format
* clean up schemas
* codegen
* second pass on schemas
* small fix
* move around checkImportStatus
* re-codegen
* getServiceAuth
* getServiceAuth impl
* importRepo impl
* handle uploadBlob for import
* allow bringing your own did on createAccount
* working on test flow
* fleshing out flow
* fix up sendPlcOP
* small fixes
* activate/deactivate account
* full flow working!
* schema tweaks
* format
* update schemas
* moar codegen
* match impl to new schemas
* email flow for signed plc operation
* add email flow for plc operations
* impl plc op request email
* proxy to entryway
* tidy activate account
* integrating account deactivated state
* fix up tests
* friendly parse on optional did auth
* admin activate/deactivate routes
* proxy relevant requests to entryway
* remove admin activation routes
* do not proxy acitvaition to entryway
* cfg for disallowing imports
* buff up test
* refactor listMissingBlobs a bit
* add validDid & activated to accoutn status
* emit event on account activation
* test creating a post after migrating
* account deactivation tests
* test name
* tests on plc operations
* fix recommended did creds
* codegen
* turn off accepting imports on createAccount
* undo prev change
* increment version
* build branch
* pr feedback
* handle errs in p-queue
* handle blob upload outside of txn
* Clean old temp account migration lexicons (#2187)
* clean old temp lexicons
* rm old test
* fix agent session test
* fix bsky test
* dont build branch
* mv appview
* copy
* finalize copy
* package names
* big WIP
* first pass at mod servce
* some tidy
* tidy & fix compiler errors
* rename to ozone, db migrations, add to dev-env & pds cfg
* getRecord & getRepo mostly working
* fix open handle
* get record tests all working
* moderation events working
* statuses working
* tidy test suite
* search repos
* server & db tests
* moderation tests
* wip daemon + push events
* pds fanout working
* fix db test
* fanning takedowns out to appview
* rm try/catch
* bsky moderation test
* introduce mod subject wrappers
* more tidy
* refactor event reversal
* tidy some db stuff
* tidy
* rename service to mod-service
* fix test
* tidy config
* refactor auth in bsky
* wip patching up auto-mod
* add label ingester in appview
* fix a couple build issues
* fix some timing bugs
* tidy polling logic
* fix up tests
* fix some pds tests
* eslint ignore
* fix ozone tests
* move seeds to dev-env
* move images around
* fix db schemas
* use service auth admin reqs
* fix remaining tests
* auth tests bsky
* another test
* random tidy
* fix up search
* clean up bsky mod service
* more tidy
* default attempts to 0
* tidy old test
* random tidy
* tidy package.json
* tidy logger
* takedownId -> takedownRef
* misc pr feedback
* split daemon out from ozone application
* fix blob takedown mgiration
* refactor ozone config
* do push event fanout on write instead of on read
* make suspend error work again
* add attempts check & add supporting index
* fix takedown test ref
* get tests working
* rm old test
* fix timing bug in event pusher tests
* attempt another fix for timing bug
* await req
* service files
* remove labelerDid cfg
* update snaps for labeler did + some cfg changes
* fix more snaps
* pnpm i
* build ozone images
* build
* make label provider optional
* fix build issues
* fix build
* fix build
* build pds
* build on ghcr
* fix syntax in entry
* another fix
* use correct import
* export logger
* remove event reverser
* adjust push event fanout
* push out multiple
* remove builds
* cleanup repeat process all
* wip
* skip actor search test
* skip actor search test
* tweak processAll
* decrease wait to 1 sec
* repo_blob -> record_blob
* simplify backlink linkTo
* return repo_root to one row
* sequence before updating repo_root
* invite code forUser -> forAccount
* ipld_block -> repo_block
* use lru-cache fetchMethod
* move did_cache to own db
* better error handling on did cache
* drop did_handle
* fix sequencer wait time
* debug
* debug
* more debug
* check something
* fix bday paradox
* fix bday paradox
* tidy up pds service auth
* rm skipped test
* retry http
* tidy
* improve fanout error handling
* fix test
* return signing key in did-web
* more tests
* tidy serivce auth checks
* user_account -> account
* remove inviteNote
* keypair per repo
* use an lru cache for keypairs as well
* clean up repo
* wip
* wrap up accoutn manager
* tidy
* tidy
* fix tests
* fix disabled codes
* fix appview tests
* add note
* set pragmas
* tidy account manager getDb
* rename pref transactor
* user pref -> account pref
* handle blob imports
* tidy imports
* add reserveSigningKey
* wip transferAccount
* clean up transferAccount
* tests
* tidy
* tidy
* configure entryway url on pds
* handle entryway in pds admin endpoints
* make importRepo temp
* fix imports
* make email optional on pds when using entryway
* handle diffs
* handle pds entryway usage for server, identity, admin endpoints
* pds support for credentials from entryway
* setup pds tests w/ entryway service
* tidy
* tidy
* update entryway version
* wip
* test handle updates w/ entryway
* split account table into two
* tidy
* tweak scripts
* tidy tests
* tidy
* better config for actorstore & dbs
* clean up cfg more
* reorg actorstore fs layout
* handle erros on actor db create
* pr tidy & fix accoutn deletion test
* pr feedback
* fix bad merge
* unskip test
* fix subscribe repos tests
* tidy repo root tables
* tidy
* fix tests
* tidy delete tokens
* tidy account getters
* tidy
* bulk deletesg
* increase chunk size
* handle racing refreshes
* wip
* fix auth test
* invert import flow
* clean up actor store on create account failure
* tweak sequencer
* prevent invite code races on createAccount
* rm note
* add back in race protection on getAccountInviteCodes
* start feature branch
* deleted app migration table
* patch up new auth test
* rm note
* g
* create accoutn delegated from entryway
* tidy
* fix test
* change plcOp type to unknown
* small fixes
* sync up w entryway branch
* Use proper error when authed account is not found (#1799)
provide proper error when account not found in access-takedown check
* build branch
* build on ghcr
* tweak service file
* tweak service file
* change where we save reserved keys
* no tmp dir in blobstore either
* fix blobstore temp location again
* handle repeat record_blobs
* create account before submitting plc op & undo if fail
* small tweak
* limit the number of local records
* push out empty commit on transfer
* fix issue with record_blob
* add push blob endpoint
* Set and validate token audiences on pds v2 (#1793)
set and validate token audience on pds v2
* merge
* include entryway did on tests
* build branch
* fix cache issue
* xrpc server blob limit
* put correct bytes
* add auth to routes
* handle quarantining/unquarantining a blob that does not exist
* tidy
* fix transfer tests
* fix email request routes for entryway
* PDS v2 entryway account deletion (#1819)
* add admin lexicon for account deletion
* implement admin account deletion endpoint
* fix entryway proxying on account email checks
* proxy to entryway for acct deletion
* read-after-write sanity check
* tweak
* wip
* finish refactor
* fix test schema
* application retry logic for busy
* pr feedback
* rm lru-cache
* fix test pg schema
* fix transfer test
* Sqlite instrumentation for pds v2 (#1838)
* sqlite instrumentation
* build
* remove build
* dont reimport blobs
* send ticks during import
* close on error
* catch handle validation error
* add log
* fix test
* return emailConfirmedAt on getAccountInfo
* Upgrade sharp on pds v2 (#1863)
upgrade sharp to 0.32.6
* read all bytes before parsing car
* Async car reader (#1867)
* asynchronously read in car
* dont buffer car
* tweak
* Gracefully handle indexing of invalid records (#1853)
* gracefully handle indexing of invalid records
* fix repo tests
* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)
fix role auth for access-or-role verifier, fix getBlob actor takedown check
* better cleanup of actor-stores
* add ability to not ensure leaves
* tidy
* allow did:web transfer
* Migration utility for actor-store (#1873)
beginnings of helper for migrating all actors
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* base case for findBlobRefs
* App-level retries for sqlite on pds (#1871)
* revamp retry helper to be more flexible re: backoff strategies
* sqlite timeout helper
* ensure sqlite wal on db creation/migration rather than every open
* layer retries for sqlite on writes outside transactions on pds
* tidy
* fix up lockfile
* tidy
* fix lex codegen
* fix timing bug in threadgate test
* No-op update handling (#1916)
do no produce commits on no-op updates
* Retry on all SQLITE_BUSY error codes (#1917)
retry on all sqlite_busy error codes
* Pds v2 ensure sqlite ready (#1918)
ensure sqlite is ready before making queries
* try something
* tidy
* dont build branch
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicon for did doc w/ auth credentials
* include did doc w/ session when configured. configure on dev-env.
* Add dynamic PDS URL adoption to the client
* remove usage of did doc field from getsession in client
* dry-up did doc type and validation
* remove explicit dep on zod by identity package
* move more did doc parsing to common-web
* go back to strings
* rollback breaking changes to identity package
* add changeset
---------
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: dholms <dtholmgren@gmail.com>
* lexicons
* codegen
* email templates
* request routes
* impl
* migration
* tidy
* tests
* tidy & bugfixes
* format
* fix api test
* fix auth test
* codegen
* add unique constraint
* Add email confirmed to AtpSessionData
* interop test files (#1529)
* initial interop-test-files
* crypto: switch signature-fixtures.json to a symlink
* syntax: test against interop files
* prettier
* Update interop-test-files/README.md
Co-authored-by: Eric Bailey <git@esb.lol>
* disable prettier on test vectors
---------
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>
* add getSuggestedFollowsByActor (#1553)
* add getSuggestedFollowsByActor lex
* remove pagination
* codegen
* add pds route
* add app view route
* first pass at likes-based suggested actors, plus tests
* format
* backfill with suggested_follow table
* combine actors queries
* fall back to popular follows, handle backfill differently
* revert seed change, update test
* lower likes threshold
* cleanup
* remove todo
* format
* optimize queries
* cover mute lists
* clean up into pipeline steps
* add changeset
* List feeds (#1557)
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* lex
* list purpose
* lex gen
* add route
* add proxy route
* seed client helpers
* tests
* mutes and blocks
* proxy test
* snapshot
* hoist actors out of composeThread()
* tidy
* tidy
* run ci on all prs
* format
* format
* fix snap name
* fix snapsh
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Improve xrpc server error handling (#1597)
improve xrpc server error handling
* Remove appview proxy runtime flags (#1590)
* remove appview proxy runtime flags
* clean up proxy tests
* getPopular hotfix (#1599)
dont pass all params
* Interaction Gating (#1561)
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* initial lexicons for interaction-gating
* add interactions view to post views
* codegen
* model bad reply/interaction check state on posts
* initial impl for checking bad reply or interaction on write
* omit invalid interactions from post thread
* support not-found list in interaction view
* hydrate can-reply state on threads
* present interaction views on posts
* misc fixes, update snaps
* tidy/reorg
* tidy
* split interaction gating into separate record in lexicon
* switch interaction-gating impl to use separate record type
* allow checking reply gate w/ root post deletion
* fix
* initial gating tests
* tighten gated reply views, tests
* reply-gating list rule tests
* allow custom post rkeys within window
* hoist actors out of composeThread()
* tidy
* update thread gate lexicons, codegen
* lex fix
* rename gate to threadgate in bsky, update views
* lex fix
* improve terminology around reply validation
* fix down migration
* remove thread gates on actor unindexing
* add back .prettierignore
* tidy
* run ci on all prs
* syntax
* run ci on all prs
* format
* fix snap
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* order by `like.indexedAt` in app view (#1592)
* order by like.indexedAt
* use keyset for ordering
* simplify
* ok ok ok I get it now
* Update packages/bsky/src/api/app/bsky/feed/getActorLikes.ts
Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
---------
Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
* Remove default value for post table invalid attrs (#1601)
remove default value for post table attrs
* Version packages (#1602)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* update Bluesky PBLLC to PBC (Public Benefit Corporation) (#1600)
* Temporarily disable filtering `invalidReplyRoot`s (#1609)
temporarily disable invalidReplyRoot check
* fix syntax docs (#1611)
* Version packages (#1612)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Allow bypass on ratelimit ip (#1613)
allow bypass on ratelimit ip
* Write rate limits (#1578)
* get rate limit ip correctly
* add write rate-limits
* Tweak createSession rate limit key (#1614)
tweak create session rl key
* Filter preferences for app passwords (#1626)
filter preferences for app passwords
* Tweak rate limit setup for multi rate limit routes (#1627)
tweak rate limit setup for multi rate limit routes
* Remove zod from xrpc-server error handling (#1631)
remove zod from xrpc-server error handling check
* Enforce properties field on lexicon object schemas (#1628)
* add empty properites to thread gate schema fragments
* tweak lexicon type
* Add feed-vew and thread-view preferences (#1638)
* Add feed and thread preference lexicons
* Add feed-view and thread-view preference APIs
* Add changeset for new preferences (#1639)
Add changeset
* Version packages (#1640)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Disable getAccountInviteCodes for app passwords (#1642)
disable getAccountInviteCodes for app passwords
* remove cruft packages (uri, nsid, identifier) (#1606)
* remove @atproto/nsid (previously moved to syntax)
* remove @atproto/uri (previously moved to syntax)
* remove @atproto/identifier (previously moved to syntax)
* bump lockfile to remove old packages
---------
Co-authored-by: Eric Bailey <git@esb.lol>
* api: update login/resumeSession examples in README (#1634)
* api: update login/resumeSession examples in README
* Update packages/api/README.md
Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
---------
Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
* small syntax lints (#1646)
* lint: remove unused imports and variables
* lint: prefix unused args with '_'
* eslint: skip no-explicit-any; ignore unused _var (prefix)
* eslint: explicitly mark ignores for tricky cases
* indicate that getPopular is deprecated (#1647)
* indicate that getPopular is deprecated
* codegen for deprecating getPopular
* tidy up package.json and READMEs (#1649)
* identity: README example and tidy
* tidy up package metadata (package.json files)
* updated README headers/stubs for several packages
* crypto: longer README, with usage
* syntax: tweak README
* Apply suggestions from code review
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>
---------
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>
* Improve the types of the thread and feed preferences APIs (#1653)
* Improve the types of the thread and feed preferences APIs
* Remove unused import
* Add changeset
* Version packages (#1654)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Disable pds appview routes (#1644)
* wip
* remove all canProxyReadc
* finish cleanup
* clean up tests
* fix up tests
* fix api tests
* fix build
* fix compression test
* update image tests
* fix dev envs
* build branch
* fix service file
* re-enable getPopular
* format
* rm unused sharp code
* dont build branch
* auto-moderator tweaks: pass along record URI, create report for takedown action (#1643)
* auto-moderator: include record URI in abyss requests
* auto-moderator: log attempt at hard takedown; create report as well
The motivation is to flag the event to mod team, and to make it easier
to confirm that takedown took place.
* auto-mod: typo fix
* auto-mod: bugfixes
* bsky: always create auto-mod report locally, not pushAgent (if possible)
* bsky: fix auto-mod build
* bsky: URL-encode scanBlob call
* Clear follow viewer state when blocking (#1659)
* clear follow viewer state when blocking
* tidy
* add `tags` to posts (#1637)
* add tags to post lex
* kiss
* add richtext facet and validation attrs
* add tag validation attrs to post
* codegen
* add maxLength for tags, add description
* validate post tags on write
* add test
* handle tags in indexer
* add tags to postView, codegen
* return tags on post thread view
* format
* revert formatting change to docs
* use establish validation pattern
* add changeset
(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)
* remove tags from postView, codegen
* remove tags from thread view
* revert unused changes
* Version packages (#1664)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* merge
* Reverse order of blocks from sync.getRepo (#1665)
* reverse order of blocks from sync.getRepo
* write to car while fetching next page
* Add hashtag detection to richtext (#1651)
* add tag detection to richtext
* fix duplicate tag index error
* add utils
* fix leading space index failures, test for them
* add changeset
* Version packages (#1669)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* proposed new search lexicons (#1594)
* proposed new search lexicons
* lexicons: lint
* lexicons: fix actors typo
* lexicons: camelCase bites again, ssssss
* lexicons: add 'q' and mark 'term' as deprecated for search endpoints
* codegen for search lexicon updates
* bsky: prefer 'q' over 'term' in existing search endpoints
* search: bugfix
* lexicons: make unspecced search endpoints return skeleton obj
* re-codegen for search skeleton obj
* Disable pds appview indexing (#1645)
* rm indexing service
* remove message queue & refactor background queue
* wip
* remove all canProxyReadc
* finish cleanup
* clean up tests
* fix up tests
* fix api tests
* fix build
* fix compression test
* update image tests
* fix dev envs
* build branch
* wip - removing labeler
* fix service file
* remove kysely tables
* re-enable getPopular
* format
* cleaning up tests
* rm unused sharp code
* rm pds build
* clean up tests
* fix build
* fix build
* migration
* tidy
* build branch
* tidy
* build branch
* small tidy
* dont build
* Refactor PDS appview routes (#1673)
move routes around
* Strip leading `#` from from detected tag facets (#1674)
ensure # is removed from facets
* Version packages (#1675)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Proxy search queries (#1676)
* proxy search
* tweak profile resp
* fix admin.searchRepos
* add mock mailer
* Fix to daniel's MOCKERY of a mock mailer
* Don't allow non-verified email updates until app feature is out (#1682)
stricter updating email until app feature is out
* changesets
---------
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* use pnpm
* fix dependency issues, replace yarn and lerna scripts
* remove the main/dist scripts
* update Dockerfiles
* use pnpm
* fix dependency issues, replace yarn and lerna scripts
* remove the main/dist scripts
* update Dockerfiles
* update bin script
* remove unused zod dep
* fix type errors in pds
* add types prop to packages
* remove unused, bump lock
* fix test running
* build before test
* fix pino types
* format
* pds depends on dev-env in test
* refer to src instead of built packages
* pds relies on bsky in test too
* remove yarn.lock
* add -r flag to root test
* test push to aws
* remove docker test
* add publishConfig to new package
* move services to top level dir
(cherry picked from commit f5012bec33435a4473e9960066807623334f3aff)
* update workflow paths
(cherry picked from commit 5c70f0176d381ca35d6be10cfa173e22373a5b5d)
* add main-to-dist script
* use script in all packages, remove old Dockerfiles
* remove old bsky service
* remove newline
* test container builds
* Revert "test container builds"
This reverts commit c228611f5e8e1624d4b124be4976c49590130f43.
* remove unused config
* test build containers
* pnpm in syntax
* bump dd-trace
* shamefully hoist
* even more shame
* hoist, externalize deps
* clean install for prod and smaller containers
* dont build branches
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* first pass on profiles
* quick test
* wip
* wip
* test post thread
* record embeds
* get author feed profiles
* wip timeline
* fix get timeline
* switch from counter to tid
* tidy into a service
* quick tid test
* pr feedback
* clock -> rev
* update image formatting
* disable migration & build branch
* add recent posts to getAuthorFeed & handle post thread not found errors
* refactor for lag header
* tidy
* rm collections check
* tidy test
* pr feedback
* fix small bug
* build branch
* get migrations into system
* enable migrations
* fix up bug in actor service
* add in hydrate methods
* build fix
* couple more fixes
* tidy
* port to appview
* dont build branch
* fix issue with items in list
* tidy
* dont build branch
* quick fix
* Add languages field to post record
* helper for parsing bcp47
* add language format to lexicon
* codegen for post record langs field
* re-export language parsing in api package
* tests and tidy for lexicon language format
* index post langs, in-progress
* update snapshots, fixes record-with-media embed issue
* index post langs on bsky appview
* don't bother indexing post langs in pds appview, tidy
* wip
* first pass done
* chnage around how channels function
* bugfixing
* fix up migration & tests
* fix dev-env
* add prep migration for stream fixing
* application code changes to use new repo_event table
* dont send evts
* add outgoing_repo_seq table + sqlite migrations
* patch up down migration
* patching up a couple things
* update test
* patchup
* re-enable subscription route
* add outgoingSeq column
* updating
* accidental add
* temporarily disable stream
* resolve merge conflict
* add migration
* migration for seq
* fix test
* id -> seq
* get schema lock ids
* tidy
* fix server test
* fix up
* use bsky uniqueLockId approach
* tidy
* gh actions build
* build branch
* build branch
* tweak test
* rm constraint rename
* small bugfix
* check for seq not null
* fix up issues
* one more small change
* tidy
* no build branches
* block lexicons
* table schema
* block processing
* Actor-block migration
* Add blocked post lexicon to getPostThread
* getBlocks routes
* Add missing lexicons to bsky
* Add block state to actor views
* throw on blocked accounts in getAuthorFeed
* blcok state in profile views
* Update getPostThread to support block views, update existing test
* strip out blocks out oof timeline & popular
* more routes
* Test blocked posts in threads
* Support blocks on record embeds
* notifs
* Update pds test snapshots w/ block state
* fix search
* Update pds actor search snapshots w/ blocks
* test
* tests
* update lex
* fix up from returning blocking as record
* ci for branch
* fix build issue in bsky
* more blocks tests
* fix actor search test
* hook getBlocks up to router
* bugfix & actually test lol
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* fix up a couple of tsc errors in app view merge
* wip
* simple proxy
* use dev-env for appview tests
* process all in blob resolver
* another test fix
* wip
* copy proxied tests & add mutes to getFollows/getFollowers
* tidy & add mutes to likes
* more routes + getAuthorFeed tests
* more testing
* tests for feed views
* thread testing
* finished tests for threads
* temporarily skip some tests
* cleaning up & updating test names
* separate db schema for appview
* rearrange
* typo
* add notifications
* re-enable notifs on pds appview
* update schemas
* updated some bsky snaps
* wip
* refactor did-resolver
* clean up deps
* some fixups + caching utilities
* fix up & move to appCtx
* neat its working
* update bsky tests to new auth
* rm unused pds config var
* tidy
* check exp in seconds
* cache dids in postgres
* add migration & did-cache
* start tests
* couple helpers around cache invalidation
* fix expired check
* wip
* change cache semantics
* did cache testing
* do some cache revalidation in indexing
* fix config
* fix issue w did-resolver test-env prototype
* use map instead of record
* stale + expired
* tests
* clear entry method
* fix up build
* expired dids
* clear missing dids
* better verify payload
* bump test timeout
* fix notifs test
* fix up proxied actor search tests
* update snaps to include labels
* fix dev env
* fix up moderation route auth
* fix more auth headers
* fix auth on getPosts
* increase jest timeouts
* fix snaps
* Add dumy check to votes-to-likes app migration, tidy
* Ensure there are writes in vote-to-like app migration
* update migration name
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* Update api package to use updated richtext facets
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* added test vectors for ipld serialization
* another test
* added method & test for ipld equality
* tidy
* test for unicode, floats & blob-refs
* remove floats
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* remove return in test
* couple of fixups in other pacakges
* Add dummy checks to declaration and follow app migrations, remove paranoid join
* update db nsid migration
* Ensure there are writes in follow app migration
* Add dumy check to votes-to-likes app migration, tidy
* Ensure there are writes in vote-to-like app migration
* update migration name
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* Update api package to use updated richtext facets
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* Fix notification reason for change from vote to like
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
