454 Commits

Author SHA1 Message Date
rafael
64100a75b3
Bookmarks (#4163) 2025-09-02 10:28:34 -03:00
Matthieu Sieben
f9dc9aa4c9
Permission set (#4108)
* Export constants and type assertion utilities

* Add permission set support to oauth provider

* improve permission set parsing

* Rename `PermissionSet` to `ScopePermissions`

* Improve performance of NSID validation

* Add support for `permission-set` in lexicon document

* Validate NSID syntax using `@atproto/syntax`

* Export all types used in public interfaces (from `lexicon-resolver`)

* Small performance improvement

* Rework scope parsing utilities to work with Lexicon defined permissions

* file rename

* fixup! Rework scope parsing utilities to work with Lexicon defined permissions

* removed outdated comment

* removed outdated comment

* fix comment typo

* Improve `SimpleStore` api

* permission-set NSID auth scopes

* Remove dev dependency on dev-env

* fix build script

* pnpm-lock

* Improve fetch-node unicast protection

* Explicitly set the `redirect: "follow"` `fetch()` option

* Add delay when building oauth-provider-ui in watch mode

* Remove external dependencies from auth-scopes

* Add customizable lexicon authority to pds (for dev purposes)

* fix pds migration

* update permission-set icon

* Add support for `include:` syntax in scopes

* tidy

* Renaming of "resource" concept to better reflect the fact that not all oauth scope values are about resources

* changeset

* ui improvmeents

* i18n

* ui imporvements

* add `AtprotoAudience` type

* Enforce proper formatting of audience (atproto supported did + fragment part)

* tidy

* tidy

* tidy

* fix ci ?

* ci fix ?

* tidy ?

* Apply consistent outline around focusable items

* Use `inheritAud: true` to control `aud` inheritance

* Update packages/oauth/oauth-provider/src/lexicon/lexicon-manager.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Review comments

* Add `nsid` property to `LexiconResolutionError`

* improve nsid validation

* i18n

* Improve oauth scope parsing

* Simplify lex scope parsing

* tidy

* docs

* tidy

* ci

* Code simplification

* tidy

* improve type safety

* improve deps graph

* naming

* Improve tests and package structure

* Improve error when resolving a non permission-set

* improve nsid parsing perfs

* benchmark

* Refactor ozone and lexicon into using a common service profile mechanism

* improve perfs

* ci fix (?)

* tidy

* Allow storage of valid lexicons in lexicon store

* Improve handling of lexicon resolution failures

* review comment

* Test both regexp and non regexp based nsid validation

* properly detect presence of port number in https did:web

* Re-enable logging of `safeFetch` requests

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2025-08-29 12:19:19 +02:00
Matthieu Sieben
c0126f4a84
Improve error handling when destroying pipethrough stream (#4133)
* Improve error handling when destroying pipethrough stream

fixes #4129

* docs

* pds: add failing test for abort handling

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2025-08-26 14:33:34 -04:00
David Buchanan
75162ffb9e
Fix putRecord auth check (#4104)
* Fix putRecord auth check

* changeset

* switch pds changeset to patch

* add test for putRecord via handle

* style fix

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2025-08-13 22:41:39 +01:00
Matthieu Sieben
1899b1fc16
OAuth scopes (#3806)
* style: prefix `id` and `uri` with `request` where applicable

* Dynamically validate OAuth scopes

* Allow configuring trusted OAuth clients

* Improve client validation

* Rework authorization to work with permissions

* Review changes

* fix permissions

* tidy

* Drop authorization result

* unused code cleanup

* fix preferences auth

* remove redundant check in `applyWrites`

* style

* Remove need to specify "scopes" in authorized auth strategy

* fixup! Remove need to specify "scopes" in authorized auth strategy

* split authorized and oauth auth methods

* Require explicit opt-in for takendown

* fix tests

* rollback redundant permissions mechanism

* tidy

* Fix tests

* tidy

* tidy

* pr changes

* remove hack allowing access to full preferences

* always specify authorize method

* Add OAuth scope parsing & matching

* tidy

* add support for oauth scopes in client

* review changes

* Small xrpc-server optimizations

* pr comments

* Review comments

* refactor: move oauth scopes parser & checker in own package

* code simplification

* Allow multiple collections in `repo` scopes.
Allow wildcard action in `repo` scopes.
Require action in `repo` scopes.

* Rename `emailUpdate` to `email-update` in `account` scope params.
Add wildcard (`*`) in `account` and `identity` scopes.

* tidy

* add oauth-scopes package to PDS Dockerfile

* unit tests

* Syntax rework

* adapt to latest scope definition

* Add missing tests

* Render scopes in UI

* fix build

* fixes and tests

* improve ui

* tidy

* tidy

* ui improvements

* tidy

* fr messages

* tidy

* improve consent screen ui

* fix test

* tidy

* improve dx

* Remove `transition:` scopes from `scopes_supported` authorization server metadata

* Hide blob scope if no repo scope present

* changeset

* Remove the `action` param from the `identity` scope

* fix html syntax

* simplified wording

* Make `account:email` scope optional (#4089)

* Make `account:email` scope optional

* tidy

* tidy

* tidy

* tidy

* fix

* tidy

* review comments

* tidy

* refactor: remove redundant tests for identity scope parsing and matching

* minor ui fixes

* fix "back" label not translated

* ui improvements

* fix tests
2025-08-12 13:13:14 +02:00
rafael
de29a346d8
Fix paginateAll mistakes and improve typing (#4062) 2025-07-22 14:51:16 -03:00
Foysal Ahamed
926193f825
Ozone support for age assurance compliance events (#4029)
* 🚧 WIP

*  Make age assurance state queryable

*  Split age assurance events into 2

*  Implement admin and user state overrides

*  Add blocked as a known value for age assurance state

*  Update test snapshot

*  Update test snapshot

*  Cleanup
2025-07-11 18:00:55 +02:00
devin ivy
5ed4a88596
Fix json and text uploads: don't parse bodies with input encoding of */* (#4027)
* xrpc-server: skip body parsing when input encoding is */*, fix json and text uploads

* changeset

* pds: add tests for text and json uploads

* tidy

* xrpc-server: only create body parser when it will be used
2025-07-09 18:55:48 -04:00
Matthieu Sieben
8ef976d385
xrpc-server & lex-cli rework (#3999)
* xrpc-server & lex-cli rework

* codegen

* tidy

* tidy

* tidy

* Update .changeset/nasty-icons-peel.md

Co-authored-by: devin ivy <devinivy@gmail.com>

* excludeErrorResult util

* Restore export of `SkeletonHandler` from `pds`

* Make `calcKey` RateLimiter option required

* Process request input after auth

* fix tests

* changeset

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2025-07-08 13:04:11 +02:00
Samuel Newman
8a0f190399
Subscribe to activity implementation (#3908)
Co-authored-by: rafael <rafael@blueskyweb.xyz>
2025-06-30 18:43:27 -03:00
Matthieu Sieben
09d90ae486
Improve OAuth Example app (#3952)
* Improve OAuth Example app

* Improve style

* bsync: Accept NSID with fragment in operation ns (#3954)

* Add `match: MuteWordMatch` to `muted-word` mod decision `cause` (#2934)

* Return MuteWordMatch instead of simple boolean

* Return full mute word with match

* Add MuteWordMatch to decision cause, update a few tests

* Backwards compat

* Tighter types

* Return all mute word matches

* Clean up types

* Rename

* More cleanup of naming

* Remove unneeded changes

* Format

* Add predicate value to matches

* Better migration path

* Changeset

* Import sort

* Tighten up addMuteWord API

Co-authored-by: Matthieu Sieben <matthieusieben@users.noreply.github.com>

* Mute words: handle `Andor` and `and/or` case (#3948)

* Handle Andor case

* Remove useless escape

* Changeset

---------

Co-authored-by: Matthieu Sieben <matthieusieben@users.noreply.github.com>

* Version packages (#3947)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update README.md to add some missing details in examples (#3254)

Update README.md

Improve code examples (some OAuth implementation details are missing in these examples)

* Increase oauth session & refresh token lifetimes (#3883)

* Allow HTTPS `redirect_uris` from any origin (#3811)

* bump MST key length from 256 to 1024 chars (#3956)

* bump MST key length from 256 to 1024 chars

* update MST key test

* add a changeset

* Version packages (#3959)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Rename `filter` -> `include` (#3966)

* rename filter -> include

* changeset

* fix tests

* Minor Fixes: Typo Correction and Comment Update (#3961)

* Update blob-resolver.ts

* Update index.ts

* Appview: sync up protos for notification prefs (#3970)

appview: sync up protos for notification prefs

* Version packages (#3969)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Fix invalid use of `invalid_client` (#3967)

* Replace slice() with subarray() in car file parsing (#3971)

* Replace slice() with subarray() in car file parsing

* changeset

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Re-export all types & utilities needed to instantiate an OAuth client (#3976)

* Re-export all types & utilities needed to instantiate an OAuth client

* Add `jwkPrivateSchema` to ensure a key is private

* Return object instead of array as result of `findPrivateKey`

* Allow override of default `handleResolver` and `runtimeImplementation` options for NodeOAuthClient

* changeset

* Allow `OAuthClient` to be instantiated with custom `didResolver` instance

* Version packages (#3975)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Perform a bi-directional check when resolving identity from did (#3977)

* Perform a bi-directional check when resolving identity from did

* tidy

* Reject did documents containing invalid `alsoKnownAs` ATProto handles

* Use error classes

* tidy

* Improve identity resolution

* tidy

* Allow non-normalized handles in did document

* pnpm-lock

* Version packages (#3979)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* repo: MST should allow tilde in keys (#3981)

* repo: MST should allow tilde in keys

* add changeset

* fic ci

* tidy

* tidy

---------

Co-authored-by: rafael <rafael@blueskyweb.xyz>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: James Futhey <kidGodzilla@users.noreply.github.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Samuel Newman <mozzius@protonmail.com>
Co-authored-by: leopardracer <136604165+leopardracer@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2025-06-23 17:31:02 +02:00
Foysal Ahamed
498f3c0738
🐛 Fetch record from pds if appview fails to find it for ozone (#3892)
* 🐛 Fetch record from pds if appview fails to find it for ozone

*  Resolve and etch from pds without auth

* ♻️ Refactor and cleanup

*  Fix tests

*  Fix tests

* 🚨 Fix linter issue

* 🧹 Cleanup
2025-05-28 21:39:28 +02:00
Samuel Newman
79a75bb1ed
Referred likes lexicon changes (#3882) 2025-05-26 19:28:59 +03:00
Foysal Ahamed
45354c84f8
❇️ Verifications manager in ozone (#3765)
* First vouch implementation

* Remove unneeded endpoints

* wip

*  wip

*  Process jetstream events through p-queue and add tests

*  Add test for cursor update

* 🐛 Use utc time to update updatedAt

* 🧹 Cleanup

* 🔨 Fix pnpm versioning issues

*  Replace jetstream lib with manual implementation

*  Remove unnecessary 3p dep

*  Add e2e test for jetstream

* 🚨 Fix import

* 🧹 Remove unnecessary property

*  Fix dev-env and add profile to verification view in ozone

*  Add profile type

*  Add backpressure handling to jetstream listener

*  Use WebSocketKeepAlive from xrpc-server and replace partysocket

*  Add a new verifier role to ozone team meber roles

* 📝 Run codegen

* 🐛 Fix auth check

* 🐛 Fix test failure check

* 🚨 Fix json formatting

* 🐛 Fix team role check

* 🚧 Checking failing test

*  Fix tests

*  Address review comments

*  Add xrpc-server to version

* 🚨 Fix linter issue

* 🚨 Fix linter issue

*  Resolve race condition in cursor update

*  Add verification check on profile

* 🐛 Fix missing cid in test and firehose cursor

*  Fix test

*  Add record validation for verification and separate xrpc-server version

*  Return error object for failed revocations

*  Add re-login on expired session case

* 📝 Fix typo

---------

Co-authored-by: rafael <rafael@blueskyweb.xyz>
2025-04-24 10:49:45 +02:00
Matthieu Sieben
371e04aad2
Account management page (#3659)
---------

Co-authored-by: Eric Bailey <git@esb.lol>
2025-04-15 17:15:27 +02:00
Daniel Holmgren
bdbd3c3e3f
Sequencer recovery (#2519)
* wip

* flesh out recoverer

* basic tests + handle uncreated repos

* add key recovery when keys are lost

* schemas

* tidy recoverer

* tidy & comment test

* work into script framework

* use account evt instead of tombstone

* add flag for updating keys

* add log

* rotate keys script

* pr feedback

* build branch

* use exact commit data from sequencer to restore

* fixing up key rotation script

* add onIdle

* build branch

* track blobs

* fix types

* fix blob tracking

* patches

* publish identity script

* fix build err

* wip

* build err

* wip

* recovery db + fix some errors

* refactor & add repair script

* dont run script

* fix test

* tidy scripts

* tidy

* change publish identity recoveyr method to rotate keys

* tidy

* pr feedback

* simple readme

* changesets
2025-03-31 17:02:11 -05:00
devin ivy
eab9c003f8
Sync event handling (#3612)
* pds: add sync event to account creation

* changeset

* fix bsky subscription handler

* add sync events to @atproto/sync package

* more sync package fixup

* fix sequencer test

* fix sync tests

* clarify firehose event type

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2025-03-07 15:35:57 -06:00
Matthieu Sieben
850e39843c
OAuth: Reset password & Sign-up (#2945)
* Adds "password reset" during OAuth flows
* Adds "Sign up" during OAuth flows
* Adds support for multiple languages in the OAuth flow
* Adds "fr" translation for the OAuth flow

Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: Eric Bailey <git@esb.lol>
2025-03-07 09:41:06 +01:00
Daniel Holmgren
38320191e5
Wrap sync semantics (#3585)
* deprecate blobs & tooBig

* add sync event, deprecate handle & tombstone

* fix up tests

* small tidy

* add test for sync account on account activation

* use new sync event in another place

* remove deprecated events from lexicons

* formatting

* pr cleanup

* changeset
2025-02-28 17:19:18 -06:00
Daniel Holmgren
7e3678c089
Send prevs on firehose (#3449)
* schema

* reset rate limit codegen

* codegen

* send prev cids on firehose

* fix test

* fix some test compiler errors & add experimental note

* fix linting

* build branch

* add prevData to commit event

* fix cbor undefined err

* add sibling proofs to relevant blocks

* bump depth of obj in test

* fix bug on right sibling proof & add some tests

* another test

* refactor proof construction

* more tests

* factor into fixtures

* fix styles in json

* lint: import ordering

* pr feedback

* add invertible op test

* remove prev from outgoing events

* return to original proof construction

* dont build branch

* changeset
2025-02-21 15:01:08 -06:00
bnewbold
01874c4be7
lexicons: use 'tid' and 'record-key' formats in sync+repo lexicons (#2378)
* lex: use 'tid' and 'record-key' formats in lexicons

* more TID formats; and remove redundant maxLength

* add maxLength back, with a comment/description

* revert accidential change to applyWrites

* codegen

* changeset

* update test for invalid record-key
2025-02-20 11:50:24 -08:00
Matthieu Sieben
c53d943c8b
Improve codegen typings (#2999)
* Make codegen types stricter
* Add .js file extension to import statements in generated code
* Fixes a bug that would clear interests prefs when updating hidden posts prefs.
2025-02-13 15:21:00 +01:00
Foysal Ahamed
709a85b0b6
Add a priority score value for ozone subjects (#3495)
*  Add a priority score value for ozone subjects

* 🚨 Bring back lock file

*  Update snapshot for pds proxied tests

*  Update takedown appeal snapshot

*  Update takedown appeal snapshot

*  Update repo and record snapshots

* :rotating_lights: Fix lint issue

*  Update moderation events snapshot

*  Update moderation statuses snapshot

* 🧹 Cleanup unnecessary durationInHours field

*  Allow filtering by priority score

*  Add test for priorityScore filter

* 🚨 Fix import order

*  Allow 0 priority score filter

* 📝 Add changeset
2025-02-06 20:54:27 +00:00
Matthieu Sieben
61dc0d60e1
Add linting rule to sort imports (#3220)
* Add linting rule to sort imports

* remove spacing between import groups

* changeset

* changeset

* prettier config fine tuning

* forbid use of deprecated imports

* tidy
2025-02-05 15:06:58 +01:00
Matthieu Sieben
53a577fd4b
PDS: ActorStore Refactor (#3492)
* Similification of the `pds` package by:
- Removing `DetailedAccountStore` class (moving its logic inside `AccountManager`)
- Factorizes image URL building into its own class (for easy re-use from `AccountManager`)
- Adds an `AppView` class that exposes an `agent: AtpAgent` and url builder function (used by the `ImageUrlBuilder`).
- Reworks the `ActorStore` to avoid circular dependency between `AccountManager` and `LocalViewerCreator` (needed because of first item)

* tidy

* move classes in their own file
2025-02-05 13:37:16 +01:00
Matthieu Sieben
fb64d50ee2
Explicitly allow "undefined" values in headers (#3456)
* Explicitly allow "undefined" values in `headers`

* fix tests types
2025-01-26 13:24:27 -08:00
Matthieu Sieben
8810885b8e
fix pds tests flakiness (#3463) 2025-01-26 13:24:03 -08:00
Matthieu Sieben
ee9779d074
Add account details to queryStatuses (#3236)
* Moderation subjectStatusView with statistics

* add account stats to queue entries

* fix

* adapt

* review changes

* style

* review comments

* wip

* add indexes

* refresh materialized view

* use the background queue to refresh materialized views

* tidy

* fix verrify

* tidy

* Add filtering based on "minAccountSuspendCount"

* fix tests

* Update test snapshots

* update materialized views when running `processAll`

* tidy

* processAll sequentially

* tidy

* tidy

* fix dead lock
2025-01-20 15:36:02 +01:00
Daniel Holmgren
3f93d8cabf
Fix flaky appeal test (#3369)
fix flaky appeal test
2025-01-14 13:02:06 -06:00
Foysal Ahamed
9dc7251fc7
Update snapshot check for tags array (#3340) 2025-01-08 17:24:31 +01:00
Matthieu Sieben
72eba67af1
Drop axios dependency (#3177)
* Minor adaptation of VerifyCidTransform implementation

* refactor: factorize content-encoding negotiation into new lib

* bsky: Use undici to stream blob

* fixup! bsky: Use undici to stream blob

* disable ssrf bsky protection in dev-env

* remove http requests to self to host "/img/"

* drop axios from tests

* fixes

* fix tests

* reviex changes

* properly handle HEAD requests

* handle client disconnection

* fix tests

* drop unrelated change

* tidy

* tidy

* tidy

* remove axios from dev-env

* remove axios from identity package

* use undici 6

* remove axios dependency from ozone

* tidy

* remove axios from PDS package

* avoid killing bsky-pds connections

* improve debugging data

* Better handle invalid CID

* tidy

* tidy

* refactor "allFulfilled" util in @atproto/common

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2025-01-06 18:34:11 +01:00
Foysal Ahamed
6d308b857b
Allow appeals on takendown account (#3251)
*  Allow appeals on takendown account

*  Update snapshot

*  Remove duplicate test

*  Respond with takendown token from createSession for takendown accounts

* 🧹 cleanup appeal account action stuff

* 📝 Add description to new field

* ♻️ Refactor authscope formatter and add test for create record with takendown token

*  Update snapshot

* add createReport route

* changeset

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2024-12-20 13:52:20 -06:00
rafael
638f5a8312
Fix avatar path resolution in dev-env (#3266)
* Fix avatar path resolution in dev-env

* changeset

* extract dev-env assets to dedicated folder

* add comment

* fix fmt
2024-12-19 11:25:41 -03:00
Foysal Ahamed
c559e7e3de
Add subject tags with report type (#3260)
*  Add subject tags with report type

*  Update snapshots

*  Update snpshot for pds admin

*  Update ozone snapshot

*  Update ozone snapshot

* 🧹 Cleanup
2024-12-18 22:57:27 +00:00
Foysal Ahamed
d6f33b4742
Add events for account and record update/delete/deactivation (#2661)
*  Add events for account and record update/delete/deactivation

*  Add handle change event

*  Reduce account events to 2 types and record events to 1

*  Store metadata from account, identity and record events

*  Add created event for record

*  Add ndd the new events to allowed types in emitEvent

*  Use string value for record op and add tombstone flag to identity event

*  Add active flag on account events

*  Change accountStatus -> status to match with firehose event

*  Make active flag required

* 🚨 fix prettier style issue

*  Track record/account delete and update data in subject status (#2804)

*  Store deleted/updated event data in subject_status

* 🐛 Fix query for recordDeletedAt and recordUpdatedAt

*  Add tombstoned status

*  Move from record to hosting term

*  Add tests for hosting params

*  Update lexicons for hostingStatuses

*  Update snapshots

*  Update snapshots

*  Update snapshots

*  Adjust hosting statuses

* 📝 Add changeset
2024-11-07 22:51:44 +01:00
dan
9e18ab6a35
Fix optimistic getPostThread for URIs with handle (#2946)
* Harden test coverge (add failing test)

* Resolve handle for post thread request

* Changeset
2024-11-04 18:28:39 -06:00
Matthieu Sieben
9ffeb5216a
Allow retrying proxied requests (#2850)
* Allow retrying proxied requests

* fix tests

* remove un-necessary code

* Only retry on socket errors
2024-10-31 00:21:05 -04:00
Ilya Siamionau
de194398ed
Fix root block missing in too big seq commit (#2894)
* Fix root block missing in too big seq commit

* Update packages/pds/src/sequencer/events.ts

* fix indentation

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
2024-10-29 17:56:22 -05:00
Matthieu Sieben
7f26b17652
Add OAuth tests (#2874)
* Improve error message when using invalid client_id during code exchange

* Extract SPA example OAuth client in own package

* wip

* remove dependency on get-port

* Properly configure jest to only transpile "get-port" from node_modules

https://jestjs.io/docs/configuration#transformignorepatterns-arraystring

* Use dynamically assigned port number during tests

* use puppeteer to run tests

* remove login input "id" attribute

* code style

* add missing declaration

* tidy

* headless

* remove get-port dependency

* fix tests/proxied/admin.test.ts

* fix tests

* Allow unsecure oauth providers through configuration

* transpile "lande" during ozone tests

* Cache Puppeteer browser binaries

* Use puppeteer cache during all workflow steps

* remove use of set-output

* use get-port in xrpc-server tests

* Renamed to allowHttp

* tidy

* tidy
2024-10-18 15:40:05 +02:00
Matthieu Sieben
4098d98901
Default to unencoded responses (#2834)
* Allow defaulting to unencoded responses when proxying client requests that do not specify accept-encoding
* fix content-encoding negotiation
2024-10-01 10:43:15 +02:00
Matthieu Sieben
b298bfd280
Prevent PDS crash when catchall proxy request are cancelled (#2824)
* Prevent PDS crash when catchall proxy request are cancelled

* fixes

* ensure stream destruction
2024-09-24 11:55:29 -05:00
Matthieu Sieben
a07b21151f
PDS pipethrough optimizations (#2770)
* Micro optimization in request proxying

* Request NSID parsing optimization

* DID document parsing optimization

* remove un-necessary call to next()

* Allow HandlerPipeThrough to be used with streams

* Refactor pipethrough to work with streams

* Expose "unicastLookup" DNS lookup and "isUnicastIp" utilities

* Use a hardened, HTTP2 compatible, client to perform proxied requests

* changeset

* tidy

* Properly handle compressed streams

* tidy

* update @types/node

* refactor

* Improved error management

* Expose parseContentEncoding() util

* use pipeline from nodejs

* Avoid decoding in read-after-write (if possible)

* Various fixes

* Return Buffer instance from streamToBytes

* fixes

* Add omit() utility

* tidy

* lint

* typo

* Use Buffer instead of ArrayBuffer form pipe through handler result

* optimization

* tidy

* refactor

* increase highWaterMark

* remove un-necessary type check

* Use undici.request where more relevant

* Improve soc in fetch utils

* feedback

* fidy

* tidy

* test refactor

* safer fetch

* changeset

* expose and re-use extractUrl util

* small optimizations

* tidy

* optimization

* build branch

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2024-09-19 18:24:20 -05:00
Eric Bailey
922b94ce37
Update email templates (#2767)
* Update email templates

* Update PLC

* Update test with new email string

* Format

* One more test update

* Use handle instead of identifier to match entryway

* Changeset
2024-09-12 10:53:31 -05:00
Daniel Holmgren
b15dec2f4f
Atproto sync package (#2752)
* first pass/port

* reworking

* authenticated commit parsing

* authenticate identity evts

* some testing

* tidy & add firehose to queue

* error handling

* fix test

* refactor sync queue + some tests

* fix race in sync queue

* rm firehose from syncqueue

* add tests for queue utils

* README

* lint readme

* filter before parsing

* pr feedback

* small fix

* changesets

* fix type

* Rework dataplane subscription (#2766)

* working sync package into appview subscription

* add restart method to subscription for tests

* fix another test

* tidy subscription utils/files

* remove dupe property

* tidy after merge

* fix start cursor on subscription

* tweak process full subscription logic

* fixes
2024-09-04 20:18:16 -05:00
devin ivy
befebc0e98
Revise repo write validation for unknown Lexicons (#2748)
* lexicon: validation status result from repo writes

* pds: return  validation status from repo writes, write results from applyWrites

* tidy
2024-08-30 12:59:55 -05:00
Matthieu Sieben
ebb318325b
Improved control over JWT's typ claim (#2743)
* Add "jwtAlg" option to verifySignature() function

* Verify service JWT header values. Add iat claim to service JWT

* Allow missing 'typ' claim in service auth jwt

* Add, and verify, a "typ" header to access and refresh tokens

* tidy

* Properly identify JWT typ missmatch

* tidy

* exclude known invalid "typ" from service auth headers

* tidy

* tidy changeset

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-27 13:50:14 -04:00
Matthieu Sieben
dee817b6e0
OAuth: Add authorization scopes & remove OpenID compatibility (#2734)
* Re-use code definition of oauthResponseTypeSchema

* Generate proper invalid_authorization_details

* Remove OpenID compatibility

* tidy

* properly verify presence of jti claim in client assertion

* Remove non-standard "sub" from OAuthTokenResponse

* Remove nonce from authorization request

* tidy

* Enforce uniqueness of code_challenge

* remove unused "atproto" scope

* Improve reporting of validation errors

* Allow empty set of scopes

* Do not remove scopes not advertised in the AS's "scopes_supported" when building the authorization request.

* Prevent empty scope string

* Remove invalid check from token response

* remove un-necessary session refresh

* Validate scopes characters according to OAuth 2.1 spec

* Mandate the use of "atproto" scope

* Disable ability to list app passwords when using an app password

* Use locally defined authPassthru in com.atproto.admin.* handlers

* provide proper production handle resolver in example

* properly compote login method

* feat(oauth-provider): always rotate session cookie on sign-in

* feat(oauth-provider): do not require consent from first party apps

* update request parameter's prompt before other param validation checks

* feat(oauth-provider): rework display of client name

* feat(oauth-client-browser:example): add token info introspection

* feat(oauth-client-browser:example): allow defining scope globally

* Display requested scopes during the auth flow

* Add, and verify, a "typ" header to access and refresh tokens

* Ignore case when checking for dpop auth scheme

* Add "jwtAlg" option to verifySignature() function

* Verify service JWT header values. Add iat claim to service JWT

* Add support for "transition:generic" and "transition:chat.bsky" oauth scopes in PDS

* oauth-client-browser(example): add scope request

* Add missing "atproto" scope

* Allow missing 'typ' claim in service auth jwt

* Improved 401 feedback

Co-authored-by: devin ivy <devinivy@gmail.com>

* Properly parse scopes upon verification

Co-authored-by: devin ivy <devinivy@gmail.com>

* Rename "atp" to "credential" auth in oauth-client-browser example

* add key to iteration items

* Make CORS protection stronger

* Allow OAuthProvider to define its own CORS policies

* Revert "Allow missing 'typ' claim in service auth jwt"

This reverts commit 15c6b9e2197064eb5de61a96de6497060edb824e.

* Revert "Verify service JWT header values. Add iat claim to service JWT"

This reverts commit 08df8df322a3f4b631c4a63a61d55b2c84c60c11.

* Revert "Add "jwtAlg" option to verifySignature() function"

This reverts commit d0f77354e6904678e7f5d76bb026f07537443ba9.

* Revert "Add, and verify, a "typ" header to access and refresh tokens"

This reverts commit 3e21be9e4b5875caa5e862c11f2196786fb2366d.

* pds: implement protected service auth methods

* Prevent app password management using sessions initiated from an app password.

* Alphabetically sort PROTECTED_METHODS

* Revert changes to app password management permissions

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-27 13:43:29 -04:00
Hailey
4ab2483547
Add quoteCount to embed view (#2735)
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-22 09:49:06 -07:00
Eric Bailey
aba664fbdf
Detached QPs and hidden replies (#2675)
* Add new postgate lex, hiddeReplies to threadgate, codegen

* Add protobufs

* Add to mock dataplane

* Add matching postgate method to feed hydration methods

* Add to getRecord

* Add to HydrationState

* Fix typo

* Add to mergeStates, fetch embeds in threads

* Integrate into embed views

* Add test for QPs in threads

* Add feed test

* Fix naming convention in protos

* Add #viewRemoved record view, rename postgate.json

* Integrate new view

* Filter hidden replies from feeds

* Filter out replies at the handler level, do not filter for author feeds

* Fix lint

* Move hidden reply check to view layer

* Reduce, reuse, recycle

* Rename to lowercase

* Rename layer vars

* Add quote gate props to postgate (#2693)

* Add quote gate props to postgate

* Consistent naming

* Fix record structure

* Codegen

* Show hidden replies in author feed

* Allow reposts of hidden replies

* Lex and codegen

* Add violates_quote_gate to proto

* Consistent naming, codegen

* Integrate violatesQuotegate and canQuotepost

* Remove rules, codegen

* Hydrate all postgates for all requested posts

* Match other impl

* Add test, need to split these out

* Format

* Hydrate first nested embeds too

* Add postgate test suite

* Add violatesQuoteGate to dataplane

* Ingest and set violatesQuoteGate, return on meta

* Return removed embed for quotes that violate gate

* Add test

* Dedupe URIs before fetching postgates

* Update snaps

* Snap

* Format

* Updating naming conventions for postgate-related attributes

* Correct naming

* Consistency

* Proto too

* Rename to viewDetached

* Codegen

* Rename everything

* Codegen

* Quotes that violate a quote gate can still be quoted themselves

* Couple more renames

* Snaps

* Ensure reply ref is tombstoned for hidden replies

* Split out hidden replies tests and create fresh fixture

* Hydrate threadgates for reply notifications, filter hidden replies

* Remove snap

* Add flaky test

* Rename violatesEmbeddingRules

* Fix flaky test

* Only write to db if violatesEmbeddingRules is true

* DRY up post uri -> gate uri logic

* isThreadgateListRule

* Don't share users object between tests

* No pascal

* Remove default params

* Find -> some

* canQuotepost -> canEmbed, remove optional

* Fix quoteee typo

* await follows

* Throw in post uri -> gate utils

* Ensure fetch threadgates for reply roots

* Don't hydrate threadgates twice

* DRY up uri -> did parsing

* Clean up parsePostgate logic

* Format

* Revert change

* Revert change

* Replace a couple more uri->did conversions

* Only filter replies from feeds if viewer hid them

* Revert, filter out replies that are hidden from feeds

* Remove old test

* Replace uri->did util

* Revert change to unused file

* Only validatePostEmbed and check postgates for post records

* Ensure notifications aren't generated down a hidden reply chain

* Changeset

* Cleanup

* Fix notification filtering logic

* Simplify

* Don't notify for invalid embeds

* Use new APIs

* Add hasPostGate and hasThreadGate flags from dataplane

* Only fetch postgates if post has one

* Only fetch threadgates if post has one or was deleted

* Remove notification filtering

* Don't hydrate threadgates for notifications

* Move hidden replies in feeds to match block handling

* Do no filtering of hidden replies in feeds

* Revert "Don't hydrate threadgates for notifications"

This reverts commit 1dcec0b239a7b9d6800427b26b8ba3e6a54210f9.

* Revert "Remove notification filtering"

This reverts commit 1e7069dfd809d1f18e9f05fd1d422e7399aa1bb0.

* Filter notifications for OP only

* Add additional check to hidden replies test

* Move noty filter logic into method handler

* Update .changeset/perfect-parrots-appear.md

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/bsky/tests/seed/postgates.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Another structuredClone

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Better comment

* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Regen protos to match dataplane

* Update quotes snap to include embeddingDisabled

* Clarify usage of post uri -> gate utils

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-21 14:36:51 -05:00
Hailey
2a0c088cc5
quote aggs, list quotes (#2658)
* add quote count to post_agg, add getPostQuotes

rework schema

rework schema

add getPostQuotes to api

use posts

use posts

codegen

use items instead of quotes

codegen

add getPostQuotes

add quoteCount to response

update lexicon for postview

increment post ags

add quote to post aggs

add quote interface

oops

add quote table migration

* update

* bufgen

* update params

* update to use v2

* logs

* rm comment

* pass cursor

* add index

* Update packages/bsky/src/data-plane/server/db/migrations/20240723T220703655Z-quotes.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* only if its a post

* tests

* Discard changes to packages/bsky/tests/views/posts.test.ts

* fix client call

* Include new quotes agg in test expectation

* Use new API for headers

* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* revert rm

* rm timeout

* cursor test

* Changeset

* Remove pds specific bump

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: Eric Bailey <git@esb.lol>
2024-08-21 12:33:05 -05:00