1610 Commits

Author SHA1 Message Date
Arthur Pinheiro
6bf6192ff2
Fix typo in README.md (#2826) 2024-09-23 16:37:35 +01:00
Matthieu Sieben
a07b21151f
PDS pipethrough optimizations (#2770)
* Micro optimization in request proxying

* Request NSID parsing optimization

* DID document parsing optimization

* remove un-necessary call to next()

* Allow HandlerPipeThrough to be used with streams

* Refactor pipethrough to work with streams

* Expose "unicastLookup" DNS lookup and "isUnicastIp" utilities

* Use a hardened, HTTP2 compatible, client to perform proxied requests

* changeset

* tidy

* Properly handle compressed streams

* tidy

* update @types/node

* refactor

* Improved error management

* Expose parseContentEncoding() util

* use pipeline from nodejs

* Avoid decoding in read-after-write (if possible)

* Various fixes

* Return Buffer instance from streamToBytes

* fixes

* Add omit() utility

* tidy

* lint

* typo

* Use Buffer instead of ArrayBuffer form pipe through handler result

* optimization

* tidy

* refactor

* increase highWaterMark

* remove un-necessary type check

* Use undici.request where more relevant

* Improve soc in fetch utils

* feedback

* fidy

* tidy

* test refactor

* safer fetch

* changeset

* expose and re-use extractUrl util

* small optimizations

* tidy

* optimization

* build branch

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2024-09-19 18:24:20 -05:00
Eric Bailey
319aa7cf6d
Add isFallback to getSuggestedFollowsByActor method (#2805)
* Add isFallback to `getSuggestedFollowsByActor`

Inferred based on returned `relativeToDid` from the suggestions
response.

* Integrate new params

* Fix logic
2024-09-13 12:37:15 -05:00
Eric Bailey
a06634ae57
Email templates at (#2813)
* Add @ to handles, update PLC copy

* Changeset

* lint

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2024-09-12 11:57:41 -05:00
Eric Bailey
922b94ce37
Update email templates (#2767)
* Update email templates

* Update PLC

* Update test with new email string

* Format

* One more test update

* Use handle instead of identifier to match entryway

* Changeset
2024-09-12 10:53:31 -05:00
github-actions[bot]
85c85350d1
Version packages (#2791)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-11 18:30:47 -05:00
Eric Bailey
33aa0c7222
NUX API (#2810)
* Codegen

* Explicitly add Zod (already a peer dep) and validation to api

* Add Nux methods

* Match naming convention

* Remove id, it won't be used

* Add tests

* Use id instead of name, little clearer

* Update API contracts

* Update tests

* Changeset

* Don't mutate
2024-09-11 18:25:05 -05:00
Foysal Ahamed
62a8225f36
Optionally ack all open subjects from the author with takedown event and get all mod subjects of a user (#2793)
*  Add acknowledgeAllSubjectsOfAccount flag with takedown event

* 📝 better documentation of forAccount and subject params

* 🧹 Cleanup tests

*  Change wording according to review

*  Refactor tests

* ♻️ Rename acknowledge flag
2024-09-11 18:26:53 +02:00
Matthieu Sieben
98711a147a
fix(xrpc-server): properly parse & process content-encoding (#2464)
* fix(xrpc-server): properly parse & process content-encoding

* Minor optimization

* code style
2024-09-11 09:46:18 +02:00
Daniel Holmgren
bcefbdbc6b
Slow down get repos (#2802)
* slow down get repos

* move wait
2024-09-08 20:32:55 -05:00
Daniel Holmgren
6c1ec149cf
PDS proxy to appview performance (#2773)
* accept entryway session tokens

* extra check + tests

* build

* build

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-09-06 18:56:37 -05:00
devin ivy
71785d31f7
Ozone: pass through unknown account details (#2794)
* ozone: pass through account info details that may not be known yet

* tidy, build
2024-09-05 21:22:48 -04:00
Eric Bailey
c5b765d043
Remove launched feature gate (#2765) 2024-09-05 10:30:37 -05:00
Matthieu Sieben
cb4abbb673
Properly validate atproto did:web (#2776)
* Properly validate atproto did:web

* explicit why there is no protection against localhost fetches in did:web resolver
2024-09-05 13:01:48 +02:00
github-actions[bot]
0a37a3cc56
Version packages (#2788)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-04 20:22:22 -05:00
Daniel Holmgren
b15dec2f4f
Atproto sync package (#2752)
* first pass/port

* reworking

* authenticated commit parsing

* authenticate identity evts

* some testing

* tidy & add firehose to queue

* error handling

* fix test

* refactor sync queue + some tests

* fix race in sync queue

* rm firehose from syncqueue

* add tests for queue utils

* README

* lint readme

* filter before parsing

* pr feedback

* small fix

* changesets

* fix type

* Rework dataplane subscription (#2766)

* working sync package into appview subscription

* add restart method to subscription for tests

* fix another test

* tidy subscription utils/files

* remove dupe property

* tidy after merge

* fix start cursor on subscription

* tweak process full subscription logic

* fixes
2024-09-04 20:18:16 -05:00
Foysal Ahamed
642c7ae968
Improve query perf on moderation event (#2787)
*  Add index on subjectDid and use subjectDid to speed up query

* 📝 Add changeset
2024-09-04 19:48:16 -04:00
devin ivy
2397067b26
Appview: retry dataplane requests on abort (#2783)
appview: retry dataplane requests on abort
2024-09-04 19:36:22 -04:00
github-actions[bot]
71305e8595
Version packages (#2764)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-04 14:23:00 +02:00
Foysal Ahamed
e4d41d66fa
❇️ Template language (#2780)
*  Throw specific error for duplicate template name

* 🧹 Cleanup console

*  Throw duplicate template name error from update too

*  Add language to templates

* 📝 Add changeset

*  Add missing event type

*  Add language format in lexicon and error checker in util

* 🚨 fix linter issues
2024-09-04 12:42:39 +02:00
Foysal Ahamed
8252c652e8
🐛 Fix appealed param's usage (#2777)
🐛 Depend on false value instead of null since the input is sent through GET request for queryStatuses
2024-09-02 19:10:54 -04:00
Daniel Holmgren
bfbac24312
Repo write commit metadata (#2745)
* schemas & codegen

* impl

* bundle commit & rev
2024-08-30 13:18:43 -05:00
devin ivy
befebc0e98
Revise repo write validation for unknown Lexicons (#2748)
* lexicon: validation status result from repo writes

* pds: return  validation status from repo writes, write results from applyWrites

* tidy
2024-08-30 12:59:55 -05:00
Foysal Ahamed
372ed4c519
Apply embed specific tag on subjects for video, image and external (#2703)
*  Refactor subject tagging to facilitate video content tagging

* ♻️ Refactor tag check

*  Fix tagging logic

* ♻️ Refactor content tagger and fix image content type check

*  Add embed tag check for video and external

*  Add tagging for both media and image embed
2024-08-29 18:23:48 +02:00
github-actions[bot]
6bc7faf087
Version packages (#2761)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-28 19:13:54 -04:00
devin ivy
80ada8f476
Video lexicons and appview views (#2751)
* lexicon: initial lexicons for video embeds in bsky app

* lexicon: fix video caption file size limit

* codegen

* appview: stub out video embed view logic

* api prerelease

* api prerelease

* lexicon: video upload/processing lexicons

* tidy

* lexicon: app.bsky.video lexicons for uploads

* codegen

* api prerelease

* appview: present video embeds on posts

* appview: snaps

* changeset

* appview: fix wiring of video url config
2024-08-28 19:03:35 -04:00
Eric Bailey
f1dad05975
Add test for viewer block (#2759)
* Add test for viewer block

* Use util
2024-08-28 17:28:03 -04:00
Eric Bailey
ea6cfc39d2
Fix viewer block reference (#2758) 2024-08-28 15:45:57 -04:00
github-actions[bot]
e49e05c0fa
Version packages (#2750)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-27 17:35:48 -04:00
devin ivy
c180cf4d86
OAuth: fix client-side crash on authorize page (#2749)
oauth: fix client-side crash on authorize page
2024-08-27 17:31:44 -04:00
github-actions[bot]
a1d8c77edd
Version packages (#2738)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-27 14:03:39 -04:00
Matthieu Sieben
ebb318325b
Improved control over JWT's typ claim (#2743)
* Add "jwtAlg" option to verifySignature() function

* Verify service JWT header values. Add iat claim to service JWT

* Allow missing 'typ' claim in service auth jwt

* Add, and verify, a "typ" header to access and refresh tokens

* tidy

* Properly identify JWT typ missmatch

* tidy

* exclude known invalid "typ" from service auth headers

* tidy

* tidy changeset

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-27 13:50:14 -04:00
Matthieu Sieben
dee817b6e0
OAuth: Add authorization scopes & remove OpenID compatibility (#2734)
* Re-use code definition of oauthResponseTypeSchema

* Generate proper invalid_authorization_details

* Remove OpenID compatibility

* tidy

* properly verify presence of jti claim in client assertion

* Remove non-standard "sub" from OAuthTokenResponse

* Remove nonce from authorization request

* tidy

* Enforce uniqueness of code_challenge

* remove unused "atproto" scope

* Improve reporting of validation errors

* Allow empty set of scopes

* Do not remove scopes not advertised in the AS's "scopes_supported" when building the authorization request.

* Prevent empty scope string

* Remove invalid check from token response

* remove un-necessary session refresh

* Validate scopes characters according to OAuth 2.1 spec

* Mandate the use of "atproto" scope

* Disable ability to list app passwords when using an app password

* Use locally defined authPassthru in com.atproto.admin.* handlers

* provide proper production handle resolver in example

* properly compote login method

* feat(oauth-provider): always rotate session cookie on sign-in

* feat(oauth-provider): do not require consent from first party apps

* update request parameter's prompt before other param validation checks

* feat(oauth-provider): rework display of client name

* feat(oauth-client-browser:example): add token info introspection

* feat(oauth-client-browser:example): allow defining scope globally

* Display requested scopes during the auth flow

* Add, and verify, a "typ" header to access and refresh tokens

* Ignore case when checking for dpop auth scheme

* Add "jwtAlg" option to verifySignature() function

* Verify service JWT header values. Add iat claim to service JWT

* Add support for "transition:generic" and "transition:chat.bsky" oauth scopes in PDS

* oauth-client-browser(example): add scope request

* Add missing "atproto" scope

* Allow missing 'typ' claim in service auth jwt

* Improved 401 feedback

Co-authored-by: devin ivy <devinivy@gmail.com>

* Properly parse scopes upon verification

Co-authored-by: devin ivy <devinivy@gmail.com>

* Rename "atp" to "credential" auth in oauth-client-browser example

* add key to iteration items

* Make CORS protection stronger

* Allow OAuthProvider to define its own CORS policies

* Revert "Allow missing 'typ' claim in service auth jwt"

This reverts commit 15c6b9e2197064eb5de61a96de6497060edb824e.

* Revert "Verify service JWT header values. Add iat claim to service JWT"

This reverts commit 08df8df322a3f4b631c4a63a61d55b2c84c60c11.

* Revert "Add "jwtAlg" option to verifySignature() function"

This reverts commit d0f77354e6904678e7f5d76bb026f07537443ba9.

* Revert "Add, and verify, a "typ" header to access and refresh tokens"

This reverts commit 3e21be9e4b5875caa5e862c11f2196786fb2366d.

* pds: implement protected service auth methods

* Prevent app password management using sessions initiated from an app password.

* Alphabetically sort PROTECTED_METHODS

* Revert changes to app password management permissions

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-27 13:43:29 -04:00
Matthieu Sieben
bbca17bc53 Deprecate Agent.accountDid in favor of Agent.assertDid 2024-08-26 09:02:10 +02:00
devin ivy
ec2e426e32
Appview: fix uri used to place threadgate in getPostThread response (#2741)
appview: fix uri used to place threadgate in getPostThread response
2024-08-23 15:05:54 -04:00
Hailey
5af9b8287b
filter posts with an embed block (#2740)
* filter posts with an embed block

* log
2024-08-23 11:18:21 -04:00
Eric Bailey
a8e1f9000d
Return ThreadgateView on response from getPostThread (#2737)
* Return `ThreadgateView` on response from `getPostThread`

* Changeset

* Format

* Add to test

* Clean up logic

* Use suggestion from Dan
2024-08-22 17:00:01 -05:00
Matthieu Sieben
d9ffa3c460
Instantiate XrpcClient from an OAuthAgent (#2714)
* Improve transformation of fetchHandler errors into XrpcError

* Add ability to instantiate XrpcClient from FetchHandlerObject type

* Remove un-necessary dev dependency

* Allow oauthAgent to be used in order to instantiate XrpcClient

* fix lock file

* Move OAuthAtpAgent  to api package

* correct doc

* docs(oauth-client): improve example

* fix example code

* Rename OAuthAgent into OAuthSession

* Allow instantiating Agent and XrpcClient with OAuthSession

* Fix changesets

* codegen

* tidy

* tidy

* tidy

* Update .changeset/chilled-jokes-relax.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update packages/oauth/oauth-client/README.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update packages/api/OAUTH.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update .changeset/old-mice-give.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update packages/api/OAUTH.md

* Update packages/api/README.md

* Update packages/api/README.md

* Update .changeset/polite-toys-happen.md

---------

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-22 17:59:22 -04:00
github-actions[bot]
f70bd6a9dc
Version packages (#2736)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-22 09:54:41 -07:00
Hailey
4ab2483547
Add quoteCount to embed view (#2735)
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-22 09:49:06 -07:00
Matthieu Sieben
5fef534610 codegen 2024-08-22 11:55:36 +02:00
devin ivy
45770bb165
Appview: sync-up protos for quotes and postgates (#2733)
appview: sync-up protos for quotes and postgates
2024-08-21 20:38:54 -04:00
Hailey
47263e9f3d
filter blocks in curate list (#2720) 2024-08-21 14:55:30 -07:00
github-actions[bot]
1572058887
Version packages (#2732)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-21 14:42:01 -05:00
Eric Bailey
aba664fbdf
Detached QPs and hidden replies (#2675)
* Add new postgate lex, hiddeReplies to threadgate, codegen

* Add protobufs

* Add to mock dataplane

* Add matching postgate method to feed hydration methods

* Add to getRecord

* Add to HydrationState

* Fix typo

* Add to mergeStates, fetch embeds in threads

* Integrate into embed views

* Add test for QPs in threads

* Add feed test

* Fix naming convention in protos

* Add #viewRemoved record view, rename postgate.json

* Integrate new view

* Filter hidden replies from feeds

* Filter out replies at the handler level, do not filter for author feeds

* Fix lint

* Move hidden reply check to view layer

* Reduce, reuse, recycle

* Rename to lowercase

* Rename layer vars

* Add quote gate props to postgate (#2693)

* Add quote gate props to postgate

* Consistent naming

* Fix record structure

* Codegen

* Show hidden replies in author feed

* Allow reposts of hidden replies

* Lex and codegen

* Add violates_quote_gate to proto

* Consistent naming, codegen

* Integrate violatesQuotegate and canQuotepost

* Remove rules, codegen

* Hydrate all postgates for all requested posts

* Match other impl

* Add test, need to split these out

* Format

* Hydrate first nested embeds too

* Add postgate test suite

* Add violatesQuoteGate to dataplane

* Ingest and set violatesQuoteGate, return on meta

* Return removed embed for quotes that violate gate

* Add test

* Dedupe URIs before fetching postgates

* Update snaps

* Snap

* Format

* Updating naming conventions for postgate-related attributes

* Correct naming

* Consistency

* Proto too

* Rename to viewDetached

* Codegen

* Rename everything

* Codegen

* Quotes that violate a quote gate can still be quoted themselves

* Couple more renames

* Snaps

* Ensure reply ref is tombstoned for hidden replies

* Split out hidden replies tests and create fresh fixture

* Hydrate threadgates for reply notifications, filter hidden replies

* Remove snap

* Add flaky test

* Rename violatesEmbeddingRules

* Fix flaky test

* Only write to db if violatesEmbeddingRules is true

* DRY up post uri -> gate uri logic

* isThreadgateListRule

* Don't share users object between tests

* No pascal

* Remove default params

* Find -> some

* canQuotepost -> canEmbed, remove optional

* Fix quoteee typo

* await follows

* Throw in post uri -> gate utils

* Ensure fetch threadgates for reply roots

* Don't hydrate threadgates twice

* DRY up uri -> did parsing

* Clean up parsePostgate logic

* Format

* Revert change

* Revert change

* Replace a couple more uri->did conversions

* Only filter replies from feeds if viewer hid them

* Revert, filter out replies that are hidden from feeds

* Remove old test

* Replace uri->did util

* Revert change to unused file

* Only validatePostEmbed and check postgates for post records

* Ensure notifications aren't generated down a hidden reply chain

* Changeset

* Cleanup

* Fix notification filtering logic

* Simplify

* Don't notify for invalid embeds

* Use new APIs

* Add hasPostGate and hasThreadGate flags from dataplane

* Only fetch postgates if post has one

* Only fetch threadgates if post has one or was deleted

* Remove notification filtering

* Don't hydrate threadgates for notifications

* Move hidden replies in feeds to match block handling

* Do no filtering of hidden replies in feeds

* Revert "Don't hydrate threadgates for notifications"

This reverts commit 1dcec0b239a7b9d6800427b26b8ba3e6a54210f9.

* Revert "Remove notification filtering"

This reverts commit 1e7069dfd809d1f18e9f05fd1d422e7399aa1bb0.

* Filter notifications for OP only

* Add additional check to hidden replies test

* Move noty filter logic into method handler

* Update .changeset/perfect-parrots-appear.md

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/bsky/tests/seed/postgates.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Another structuredClone

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Better comment

* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Regen protos to match dataplane

* Update quotes snap to include embeddingDisabled

* Clarify usage of post uri -> gate utils

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-21 14:36:51 -05:00
Hailey
2a0c088cc5
quote aggs, list quotes (#2658)
* add quote count to post_agg, add getPostQuotes

rework schema

rework schema

add getPostQuotes to api

use posts

use posts

codegen

use items instead of quotes

codegen

add getPostQuotes

add quoteCount to response

update lexicon for postview

increment post ags

add quote to post aggs

add quote interface

oops

add quote table migration

* update

* bufgen

* update params

* update to use v2

* logs

* rm comment

* pass cursor

* add index

* Update packages/bsky/src/data-plane/server/db/migrations/20240723T220703655Z-quotes.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* only if its a post

* tests

* Discard changes to packages/bsky/tests/views/posts.test.ts

* fix client call

* Include new quotes agg in test expectation

* Use new API for headers

* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* revert rm

* rm timeout

* cursor test

* Changeset

* Remove pds specific bump

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: Eric Bailey <git@esb.lol>
2024-08-21 12:33:05 -05:00
Eric Bailey
922b2e82ac
Compute 3p blocks for ReplyRef.root (#2721)
* Add grandparent to reply ref

* Hydrate grandparent on reply ref

* Changeset

* Check for 3p blocks on ReplyRef.root

* Revert "Add grandparent to reply ref"

This reverts commit 73012b67262eced0f626c2f05592aaea655d6f79.

* Revert "Changeset"

This reverts commit 4264b085ae7c4cb9bd833528f354973be1a05305.

* Remove redundant hydration

* More tests

* Check for 3p blocks between grandparent and root

* Just kidding, ensure we only use first slice to compute from

* Clarify block relationship

* Add anti-test

* Rename test file to be more precise

* Make sure we get child->root block relationships

* Rename postBlocks.reply to parent

* Ensure first slice relationships are still correct if replyRef data is outside first page of results

* Remove unnecessary check

* Format

* Use latest APIs

* Prevent [creator, creator] checks
2024-08-20 14:00:07 -05:00
github-actions[bot]
5e2f2617ab
Version packages (#2726)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-20 11:41:33 -04:00
Matthieu Sieben
35a1264297
Remove non-standard *_endpoint_auth_method (#2729) 2024-08-20 17:26:19 +02:00
Matthieu Sieben
5131b027f0
Allow charset in content-type header of incoming requests (#2728)
* Allow charset in content-type header of incoming requests
2024-08-20 11:24:56 +02:00