63 Commits

Author SHA1 Message Date
Matthieu Sieben
2889c76995
Improve type safety and compatibility with Bun (#2879)
* jwk: Improve type safety and compatibility with Bun
* improve type safety of jwk keys
* improve typing of verifyAccessToken
* update @types/http-errors
* Better report invalid content-encoding errors
* Mark jwk key fields as readonly
2025-01-09 14:26:07 +01:00
Daniel Holmgren
1abfd74ec7
Update crypto libraries (#3335)
* update crypto libs & use new format option

* reinstall deps

* changeset
2025-01-07 11:59:21 -06:00
Matthieu Sieben
72eba67af1
Drop axios dependency (#3177)
* Minor adaptation of VerifyCidTransform implementation

* refactor: factorize content-encoding negotiation into new lib

* bsky: Use undici to stream blob

* fixup! bsky: Use undici to stream blob

* disable ssrf bsky protection in dev-env

* remove http requests to self to host "/img/"

* drop axios from tests

* fixes

* fix tests

* reviex changes

* properly handle HEAD requests

* handle client disconnection

* fix tests

* drop unrelated change

* tidy

* tidy

* tidy

* remove axios from dev-env

* remove axios from identity package

* use undici 6

* remove axios dependency from ozone

* tidy

* remove axios from PDS package

* avoid killing bsky-pds connections

* improve debugging data

* Better handle invalid CID

* tidy

* tidy

* refactor "allFulfilled" util in @atproto/common

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2025-01-06 18:34:11 +01:00
Daniel Holmgren
0bec389a1c
Sync: pass through websocket opts (#2992)
* pass through websocket options in firehose

* changeset

* import type
2024-12-05 19:01:23 -06:00
Matthieu Sieben
1e367cba2b
Update email validation logic (#3092) 2024-11-25 08:52:11 +01:00
Pyrox
124eaee8f2
bsky,pds: sharp 0.32.6 -> 0.33.5 (#2958) 2024-11-07 19:16:33 -05:00
Matthieu Sieben
9d40ccbb69
Various OAuth related fixes (#2871)
* wip

* tidy

* tidy

* tidy

* Update packages/oauth/oauth-client/src/session-getter.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix combineSignals

* tidy

* tidy

* improve typing of atprotoScopeSchema

* stronger typings

* tidy

* ci

* Fix cors error

* downgrade ioredis dependency

* fix ioredis version

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-10-18 20:23:33 +02:00
Matthieu Sieben
7f26b17652
Add OAuth tests (#2874)
* Improve error message when using invalid client_id during code exchange

* Extract SPA example OAuth client in own package

* wip

* remove dependency on get-port

* Properly configure jest to only transpile "get-port" from node_modules

https://jestjs.io/docs/configuration#transformignorepatterns-arraystring

* Use dynamically assigned port number during tests

* use puppeteer to run tests

* remove login input "id" attribute

* code style

* add missing declaration

* tidy

* headless

* remove get-port dependency

* fix tests/proxied/admin.test.ts

* fix tests

* Allow unsecure oauth providers through configuration

* transpile "lande" during ozone tests

* Cache Puppeteer browser binaries

* Use puppeteer cache during all workflow steps

* remove use of set-output

* use get-port in xrpc-server tests

* Renamed to allowHttp

* tidy

* tidy
2024-10-18 15:40:05 +02:00
Matthieu Sieben
fabc8a9381
Update typescript to version 5.6.2 (#2863) 2024-10-11 14:05:53 +02:00
Hailey
95cb897751
Send mark-read-generic notification on updateSeen (#2567)
* send `mark-read-generic` notification on `updateSeen`

add `reason`

add `recipientDid`

push `mark-read-generic` notification on `updateSeen`

add `client_controlled`

* unique id, change `alwaysDeliver` to false

* use murmur id

* organize import

* bsky: fix tests, making courier config optional. fix unread count query.

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-10-04 14:35:37 -04:00
Matthieu Sieben
08ed0a5a91
Fix build (#2862)
add missing ts dependency
2024-10-04 11:40:50 +02:00
Foysal Ahamed
a2bad977a8
Ozone batch repo and record getters (#2836)
*  Add getRepos and getRecords endpoints for bulk fetching

*  Fix issues and add tests for get repos and get records

*  Use the right lxm

* 🐛 Revert changes in lockfile

*  Add getAccountInfos in PDS

* 🐛 Fix type def for repo and record view detail

*  Update snapshots

*  Update snapshots

*  Consolidate error type for com.atproto and tools.ozone getRecord error type

* 🧹 Cleanup

*  Update snapshots

*  Update snapshots

*  Changeset
2024-10-01 19:37:23 +02:00
Matthieu Sieben
a07b21151f
PDS pipethrough optimizations (#2770)
* Micro optimization in request proxying

* Request NSID parsing optimization

* DID document parsing optimization

* remove un-necessary call to next()

* Allow HandlerPipeThrough to be used with streams

* Refactor pipethrough to work with streams

* Expose "unicastLookup" DNS lookup and "isUnicastIp" utilities

* Use a hardened, HTTP2 compatible, client to perform proxied requests

* changeset

* tidy

* Properly handle compressed streams

* tidy

* update @types/node

* refactor

* Improved error management

* Expose parseContentEncoding() util

* use pipeline from nodejs

* Avoid decoding in read-after-write (if possible)

* Various fixes

* Return Buffer instance from streamToBytes

* fixes

* Add omit() utility

* tidy

* lint

* typo

* Use Buffer instead of ArrayBuffer form pipe through handler result

* optimization

* tidy

* refactor

* increase highWaterMark

* remove un-necessary type check

* Use undici.request where more relevant

* Improve soc in fetch utils

* feedback

* fidy

* tidy

* test refactor

* safer fetch

* changeset

* expose and re-use extractUrl util

* small optimizations

* tidy

* optimization

* build branch

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2024-09-19 18:24:20 -05:00
Eric Bailey
33aa0c7222
NUX API (#2810)
* Codegen

* Explicitly add Zod (already a peer dep) and validation to api

* Add Nux methods

* Match naming convention

* Remove id, it won't be used

* Add tests

* Use id instead of name, little clearer

* Update API contracts

* Update tests

* Changeset

* Don't mutate
2024-09-11 18:25:05 -05:00
Daniel Holmgren
6c1ec149cf
PDS proxy to appview performance (#2773)
* accept entryway session tokens

* extra check + tests

* build

* build

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-09-06 18:56:37 -05:00
Daniel Holmgren
b15dec2f4f
Atproto sync package (#2752)
* first pass/port

* reworking

* authenticated commit parsing

* authenticate identity evts

* some testing

* tidy & add firehose to queue

* error handling

* fix test

* refactor sync queue + some tests

* fix race in sync queue

* rm firehose from syncqueue

* add tests for queue utils

* README

* lint readme

* filter before parsing

* pr feedback

* small fix

* changesets

* fix type

* Rework dataplane subscription (#2766)

* working sync package into appview subscription

* add restart method to subscription for tests

* fix another test

* tidy subscription utils/files

* remove dupe property

* tidy after merge

* fix start cursor on subscription

* tweak process full subscription logic

* fixes
2024-09-04 20:18:16 -05:00
Matthieu Sieben
dee817b6e0
OAuth: Add authorization scopes & remove OpenID compatibility (#2734)
* Re-use code definition of oauthResponseTypeSchema

* Generate proper invalid_authorization_details

* Remove OpenID compatibility

* tidy

* properly verify presence of jti claim in client assertion

* Remove non-standard "sub" from OAuthTokenResponse

* Remove nonce from authorization request

* tidy

* Enforce uniqueness of code_challenge

* remove unused "atproto" scope

* Improve reporting of validation errors

* Allow empty set of scopes

* Do not remove scopes not advertised in the AS's "scopes_supported" when building the authorization request.

* Prevent empty scope string

* Remove invalid check from token response

* remove un-necessary session refresh

* Validate scopes characters according to OAuth 2.1 spec

* Mandate the use of "atproto" scope

* Disable ability to list app passwords when using an app password

* Use locally defined authPassthru in com.atproto.admin.* handlers

* provide proper production handle resolver in example

* properly compote login method

* feat(oauth-provider): always rotate session cookie on sign-in

* feat(oauth-provider): do not require consent from first party apps

* update request parameter's prompt before other param validation checks

* feat(oauth-provider): rework display of client name

* feat(oauth-client-browser:example): add token info introspection

* feat(oauth-client-browser:example): allow defining scope globally

* Display requested scopes during the auth flow

* Add, and verify, a "typ" header to access and refresh tokens

* Ignore case when checking for dpop auth scheme

* Add "jwtAlg" option to verifySignature() function

* Verify service JWT header values. Add iat claim to service JWT

* Add support for "transition:generic" and "transition:chat.bsky" oauth scopes in PDS

* oauth-client-browser(example): add scope request

* Add missing "atproto" scope

* Allow missing 'typ' claim in service auth jwt

* Improved 401 feedback

Co-authored-by: devin ivy <devinivy@gmail.com>

* Properly parse scopes upon verification

Co-authored-by: devin ivy <devinivy@gmail.com>

* Rename "atp" to "credential" auth in oauth-client-browser example

* add key to iteration items

* Make CORS protection stronger

* Allow OAuthProvider to define its own CORS policies

* Revert "Allow missing 'typ' claim in service auth jwt"

This reverts commit 15c6b9e2197064eb5de61a96de6497060edb824e.

* Revert "Verify service JWT header values. Add iat claim to service JWT"

This reverts commit 08df8df322a3f4b631c4a63a61d55b2c84c60c11.

* Revert "Add "jwtAlg" option to verifySignature() function"

This reverts commit d0f77354e6904678e7f5d76bb026f07537443ba9.

* Revert "Add, and verify, a "typ" header to access and refresh tokens"

This reverts commit 3e21be9e4b5875caa5e862c11f2196786fb2366d.

* pds: implement protected service auth methods

* Prevent app password management using sessions initiated from an app password.

* Alphabetically sort PROTECTED_METHODS

* Revert changes to app password management permissions

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-27 13:43:29 -04:00
Matthieu Sieben
d9ffa3c460
Instantiate XrpcClient from an OAuthAgent (#2714)
* Improve transformation of fetchHandler errors into XrpcError

* Add ability to instantiate XrpcClient from FetchHandlerObject type

* Remove un-necessary dev dependency

* Allow oauthAgent to be used in order to instantiate XrpcClient

* fix lock file

* Move OAuthAtpAgent  to api package

* correct doc

* docs(oauth-client): improve example

* fix example code

* Rename OAuthAgent into OAuthSession

* Allow instantiating Agent and XrpcClient with OAuthSession

* Fix changesets

* codegen

* tidy

* tidy

* tidy

* Update .changeset/chilled-jokes-relax.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update packages/oauth/oauth-client/README.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update packages/api/OAUTH.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update .changeset/old-mice-give.md

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>

* Update packages/api/OAUTH.md

* Update packages/api/README.md

* Update packages/api/README.md

* Update .changeset/polite-toys-happen.md

---------

Co-authored-by: surfdude29 <149612116+surfdude29@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-22 17:59:22 -04:00
Matthieu Sieben
5131b027f0
Allow charset in content-type header of incoming requests (#2728)
* Allow charset in content-type header of incoming requests
2024-08-20 11:24:56 +02:00
Matthieu Sieben
b934b396b1
Client SDK rework (#2483)
* feat(api): support creation of oauth based AtpAgents

* oauth: misc fixes for confidential clients

* fix(xprc): remove ReadableStream.from polyfill

* OAuth docs tweaks (#2679)

* OAuth: clarification about client_name being shown

* OAuth: re-write handle resolution privacy concern

* avoid relying on ReadableStream.from in xrpc-server tests

* feat(oauth-types): expose "ALLOW_UNSECURE_ORIGINS" constant

* feat(handle-resolver): expose "AtprotoIdentityDidMethods" type

* fix(oauth-client): ensure that the oauth metadata document contains client_id_metadata_document_supported

* fix(oauth-types): prevent unknown query string in loopback client id

* fix(identity-resolver): check that handle is in did doc's "alsoKnownAs"

* feat(oauth-client:oauth-resolver): allow logging in using either the PDS URL or Entryway URL

* fix(oauth-client): return better error in case of invalid "oauth-protected-resource" status code

* refactor(did): group atproto specific checks in own

* feat(api): relax typing of "appLabelers" and "labelers" AtpClient properties

* allow any did as labeller (for tests mainly)

* fix(api): allow to override "atproto-proxy" on a per-request basis

* remove release candidate versions from changelog

* update changeset for api and xrpc packages

* Add missing changeset

* revert RC versions

* Proper wording in OAUTH.md api example

* remove "pre" changeset file

* xrpc: restore original behavior of setHEader and unsetHeader

* docs: add comment for XrpcClient 's constructor arg

* feat(api): expose "schemas" publicly

* feat(api): allow customizing the whatwg fetch function of the AtpAgent

* docs(api): improve migration docs

* docs: change reference to BskyAgent to AtpAgent

* docs: mention the breaking change regarding setSessionPersistHandler

* fix(api): better split AtpClient concerns

* fix(xrpc): remove unused import

* refactor(api): simplify class hierarchu by removeing AtpClient

* fix(api): mock proper method for facets detection

* restore ability to restore session asynchronously

* feat(api): allow instantiating Agent with same argument as super class

* docs(api): properly extend Agent class

* style(xrpc): var name

* docs(api): remove "async" to header getter

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Hailey <me@haileyok.com>
2024-08-12 19:57:21 +02:00
Daniel Holmgren
76c91f8325
Priority notification setting (#2648)
* priority notif settings in bsync

* lint

* priority notifications lexicon update

* codegen

* putNotificationPreferences -> putPreferences

* bsync: reorg around notif "priority", fix build, add validation & tests

* bsync: notif channel fix, tests fix

* bsky: update protos for priority notifs

* api prerelease

* add priority notif to actor state table

* dataplane impl

* appview: wire-up notif priority params

* appview: notif priority tests

* dataplane impl

* fix up tests

* tidy

* add changeset

---------

Co-authored-by: Samuel Newman <10959775+mozzius@users.noreply.github.com>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-07-23 12:33:01 -07:00
Matthieu Sieben
acc9093d28
OAuth: 2FA (#2633)
* chore(ci): update setup-node & checkout actions to v4

* refactor(oauth): rename internal types to avoid conflicting types
fix(oauth): support building from parcel
feat(oauth): add runtime lock support to prevent concurrent session updates
feat(oauth): improve metadata validation
fix(oauth): allow use of handle as login hint
fix: proper parsing of authorization header
feat(oauth): add email 2fa support
feat(oauth): adapt auth UI to match app UI

* fix(oauth): improve parsing of digest algo

* fix(oauth-provider): dead code cleanup

* fix(oauth-provider): avoid inconsistent use of "id" prop in InputCheckbox

* style(oauth-provider): use if/else instead of switch

* feat(oauth-provider): stronger validation of customization data

Invalid oauth customization would cause the server to crash at startup.

* docs(oauth-client): explain why the abortRequest method is not mandatory

* fix(oauth-client): cancel fetch response body when not used

* docs: typo

Co-authored-by: devin ivy <devinivy@gmail.com>

* feat(oauth-provider:metadata): add client_id_metadata_document_supported metadata

* fix(oauth-provider): require the content-type to be set on client metadata response

* feat(common): add obfuscation utilities
fix(pds): show user did in logs
fix(ozone): show user did in logs

* tidy

* fix(simple-store): avoid leaking context when calling hooks

* fix: use patch level changeset

* chore(oauth-types): add changeset regarding client_id_metadata_document_supported

* chore: add changeset for bsky & ozone

* unify loggerMiddleware instantiation

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-07-12 17:28:03 +02:00
Eric Bailey
8f22a2521c
Update data source for getSuggestedFollowsByActor (#2630)
* Update lex

* Codegen

* Set up StatSig

* Integrate new implementation into old endpoint

* Add todo to crypto module

* Format

* Specify StatSig env

* Downgrade pnpm to match CI, bump lock

* Catch StatSig errors

* Use sep env

* Reset lockfile

* Re-add new dep using correct pnpm version

* tidy

* Integrate into AppContext and lifecycle

* Use camelCase

* Switcheroo

Co-authored-by: devin ivy <devinivy@gmail.com>

* Init prior to server listen start

* Move test env check up to server config

* Add logger and log

* Better comment

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-07-11 17:14:54 -04:00
Paul Frazee
bc861a2c25
Add new preference and api for bsky app state; also put preference updates within transactional lock regions (#2492)
* Add bskyAppState preference and api for dismissed nudges

* Add tests

* Wrap preference updates in an lock region to avoid clobbering updates

* Move to queuedNudges and activeProgressGuide

* Fix type error

* Update test

* Add changeset
2024-07-01 18:23:21 -07:00
Matthieu Sieben
a8d6c11235
🚧 OAuth2 - Authorization Server (#2482)
* chore(deps): update zod

* chore(deps): update pino to match entryway version

* chore(tsconfig): remove truncation of types through noErrorTruncation

* add support for DPoP token type when logging

* fix(bsky): JSON.parse does not return value of type JSON

* fix(pds): add res property to ReqCtx

* fix(pds): properly type getPreferences return value

* chore(tsconfig): disable noFallthroughCasesInSwitch

* refactor(pds): move tracer config in own file

* feat(dev-env): start with "pnpm dev"

* feat(oauth): add oauth provider & client libs

* feat(pds): add oauth provider

* chore: changeset

* feat: various fixes and improvements

* chore(deps): update better-sqlite3 to version 10.0.0 for node 22 compatibility

* chore(deps): drop unused tslib

* fix(did): normalize service IDs before looking for duplicates

* fix(did): avoid minor type casting

* fix(did): improve argument validation

* fix(fetch): explicit use of negation around number comparison

* fix(oauth-provider): improve argument validation

* feat(did): add ATPROTO specific "isAtprotoDidWeb" method

* feat(rollup-plugin-bundle-manifest): add readme

* feat(lint): add eqeqeq rule (only allow == and != with null)

* fix(oauth-client-browser): typo in gitignore

* fix(oauth-provider): properly name error class file

* fix(oauth-provider): remove un-necessary useMemo

* fix(did-resolver): properly build did:web document url

* fix(did-resolver): remove unused types

* fix(fetch): remove unused utils

* fix(pds): remove unused script and dependency

* fix(oauth-provider): simplify isSubPath util

* fix(oauth-provider): add InvalidRedirectUriError static constructor

* fix(jwk): improve JWT validation to provide better error messages and distinguish between signed and unsigned tokens

* fix(pds): use "debug" log level for fetch method

* fix(pds): allow access tokens to contain an unknown "typ" claim (with the exception of "dpop+jwt")

* fix(jwk): remove un-necessary code

* fix(pds): account for whitespace chars when checking JSON

* fix(pds): remove oauth specific config

* fix(pds): run all write queries through transaction or executeWithRetry
fix(pds): remove outdated comments
fix(pds): rename used_refresh_token columns & added primary key
fix(pds): run cleanup task through backgroundQueue
fix(pds): add device.id foreign key to device_account
fix(pds): add comment on cleanup of used_refresh_token
fix(pds): add primary key on device_account

* fix(oauth-provider:time): simplify constantTime util

* fix(pds): rename disableSsrf into disableSsrfProtection

* fix(oauth-client-react-native): remove incomplete package

* refactor(pds): remove status & active from ActorAccount

* fix(pds): invalidate all oauth tokens on takedown

* fix(oauth-provider): enforce token expiry

* fix(pds): properly support deactivated accounts

* perf(pds:db): allow transaction function to be sync

* refactor(psq:account-manager): expose only query builders & data transformations utils from helpers

* fix(oauth-provider): imports from self

* fix(ci): add nested packages to build artifacts

* style(fetch): rename TODO into @TODO

* style(rollup-plugin-bundle-manifest): remove "TODO" from comment

* style(oauth-client): rename TODO into @TODO

* style(oauth-provider): rename TODO into @TODO

* refactor(oauth-client): remove "OAuth" prefix from types

* fix(oauth-client-browser): better type SessionListener

* style(oauth): rename TODO into @TODO

* fix(oauth-provider): enforce provider max session age

* fix(oauth-provider): check authentication parameters against all client metadata

* fix(api): tests

* fix(pds): remove .js from imports for tests

* fix(pds): change account status to match tests

* chore(deps): make all packages depend on the same zod version

* fix(common-web): remove un-necessary binding of Checkable to "zod"

* refactor(jwk): infer jwt schema from refinement definition

* fix(handle-resolver): allow resolution errors to propagate
docs(handle-resolver): better handling of DNS resolution errors
fix(handle-resolver): properly handle DOH responses

* fix(did): service endpoint arrays must contain "one or more" element

* refactor(pipe): simplify implementation

* fix(pds): add missing DB indexes

* feat(oauth): Resolve Authorization Server URI through Protected Resource Metadata

* style:(oauth-client): import order

* docs(oauth-provider:redirect-uri): add reference url

* feat(oauth): implement "OAuth Client ID Metadata Document" from draft-parecki-oauth-client-id-metadata-document-latest internet draft

* feat(oauth-client): backport changes from feat-oauth-client

* docs(simple-store): improve comments

* feat(lexicons): add iterable capabilities

* fix(pds): type error in dev mode

* feat(oauth-provider): improved error reporting

* fix(oauth-types): allow insecure issuer during tests

* fix(xrpc-server): allow upload of empty files

* fix: lint

* feat(fetch): keep request reference in errors
feat(fetch): utilities improvements

* fix(pds): allow more than one session token per user

* feat(ozone): improve env validation error messages

* fix(oauth-client): account for DPoP when checking for invalid_token errors

* fixup! feat(fetch): keep request reference in errors feat(fetch): utilities improvements

* fixup! feat(fetch): keep request reference in errors feat(fetch): utilities improvements

* fix(oauth): various validation fixes
feat(oauth): share client_id validation and parsing utilities between client & provider

* feat(dev-env): fix ozone port number

* fix(fetch-node): prevent fetch against invalid domain names

* fix(oauth-provider): add typings for psl dep

* feat(jwk): make type def compatible with TS 4.x

* fix(oauth): fixed various spec compliance
fix(oauth): return "sub" in refresh token response
fix(oauth): limit token validity for third party clients
fix(oauth): hide client image when not trusted

* fix(oauth): lint

* pds: switch changeset to patch, no breaking changes

* changeset and config for new oauth deps

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-06-18 15:11:37 -04:00
Foysal Ahamed
1f560f021c
Allow dynamic labeler header through ozone (#2442)
*  Allow dynamic labeler header through ozone

*  Close third party ozone

*  Add 3p labels in for repos

*  Update snapshots

*  Update pds snapshots

*  Update ozone snapshots

*  Fix typing for labels

*  Add queryLabels endpoint in appview

*  Use queryLabels to hydrate labels

* 🧹 Cleanup

*  Update pds snapshots

 Update pds snapshots

 Update ozone snapshots

 Update pds snapshots

 Update pds snapshots

 Update pds snapshots

 Update pds snapshots

*  Update pds snapshots

*  Checking if tests fail

*  Don't default to labelers in header

*  Adjust tests

* 🧹 Cleanup

* 🧹 Remove unnecessary auth header

* 📝 Add changeset

* 📝 Add @atproto/api to patch update list
2024-05-22 17:55:50 +02:00
Daniel Holmgren
c812902107
Dev-env introspection server (#2470)
* add dev-env introspection server

* Update packages/dev-env/src/introspect.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* lost a bracket

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-05-08 16:31:17 -05:00
Matthieu Sieben
173ef27cf8
fix(xrpc-server): add missing runtime dependency (#2463) 2024-05-02 21:18:34 +02:00
Foysal Ahamed
6f1f54493e
Detect language from record content if lang property is not set (#2301)
*  Detect language from record content if lang property is not set

*  Update test snapshots for auto detected language

*  Update pds test snapshots

*  Adjust test to use the right method

* 🧹 Make list override param optional

* 🧹 Fix import

* 🚨 fix linter issues

* ♻️ Use record type for getting text from record

*  process records for appview to pick it up

* 🧹 Cleanup
2024-04-18 20:15:39 +02:00
Matthieu Sieben
b8f8f06a12
fix(codegen): fix use of prettier.format for codegen (#2393)
* fix(codegen): fix use of prettier.format for codegen

* chore(deps): remove unused "babel" dependency

* chore(deps): update typescript to latest (5.4.4)
2024-04-09 10:07:47 -04:00
Matthieu Sieben
97482da17a
Update prettier & eslint (#2373)
* chore(deps): update linting tools

* style(lint): format code
2024-04-03 21:39:11 +02:00
devin ivy
c28e374868
Appview: add associated.labeler info to all profile views (#2341)
* lexicon: add associated field to profile and profile-basic views, for indicating whether labeler

* appview: add associated.labeler to profile and profile-basic views

* dev-env: downgrade get-port, non-esm-only version
2024-03-19 12:26:14 -04:00
Matthieu Sieben
f689bd51a2
Build system rework (#2169)
* refactor(crypto): remove circular dependency

* refactor(crypto): expose compress/decompress as part of the DidKeyPlugin interface

* fix(crypto): remove import from private file

* refactor: isolate tsconfig

* fix: remove unused bench file

* chore(repo): remove unused deps

* fix(ozone): properly list dependencies

* fix(services): do lint js files

* fix(services/pds): remove unused deps

* chore(pds): remove bench

* chore(dev-env): remove unused deps

* chore(api): remove bench

* remove unused babel.config.js files

* fix: remove .ts extension from import

* fix(pds): remove imports of src files

* fix(tsconfig): properly list all projects

* fix(dev-env): remove imports of src files

* fix(bsky): remove direct import to crypto src

* fix(api): remove imports to api internals

* chore(build): prevent bundling of built output

* chore(dev): add "dev" script to build in watch mode

* chore(deps): move ts-node dependency where it is actually used

* fix(deps): add dev-env as project dependency

* fix(xrpc-server): properly type kexicon

* fix(bsky): improve typings

* fix(pds): fully type formatRecordEmbedInternal return value

* fix(repo): remove imports from @ipld/car/api

* feat(dev-env): re-export BskyIngester

* fix: properly lint & type jest config & test files

* fix(ci): test after build

* fix(types): use NodeJS.Timeout instead of NodeJS.Timer

* fix(bsky): make types exportable

* fix(ozone): make types exportable

* fix(xrpc-server): make types exportable

* fix(xprc-server): make code compliant with "node" types

* fix(xrpc-server): avoid accessing properties of unknown

* chore(deps): update @types/node

* feat(tsconfig): narrow down available types depending on the package's target environment

* fix(pds): remove unused prop

* fix(bsync): Database's migrator not always initialized

* fix(dev-env): remove unreachable code

* fix(xrpc-server): remove unused import

* fix(xrpc-server): mark header property as abstract

* fix(pds): initialize LeakyTxPlugin's txOver property

* fix(bsky): initialize LeakyTxPlugin's txOver property

* fix(bsky): remove unused migrator from DatabaseCoordinator

* fix(bsky): Properly initialize LabelService's cache property

* fix(ozone): Database's migrator not initialized

* fix(ozone): initialize LeakyTxPlugin's txOver property

* fix(crypto): ignore unused variable error

* feat(tsconfig): use stricter rules

* feat(tsconfig): enable useDefineForClassFields

* feat(xrpc-server): add support for brotli incoming payload

* fix(xrpc-server): properly parse & process content-encoding

* fix(common:stream): always call cb in _transform

* tidy/fix tests and service entrypoints

* Revert "fix(xrpc-server): properly parse & process content-encoding"

This reverts commit 2b1c66e153820d3e128fc839fcc1834d52a66686.

* Revert "feat(xrpc-server): add support for brotli incoming payload"

This reverts commit e710c21e6118214ddf215b0515e68cb87299a952.

* remove special node env for tests (defaults to jest val of "test")

* kill mute sync handler on disconnect

* work around connect-es bug w/ request aborts

* style(crypto): rename imports from uint8arrays

* fix update package-lock

* fix lint

* force hbs files to be bundled as cjs

* fix: use concurrently instead of npm-run-all

npm-run-all seems not to be maintained anymore. Additionally, concurrently better forwards signals to child processes.

* remove concurrently alltogether

* ignore sqlite files in services/pds

* fix verify

* fix verify

* tidy, fix verify

* fix blob diversion test

* build rework changeset

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-03-18 17:10:58 -04:00
futur
0a2464cb67
Remove unused dependencies (#2327) 2024-03-18 16:01:15 -04:00
Paul Frazee
00a567daa0
Additional 3p labeler updates (#2286)
* Rename bsky_labeler_did

* Use labeldef default setting

* Add definedBy to interpretted label defs

* Improve dev-env mocks for labels

* Remove global label defs for dmca-violation, doxxing, and !no-promote

* Change nudity global label def to default to ignore and no longer be adult-only

* Remove old !no-promote tests

* Add mod authorities header and move bsky labeler into it

* Rename modsPref and addModService/removeModService to labelersPref and add/removeLabeler

* Add defaultSetting and adultOnly to custom label value definitions

* Rename InterprettedLabelValueDefinition to InterpretedLabelValueDefinition

* Update dev-env mock

* Move muted words and hidden posts into moderationPrefs

* Add muted word and hidden post handling to moderatePost

* Add mutewords tests

* Finish muteword tests

* Add mod-authority.test to dev-env

* Rename global label value def gore to graphic-media

* Fix typo

* Stop converting old label values

* Update label target=profile behaviors: dont filter content on hide, dont blur display names, but do show the info cards

* Update label target=account behaviors: dont blur media of content

* Add muteword moderation behaviors

* Fix mock label defs

* Implement quote-post moderation handling

* Add adult content test

* Handle sync legacy labels (#2291)

* Handle sync legacy labels

* Remap values on read

* Filter out double-written legacy label values

* Better naming, fix types

* Fix test

* Update moderation docs in sdk

* Update to new atproto-accept-labelers header behavior

* Add getLabelDefinitions() helper method

* Add proxy header support to agent

* Update mock moderation

* lint

* Implement moderation for userlists and feedgens

* Add another test label

* fix labeler in dev-env agents

* fix label hydration test

* fix lint error

* fix agent test

* fix takedown labels test

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>
2024-03-12 21:10:19 -07:00
Daniel Holmgren
7b146605c9
Updated semantics for atproto labelers header (#2292)
* update labelers header semantics

* add response header

* pds: pipe through res headers

* fix up tests

* revamp parsing
2024-03-12 16:04:12 -05:00
Foysal Ahamed
2802880a97
Add modEventDivertBlobs event to send blobs to abyss (#2238)
* 🚧 Working through an nullable review state

*  Update snapshots on some tests

*  Update snapshots on some tests

*  Add test for reviewOptional status mutation

*  Add divertBlobs event to send blobs to abyss

* ♻️ Rename reviewOptional -> reviewNone

* ♻️ Rename modEventDivertBlobs -> modEventDivert

* 🐛 Rename event type checker

*  Use pds resolver to get blob straight from pds

*  Use FOR UPDATE to respect db transactions

* ♻️ Refactor to use event_pusher table instead of new table

*  Bring back missing lines in pnpm-lock

* 🔨 Rebuild?

* 🚨 Formatting

* ♻️ Refactor to divert blob sync

* 🧹 Cleanup

*  Use modClient seed client in blob-divert test

* update divert blob config to use basic admin auth

* fix

* build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-03-12 10:40:32 -04:00
Daniel Holmgren
71f9cc9d4f
Clean up role-based auth (#2255)
* tidy bsky auth

* hook up new auth verifier

* update auth throughout ozone

* handle mod signing keys

* add client proxy heads to pds

* hook up rest of routes

* simplify pipethrough & add some SSRF protection

* tests

* fix bad var

* remove basic auth in ozone

* wip

* fix key parsing in pds

* fix up all ozone tests

* fix admin auth test

* rename test

* fix ozone test

* clean up tokens in pds

* fix up pds tests

* fix up ozone tests

* add pipethrough to write routes

* reenable proxied admin test

* add moderator accounts to ozone in dev-env

* update did doc id values

* null creds string -> `none`

* fix fetchLabels auth check

*  Add a couple more proxied requests that we use in ozone ui

* Add runit to the services/bsky Dockerfile (#2254)

add runit to the services/bsky Dockerfile

* Improve tag detection (#2260)

* Allow tags to lead with and contain only numbers

* Break tags on other whitespace characters

* Export regexes from rich text detection

* Add test

* Add test

* Disallow number-only tags

* Avoid combining enclosing screen chars

* Allow full-width number sign

* Clarify tests

* Fix punctuation edge case

* Reorder

* Simplify, add another test

* Another test, comment

* Version packages (#2261)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* 🐛 Increment attempt count after each attempt to push ozone event (#2239)

* Ozone delegates email sending to actor's pds (#2272)

* ozone delegates email sending to user's pds

* lexicon: add content field to mod email event

* test email sending via mod event

* add dev dep for nodemailer in ozone

* fix auth verifier method

* build branch

* build branch

* fix url check

* better error handling for get account infos

* fix labeler service id

* fix iss on auth headers

* fix dev-env ozone did

* fix tests & another jwt issuer

* fix proxy auth

* ozone: fix ip check

* fix aud check on pds mod service auth

* tidy

* Update packages/pds/tests/proxied/admin.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix pipethrough of headers

* fix moderation status tests

* fix auth on ozone routes

* update iss on daemon

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-03-06 19:53:52 -06:00
Daniel Holmgren
81370d7618
Ozone cdn invalidation (#2087)
* hooking up invalidator to ozone

* test + fix

* wire up service entry

* add cid to invalidation url

* add aws to ozone service package.json

* build branch

* fix build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-02-29 15:28:26 -05:00
Daniel Holmgren
f65de89eed
Feature: Appview v2 (#1924)
* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* filler commit

* rm filler

* server boilerplate

* follows impl

* posts impl

* posts & likes impl

* repost impl

* profiles & handle null values

* list impl

* mutes impl

* blocks impl

* misc

* feed gen impl

* label impl

* notifs impl

* feeds impl

* threads impl

* early sketchwork

* wip

* stub out thick client

* in-progress work on hydrator

* tweak

* hydrate profile labels, detail lists

* feedgen hydration

* protobuf tweaks

* more protobuf tweaks

* wip

* snake case

* moar snake case

* tidy actor hydration

* tidy parsing

* type fixes, renaming, comments in hydrator

* hydrate list items and likes

* hydrate notifications

* feed hydration

* graph & label hydration

* more record protobufs

* pluralize

* tweak pbs

* use new methods

* Setup dataplane grpc client/mock server (#1921)

* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* add record getter mocks

* post view hydration

* fix up mock dataplane to match new protos

* missed one

* wire up dataplane in ctx & dev-env

* adding some basic views

* feed hydration, add blocks to post hydration

* pass over notification hydration

* tidy

* merge

* implement getProfile

* hydrate post aggregation and viewer state

* fix

* fix codegen

* get some tests passing!

* add takedowns & some like bugfixing

* all profile tests passing!

* likes test

* follow endpoints using data plane

* reorg follow block rules

* reposts

* post views!

* implement getList w/ dataplane caveat

* adjust dataplane getListMembers to return listitem uris

* implement getListMutes and -Blocks w/ dataplane

* suggestions

* timeline

* misc view fixes

* view fixes for mutes, self-mute/block

* author feed

* feed gen routes

* tidy

* misc block/mute fixes

* list feed & actor likes

* implement getLists, fix some empty cursors

* implement getMutes, empty profile description fix

* implement getBlocks, block application fix

* implement getSuggestedFollowsByActor, needs some fixes

* feed generation

* search routes

* threads

* tidy

* fix some snaps

* fix getSuggestedFollowsByActor

* implement listNotifications

* implement getUnreadCount w/ dataplane

* implement notifications.updateSeen w/ dataplane

* 3rd party blocking tests

* blocked profile viewer

* add search mocks

* refactor getFeed

* createPipeline -> createPipelineNew

* basic replygating functionality on dataplane w/o filtering violating replies

* hack threadgates into dataplane, apply gates

* deterministic thread orders in dataplane

* misc cleanup around dataplane

* upgrade typescript to v5.3

* update typescript linter deps

* sync bsky proto, codegen

* update dataplane, sync with bsky proto updates

* remove indexer, ingester, daemon, moderation services from appview

* convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane

* remove bsky services. db and indexing logic into mock dataplane.

* remove tests not needed by appview v2, misc reorg

* add basic in-mem repo subscription to dataplane mock

* fix dev-env, bsky tests, bsky build

* cull bsky service entrypoint

* add bsky service readme

* build

* tidy

* tidy, fix pds proxy tests

* fix

* fix bsky entrypoint deps

* support http2 grpc client

* build

* fix dataplane bad tls config/default

* support multiple dataplane urls, retry when unavailable

* build

* tidy/fix

* move dataplane mock tests into their own dir

* cover label hydration through timeline test

* bring back labels in appview tests

* remove unused db primary/replica/coordinator from bsky dataplane

* bsky proto add cids to contracts, buf codegen

* sync-up bsky data-plane w/ codegen updates

* start using dataplane interaction endpoints

* add file

* avoid overfetching from dataplane, plumb feed items and cids

* pass refs through for post viewer state

* switch list feeds to use feed item in dataplane

* handle not found err on get-thread dataplane call

* support use of search service rather than dataplane methods

* mark some appview v2 todos

* tidy

* still use dataplane on search endpoints when search service is not configured

* fix pds test

* fix up bsky tests & snaps

* tidy migrations

* fix appview-v2 docker build

* Support label issuer tied to appview v2 (#2033)

support label issuer tied to appview

* Appview v2: handle empty cursor on list notifications (#2017)

handle empty cursor on appview listnotifs

* Update appview v2 to use author feed enum (#2047)

* update bsky protos with author feed enum, misc feed item changes

* support new author feed enums in dataplane

* fix build

* Appview v2: utilize sorted-at field in bsky protos (#2050)

utilize new sorted-at field in bsky protos

* remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts

* Appview v2, sync w/ changes to protos (#2071)

* sync bsky protos

* sync-up bsky implementation w/ proto changes

* Appview v2 initial implementation for getPopularFeedGenerators (#2072)

add an initial implementation for getPopularFeedGenerators on appview v2

* merge

* fixes

* fix feed tests

* fix bsync mock

* format

* remove unused config

* fix lockfile

* another lockfile fix

* fix duplicate type

* fix dupplicate test

* Appview v2 handling clearly bad cursors (#2092)

* make mock dataplane cursors different from v1 cursors

* fail open on clearly bad appview cursors

* fix pds appview proxy snaps

* Appview v2 no notifs seen behavior (#2096)

* alter behavior for presenting notifications w/ no last-seen time

* fix pds proxy tests

* Appview v2 dataplane retries based on client host (#2098)

choose dataplane client for retries based on host when possible/relevant

* don't apply negated labels

* display suspensions on actor profile in appview v2

* Appview v2 use dataplane for identity lookups (#2095)

* update bsky proto w/ identity methods

* setup identity endpoints on mock dataplane

* move from idresolver to dataplane for identity lookups on appview

* tidy

* Appview v2: apply safe takedown refs to records, actors (#2107)

apply safe takedown refs to records, actors

* Fix timing on appview v2 repo rev header (#2113)

fix timing on appview repo rev

* fix post thread responses

* Appview v2 don't apply 3p self blocks (#2112)

do not apply 3p self-blocks

* Appview v2 search for feed generators (#2118)

* add protos for feedgen search

* support feed search on getPopularFeedGenerators

* Appview v2 config tidy (#2117)

* remove mod and triage roles from appview

* rename cdn and search config

* remove custom feed harness from appview v2

* Appview v2: don't apply missing modlists (#2122)

* dont apply missing mod lists

* update mock dataplane

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* refactor & document a bit better

* fix up other routes

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* Appview v2 enforce post thread root boundary (#2120)

* enforce post thread root boundary

* test thread root boundary

* Appview v2 fix admin environment variable (#2137)

fix admin env in appview v2

* Remove re-pagination from getSuggestions (#2145)

* remove re-pagination from getSuggestions

* fix test

* Adjust wording for account suspension (#2153)

adjust wording for account suspension

* Appview v2: fix not-found and blocked uris in threads (#2201)

* fix uris of not-found and blocked posts in threads

* update snaps

*  Show author feed of takendown author to admins only (#2197)

* fold in cid, auth, tracing, node version changes

* remove dead config from bsky service entrypoint

* build

* remove ozone test codepaths for appview v2

* tidy, docs fix

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-02-27 15:22:55 -05:00
Foysal Ahamed
d81573232b
Allow filtering moderation queue by language (#2161)
*  Store languages on subjects from record data

*  Fetch lang from author feed for repo subject

*  Regenerate lex

*  Add lang to profile subject

*  Move lang to flag

*  Add comment to flag event

*  Update pds test snapshot

* 🐛 Fix broken import

*  Rename flag to tag

*  Update snapshot for event by id

*  Bring back skipped test

*  Move tags to dedicated columns

*  Update test snapshots, use logger

*  Change lang:unknown to lang:und

*  Update test snapshots

* 🧹 Cleanup

*  Add filter params for tags on queryModerationEvents endpoint

*  Add tags property to subject status model

*  Update test code

*  Add filter to get subjects that do not have certain tags
2024-02-19 10:35:03 -05:00
devin ivy
d0be052e12
Support node v20 on pds distribution (#2157)
* upgrade node to v20, better-sqlite3 to v9

* fix tests, deps for node v20

* build

* add io_uring setting to pds dockerfile
2024-02-09 19:05:29 -05:00
devin ivy
51aa369cd0
Include dids in bsky logs (#2015)
* include dids in bsky logs

* tidy
2024-01-26 16:43:33 -05:00
devin ivy
6a318b9f76
Appview v1 maintaining device tokens and pushing notifications w/ courier (#2073)
* add courier proto to bsky, build

* update registerPush on appview to support registering device tokens with courier

* setup bsky notifications to send to either gorush or courier

* wire courier push into indexer, test

* courier push retries

* tidy and build
2024-01-23 21:17:32 -05:00
devin ivy
d108310575
Appview v1 generating and ingesting mute ops w/ bsync (#2067)
* build bsync protos in appview, standardize per-package proto gen directory

* configure appview with bsync, allow mute endpoints to use bsync

* import fixes in bsync

* configure appview ingester with bsync, ingest mute ops into db

* test bsync mutes roundtrip w/ appview, setup bsync in dev-env

* build

* ensure to propagate errors when bsyncOnlyMutes is on
2024-01-22 15:48:31 -05:00
devin ivy
c2d0578b5a
Bsky appview sync service (#2031)
* init bsky-sync

* add bsync models and config

* rename bsky-sync to bsync

* protos and gen for bsync service

* start roughing-out bsync routes

* adjust bsync model, validation

* bsync auth, context, notify

* implement bsync scan mute ops, listen for mute op event

* setup basic bsync tests, misc fixes

* rename some files

* reorg bsync server routes

* reorg bsync server routes

* tests

* test input validation to addMuteOperation

* add db stats to bsync

* add bsync service

* redact bsync auth header from logs

* upgrade typescript to v5.3

* prettier on codegened bsync files
2024-01-09 15:40:35 -05:00
Daniel Holmgren
de2dbc2903
Split out moderation backend (#1970)
* mv appview

* copy

* finalize copy

* package names

* big WIP

* first pass at mod servce

* some tidy

* tidy & fix compiler errors

* rename to ozone, db migrations, add to dev-env & pds cfg

* getRecord & getRepo mostly working

* fix open handle

* get record tests all working

* moderation events working

* statuses working

* tidy test suite

* search repos

* server & db tests

* moderation tests

* wip daemon + push events

* pds fanout working

* fix db test

* fanning takedowns out to appview

* rm try/catch

* bsky moderation test

* introduce mod subject wrappers

* more tidy

* refactor event reversal

* tidy some db stuff

* tidy

* rename service to mod-service

* fix test

* tidy config

* refactor auth in bsky

* wip patching up auto-mod

* add label ingester in appview

* fix a couple build issues

* fix some timing bugs

* tidy polling logic

* fix up tests

* fix some pds tests

* eslint ignore

* fix ozone tests

* move seeds to dev-env

* move images around

* fix db schemas

* use service auth admin reqs

* fix remaining tests

* auth tests bsky

* another test

* random tidy

* fix up search

* clean up bsky mod service

* more tidy

* default attempts to 0

* tidy old test

* random tidy

* tidy package.json

* tidy logger

* takedownId -> takedownRef

* misc pr feedback

* split daemon out from ozone application

* fix blob takedown mgiration

* refactor ozone config

* do push event fanout on write instead of on read

* make suspend error work again

* add attempts check & add supporting index

* fix takedown test ref

* get tests working

* rm old test

* fix timing bug in event pusher tests

* attempt another fix for timing bug

* await req

* service files

* remove labelerDid cfg

* update snaps for labeler did + some cfg changes

* fix more snaps

* pnpm i

* build ozone images

* build

* make label provider optional

* fix build issues

* fix build

* fix build

* build pds

* build on ghcr

* fix syntax in entry

* another fix

* use correct import

* export logger

* remove event reverser

* adjust push event fanout

* push out multiple

* remove builds
2024-01-05 17:06:54 -06:00
devin ivy
16b31e3ce2
Proper validation of entryway op in account creation (#1975)
* ensure entryway plc rotation key

* update plc lib on pds, test
2023-12-18 12:02:17 -05:00
Matthieu Sieben
9cb81426dd
pre-compile handlebar templates at build time (#1833)
feat(pds): pre-compile handlebar templates at build time
2023-12-05 11:48:22 -05:00
Daniel Holmgren
f9fd3e68ca
Feature branch: PDS v2 (#1789)
* cleanup repeat process all

* wip

* skip actor search test

* skip actor search test

* tweak processAll

* decrease wait to 1 sec

* repo_blob -> record_blob

* simplify backlink linkTo

* return repo_root to one row

* sequence before updating repo_root

* invite code forUser -> forAccount

* ipld_block -> repo_block

* use lru-cache fetchMethod

* move did_cache to own db

* better error handling on did cache

* drop did_handle

* fix sequencer wait time

* debug

* debug

* more debug

* check something

* fix bday paradox

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* user_account -> account

* remove inviteNote

* keypair per repo

* use an lru cache for keypairs as well

* clean up repo

* wip

* wrap up accoutn manager

* tidy

* tidy

* fix tests

* fix disabled codes

* fix appview tests

* add note

* set pragmas

* tidy account manager getDb

* rename pref transactor

* user pref -> account pref

* handle blob imports

* tidy imports

* add reserveSigningKey

* wip transferAccount

* clean up transferAccount

* tests

* tidy

* tidy

* configure entryway url on pds

* handle entryway in pds admin endpoints

* make importRepo temp

* fix imports

* make email optional on pds when using entryway

* handle diffs

* handle pds entryway usage for server, identity, admin endpoints

* pds support for credentials from entryway

* setup pds tests w/ entryway service

* tidy

* tidy

* update entryway version

* wip

* test handle updates w/ entryway

* split account table into two

* tidy

* tweak scripts

* tidy tests

* tidy

* better config for actorstore & dbs

* clean up cfg more

* reorg actorstore fs layout

* handle erros on actor db create

* pr tidy & fix accoutn deletion test

* pr feedback

* fix bad merge

* unskip test

* fix subscribe repos tests

* tidy repo root tables

* tidy

* fix tests

* tidy delete tokens

* tidy account getters

* tidy

* bulk deletesg

* increase chunk size

* handle racing refreshes

* wip

* fix auth test

* invert import flow

* clean up actor store on create account failure

* tweak sequencer

* prevent invite code races on createAccount

* rm note

* add back in race protection on getAccountInviteCodes

* start feature branch

* deleted app migration table

* patch up new auth test

* rm note

* g

* create accoutn delegated from entryway

* tidy

* fix test

* change plcOp type to unknown

* small fixes

* sync up w entryway branch

* Use proper error when authed account is not found (#1799)

provide proper error when account not found in access-takedown check

* build branch

* build on ghcr

* tweak service file

* tweak service file

* change where we save reserved keys

* no tmp dir in blobstore either

* fix blobstore temp location again

* handle repeat record_blobs

* create account before submitting plc op & undo if fail

* small tweak

* limit the number of local records

* push out empty commit on transfer

* fix issue with record_blob

* add push blob endpoint

* Set and validate token audiences on pds v2 (#1793)

set and validate token audience on pds v2

* merge

* include entryway did on tests

* build branch

* fix cache issue

* xrpc server blob limit

* put correct bytes

* add auth to routes

* handle quarantining/unquarantining a blob that does not exist

* tidy

* fix transfer tests

* fix email request routes for entryway

* PDS v2 entryway account deletion (#1819)

* add admin lexicon for account deletion

* implement admin account deletion endpoint

* fix entryway proxying on account email checks

* proxy to entryway for acct deletion

* read-after-write sanity check

* tweak

* wip

* finish refactor

* fix test schema

* application retry logic for busy

* pr feedback

* rm lru-cache

* fix test pg schema

* fix transfer test

* Sqlite instrumentation for pds v2 (#1838)

* sqlite instrumentation

* build

* remove build

* dont reimport blobs

* send ticks during import

* close on error

* catch handle validation error

* add log

* fix test

* return emailConfirmedAt on getAccountInfo

* Upgrade sharp on pds v2 (#1863)

upgrade sharp to 0.32.6

* read all bytes before parsing car

* Async car reader (#1867)

* asynchronously read in car

* dont buffer car

* tweak

* Gracefully handle indexing of invalid records (#1853)

* gracefully handle indexing of invalid records

* fix repo tests

* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)

fix role auth for access-or-role verifier, fix getBlob actor takedown check

* better cleanup of actor-stores

* add ability to not ensure leaves

* tidy

* allow did:web transfer

* Migration utility for actor-store (#1873)

beginnings of helper for migrating all actors

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* base case for findBlobRefs

* App-level retries for sqlite on pds (#1871)

* revamp retry helper to be more flexible re: backoff strategies

* sqlite timeout helper

* ensure sqlite wal on db creation/migration rather than every open

* layer retries for sqlite on writes outside transactions on pds

* tidy

* fix up lockfile

* tidy

* fix lex codegen

* fix timing bug in threadgate test

* No-op update handling (#1916)

do no produce commits on no-op updates

* Retry on all SQLITE_BUSY error codes (#1917)

retry on all sqlite_busy error codes

* Pds v2 ensure sqlite ready (#1918)

ensure sqlite is ready before making queries

* try something

* tidy

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-12-04 18:00:09 -06:00