8 Commits

Author SHA1 Message Date
Daniel Holmgren
f9fd3e68ca
Feature branch: PDS v2 (#1789)
* cleanup repeat process all

* wip

* skip actor search test

* skip actor search test

* tweak processAll

* decrease wait to 1 sec

* repo_blob -> record_blob

* simplify backlink linkTo

* return repo_root to one row

* sequence before updating repo_root

* invite code forUser -> forAccount

* ipld_block -> repo_block

* use lru-cache fetchMethod

* move did_cache to own db

* better error handling on did cache

* drop did_handle

* fix sequencer wait time

* debug

* debug

* more debug

* check something

* fix bday paradox

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* user_account -> account

* remove inviteNote

* keypair per repo

* use an lru cache for keypairs as well

* clean up repo

* wip

* wrap up accoutn manager

* tidy

* tidy

* fix tests

* fix disabled codes

* fix appview tests

* add note

* set pragmas

* tidy account manager getDb

* rename pref transactor

* user pref -> account pref

* handle blob imports

* tidy imports

* add reserveSigningKey

* wip transferAccount

* clean up transferAccount

* tests

* tidy

* tidy

* configure entryway url on pds

* handle entryway in pds admin endpoints

* make importRepo temp

* fix imports

* make email optional on pds when using entryway

* handle diffs

* handle pds entryway usage for server, identity, admin endpoints

* pds support for credentials from entryway

* setup pds tests w/ entryway service

* tidy

* tidy

* update entryway version

* wip

* test handle updates w/ entryway

* split account table into two

* tidy

* tweak scripts

* tidy tests

* tidy

* better config for actorstore & dbs

* clean up cfg more

* reorg actorstore fs layout

* handle erros on actor db create

* pr tidy & fix accoutn deletion test

* pr feedback

* fix bad merge

* unskip test

* fix subscribe repos tests

* tidy repo root tables

* tidy

* fix tests

* tidy delete tokens

* tidy account getters

* tidy

* bulk deletesg

* increase chunk size

* handle racing refreshes

* wip

* fix auth test

* invert import flow

* clean up actor store on create account failure

* tweak sequencer

* prevent invite code races on createAccount

* rm note

* add back in race protection on getAccountInviteCodes

* start feature branch

* deleted app migration table

* patch up new auth test

* rm note

* g

* create accoutn delegated from entryway

* tidy

* fix test

* change plcOp type to unknown

* small fixes

* sync up w entryway branch

* Use proper error when authed account is not found (#1799)

provide proper error when account not found in access-takedown check

* build branch

* build on ghcr

* tweak service file

* tweak service file

* change where we save reserved keys

* no tmp dir in blobstore either

* fix blobstore temp location again

* handle repeat record_blobs

* create account before submitting plc op & undo if fail

* small tweak

* limit the number of local records

* push out empty commit on transfer

* fix issue with record_blob

* add push blob endpoint

* Set and validate token audiences on pds v2 (#1793)

set and validate token audience on pds v2

* merge

* include entryway did on tests

* build branch

* fix cache issue

* xrpc server blob limit

* put correct bytes

* add auth to routes

* handle quarantining/unquarantining a blob that does not exist

* tidy

* fix transfer tests

* fix email request routes for entryway

* PDS v2 entryway account deletion (#1819)

* add admin lexicon for account deletion

* implement admin account deletion endpoint

* fix entryway proxying on account email checks

* proxy to entryway for acct deletion

* read-after-write sanity check

* tweak

* wip

* finish refactor

* fix test schema

* application retry logic for busy

* pr feedback

* rm lru-cache

* fix test pg schema

* fix transfer test

* Sqlite instrumentation for pds v2 (#1838)

* sqlite instrumentation

* build

* remove build

* dont reimport blobs

* send ticks during import

* close on error

* catch handle validation error

* add log

* fix test

* return emailConfirmedAt on getAccountInfo

* Upgrade sharp on pds v2 (#1863)

upgrade sharp to 0.32.6

* read all bytes before parsing car

* Async car reader (#1867)

* asynchronously read in car

* dont buffer car

* tweak

* Gracefully handle indexing of invalid records (#1853)

* gracefully handle indexing of invalid records

* fix repo tests

* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)

fix role auth for access-or-role verifier, fix getBlob actor takedown check

* better cleanup of actor-stores

* add ability to not ensure leaves

* tidy

* allow did:web transfer

* Migration utility for actor-store (#1873)

beginnings of helper for migrating all actors

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* base case for findBlobRefs

* App-level retries for sqlite on pds (#1871)

* revamp retry helper to be more flexible re: backoff strategies

* sqlite timeout helper

* ensure sqlite wal on db creation/migration rather than every open

* layer retries for sqlite on writes outside transactions on pds

* tidy

* fix up lockfile

* tidy

* fix lex codegen

* fix timing bug in threadgate test

* No-op update handling (#1916)

do no produce commits on no-op updates

* Retry on all SQLITE_BUSY error codes (#1917)

retry on all sqlite_busy error codes

* Pds v2 ensure sqlite ready (#1918)

ensure sqlite is ready before making queries

* try something

* tidy

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-12-04 18:00:09 -06:00
Daniel Holmgren
fcb19c9c51
Simplify PDS moderation (#1723)
* spec out new simple pds mod routes

* introduce new admin state endpoints

* wire up routes

* clean up pds

* revoke refresh tokens

* getUserAccountInfo

* pr tidy

* fixing some tests

* fixing up more tests

* fanout takedowns to pds

* fanout admin reqs to pds

* tidy

* more tidy & add more pds moderation tests

* getUserAccountInfo -> getAccountInfo

* dont hydrate pds info on searchRepos

* fix build

* port admin tests to bsky package

* clean up old snaps

* tests on fanout

* tweak naming

* missed a rename

* tidy renames

* fix lex name

* tidy & move snap

* fix build

* cleanup repeat process all

* skip actor search test

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* change takedownId col to takedownRef

* build branch

* fix bsky test

* add service key to indexer

* move signing key to api entry

* dont build
2023-10-30 16:56:17 -05:00
Daniel Holmgren
d425061179
Fix test seed & snap (#1733)
fix test seed & snap
2023-10-11 10:26:00 -05:00
Daniel Holmgren
d664b51c64
Finalize PDS in-process AppView removal (#1198)
* rm tables

* rm event-stream & proxied

* Remove appview services, move label service to pds

* only proxy appview stuff

* delete more tables

* Start removing message dispatched from pds

* more syncing-up removal of message dispatcher in pds

* merged

* remove feedgens from pds, remove getPopular

* remove unused image helper from pds

* fixing compiler errors

* clean up sharp

* rm label service

* first pass on cleaning up tests

* fix up a bunch of tests

* moderation view tests

* last admin tests

* got a lil overzealous in deletes

* clean up unused cfg

* clean up label table

* simplify admin repo search query/logic

* tidy pds entrypoint

* in-progress pds config changes

* cfg fiddling

* finish cleaning up cfg/ctx

* comments

* building

* pds prefix on env

* test env

* collapse pds migrations down into a single migration

* fix up dev-env

* tidy

* cleanup

* fix pds admin tests

* fix handle test

* fix pds proxy tests

* fix subscribe repos test

* fix sqlite config in pds tests

* add sqlite clause in sequencer-leader

* fix actor search w/ sqlite on pds

* fixes

* fix dev env build

* update pds service entrypoint

* simple env example

* make takedown ids opaque identifiers in the pds

* use pds routes for api tests

* update pds dockerfile with volume and correct port env var

* add a couple env vars to example

* add comments to env example

* @atproto/pds 0.2.0-beta.0

* @atproto/aws 0.0.1-beta.0

* appview did

* @atproto/aws 0.0.1

* enable logs by default

* update env example

* bugfixing sandbox issues

* consistency in pds env var name for appview url

* log on pds start and stop, configure version at runtime

* @atproto/pds 0.2.0-beta.1

* fix semver matching for pds beta version

* v0.2.0-beta.2

* default invites to being not required

* fix flaky test

* limit db connections in tests

* publish 0.2.0-beta.d3

* fix invite required parsing

* @atproto/pds 0.2.0-beta.5

* Proxy getPopularFeedGenerators on simplified pds (#1222)

proxy getPopularFeedGenerators on pds

Co-authored-by: dholms <dtholmgren@gmail.com>

* tidy migrations

* fix service entry

* bump version

* change auth order

* bump version

* bump version

* add upgradeRepoVersion & fallback url for cdn

* bump version

* merging

* merge pds

* building dev-env

* merging tests

* merge service entry

* test fixing

* tidy

* fix admin search

* tidy

* tidy

* add snap for getListFeed

* add backup nameserver cfg

* tidy + pr feedback

* tidy

* tidy env

* bit more

* re-add dotenv to root package.json

* fix dep

* build branch

* fix tests

* Refactor tests to make better use of dev-env (#1690)

* refactor pds tests to use dev env

* refactor bsky tests

* fix pds test

* tidy bsky tests

* build pds correctly

* fix entry point

* default logging to false (for now)

* format service entry

* Switch takedown ids back to ints on pds distribution (#1694)

* switch takedown ids back to ints, consistent with live pds

* tidy/fix migration

* update migration for sqlite

* export moderation action reversal

* takedown tests

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-10-02 13:27:45 -05:00
Paul Frazee
ab50816461
Self-labeling (#1444)
* Add self-label schemas

* switch around array and union for self-labels

* apply self-labels to post and profile views

* test for self labels in pds

* apply self-labels to post and profile views in bsky appview

* test for self labels in bsky appview

* update pds proxied test snapshots

* Add support for self-labels to the mdoeration sdk

* Disable unknown-labeler filtering until 3P support is added

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-09 19:06:29 -04:00
devin ivy
3f3ae4cfd5
Support proxying moderation through to appview (#1233)
* in-progress work on proxying moderation to appview

* tidy

* proxy reports pds to appview, misc tidy

* test proxying of moderation endpoints

* remove report action fkeys from pds for appview sync

* test appview/pds moderation synchronization

* tidy

* tidy

* fix admin proxy tests, build

* temp disable migration

* rm relative @atproto/api imports

* fix a couple more relative imports

* tidy

* reenable migration, comment contents temporarily

* fully enable migration, remove build, misc test fixes

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-04 16:06:04 -04:00
Daniel Holmgren
d6ac47d76e
Label profiles in feeds (#805)
* Label profiles in feeds

* send empty array
2023-04-12 18:14:23 -05:00
Daniel Holmgren
962041e6c1
Scene trending (#327)
* wip

* views

* trending schema

* starting message queue

* scene processor

* wip

* send mq messages from db

* db events

* undo screwing up codegen lol

* setup queue

* db migrations

* fixing up message processing

* div by 0 check

* tx issue

* queue use cursor

* update not insert

* sql bugfix + tests

* trying to linearize tests

* correclty serializing txs

* attempt update before insert

* log errors

* handle gaps in cursor

* cleanup

* oops reenable test

* correctly incr cursor
2022-11-11 12:29:48 -06:00