440 Commits

Author SHA1 Message Date
Daniel Holmgren
b75436b9cb
Delete user state on account delete (#970)
* Delete user state on account delete

* blocks too
2023-05-03 22:39:57 -05:00
Daniel Holmgren
015fe1226c
Delete dereferenced blobs (#959)
* delete dereferenced blobs

* invalidate cache for delete blobs

* start testing

* test & cleanup

* rm blob cleanup logic on rebases

* make disk blob deletes idempotent

* move blob deletion to background queue
2023-05-03 18:34:55 -05:00
devin ivy
b1493650f4
Perform account deletion async (#951)
* Perform takedown then background hard-deletion process for deleteAccount

* Remove transactions from account deletion process, perform steps sequentially
2023-05-02 14:24:59 -04:00
devin ivy
142c77cadd
Disallow writes of bad records (#955)
* Disallow writes of bad records

* Increase obj depth to hit stack issue on more systems
2023-05-02 13:44:07 -04:00
Daniel Holmgren
86fae264d3
Sequencer race condition (#906)
* wip

* split out seq invalidations

* wip

* fix

* comments

* 2 db hits

* cleanup

* re-add comment
2023-05-02 11:39:25 -05:00
Daniel Holmgren
bff96544d1
Tweak whats hot (#945)
* tweak whats hot

* couple fixes & test
2023-05-01 14:10:48 -05:00
Devin Ivy
3d1b3b3675 Add csp and nosniff when serving user uploads 2023-04-29 11:47:50 -04:00
Daniel Holmgren
db1042cf0d
Blocks (#922)
* block lexicons

* table schema

* block processing

* Actor-block migration

* Add blocked post lexicon to getPostThread

* getBlocks routes

* Add missing lexicons to bsky

* Add block state to actor views

* throw on blocked accounts in getAuthorFeed

* blcok state in profile views

* Update getPostThread to support block views, update existing test

* strip out blocks out oof timeline & popular

* more routes

* Test blocked posts in threads

* Support blocks on record embeds

* notifs

* Update pds test snapshots w/ block state

* fix search

* Update pds actor search snapshots w/ blocks

* test

* tests

* update lex

* fix up from returning blocking as record

* ci for branch

* fix build issue in bsky

* more blocks tests

* fix actor search test

* hook getBlocks up to router

* bugfix & actually test lol

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-28 19:36:02 -05:00
devin ivy
6b3d5c133e
Misc notification and suggestion optimizations (#920)
* Background notification creation/deletion

* Add todo

* Update tests for backgrounded notifs

* Add index to notifications supporting record deletion

* Use profile_agg to power getSuggestions

* Fetch record blocks separately from getting notification resultset

* Update background queue to drain pending tasks
2023-04-27 19:10:21 -04:00
devin ivy
da6ee3908a
Background notification creation and deletion (#919)
* Background notification creation/deletion

* Add todo

* Update tests for backgrounded notifs

* Undo backgrounding of notification creation

* Revert "Undo backgrounding of notification creation"

This reverts commit 507353a9802da75a232fc245ede30d47dab74e5f.
2023-04-27 17:12:18 -04:00
devin ivy
5249e52f36
Lengthen pass reset and invite codes (#912)
* Lengthen pass reset and invite codes, invalidation on reset

* Tidy

* Adjust token format for reset and deletion

* Update invite codes to use same token gen as pass reset and acct deletion
2023-04-27 14:26:31 -04:00
Daniel Holmgren
384e739a3b
Allow admins to update handles to reserved handles (#916)
* allow admins to update handles to reserved handles

* tidy
2023-04-27 12:02:47 -05:00
Daniel Holmgren
4b7f7a5294
Appview proxying bugfixes (#897)
* fix some bugs in proxying reqs to appview

* update snap
2023-04-25 17:24:46 -05:00
Devin Ivy
01501f0d49 Fix snapshots 2023-04-25 17:33:29 -04:00
devin ivy
eb7667e4fc
Add labels onto record embeds (#854)
* Add lexicons for labeling embeds

* Hydrate labels onto record embeds in pds

* Support labels on embeds in bsky appview
2023-04-25 17:19:40 -04:00
Daniel Holmgren
68004c617f
Appview proxy (#827)
* fix up a couple of tsc errors in app view merge

* wip

* simple proxy

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* wip

* copy proxied tests & add mutes to getFollows/getFollowers

* tidy & add mutes to likes

* more routes + getAuthorFeed tests

* more testing

* tests for feed views

* thread testing

* finished tests for threads

* temporarily skip some tests

* cleaning up & updating test names

* separate db schema for appview

* rearrange

* typo

* add notifications

* re-enable notifs on pds appview

* update schemas

* updated some bsky snaps

* wip

* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* fix up & move to appCtx

* neat its working

* update bsky tests to new auth

* rm unused pds config var

* tidy

* check exp in seconds

* cache dids in postgres

* add migration & did-cache

* start tests

* couple helpers around cache invalidation

* fix expired check

* wip

* change cache semantics

* did cache testing

* do some cache revalidation in indexing

* fix config

* fix issue w did-resolver test-env prototype

* use map instead of record

* stale + expired

* tests

* clear entry method

* fix up build

* expired dids

* clear missing dids

* better verify payload

* bump test timeout

* fix notifs test

* fix up proxied actor search tests

* update snaps to include labels

* fix dev env

* fix up moderation route auth

* fix more auth headers

* fix auth on getPosts

* increase jest timeouts

* fix snaps
2023-04-25 15:26:14 -05:00
Daniel Holmgren
cae6779916
Fix leaky database transactions (#890)
* fix leaky txs

* tidy

* end tx in finally
2023-04-25 10:51:59 -05:00
devin ivy
0c02804a73
Ensure indexes are used on actor search queries (#879)
* In-progress work on optimizing actor search

* Tidy-up approach to using trgm index on actor search, apply to bsky appview

* Fix comment typo
2023-04-24 11:17:30 -04:00
Daniel Holmgren
553f59923c
Get posts method (#876)
* implement getPosts method

* tests

* bsky tests

* comment out test

* Handle stripping record-with-media viewer info in bsky tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-23 23:43:37 -05:00
Daniel Holmgren
f30887bc1c
Clean up Did-resolver (#842)
* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* couple helpers around cache invalidation

* fix expired check

* change cache semantics

* use map instead of record

* stale + expired

* tests

* clear entry method

* clear missing dids

* add missing type
2023-04-23 13:36:01 -05:00
Daniel Holmgren
2b5347b534
Remove profile labels (#866)
* moderation tweks

* fix: typo. (#818)

* Create invite codes for many accounts (#825)

* create invite codes for many accounts

* test

* Revamp dev env (#796)

* fix up a couple of tsc errors in app view merge

* wip

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* fixed missed merge conflict

* fix one more merge conflict

* fix popular test

* Sync test env with labeler changes (#836)

fix test-env

* Remove extraneous info on getRecord (#835)

* remove extraneous info on getRecord

* fix test-env

* Fix duplication of constants in the crypto package (#819)

* App passwords (#826)

* app password lex & auth chnages

* scrypt things

* implemented app password refresh tokens

* db tidy & migration

* revocation + bugfixin

* tests, listing passwords & cleanup

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* remove profile labels

* rm temp branch from workflow

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: S. Ota <1632335+susumuota@users.noreply.github.com>
Co-authored-by: Ilya Siamionau <ilya@marshal.dev>
2023-04-23 12:18:24 -05:00
devin ivy
a832c54de7
PDS moderator credentials (#863)
* Setup config and auth verifiers for moderators

* Enforce admin vs. moderator access on PDS admin/server endpoints

* Tidy
2023-04-21 20:58:34 -04:00
Daniel Holmgren
89109ef32d
Format commits outside of tx (#860)
* format commits outside of tx

* fix tests

* fix bsky test

* use persistent cache

* reuse repo storage between attempts to get caching benefits

* better error handling

* tests

* rm failing sqlite test

* pr feedback
2023-04-21 18:13:04 -05:00
Daniel Holmgren
4f1ad109af
Increase getPopular likes to 8 (#862)
* increase get popular likes to 8

* use new like counts agg
2023-04-21 18:12:44 -05:00
devin ivy
a0f0301241
Serve materialized post and profile aggregations (#859)
* Add migration for post & profile aggs to pds

* Post/profile agg db tweaks

* Post/profile aggregation queries on pds

* Add facility for performing work on db commit in pds

* Add background queue to pds

* Perform aggregations during indexing on background queue

* Tidy pds background queue, incorporate into labeler

* Fix pds onCommit, remove unused indexing dep, fix zeros for aggs

* Finish background tasks for view tests in pds

* Update aggregations on account deletion, tidy

* Update profile and post views to serve materialized aggregations
2023-04-21 12:01:23 -04:00
devin ivy
587b24352d
Materialize post and profile aggregations on write (#858)
* Add migration for post & profile aggs to pds

* Post/profile agg db tweaks

* Post/profile aggregation queries on pds

* Add facility for performing work on db commit in pds

* Add background queue to pds

* Perform aggregations during indexing on background queue

* Tidy pds background queue, incorporate into labeler

* Fix pds onCommit, remove unused indexing dep, fix zeros for aggs

* Finish background tasks for view tests in pds

* Update aggregations on account deletion, tidy
2023-04-21 11:52:45 -04:00
Daniel Holmgren
9a8d7529a3
Skip locked repos instead of waiting for lock (#847)
* skip locked repos instead of waiting for lock

* some tests

* rm racing tests on profile

* subscribe repos test
2023-04-19 21:57:06 -05:00
Daniel Holmgren
c71e351c38
Do not give notifs for a deleted subject (#843)
generally do not give notifs for a deleted subject
2023-04-19 16:24:18 -05:00
Daniel Holmgren
6446e8d1d6
App passwords (#826)
* app password lex & auth chnages

* scrypt things

* implemented app password refresh tokens

* db tidy & migration

* revocation + bugfixin

* tests, listing passwords & cleanup

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-18 13:47:04 -05:00
Daniel Holmgren
9e1bb49957
Create invite codes for many accounts (#825)
* create invite codes for many accounts

* test
2023-04-18 10:44:34 -05:00
devin ivy
6a39ebf9f7
Stop sequencer listener in outbox when subscription aborts (#828)
* Update types to allow for abort signal in xrpc subscriptions

* Support abort signal in xrpc-server subscriptions

* Stop sequencer listener in outbox when subscription aborts
2023-04-17 14:22:03 -04:00
Daniel Holmgren
d8b50c73e4
Add admin.updateAccountEamil (#812)
* -add admin capability to update account email

* pr feedback
2023-04-13 14:43:53 -05:00
devin ivy
4aa9103fb5
Minor query optimization for getBlocksForCommits() (#811) 2023-04-13 14:39:49 -05:00
Daniel Holmgren
efb1cac2bf
Label reversal fix (#809)
* Reproduce issue with empty negateLabelVals

* normalize str arrays

* Reproduce issue with reversing labels on repo

* Use multiple labels for record labeling test

* join with space

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-12 23:48:47 -05:00
Daniel Holmgren
aa46ad1e1c
Admin labeling (#801)
* adding to moderation flows

* some bugfixing in labels

* hardcode profile nsid

* make labels off moderation action

* db updates

* wip

* report labels in admin views & reverse moderation actions

* Test admin get record and repo w/ labels

* update db

* fix

* exclude negs from labels

* exclude neg on moderation views as well

* Check-in missing lex

* Check-in missing lex

* In-progress admin label tests

* Test label creation/reversal via actions

* Admin label test snapshots (#808)

* new snapshots for label on user

* fix get moderation action snap

* fix dev-env

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-12 22:53:32 -05:00
Daniel Holmgren
d6ac47d76e
Label profiles in feeds (#805)
* Label profiles in feeds

* send empty array
2023-04-12 18:14:23 -05:00
Daniel Holmgren
521c1c1edb
Admin update handle (#803)
* admin route for updating accoutn handle on services domain

* tests

* Update lexicons/com/atproto/admin/updateAccountHandle.json

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-12 17:23:58 -05:00
Daniel Holmgren
0762e8b7f6
Label bugfixes (#802)
* some bugfixing in labels

* hardcode profile nsid
2023-04-12 17:23:32 -05:00
Daniel Holmgren
c41232b2ef
Fix DB schema in rebase tests (#799)
fix typoed db schema in test
2023-04-12 14:34:39 -05:00
Daniel Holmgren
8b1da9fed8
Labeling (#789)
* take notifs off of message queue

* fix order of notif handling

* table

* reintroducing message queue

* setting up user

* added labeler classes

* tidy + hook up labeler to config / service start

* hooking up p-queue to labeler

* rip out message queue

* drop mq tables

* rm stream consumers

* keyword labeler

* tidy + some tests

* work labels into views

* update snaps

* labeled images in dev-env

* snaps

* labeler tests

* more labels

* Update packages/pds/src/api/app/bsky/notification/listNotifications.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/dev-env/src/mock/index.ts

* Update packages/dev-env/src/mock/index.ts

* Update packages/pds/src/labeler/util.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/pds/tests/labeler/hive.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback + migration change

* fix lower case keywords issue

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-12 14:34:18 -05:00
Daniel Holmgren
d8b975d8dd
Normalize handles on resolve (#795)
* normalize handles on resolve

* tidy & test
2023-04-12 10:49:49 -07:00
Daniel Holmgren
286c6bbef5
Remove mutes & takedowns from getPopular (#793)
remove mutes & takedowns from getPopular
2023-04-11 22:58:06 -07:00
Daniel Holmgren
95ac1808ac
No notifs for any records from muted accounts (#791) 2023-04-11 21:55:20 -07:00
Daniel Holmgren
15762260fd
Sync/listRepos (#784)
* add getHostedRepos lex & route

* tests

* tidy

* rename
2023-04-11 17:16:13 -07:00
Daniel Holmgren
fc4a34caf2
Fully deprecate handle on createSession (#761)
* fully deprecate handle on createSession

* fix tests
2023-04-11 17:10:35 -07:00
Daniel Holmgren
8ecc232796
Synchronous notifs (#785)
* take notifs off of message queue

* fix order of notif handling

* rip out message queue

* drop mq tables

* rm stream consumers

* Tidy

* Temporarily remove drop message queue migration

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-11 17:05:39 -07:00
devin ivy
6b55a95c88
Build timeline and author feeds from feed items index (#774)
* Index posts and reposts into feed_item table for building feeds

* Use feed_item table for building timeline and author feeds

* Apply feed item indexing to bsky app view

* Fix bsky appview tests, test getPopular

* Use feed item index to build feeds in bsky app view
2023-04-09 13:21:48 -07:00
devin ivy
2b8e1ac387
Index posts and reposts as feed items (#773)
* Index posts and reposts into feed_item table for building feeds

* Apply feed item indexing to bsky app view
2023-04-09 12:13:18 -07:00
Daniel Holmgren
31d5258922
Fix pg sequencer test (#777)
* fix pg sequencer test

* tidy
2023-04-07 10:43:19 -05:00
devin ivy
4e83748545
Bsky app view (#716)
* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect

* Cull bsky config, services, auth, etc.

* Sweep app view xrpc methods, tidy deps, add storage back for img server

* Run repo subscription on bsky app view

* Collapse db migrations down for bsky app view

* Tidy app view bin

* Remove mute functionality from app view, delegate to pds

* Initial tidy/culling of bsky app view tests

* Passing bsky app view db, server, and repo subscription tests

* Passing bsky app view duplicate-records tests

* Bsky app view test tidy/cull

* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.

* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.

* Setup actor handles by did in bsky app view

* Passing indexing tests on bsky app view

* Passing image tests on bsky app view

* Fix bsky actor reindexing, support custom lock id for testing repo subs

* Sweep bsky view tests, misc tests, passing

* Tidy bsky deps

* Include did in resized image uris

* Update bsky image process server to use getBlob

* Update image server tests, misc fixes

* Implement bsky blob resolver

* Wire local image processing server to local blob resolver, test blob resolver

* Tidy

* Tidy

* Tidy

* Tidy app view init

* Fix handle resolution, tidy

* Add utils for partitioning indexing by did

* Update repo sub to parallelize work per repo

* Dep tidy

* Tidy bsky tests for updated repo sub destroy()

* Update thead indexing to handle out-of-order posts

* Sketch out strategy in bsky for handling too-big commits

* Set content-type on sync.getBlob

* Add logging for failed transmissions in bsky blob resolver

* Tidy

* Tidy bsky repo indexing and supporting repo interfaces

* Sort in app view based on combo of creation and indexing times

* Fix types

* Add retry utils to bsky

* Add retries to http requests made by bsky

* Test repo indexing

* Update bsky db/model for lex refactor

* Update bsky lexicons for lex refactor

* Update bsky actor service for lex refactor

* Update bsky feed service for lex refactor

* Update bsky indexing service for lex refactor

* Update bsky repo subscription for lex refactor

* Tidy bsky repo sub

* Add unspecced endpoints to bsky app view, update entrypoint

* Update bsky xrpc utils for lex refactor

* Update bsky xrpc methods for lex refactor

* Update bsky test seeds for lex refactor, tidy api entrypoint

* Update bsky non-view tests for lex refactor

* Update bsky likes view test for lex refactor, minor fix

* Update bsky author feed tests for lex refactor, minor test util fix

* Update bsky follow, profile, repost, search view tests for lex refactor

* Update bsky timeline view tests for lex refactor

* Replace bsky out-of-order thread indexing logic

* Update bsky thread view tests for lex refactor, general test tidying

* Handle rebases and too-big commits in repo subscription, tracking commit data cid

* Tidy

* Ensure did resolver reports "not found" only when positively not found

* Handle tombstones and handle updates in bsky

* Test indexing handle updates and did tombstones

* Support cors on bsky

* Allow app view to serve most routes unauthed

* Tests for bsky unauthed views

* Tidy bsky service entrypoint and dockerfile

* Remove unused storage interfaces from bsky

* Bsky entrypoint and dockerfile fixes, tidy

* Add workflow for bsky build to aws

* Use more standard db env variables, make migration creds optional

* Make bsky repo subscription optional

* Fix lex->json serialization in bsky

* Split bsky actor sync state into its own table

* Skip invalid records on indexing full repo, tidy

* Tidy

* Leader test timing

* Tidy/lint

* Fix bsky config overrides
2023-04-06 18:47:46 -04:00