454 Commits

Author SHA1 Message Date
Foysal Ahamed
b9ca76f012
Allow searching reports by moderator did (#1283)
*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions
2023-07-14 16:05:56 -07:00
devin ivy
53c4956146
Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
2023-07-14 14:37:32 -07:00
Daniel Holmgren
6b51ecbbb2
Handle validation improvements (#1336)
* Handle matches and false positives for unacceptable words in handles

* move handle validation logic to pds

* missed merge

* add cfg flag

* encode lists

* fix build issues

* move words to cfg

* tidy

---------

Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
2023-07-13 22:29:01 -05:00
Ansh
08dc2b725d
Pagination on getPopularFeedGenerators (#1293)
* add temporary page-based pagination

* use cursor & limit for pagination

* tests

* add to appview

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-07-07 13:49:02 -05:00
devin ivy
3ea892bc57
Add administrative triage role, update moderator role (#1216)
* setup triage user on pds, update moderator username (invalidates old token)

* initial pass on triage access on pds, limit access to email addrs

* apply moderator vs triage rules on taking and reversing mod actions for pds

* update pds tests for triage auth role

* setup moderator and triage roles on bsky appview

* apply mod and triage access rules to bsky admin endpoints

* reframe admin auth as role-based auth, tidy auth apis

* tidy

* build

* revert change to basic auth username for role-based auth
2023-07-06 15:18:52 -04:00
Daniel Holmgren
60e8284b15
Fix flaky db test (#1287)
fix flaky db test
2023-07-06 09:57:41 -04:00
Daniel Holmgren
4d1f8d3289
Misc scaling (#1284)
* limit backsearch to 1 day instead of 3

* lower like count threshold

* bump to 6

* disable like count check

* disable with friends

* preemptively cache last commit

* inline list mutes

* actor service

* label cache

* placehodler on popular with friends

* bulk sequence

* no limit but chunk

* bump chunk to 5k

* try 10k

* fix notify

* tweaking

* syntax

* one more fix

* increase backfill allowance

* full refresh label cache

* limit 1 on mute list

* reserve aclu handle

* clean up testing with label cache

* note on with-friends

* rm defer from label cache

* label cache error handling

* rm branch build
2023-07-05 16:41:08 -05:00
Foysal Ahamed
03200c1d8b
🐛 Only ignore reports for specific at-uri when ignoreSubject contains at-uri (#1251) 2023-07-03 15:21:16 -04:00
Daniel Holmgren
c793ff9103
Add boolean for enabling generic appview proxying (#1273)
* add boolean config for enabling generic proxying

* tweak

* tweak cfg var name

* tweak
2023-07-03 14:14:49 -05:00
Daniel Holmgren
e7a0d27f1f
Proxy timeline skeleton construction (#1264)
proxy timeline skeleton construction to appview
2023-07-03 11:28:39 -05:00
Daniel Holmgren
4db7aeca57
Rework record embeds (#1262)
* wip

wip

wip

fix tests

tidy

* small fix

* fix flaky proxy test
2023-07-02 20:37:27 -05:00
Daniel Holmgren
e5dc790eda
Ensure not to serve repo endpoints for taken-down actor (#1263)
ensure not to serve repo endpoints for taken-down actor

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-07-02 20:22:35 -05:00
Daniel Holmgren
ee68a4037b
Appview - Invalid handles (#1244)
* run with-test-db in monorepo test

* improve pg script

* tidy

* namespace bsky pg schemas

* differentiate schemas

* clean up script

* first pass at invalid handles in appview

* tests for handle invalidation

* move mock to instance instead of prototype

* change network mocks in general

* fixing pagination on actor-search

* fix snap & normalize handles on index

* handling did pagination + update tests

* one last update
2023-06-30 16:02:47 -05:00
Daniel Holmgren
7cb8c62b54
Repo update advisory lock (#1230)
* use for no key update

* use a tx advisory lock for repo updates

* skip tests for sqlite

* move check of commit swap on rebase

* do lock before formatting rebase

* hash schema in for lock id

* no tx lock in sqlite

* move rebase formatting to tx

* move dialect check

* rm log

* make the lock ids a bit safer

* change how we do lock id

* refactor id generator
2023-06-29 13:42:10 -05:00
Daniel Holmgren
9263ddc64d
Add compression in http res (#1022)
* add compression

* compress carfiles

* update package version

* add tests

* one more test

* remove compressible dep

* add gzip to appview

* fix package.json
2023-06-29 10:24:48 -05:00
Daniel Holmgren
ba870d3dbe
Run test DB for full monorepo tests (#1225)
* run with-test-db in monorepo test

* improve pg script

* tidy

* namespace bsky pg schemas

* differentiate schemas

* clean up script

* tweaking script
2023-06-28 15:12:37 -05:00
Jess Porter
d9ac72e647
firehose tombstones (#1017)
* firehose tombstones

* return tombstones from sequencer requestSeqRange

* firehose tombstone test

* oops semicolons :3

* yes this can be const

* fix verifyTombstoneEvent func signature

* invalidate all sequence events upon tombstone

* don't manually purge all seqs, test for tombstone after delete

* actually fully delete account

* fix linting

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-26 18:48:00 -05:00
Daniel Holmgren
2e94c43010
Cleanup @atproto/crypto (#1218)
* remove webcrypto, upgrade @noble/curves, normalize p256 interface

* cleanup

* Test vectors for secp and p256 signature verification (#737)

Add test vectors for secp and p256 signature verification

* fix up test vectors

* add explicit test vectors for high-s signatures

* tidy json to pass verify check

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-06-26 18:47:28 -05:00
Daniel Holmgren
0ee6dc45a5
Improve invite code counting (#1234)
* dont count admin codes for routine codes

* better tests for invites

* variable naming

* change approach

* update comment

* tidy
2023-06-24 13:07:37 -05:00
Ansh
0306f81d37
Add ability to embed lists (#1188)
* add cid to ListView and ListViewBasic

* add test for mute list embed

* add support for list embeds views

* test

* port to appview

* update missing snap

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-06-23 16:30:52 -04:00
devin ivy
3da0324873
Post record language tagging, lexicon language format (#1219)
* Add languages field to post record

* helper for parsing bcp47

* add language format to lexicon

* codegen for post record langs field

* re-export language parsing in api package

* tests and tidy for lexicon language format

* index post langs, in-progress

* update snapshots, fixes record-with-media embed issue

* index post langs on bsky appview

* don't bother indexing post langs in pds appview, tidy
2023-06-23 16:23:16 -04:00
devin ivy
e651ed67a4
Respect takedowns on feedgen records, sync routes (#1119)
* Don't serve high-level details about taken-down feedgens

* Takedown checks on sync routes

* Use simpler check for takedowns on sync routes

* ensure takendown feedgens are handled in bsky appview
2023-06-21 15:07:16 -04:00
Daniel Holmgren
5d85188509
Invite epochs (#1205)
* add invite epoch

* reorg test

* fix dev env

* tidy
2023-06-13 17:49:47 -05:00
Daniel Holmgren
8c4fa530d6
Pds proxy to appview based on headers (#1191)
* pds proxy to appview based on headers

* reorg appview routes forwarding mutes to appview

* tidy

* clean up ctx

* track notif state

* use notif state for notif routes

* add back in missing file

* pr feedback

* wip

* more views

* more views

* update snaps & suggestions test

* better choices for like & reposted

* note

* better block test

* Test pds-proxied mute and notif state

* Update bsky snaps, remove unnecessary #postView

* Update bsky notification tests to use updateSeen

* add getList getFeedGen tests

* Re-disconnect pds from in-process appview indexing for tests

* clean up forSnapshot

* tidy

* feed gen view tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 17:29:32 -05:00
devin ivy
0ed0eb5d5f
Remove appview repo subscription from PDS (#1195)
* Remove repo subscription from pds appview

* Remove appview repo subscription config from pds
2023-06-12 17:01:59 -04:00
Daniel Holmgren
08fbb33fdf
Appview blocks (#1187)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* hook up indexing

* Setup custom feeds on bsky

* graph service + getList routes

* migration

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* Test custom algos on bsky

* add mute state to snapshots

* table & indexing plugin

* getBlocks

* bail out of mute subquery when no viewer

* fleshing out services

* Test pds getFeed proxy

* tidy, fix build

* tidy

* Fix minor race in bsky mutelist tests

* blocks in feed views

* db migration, impl block logic & tests

* Update pds proxy snaps based on appview mutes

* missed some files on last commit

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

* apply blocks in getFeed

* update snaps

* upadate one last snap

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 11:05:22 -05:00
devin ivy
7981c5948f
Misc. syncing-up bsky appview with on-pds appview (#1189)
* Misc notification and optimization in bsky, tidy

* Handle notifs on message queue in bsky appview

* Sync-up profile labeling behavior in bsky

* No async work during actor unindexing in bsky, include additional indexes

* Prevent deeply nested notifs w/ hellthread exception in bsky appview

* Add indexes to bsky appview for more efficient actor deletion/unindexing

* Update pds proxy snaps for updated bsky profile labeling

* Return custom suggestions in bsky appview
2023-06-11 14:55:43 -04:00
Daniel Holmgren
4f2f3d8d17
Appview mutes (#1184)
* hook up indexing

* graph service + getList routes

* migration

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* add mute state to snapshots

* bail out of mute subquery when no viewer

* Fix minor race in bsky mutelist tests

* Update pds proxy snaps based on appview mutes

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-10 17:12:15 -04:00
devin ivy
283fb1d8d6
Custom feeds on bsky appview (#1185)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* Setup custom feeds on bsky

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* Test custom algos on bsky

* Test pds getFeed proxy

* tidy, fix build

* tidy
2023-06-09 11:48:17 -04:00
Foysal Ahamed
8857fb0845
Filter reports by reporter did (#1165)
*  Hydrate subject author's repo in moderation reports

*  Allow filtering moderation reports by reporter DIDs

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package

*  Port over filter by reporter implementation to bsky package

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-06-09 09:21:09 -04:00
Foysal Ahamed
9a2dc7119e
Hydrate subject author's repo in moderation reports (#1135)
*  Hydrate subject author's repo in moderation reports

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package
2023-06-08 10:17:52 -04:00
Daniel Holmgren
35e035dad2
Disable resolution of handles through xrpc (#1179)
* disable resolution of handles through xrpc

* tweak admin.updateHandle
2023-06-07 16:00:52 -05:00
Daniel Holmgren
6fb9a8faea
Fix dropped events in sync.subscribeRepos (#1167)
* wip

* first pass done

* chnage around how channels function

* bugfixing

* fix up migration & tests

* fix dev-env

* add prep migration for stream fixing

* application code changes to use new repo_event table

* dont send evts

* add outgoing_repo_seq table + sqlite migrations

* patch up down migration

* patching up a couple things

* update test

* patchup

* re-enable subscription route

* add outgoingSeq column

* updating

* accidental add

* temporarily disable stream

* resolve merge conflict

* add migration

* migration for seq

* fix test

* id -> seq

* get schema lock ids

* tidy

* fix server test

* fix up

* use bsky uniqueLockId approach

* tidy

* gh actions build

* build branch

* build branch

* tweak test

* rm constraint rename

* small bugfix

* check for seq not null

* fix up issues

* one more small change

* tidy

* no build branches
2023-06-06 19:18:44 -05:00
Foysal Ahamed
46e1f051e1
Allow searching repo by DID (#1084)
*  Allow searching repo by DID

*  Port over did search from pds

*  Add tests for bsky implementation of admin searchRepos

* 🧹 Cleanup according to PR review
2023-06-06 00:12:14 -04:00
bnewbold
a4a4af1693
all packages: prettier applies to tests/ in addition to src/ (#1170)
* pds: prettier:fix (on a JSON file)

* package.json (multiple): run prettier on tests/

* bsky: prettier:fix (on a JSON file)
2023-06-05 19:26:32 -07:00
devin ivy
53a107b237
Fix pagination limit when not respected by feedgens (#1129)
* Enforce limit param in the case that the feedgen does not

* Tidy
2023-06-01 17:36:22 -04:00
Daniel Holmgren
87fc4ce1bc
Return custom suggestions (#1118)
* return custom suggestions on server

* allow ordered suggestions
2023-05-30 13:23:22 -05:00
Foysal Ahamed
4a6c9764e3
Allow reverting chronological order of moderation reports (#1113)
*  Allow reverting chronological order of moderation reports

*  Port over report sorting to bsky package
2023-05-30 11:50:46 -04:00
Foysal Ahamed
3414fcedbf
Allow ignoring moderation reports by subject (#1091)
*  Allow ignoring moderation reports by subject

* 🔧 Remove unwanted package.json change

*  Remove subject format specifier

*  Port over ignore subject implementation to bsky package

* Revert " Port over ignore subject implementation to bsky package"

This reverts commit 3c782c1805548d1ebdbbd08b68b6e92ac97fdee5.

* 🚧 Port over ignore subject implementation to bsky package

*  Generate lexicons for bsky
2023-05-29 00:39:03 -04:00
Daniel Holmgren
693c703d78
Skip popular with friends test (#1109)
skip test
2023-05-26 18:48:35 -05:00
Daniel Holmgren
ec15af5de8
Algo tweaks (#1108)
* Temporarily skip online/valid checks for feedgens while external feedgens catch-up

* tweak algos

* hook up did resolver with cache

* Re-enable whats-hot w/ materialization

* pulling a fast one on the query optimizer

* clean up

* disable branch build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-05-26 18:27:23 -05:00
devin ivy
9eb817d488
Materialize candidates for whats-hot algo (#1086)
* Setup whats-hot materialized view w/ params, run view maintainer in service entrypoint

* Update whats-hot to use new materialized view, tidy

* Update migration to create with no data

* Revert whats-hot migration change for no data

* Bump refresh rate for view down to 1min
2023-05-26 15:05:53 -04:00
Daniel Holmgren
03a74b4259
Catch missed txn rollbacks (#1094)
* catch missed rollbacks

* change approach

* revert back to simple way

* test

* add to bsky package as well

* improve handler & tests
2023-05-26 11:04:26 -05:00
devin ivy
7a7c9c75af
Improved custom algo debugging (#1096)
* Better feedgen errors in getFeed and getFeedGenerator

* Support handler output headers in codegen

* Support handler output headers in xrpc-server

* Implement server-timing helpers and apply them to getFeed

* fix codegen issue

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-05-26 11:59:05 -04:00
David Buchanan
c62964b7b4
Limit getPostThread recursion into parents (#1062)
* limit getPostThread recursion into parents

* add parentHeight parameter

* Lexicons: specify default/min/max values for depth and parentHeight, for getPostThread

* prettier formatting

* re-run lexicon codegen

* do the rest of lexicon codegen (oops)

* add 'shallow ancestors' fetch test
2023-05-25 19:22:05 -05:00
Daniel Holmgren
743eaf1cf0
Move handle resolution to .well-known (#1048)
* move handle resolution to .well-known

* required handle on resolveHandle

* rm test

* tidy

* tidy

* fix up appview

* missing await

* atproto-handle -> atproto-did

* shift did & handle resolution to new identity package

* fix up network mocks

* fix up another test

* one more

* drop lex comment

* rm handle param

* Update packages/identity/src/handle/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* still temporarily support xrpc handle resolution

* typo

* ensure return value is a string

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-05-25 19:04:10 -05:00
Daniel Holmgren
5df2903061
Feed publishing tweaks (#1087)
* add feed gen did to cfg

* allow puts for generator records

* add a quick test

* quick test for updates

* fix up test
2023-05-24 17:38:01 -05:00
devin ivy
53044ae48e
Get popular feedgens (#1090)
* Lexicon for unspecced popular feed gen method

* Endpoint impl for popular feed gens
2023-05-24 17:37:53 -05:00
devin ivy
f7c502b145
Improve performance of whats-hot custom algo (#1078)
Improve whats-hot algo perf by reorganizing query, excluding some unpopular results more proactively
2023-05-23 23:01:51 -04:00
devin ivy
7e22719967
Tweaks to with-friends algo (#1082)
* Iteration on with-friends algo

* Fix with-friends tests
2023-05-23 15:50:45 -04:00