426 Commits

Author SHA1 Message Date
devin ivy
283fb1d8d6
Custom feeds on bsky appview (#1185)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* Setup custom feeds on bsky

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* Test custom algos on bsky

* Test pds getFeed proxy

* tidy, fix build

* tidy
2023-06-09 11:48:17 -04:00
Foysal Ahamed
8857fb0845
Filter reports by reporter did (#1165)
*  Hydrate subject author's repo in moderation reports

*  Allow filtering moderation reports by reporter DIDs

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package

*  Port over filter by reporter implementation to bsky package

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-06-09 09:21:09 -04:00
Foysal Ahamed
9a2dc7119e
Hydrate subject author's repo in moderation reports (#1135)
*  Hydrate subject author's repo in moderation reports

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package
2023-06-08 10:17:52 -04:00
Daniel Holmgren
35e035dad2
Disable resolution of handles through xrpc (#1179)
* disable resolution of handles through xrpc

* tweak admin.updateHandle
2023-06-07 16:00:52 -05:00
Daniel Holmgren
6fb9a8faea
Fix dropped events in sync.subscribeRepos (#1167)
* wip

* first pass done

* chnage around how channels function

* bugfixing

* fix up migration & tests

* fix dev-env

* add prep migration for stream fixing

* application code changes to use new repo_event table

* dont send evts

* add outgoing_repo_seq table + sqlite migrations

* patch up down migration

* patching up a couple things

* update test

* patchup

* re-enable subscription route

* add outgoingSeq column

* updating

* accidental add

* temporarily disable stream

* resolve merge conflict

* add migration

* migration for seq

* fix test

* id -> seq

* get schema lock ids

* tidy

* fix server test

* fix up

* use bsky uniqueLockId approach

* tidy

* gh actions build

* build branch

* build branch

* tweak test

* rm constraint rename

* small bugfix

* check for seq not null

* fix up issues

* one more small change

* tidy

* no build branches
2023-06-06 19:18:44 -05:00
Foysal Ahamed
46e1f051e1
Allow searching repo by DID (#1084)
*  Allow searching repo by DID

*  Port over did search from pds

*  Add tests for bsky implementation of admin searchRepos

* 🧹 Cleanup according to PR review
2023-06-06 00:12:14 -04:00
bnewbold
a4a4af1693
all packages: prettier applies to tests/ in addition to src/ (#1170)
* pds: prettier:fix (on a JSON file)

* package.json (multiple): run prettier on tests/

* bsky: prettier:fix (on a JSON file)
2023-06-05 19:26:32 -07:00
devin ivy
53a107b237
Fix pagination limit when not respected by feedgens (#1129)
* Enforce limit param in the case that the feedgen does not

* Tidy
2023-06-01 17:36:22 -04:00
Daniel Holmgren
87fc4ce1bc
Return custom suggestions (#1118)
* return custom suggestions on server

* allow ordered suggestions
2023-05-30 13:23:22 -05:00
Foysal Ahamed
4a6c9764e3
Allow reverting chronological order of moderation reports (#1113)
*  Allow reverting chronological order of moderation reports

*  Port over report sorting to bsky package
2023-05-30 11:50:46 -04:00
Foysal Ahamed
3414fcedbf
Allow ignoring moderation reports by subject (#1091)
*  Allow ignoring moderation reports by subject

* 🔧 Remove unwanted package.json change

*  Remove subject format specifier

*  Port over ignore subject implementation to bsky package

* Revert " Port over ignore subject implementation to bsky package"

This reverts commit 3c782c1805548d1ebdbbd08b68b6e92ac97fdee5.

* 🚧 Port over ignore subject implementation to bsky package

*  Generate lexicons for bsky
2023-05-29 00:39:03 -04:00
Daniel Holmgren
693c703d78
Skip popular with friends test (#1109)
skip test
2023-05-26 18:48:35 -05:00
Daniel Holmgren
ec15af5de8
Algo tweaks (#1108)
* Temporarily skip online/valid checks for feedgens while external feedgens catch-up

* tweak algos

* hook up did resolver with cache

* Re-enable whats-hot w/ materialization

* pulling a fast one on the query optimizer

* clean up

* disable branch build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-05-26 18:27:23 -05:00
devin ivy
9eb817d488
Materialize candidates for whats-hot algo (#1086)
* Setup whats-hot materialized view w/ params, run view maintainer in service entrypoint

* Update whats-hot to use new materialized view, tidy

* Update migration to create with no data

* Revert whats-hot migration change for no data

* Bump refresh rate for view down to 1min
2023-05-26 15:05:53 -04:00
Daniel Holmgren
03a74b4259
Catch missed txn rollbacks (#1094)
* catch missed rollbacks

* change approach

* revert back to simple way

* test

* add to bsky package as well

* improve handler & tests
2023-05-26 11:04:26 -05:00
devin ivy
7a7c9c75af
Improved custom algo debugging (#1096)
* Better feedgen errors in getFeed and getFeedGenerator

* Support handler output headers in codegen

* Support handler output headers in xrpc-server

* Implement server-timing helpers and apply them to getFeed

* fix codegen issue

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-05-26 11:59:05 -04:00
David Buchanan
c62964b7b4
Limit getPostThread recursion into parents (#1062)
* limit getPostThread recursion into parents

* add parentHeight parameter

* Lexicons: specify default/min/max values for depth and parentHeight, for getPostThread

* prettier formatting

* re-run lexicon codegen

* do the rest of lexicon codegen (oops)

* add 'shallow ancestors' fetch test
2023-05-25 19:22:05 -05:00
Daniel Holmgren
743eaf1cf0
Move handle resolution to .well-known (#1048)
* move handle resolution to .well-known

* required handle on resolveHandle

* rm test

* tidy

* tidy

* fix up appview

* missing await

* atproto-handle -> atproto-did

* shift did & handle resolution to new identity package

* fix up network mocks

* fix up another test

* one more

* drop lex comment

* rm handle param

* Update packages/identity/src/handle/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* still temporarily support xrpc handle resolution

* typo

* ensure return value is a string

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-05-25 19:04:10 -05:00
Daniel Holmgren
5df2903061
Feed publishing tweaks (#1087)
* add feed gen did to cfg

* allow puts for generator records

* add a quick test

* quick test for updates

* fix up test
2023-05-24 17:38:01 -05:00
devin ivy
53044ae48e
Get popular feedgens (#1090)
* Lexicon for unspecced popular feed gen method

* Endpoint impl for popular feed gens
2023-05-24 17:37:53 -05:00
devin ivy
f7c502b145
Improve performance of whats-hot custom algo (#1078)
Improve whats-hot algo perf by reorganizing query, excluding some unpopular results more proactively
2023-05-23 23:01:51 -04:00
devin ivy
7e22719967
Tweaks to with-friends algo (#1082)
* Iteration on with-friends algo

* Fix with-friends tests
2023-05-23 15:50:45 -04:00
Daniel Holmgren
8059e07d8a
Custom feeds (#1001)
* actor type in profiles

* skeleton lex

* more schemas

* first pass at feed composition

* cleanup services

* more tidying

* fix up tests

* bookmarking schemas

* make feeds records instead of actors

* some cleanup

* fleshing out feeds as records

* wip

* working into cfg & dev-env

* fix appview test util

* bookmarking + views

* tidy

* add displayName

* db migration

* wip test

* blob processing

* first test working!

* Sync-up snapshots

* wip

* Fix pds ctx type

* Sync lexicons on pds and api

* wip

* fully refactor

* Prep for open union in reply data in bsky appview, sync-up snapshots

* msgs & ports

* Make pds did cache sqlite-friendly

* Start scaffolding more feedgen tests

* revamp appview tests

* revamp pds tests

* merge in dev env utils

* ensure plc close

* add feed gen to dev-env

* second feed gen

* named feeds in mock

* fix issue with dev-env pds appview

* update bsky snaps

* Flesh-out some feed gen tests

* Fix feed gen reply test

* Test feed bookmarking

* update skeleton defs

* merge

* Sync-up pds with updated feed skeleton contract

* Fix feed bookmark viewer state

* add feed embeds

* Test inaccurate feed gen repost

* Tighten some error behaviors around feed gens

* generator view in record embed

* test

* bookmarked/subscribed -> saved

* updated snaps

* name & descript on alices feed

* add like counts

* proposed lex

* tweak

* fetch feed generator details

* add describeFeedGen to dev-env

* adds tests for getFeedGenerator

* Update lexicons/app/bsky/feed/getFeedGenerator.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* update describe

* add cid to feed gen view

* add alice feed gen avatar

* generator displayName required

* update snaps

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* cleanup lex

* mount route

* Starter bsky algos (#1046)

* implement some starter bsky algos

* add team dids

* tidy

* add quotepost check to whats hot classic

* tests

* carve out feedgen section of pds

* apply limits

* add describeFeedGenerator route

* Whats hot new (#1060)

* map out new whats hot

* only available in postgres

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* build algos branch

* remove relative imports to @atproto/api

* update hot classic name

* rm build

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>

* v0.3.6

* v0.3.7

* v0.1.10

* update name of skyline

* build branch

* aws build

* getFeedGenerators test

* For now only use new replyRef union for custom feeds, for compat

* Don't use replyRef union on new feeds yet

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-05-19 18:31:51 -04:00
devin ivy
e57c297a82
Handling deleted content on admin/moderation methods (#1045)
* Update lexicon for missing repo/record data in admin views

* Serve missing repo/record variant in admin views

* Test actioning and presenting unknown records/repos, fix on pds
2023-05-17 15:28:52 -04:00
Daniel Holmgren
47d30c1796
Revamp dev env (#1031)
* wip

* wip

* fully refactor

* msgs & ports

* revamp appview tests

* revamp pds tests

* ensure plc close

* ensure pds appview is hooked up when not running appview

* move service setup to dev env

* move admin auth headers to dev-env

* fix service headers

* missing port

* fixing ports

* start services in order

* startup plc immediately

* rm dead code

* fix up build

* rm log
2023-05-15 12:28:36 -05:00
Daniel Holmgren
231729db72
Return email in mod views according to lexicon (#885)
* return email in mod views according to lexicon

* Update packages/pds/src/services/moderation/views.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* updated snaps

* tidy

* Fix snapshot

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-05-15 11:43:06 -04:00
devin ivy
1cbffd63ef
Reports by active action type (#1028)
* Lexicon updates for getting mod reports by active action type

* Update pds and bsky to support getting mod reports by active action type
2023-05-12 16:45:15 -05:00
Daniel Holmgren
5804716504
Bring your own DID (#1011)
* allow bringing your own did

* tests + tidy

* one more check/test

* fix typo

* better err names
2023-05-12 16:18:49 -05:00
Paul Frazee
df6ed7d5c0
Add account preferences APIs (#1013)
* Add lexicons for account preferences

* Move prefs to app.bsky, codegen

* Setup model and services for user prefs

* Setup xrpc methods for prefs

* Test preferences, fixes

* Tidy

* Tidy

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-05-11 15:44:00 -05:00
Daniel Holmgren
e32dc025e8
Mute lists (#986)
* possible schemas

* more lexicons

* codegen

* tables

* hook up processor

* getlist route

* add getListBlocks route

* migration

* didnt save something

* add reason + facets

* description facets

* get facets into db

* add to accnt deletion tests before i forget

* tests & bugfixes

* quickfix

* move defs to .graph

* add purpose

* remove unused actor defs

* add creator info to list view

* refactor to graph service

* lexicons

* impl

* writer methods

* bug in accnt service

* handle blobs

* tests

* fix typos

* tidy

* fix snap

* pr feedback

* missed drop index in migration

* remove listitem reason

* allow puts for lists

* make mute lists idempotent

* test mute list updates

* fix up

* fix err test

* update snap
2023-05-11 14:29:08 -05:00
Patryk
a7c2b4ae11
Add block record duplicate detection (#1027)
* Add block record duplicate detection

* Update packages/pds/src/services/record/index.ts

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
2023-05-11 11:00:24 -04:00
Daniel Holmgren
89b783eec9
Update how sequencer invalidation works (#1014)
* invalidatedBy -> invalidated

* fix test

* dont remake fkey constraint sqlite

* sqlite migration

* appeasing sqlite

* sqlite finally happy?

* Fix sqlite migration, run proxy tests thru postgres when configured

* Fix pg schemas in proxy tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-05-10 11:01:56 -04:00
devin ivy
977202fa2b
Fix indexed record datetimes (#1009)
* Normalize record datetimes used for sort to simplified iso

* Update tests and snapshots for alternate valid datetime formats

* Fix notifications race during record update
2023-05-09 12:53:33 -04:00
Daniel Holmgren
d30b0f93e1
Disable account invites (#1004)
* enable disabling account invites

* xtra test

* couple fixes

* allow re-enabling user accoutn invites

* add in getter route for if invites are disabled or not

* fix admin search test

* gneerated type on user account
2023-05-09 09:24:12 -05:00
Daniel Holmgren
b20a5045a2
Rebase route (#948)
* tidy rebase query

* add rebase route

* bring tests up to speed

* add tx check
2023-05-09 05:54:48 -05:00
Daniel Holmgren
c8e6adbbf5
Clean up what's hot (#1002)
* changes to clean up whats hot

* fix test

* add quote posts back in & use underwear label instead of no-promote

* tidy
2023-05-08 18:26:11 -05:00
Daniel Holmgren
b75436b9cb
Delete user state on account delete (#970)
* Delete user state on account delete

* blocks too
2023-05-03 22:39:57 -05:00
Daniel Holmgren
015fe1226c
Delete dereferenced blobs (#959)
* delete dereferenced blobs

* invalidate cache for delete blobs

* start testing

* test & cleanup

* rm blob cleanup logic on rebases

* make disk blob deletes idempotent

* move blob deletion to background queue
2023-05-03 18:34:55 -05:00
devin ivy
b1493650f4
Perform account deletion async (#951)
* Perform takedown then background hard-deletion process for deleteAccount

* Remove transactions from account deletion process, perform steps sequentially
2023-05-02 14:24:59 -04:00
devin ivy
142c77cadd
Disallow writes of bad records (#955)
* Disallow writes of bad records

* Increase obj depth to hit stack issue on more systems
2023-05-02 13:44:07 -04:00
Daniel Holmgren
86fae264d3
Sequencer race condition (#906)
* wip

* split out seq invalidations

* wip

* fix

* comments

* 2 db hits

* cleanup

* re-add comment
2023-05-02 11:39:25 -05:00
Daniel Holmgren
bff96544d1
Tweak whats hot (#945)
* tweak whats hot

* couple fixes & test
2023-05-01 14:10:48 -05:00
Devin Ivy
3d1b3b3675 Add csp and nosniff when serving user uploads 2023-04-29 11:47:50 -04:00
Daniel Holmgren
db1042cf0d
Blocks (#922)
* block lexicons

* table schema

* block processing

* Actor-block migration

* Add blocked post lexicon to getPostThread

* getBlocks routes

* Add missing lexicons to bsky

* Add block state to actor views

* throw on blocked accounts in getAuthorFeed

* blcok state in profile views

* Update getPostThread to support block views, update existing test

* strip out blocks out oof timeline & popular

* more routes

* Test blocked posts in threads

* Support blocks on record embeds

* notifs

* Update pds test snapshots w/ block state

* fix search

* Update pds actor search snapshots w/ blocks

* test

* tests

* update lex

* fix up from returning blocking as record

* ci for branch

* fix build issue in bsky

* more blocks tests

* fix actor search test

* hook getBlocks up to router

* bugfix & actually test lol

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-28 19:36:02 -05:00
devin ivy
6b3d5c133e
Misc notification and suggestion optimizations (#920)
* Background notification creation/deletion

* Add todo

* Update tests for backgrounded notifs

* Add index to notifications supporting record deletion

* Use profile_agg to power getSuggestions

* Fetch record blocks separately from getting notification resultset

* Update background queue to drain pending tasks
2023-04-27 19:10:21 -04:00
devin ivy
da6ee3908a
Background notification creation and deletion (#919)
* Background notification creation/deletion

* Add todo

* Update tests for backgrounded notifs

* Undo backgrounding of notification creation

* Revert "Undo backgrounding of notification creation"

This reverts commit 507353a9802da75a232fc245ede30d47dab74e5f.
2023-04-27 17:12:18 -04:00
devin ivy
5249e52f36
Lengthen pass reset and invite codes (#912)
* Lengthen pass reset and invite codes, invalidation on reset

* Tidy

* Adjust token format for reset and deletion

* Update invite codes to use same token gen as pass reset and acct deletion
2023-04-27 14:26:31 -04:00
Daniel Holmgren
384e739a3b
Allow admins to update handles to reserved handles (#916)
* allow admins to update handles to reserved handles

* tidy
2023-04-27 12:02:47 -05:00
Daniel Holmgren
4b7f7a5294
Appview proxying bugfixes (#897)
* fix some bugs in proxying reqs to appview

* update snap
2023-04-25 17:24:46 -05:00
Devin Ivy
01501f0d49 Fix snapshots 2023-04-25 17:33:29 -04:00