Commit Graph

2335 Commits

Author SHA1 Message Date
devin ivy 0ed0eb5d5f Remove appview repo subscription from PDS (#1195)
* Remove repo subscription from pds appview

* Remove appview repo subscription config from pds
2023-06-12 17:01:59 -04:00
Daniel Holmgren 7f9d2c1ebe Notify crawlers on write (#1194)
* notify crawlers on write

* pr feedback

* more resilient config
2023-06-12 15:00:17 -05:00
Daniel Holmgren 08fbb33fdf Appview blocks (#1187)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* hook up indexing

* Setup custom feeds on bsky

* graph service + getList routes

* migration

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* Test custom algos on bsky

* add mute state to snapshots

* table & indexing plugin

* getBlocks

* bail out of mute subquery when no viewer

* fleshing out services

* Test pds getFeed proxy

* tidy, fix build

* tidy

* Fix minor race in bsky mutelist tests

* blocks in feed views

* db migration, impl block logic & tests

* Update pds proxy snaps based on appview mutes

* missed some files on last commit

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

* apply blocks in getFeed

* update snaps

* upadate one last snap

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 11:05:22 -05:00
devin ivy 5951d9347f Misc. syncing-up bsky appview with on-pds appview (pt. 2) (#1190)
* Misc notification and optimization in bsky, tidy

* Handle notifs on message queue in bsky appview

* Sync-up profile labeling behavior in bsky

* No async work during actor unindexing in bsky, include additional indexes

* Prevent deeply nested notifs w/ hellthread exception in bsky appview

* Add indexes to bsky appview for more efficient actor deletion/unindexing

* Update pds proxy snaps for updated bsky profile labeling

* Return custom suggestions in bsky appview

* Test out-of-order indexing of aggregations on bsky appview

* Proxy pds getPopular to hot-classic feed on appview. Proxy getPopularFeedGenerators to appview.

* Make did field required on feedgen view
2023-06-11 15:48:20 -04:00
devin ivy 7981c5948f Misc. syncing-up bsky appview with on-pds appview (#1189)
* Misc notification and optimization in bsky, tidy

* Handle notifs on message queue in bsky appview

* Sync-up profile labeling behavior in bsky

* No async work during actor unindexing in bsky, include additional indexes

* Prevent deeply nested notifs w/ hellthread exception in bsky appview

* Add indexes to bsky appview for more efficient actor deletion/unindexing

* Update pds proxy snaps for updated bsky profile labeling

* Return custom suggestions in bsky appview
2023-06-11 14:55:43 -04:00
Daniel Holmgren 4f2f3d8d17 Appview mutes (#1184)
* hook up indexing

* graph service + getList routes

* migration

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* add mute state to snapshots

* bail out of mute subquery when no viewer

* Fix minor race in bsky mutelist tests

* Update pds proxy snaps based on appview mutes

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-10 17:12:15 -04:00
devin ivy 283fb1d8d6 Custom feeds on bsky appview (#1185)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* Setup custom feeds on bsky

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* Test custom algos on bsky

* Test pds getFeed proxy

* tidy, fix build

* tidy
2023-06-09 11:48:17 -04:00
Daniel Holmgren 9a873148bd Appview lists & listmutes (#1183)
* hook up indexing

* graph service + getList routes

* migration

* mute lists

* add getListMutes

* mute list tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-09 10:46:03 -04:00
devin ivy f46d95617d Feed gen indexing and view endpoints on bsky appview (#1182)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* tidy, fix build

* tidy/fix bsky migration
2023-06-09 10:30:54 -04:00
Devin Ivy 1033eddc00 v0.3.12 2023-06-09 09:28:32 -04:00
Foysal Ahamed 8857fb0845 Filter reports by reporter did (#1165)
*  Hydrate subject author's repo in moderation reports

*  Allow filtering moderation reports by reporter DIDs

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package

*  Port over filter by reporter implementation to bsky package

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-06-09 09:21:09 -04:00
devin ivy 98fdd5c376 Materialize post and profile aggregations on bsky appview (#1177)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky
2023-06-08 13:18:30 -04:00
Devin Ivy 5eb777c3f2 v0.3.11 2023-06-08 10:32:52 -04:00
Foysal Ahamed 9a2dc7119e Hydrate subject author's repo in moderation reports (#1135)
*  Hydrate subject author's repo in moderation reports

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package
2023-06-08 10:17:52 -04:00
dholms 85e3cdaf7b publish identity@0.0.1 2023-06-07 18:31:01 -05:00
Daniel Holmgren 35e035dad2 Disable resolution of handles through xrpc (#1179)
* disable resolution of handles through xrpc

* tweak admin.updateHandle
2023-06-07 16:00:52 -05:00
devin ivy dd38c8b3b5 Normalize creation times on list, list item, feedgen records (#1178)
Normalize creation times on lists, list items, feedgens
2023-06-07 16:58:15 -04:00
devin ivy ff78a30b9e Ensure keyset pagination makes use of indexes on timeline (#1174)
Alternative pagination query snippet for better use of indexes
2023-06-07 11:46:33 -04:00
devin ivy a23fb1eac3 Fix lists pagination, ensure paginating on correct column (#1175)
Ensure lists pagination uses createdAt both in query and for building cursor
2023-06-07 11:41:02 -04:00
Daniel Holmgren 6fb9a8faea Fix dropped events in sync.subscribeRepos (#1167)
* wip

* first pass done

* chnage around how channels function

* bugfixing

* fix up migration & tests

* fix dev-env

* add prep migration for stream fixing

* application code changes to use new repo_event table

* dont send evts

* add outgoing_repo_seq table + sqlite migrations

* patch up down migration

* patching up a couple things

* update test

* patchup

* re-enable subscription route

* add outgoingSeq column

* updating

* accidental add

* temporarily disable stream

* resolve merge conflict

* add migration

* migration for seq

* fix test

* id -> seq

* get schema lock ids

* tidy

* fix server test

* fix up

* use bsky uniqueLockId approach

* tidy

* gh actions build

* build branch

* build branch

* tweak test

* rm constraint rename

* small bugfix

* check for seq not null

* fix up issues

* one more small change

* tidy

* no build branches
2023-06-06 19:18:44 -05:00
Foysal Ahamed 46e1f051e1 Allow searching repo by DID (#1084)
*  Allow searching repo by DID

*  Port over did search from pds

*  Add tests for bsky implementation of admin searchRepos

* 🧹 Cleanup according to PR review
2023-06-06 00:12:14 -04:00
bnewbold a4a4af1693 all packages: prettier applies to tests/ in addition to src/ (#1170)
* pds: prettier:fix (on a JSON file)

* package.json (multiple): run prettier on tests/

* bsky: prettier:fix (on a JSON file)
2023-06-05 19:26:32 -07:00
devin ivy c20ed66f2a Optimization for label subqueries on feeds (#1164)
* temp disable hot-classic

* Switch labels in whats-hot to values list

* Switch labels in getPopular (legacy hot-classic) to values list

* Reenable hot-classic, switch labels to values list

* remove branch build in workflow
2023-06-03 14:10:57 -04:00
Daniel Holmgren bf4ac79cef Fix up what's hot (#1133)
* do whats hot off of candidate feed

* tidy
2023-06-01 21:26:40 -05:00
devin ivy 53a107b237 Fix pagination limit when not respected by feedgens (#1129)
* Enforce limit param in the case that the feedgen does not

* Tidy
2023-06-01 17:36:22 -04:00
Daniel Holmgren 243972bd69 Rebase indices (#1126)
* prepping for rebases

* add down migration
2023-06-01 15:34:41 -05:00
bnewbold a280ae21e1 identifier tweaks (#1085)
* nsid: reduce 'name' length from 128 chars to 63 chars

* nsid: remove 'nsid-ns' (glob pattern) from main NSID syntax validation

This special variant syntax should be handled separately. Eg, an
"nsid-ns" should not be accepted as an NSID in Lexicons looking for a
regular NSID.

* nsid: update syntax check to match domain rules

This is to be closer to actual underlying specification needs. For
example, domains are allowed to start with a digit (like 4chan.org), and
can't start *or* end with a hyphen.

Restricts the "name" part further, to be alphabetic only.

Note that starts-with-digit domains could break some lang
auto-generation variable name rules. The docs/spec strongly recommend
against such domains for use with NSID, but it seems incorrect to
restrict at the standard/lexicon level without a clear pan-language
consistent set of rules.

* did-resolver: disallow did:web with path segments

* identifier: reduce max length of DID from 8 KB to 2 KB

* identifier: add DID test with escaped ':' in identifier

* did-resolver: have prettier run on tests/

* identifier: add additional punycode test cases

* identifiers: small tweaks from review

- type in did:web comment
- include actual max DID string length in error message

* xrpc-server tests: remove digits from NSIDs

* identity: fix import and function errors

* xrpc-server tests: remove digits from NSIDs

* xrpc-server: include tests/ in prettier:fix

* xrpc-server: lint fix
2023-05-31 16:56:08 -07:00
devin ivy feb320c93e Revert perf tweak to with-friends (#1125)
Revert change to with-friends
2023-05-31 12:37:40 -04:00
devin ivy 4e592ef7f8 Perf tweaks to with-friends, mutuals, label batching (#1124)
* Tweak with-friends and mutuals queries to be less constrained for planner, omit reposts in mutuals feed

* More batching of label queries
2023-05-31 11:09:45 -04:00
Daniel Holmgren 9d33cbb945 Don't throw on dns lookup errors (#1122)
discard dns lookup errors
2023-05-30 19:09:02 -05:00
Daniel Holmgren 756658454d Mount admin.rebaseRepo (#1121)
hook up admin rebase route
2023-05-30 15:53:58 -05:00
devin ivy 95252ddb4f Batch fetching of labels across posts and actors (#1117)
* Rename getLabelsForSubjects to getLabelsForUris

* Generic method for getting profiles organized by did or aturi subject

* Batch label fetching for posts and authors in feeds and threads

* Tidy
2023-05-30 13:23:33 -05:00
Daniel Holmgren 87fc4ce1bc Return custom suggestions (#1118)
* return custom suggestions on server

* allow ordered suggestions
2023-05-30 13:23:22 -05:00
devin ivy 3c9646bee6 Revert feed date threshold for with-friends (#1120)
Revert feed date threshold for with-friends, not beneficial
2023-05-30 14:14:43 -04:00
Devin Ivy 85765852b5 v0.3.10 2023-05-30 12:18:36 -04:00
Foysal Ahamed 4a6c9764e3 Allow reverting chronological order of moderation reports (#1113)
*  Allow reverting chronological order of moderation reports

*  Port over report sorting to bsky package
2023-05-30 11:50:46 -04:00
devin ivy 01cb3831f4 Followup to getRepo and getTimeline perf tweaks (#1116)
* To avoid performing many view refreshes in a row w/ different leaders, schedule them upfront

* Apply feed threshold to with-friends and mutuals algos

* Revert increase to getRepo commit chunk size

* Alter feed threshold approach to be tighter and relative to cursor
2023-05-30 11:46:11 -04:00
devin ivy a900e955a6 Disallow scheduling repeated view refreshes (#1114)
To avoid performing many view refreshes in a row w/ different leaders, schedule them upfront
2023-05-29 14:40:45 -04:00
Daniel Holmgren 39cde47c11 Admin rebases (#1112)
add admin route for kicking off rebases
2023-05-29 12:42:39 -05:00
devin ivy 6ee810c6d8 Minor perf tweaks for getTimeline and getRepo (#1111)
* Avoid long scans for users w/ sparse feeds on getTimeline

* Increase getRepo commit chunk size
2023-05-29 11:33:46 -04:00
Devin Ivy e0d3c58113 v0.3.9 2023-05-29 00:54:16 -04:00
Foysal Ahamed 3414fcedbf Allow ignoring moderation reports by subject (#1091)
*  Allow ignoring moderation reports by subject

* 🔧 Remove unwanted package.json change

*  Remove subject format specifier

*  Port over ignore subject implementation to bsky package

* Revert " Port over ignore subject implementation to bsky package"

This reverts commit 3c782c1805548d1ebdbbd08b68b6e92ac97fdee5.

* 🚧 Port over ignore subject implementation to bsky package

*  Generate lexicons for bsky
2023-05-29 00:39:03 -04:00
devin ivy 237bb97e20 Log backgrounded error when refreshing did cache (#1110)
* Log backgrounded error when refreshing did cache

* remove branch build
2023-05-29 00:23:17 -04:00
Daniel Holmgren 693c703d78 Skip popular with friends test (#1109)
skip test
2023-05-26 18:48:35 -05:00
Daniel Holmgren ec15af5de8 Algo tweaks (#1108)
* Temporarily skip online/valid checks for feedgens while external feedgens catch-up

* tweak algos

* hook up did resolver with cache

* Re-enable whats-hot w/ materialization

* pulling a fast one on the query optimizer

* clean up

* disable branch build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-05-26 18:27:23 -05:00
devin ivy d00e3112d0 Update pool size for view maintainer (#1107)
Update pool size for view maintainer, prettier format
2023-05-26 19:19:58 -04:00
Daniel Holmgren 43e83a8681 Improving mutuals again (#1106)
* another go at it

* one more go at it

* add self to best of follows
2023-05-26 17:34:15 -05:00
Daniel Holmgren a9c70050fd Improve mutuals feed (#1105)
fix
2023-05-26 16:25:07 -05:00
Daniel Holmgren 12dc71931a Best of follows feed (#1102)
add best of follows feed
2023-05-26 15:02:36 -05:00
devin ivy 212a3a7456 Improve logging around materialized view refresh (#1104) 2023-05-26 16:02:01 -04:00