1610 Commits

Author SHA1 Message Date
Daniel Holmgren
0c78bb4992
Appview - use sortAt (#1242)
* appview return sortAt as indexedAt

* rm logs
2023-06-26 20:01:18 -05:00
Daniel Holmgren
e47bbed270
Fix uint8arrays version mismatch (#1243)
fix ui8 version
2023-06-26 19:20:20 -05:00
Jess Porter
d9ac72e647
firehose tombstones (#1017)
* firehose tombstones

* return tombstones from sequencer requestSeqRange

* firehose tombstone test

* oops semicolons :3

* yes this can be const

* fix verifyTombstoneEvent func signature

* invalidate all sequence events upon tombstone

* don't manually purge all seqs, test for tombstone after delete

* actually fully delete account

* fix linting

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-26 18:48:00 -05:00
Daniel Holmgren
2e94c43010
Cleanup @atproto/crypto (#1218)
* remove webcrypto, upgrade @noble/curves, normalize p256 interface

* cleanup

* Test vectors for secp and p256 signature verification (#737)

Add test vectors for secp and p256 signature verification

* fix up test vectors

* add explicit test vectors for high-s signatures

* tidy json to pass verify check

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-06-26 18:47:28 -05:00
devin ivy
733325317e
Miscellaneous perf improvements and bsky sync-up (#1239)
* sync-up bsky appview to apply takedowns during hydration from #1232

* apply blocks and mutes on bsky appview custom algos

* hit index on with-friends algo pagination

* more performant query for fetching invite codes with uses

* tidy

* simplify query for actor typeahead for maximal use of trigram index
2023-06-26 14:36:16 -04:00
Daniel Holmgren
0ee6dc45a5
Improve invite code counting (#1234)
* dont count admin codes for routine codes

* better tests for invites

* variable naming

* change approach

* update comment

* tidy
2023-06-24 13:07:37 -05:00
Daniel Holmgren
c96777df71
Appview limit thread recursion (#1235)
add base case for thread recursion
2023-06-23 16:51:57 -05:00
Devin Ivy
e522ece8f3 v0.3.13 2023-06-23 16:41:27 -04:00
Ansh
0306f81d37
Add ability to embed lists (#1188)
* add cid to ListView and ListViewBasic

* add test for mute list embed

* add support for list embeds views

* test

* port to appview

* update missing snap

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-06-23 16:30:52 -04:00
devin ivy
3da0324873
Post record language tagging, lexicon language format (#1219)
* Add languages field to post record

* helper for parsing bcp47

* add language format to lexicon

* codegen for post record langs field

* re-export language parsing in api package

* tests and tidy for lexicon language format

* index post langs, in-progress

* update snapshots, fixes record-with-media embed issue

* index post langs on bsky appview

* don't bother indexing post langs in pds appview, tidy
2023-06-23 16:23:16 -04:00
Daniel Holmgren
57b87908c2
Optimize reverse chron feeds (#1232)
* temporarily lower scroll window to 3 days

* lower feed date threshold to 1 day

* verify at least one follow

* tidy

* follow count level

* fix limi

* up low follow count

* pare down with-friends query, control join order

* do takedown checks in hydration

* move more takedown checks

* don't include placeholders for takendown replies in threads

* remove follow count logic

* bump cutoff from 1 to 3

* tidy

* add back in date threshold

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-22 18:41:51 -05:00
devin ivy
e651ed67a4
Respect takedowns on feedgen records, sync routes (#1119)
* Don't serve high-level details about taken-down feedgens

* Takedown checks on sync routes

* Use simpler check for takedowns on sync routes

* ensure takendown feedgens are handled in bsky appview
2023-06-21 15:07:16 -04:00
devin ivy
23c623583a
Surface sequencer leader pg client errors in the logs (#1221)
surface sequencer leader pg client errors in the logs
2023-06-20 11:12:29 -04:00
Daniel Holmgren
2768fb9e45
Make requestCrawl & notifyOfUpdate procedures (#1217)
make requestCrawl & notifyOfUpdate procedures
2023-06-20 09:36:09 -05:00
devin ivy
104799fb28
Allow optional auth for reporting on bsky appview (#1215)
temporarily accept anonymous reports on bsky appview
2023-06-19 14:00:10 -04:00
bnewbold
6a1b24f20f
Remove what seems to be cruft from Lexicon types (#1202)
lexicon: remove 'infos' field from subscriptions

This is unused in any current lexicons, and not actually implemented
anywhere in code. We refactored "info" messages in to regular messages
within a union.
2023-06-19 09:33:33 -07:00
devin ivy
ffc6451254
Backport: fallback to getRecord on appview, misc fixes (#1210)
* bugfixing sandbox issues

* tidy after backport

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-06-15 16:41:45 -04:00
devin ivy
a3ec354dff
Improve handle indexing on appview (#1208)
* Reindex handle when commit contains invalid ops, improve error handling

* test actor handling on bad commit event
2023-06-15 14:01:05 -04:00
devin ivy
fc577f5dc0
Allow pds to fallback to appview for handle resolution (#1209)
Allow pds to fallback to appview for handle resolution before going to the source
2023-06-15 13:34:27 -04:00
bnewbold
5d6e6af146
nsid: update package README (no blob pattern) (#1172) 2023-06-14 09:38:59 -04:00
Daniel Holmgren
08c6bffc9f
Fix epoch env (#1206)
fix epoch env
2023-06-13 18:26:03 -05:00
Daniel Holmgren
5d85188509
Invite epochs (#1205)
* add invite epoch

* reorg test

* fix dev env

* tidy
2023-06-13 17:49:47 -05:00
Daniel Holmgren
33f3a0bf12
Cleanup PDS proxy (#1197)
remove proxy file
2023-06-12 19:21:10 -05:00
Daniel Holmgren
8c4fa530d6
Pds proxy to appview based on headers (#1191)
* pds proxy to appview based on headers

* reorg appview routes forwarding mutes to appview

* tidy

* clean up ctx

* track notif state

* use notif state for notif routes

* add back in missing file

* pr feedback

* wip

* more views

* more views

* update snaps & suggestions test

* better choices for like & reposted

* note

* better block test

* Test pds-proxied mute and notif state

* Update bsky snaps, remove unnecessary #postView

* Update bsky notification tests to use updateSeen

* add getList getFeedGen tests

* Re-disconnect pds from in-process appview indexing for tests

* clean up forSnapshot

* tidy

* feed gen view tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 17:29:32 -05:00
devin ivy
84032a6c77
Bsky appview: resolve long transactions, tombstone behavior (#1192)
* Loosen indexing txn boundaries to record-level in bsky appview, no long transactions

* Add explicit HeadNotFound error to sync.getHead

* Update tombstone behavior on bsky appview, use more specific error on sync.getHead
2023-06-12 17:51:38 -04:00
devin ivy
0ed0eb5d5f
Remove appview repo subscription from PDS (#1195)
* Remove repo subscription from pds appview

* Remove appview repo subscription config from pds
2023-06-12 17:01:59 -04:00
Daniel Holmgren
7f9d2c1ebe
Notify crawlers on write (#1194)
* notify crawlers on write

* pr feedback

* more resilient config
2023-06-12 15:00:17 -05:00
Daniel Holmgren
08fbb33fdf
Appview blocks (#1187)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* hook up indexing

* Setup custom feeds on bsky

* graph service + getList routes

* migration

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* Test custom algos on bsky

* add mute state to snapshots

* table & indexing plugin

* getBlocks

* bail out of mute subquery when no viewer

* fleshing out services

* Test pds getFeed proxy

* tidy, fix build

* tidy

* Fix minor race in bsky mutelist tests

* blocks in feed views

* db migration, impl block logic & tests

* Update pds proxy snaps based on appview mutes

* missed some files on last commit

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

* apply blocks in getFeed

* update snaps

* upadate one last snap

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 11:05:22 -05:00
devin ivy
5951d9347f
Misc. syncing-up bsky appview with on-pds appview (pt. 2) (#1190)
* Misc notification and optimization in bsky, tidy

* Handle notifs on message queue in bsky appview

* Sync-up profile labeling behavior in bsky

* No async work during actor unindexing in bsky, include additional indexes

* Prevent deeply nested notifs w/ hellthread exception in bsky appview

* Add indexes to bsky appview for more efficient actor deletion/unindexing

* Update pds proxy snaps for updated bsky profile labeling

* Return custom suggestions in bsky appview

* Test out-of-order indexing of aggregations on bsky appview

* Proxy pds getPopular to hot-classic feed on appview. Proxy getPopularFeedGenerators to appview.

* Make did field required on feedgen view
2023-06-11 15:48:20 -04:00
devin ivy
7981c5948f
Misc. syncing-up bsky appview with on-pds appview (#1189)
* Misc notification and optimization in bsky, tidy

* Handle notifs on message queue in bsky appview

* Sync-up profile labeling behavior in bsky

* No async work during actor unindexing in bsky, include additional indexes

* Prevent deeply nested notifs w/ hellthread exception in bsky appview

* Add indexes to bsky appview for more efficient actor deletion/unindexing

* Update pds proxy snaps for updated bsky profile labeling

* Return custom suggestions in bsky appview
2023-06-11 14:55:43 -04:00
Daniel Holmgren
4f2f3d8d17
Appview mutes (#1184)
* hook up indexing

* graph service + getList routes

* migration

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* add mute state to snapshots

* bail out of mute subquery when no viewer

* Fix minor race in bsky mutelist tests

* Update pds proxy snaps based on appview mutes

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-10 17:12:15 -04:00
devin ivy
283fb1d8d6
Custom feeds on bsky appview (#1185)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* Setup custom feeds on bsky

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* Test custom algos on bsky

* Test pds getFeed proxy

* tidy, fix build

* tidy
2023-06-09 11:48:17 -04:00
Daniel Holmgren
9a873148bd
Appview lists & listmutes (#1183)
* hook up indexing

* graph service + getList routes

* migration

* mute lists

* add getListMutes

* mute list tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-09 10:46:03 -04:00
devin ivy
f46d95617d
Feed gen indexing and view endpoints on bsky appview (#1182)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* tidy, fix build

* tidy/fix bsky migration
2023-06-09 10:30:54 -04:00
Devin Ivy
1033eddc00 v0.3.12 2023-06-09 09:28:32 -04:00
Foysal Ahamed
8857fb0845
Filter reports by reporter did (#1165)
*  Hydrate subject author's repo in moderation reports

*  Allow filtering moderation reports by reporter DIDs

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package

*  Port over filter by reporter implementation to bsky package

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

* Simplify query builder

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-06-09 09:21:09 -04:00
devin ivy
98fdd5c376
Materialize post and profile aggregations on bsky appview (#1177)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky
2023-06-08 13:18:30 -04:00
Devin Ivy
5eb777c3f2 v0.3.11 2023-06-08 10:32:52 -04:00
Foysal Ahamed
9a2dc7119e
Hydrate subject author's repo in moderation reports (#1135)
*  Hydrate subject author's repo in moderation reports

* 🚧 Expose only handle of the subject repo on moderation reports

* 🚧 Regenerate lexicons off main

* 🧹 Cleanup unnecessary joins

*  Updating snapshots

*  Make handle optional to allow serving reports even if handle doesn't exist

*  Port over subjectHandle implementation to bsky package
2023-06-08 10:17:52 -04:00
dholms
85e3cdaf7b publish identity@0.0.1 2023-06-07 18:31:01 -05:00
Daniel Holmgren
35e035dad2
Disable resolution of handles through xrpc (#1179)
* disable resolution of handles through xrpc

* tweak admin.updateHandle
2023-06-07 16:00:52 -05:00
devin ivy
dd38c8b3b5
Normalize creation times on list, list item, feedgen records (#1178)
Normalize creation times on lists, list items, feedgens
2023-06-07 16:58:15 -04:00
devin ivy
ff78a30b9e
Ensure keyset pagination makes use of indexes on timeline (#1174)
Alternative pagination query snippet for better use of indexes
2023-06-07 11:46:33 -04:00
devin ivy
a23fb1eac3
Fix lists pagination, ensure paginating on correct column (#1175)
Ensure lists pagination uses createdAt both in query and for building cursor
2023-06-07 11:41:02 -04:00
Daniel Holmgren
6fb9a8faea
Fix dropped events in sync.subscribeRepos (#1167)
* wip

* first pass done

* chnage around how channels function

* bugfixing

* fix up migration & tests

* fix dev-env

* add prep migration for stream fixing

* application code changes to use new repo_event table

* dont send evts

* add outgoing_repo_seq table + sqlite migrations

* patch up down migration

* patching up a couple things

* update test

* patchup

* re-enable subscription route

* add outgoingSeq column

* updating

* accidental add

* temporarily disable stream

* resolve merge conflict

* add migration

* migration for seq

* fix test

* id -> seq

* get schema lock ids

* tidy

* fix server test

* fix up

* use bsky uniqueLockId approach

* tidy

* gh actions build

* build branch

* build branch

* tweak test

* rm constraint rename

* small bugfix

* check for seq not null

* fix up issues

* one more small change

* tidy

* no build branches
2023-06-06 19:18:44 -05:00
Foysal Ahamed
46e1f051e1
Allow searching repo by DID (#1084)
*  Allow searching repo by DID

*  Port over did search from pds

*  Add tests for bsky implementation of admin searchRepos

* 🧹 Cleanup according to PR review
2023-06-06 00:12:14 -04:00
bnewbold
a4a4af1693
all packages: prettier applies to tests/ in addition to src/ (#1170)
* pds: prettier:fix (on a JSON file)

* package.json (multiple): run prettier on tests/

* bsky: prettier:fix (on a JSON file)
2023-06-05 19:26:32 -07:00
devin ivy
c20ed66f2a
Optimization for label subqueries on feeds (#1164)
* temp disable hot-classic

* Switch labels in whats-hot to values list

* Switch labels in getPopular (legacy hot-classic) to values list

* Reenable hot-classic, switch labels to values list

* remove branch build in workflow
2023-06-03 14:10:57 -04:00
Daniel Holmgren
bf4ac79cef
Fix up what's hot (#1133)
* do whats hot off of candidate feed

* tidy
2023-06-01 21:26:40 -05:00
devin ivy
53a107b237
Fix pagination limit when not respected by feedgens (#1129)
* Enforce limit param in the case that the feedgen does not

* Tidy
2023-06-01 17:36:22 -04:00