Commit Graph

2335 Commits

Author SHA1 Message Date
devin ivy bb2848e7f5 Bsky log commit error (#1275)
* don't bail on bad record index

* add build

* temporarily disable check, full reindex on rabase

* don't bail on bad record index during rebase, track last commit on rebase

* log bsky repo subscription stats

* add running and waiting count to repo sub stats

* re-enable fast path for happy rebases

* only hold onto seq in cursor consecutivelist, don't hold onto whole completed messages
2023-07-05 15:01:13 -04:00
Daniel Holmgren 4f7fd8b118 Thread through id-resolver cfg (#1282)
thread through id-resolver-cfg
2023-07-04 20:58:06 -05:00
Daniel Holmgren 0ceed96b11 Explicit dns servers (#1281)
* add ability to setup explicit dns servers

* cleanup

* fix

* reorder

* pr feedback
2023-07-04 18:40:43 -05:00
devin ivy dd4d91ce6b Log pds sequencer leader stats (#1277)
log pds sequencer leader stats
2023-07-04 00:15:13 -04:00
Daniel Holmgren 24be348dfb Better propagate errors on repo streams (#1276)
better propgate errors on repo streams
2023-07-03 20:07:20 -05:00
Daniel Holmgren 8815efd86e Move timeline construction to appview (#1274)
full switch timeline construction to appview
2023-07-03 15:03:04 -05:00
Foysal Ahamed 03200c1d8b 🐛 Only ignore reports for specific at-uri when ignoreSubject contains at-uri (#1251) 2023-07-03 15:21:16 -04:00
Daniel Holmgren c793ff9103 Add boolean for enabling generic appview proxying (#1273)
* add boolean config for enabling generic proxying

* tweak

* tweak cfg var name

* tweak
2023-07-03 14:14:49 -05:00
Daniel Holmgren 9610ba061c Require headers on getRecord proxy (#1271)
require headers on getRecord proxy
2023-07-03 13:19:56 -05:00
Daniel Holmgren 6695e6c9e4 Only pass through known params on timeline skeleton (#1270)
only pass through own params
2023-07-03 12:38:33 -05:00
Daniel Holmgren e7a0d27f1f Proxy timeline skeleton construction (#1264)
proxy timeline skeleton construction to appview
2023-07-03 11:28:39 -05:00
devin ivy 7637fdbf10 backpressure on bsky backfill indexing (#1268)
* backpressure on bsky backfill indexing

* skip image resolution for text labeler

* increase background queue concurrency for backfill

* tidy
2023-07-03 09:48:27 -04:00
Daniel Holmgren 4db7aeca57 Rework record embeds (#1262)
* wip

wip

wip

fix tests

tidy

* small fix

* fix flaky proxy test
2023-07-02 20:37:27 -05:00
Daniel Holmgren e5dc790eda Ensure not to serve repo endpoints for taken-down actor (#1263)
ensure not to serve repo endpoints for taken-down actor

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-07-02 20:22:35 -05:00
devin ivy 2e4a114379 Remove post hierarchy indexing from bsky appview (#1257)
* remove post_hierarchy from db model and indexing in bsky appview

* update bsky appview getPostThread to use recursive query to build thread

* add covering index to speed-up descendents query

* tidy post/notification processing w/o post_hierarchy

* tidy, disallow infinitely following reply cycles
2023-06-30 18:18:04 -04:00
Daniel Holmgren ee68a4037b Appview - Invalid handles (#1244)
* run with-test-db in monorepo test

* improve pg script

* tidy

* namespace bsky pg schemas

* differentiate schemas

* clean up script

* first pass at invalid handles in appview

* tests for handle invalidation

* move mock to instance instead of prototype

* change network mocks in general

* fixing pagination on actor-search

* fix snap & normalize handles on index

* handling did pagination + update tests

* one last update
2023-06-30 16:02:47 -05:00
Daniel Holmgren 8f8f2c4b73 Prevent duplicate blocks in checkout (#1256)
* prevent duplicate cids in checkout

* tidy
2023-06-29 19:33:04 -05:00
Daniel Holmgren e81ff25f02 Throw concurrent write error on prev mismatch (#1254)
* throw concurrent write error

* build branch

* remove branch build
2023-06-29 15:44:19 -05:00
Daniel Holmgren 7cb8c62b54 Repo update advisory lock (#1230)
* use for no key update

* use a tx advisory lock for repo updates

* skip tests for sqlite

* move check of commit swap on rebase

* do lock before formatting rebase

* hash schema in for lock id

* no tx lock in sqlite

* move rebase formatting to tx

* move dialect check

* rm log

* make the lock ids a bit safer

* change how we do lock id

* refactor id generator
2023-06-29 13:42:10 -05:00
Daniel Holmgren 9263ddc64d Add compression in http res (#1022)
* add compression

* compress carfiles

* update package version

* add tests

* one more test

* remove compressible dep

* add gzip to appview

* fix package.json
2023-06-29 10:24:48 -05:00
Daniel Holmgren ab6621e017 Ensure rebase recovers repos in a broken state (#1247)
rebase recovers from repos in a broken state
2023-06-29 10:24:13 -05:00
Daniel Holmgren 207d44796d Fix dev env package (#1245)
* v0.2.1

* fix dev-env publish

* v0.2.2

* fix package.json
2023-06-28 16:38:26 -05:00
Daniel Holmgren ba870d3dbe Run test DB for full monorepo tests (#1225)
* run with-test-db in monorepo test

* improve pg script

* tidy

* namespace bsky pg schemas

* differentiate schemas

* clean up script

* tweaking script
2023-06-28 15:12:37 -05:00
dholms b852d2e3b4 publish bsky 2023-06-27 19:45:43 -05:00
dholms 09385bbb55 v0.2.0 2023-06-27 19:35:26 -05:00
Daniel Holmgren 0c78bb4992 Appview - use sortAt (#1242)
* appview return sortAt as indexedAt

* rm logs
2023-06-26 20:01:18 -05:00
Daniel Holmgren e47bbed270 Fix uint8arrays version mismatch (#1243)
fix ui8 version
2023-06-26 19:20:20 -05:00
Jess Porter d9ac72e647 firehose tombstones (#1017)
* firehose tombstones

* return tombstones from sequencer requestSeqRange

* firehose tombstone test

* oops semicolons :3

* yes this can be const

* fix verifyTombstoneEvent func signature

* invalidate all sequence events upon tombstone

* don't manually purge all seqs, test for tombstone after delete

* actually fully delete account

* fix linting

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-26 18:48:00 -05:00
Daniel Holmgren 2e94c43010 Cleanup @atproto/crypto (#1218)
* remove webcrypto, upgrade @noble/curves, normalize p256 interface

* cleanup

* Test vectors for secp and p256 signature verification (#737)

Add test vectors for secp and p256 signature verification

* fix up test vectors

* add explicit test vectors for high-s signatures

* tidy json to pass verify check

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-06-26 18:47:28 -05:00
devin ivy 733325317e Miscellaneous perf improvements and bsky sync-up (#1239)
* sync-up bsky appview to apply takedowns during hydration from #1232

* apply blocks and mutes on bsky appview custom algos

* hit index on with-friends algo pagination

* more performant query for fetching invite codes with uses

* tidy

* simplify query for actor typeahead for maximal use of trigram index
2023-06-26 14:36:16 -04:00
Daniel Holmgren 0ee6dc45a5 Improve invite code counting (#1234)
* dont count admin codes for routine codes

* better tests for invites

* variable naming

* change approach

* update comment

* tidy
2023-06-24 13:07:37 -05:00
Daniel Holmgren c96777df71 Appview limit thread recursion (#1235)
add base case for thread recursion
2023-06-23 16:51:57 -05:00
Devin Ivy e522ece8f3 v0.3.13 2023-06-23 16:41:27 -04:00
Ansh 0306f81d37 Add ability to embed lists (#1188)
* add cid to ListView and ListViewBasic

* add test for mute list embed

* add support for list embeds views

* test

* port to appview

* update missing snap

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-06-23 16:30:52 -04:00
devin ivy 3da0324873 Post record language tagging, lexicon language format (#1219)
* Add languages field to post record

* helper for parsing bcp47

* add language format to lexicon

* codegen for post record langs field

* re-export language parsing in api package

* tests and tidy for lexicon language format

* index post langs, in-progress

* update snapshots, fixes record-with-media embed issue

* index post langs on bsky appview

* don't bother indexing post langs in pds appview, tidy
2023-06-23 16:23:16 -04:00
Daniel Holmgren 57b87908c2 Optimize reverse chron feeds (#1232)
* temporarily lower scroll window to 3 days

* lower feed date threshold to 1 day

* verify at least one follow

* tidy

* follow count level

* fix limi

* up low follow count

* pare down with-friends query, control join order

* do takedown checks in hydration

* move more takedown checks

* don't include placeholders for takendown replies in threads

* remove follow count logic

* bump cutoff from 1 to 3

* tidy

* add back in date threshold

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-22 18:41:51 -05:00
devin ivy e651ed67a4 Respect takedowns on feedgen records, sync routes (#1119)
* Don't serve high-level details about taken-down feedgens

* Takedown checks on sync routes

* Use simpler check for takedowns on sync routes

* ensure takendown feedgens are handled in bsky appview
2023-06-21 15:07:16 -04:00
devin ivy 23c623583a Surface sequencer leader pg client errors in the logs (#1221)
surface sequencer leader pg client errors in the logs
2023-06-20 11:12:29 -04:00
Daniel Holmgren 2768fb9e45 Make requestCrawl & notifyOfUpdate procedures (#1217)
make requestCrawl & notifyOfUpdate procedures
2023-06-20 09:36:09 -05:00
devin ivy 104799fb28 Allow optional auth for reporting on bsky appview (#1215)
temporarily accept anonymous reports on bsky appview
2023-06-19 14:00:10 -04:00
bnewbold 6a1b24f20f Remove what seems to be cruft from Lexicon types (#1202)
lexicon: remove 'infos' field from subscriptions

This is unused in any current lexicons, and not actually implemented
anywhere in code. We refactored "info" messages in to regular messages
within a union.
2023-06-19 09:33:33 -07:00
devin ivy ffc6451254 Backport: fallback to getRecord on appview, misc fixes (#1210)
* bugfixing sandbox issues

* tidy after backport

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-06-15 16:41:45 -04:00
devin ivy a3ec354dff Improve handle indexing on appview (#1208)
* Reindex handle when commit contains invalid ops, improve error handling

* test actor handling on bad commit event
2023-06-15 14:01:05 -04:00
devin ivy fc577f5dc0 Allow pds to fallback to appview for handle resolution (#1209)
Allow pds to fallback to appview for handle resolution before going to the source
2023-06-15 13:34:27 -04:00
bnewbold 5d6e6af146 nsid: update package README (no blob pattern) (#1172) 2023-06-14 09:38:59 -04:00
Daniel Holmgren 08c6bffc9f Fix epoch env (#1206)
fix epoch env
2023-06-13 18:26:03 -05:00
Daniel Holmgren 5d85188509 Invite epochs (#1205)
* add invite epoch

* reorg test

* fix dev env

* tidy
2023-06-13 17:49:47 -05:00
Daniel Holmgren 33f3a0bf12 Cleanup PDS proxy (#1197)
remove proxy file
2023-06-12 19:21:10 -05:00
Daniel Holmgren 8c4fa530d6 Pds proxy to appview based on headers (#1191)
* pds proxy to appview based on headers

* reorg appview routes forwarding mutes to appview

* tidy

* clean up ctx

* track notif state

* use notif state for notif routes

* add back in missing file

* pr feedback

* wip

* more views

* more views

* update snaps & suggestions test

* better choices for like & reposted

* note

* better block test

* Test pds-proxied mute and notif state

* Update bsky snaps, remove unnecessary #postView

* Update bsky notification tests to use updateSeen

* add getList getFeedGen tests

* Re-disconnect pds from in-process appview indexing for tests

* clean up forSnapshot

* tidy

* feed gen view tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 17:29:32 -05:00
devin ivy 84032a6c77 Bsky appview: resolve long transactions, tombstone behavior (#1192)
* Loosen indexing txn boundaries to record-level in bsky appview, no long transactions

* Add explicit HeadNotFound error to sync.getHead

* Update tombstone behavior on bsky appview, use more specific error on sync.getHead
2023-06-12 17:51:38 -04:00