Commit Graph

1347 Commits

Author SHA1 Message Date
bryan newbold 71d6ce8c5b pds and plc: example .env files for development 2023-02-09 16:22:20 -08:00
Daniel Holmgren 7d92eb0513 Feature branch: streaming repo sync (#542)
* Scaffold xrpc-stream

* Write and test stream frames

* Write and test stream server

* Test bad stream frame parsing cases

* Proof of concept streaming on xrpc-server

* Test bad streaming endpoint

* Make frame stream to fix buffering frame bytes together

* reorg

* some lex-cli things

* better codegen for subscriptions

* scaffolding repo subscriptions

* wip

* remove repo ops

* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* basic sequencer

* some refactoring

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* subscribeRepo outbox

* some cleanup

* wip

* wip

* bugfixin

* only send msgs after tx is committed

* better handle event-emitter -> generator

* max buffer size

* cleanup

* Sync-up xrpc subscriptions with current iteration of the spec

* Allow missing encoding for subscriptions

* track sequencedAt & eventType as well

* Resolve streaming type codes, streaming tests

* Fix interface for hooking into http server for streaming, fix stream routing

* Minor reorg how streaming is enabled in xrpc-server

* Server codegen for xrpc subscriptions, supporting types in xrpc-server

* fix up buffer overloading

* Rename data frame to message frame, rename subscription lex output to message

* Move body param first for streaming frames

* Tidy

* clean up evt types

* buff up tests

* missed merge conflict

* new schema

* blobs on subscriptions

* rm genned client subscription methods

* backfill limits

* testing subscription route & quick outbox bugfix

* fix up migration

* cascade on delete

* comments & naming

* fix dev env

* delete seqs on account deletion

* tidy

* fixing things up with db notify system for schemas

* fix duplicates in outbox

* tidy

* Break out of stream handler when socket closes

* fixing up some timing issues

* tidy

* terminate ws after using

* bump up timer on async reader

* fixing up NOTIFY in txns

* pr feedback

* pr bugfixes

* make order asc explicit

* bringing tests up to speed w atpagent

* bump up max listeners on sequencer

* increase timeouts for reading from generators

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-09 16:17:06 -06:00
Daniel Holmgren 7ac555bd41 Allow syncing a commit range (#535)
allow syncing a commit range
2023-02-09 11:27:44 -06:00
devin ivy 71d812abe6 Get profiles xrpc method (#532)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Move ArrayEl type helper into common
2023-02-09 11:11:46 -05:00
Paul Frazee 2242e8a313 Refactor @atproto/api to the AtpAgent interface (#529)
* Refactor @atproto/api to use the simplified AtpAgent API

* xrpc package: Export the defaultFetchHandler to reuse in api

* api package: Use the defaultFetchHandler defined in xrpc

* Update all usages of the api for the new AtpAgent

* Clear promise on thrown codepath

* Avoid updating the atpagent session until ready to return
2023-02-07 15:30:29 -06:00
devin ivy b91e6e96b7 Incorporate blobs into moderation actions (#511)
* Lexicon updates for moderating blobs

* Add tracking of blob takedowns to moderation system

* Handle blobs in record takedown reversal

* Apply blob takedowns w/ image invalidation

* Test blob takedowns

* Implement cloudfront image invalidator

* Update lexicons for current moderation action on records and blobs

* Fix migration for postgres

* Enforce single current action per repo and blob, present current action on repo and blob

* Test single active mod action per repo, record, and blob

* Continue to resolve handles for moderated repos
2023-02-06 11:44:26 -05:00
Daniel Holmgren 64f5b4a022 Refactor repo indexing (#521)
* refactor repo indexing

* bit more
2023-02-03 17:11:39 -06:00
devin ivy 216e434c73 Handle client abort on image upload (#524) 2023-02-03 11:19:45 -05:00
Daniel Holmgren 55234b4b0d Account deletion hotfix (#520)
* account deletion hotfix

* dont end pool

* set array val
2023-02-02 16:47:14 -06:00
Daniel Holmgren 396698f218 Filter out self & follows from suggestions (#498)
dont suggest self or follows
2023-02-02 16:48:41 -05:00
Daniel Holmgren 134920d872 Database notify/listen (#514)
* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* poor guy cant spell
2023-02-02 15:04:38 -06:00
Daniel Holmgren 773f9e304d Account deletion (#488)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* some lexicons

reworking routes

request deletion flows

delete actor rows

migration for user-partitioned-cids

move creator to be on ipld_block

migration tests

* delete records & repos

* delete blobs

* hook it up in route

* pettier ignore email templates

* testing & bugfixes

* testing blobs & bugfixes

* pr feedback

* make deletion test more robust

* change out handle for did on account deletion

* small cleanup

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-02-02 12:34:00 -06:00
Paul Frazee eb04cd333a Revert the feed_thumbnail to fit:inside (#510)
* Revert the feed_thumbnail to fit:inside

* Update test snapshots for thumbnail sizing
2023-02-02 10:42:22 -06:00
dholms dad13229c9 update pg schema name 2023-02-01 15:15:31 -06:00
dholms 62499d69ca testing backfill 2023-02-01 15:01:32 -06:00
dholms 54d51e83a1 unskip tests 2023-02-01 14:34:16 -06:00
dholms 0859119d93 Merge branch 'repo-sync-migration-reorg' into repo-sync-migration-reorg-pt2 2023-02-01 14:33:15 -06:00
dholms 572b38ae76 skipping tests 2023-02-01 14:33:02 -06:00
dholms 79dd8ed5e5 updated for merge 2023-02-01 14:23:04 -06:00
dholms 709257e83f Merge branch 'repo-sync-migration-reorg' into repo-sync-migration-reorg-pt2 2023-02-01 14:22:22 -06:00
dholms bce8406e97 errors in storage instead of routes 2023-02-01 14:22:04 -06:00
dholms d628718e49 no more need for catchup code 2023-02-01 14:15:20 -06:00
dholms b7a2bd09f8 merged pt1 2023-02-01 14:11:57 -06:00
dholms 2842954662 p1 of migration 2023-02-01 14:10:08 -06:00
dholms 265a519f85 pt2 of migration 2023-02-01 14:07:48 -06:00
dholms eb14c5fa53 couple logs & lock on final check 2023-02-01 11:15:22 -06:00
dholms 7d4a1ca14b do a check at end of migration for any new commits 2023-02-01 11:02:11 -06:00
Devin Ivy ffabda139b Run repo-sync-data migration one user at a time, serially-batched record inserts 2023-02-01 11:32:14 -05:00
Daniel Holmgren 92c1a15d44 Sync revamp migration fix (#500)
* moved indexedAt time to record instead of ipld-block

* add in necessary ipld_block_creators

* fix test name
2023-01-31 18:53:28 -05:00
Daniel Holmgren b455b91b0c Moved indexedAt time to record instead of ipld-block (#501)
* moved indexedAt time to record instead of ipld-block

* use underlying kysely obj in tests

* pr feedback
2023-01-31 18:23:31 -05:00
Paul Frazee 5821a671cf Bump @atproto/pds@0.0.2 (#507) 2023-01-31 09:01:57 -06:00
Daniel Holmgren 05855fcff3 Loosening image constraints (#503)
* loosening img constraints

* fixing up tests
2023-01-30 18:59:19 -05:00
devin ivy df74970bef Feed thumbnail sizing, quality param bugfix (#495)
* Fix support for quality param in image urls

* Use cover sizing for feed thumbnails

* Update snapshots for image thumb urls
2023-01-27 10:19:42 -05:00
Daniel Holmgren 75e14ae102 Feature branch: storage & sync revamp (#446)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-01-26 18:09:55 -06:00
Daniel Holmgren 6f780bf670 Remove scenes (#483)
* wip

* wip

* remove getAssertions & fix up notifs test

* remove did param on updateProfile
2023-01-26 15:38:45 -06:00
Paul Frazee ff942976ea Include the user's handle in the reset password email (#489)
* Include the user's handle in the reset password email (close #486)

* Add test for handle in reset-code email

* Linter fix
2023-01-26 14:14:20 -06:00
Paul Frazee 49d31f0a2c Add missing 'avatar' field in app.bsky.graph.getFollows (#494) 2023-01-26 14:14:00 -06:00
devin ivy 9834c2f5ef Allow login using email (#492)
* Allow login using email, deprecate handle input in session.create

* Remove handle input from session.create lexicon
2023-01-26 13:08:56 -05:00
Daniel Holmgren e8c218b5c9 Sort "suggested follows" by number of posts (#477)
* return suggestions by post count

* pr feedback

* fix up PG pagination issue
2023-01-24 14:03:51 -06:00
Paul Frazee e44c98e717 Update module publish scripts (#478)
* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts
2023-01-24 10:11:20 -06:00
Daniel Holmgren 292f41c807 Fix migration name (#475)
fix migration name
2023-01-19 18:07:24 -06:00
devin ivy 13dd58bb15 Feature: admin moderation views (#472)
* Model mod actions for actor takedowns

* Add administration lex schemas for moderation actions

* Initial implementation of takedown mod action and reversal

* Refactor admin actions into a service

* Check for account takedown when creating new session or refreshing session

* Sweep usage of getUser() and getUserByEmail() for takedowns

* Begin sweep for account takedowns

* Tidy takedown checks

* Handle post author takedowns

* Fix tests and migrations for mod takedown

* Fix xrpc-server delayed read of blob payloads

* Test takedown mod action, misc fixes

* Remodel lex schemas for admin moderation actions

* Remove delcaration cid from actor takedowns, lex schema tidy

* Add notes

* Move moderation methods to com.atproto, based around repo rather than actor

* Moderation reports of repos and records

* Implement moderation report resolution method

* Tidy

* Place moderation-init migration after mute-init

* Allow services to reference each other

* Model record takedowns in db

* Lex schemas for record takedowns

* Service/methods for record takedowns, fix report view subject

* Apply record takedown to posts in feeds

* Test record takedown effects, fix reversal

* Support server moderation flag and acknowledge actions, fix action/report resolution for records

* Track repo takedowns on repo_root rather than did_handle

* Move repo/record reporting under com.atproto.report

* Colocate all moderation functionality into moderation service, combine from admin and repo services

* Tidy

* Tidy

* Unify mod report and action methods with new repo/record ref schemas

* Schemas for admin moderation views

* Tidy/codegen

* Fix test snapshot

* Break pds admin methods into separate files

* Implment admin repo search method

* Reorganize existing moderation views into the ModerationViews

* Lex schemas for admin get repo/record, implement get repo

* Implement admin get record view

* Tidy/codegen

* Tidy, fix snapshot

* Tidy, swap filenames

* Implement admin get moderation action view

* Implement admin get moderation report view

* Implement and test admin views for mod actions and reports

* Dry-up some admin and moderation-related seeds in the tests

* Fix xrpc fetch usage for newly required duplex option

* Admin view of repo and record details (#463)

* Reorganize existing moderation views into the ModerationViews

* Lex schemas for admin get repo/record, implement get repo

* Implement admin get record view

* Tidy/codegen

* Tidy, fix snapshot
2023-01-18 18:01:04 -05:00
Dimitris Apostolou 422008be3e Fix typos (#467) 2023-01-09 21:34:07 -05:00
devin ivy 1007fab49a Fix out-of-date test snapshots (#464)
Fix out-of-date snapshots
2023-01-06 17:37:56 -05:00
devin ivy ef487900f6 Server moderation action/resolution and reporting (#458)
* Model mod actions for actor takedowns

* Add administration lex schemas for moderation actions

* Initial implementation of takedown mod action and reversal

* Refactor admin actions into a service

* Check for account takedown when creating new session or refreshing session

* Sweep usage of getUser() and getUserByEmail() for takedowns

* Begin sweep for account takedowns

* Tidy takedown checks

* Handle post author takedowns

* Fix tests and migrations for mod takedown

* Fix xrpc-server delayed read of blob payloads

* Test takedown mod action, misc fixes

* Remodel lex schemas for admin moderation actions

* Remove delcaration cid from actor takedowns, lex schema tidy

* Add notes

* Move moderation methods to com.atproto, based around repo rather than actor

* Moderation reports of repos and records

* Implement moderation report resolution method

* Tidy

* Place moderation-init migration after mute-init

* Allow services to reference each other

* Model record takedowns in db

* Lex schemas for record takedowns

* Service/methods for record takedowns, fix report view subject

* Apply record takedown to posts in feeds

* Test record takedown effects, fix reversal

* Support server moderation flag and acknowledge actions, fix action/report resolution for records

* Track repo takedowns on repo_root rather than did_handle

* Move repo/record reporting under com.atproto.report

* Colocate all moderation functionality into moderation service, combine from admin and repo services

* Tidy

* Tidy

* Unify mod report and action methods with new repo/record ref schemas
2023-01-06 12:59:47 -05:00
devin ivy e9f5f23edc Fix lexicon validation, do not allow undefined when required (#457)
* Fix lexicon validation, do not allow undefined when required

* Make term optional for actor search/typeahead
2023-01-05 12:04:31 -05:00
Devin Ivy 9d32cf130e Fix resolution of server did in com.atproto.handle.resolve 2023-01-04 17:39:30 -05:00
Paul Frazee 2ed0c31254 Datetime validation fix (#454)
* Fix datetime validation (close #402)

* Fix lexicon publishing flow

* lexicon@0.0.3

* lexicon@0.0.4

* Fix xrpc publishing scripts

* xrpc@0.0.3

* api@0.0.6

* Update thread test snapshot
2023-01-03 16:37:31 -06:00
Daniel Holmgren 2602c00e85 Remove "bksy" (#453)
done once & for all
2023-01-02 21:40:56 -06:00
Paul Frazee 3f69270dfe API & client improvements (#451)
* Add typeguard helpers to codegen

* Add option to disable validation on XRPC responses

* Add xrpc client-side validation of responses

* Add general-purpose validate() method to Lexicons

* Add validation helpers to lex-cli codegen
2023-01-02 17:38:30 -06:00