2251 Commits

Author SHA1 Message Date
bryan newbold
08a316b46e gitignore: some text editor tmp files 2023-02-09 16:40:06 -08:00
bryan newbold
4fa54d6003 LICENSE: bump year range for 2023 2023-02-09 16:40:06 -08:00
bryan newbold
eedcc5b60e add a Makefile with development helpers 2023-02-09 16:40:06 -08:00
bryan newbold
71d6ce8c5b pds and plc: example .env files for development 2023-02-09 16:22:20 -08:00
Daniel Holmgren
7d92eb0513
Feature branch: streaming repo sync ()
* Scaffold xrpc-stream

* Write and test stream frames

* Write and test stream server

* Test bad stream frame parsing cases

* Proof of concept streaming on xrpc-server

* Test bad streaming endpoint

* Make frame stream to fix buffering frame bytes together

* reorg

* some lex-cli things

* better codegen for subscriptions

* scaffolding repo subscriptions

* wip

* remove repo ops

* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* basic sequencer

* some refactoring

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* subscribeRepo outbox

* some cleanup

* wip

* wip

* bugfixin

* only send msgs after tx is committed

* better handle event-emitter -> generator

* max buffer size

* cleanup

* Sync-up xrpc subscriptions with current iteration of the spec

* Allow missing encoding for subscriptions

* track sequencedAt & eventType as well

* Resolve streaming type codes, streaming tests

* Fix interface for hooking into http server for streaming, fix stream routing

* Minor reorg how streaming is enabled in xrpc-server

* Server codegen for xrpc subscriptions, supporting types in xrpc-server

* fix up buffer overloading

* Rename data frame to message frame, rename subscription lex output to message

* Move body param first for streaming frames

* Tidy

* clean up evt types

* buff up tests

* missed merge conflict

* new schema

* blobs on subscriptions

* rm genned client subscription methods

* backfill limits

* testing subscription route & quick outbox bugfix

* fix up migration

* cascade on delete

* comments & naming

* fix dev env

* delete seqs on account deletion

* tidy

* fixing things up with db notify system for schemas

* fix duplicates in outbox

* tidy

* Break out of stream handler when socket closes

* fixing up some timing issues

* tidy

* terminate ws after using

* bump up timer on async reader

* fixing up NOTIFY in txns

* pr feedback

* pr bugfixes

* make order asc explicit

* bringing tests up to speed w atpagent

* bump up max listeners on sequencer

* increase timeouts for reading from generators

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-09 16:17:06 -06:00
devin ivy
5554d80d20
Containerize PLC build ()
* Add plc service

* Setup dockerfile for plc

* add build-and-push GitHub Action

* Try making the tag plc:sha-

* try with the tags in the correct place

* add push to push!

* use a long sha for the container tag

* only build-and-push containers on main branch

* only build-and-push containers on main branch

---------

Co-authored-by: Jake Gold <jake@blueskyweb.xyz>
2023-02-09 16:57:42 -05:00
devin ivy
262d63d852
Add utility for testing PLC ()
Add script to plc for did creation
2023-02-09 15:33:28 -05:00
Paul Frazee
3dcfc047cf
Publish @atproto/api@0.1.1 ()
@atproto/api@0.1.1
2023-02-09 13:37:43 -06:00
Daniel Holmgren
7ac555bd41
Allow syncing a commit range ()
allow syncing a commit range
2023-02-09 11:27:44 -06:00
devin ivy
71d812abe6
Get profiles xrpc method ()
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Move ArrayEl type helper into common
2023-02-09 11:11:46 -05:00
Daniel Holmgren
a1240f0a37
Handle SIGINT in postgres script ()
test:pg sigint
2023-02-08 15:45:13 -06:00
Paul Frazee
e8b8d081ae
Bump @atproto/xrpc@0.0.4, @atproto/api@0.1.0 ()
* @atproto/xrpc@0.0.4

* @atproto/api@0.1.0
2023-02-07 15:35:09 -06:00
Paul Frazee
2242e8a313
Refactor @atproto/api to the AtpAgent interface ()
* Refactor @atproto/api to use the simplified AtpAgent API

* xrpc package: Export the defaultFetchHandler to reuse in api

* api package: Use the defaultFetchHandler defined in xrpc

* Update all usages of the api for the new AtpAgent

* Clear promise on thrown codepath

* Avoid updating the atpagent session until ready to return
2023-02-07 15:30:29 -06:00
Devin Ivy
0c58a937c9 Fix aws package exports 2023-02-06 19:39:56 -05:00
devin ivy
b91e6e96b7
Incorporate blobs into moderation actions ()
* Lexicon updates for moderating blobs

* Add tracking of blob takedowns to moderation system

* Handle blobs in record takedown reversal

* Apply blob takedowns w/ image invalidation

* Test blob takedowns

* Implement cloudfront image invalidator

* Update lexicons for current moderation action on records and blobs

* Fix migration for postgres

* Enforce single current action per repo and blob, present current action on repo and blob

* Test single active mod action per repo, record, and blob

* Continue to resolve handles for moderated repos
2023-02-06 11:44:26 -05:00
Daniel Holmgren
64f5b4a022
Refactor repo indexing ()
* refactor repo indexing

* bit more
2023-02-03 17:11:39 -06:00
devin ivy
216e434c73
Handle client abort on image upload () 2023-02-03 11:19:45 -05:00
Daniel Holmgren
55234b4b0d
Account deletion hotfix ()
* account deletion hotfix

* dont end pool

* set array val
2023-02-02 16:47:14 -06:00
Daniel Holmgren
396698f218
Filter out self & follows from suggestions ()
dont suggest self or follows
2023-02-02 16:48:41 -05:00
Daniel Holmgren
134920d872
Database notify/listen ()
* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* poor guy cant spell
2023-02-02 15:04:38 -06:00
Paul Frazee
817ff92c66
Bump @atproto/api@0.0.8 () 2023-02-02 14:30:03 -06:00
Daniel Holmgren
773f9e304d
Account deletion ()
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts ()

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts ()

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* some lexicons

reworking routes

request deletion flows

delete actor rows

migration for user-partitioned-cids

move creator to be on ipld_block

migration tests

* delete records & repos

* delete blobs

* hook it up in route

* pettier ignore email templates

* testing & bugfixes

* testing blobs & bugfixes

* pr feedback

* make deletion test more robust

* change out handle for did on account deletion

* small cleanup

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-02-02 12:34:00 -06:00
Paul Frazee
eb04cd333a
Revert the feed_thumbnail to fit:inside ()
* Revert the feed_thumbnail to fit:inside

* Update test snapshots for thumbnail sizing
2023-02-02 10:42:22 -06:00
dholms
dad13229c9 update pg schema name 2023-02-01 15:15:31 -06:00
dholms
62499d69ca testing backfill 2023-02-01 15:01:32 -06:00
dholms
54d51e83a1 unskip tests 2023-02-01 14:34:16 -06:00
dholms
0859119d93 Merge branch 'repo-sync-migration-reorg' into repo-sync-migration-reorg-pt2 2023-02-01 14:33:15 -06:00
dholms
572b38ae76 skipping tests 2023-02-01 14:33:02 -06:00
dholms
79dd8ed5e5 updated for merge 2023-02-01 14:23:04 -06:00
dholms
709257e83f Merge branch 'repo-sync-migration-reorg' into repo-sync-migration-reorg-pt2 2023-02-01 14:22:22 -06:00
dholms
bce8406e97 errors in storage instead of routes 2023-02-01 14:22:04 -06:00
dholms
d628718e49 no more need for catchup code 2023-02-01 14:15:20 -06:00
dholms
b7a2bd09f8 merged pt1 2023-02-01 14:11:57 -06:00
dholms
2842954662 p1 of migration 2023-02-01 14:10:08 -06:00
dholms
265a519f85 pt2 of migration 2023-02-01 14:07:48 -06:00
dholms
eb14c5fa53 couple logs & lock on final check 2023-02-01 11:15:22 -06:00
dholms
7d4a1ca14b do a check at end of migration for any new commits 2023-02-01 11:02:11 -06:00
Devin Ivy
ffabda139b Run repo-sync-data migration one user at a time, serially-batched record inserts 2023-02-01 11:32:14 -05:00
Daniel Holmgren
92c1a15d44
Sync revamp migration fix ()
* moved indexedAt time to record instead of ipld-block

* add in necessary ipld_block_creators

* fix test name
2023-01-31 18:53:28 -05:00
Daniel Holmgren
b455b91b0c
Moved indexedAt time to record instead of ipld-block ()
* moved indexedAt time to record instead of ipld-block

* use underlying kysely obj in tests

* pr feedback
2023-01-31 18:23:31 -05:00
Paul Frazee
5821a671cf
Bump @atproto/pds@0.0.2 () 2023-01-31 09:01:57 -06:00
Daniel Holmgren
05855fcff3
Loosening image constraints ()
* loosening img constraints

* fixing up tests
2023-01-30 18:59:19 -05:00
Devin Ivy
0274c311b8 @atproto/api version 0.0.7 2023-01-30 10:46:42 -05:00
devin ivy
df74970bef
Feed thumbnail sizing, quality param bugfix ()
* Fix support for quality param in image urls

* Use cover sizing for feed thumbnails

* Update snapshots for image thumb urls
2023-01-27 10:19:42 -05:00
Daniel Holmgren
75e14ae102
Feature branch: storage & sync revamp ()
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts ()

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts ()

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-01-26 18:09:55 -06:00
Daniel Holmgren
6f780bf670
Remove scenes ()
* wip

* wip

* remove getAssertions & fix up notifs test

* remove did param on updateProfile
2023-01-26 15:38:45 -06:00
Paul Frazee
ff942976ea
Include the user's handle in the reset password email ()
* Include the user's handle in the reset password email (close )

* Add test for handle in reset-code email

* Linter fix
2023-01-26 14:14:20 -06:00
Paul Frazee
49d31f0a2c
Add missing 'avatar' field in app.bsky.graph.getFollows () 2023-01-26 14:14:00 -06:00
devin ivy
73de6d8668
Add some reporting mocks to dev-env () 2023-01-26 13:09:08 -05:00
devin ivy
9834c2f5ef
Allow login using email ()
* Allow login using email, deprecate handle input in session.create

* Remove handle input from session.create lexicon
2023-01-26 13:08:56 -05:00