1610 Commits

Author SHA1 Message Date
bnewbold
6ec452450a
pds sql: add indices for feed and post aggregations (#538)
* pds sql: add indices for feed and post aggregations

Seems like aggregations/stats/counts are likely to get changed up in the
near term future, but in the very short term, these simple index
additions yielded non-trivial speedups even at small database sizes, in
my laptop testing.

* pds migration: fix index drops in 'down'
2023-02-09 21:01:05 -05:00
Paul Frazee
d252fd3e95
Fix knownValues for reasonType lexicon (#527) 2023-02-09 20:59:32 -05:00
bnewbold
a3c1105aca
handles: reserve some top-100 handles; prevent new 2-char handle registrations (#531)
* handle: disallow new 2-char handle registration

This should not impact existing handles, only new registrations.

We discussed blocking these to prevent possible future URL path
collisions with, eg, ISO country or language codes.

* handle: reserve some more famous handles

This isn't the full list of thousands of twitter handles that have more
than 5 million or more than 1 million followers. Instead, it is only the
top 100 or so, plus a handful of additional top-50 instagram accounts (when
different), and a couple recent famous figures. Pretty subjective.

* handle: overall and segment lengths; enforce ASCII; test dashes

The 'address' package we are using to check domains doesn't seem to do
any overall length checks. The values and examples i'm testing here are
based on general domain limits. Technically a web domain could be 255,
it is email domains limited to 253 chars, but that seems like a
reasonable value to cut off at.
2023-02-09 20:58:42 -05:00
bryan newbold
7b5f8906d1 plc: run dotenv before any other imports
This fixes, eg, logging configuration via .env variables.
2023-02-09 16:40:28 -08:00
bryan newbold
8b803e013c pds: run dotenv before any other imports
This fixes logging configuration via env vars set in .env file
2023-02-09 16:40:14 -08:00
bryan newbold
d545b3195e pds: add missing 'license' to package.json 2023-02-09 16:40:06 -08:00
bryan newbold
71d6ce8c5b pds and plc: example .env files for development 2023-02-09 16:22:20 -08:00
Daniel Holmgren
7d92eb0513
Feature branch: streaming repo sync (#542)
* Scaffold xrpc-stream

* Write and test stream frames

* Write and test stream server

* Test bad stream frame parsing cases

* Proof of concept streaming on xrpc-server

* Test bad streaming endpoint

* Make frame stream to fix buffering frame bytes together

* reorg

* some lex-cli things

* better codegen for subscriptions

* scaffolding repo subscriptions

* wip

* remove repo ops

* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* basic sequencer

* some refactoring

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* subscribeRepo outbox

* some cleanup

* wip

* wip

* bugfixin

* only send msgs after tx is committed

* better handle event-emitter -> generator

* max buffer size

* cleanup

* Sync-up xrpc subscriptions with current iteration of the spec

* Allow missing encoding for subscriptions

* track sequencedAt & eventType as well

* Resolve streaming type codes, streaming tests

* Fix interface for hooking into http server for streaming, fix stream routing

* Minor reorg how streaming is enabled in xrpc-server

* Server codegen for xrpc subscriptions, supporting types in xrpc-server

* fix up buffer overloading

* Rename data frame to message frame, rename subscription lex output to message

* Move body param first for streaming frames

* Tidy

* clean up evt types

* buff up tests

* missed merge conflict

* new schema

* blobs on subscriptions

* rm genned client subscription methods

* backfill limits

* testing subscription route & quick outbox bugfix

* fix up migration

* cascade on delete

* comments & naming

* fix dev env

* delete seqs on account deletion

* tidy

* fixing things up with db notify system for schemas

* fix duplicates in outbox

* tidy

* Break out of stream handler when socket closes

* fixing up some timing issues

* tidy

* terminate ws after using

* bump up timer on async reader

* fixing up NOTIFY in txns

* pr feedback

* pr bugfixes

* make order asc explicit

* bringing tests up to speed w atpagent

* bump up max listeners on sequencer

* increase timeouts for reading from generators

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-09 16:17:06 -06:00
devin ivy
5554d80d20
Containerize PLC build (#533)
* Add plc service

* Setup dockerfile for plc

* add build-and-push GitHub Action

* Try making the tag plc:sha-

* try with the tags in the correct place

* add push to push!

* use a long sha for the container tag

* only build-and-push containers on main branch

* only build-and-push containers on main branch

---------

Co-authored-by: Jake Gold <jake@blueskyweb.xyz>
2023-02-09 16:57:42 -05:00
devin ivy
262d63d852
Add utility for testing PLC (#540)
Add script to plc for did creation
2023-02-09 15:33:28 -05:00
Paul Frazee
3dcfc047cf
Publish @atproto/api@0.1.1 (#541)
@atproto/api@0.1.1
2023-02-09 13:37:43 -06:00
Daniel Holmgren
7ac555bd41
Allow syncing a commit range (#535)
allow syncing a commit range
2023-02-09 11:27:44 -06:00
devin ivy
71d812abe6
Get profiles xrpc method (#532)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Move ArrayEl type helper into common
2023-02-09 11:11:46 -05:00
Daniel Holmgren
a1240f0a37
Handle SIGINT in postgres script (#534)
test:pg sigint
2023-02-08 15:45:13 -06:00
Paul Frazee
e8b8d081ae
Bump @atproto/xrpc@0.0.4, @atproto/api@0.1.0 (#530)
* @atproto/xrpc@0.0.4

* @atproto/api@0.1.0
2023-02-07 15:35:09 -06:00
Paul Frazee
2242e8a313
Refactor @atproto/api to the AtpAgent interface (#529)
* Refactor @atproto/api to use the simplified AtpAgent API

* xrpc package: Export the defaultFetchHandler to reuse in api

* api package: Use the defaultFetchHandler defined in xrpc

* Update all usages of the api for the new AtpAgent

* Clear promise on thrown codepath

* Avoid updating the atpagent session until ready to return
2023-02-07 15:30:29 -06:00
Devin Ivy
0c58a937c9 Fix aws package exports 2023-02-06 19:39:56 -05:00
devin ivy
b91e6e96b7
Incorporate blobs into moderation actions (#511)
* Lexicon updates for moderating blobs

* Add tracking of blob takedowns to moderation system

* Handle blobs in record takedown reversal

* Apply blob takedowns w/ image invalidation

* Test blob takedowns

* Implement cloudfront image invalidator

* Update lexicons for current moderation action on records and blobs

* Fix migration for postgres

* Enforce single current action per repo and blob, present current action on repo and blob

* Test single active mod action per repo, record, and blob

* Continue to resolve handles for moderated repos
2023-02-06 11:44:26 -05:00
Daniel Holmgren
64f5b4a022
Refactor repo indexing (#521)
* refactor repo indexing

* bit more
2023-02-03 17:11:39 -06:00
devin ivy
216e434c73
Handle client abort on image upload (#524) 2023-02-03 11:19:45 -05:00
Daniel Holmgren
55234b4b0d
Account deletion hotfix (#520)
* account deletion hotfix

* dont end pool

* set array val
2023-02-02 16:47:14 -06:00
Daniel Holmgren
396698f218
Filter out self & follows from suggestions (#498)
dont suggest self or follows
2023-02-02 16:48:41 -05:00
Daniel Holmgren
134920d872
Database notify/listen (#514)
* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* poor guy cant spell
2023-02-02 15:04:38 -06:00
Paul Frazee
817ff92c66
Bump @atproto/api@0.0.8 (#517) 2023-02-02 14:30:03 -06:00
Daniel Holmgren
773f9e304d
Account deletion (#488)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* some lexicons

reworking routes

request deletion flows

delete actor rows

migration for user-partitioned-cids

move creator to be on ipld_block

migration tests

* delete records & repos

* delete blobs

* hook it up in route

* pettier ignore email templates

* testing & bugfixes

* testing blobs & bugfixes

* pr feedback

* make deletion test more robust

* change out handle for did on account deletion

* small cleanup

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-02-02 12:34:00 -06:00
Paul Frazee
eb04cd333a
Revert the feed_thumbnail to fit:inside (#510)
* Revert the feed_thumbnail to fit:inside

* Update test snapshots for thumbnail sizing
2023-02-02 10:42:22 -06:00
dholms
dad13229c9 update pg schema name 2023-02-01 15:15:31 -06:00
dholms
62499d69ca testing backfill 2023-02-01 15:01:32 -06:00
dholms
54d51e83a1 unskip tests 2023-02-01 14:34:16 -06:00
dholms
0859119d93 Merge branch 'repo-sync-migration-reorg' into repo-sync-migration-reorg-pt2 2023-02-01 14:33:15 -06:00
dholms
572b38ae76 skipping tests 2023-02-01 14:33:02 -06:00
dholms
79dd8ed5e5 updated for merge 2023-02-01 14:23:04 -06:00
dholms
709257e83f Merge branch 'repo-sync-migration-reorg' into repo-sync-migration-reorg-pt2 2023-02-01 14:22:22 -06:00
dholms
bce8406e97 errors in storage instead of routes 2023-02-01 14:22:04 -06:00
dholms
d628718e49 no more need for catchup code 2023-02-01 14:15:20 -06:00
dholms
b7a2bd09f8 merged pt1 2023-02-01 14:11:57 -06:00
dholms
2842954662 p1 of migration 2023-02-01 14:10:08 -06:00
dholms
265a519f85 pt2 of migration 2023-02-01 14:07:48 -06:00
dholms
eb14c5fa53 couple logs & lock on final check 2023-02-01 11:15:22 -06:00
dholms
7d4a1ca14b do a check at end of migration for any new commits 2023-02-01 11:02:11 -06:00
Devin Ivy
ffabda139b Run repo-sync-data migration one user at a time, serially-batched record inserts 2023-02-01 11:32:14 -05:00
Daniel Holmgren
92c1a15d44
Sync revamp migration fix (#500)
* moved indexedAt time to record instead of ipld-block

* add in necessary ipld_block_creators

* fix test name
2023-01-31 18:53:28 -05:00
Daniel Holmgren
b455b91b0c
Moved indexedAt time to record instead of ipld-block (#501)
* moved indexedAt time to record instead of ipld-block

* use underlying kysely obj in tests

* pr feedback
2023-01-31 18:23:31 -05:00
Paul Frazee
5821a671cf
Bump @atproto/pds@0.0.2 (#507) 2023-01-31 09:01:57 -06:00
Daniel Holmgren
05855fcff3
Loosening image constraints (#503)
* loosening img constraints

* fixing up tests
2023-01-30 18:59:19 -05:00
Devin Ivy
0274c311b8 @atproto/api version 0.0.7 2023-01-30 10:46:42 -05:00
devin ivy
df74970bef
Feed thumbnail sizing, quality param bugfix (#495)
* Fix support for quality param in image urls

* Use cover sizing for feed thumbnails

* Update snapshots for image thumb urls
2023-01-27 10:19:42 -05:00
Daniel Holmgren
75e14ae102
Feature branch: storage & sync revamp (#446)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-01-26 18:09:55 -06:00
Daniel Holmgren
6f780bf670
Remove scenes (#483)
* wip

* wip

* remove getAssertions & fix up notifs test

* remove did param on updateProfile
2023-01-26 15:38:45 -06:00
Paul Frazee
ff942976ea
Include the user's handle in the reset password email (#489)
* Include the user's handle in the reset password email (close #486)

* Add test for handle in reset-code email

* Linter fix
2023-01-26 14:14:20 -06:00