64 Commits

Author SHA1 Message Date
devin ivy
53f6db7533
Test rebuilding pds timeline indexes from repos ()
* Test rebuilding pds timeline indexes from repos

* Tidy

* Sync-up with main

* Post-merge update event stream sync test
2022-12-15 21:04:09 -05:00
Paul Frazee
8a92990268
Reset password updates ()
* Log emails to stdout if no smtp mailer is set (useful for debugging)

* Fancier reset-password email template

* Update reset password flow to use a human-enterable OTP

* Tidy

* Run the email template through prettier

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-15 17:43:36 -06:00
devin ivy
809c9a38f4
Message queue blocking and graceful teardown ()
* Fix db blocking when processing from the message queue

* Make message queue concurrency test not pg-only

* Rename test

* Dedupe test

* Tidy
2022-12-15 13:54:05 -05:00
Daniel Holmgren
53d01564ed
Clean up record validation ()
clean up record validation

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-15 11:32:58 -06:00
Daniel Holmgren
7425fba702
Bad password test ()
quick test for bad pass
2022-12-15 11:02:31 -06:00
devin ivy
ed9556f049
Streaming: decouple producer side more, refactor db class ()
* Begin to decouple message queue consumers from the queue itself

* Tidy

* Reorganize pds message queue code out of db

* Decouple message queue, repo, and actor functionality from db instance w/ services

* Move repo processing into repo service

* Tidy

* Move repo blobs functionality into service

* Tidy

* Ensure to close message queue in all pds tests

* Fix typo

* Force specifying a db when using a service

* Reorg pds record plugins into record service

* Rename pds stream/ to event-stream/

* Tidy and fixes
2022-12-15 00:00:14 -05:00
Paul Frazee
f5302ce139
Fixes to post media ()
* Fix typo

* Add failing test for attaching image to a post

* Tidy failing test

* Fix

* Expand the available image resize fit modes

* Add proper validation of glob-prefixed mimes (eg image/*)

* Detect and handle blobs in post embeds

* evert "Expand the available image resize fit modes"

This reverts commit 3390fbe4d091a04f9b67a2885eae19d025747fc4.

* Use fit: inside for post media

* Update tests

* Make the thumbnail optional in external embeds

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-14 14:28:59 -06:00
Daniel Holmgren
57bec34cf5
Post media ()
* adding in banners & some lex codegen improvements for server

* fix up spacing

* couple small fixes

* client codegen & remove media embed schemas

* hook up banners

* missed a couple of lexicons

* sketching out embed schemas

* change wrapper obj property name

* setting up indexing for post media

* adding to views

* codegen client

* some tests

* Update lexicons/app/bsky/feed/getAuthorFeed.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update lexicons/app/bsky/feed/getPostThread.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update lexicons/app/bsky/feed/getTimeline.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* exhaustive check

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-12-13 13:17:53 -06:00
devin ivy
789fbb27c4
Fix size/url for profile banner () 2022-12-12 14:55:27 -05:00
devin ivy
f3eac9b2e8
Support compression in requests to xrpc-server ()
* Support compression in requests to xrpc-server

* Check payload streaming payload sizes in xrpc-server

* Tidy tests

* Move MaxSizeChecker into common stream utils
2022-12-12 12:52:14 -05:00
Daniel Holmgren
a125f4e36d
Banners ()
* adding in banners & some lex codegen improvements for server

* fix up spacing

* couple small fixes

* client codegen & remove media embed schemas

* hook up banners

* missed a couple of lexicons
2022-12-12 10:18:50 -06:00
Daniel Holmgren
91215f4f67
Verify content type of files ()
check content type of files
2022-12-08 17:00:11 -06:00
Daniel Holmgren
c74262299b
Rename upload blob NSID ()
* rename blob route

* Tweak to codegen after rebase

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2022-12-08 13:30:10 -06:00
Paul Frazee
79c3f31f9e
Add 'avatar' to the outputs of views ()
* Add sharp to the dev-env deps

* Add 'avatar' to the outputs of views
2022-12-08 11:40:19 -06:00
devin ivy
bab3b1f783
Fix issue with duped blobs () 2022-12-08 11:53:58 -05:00
Daniel Holmgren
d0b7497a08
File uploads & profile image support ()
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* wip

* improve repo processing & add blbos

* more blob processing

* work into routes & config

* testing out w profiles

* Implement initial pds image uri builder/signer/verifier

* pr fixup & tests

* streaming

* better streaming interface

* s3 interface

* fix s3 prefixes

* readable streams as xrpc inputs

* more tests + cleaning up apis

* tests for failed references

* Initial implementation of image processing server

* Update node types for node v18

* fix clone issue

* add getStream to blobstore

* fixing up tests & dev-env

* get img info on upload

* integrating img processing

* hex for img uri & fixing build errors

* improve streams

* hook up uris to getProfile

* pr feedback on tmp

* test on getprofile

* testing avatars

* fix snapshot after hmac changed

* quick cleanup

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-07 18:08:29 -06:00
devin ivy
d19fbe5473
Image processing interface ()
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* Implement initial pds image uri builder/signer/verifier

* Initial implementation of image processing server

* Update node types for node v18

* Add disk caching to pds image service

* Test pds image process caching, tidy

* Move pds image processor away from an interface

* Add sharp to pds build externals

* Move away from xrpc-server errors to http-errors for pds image server

* Tidy pds image exports

* Tidy
2022-12-07 09:44:34 -06:00
Paul Frazee
d7b4697b2d
Add terms-of-service and privacy-policy links ()
* Add privacy-policy and terms-of-service to getAccountsConfig

* Add privacy policy and tos urls to dev environment
2022-12-06 13:05:05 -06:00
devin ivy
aea590ef37
Auth in XRPC server ()
* Add auth to xrpc-server, refactor to use per-route middleware stack

* Add xrpc-server error middleware, tidy

* Tests in xrpc for for catchall errors and auth

* Tidy

* Update lex-cli for new xrpc-server auth interface

* Update pds lexicon w/ codegen for xrpc auth

* Use xrpc-server auth in pds

* Update tests for newly-required auth
2022-12-05 23:46:21 -05:00
Paul Frazee
c966185f01
Lexicon tweaks: encoding and union $type discriminators ()
* Update LexXrpcBody encoding to always be a single string with comma-separated values

* Update lexicon union-type to use  as a discriminator, and add closed flag to establish if new union members may be added

* Fix to generated code for open unions
2022-12-05 09:38:51 -06:00
Paul Frazee
63b9873bb1
Lexicon rework ()
* Implement types and validation for new NIH lexicon DSL

* Update xprc and xrpc-server to use new NIH lexicon DSL

* Simplification: dont allow lexicon definitions to be refs or ref arrays

* Add 'unknown' primitive type to NIH lexicon

* Implement client codegen for NIH Lexicon

* Implement server codegen for NIH Lexicon

* Add the 'datetime' primitive type to the NIH Lexicon

* Rework lexicons to use new NIH Lexicon DSL

* Readd tokens to code generation

* Update all code to use new NIH Lexicon (all tests passing!)

* Add imports from external files to lex-cli codegen

* Do not allow records, queries, or procedures outside of the main def

* Update packages/lex-cli/src/util.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Linter pass

* Add knownValue output to codegen

* Add comments to codegen

* Use enum and const in codegen

* Rework lexicon 'properties' structure

* Treat falsy query params as undefined

Co-authored-by: devin ivy <devinivy@gmail.com>

* Remove now-redundant param presence validation

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-12-01 12:26:18 -06:00
devin ivy
645a2ca5f7
Enforce max payload size for PLC and PDS ()
* Sketching out max payload size in xrpc-server

* Fixes and testing for external payload parsing

* Apply max json payload limit to pds and plc

* Move parsing middleware into xrpc-server w/ options
2022-11-29 13:02:03 -05:00
Paul Frazee
47c69f5469
Update getPostThread to handle missing parents () 2022-11-28 21:35:40 -06:00
Paul Frazee
ebf8e5c9ed
Add NotFoundError to getPostThread () 2022-11-28 21:33:32 -06:00
devin ivy
9d29ef96f6
Well-defined pagination in PDS ()
* Implement tuple keyset cursor to break pagination ties, apply to feeds and assertions

* Tuple keyset cursor on rest of views

* Tidy keysets for postgres

* Add default implementation to Keyset to remove some boilerplate

* Simplify Keyset interface

* Remove dead code

* Move to new Keyset interface, incl. TimeCidKeyset

* Make pds assertion view tests deterministic

* Apply pds keyset interface to user search
2022-11-28 11:51:27 -05:00
Daniel Holmgren
6fd50bd46c
Wait for servers to close at end of test suite ()
wait for servers to close
2022-11-18 15:47:03 -06:00
Daniel Holmgren
0f2562419b
Return post ancestors ()
* recursively get post ancestors

* tests
2022-11-17 17:06:44 -06:00
Daniel Holmgren
227b85608e
Deduping indexing ()
* record processor

* moving records to new processor

* plugins finished

* hook up to db

* migration

* yay tests working

* Tx migration ()

* migrations in txs

* fix tx issue

* testing

* test deduping

* test assertions

* rm types on migrations

* dont do migrations in txs, fixes pg issue
2022-11-17 14:10:37 -06:00
devin ivy
4fa2110553
Use public hostname in invite codes () 2022-11-17 12:03:17 -05:00
Daniel Holmgren
c207947bd5
Stagable blockstore ()
* stagable blockstore + remove mutable repo

* pds wip

* nearly there

* bugfix

* assertion ordering

* del old code

* save staged if no callback

* removing level

* pr fixup
2022-11-17 09:51:36 -06:00
Daniel Holmgren
43788cdc5a
Show own reposts in timeline views ()
show own reposts in timelines
2022-11-14 13:42:12 -06:00
Paul Frazee
60071e0770
Handle updateProfile when no record exists ()
* WIP handle updateProfile when no record exists

* create new profile in updateProfile

Co-authored-by: dholms <dtholmgren@gmail.com>
2022-11-11 14:10:56 -06:00
devin ivy
5b4e2b9ee3
Fix test snapshots post-merge () 2022-11-11 14:59:29 -05:00
Daniel Holmgren
2d947be9c2
Allow updating scene profiles ()
allow updating scene profiles
2022-11-11 13:36:36 -06:00
Paul Frazee
a1bee406e2
Add app.bsky.graph.getAssertions() ()
* Add app.bsky.graph.getAssertions()

* Update getAssertions to support query by author or subject

* Revert accidental change to members test
2022-11-11 12:38:25 -06:00
Daniel Holmgren
962041e6c1
Scene trending ()
* wip

* views

* trending schema

* starting message queue

* scene processor

* wip

* send mq messages from db

* db events

* undo screwing up codegen lol

* setup queue

* db migrations

* fixing up message processing

* div by 0 check

* tx issue

* queue use cursor

* update not insert

* sql bugfix + tests

* trying to linearize tests

* correclty serializing txs

* attempt update before insert

* log errors

* handle gaps in cursor

* cleanup

* oops reenable test

* correctly incr cursor
2022-11-11 12:29:48 -06:00
devin ivy
8ab87ff1f7
Standard actor declarations in app.bsky ()
* Include scene trends in feeds

* Tidy

* Tidy

* Sweep app.bsky schemas for standard actor declarations, remove reference from com.atproto

* Replace declarationCid on follow, assertion, confirmation records

* Sweep pds for standard app.bsky actor declarations

* Fix dev-env

* Tidy after merge
2022-11-10 15:46:16 -05:00
devin ivy
c52b9f0771
Include scene trends in feeds ()
* Include scene trends in feeds

* Tidy

* Tidy
2022-11-10 14:26:57 -05:00
devin ivy
1bd8ef1811
Implement setVote() method ()
* Model procedure for setting a vote on a post

* Implement app.bsky.feed.setVote() method

* Ensure setVote() no-ops when already in correct state
2022-11-09 15:24:35 -05:00
Paul Frazee
e1150465ab
Add 'member' to getProfile state output, and add some dev-env mock data ()
* Add a scene to the dev-env mock data

* Add 'member' to myState output of getProfile

* Await db writes in dev env mock data

Co-authored-by: devin ivy <devinivy@gmail.com>

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-11-09 14:09:16 -06:00
Daniel Holmgren
8b689de071
Reenable account tests + quick fix ()
reenable + quick fix
2022-11-08 16:12:11 -06:00
Daniel Holmgren
58634c47d7
Scene membership ()
* schemas

* routes

* bugfix

* tests

* test fixups

* delcaration in member views

* sort notif output
2022-11-08 14:22:16 -06:00
Daniel Holmgren
0d98291c42
Indexing rework ()
* remove unnecessary db plugin methods

* plugin types & remove declaration table

* assertion things

* wrap up things

* simplify table names
2022-11-08 12:14:15 -06:00
devin ivy
b117ae4568
Move from likes to votes ()
* Model votes in lexicon, remove likes

* Model votes in db, setup notifs, fix schema

* Update pds logic, move likes to votes

* Codegen

* Update tests for votes

* Test vote filter by direction

* Fix dev-env
2022-11-08 11:02:41 -05:00
Paul Frazee
cd1b64e600
Add app.bsky.actor.getSuggestions ()
* Fix some lexicon filenames

* Add app.bsky.actor.getSuggestions
2022-11-08 09:40:52 -06:00
Daniel Holmgren
b530a13d09
Scene creation & membership ()
* scene creation w unique handles

* wip

* user_dids -> did_handle

* invites -> assertion/confirmation

* do all the ops on scene create

* start tests + bug fixin

* user can make requests on behalf of scene

* accepting invites

* fix schema names

* get scene profiles

* return actor type in getProfile

* scene views test

* notifications test

* patch up a few db things

* rework handle & email noramlization

* codegen createscene errors
2022-11-07 14:33:33 -06:00
Paul Frazee
70c525f806
Rework post text slices to be objects () 2022-11-04 18:26:33 -05:00
devin ivy
d6a189f59a
Healthchecks for PDS and PLC ()
* Add plc healthcheck, unify plc server interface with pds

* Add pds healthcheck

* Make plc test util file consistent with other packages

* Fix types in dev-env

* Type fixes

* Tidy

* Rename healthcheck routes, place pds's under /xrpc
2022-11-04 16:26:56 -04:00
Paul Frazee
e61127111a
Split lexicons into more specific namespaces ()
* Segment lexicons into more specific namespaces and tighten up some naming conventions

* Update API to use new lexicons

* Update pds and dev-env to use new lexicons
2022-11-04 12:59:00 -05:00
Daniel Holmgren
7fdbb4dbb5
Username -> Handle ()
* schemas

* db

* atproto xrpc

* bsky xrpc

* got er done
2022-11-03 11:35:21 -05:00