Commit Graph

2335 Commits

Author SHA1 Message Date
devin ivy b686b3baff Create PDS with ServerConfig rather than ServerConfigValues (#435)
Create pds with ServerConfig rather than ServerConfigValues
2022-12-21 13:32:10 -05:00
Daniel Holmgren 30ab0d341b Refactor feeds & post threads (#423)
* factor out feed items & mystate

* propose feed items rewordk

* pr feedback

* revamping getTimeline

* feed service & cleanup

* bugfixin

* update timeline snapshots

* update getAuthFeed snapshot

* bugfixin

* revamp getPostThread

* fix up sync & votes tests

* bug in post thread

* fix replyCount

* fix ordering

* move uriImgBuilder to createServices
2022-12-20 20:38:37 -06:00
Daniel Holmgren f0ee51f151 App context (#424)
* introducing appcontext & pds class

* revamp rest of server + auth

* revamp tests

* add to dev-env

* fix api test

* add start method to pds

* hook up terminator before

* PLC AppContext (#425)

* convert plc server to app context

* add start method to plc

* hooke up terminator on server setup

* fix test

* cleanup & pr feedback

* fix up one more test
2022-12-16 11:23:36 -06:00
devin ivy 53f6db7533 Test rebuilding pds timeline indexes from repos (#413)
* Test rebuilding pds timeline indexes from repos

* Tidy

* Sync-up with main

* Post-merge update event stream sync test
2022-12-15 21:04:09 -05:00
Paul Frazee 8a92990268 Reset password updates (#422)
* Log emails to stdout if no smtp mailer is set (useful for debugging)

* Fancier reset-password email template

* Update reset password flow to use a human-enterable OTP

* Tidy

* Run the email template through prettier

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-15 17:43:36 -06:00
devin ivy 809c9a38f4 Message queue blocking and graceful teardown (#421)
* Fix db blocking when processing from the message queue

* Make message queue concurrency test not pg-only

* Rename test

* Dedupe test

* Tidy
2022-12-15 13:54:05 -05:00
Daniel Holmgren b52f53e9fe Cleanup write prep (#419)
* clean up record validation

* simplify & reorg write prepares
2022-12-15 12:03:12 -06:00
Daniel Holmgren 53d01564ed Clean up record validation (#417)
clean up record validation

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-15 11:32:58 -06:00
Daniel Holmgren 7425fba702 Bad password test (#418)
quick test for bad pass
2022-12-15 11:02:31 -06:00
devin ivy ed9556f049 Streaming: decouple producer side more, refactor db class (#414)
* Begin to decouple message queue consumers from the queue itself

* Tidy

* Reorganize pds message queue code out of db

* Decouple message queue, repo, and actor functionality from db instance w/ services

* Move repo processing into repo service

* Tidy

* Move repo blobs functionality into service

* Tidy

* Ensure to close message queue in all pds tests

* Fix typo

* Force specifying a db when using a service

* Reorg pds record plugins into record service

* Rename pds stream/ to event-stream/

* Tidy and fixes
2022-12-15 00:00:14 -05:00
Daniel Holmgren 91e7828021 embedsForPost fix (#416)
get embeds fix
2022-12-14 15:37:38 -06:00
Daniel Holmgren 01760b994e Patch external embed table type (#415)
* patch external embed table type

* missed type
2022-12-14 15:02:20 -06:00
Paul Frazee f5302ce139 Fixes to post media (#412)
* Fix typo

* Add failing test for attaching image to a post

* Tidy failing test

* Fix

* Expand the available image resize fit modes

* Add proper validation of glob-prefixed mimes (eg image/*)

* Detect and handle blobs in post embeds

* evert "Expand the available image resize fit modes"

This reverts commit 3390fbe4d091a04f9b67a2885eae19d025747fc4.

* Use fit: inside for post media

* Update tests

* Make the thumbnail optional in external embeds

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-14 14:28:59 -06:00
Daniel Holmgren 9598a22ef5 Cleanup dependencies (#410)
* cleaning up & deleting unused deps

* oops we were using npm-run-all
2022-12-14 12:31:28 -06:00
devin ivy a09483dcd1 Streaming: support multi-stage processing (#404)
* Begin to decouple message queue consumers from the queue itself

* Tidy

* Support multi-stage stream processing by consumers returning effects
2022-12-13 14:22:42 -05:00
Daniel Holmgren 57bec34cf5 Post media (#400)
* adding in banners & some lex codegen improvements for server

* fix up spacing

* couple small fixes

* client codegen & remove media embed schemas

* hook up banners

* missed a couple of lexicons

* sketching out embed schemas

* change wrapper obj property name

* setting up indexing for post media

* adding to views

* codegen client

* some tests

* Update lexicons/app/bsky/feed/getAuthorFeed.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update lexicons/app/bsky/feed/getPostThread.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update lexicons/app/bsky/feed/getTimeline.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* exhaustive check

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-12-13 13:17:53 -06:00
Ryan Barrett eb6f295c52 Update example code in xrpc and xrpc-server READMEs (#409)
* addSchema => addLexicon
* update lexicons for new defs.main structure
* update server ping method implementation to match XRPCHandler type signature
2022-12-13 09:09:57 -06:00
devin ivy 4bc9af4c37 Downgrade file-type dep version, fix build (#407)
Downgrade file-type dep since it is now ESM-only
2022-12-12 15:53:05 -05:00
devin ivy 789fbb27c4 Fix size/url for profile banner (#406) 2022-12-12 14:55:27 -05:00
Paul Frazee 3636bd19a8 A couple lexicon-related fixes (#405)
* Fix the generated type for encoding when the value is */* (close #391)

* Add 'toKnownError' test and fix named error for updateProfile

* Improve blob errors
2022-12-12 13:48:59 -06:00
devin ivy 2ab2dbf75a Streaming: decouple consumers from the message queue (#403)
* Begin to decouple message queue consumers from the queue itself

* Tidy

* Get rid of callback variant of Listener
2022-12-12 14:32:06 -05:00
devin ivy f3eac9b2e8 Support compression in requests to xrpc-server (#396)
* Support compression in requests to xrpc-server

* Check payload streaming payload sizes in xrpc-server

* Tidy tests

* Move MaxSizeChecker into common stream utils
2022-12-12 12:52:14 -05:00
Daniel Holmgren a125f4e36d Banners (#397)
* adding in banners & some lex codegen improvements for server

* fix up spacing

* couple small fixes

* client codegen & remove media embed schemas

* hook up banners

* missed a couple of lexicons
2022-12-12 10:18:50 -06:00
Daniel Holmgren 91215f4f67 Verify content type of files (#395)
check content type of files
2022-12-08 17:00:11 -06:00
Daniel Holmgren c74262299b Rename upload blob NSID (#393)
* rename blob route

* Tweak to codegen after rebase

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2022-12-08 13:30:10 -06:00
Daniel Holmgren 2a4a1ec4d1 Various blob fixes (#394)
* fix byte streaming from mem blobstore

* Enlarge avatar maxSize to 300kb

* Skip the cache on empty files

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2022-12-08 13:26:05 -06:00
Paul Frazee 79c3f31f9e Add 'avatar' to the outputs of views (#392)
* Add sharp to the dev-env deps

* Add 'avatar' to the outputs of views
2022-12-08 11:40:19 -06:00
devin ivy 13f8fae5c9 Fix S3 blob store, issue putting object from a stream (#389)
Fix s3.putObject when used w/ streams by using Upload
2022-12-08 12:00:33 -05:00
devin ivy bab3b1f783 Fix issue with duped blobs (#390) 2022-12-08 11:53:58 -05:00
Daniel Holmgren d0b7497a08 File uploads & profile image support (#383)
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* wip

* improve repo processing & add blbos

* more blob processing

* work into routes & config

* testing out w profiles

* Implement initial pds image uri builder/signer/verifier

* pr fixup & tests

* streaming

* better streaming interface

* s3 interface

* fix s3 prefixes

* readable streams as xrpc inputs

* more tests + cleaning up apis

* tests for failed references

* Initial implementation of image processing server

* Update node types for node v18

* fix clone issue

* add getStream to blobstore

* fixing up tests & dev-env

* get img info on upload

* integrating img processing

* hex for img uri & fixing build errors

* improve streams

* hook up uris to getProfile

* pr feedback on tmp

* test on getprofile

* testing avatars

* fix snapshot after hmac changed

* quick cleanup

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-07 18:08:29 -06:00
devin ivy d19fbe5473 Image processing interface (#378)
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* Implement initial pds image uri builder/signer/verifier

* Initial implementation of image processing server

* Update node types for node v18

* Add disk caching to pds image service

* Test pds image process caching, tidy

* Move pds image processor away from an interface

* Add sharp to pds build externals

* Move away from xrpc-server errors to http-errors for pds image server

* Tidy pds image exports

* Tidy
2022-12-07 09:44:34 -06:00
Paul Frazee d7b4697b2d Add terms-of-service and privacy-policy links (#386)
* Add privacy-policy and terms-of-service to getAccountsConfig

* Add privacy policy and tos urls to dev environment
2022-12-06 13:05:05 -06:00
devin ivy aea590ef37 Auth in XRPC server (#384)
* Add auth to xrpc-server, refactor to use per-route middleware stack

* Add xrpc-server error middleware, tidy

* Tests in xrpc for for catchall errors and auth

* Tidy

* Update lex-cli for new xrpc-server auth interface

* Update pds lexicon w/ codegen for xrpc auth

* Use xrpc-server auth in pds

* Update tests for newly-required auth
2022-12-05 23:46:21 -05:00
Paul Frazee c966185f01 Lexicon tweaks: encoding and union $type discriminators (#382)
* Update LexXrpcBody encoding to always be a single string with comma-separated values

* Update lexicon union-type to use  as a discriminator, and add closed flag to establish if new union members may be added

* Fix to generated code for open unions
2022-12-05 09:38:51 -06:00
Paul Frazee dd180f3dd1 Refactor lexicon definitions to reuse more shared types (#381)
* Factor out app.bsky.system.declRef

* Factor out app.bsky.actor.ref and app.bsky.feed.embed

* Factor out com.atproto.repo.strongRef and use app.bsky.actor.ref in more places

* Modify app.bsky.actor.ref to be light reference, add #withInfo for indexed info
2022-12-02 09:25:03 -06:00
Paul Frazee f2aa6ddbea Rework lexicon references to use a more consistent structure (#380) 2022-12-01 16:09:41 -06:00
Paul Frazee 63b9873bb1 Lexicon rework (#379)
* Implement types and validation for new NIH lexicon DSL

* Update xprc and xrpc-server to use new NIH lexicon DSL

* Simplification: dont allow lexicon definitions to be refs or ref arrays

* Add 'unknown' primitive type to NIH lexicon

* Implement client codegen for NIH Lexicon

* Implement server codegen for NIH Lexicon

* Add the 'datetime' primitive type to the NIH Lexicon

* Rework lexicons to use new NIH Lexicon DSL

* Readd tokens to code generation

* Update all code to use new NIH Lexicon (all tests passing!)

* Add imports from external files to lex-cli codegen

* Do not allow records, queries, or procedures outside of the main def

* Update packages/lex-cli/src/util.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Linter pass

* Add knownValue output to codegen

* Add comments to codegen

* Use enum and const in codegen

* Rework lexicon 'properties' structure

* Treat falsy query params as undefined

Co-authored-by: devin ivy <devinivy@gmail.com>

* Remove now-redundant param presence validation

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-12-01 12:26:18 -06:00
devin ivy 645a2ca5f7 Enforce max payload size for PLC and PDS (#376)
* Sketching out max payload size in xrpc-server

* Fixes and testing for external payload parsing

* Apply max json payload limit to pds and plc

* Move parsing middleware into xrpc-server w/ options
2022-11-29 13:02:03 -05:00
Paul Frazee 47c69f5469 Update getPostThread to handle missing parents (#373) 2022-11-28 21:35:40 -06:00
Paul Frazee ebf8e5c9ed Add NotFoundError to getPostThread (#375) 2022-11-28 21:33:32 -06:00
devin ivy 134595e26e Standardize limit schema across lexicons (#377) 2022-11-28 21:34:55 -05:00
devin ivy 9d29ef96f6 Well-defined pagination in PDS (#365)
* Implement tuple keyset cursor to break pagination ties, apply to feeds and assertions

* Tuple keyset cursor on rest of views

* Tidy keysets for postgres

* Add default implementation to Keyset to remove some boilerplate

* Simplify Keyset interface

* Remove dead code

* Move to new Keyset interface, incl. TimeCidKeyset

* Make pds assertion view tests deterministic

* Apply pds keyset interface to user search
2022-11-28 11:51:27 -05:00
devin ivy fd359bada0 Use integers in lexicons (#367)
Use json schema integers rather than numbers where appropriate in lexicons
2022-11-23 16:13:37 -05:00
Daniel Holmgren 083f53078a Fix a few MST bugs (#364)
* fix MST stage & getPointer operations

* test

* comments

* fix pointer issue on higher level splits
2022-11-22 14:37:38 -06:00
Daniel Holmgren 6fd50bd46c Wait for servers to close at end of test suite (#362)
wait for servers to close
2022-11-18 15:47:03 -06:00
Daniel Holmgren 0f2562419b Return post ancestors (#361)
* recursively get post ancestors

* tests
2022-11-17 17:06:44 -06:00
Daniel Holmgren 40458a6249 Fix unclear indexedAt in processor (#360)
fix unclear indexedAt in processor
2022-11-17 15:36:05 -06:00
Daniel Holmgren dafeeee4af MessageQueue fix (#358)
* keep message queue from getting behind

* better error hadnling that doesnt crash sanity loop

* 3 -> 1 min

* timeout + destory on mq

* destory mq on db close
2022-11-17 15:34:18 -06:00
Daniel Holmgren 1abe9e7e0d Remove Level (#357)
remove level & move did-web server to tests
2022-11-17 15:22:33 -06:00
Daniel Holmgren 53d8209c9c Don't give notifications for mentions & replies to self (#359)
Dont give notifciations for metnions & replies to self
2022-11-17 15:22:14 -06:00