116 Commits

Author SHA1 Message Date
Daniel Holmgren
05855fcff3
Loosening image constraints ()
* loosening img constraints

* fixing up tests
2023-01-30 18:59:19 -05:00
devin ivy
df74970bef
Feed thumbnail sizing, quality param bugfix ()
* Fix support for quality param in image urls

* Use cover sizing for feed thumbnails

* Update snapshots for image thumb urls
2023-01-27 10:19:42 -05:00
Daniel Holmgren
75e14ae102
Feature branch: storage & sync revamp ()
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts ()

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts ()

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-01-26 18:09:55 -06:00
Daniel Holmgren
6f780bf670
Remove scenes ()
* wip

* wip

* remove getAssertions & fix up notifs test

* remove did param on updateProfile
2023-01-26 15:38:45 -06:00
Paul Frazee
ff942976ea
Include the user's handle in the reset password email ()
* Include the user's handle in the reset password email (close )

* Add test for handle in reset-code email

* Linter fix
2023-01-26 14:14:20 -06:00
Paul Frazee
49d31f0a2c
Add missing 'avatar' field in app.bsky.graph.getFollows () 2023-01-26 14:14:00 -06:00
devin ivy
9834c2f5ef
Allow login using email ()
* Allow login using email, deprecate handle input in session.create

* Remove handle input from session.create lexicon
2023-01-26 13:08:56 -05:00
Daniel Holmgren
e8c218b5c9
Sort "suggested follows" by number of posts ()
* return suggestions by post count

* pr feedback

* fix up PG pagination issue
2023-01-24 14:03:51 -06:00
Paul Frazee
e44c98e717
Update module publish scripts ()
* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts
2023-01-24 10:11:20 -06:00
Daniel Holmgren
292f41c807
Fix migration name ()
fix migration name
2023-01-19 18:07:24 -06:00
devin ivy
13dd58bb15
Feature: admin moderation views ()
* Model mod actions for actor takedowns

* Add administration lex schemas for moderation actions

* Initial implementation of takedown mod action and reversal

* Refactor admin actions into a service

* Check for account takedown when creating new session or refreshing session

* Sweep usage of getUser() and getUserByEmail() for takedowns

* Begin sweep for account takedowns

* Tidy takedown checks

* Handle post author takedowns

* Fix tests and migrations for mod takedown

* Fix xrpc-server delayed read of blob payloads

* Test takedown mod action, misc fixes

* Remodel lex schemas for admin moderation actions

* Remove delcaration cid from actor takedowns, lex schema tidy

* Add notes

* Move moderation methods to com.atproto, based around repo rather than actor

* Moderation reports of repos and records

* Implement moderation report resolution method

* Tidy

* Place moderation-init migration after mute-init

* Allow services to reference each other

* Model record takedowns in db

* Lex schemas for record takedowns

* Service/methods for record takedowns, fix report view subject

* Apply record takedown to posts in feeds

* Test record takedown effects, fix reversal

* Support server moderation flag and acknowledge actions, fix action/report resolution for records

* Track repo takedowns on repo_root rather than did_handle

* Move repo/record reporting under com.atproto.report

* Colocate all moderation functionality into moderation service, combine from admin and repo services

* Tidy

* Tidy

* Unify mod report and action methods with new repo/record ref schemas

* Schemas for admin moderation views

* Tidy/codegen

* Fix test snapshot

* Break pds admin methods into separate files

* Implment admin repo search method

* Reorganize existing moderation views into the ModerationViews

* Lex schemas for admin get repo/record, implement get repo

* Implement admin get record view

* Tidy/codegen

* Tidy, fix snapshot

* Tidy, swap filenames

* Implement admin get moderation action view

* Implement admin get moderation report view

* Implement and test admin views for mod actions and reports

* Dry-up some admin and moderation-related seeds in the tests

* Fix xrpc fetch usage for newly required duplex option

* Admin view of repo and record details ()

* Reorganize existing moderation views into the ModerationViews

* Lex schemas for admin get repo/record, implement get repo

* Implement admin get record view

* Tidy/codegen

* Tidy, fix snapshot
2023-01-18 18:01:04 -05:00
Dimitris Apostolou
422008be3e
Fix typos () 2023-01-09 21:34:07 -05:00
devin ivy
1007fab49a
Fix out-of-date test snapshots ()
Fix out-of-date snapshots
2023-01-06 17:37:56 -05:00
devin ivy
ef487900f6
Server moderation action/resolution and reporting ()
* Model mod actions for actor takedowns

* Add administration lex schemas for moderation actions

* Initial implementation of takedown mod action and reversal

* Refactor admin actions into a service

* Check for account takedown when creating new session or refreshing session

* Sweep usage of getUser() and getUserByEmail() for takedowns

* Begin sweep for account takedowns

* Tidy takedown checks

* Handle post author takedowns

* Fix tests and migrations for mod takedown

* Fix xrpc-server delayed read of blob payloads

* Test takedown mod action, misc fixes

* Remodel lex schemas for admin moderation actions

* Remove delcaration cid from actor takedowns, lex schema tidy

* Add notes

* Move moderation methods to com.atproto, based around repo rather than actor

* Moderation reports of repos and records

* Implement moderation report resolution method

* Tidy

* Place moderation-init migration after mute-init

* Allow services to reference each other

* Model record takedowns in db

* Lex schemas for record takedowns

* Service/methods for record takedowns, fix report view subject

* Apply record takedown to posts in feeds

* Test record takedown effects, fix reversal

* Support server moderation flag and acknowledge actions, fix action/report resolution for records

* Track repo takedowns on repo_root rather than did_handle

* Move repo/record reporting under com.atproto.report

* Colocate all moderation functionality into moderation service, combine from admin and repo services

* Tidy

* Tidy

* Unify mod report and action methods with new repo/record ref schemas
2023-01-06 12:59:47 -05:00
devin ivy
e9f5f23edc
Fix lexicon validation, do not allow undefined when required ()
* Fix lexicon validation, do not allow undefined when required

* Make term optional for actor search/typeahead
2023-01-05 12:04:31 -05:00
Devin Ivy
9d32cf130e Fix resolution of server did in com.atproto.handle.resolve 2023-01-04 17:39:30 -05:00
Paul Frazee
2ed0c31254
Datetime validation fix ()
* Fix datetime validation (close )

* Fix lexicon publishing flow

* lexicon@0.0.3

* lexicon@0.0.4

* Fix xrpc publishing scripts

* xrpc@0.0.3

* api@0.0.6

* Update thread test snapshot
2023-01-03 16:37:31 -06:00
Daniel Holmgren
2602c00e85
Remove "bksy" ()
done once & for all
2023-01-02 21:40:56 -06:00
Paul Frazee
3f69270dfe
API & client improvements ()
* Add typeguard helpers to codegen

* Add option to disable validation on XRPC responses

* Add xrpc client-side validation of responses

* Add general-purpose validate() method to Lexicons

* Add validation helpers to lex-cli codegen
2023-01-02 17:38:30 -06:00
devin ivy
149da8218e
Remove unmapped post.viewer.muted property ()
Remove unused/unmapped post.viewer.muted property
2023-01-02 15:42:04 -05:00
devin ivy
aa626c3dc4
User muting ()
* Procedures and view for user muting

* Update profile and feed/thread methods for muting, update tests

* Tidy

* Test muting for timeline, actor feed, thread, profile views, misc fixes

* Test mute views, misc fixes

* Add missing file

* Add per-user notification index

* Omit mention and reply notifications from muted users

* Test timing leeway
2022-12-30 14:27:48 -05:00
Paul Frazee
1ede96c4ad
Improve image quality ()
* Increase the allowed dimenions of all images

* Increase the size and quality of served images

* Update tests for larger res
2022-12-26 15:51:54 -06:00
devin ivy
b686b3baff
Create PDS with ServerConfig rather than ServerConfigValues ()
Create pds with ServerConfig rather than ServerConfigValues
2022-12-21 13:32:10 -05:00
Daniel Holmgren
30ab0d341b
Refactor feeds & post threads ()
* factor out feed items & mystate

* propose feed items rewordk

* pr feedback

* revamping getTimeline

* feed service & cleanup

* bugfixin

* update timeline snapshots

* update getAuthFeed snapshot

* bugfixin

* revamp getPostThread

* fix up sync & votes tests

* bug in post thread

* fix replyCount

* fix ordering

* move uriImgBuilder to createServices
2022-12-20 20:38:37 -06:00
Daniel Holmgren
f0ee51f151
App context ()
* introducing appcontext & pds class

* revamp rest of server + auth

* revamp tests

* add to dev-env

* fix api test

* add start method to pds

* hook up terminator before

* PLC AppContext ()

* convert plc server to app context

* add start method to plc

* hooke up terminator on server setup

* fix test

* cleanup & pr feedback

* fix up one more test
2022-12-16 11:23:36 -06:00
devin ivy
53f6db7533
Test rebuilding pds timeline indexes from repos ()
* Test rebuilding pds timeline indexes from repos

* Tidy

* Sync-up with main

* Post-merge update event stream sync test
2022-12-15 21:04:09 -05:00
Paul Frazee
8a92990268
Reset password updates ()
* Log emails to stdout if no smtp mailer is set (useful for debugging)

* Fancier reset-password email template

* Update reset password flow to use a human-enterable OTP

* Tidy

* Run the email template through prettier

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-15 17:43:36 -06:00
devin ivy
809c9a38f4
Message queue blocking and graceful teardown ()
* Fix db blocking when processing from the message queue

* Make message queue concurrency test not pg-only

* Rename test

* Dedupe test

* Tidy
2022-12-15 13:54:05 -05:00
Daniel Holmgren
b52f53e9fe
Cleanup write prep ()
* clean up record validation

* simplify & reorg write prepares
2022-12-15 12:03:12 -06:00
Daniel Holmgren
53d01564ed
Clean up record validation ()
clean up record validation

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-15 11:32:58 -06:00
Daniel Holmgren
7425fba702
Bad password test ()
quick test for bad pass
2022-12-15 11:02:31 -06:00
devin ivy
ed9556f049
Streaming: decouple producer side more, refactor db class ()
* Begin to decouple message queue consumers from the queue itself

* Tidy

* Reorganize pds message queue code out of db

* Decouple message queue, repo, and actor functionality from db instance w/ services

* Move repo processing into repo service

* Tidy

* Move repo blobs functionality into service

* Tidy

* Ensure to close message queue in all pds tests

* Fix typo

* Force specifying a db when using a service

* Reorg pds record plugins into record service

* Rename pds stream/ to event-stream/

* Tidy and fixes
2022-12-15 00:00:14 -05:00
Daniel Holmgren
91e7828021
embedsForPost fix ()
get embeds fix
2022-12-14 15:37:38 -06:00
Daniel Holmgren
01760b994e
Patch external embed table type ()
* patch external embed table type

* missed type
2022-12-14 15:02:20 -06:00
Paul Frazee
f5302ce139
Fixes to post media ()
* Fix typo

* Add failing test for attaching image to a post

* Tidy failing test

* Fix

* Expand the available image resize fit modes

* Add proper validation of glob-prefixed mimes (eg image/*)

* Detect and handle blobs in post embeds

* evert "Expand the available image resize fit modes"

This reverts commit 3390fbe4d091a04f9b67a2885eae19d025747fc4.

* Use fit: inside for post media

* Update tests

* Make the thumbnail optional in external embeds

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-14 14:28:59 -06:00
Daniel Holmgren
9598a22ef5
Cleanup dependencies ()
* cleaning up & deleting unused deps

* oops we were using npm-run-all
2022-12-14 12:31:28 -06:00
devin ivy
a09483dcd1
Streaming: support multi-stage processing ()
* Begin to decouple message queue consumers from the queue itself

* Tidy

* Support multi-stage stream processing by consumers returning effects
2022-12-13 14:22:42 -05:00
Daniel Holmgren
57bec34cf5
Post media ()
* adding in banners & some lex codegen improvements for server

* fix up spacing

* couple small fixes

* client codegen & remove media embed schemas

* hook up banners

* missed a couple of lexicons

* sketching out embed schemas

* change wrapper obj property name

* setting up indexing for post media

* adding to views

* codegen client

* some tests

* Update lexicons/app/bsky/feed/getAuthorFeed.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update lexicons/app/bsky/feed/getPostThread.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update lexicons/app/bsky/feed/getTimeline.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* exhaustive check

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-12-13 13:17:53 -06:00
devin ivy
4bc9af4c37
Downgrade file-type dep version, fix build ()
Downgrade file-type dep since it is now ESM-only
2022-12-12 15:53:05 -05:00
devin ivy
789fbb27c4
Fix size/url for profile banner () 2022-12-12 14:55:27 -05:00
Paul Frazee
3636bd19a8
A couple lexicon-related fixes ()
* Fix the generated type for encoding when the value is */* (close )

* Add 'toKnownError' test and fix named error for updateProfile

* Improve blob errors
2022-12-12 13:48:59 -06:00
devin ivy
2ab2dbf75a
Streaming: decouple consumers from the message queue ()
* Begin to decouple message queue consumers from the queue itself

* Tidy

* Get rid of callback variant of Listener
2022-12-12 14:32:06 -05:00
devin ivy
f3eac9b2e8
Support compression in requests to xrpc-server ()
* Support compression in requests to xrpc-server

* Check payload streaming payload sizes in xrpc-server

* Tidy tests

* Move MaxSizeChecker into common stream utils
2022-12-12 12:52:14 -05:00
Daniel Holmgren
a125f4e36d
Banners ()
* adding in banners & some lex codegen improvements for server

* fix up spacing

* couple small fixes

* client codegen & remove media embed schemas

* hook up banners

* missed a couple of lexicons
2022-12-12 10:18:50 -06:00
Daniel Holmgren
91215f4f67
Verify content type of files ()
check content type of files
2022-12-08 17:00:11 -06:00
Daniel Holmgren
c74262299b
Rename upload blob NSID ()
* rename blob route

* Tweak to codegen after rebase

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2022-12-08 13:30:10 -06:00
Daniel Holmgren
2a4a1ec4d1
Various blob fixes ()
* fix byte streaming from mem blobstore

* Enlarge avatar maxSize to 300kb

* Skip the cache on empty files

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2022-12-08 13:26:05 -06:00
Paul Frazee
79c3f31f9e
Add 'avatar' to the outputs of views ()
* Add sharp to the dev-env deps

* Add 'avatar' to the outputs of views
2022-12-08 11:40:19 -06:00
devin ivy
bab3b1f783
Fix issue with duped blobs () 2022-12-08 11:53:58 -05:00
Daniel Holmgren
d0b7497a08
File uploads & profile image support ()
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* wip

* improve repo processing & add blbos

* more blob processing

* work into routes & config

* testing out w profiles

* Implement initial pds image uri builder/signer/verifier

* pr fixup & tests

* streaming

* better streaming interface

* s3 interface

* fix s3 prefixes

* readable streams as xrpc inputs

* more tests + cleaning up apis

* tests for failed references

* Initial implementation of image processing server

* Update node types for node v18

* fix clone issue

* add getStream to blobstore

* fixing up tests & dev-env

* get img info on upload

* integrating img processing

* hex for img uri & fixing build errors

* improve streams

* hook up uris to getProfile

* pr feedback on tmp

* test on getprofile

* testing avatars

* fix snapshot after hmac changed

* quick cleanup

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-07 18:08:29 -06:00