Commit Graph

135 Commits

Author SHA1 Message Date
devin ivy b91e6e96b7 Incorporate blobs into moderation actions (#511)
* Lexicon updates for moderating blobs

* Add tracking of blob takedowns to moderation system

* Handle blobs in record takedown reversal

* Apply blob takedowns w/ image invalidation

* Test blob takedowns

* Implement cloudfront image invalidator

* Update lexicons for current moderation action on records and blobs

* Fix migration for postgres

* Enforce single current action per repo and blob, present current action on repo and blob

* Test single active mod action per repo, record, and blob

* Continue to resolve handles for moderated repos
2023-02-06 11:44:26 -05:00
Daniel Holmgren 64f5b4a022 Refactor repo indexing (#521)
* refactor repo indexing

* bit more
2023-02-03 17:11:39 -06:00
Daniel Holmgren 773f9e304d Account deletion (#488)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* some lexicons

reworking routes

request deletion flows

delete actor rows

migration for user-partitioned-cids

move creator to be on ipld_block

migration tests

* delete records & repos

* delete blobs

* hook it up in route

* pettier ignore email templates

* testing & bugfixes

* testing blobs & bugfixes

* pr feedback

* make deletion test more robust

* change out handle for did on account deletion

* small cleanup

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-02-02 12:34:00 -06:00
Daniel Holmgren 75e14ae102 Feature branch: storage & sync revamp (#446)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-01-26 18:09:55 -06:00
Paul Frazee e44c98e717 Update module publish scripts (#478)
* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts
2023-01-24 10:11:20 -06:00
devin ivy 53f6db7533 Test rebuilding pds timeline indexes from repos (#413)
* Test rebuilding pds timeline indexes from repos

* Tidy

* Sync-up with main

* Post-merge update event stream sync test
2022-12-15 21:04:09 -05:00
Paul Frazee f5302ce139 Fixes to post media (#412)
* Fix typo

* Add failing test for attaching image to a post

* Tidy failing test

* Fix

* Expand the available image resize fit modes

* Add proper validation of glob-prefixed mimes (eg image/*)

* Detect and handle blobs in post embeds

* evert "Expand the available image resize fit modes"

This reverts commit 3390fbe4d091a04f9b67a2885eae19d025747fc4.

* Use fit: inside for post media

* Update tests

* Make the thumbnail optional in external embeds

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-14 14:28:59 -06:00
Daniel Holmgren 9598a22ef5 Cleanup dependencies (#410)
* cleaning up & deleting unused deps

* oops we were using npm-run-all
2022-12-14 12:31:28 -06:00
Daniel Holmgren d0b7497a08 File uploads & profile image support (#383)
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* wip

* improve repo processing & add blbos

* more blob processing

* work into routes & config

* testing out w profiles

* Implement initial pds image uri builder/signer/verifier

* pr fixup & tests

* streaming

* better streaming interface

* s3 interface

* fix s3 prefixes

* readable streams as xrpc inputs

* more tests + cleaning up apis

* tests for failed references

* Initial implementation of image processing server

* Update node types for node v18

* fix clone issue

* add getStream to blobstore

* fixing up tests & dev-env

* get img info on upload

* integrating img processing

* hex for img uri & fixing build errors

* improve streams

* hook up uris to getProfile

* pr feedback on tmp

* test on getprofile

* testing avatars

* fix snapshot after hmac changed

* quick cleanup

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-07 18:08:29 -06:00
Daniel Holmgren 083f53078a Fix a few MST bugs (#364)
* fix MST stage & getPointer operations

* test

* comments

* fix pointer issue on higher level splits
2022-11-22 14:37:38 -06:00
Daniel Holmgren c207947bd5 Stagable blockstore (#351)
* stagable blockstore + remove mutable repo

* pds wip

* nearly there

* bugfix

* assertion ordering

* del old code

* save staged if no callback

* removing level

* pr fixup
2022-11-17 09:51:36 -06:00
devin ivy 13aead171c Support shallow builds and sourcemaps (#352)
* Enable all modules to be built shallowly, externalizing their deps

* Setup shallow builds to modify package.json main

* Hoist update-main-to-dist to separate call, fix lex-cli and plc shallow builds

* Generate sourcemaps during build

* Tidy

* Hoist new build deps up to root
2022-11-15 19:24:58 -05:00
Daniel Holmgren 962041e6c1 Scene trending (#327)
* wip

* views

* trending schema

* starting message queue

* scene processor

* wip

* send mq messages from db

* db events

* undo screwing up codegen lol

* setup queue

* db migrations

* fixing up message processing

* div by 0 check

* tx issue

* queue use cursor

* update not insert

* sql bugfix + tests

* trying to linearize tests

* correclty serializing txs

* attempt update before insert

* log errors

* handle gaps in cursor

* cleanup

* oops reenable test

* correctly incr cursor
2022-11-11 12:29:48 -06:00
Daniel Holmgren 94a4f7ec41 simple repo benchmarks (#246)
* simple repo benchmarks

* cleanup
2022-10-31 12:13:38 -05:00
Daniel Holmgren 9879f673ad SQL Blockstore (#276)
* setting up sql-blockstore

* drop table migration

* unblock sqlite tx

* correct binary types

* lol woops

* block encoding utilities

* no longer store raw record in record table

* fix dev-env

* pr feedback

* tx check
2022-10-27 13:26:17 -05:00
devin ivy 4a65804db5 Miscellaneous dep and build fixes (#272)
Misc dep and build fixes
2022-10-26 14:19:14 -04:00
devin ivy e3c4efd7d9 PDS transactions (#245)
* Run getProfile in transaction, setup/test transaction helpers

* Test tidy

* Transactionalize invite code usage in acct creation, add transaction assertions

* Transactionalize account creation

* crud txns

* test txns for account

* cleanup

* lock table

* fixed up crud operations

* fixed updateProfile route

* test data races

* cleanup

* Transactional invite code use, username/email creation (#253)

* Remove table lock for invite code use, surface username race

* Add and use db constraint for unique account username/email

* Update packages/server/src/db/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/server/src/db/records/badgeAccept.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/server/src/db/util.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr fixups

* remove old todos

* test for updateProfile races

* logging on repo update errors

* give feedback on misuse of forUpdate

Co-authored-by: dholms <dtholmgren@gmail.com>
2022-10-24 15:59:40 -05:00
Daniel Holmgren 960a0fce1e Immutable repo structure (#247)
* immutable repo

* wrap mutable repo around immutable

* fixing up repo & structure

* added back in safeCommit

* fix up server

* return rebase cid on commit callback

* getReocrd on structure + a bit of cleanup

* pr cleanup
2022-10-20 11:11:51 -05:00
Paul Frazee feca3622c1 Rename ADX -> ATP (#244)
* Rename ADX to ATP

* Remove old docs (now in atproto website repo)

* Update readme

* Update README.md

Co-authored-by: devin ivy <devinivy@gmail.com>

* A couple of @ -> AT

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-10-18 12:21:59 -05:00
Daniel Holmgren fd18da2cb5 Schema keyed hint (#243)
* update schemas for keyed

* codegen dict

* codgen record ids

* determine key type from schema

* allow non-tid keys

* rm tids from batch write schema

* use schema ids record in db layouts

* standardize on rkey

* doc update

* crud test

* repo tests for custom rkeys
2022-10-18 11:14:47 -05:00
Daniel Holmgren 60a537a6e4 MIT license (#241)
mit license errything
2022-10-17 13:20:06 -05:00
Daniel Holmgren 83691da3f6 Repo metadata (#240)
* repo metadata

* tests & write meta to car file
2022-10-17 13:11:09 -05:00
Daniel Holmgren 3679719571 Badging utilities (#237)
* record type for accepted badge

* getBadgeMembers view

* updateProfile procedure

* better return type on updateProfile & gen client

* add badges to getProfile view

* testing & bug fixes

* tests for updating profile & linking badges

* errors

* badgeOffer record type

* getProfile view ensures badges are both offered & accepted

* getBadgeMembers test

* PR feedback

* cid check
2022-10-17 11:18:08 -05:00
Daniel Holmgren bfaf3a86f9 Handling updates (#231)
* disabled udpates

* remove update tests

* update schemas

* crud operations

* working cids into db layer

* crud working

* views tests working & updated snapshots

* rm tests for likes on reposts

* fix up repo tests

* quick pr feedback

* recordKey -> rkey & nameOrDid -> user

* reference records by uri & cid obj

* fix dev-env
2022-10-12 18:36:40 -05:00
Daniel Holmgren b26eb75277 Logging (#209)
* messing around w pino

* rework locals & add pino-http

* subsystem logging off env vars

* remove auth tokens from logs

* lots of logging

* re-enable tests

* update enabled check

* fix subsystem logging

* pr fixups
2022-10-05 17:47:00 -05:00
Daniel Holmgren 670c419c18 Plc auth lib integration (#195)
* parsing dids to something we understand

* revamping did-sdk with did-resolver

* testing

* fixing some thigns & finishing tests

* fix type error & remove old test

* wip

* create did on server & add email

* check username available & store email

* better test utils

* remove did:test

* allow authstore to sign for controlled dids

* plc ucan integration wip

* wip

* finished auth lib refactor

* integrating to repo & server

* couple fix ups

* couple more fixups

* disable dev-env test

* comments
2022-09-28 17:52:18 -05:00
Daniel Holmgren 55e6e54fcf Did PLC (#181)
* validate the signatures
add the id to the did doc
pid from sha256 of oragin doc
added the tick to diffs extractor
added the key authorisation
add lots of types
add routes for calling the aic
refactor to jest based repo structure

* WIP

* migrate the tests

* more test fixes

* test.

* replace snake case with camel case

* moved to use isolated server for aic

* remove the aic test from the PDS

* clean up some changes outside the aic folder

* fix CaS for saving ticks

* rearange the folder

* missed  one

* cleaned up aic package.json

* broken build

* test passing

* prettier

* pid tests refactor

* fix eslint/prettier setup, add eslint-config-prettier

* misc linting and style tweaks in aic

* minor touchups to aic sign/verify

* tidy error-handling in aic

* tidy-up crypto init in aic

* add aic route for getting a did doc, update /tick routes

* knex -> typeorm

* refactor crypto

* error handling & dependencies

* switching op model & some other cleanup

* change db to use operations

* add cid references to ops & ensure no branching in DB tx

* cleaning up

* rotues

* working on client & tests

* document & oepration tests

* server tests

* cleanup

* remove lobby page & static

* formatting did documents

* aic/pch -> plc

* fixing up some doc semantics

* tooling/test server

* added recovery window

* tx mutex

* test on document formatting

* take validate op out of write tx

* error handling

* readme

* couple bug fixes

* quick readme edit

Co-authored-by: Aaron Goldman <aaron@aarons-mbp.lan>
Co-authored-by: Aaron D Goldman <aaron@blueskyweb.xyz>
Co-authored-by: Devin Ivy <devin@bigroomstudios.com>
2022-09-20 21:28:00 -05:00
Paul Frazee 1425b0cd4f New XRPC-based Server and API implementations (#180)
* Add xrpc-cli package; implement 'new' and 'gen-md' commands

* Add gen-ts-obj to xrpc-cli

* Add gen-api command to xrpc-cli

* Add gen-server to xrpc-cli

* Update codegen from xrpc-cli to run prettier and include a banner comment

* Use 'void' instead of 'undefined' for return type

* Skip invalid schemas when generating code

* Rewrite schemas package to use new simplified schema

* Output type definitions in nested folders

* Add record schema code-generation

* Change xrcp client call() signature to be easier to use

* Update client codegen to match new xrpc call signature

* Fixes and improvements to generated record APIs

* Update generated api to use per-service instances

* Add xrpc-cli readme

* (WIP) Rewrite server to use xrpc

* Add missing references to server/tsconfig.json

* Replace api implementation with xrpc generated client

* Add /schemas folder with todo.adx

* Add /schemas/todo.social

* Update api package to include todo.social xrpc and adx schemas

* Update server package to include todo.social xrpc and adx schemas

* Fix some schema definitions

* (WIP) Work toward xrpc update of server

* Update auth package to use nsids for collections

* Update repo package to use nsids for collections

* Update AdxUri to use nsid collection IDs

* Bring server CRUD tests to passing state

* Move API to simpler xrpc call signature

* Apply fixes and improvements to record apis

* Update to new generated api with per-service instances

* Rename the todo.social.* view methods

* Rewrite the views system to use xrpc

* Remove paths accidentally added to tsconfig

* Remove microblog package

* Add simple API readme

* todo about deduping ucan prfs

* increase postThread default depth to fix failing test. & fix USE_TEST_SERVER config

Co-authored-by: dholms <dtholmgren@gmail.com>
2022-09-20 17:39:05 -05:00
dholms 275c3df202 lock uint8arrays at 3.0.0 2022-09-13 15:16:57 -05:00
dholms c9a6609b93 fixed up common & repo tests 2022-09-13 11:15:56 -05:00
Paul Frazee cd431c58c8 Add some readmes (#169) 2022-09-13 10:54:56 -05:00
dholms 935481c566 missed a couple of type updates 2022-09-12 21:18:08 -05:00
dholms d5fe1880ff fix up api package as well 2022-09-12 21:14:14 -05:00
dholms 22f069f5d3 cleaning up dependencies 2022-09-12 21:07:06 -05:00
dholms 94babb38b7 split common and repo out into separate packages 2022-09-12 21:01:56 -05:00