141 Commits

Author SHA1 Message Date
Daniel Holmgren
134920d872
Database notify/listen (#514)
* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* poor guy cant spell
2023-02-02 15:04:38 -06:00
Daniel Holmgren
75e14ae102
Feature branch: storage & sync revamp (#446)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-01-26 18:09:55 -06:00
Paul Frazee
e44c98e717
Update module publish scripts (#478)
* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts
2023-01-24 10:11:20 -06:00
Daniel Holmgren
9598a22ef5
Cleanup dependencies (#410)
* cleaning up & deleting unused deps

* oops we were using npm-run-all
2022-12-14 12:31:28 -06:00
devin ivy
f3eac9b2e8
Support compression in requests to xrpc-server (#396)
* Support compression in requests to xrpc-server

* Check payload streaming payload sizes in xrpc-server

* Tidy tests

* Move MaxSizeChecker into common stream utils
2022-12-12 12:52:14 -05:00
Daniel Holmgren
91215f4f67
Verify content type of files (#395)
check content type of files
2022-12-08 17:00:11 -06:00
Daniel Holmgren
2a4a1ec4d1
Various blob fixes (#394)
* fix byte streaming from mem blobstore

* Enlarge avatar maxSize to 300kb

* Skip the cache on empty files

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2022-12-08 13:26:05 -06:00
Daniel Holmgren
d0b7497a08
File uploads & profile image support (#383)
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* wip

* improve repo processing & add blbos

* more blob processing

* work into routes & config

* testing out w profiles

* Implement initial pds image uri builder/signer/verifier

* pr fixup & tests

* streaming

* better streaming interface

* s3 interface

* fix s3 prefixes

* readable streams as xrpc inputs

* more tests + cleaning up apis

* tests for failed references

* Initial implementation of image processing server

* Update node types for node v18

* fix clone issue

* add getStream to blobstore

* fixing up tests & dev-env

* get img info on upload

* integrating img processing

* hex for img uri & fixing build errors

* improve streams

* hook up uris to getProfile

* pr feedback on tmp

* test on getprofile

* testing avatars

* fix snapshot after hmac changed

* quick cleanup

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-12-07 18:08:29 -06:00
devin ivy
13aead171c
Support shallow builds and sourcemaps (#352)
* Enable all modules to be built shallowly, externalizing their deps

* Setup shallow builds to modify package.json main

* Hoist update-main-to-dist to separate call, fix lex-cli and plc shallow builds

* Generate sourcemaps during build

* Tidy

* Hoist new build deps up to root
2022-11-15 19:24:58 -05:00
Paul Frazee
60071e0770
Handle updateProfile when no record exists (#343)
* WIP handle updateProfile when no record exists

* create new profile in updateProfile

Co-authored-by: dholms <dtholmgren@gmail.com>
2022-11-11 14:10:56 -06:00
Daniel Holmgren
9879f673ad
SQL Blockstore (#276)
* setting up sql-blockstore

* drop table migration

* unblock sqlite tx

* correct binary types

* lol woops

* block encoding utilities

* no longer store raw record in record table

* fix dev-env

* pr feedback

* tx check
2022-10-27 13:26:17 -05:00
Daniel Holmgren
21ff052da9
Dont reexport CID type (#277)
dont reexport cid
2022-10-26 23:13:50 -05:00
Paul Frazee
feca3622c1
Rename ADX -> ATP (#244)
* Rename ADX to ATP

* Remove old docs (now in atproto website repo)

* Update readme

* Update README.md

Co-authored-by: devin ivy <devinivy@gmail.com>

* A couple of @ -> AT

Co-authored-by: devin ivy <devinivy@gmail.com>
2022-10-18 12:21:59 -05:00
Daniel Holmgren
fd18da2cb5
Schema keyed hint (#243)
* update schemas for keyed

* codegen dict

* codgen record ids

* determine key type from schema

* allow non-tid keys

* rm tids from batch write schema

* use schema ids record in db layouts

* standardize on rkey

* doc update

* crud test

* repo tests for custom rkeys
2022-10-18 11:14:47 -05:00
Daniel Holmgren
60a537a6e4
MIT license (#241)
mit license errything
2022-10-17 13:20:06 -05:00
Daniel Holmgren
7233ce070f
Cleanup (#210)
* messing around w pino

* rework locals & add pino-http

* subsystem logging off env vars

* remove auth tokens from logs

* lots of logging

* re-enable tests

* update enabled check

* fix subsystem logging

* some quick cleanup
2022-10-05 17:55:08 -05:00
Daniel Holmgren
b26eb75277
Logging (#209)
* messing around w pino

* rework locals & add pino-http

* subsystem logging off env vars

* remove auth tokens from logs

* lots of logging

* re-enable tests

* update enabled check

* fix subsystem logging

* pr fixups
2022-10-05 17:47:00 -05:00
Daniel Holmgren
55e6e54fcf
Did PLC (#181)
* validate the signatures
add the id to the did doc
pid from sha256 of oragin doc
added the tick to diffs extractor
added the key authorisation
add lots of types
add routes for calling the aic
refactor to jest based repo structure

* WIP

* migrate the tests

* more test fixes

* test.

* replace snake case with camel case

* moved to use isolated server for aic

* remove the aic test from the PDS

* clean up some changes outside the aic folder

* fix CaS for saving ticks

* rearange the folder

* missed  one

* cleaned up aic package.json

* broken build

* test passing

* prettier

* pid tests refactor

* fix eslint/prettier setup, add eslint-config-prettier

* misc linting and style tweaks in aic

* minor touchups to aic sign/verify

* tidy error-handling in aic

* tidy-up crypto init in aic

* add aic route for getting a did doc, update /tick routes

* knex -> typeorm

* refactor crypto

* error handling & dependencies

* switching op model & some other cleanup

* change db to use operations

* add cid references to ops & ensure no branching in DB tx

* cleaning up

* rotues

* working on client & tests

* document & oepration tests

* server tests

* cleanup

* remove lobby page & static

* formatting did documents

* aic/pch -> plc

* fixing up some doc semantics

* tooling/test server

* added recovery window

* tx mutex

* test on document formatting

* take validate op out of write tx

* error handling

* readme

* couple bug fixes

* quick readme edit

Co-authored-by: Aaron Goldman <aaron@aarons-mbp.lan>
Co-authored-by: Aaron D Goldman <aaron@blueskyweb.xyz>
Co-authored-by: Devin Ivy <devin@bigroomstudios.com>
2022-09-20 21:28:00 -05:00
Paul Frazee
1425b0cd4f
New XRPC-based Server and API implementations (#180)
* Add xrpc-cli package; implement 'new' and 'gen-md' commands

* Add gen-ts-obj to xrpc-cli

* Add gen-api command to xrpc-cli

* Add gen-server to xrpc-cli

* Update codegen from xrpc-cli to run prettier and include a banner comment

* Use 'void' instead of 'undefined' for return type

* Skip invalid schemas when generating code

* Rewrite schemas package to use new simplified schema

* Output type definitions in nested folders

* Add record schema code-generation

* Change xrcp client call() signature to be easier to use

* Update client codegen to match new xrpc call signature

* Fixes and improvements to generated record APIs

* Update generated api to use per-service instances

* Add xrpc-cli readme

* (WIP) Rewrite server to use xrpc

* Add missing references to server/tsconfig.json

* Replace api implementation with xrpc generated client

* Add /schemas folder with todo.adx

* Add /schemas/todo.social

* Update api package to include todo.social xrpc and adx schemas

* Update server package to include todo.social xrpc and adx schemas

* Fix some schema definitions

* (WIP) Work toward xrpc update of server

* Update auth package to use nsids for collections

* Update repo package to use nsids for collections

* Update AdxUri to use nsid collection IDs

* Bring server CRUD tests to passing state

* Move API to simpler xrpc call signature

* Apply fixes and improvements to record apis

* Update to new generated api with per-service instances

* Rename the todo.social.* view methods

* Rewrite the views system to use xrpc

* Remove paths accidentally added to tsconfig

* Remove microblog package

* Add simple API readme

* todo about deduping ucan prfs

* increase postThread default depth to fix failing test. & fix USE_TEST_SERVER config

Co-authored-by: dholms <dtholmgren@gmail.com>
2022-09-20 17:39:05 -05:00
dholms
c9a6609b93 fixed up common & repo tests 2022-09-13 11:15:56 -05:00
Paul Frazee
cd431c58c8
Add some readmes (#169) 2022-09-13 10:54:56 -05:00
Daniel Holmgren
00d691d70f
Merge pull request #167 from bluesky-social/repo-package
Split out repo package
2022-09-13 08:54:01 -05:00
Daniel Holmgren
a47f1c85b4
Merge pull request #162 from bluesky-social/pds-api
PDS revamp
2022-09-13 08:53:40 -05:00
dholms
22f069f5d3 cleaning up dependencies 2022-09-12 21:07:06 -05:00
dholms
94babb38b7 split common and repo out into separate packages 2022-09-12 21:01:56 -05:00
dholms
550136f05c cleanup 2022-09-12 15:58:25 -05:00
dholms
1cb552a46b bit of cleanup 2022-09-12 13:44:05 -05:00
dholms
dad56e802f crud tests 2022-09-09 15:25:36 -05:00
dholms
0760d07a80 fixing badge refs 2022-09-08 11:30:35 -05:00
dholms
9529ebe886 posts go in -> posts come out 2022-08-31 19:09:21 -05:00
dholms
300e68bd52 del old microblogging code & allow sigs with no auth token from root did 2022-08-31 17:41:45 -05:00
dholms
aede15bd29 api routes 2022-08-30 19:15:09 -05:00
dholms
7271e8d2f3 repo roots & http-types 2022-08-19 15:52:57 -07:00
dholms
8ec37c960a wip 2022-08-18 18:11:44 -07:00
dholms
aab1a9ef42 add namespace/datasets to adxuri 2022-08-18 18:00:50 -07:00
dholms
59ef9fa391 add safecommit test 2022-08-18 10:38:12 -07:00
dholms
7a062e9877 unnecessary todo 2022-08-16 13:14:02 -07:00
dholms
d5c8806fdd cleanup + basic repo fn tests 2022-08-16 13:11:39 -07:00
dholms
1f3e5c309a some ucan cleanup 2022-08-16 12:30:03 -07:00
dholms
e9b558469a testing diffs 2022-08-16 12:18:23 -07:00
dholms
0c025b1723 test failure cases 2022-08-16 11:58:42 -07:00
dholms
4a34dc576e yay diffs & sync working 2022-08-11 18:10:14 -05:00
dholms
f44f1fb102 cleanup 2022-08-11 16:26:13 -05:00
dholms
60a6ff2c87 simplified diff class 2022-08-11 16:09:22 -05:00
dholms
49cdb5074c fix some subtle mst bugs & get diff working 2022-08-11 16:04:24 -05:00
dholms
84d56dc742 fixing up diff algo (wip) 2022-08-11 13:50:56 -05:00
dholms
4e03fba102 refactor blockstore 2022-08-10 16:26:26 -05:00
dholms
26cb3a35eb better tests 2022-08-10 16:02:21 -05:00
dholms
28a159869c fixed up writing car file 2022-08-09 20:04:45 -05:00
dholms
f9f13da614 fixing up verify 2022-08-09 19:48:01 -05:00