1610 Commits

Author SHA1 Message Date
Daniel Holmgren
bfaf3a86f9
Handling updates (#231)
* disabled udpates

* remove update tests

* update schemas

* crud operations

* working cids into db layer

* crud working

* views tests working & updated snapshots

* rm tests for likes on reposts

* fix up repo tests

* quick pr feedback

* recordKey -> rkey & nameOrDid -> user

* reference records by uri & cid obj

* fix dev-env
2022-10-12 18:36:40 -05:00
devin ivy
0b6bf87b66
Run PDS using postgres (#233)
* Setup dockerization of postgres db

* Fix typo

* Setup postgres dialect config in pds, fix migrations for pg

* Convert home and author feed to work on pg

* Convert pds follow views to work on pg

* Convert pds liked-by and notifications views to work on pg

* Unify bigint handling between sqlite and pg

* Convert pds account, crud, and profile, reposts, thread views to work on pg

* Ensure a more complete reset of containers/volumes in with-test-db.sh, set libpq env vars

* Add readme for dockerized postgres

* Fix build issues
2022-10-12 16:43:19 -04:00
Paul Frazee
45ddf23c58
Fixes to dev env (#234) 2022-10-12 15:07:31 -05:00
Paul Frazee
90cdb99bdd
Update NSID semantics to final (#232)
* Improve lex-cli to remove files that no longer exist

* Update lex-cli to use new ns

* Rename lexicons: todo.adx to com.atproto, todo.social to app.bsky

* Improve lex-cli md generator
2022-10-12 14:32:18 -05:00
Daniel Holmgren
7f2dfb8d50
Switch TypeORM to Kysely (#229)
* kysely interfaces

* migrated record plugins

* db types + notifications

* index record emthods on db

* Convert author feed to kysely

* rest of db index

* Convert home feed to kysely

* Remove references to sqlite3 package

* Convert liked by view to kysely

* create tables

* Convert notif count view to kysely

* move scrypt to own file

* Fix typo

* Convert notifs view to kysely

* Convert post thread to kysely

* Convert profile view to kysely

* accounts tests passing!

* Convert reposted by view to kysely

* Convert followers view to kysely

* Convert follows view to kysely

* Convert notifs last seen to kysely

* Tidy

* Re-enable social methods

* Minor tidy of author and home feed queries

* Tidying and minor fixes for pds view queries

* fix up crud, disabled PUTs for now

* Fix profile view, use nulls in table interfaces

* Fix pds follow/follower views, remove special db date logic

* Fix pds notification count view

* Fix pds feed views, reposted by join

* Remove typeorm from server

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2022-10-11 13:39:11 -05:00
Paul Frazee
72add11740
Fix the external better-sqlite3 path in dev-env build (#230) 2022-10-10 19:05:50 -05:00
devin ivy
ef69a725f0
Fix PLC server usage in dev-env (#228)
Fix PLC server usage in dev-env, ensure better-sqlite3 bindings are external
2022-10-10 18:58:24 -05:00
devin ivy
af5daacb90
More comprehensive testing for PDS views, using snapshots (#223)
* Create basic seed for server based on view tests

* Utilize basic seed in view tests

* Split-up pds view tests

* Trying out snapshot testing for author feed

* Tidy

* Generalize normalization for snapshot testing

* Snapshot test home feed, fix inclusion of own reposts

* Fix typo

* Add follow snapshot tests, fix ordering and db date handling

* Add like snapshot tests, fix ordering and db date handling

* Add notification snapshot tests

* Add profile snapshot tests

* Add reposts snapshot tests, fix ordering and db date handling

* Add thread snapshot tests

* Fix typo

* Remove unneeded space
2022-10-10 11:38:57 -04:00
Daniel Holmgren
214e4a14bb
Plc tests & cleanup (#226)
* ensure we understand the key type a user is rotating to

* race condition test

* handle username/pds with or without https://

* fix a quick types issue

* adding logging

* fix better-sqlite3 build issues

* bring did-resolver up to date w plc

* fix pds tests for plc
2022-10-10 09:50:49 -05:00
Daniel Holmgren
47c1e1f327
Use Kysely for PLC DB (#220)
* switching out typeorm for kysely

* refigure migrations
2022-10-07 17:16:23 -05:00
devin ivy
604f4d2ab2
Seeds for testing server views (#222)
* Create basic seed for server based on view tests

* Utilize basic seed in view tests

* Split-up pds view tests
2022-10-07 18:11:34 -04:00
Daniel Holmgren
92ab06941d
Tests for did conversion (#225)
added tests for did conversion
2022-10-07 16:45:45 -05:00
devin ivy
5a9c0e9058
Fix dev-env build, account for server's hbs templates (#218)
* Fix dev-env build, account for server's hbs templates

* Remove out-of-date comments
2022-10-06 16:02:17 -04:00
devin ivy
e30c45d38a
Enforce single user per email, usernames case-insensitive (#217)
* Schemas and scaffolding for reset password methods

* Initial handler for todo.adx.requestAccountPasswordReset

* Initial handler for todo.adx.resetAccountPassword

* Implement server mailer

* Configure server for mailer and testing w/ mailer

* Test happy path of pass reset, fix reset bug

* Update lex to fix types bug for requestAccountPasswordReset

* Fix handlebars reference to config getters

* Test some negative password reset flows

* Minor cleanup to pass reset

* Tidy handlebars file with prettier, supporting double-quotes for html

* Fix esbuild of server for mailer templates, fix test issue

* Misc tidying for password reset

* Misc tidying for password reset

* Enforce single user per email, test unique email and username

* Remove resolved TODO re: duplicate emails
2022-10-06 14:28:20 -04:00
devin ivy
59f18ac707
Forgot password methods (#216)
* Schemas and scaffolding for reset password methods

* Initial handler for todo.adx.requestAccountPasswordReset

* Initial handler for todo.adx.resetAccountPassword

* Implement server mailer

* Configure server for mailer and testing w/ mailer

* Test happy path of pass reset, fix reset bug

* Update lex to fix types bug for requestAccountPasswordReset

* Fix handlebars reference to config getters

* Test some negative password reset flows

* Minor cleanup to pass reset

* Tidy handlebars file with prettier, supporting double-quotes for html

* Fix esbuild of server for mailer templates, fix test issue

* Misc tidying for password reset

* Misc tidying for password reset
2022-10-06 14:24:26 -04:00
dholms
904fa6448d SDK -> Resolver 2022-10-05 18:07:34 -05:00
Daniel Holmgren
7233ce070f
Cleanup (#210)
* messing around w pino

* rework locals & add pino-http

* subsystem logging off env vars

* remove auth tokens from logs

* lots of logging

* re-enable tests

* update enabled check

* fix subsystem logging

* some quick cleanup
2022-10-05 17:55:08 -05:00
Daniel Holmgren
b26eb75277
Logging (#209)
* messing around w pino

* rework locals & add pino-http

* subsystem logging off env vars

* remove auth tokens from logs

* lots of logging

* re-enable tests

* update enabled check

* fix subsystem logging

* pr fixups
2022-10-05 17:47:00 -05:00
devin ivy
685c656729
Fixes for handling of posts vs. reposts in feeds (#208)
* Fix ordering of reposts in feed

* Preserve separate posts and reposts in feeds

* Include users' own posts in their feed

* Remove unused dep from test file
2022-10-04 17:52:44 -04:00
Daniel Holmgren
b3c60696da
Invite codes (#204)
* implement invite codes

* siwtch to admin basic auth & tests

* signular InviteCodeUse for table name

* comment

* pr fixups

* tx TODOs
2022-10-04 15:13:55 -05:00
Paul Frazee
11928993dc
Make the cursor field a required output in get*Feed (#206) 2022-10-04 12:28:49 -05:00
devin ivy
d64e4f6a75
Support firehose feed algorithm (#205)
* Split getFeed() into getAuthorFeed() and getHomeFeed()

* Remove unused getFeed codegen file

* Unify queries for getHomeFeed and getAuthorFeed

* Factor out common query parts for building feeds

* Test firehose and default feed algorithm

* Fix api tsconfig reference
2022-10-04 12:18:34 -05:00
Paul Frazee
c70911ab20
Fix: support multiple post entities per post (#203) 2022-10-04 09:29:07 -05:00
Paul Frazee
2cb0e8d36c
Feed bugfixes (#202)
* Fix feed test

* Fixes to repost information in getFeed
2022-09-30 11:57:38 -05:00
Daniel Holmgren
8b0884a315
Notifications (#200)
* notification methods on tables

* process notifs

* last seen

* notifs xrpc methods

* seen notifs route

* register route

* notifications tests

* rm unused db method
2022-09-30 11:09:42 -05:00
Paul Frazee
fe2bf32874
Fix bug in test name registry update (#199) 2022-09-29 14:23:24 -05:00
Paul Frazee
8ed0000c28
Fix: query correct table to get like state in getFeed and getPostThread (#198)
Fix: query correct table to get like state in getFeed and getPostThread
2022-09-29 14:17:58 -05:00
dholms
5392501847 Updating some dev tools 2022-09-29 13:11:03 -05:00
Daniel Holmgren
6bb82894e3
Add PLC server to dev-env (#197) 2022-09-28 19:30:33 -05:00
Daniel Holmgren
670c419c18
Plc auth lib integration (#195)
* parsing dids to something we understand

* revamping did-sdk with did-resolver

* testing

* fixing some thigns & finishing tests

* fix type error & remove old test

* wip

* create did on server & add email

* check username available & store email

* better test utils

* remove did:test

* allow authstore to sign for controlled dids

* plc ucan integration wip

* wip

* finished auth lib refactor

* integrating to repo & server

* couple fix ups

* couple more fixups

* disable dev-env test

* comments
2022-09-28 17:52:18 -05:00
Daniel Holmgren
ee68e6977b
Pds Did integration (#190)
* parsing dids to something we understand

* revamping did-sdk with did-resolver

* testing

* fixing some thigns & finishing tests

* fix type error & remove old test

* wip

* create did on server & add email

* check username available & store email

* better test utils

* remove did:test

* allow authstore to sign for controlled dids
2022-09-28 17:36:53 -05:00
Paul Frazee
6b47ad7622
Add better cursor pagination behavior to getFeed (#196) 2022-09-28 17:33:18 -05:00
Paul Frazee
a21417ba91
Improved XRPC error protocol (#194)
* Add lexicon doc

* Update error-handling spec

* Implement new error-behaviors in xrpc and xrpc-server packages

* Update lexicon and lex-cli packages to add xrpc error behaviors

* Generate new API and test an error behavior
2022-09-28 11:55:43 -05:00
Paul Frazee
31bd54e69c
More client integration updates (#192)
* Update dev-env to use new authorization system

* Add did and name to createSession response body

* Handle not found in getProfile api

* Stop putting a timestamp in the generated code, ffs

* Add todo.adx.getAccountsConfig

* Remove timestamps from schema codegen
2022-09-27 12:32:17 -05:00
Daniel Holmgren
257b906bdf
Did sdk revamp (#189)
* parsing dids to something we understand

* revamping did-sdk with did-resolver

* testing

* fixing some thigns & finishing tests

* fix type error & remove old test
2022-09-27 09:33:30 -05:00
Daniel Holmgren
100319f59e
Bump ucans to stable version (#191)
Bump UCANs to stable version
2022-09-26 18:55:54 -05:00
Paul Frazee
dcba744dab
Various fixes related to integration with client software (#188)
* Refactor xrpc to use native fetch and be able to replace it (for react native)

* Factor out uri package

* Fixes to getPostThread

* Manually construct indexedAt and receivedAt timestamps to ensure theyre properly encoded
2022-09-23 11:33:26 -05:00
Daniel Holmgren
1d136eba17
JWT auth (#187)
* setting up auth

* sticking auth on all the routes

* fixed up tests

* add util.getAuthstore fn

* rm fixed TODO

* use nodes native scrypt
2022-09-23 11:11:16 -05:00
Paul Frazee
3a4e4c0759
Dev env tooling, server package restructuring, add did:test and *.test rules (#186)
* Add ServerConfig class

* Update todo.adx.resolveName spec to make name param optional

* Add .test username and did:test behaviors

* Document name/did resolution and the .test/did:test behaviors

* Move did:test registry into server config to share across servers in the dev env

* Rework the server package structure to make it easier to include as a library

* Fix dev-env build issues

* Give better error when repo API is called on an invalid DID

* Add mock data environment to dev-env
2022-09-22 13:34:46 -05:00
Paul Frazee
735e201028
Rename the generated-code folder in the server package (#185) 2022-09-21 11:56:43 -05:00
Daniel Holmgren
55e6e54fcf
Did PLC (#181)
* validate the signatures
add the id to the did doc
pid from sha256 of oragin doc
added the tick to diffs extractor
added the key authorisation
add lots of types
add routes for calling the aic
refactor to jest based repo structure

* WIP

* migrate the tests

* more test fixes

* test.

* replace snake case with camel case

* moved to use isolated server for aic

* remove the aic test from the PDS

* clean up some changes outside the aic folder

* fix CaS for saving ticks

* rearange the folder

* missed  one

* cleaned up aic package.json

* broken build

* test passing

* prettier

* pid tests refactor

* fix eslint/prettier setup, add eslint-config-prettier

* misc linting and style tweaks in aic

* minor touchups to aic sign/verify

* tidy error-handling in aic

* tidy-up crypto init in aic

* add aic route for getting a did doc, update /tick routes

* knex -> typeorm

* refactor crypto

* error handling & dependencies

* switching op model & some other cleanup

* change db to use operations

* add cid references to ops & ensure no branching in DB tx

* cleaning up

* rotues

* working on client & tests

* document & oepration tests

* server tests

* cleanup

* remove lobby page & static

* formatting did documents

* aic/pch -> plc

* fixing up some doc semantics

* tooling/test server

* added recovery window

* tx mutex

* test on document formatting

* take validate op out of write tx

* error handling

* readme

* couple bug fixes

* quick readme edit

Co-authored-by: Aaron Goldman <aaron@aarons-mbp.lan>
Co-authored-by: Aaron D Goldman <aaron@blueskyweb.xyz>
Co-authored-by: Devin Ivy <devin@bigroomstudios.com>
2022-09-20 21:28:00 -05:00
Paul Frazee
f67626e5f2
Unify schemas under lexicon package (#183)
Unify schemas under lexicon package; remove schemas package; rename xrpc-cli to lex-cli
2022-09-20 18:08:30 -05:00
Paul Frazee
1425b0cd4f
New XRPC-based Server and API implementations (#180)
* Add xrpc-cli package; implement 'new' and 'gen-md' commands

* Add gen-ts-obj to xrpc-cli

* Add gen-api command to xrpc-cli

* Add gen-server to xrpc-cli

* Update codegen from xrpc-cli to run prettier and include a banner comment

* Use 'void' instead of 'undefined' for return type

* Skip invalid schemas when generating code

* Rewrite schemas package to use new simplified schema

* Output type definitions in nested folders

* Add record schema code-generation

* Change xrcp client call() signature to be easier to use

* Update client codegen to match new xrpc call signature

* Fixes and improvements to generated record APIs

* Update generated api to use per-service instances

* Add xrpc-cli readme

* (WIP) Rewrite server to use xrpc

* Add missing references to server/tsconfig.json

* Replace api implementation with xrpc generated client

* Add /schemas folder with todo.adx

* Add /schemas/todo.social

* Update api package to include todo.social xrpc and adx schemas

* Update server package to include todo.social xrpc and adx schemas

* Fix some schema definitions

* (WIP) Work toward xrpc update of server

* Update auth package to use nsids for collections

* Update repo package to use nsids for collections

* Update AdxUri to use nsid collection IDs

* Bring server CRUD tests to passing state

* Move API to simpler xrpc call signature

* Apply fixes and improvements to record apis

* Update to new generated api with per-service instances

* Rename the todo.social.* view methods

* Rewrite the views system to use xrpc

* Remove paths accidentally added to tsconfig

* Remove microblog package

* Add simple API readme

* todo about deduping ucan prfs

* increase postThread default depth to fix failing test. & fix USE_TEST_SERVER config

Co-authored-by: dholms <dtholmgren@gmail.com>
2022-09-20 17:39:05 -05:00
Paul Frazee
c15f71b17c
XRPC CLI (code and doc generator) (#179)
* Add xrpc-cli package; implement 'new' and 'gen-md' commands

* Add gen-ts-obj to xrpc-cli

* Add gen-api command to xrpc-cli

* Add gen-server to xrpc-cli

* Update codegen from xrpc-cli to run prettier and include a banner comment

* Use 'void' instead of 'undefined' for return type

* Skip invalid schemas when generating code

* Rewrite schemas package to use new simplified schema

* Output type definitions in nested folders

* Add record schema code-generation

* Change xrcp client call() signature to be easier to use

* Update client codegen to match new xrpc call signature

* Fixes and improvements to generated record APIs

* Update generated api to use per-service instances

* Add xrpc-cli readme
2022-09-20 17:13:54 -05:00
Paul Frazee
6b72878319
Rename fedrpc to xrpc; implement xrpc and xrpc-server packages (#178)
* Rename fedrpc to xrpc; implement xrpc and xrpc-server packages

* Add json-schema validation to xrpc

* Add xrpc and xrpc-server to base tsconfig.json

* Fix type signature
2022-09-19 15:58:09 -05:00
Paul Frazee
f5b83b947b
Add NSID package (#177)
* Add NSID package

* Move NSID functions to static methods

* NSID - Expect authorities to be at least 2 segments long
2022-09-19 15:53:39 -05:00
dholms
275c3df202 lock uint8arrays at 3.0.0 2022-09-13 15:16:57 -05:00
Paul Frazee
6283330c3c
Deprecate did:ion, replace did:key implementation, bring dev-env to a working state (#172)
* Remove 'key manager' service from dev-env

* Remove did:web service from dev-env

* Update dev-env to use new server api

* Replace node-fetch with axios

* Remove did:ion from did-sdk

* Disable did:key implementation in did-sdk and remove difficult upstream deps

* Fix dev-env build

* Fixes to dev-env and add user() env function

* Switch to a purely js-function REPL in the dev-env to simplify
2022-09-13 15:01:09 -05:00
dholms
c9a6609b93 fixed up common & repo tests 2022-09-13 11:15:56 -05:00
Paul Frazee
d247f7c218
Fix: remove references to some removed files (#170) 2022-09-13 10:59:14 -05:00