195 Commits

Author SHA1 Message Date
Daniel Holmgren
743eaf1cf0
Move handle resolution to .well-known (#1048)
* move handle resolution to .well-known

* required handle on resolveHandle

* rm test

* tidy

* tidy

* fix up appview

* missing await

* atproto-handle -> atproto-did

* shift did & handle resolution to new identity package

* fix up network mocks

* fix up another test

* one more

* drop lex comment

* rm handle param

* Update packages/identity/src/handle/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* still temporarily support xrpc handle resolution

* typo

* ensure return value is a string

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-05-25 19:04:10 -05:00
dholms
85bcd18a7b v0.1.11 2023-05-24 17:43:02 -05:00
Daniel Holmgren
8059e07d8a
Custom feeds (#1001)
* actor type in profiles

* skeleton lex

* more schemas

* first pass at feed composition

* cleanup services

* more tidying

* fix up tests

* bookmarking schemas

* make feeds records instead of actors

* some cleanup

* fleshing out feeds as records

* wip

* working into cfg & dev-env

* fix appview test util

* bookmarking + views

* tidy

* add displayName

* db migration

* wip test

* blob processing

* first test working!

* Sync-up snapshots

* wip

* Fix pds ctx type

* Sync lexicons on pds and api

* wip

* fully refactor

* Prep for open union in reply data in bsky appview, sync-up snapshots

* msgs & ports

* Make pds did cache sqlite-friendly

* Start scaffolding more feedgen tests

* revamp appview tests

* revamp pds tests

* merge in dev env utils

* ensure plc close

* add feed gen to dev-env

* second feed gen

* named feeds in mock

* fix issue with dev-env pds appview

* update bsky snaps

* Flesh-out some feed gen tests

* Fix feed gen reply test

* Test feed bookmarking

* update skeleton defs

* merge

* Sync-up pds with updated feed skeleton contract

* Fix feed bookmark viewer state

* add feed embeds

* Test inaccurate feed gen repost

* Tighten some error behaviors around feed gens

* generator view in record embed

* test

* bookmarked/subscribed -> saved

* updated snaps

* name & descript on alices feed

* add like counts

* proposed lex

* tweak

* fetch feed generator details

* add describeFeedGen to dev-env

* adds tests for getFeedGenerator

* Update lexicons/app/bsky/feed/getFeedGenerator.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* update describe

* add cid to feed gen view

* add alice feed gen avatar

* generator displayName required

* update snaps

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* cleanup lex

* mount route

* Starter bsky algos (#1046)

* implement some starter bsky algos

* add team dids

* tidy

* add quotepost check to whats hot classic

* tests

* carve out feedgen section of pds

* apply limits

* add describeFeedGenerator route

* Whats hot new (#1060)

* map out new whats hot

* only available in postgres

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* build algos branch

* remove relative imports to @atproto/api

* update hot classic name

* rm build

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>

* v0.3.6

* v0.3.7

* v0.1.10

* update name of skyline

* build branch

* aws build

* getFeedGenerators test

* For now only use new replyRef union for custom feeds, for compat

* Don't use replyRef union on new feeds yet

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-05-19 18:31:51 -04:00
dholms
2753a5a739 v0.1.8 2023-05-11 16:30:36 -05:00
dholms
7f72478e15 v0.1.7 2023-05-11 16:28:48 -05:00
dholms
5fd5c869ea bump pds version 2023-05-11 15:01:01 -05:00
Wes Todd
b01e47b617
fix: updated repository field for packages (#994) 2023-05-08 11:38:42 -04:00
Wes Bos
958c9ef6b3
add repository field to package.json (#895) 2023-05-01 20:43:32 -05:00
Paul Frazee
01b5971907
api@0.2.11, pds@0.1.5 (#925)
* @atproto/api@0.2.11

* @atproto/pds@0.1.5
2023-04-28 19:41:01 -05:00
Daniel Holmgren
68004c617f
Appview proxy (#827)
* fix up a couple of tsc errors in app view merge

* wip

* simple proxy

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* wip

* copy proxied tests & add mutes to getFollows/getFollowers

* tidy & add mutes to likes

* more routes + getAuthorFeed tests

* more testing

* tests for feed views

* thread testing

* finished tests for threads

* temporarily skip some tests

* cleaning up & updating test names

* separate db schema for appview

* rearrange

* typo

* add notifications

* re-enable notifs on pds appview

* update schemas

* updated some bsky snaps

* wip

* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* fix up & move to appCtx

* neat its working

* update bsky tests to new auth

* rm unused pds config var

* tidy

* check exp in seconds

* cache dids in postgres

* add migration & did-cache

* start tests

* couple helpers around cache invalidation

* fix expired check

* wip

* change cache semantics

* did cache testing

* do some cache revalidation in indexing

* fix config

* fix issue w did-resolver test-env prototype

* use map instead of record

* stale + expired

* tests

* clear entry method

* fix up build

* expired dids

* clear missing dids

* better verify payload

* bump test timeout

* fix notifs test

* fix up proxied actor search tests

* update snaps to include labels

* fix dev env

* fix up moderation route auth

* fix more auth headers

* fix auth on getPosts

* increase jest timeouts

* fix snaps
2023-04-25 15:26:14 -05:00
Daniel Holmgren
2c24cf6c15
Bump node-postgres version (#846)
* Bump node-postgres version

* fix package.json things
2023-04-19 19:06:56 -05:00
Devin Ivy
3f632d8bdb v0.1.4 2023-04-12 18:09:48 -07:00
Daniel Holmgren
8b1da9fed8
Labeling (#789)
* take notifs off of message queue

* fix order of notif handling

* table

* reintroducing message queue

* setting up user

* added labeler classes

* tidy + hook up labeler to config / service start

* hooking up p-queue to labeler

* rip out message queue

* drop mq tables

* rm stream consumers

* keyword labeler

* tidy + some tests

* work labels into views

* update snaps

* labeled images in dev-env

* snaps

* labeler tests

* more labels

* Update packages/pds/src/api/app/bsky/notification/listNotifications.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/dev-env/src/mock/index.ts

* Update packages/dev-env/src/mock/index.ts

* Update packages/pds/src/labeler/util.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/pds/tests/labeler/hive.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback + migration change

* fix lower case keywords issue

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-12 14:34:18 -05:00
Paul Frazee
831b7dade9
api@0.2.4, pds@0.1.3 (#765)
* @atproto/api@0.2.4

* @atproto/pds@0.1.3
2023-04-05 18:12:14 -05:00
Paul Frazee
47cb6402cd
Bump pds@0.1.2, api@0.2.3 (#762)
* @atproto/pds@0.1.2

* @atproto/api@0.2.3
2023-04-04 17:05:51 -05:00
Paul Frazee
1014938520
Pkg bumps after #756 (#759)
* @atproto/0.2.2

* @atproto/pds@0.1.1
2023-04-04 14:33:23 -05:00
Paul Frazee
9bad971e21
Post lex refactor pkg publishes (#738)
* @atproto/common-web@0.1.0

* @atproto/identifier@0.1.0

* @atproto/crypto@0.1.1

* @atproto/common@0.2.0

* @atproto/repo@0.1.0

* @atproto/lexicon@0.1.0

* @atproto/xrpc@0.1.0

* @atproto/xrpc-server@0.1.0

* @atproto/lex-cli@0.1.0

* @atproto/pds@0.1.0

* @atproto/dev-env@0.1.0

* @atproto/api@0.2.0
2023-03-31 14:54:25 -05:00
bnewbold
228431edf3
lexicon-level identifier validation helpers and test cases (#576)
* hardening: permissive tests for handles

* identifers: permissive tests for NSIDs

* identifiers: add DID validation

* identifiers: wrong about domain syntax (duh, 4chan.com)

In short, labels can start with digits except for the final TLD part.
I think that is all for "DNS domains as hostnames which are actually
used in the real world", though there might be other modern things.
Underscores are obviously allowed in DNS for things like SRV records,
but I don't think as "regular hostnames".

Not sure we want this for NSIDs, so not updating that code or tests yet.

* identifiers: ATURI test corner cases

* identifiers: remove TODOs (moved to formalism doc)

* identifiers: small comment typos and corrections

* move around & integrate indentifier hardneing

* fixed up some tests

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-03-17 17:16:31 -05:00
Daniel Holmgren
8dfcb4f996
Integrating PLC lib (#607)
* integrating new plc lib

* patching up did-resolver

* buffing up pds tests

* didResolver on ctx & plc in postgres

* bring dev env up to date

* re-add extension for linting

* tidy

* use current env vars

* pr feedback
2023-03-04 19:43:59 -06:00
Paul Frazee
5e5d3df615
Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
Daniel Holmgren
d4b1262f28
DNS handles (#555)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* basic support for dns resolution on handles

* handle resolution

* fix up account create

* tsconfig

* workshop handle constraints

* bring pds up to speed w handle changes

* change dns subdomain

* another lil test

* bugfix

* update dns record format

* typo
2023-02-16 13:59:56 -06:00
Paul Frazee
5737e653e3
Bump @atproto/pds@0.0.3, @atproto/api@0.1.2 (#558)
* @atproto/pds@0.0.3

* @atproto/api@0.1.2
2023-02-16 12:29:01 -06:00
bryan newbold
d545b3195e pds: add missing 'license' to package.json 2023-02-09 16:40:06 -08:00
Daniel Holmgren
134920d872
Database notify/listen (#514)
* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* poor guy cant spell
2023-02-02 15:04:38 -06:00
Paul Frazee
5821a671cf
Bump @atproto/pds@0.0.2 (#507) 2023-01-31 09:01:57 -06:00
Daniel Holmgren
75e14ae102
Feature branch: storage & sync revamp (#446)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* fix migration type

* switch pk indexes

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-01-26 18:09:55 -06:00
Paul Frazee
e44c98e717
Update module publish scripts (#478)
* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts
2023-01-24 10:11:20 -06:00
Daniel Holmgren
f0ee51f151
App context (#424)
* introducing appcontext & pds class

* revamp rest of server + auth

* revamp tests

* add to dev-env

* fix api test

* add start method to pds

* hook up terminator before

* PLC AppContext (#425)

* convert plc server to app context

* add start method to plc

* hooke up terminator on server setup

* fix test

* cleanup & pr feedback

* fix up one more test
2022-12-16 11:23:36 -06:00
devin ivy
809c9a38f4
Message queue blocking and graceful teardown (#421)
* Fix db blocking when processing from the message queue

* Make message queue concurrency test not pg-only

* Rename test

* Dedupe test

* Tidy
2022-12-15 13:54:05 -05:00
Daniel Holmgren
9598a22ef5
Cleanup dependencies (#410)
* cleaning up & deleting unused deps

* oops we were using npm-run-all
2022-12-14 12:31:28 -06:00
devin ivy
4bc9af4c37
Downgrade file-type dep version, fix build (#407)
Downgrade file-type dep since it is now ESM-only
2022-12-12 15:53:05 -05:00
Paul Frazee
3636bd19a8
A couple lexicon-related fixes (#405)
* Fix the generated type for encoding when the value is */* (close #391)

* Add 'toKnownError' test and fix named error for updateProfile

* Improve blob errors
2022-12-12 13:48:59 -06:00
Daniel Holmgren
91215f4f67
Verify content type of files (#395)
check content type of files
2022-12-08 17:00:11 -06:00
devin ivy
d19fbe5473
Image processing interface (#378)
* Setup interface for image processing

* Implement getInfo() on SharpImageProcessor

* Reorganize pds image processing code

* Implement initial resize() on SharpImageProcessor

* Test sharp image processor, apply a couple fixes

* Tidy

* Implement initial pds image uri builder/signer/verifier

* Initial implementation of image processing server

* Update node types for node v18

* Add disk caching to pds image service

* Test pds image process caching, tidy

* Move pds image processor away from an interface

* Add sharp to pds build externals

* Move away from xrpc-server errors to http-errors for pds image server

* Tidy pds image exports

* Tidy
2022-12-07 09:44:34 -06:00
devin ivy
9d29ef96f6
Well-defined pagination in PDS (#365)
* Implement tuple keyset cursor to break pagination ties, apply to feeds and assertions

* Tuple keyset cursor on rest of views

* Tidy keysets for postgres

* Add default implementation to Keyset to remove some boilerplate

* Simplify Keyset interface

* Remove dead code

* Move to new Keyset interface, incl. TimeCidKeyset

* Make pds assertion view tests deterministic

* Apply pds keyset interface to user search
2022-11-28 11:51:27 -05:00
Daniel Holmgren
6fd50bd46c
Wait for servers to close at end of test suite (#362)
wait for servers to close
2022-11-18 15:47:03 -06:00
Daniel Holmgren
c207947bd5
Stagable blockstore (#351)
* stagable blockstore + remove mutable repo

* pds wip

* nearly there

* bugfix

* assertion ordering

* del old code

* save staged if no callback

* removing level

* pr fixup
2022-11-17 09:51:36 -06:00
devin ivy
13aead171c
Support shallow builds and sourcemaps (#352)
* Enable all modules to be built shallowly, externalizing their deps

* Setup shallow builds to modify package.json main

* Hoist update-main-to-dist to separate call, fix lex-cli and plc shallow builds

* Generate sourcemaps during build

* Tidy

* Hoist new build deps up to root
2022-11-15 19:24:58 -05:00
devin ivy
d6a189f59a
Healthchecks for PDS and PLC (#309)
* Add plc healthcheck, unify plc server interface with pds

* Add pds healthcheck

* Make plc test util file consistent with other packages

* Fix types in dev-env

* Type fixes

* Tidy

* Rename healthcheck routes, place pds's under /xrpc
2022-11-04 16:26:56 -04:00
Paul Frazee
e61127111a
Split lexicons into more specific namespaces (#306)
* Segment lexicons into more specific namespaces and tighten up some naming conventions

* Update API to use new lexicons

* Update pds and dev-env to use new lexicons
2022-11-04 12:59:00 -05:00
Daniel Holmgren
7ee821312a
Handle package (#305)
* handle package

* forgot a couple things

* few more fixes
2022-11-03 17:55:33 -05:00
devin ivy
a6c5737bb5
Session management in PDS client (#288)
* Move codegen api client into its own directory

* Implement session-aware client for pds

* Test pds session client, fixes

* Use pds lexicon rather than api types where possible

* Tidy
2022-11-02 09:55:51 -04:00
Daniel Holmgren
8982e13ff4
Username validation (#282)
* user-did table

* yay fixed it

* resolve available domains from db

* serverDid + tests

* check for invalid domains

* username validation

* woopsied on merge

* fix migration

* lower indexes

* even moar validation
2022-11-01 14:30:37 -05:00
devin ivy
4a65804db5
Miscellaneous dep and build fixes (#272)
Misc dep and build fixes
2022-10-26 14:19:14 -04:00
Daniel Holmgren
d6fd69d3a3
Rename pds (#266)
* clean up old todos

* some more cleanup

* rename server -> pds
2022-10-25 15:52:46 -05:00