Commit Graph

2335 Commits

Author SHA1 Message Date
devin ivy 8826ada589 Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold 627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
bnewbold f08c08a675 MST review and tests (#556)
* mst: more test coverage

In particular, ports the older fanout=32 corner-case tests to fanout=16.

I generated new paths with correct level and ordering, but haven't
actually verified that the resulting structures have the shape of the
ASCII diagrams in text.

* mst: additional comments, including TODO questions

* mst: default new MST layer to 'null' not 0

Looks like all current call sites of this 'create()' function override
the layer already. This is just making it explicit that the default,
null, means the layer has not been computed yet, distinct from being
layer=0.

* mst: remove one TODO and reformat another

From review.
2023-02-23 10:47:16 -08:00
Daniel Holmgren f29c08f0bb Bump package versions (#578)
* bump pkgs

* fix package
2023-02-22 19:54:21 -06:00
devin ivy ca87aeb93f Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren 136b4b5933 Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
devin ivy 24fd3b17e2 Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren c2c87c3226 Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
devin ivy 3a6d785ae0 Fix PDS build, updating main to dist in api package (#566) 2023-02-18 16:15:19 -05:00
Daniel Holmgren 66f731ecee Repo sync semantic changes (#563)
* switch to cursor & only one event type

* quick bugfix

* no dangling async stuff

* didnt rebuild lex-cli
2023-02-17 17:33:31 -05:00
Daniel Holmgren 20a35b31c1 Add com.atproto.sync.getBlocks XRPC method (#564)
* add sync getBlocks method

* oops didnt build lex-cli
2023-02-17 17:18:13 -05:00
Devin Ivy 3f6482d0d2 Tidy snapshots 2023-02-16 16:32:17 -05:00
Daniel Holmgren 6c4f6a27f3 Bugfix: Flaky readFromGenerator test (#560)
fix flaky test
2023-02-16 14:34:26 -06:00
Daniel Holmgren d4b1262f28 DNS handles (#555)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* basic support for dns resolution on handles

* handle resolution

* fix up account create

* tsconfig

* workshop handle constraints

* bring pds up to speed w handle changes

* change dns subdomain

* another lil test

* bugfix

* update dns record format

* typo
2023-02-16 13:59:56 -06:00
bnewbold e3cd9c23a1 Small PDS and XRPC server fixes (#550)
* pds: fix logging of Bearer auth subject (DID)

* xrpc-server: handle missing and mangled Content-Type better

Eg, for blob uploads.

Also adds a test for incorrect mimetype supplied for a regular endpoint
(one not allowing "*/*").

* skip failing test

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-02-16 13:59:34 -06:00
devin ivy 9c119fb9c6 Allow unsetting profile images and description in updateProfile (#554)
* Add support for nullable on lex objects

* Support clearing fields in updateProfile lexicon

* Support clearing fields in updateProfile method
2023-02-16 14:32:25 -05:00
Paul Frazee 5737e653e3 Bump @atproto/pds@0.0.3, @atproto/api@0.1.2 (#558)
* @atproto/pds@0.0.3

* @atproto/api@0.1.2
2023-02-16 12:29:01 -06:00
devin ivy 850b54fad6 Add following state to profile/actor views, reorg views (#539)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Cull actor/profile view lexicons down to three variants

* Map viewer in profile view, mark todos, lex tweak

* Apply basic profile view to methods, lex fix

* Switch searchTypeahead to actor-with-info view

* Switch getVotes to actor-with-info view

* Switch getMutes to actor-with-info view

* Switch getFollows and getFollowers to actor-with-info view

* Switch getRepostedBy to actor-with-info view

* Switch feed and notification views to use updated actor-with-info view
2023-02-16 11:40:05 -05:00
Daniel Holmgren 57da2270b9 Update handles (#547)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* Fix did pkey down migration

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-15 14:50:29 -05:00
devin ivy a24be34c27 Notify all ancestors of reply, optimize getPostThread method (#545)
* Initialize post_hierarchy index

* Build post_hierarchy index from posts

* Power getPostThread method with post hierarchy index

* Generate ancestor reply notifications, fix tests and down migration

* Generate notifications for all reply ancestors
2023-02-15 14:16:35 -05:00
Daniel Holmgren ac48330fdb User table fixes (#536)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing
2023-02-15 14:15:32 -05:00
devin ivy 807e2a783c Apply lexicon defaults (#552)
* Thread values w/ applied defaults through validation, use in xrpc-server

* Adjust codegen to reflect xrpc defaults
2023-02-14 21:20:20 -05:00
devin ivy 751f16527d Use DAG-CBOR for XRPC streaming (#546)
* Use canonical dag cbor for xrpc streaming

* Remove undefineds from subscribeAllRepos for dag-cbor
2023-02-13 15:08:26 -05:00
bnewbold 19c54fe55b Merge pull request #537 from bluesky-social/bnewbold/dev-misc
some small low-priority patches
2023-02-12 20:23:05 -08:00
devin ivy 9a7b2c39e4 Mod action and report params as integers rather than floats (#544) 2023-02-10 16:40:32 -05:00
bnewbold 6ec452450a pds sql: add indices for feed and post aggregations (#538)
* pds sql: add indices for feed and post aggregations

Seems like aggregations/stats/counts are likely to get changed up in the
near term future, but in the very short term, these simple index
additions yielded non-trivial speedups even at small database sizes, in
my laptop testing.

* pds migration: fix index drops in 'down'
2023-02-09 21:01:05 -05:00
Paul Frazee d252fd3e95 Fix knownValues for reasonType lexicon (#527) 2023-02-09 20:59:32 -05:00
bnewbold a3c1105aca handles: reserve some top-100 handles; prevent new 2-char handle registrations (#531)
* handle: disallow new 2-char handle registration

This should not impact existing handles, only new registrations.

We discussed blocking these to prevent possible future URL path
collisions with, eg, ISO country or language codes.

* handle: reserve some more famous handles

This isn't the full list of thousands of twitter handles that have more
than 5 million or more than 1 million followers. Instead, it is only the
top 100 or so, plus a handful of additional top-50 instagram accounts (when
different), and a couple recent famous figures. Pretty subjective.

* handle: overall and segment lengths; enforce ASCII; test dashes

The 'address' package we are using to check domains doesn't seem to do
any overall length checks. The values and examples i'm testing here are
based on general domain limits. Technically a web domain could be 255,
it is email domains limited to 253 chars, but that seems like a
reasonable value to cut off at.
2023-02-09 20:58:42 -05:00
bryan newbold 7b5f8906d1 plc: run dotenv before any other imports
This fixes, eg, logging configuration via .env variables.
2023-02-09 16:40:28 -08:00
bryan newbold 8b803e013c pds: run dotenv before any other imports
This fixes logging configuration via env vars set in .env file
2023-02-09 16:40:14 -08:00
bryan newbold d545b3195e pds: add missing 'license' to package.json 2023-02-09 16:40:06 -08:00
bryan newbold 71d6ce8c5b pds and plc: example .env files for development 2023-02-09 16:22:20 -08:00
Daniel Holmgren 7d92eb0513 Feature branch: streaming repo sync (#542)
* Scaffold xrpc-stream

* Write and test stream frames

* Write and test stream server

* Test bad stream frame parsing cases

* Proof of concept streaming on xrpc-server

* Test bad streaming endpoint

* Make frame stream to fix buffering frame bytes together

* reorg

* some lex-cli things

* better codegen for subscriptions

* scaffolding repo subscriptions

* wip

* remove repo ops

* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* basic sequencer

* some refactoring

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* subscribeRepo outbox

* some cleanup

* wip

* wip

* bugfixin

* only send msgs after tx is committed

* better handle event-emitter -> generator

* max buffer size

* cleanup

* Sync-up xrpc subscriptions with current iteration of the spec

* Allow missing encoding for subscriptions

* track sequencedAt & eventType as well

* Resolve streaming type codes, streaming tests

* Fix interface for hooking into http server for streaming, fix stream routing

* Minor reorg how streaming is enabled in xrpc-server

* Server codegen for xrpc subscriptions, supporting types in xrpc-server

* fix up buffer overloading

* Rename data frame to message frame, rename subscription lex output to message

* Move body param first for streaming frames

* Tidy

* clean up evt types

* buff up tests

* missed merge conflict

* new schema

* blobs on subscriptions

* rm genned client subscription methods

* backfill limits

* testing subscription route & quick outbox bugfix

* fix up migration

* cascade on delete

* comments & naming

* fix dev env

* delete seqs on account deletion

* tidy

* fixing things up with db notify system for schemas

* fix duplicates in outbox

* tidy

* Break out of stream handler when socket closes

* fixing up some timing issues

* tidy

* terminate ws after using

* bump up timer on async reader

* fixing up NOTIFY in txns

* pr feedback

* pr bugfixes

* make order asc explicit

* bringing tests up to speed w atpagent

* bump up max listeners on sequencer

* increase timeouts for reading from generators

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-09 16:17:06 -06:00
devin ivy 5554d80d20 Containerize PLC build (#533)
* Add plc service

* Setup dockerfile for plc

* add build-and-push GitHub Action

* Try making the tag plc:sha-

* try with the tags in the correct place

* add push to push!

* use a long sha for the container tag

* only build-and-push containers on main branch

* only build-and-push containers on main branch

---------

Co-authored-by: Jake Gold <jake@blueskyweb.xyz>
2023-02-09 16:57:42 -05:00
devin ivy 262d63d852 Add utility for testing PLC (#540)
Add script to plc for did creation
2023-02-09 15:33:28 -05:00
Paul Frazee 3dcfc047cf Publish @atproto/api@0.1.1 (#541)
@atproto/api@0.1.1
2023-02-09 13:37:43 -06:00
Daniel Holmgren 7ac555bd41 Allow syncing a commit range (#535)
allow syncing a commit range
2023-02-09 11:27:44 -06:00
devin ivy 71d812abe6 Get profiles xrpc method (#532)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Move ArrayEl type helper into common
2023-02-09 11:11:46 -05:00
Daniel Holmgren a1240f0a37 Handle SIGINT in postgres script (#534)
test:pg sigint
2023-02-08 15:45:13 -06:00
Paul Frazee e8b8d081ae Bump @atproto/xrpc@0.0.4, @atproto/api@0.1.0 (#530)
* @atproto/xrpc@0.0.4

* @atproto/api@0.1.0
2023-02-07 15:35:09 -06:00
Paul Frazee 2242e8a313 Refactor @atproto/api to the AtpAgent interface (#529)
* Refactor @atproto/api to use the simplified AtpAgent API

* xrpc package: Export the defaultFetchHandler to reuse in api

* api package: Use the defaultFetchHandler defined in xrpc

* Update all usages of the api for the new AtpAgent

* Clear promise on thrown codepath

* Avoid updating the atpagent session until ready to return
2023-02-07 15:30:29 -06:00
Devin Ivy 0c58a937c9 Fix aws package exports 2023-02-06 19:39:56 -05:00
devin ivy b91e6e96b7 Incorporate blobs into moderation actions (#511)
* Lexicon updates for moderating blobs

* Add tracking of blob takedowns to moderation system

* Handle blobs in record takedown reversal

* Apply blob takedowns w/ image invalidation

* Test blob takedowns

* Implement cloudfront image invalidator

* Update lexicons for current moderation action on records and blobs

* Fix migration for postgres

* Enforce single current action per repo and blob, present current action on repo and blob

* Test single active mod action per repo, record, and blob

* Continue to resolve handles for moderated repos
2023-02-06 11:44:26 -05:00
Daniel Holmgren 64f5b4a022 Refactor repo indexing (#521)
* refactor repo indexing

* bit more
2023-02-03 17:11:39 -06:00
devin ivy 216e434c73 Handle client abort on image upload (#524) 2023-02-03 11:19:45 -05:00
Daniel Holmgren 55234b4b0d Account deletion hotfix (#520)
* account deletion hotfix

* dont end pool

* set array val
2023-02-02 16:47:14 -06:00
Daniel Holmgren 396698f218 Filter out self & follows from suggestions (#498)
dont suggest self or follows
2023-02-02 16:48:41 -05:00
Daniel Holmgren 134920d872 Database notify/listen (#514)
* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* poor guy cant spell
2023-02-02 15:04:38 -06:00
Paul Frazee 817ff92c66 Bump @atproto/api@0.0.8 (#517) 2023-02-02 14:30:03 -06:00
Daniel Holmgren 773f9e304d Account deletion (#488)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* some lexicons

reworking routes

request deletion flows

delete actor rows

migration for user-partitioned-cids

move creator to be on ipld_block

migration tests

* delete records & repos

* delete blobs

* hook it up in route

* pettier ignore email templates

* testing & bugfixes

* testing blobs & bugfixes

* pr feedback

* make deletion test more robust

* change out handle for did on account deletion

* small cleanup

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-02-02 12:34:00 -06:00