979 Commits

Author SHA1 Message Date
Daniel Holmgren
07d941639f
Repo ops on sync stream (#614)
* repo op schema

* repo ops on sync stream

* delete old seqs

* nullable cids

* pr feedback
2023-03-02 12:18:49 -06:00
Paul Frazee
5e5d3df615
Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
devin ivy
13ff521bf5
Ensure subscribeAllRepos syncs from cursor 0 (#611)
Ensure subscribeAllRepos allows sync from sequence 0
2023-03-01 16:25:17 -05:00
devin ivy
a9cab00ea1
XRPC subscription consumer (#603)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Yield info frames in byMessage()
2023-03-01 11:28:11 -05:00
Daniel Holmgren
6ee3a8cfda
Crawler schemas (#602)
* sketch of crawl schemas

* codgen

* hostname param

* Update lexicons/com/atproto/sync/requestCrawl.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* codegen

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-02-28 18:24:28 -06:00
devin ivy
5d2895f1ea
Running a task as a leader in the app view (#604)
Interface for picking a pds leader
2023-02-27 14:24:08 -05:00
Daniel Holmgren
bb05232720
Sync subscription tweaks (#599)
* tweaks to sync subscription

* fix test
2023-02-27 10:22:54 -06:00
devin ivy
8826ada589
Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold
627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
devin ivy
ca87aeb93f
Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren
136b4b5933
Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
devin ivy
24fd3b17e2
Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren
c2c87c3226
Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
Daniel Holmgren
66f731ecee
Repo sync semantic changes (#563)
* switch to cursor & only one event type

* quick bugfix

* no dangling async stuff

* didnt rebuild lex-cli
2023-02-17 17:33:31 -05:00
Daniel Holmgren
20a35b31c1
Add com.atproto.sync.getBlocks XRPC method (#564)
* add sync getBlocks method

* oops didnt build lex-cli
2023-02-17 17:18:13 -05:00
Devin Ivy
3f6482d0d2 Tidy snapshots 2023-02-16 16:32:17 -05:00
Daniel Holmgren
d4b1262f28
DNS handles (#555)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* basic support for dns resolution on handles

* handle resolution

* fix up account create

* tsconfig

* workshop handle constraints

* bring pds up to speed w handle changes

* change dns subdomain

* another lil test

* bugfix

* update dns record format

* typo
2023-02-16 13:59:56 -06:00
bnewbold
e3cd9c23a1
Small PDS and XRPC server fixes (#550)
* pds: fix logging of Bearer auth subject (DID)

* xrpc-server: handle missing and mangled Content-Type better

Eg, for blob uploads.

Also adds a test for incorrect mimetype supplied for a regular endpoint
(one not allowing "*/*").

* skip failing test

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-02-16 13:59:34 -06:00
devin ivy
9c119fb9c6
Allow unsetting profile images and description in updateProfile (#554)
* Add support for nullable on lex objects

* Support clearing fields in updateProfile lexicon

* Support clearing fields in updateProfile method
2023-02-16 14:32:25 -05:00
Paul Frazee
5737e653e3
Bump @atproto/pds@0.0.3, @atproto/api@0.1.2 (#558)
* @atproto/pds@0.0.3

* @atproto/api@0.1.2
2023-02-16 12:29:01 -06:00
devin ivy
850b54fad6
Add following state to profile/actor views, reorg views (#539)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Cull actor/profile view lexicons down to three variants

* Map viewer in profile view, mark todos, lex tweak

* Apply basic profile view to methods, lex fix

* Switch searchTypeahead to actor-with-info view

* Switch getVotes to actor-with-info view

* Switch getMutes to actor-with-info view

* Switch getFollows and getFollowers to actor-with-info view

* Switch getRepostedBy to actor-with-info view

* Switch feed and notification views to use updated actor-with-info view
2023-02-16 11:40:05 -05:00
Daniel Holmgren
57da2270b9
Update handles (#547)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* Fix did pkey down migration

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-15 14:50:29 -05:00
devin ivy
a24be34c27
Notify all ancestors of reply, optimize getPostThread method (#545)
* Initialize post_hierarchy index

* Build post_hierarchy index from posts

* Power getPostThread method with post hierarchy index

* Generate ancestor reply notifications, fix tests and down migration

* Generate notifications for all reply ancestors
2023-02-15 14:16:35 -05:00
Daniel Holmgren
ac48330fdb
User table fixes (#536)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing
2023-02-15 14:15:32 -05:00
devin ivy
807e2a783c
Apply lexicon defaults (#552)
* Thread values w/ applied defaults through validation, use in xrpc-server

* Adjust codegen to reflect xrpc defaults
2023-02-14 21:20:20 -05:00
devin ivy
751f16527d
Use DAG-CBOR for XRPC streaming (#546)
* Use canonical dag cbor for xrpc streaming

* Remove undefineds from subscribeAllRepos for dag-cbor
2023-02-13 15:08:26 -05:00
bnewbold
19c54fe55b
Merge pull request #537 from bluesky-social/bnewbold/dev-misc
some small low-priority patches
2023-02-12 20:23:05 -08:00
devin ivy
9a7b2c39e4
Mod action and report params as integers rather than floats (#544) 2023-02-10 16:40:32 -05:00
bnewbold
6ec452450a
pds sql: add indices for feed and post aggregations (#538)
* pds sql: add indices for feed and post aggregations

Seems like aggregations/stats/counts are likely to get changed up in the
near term future, but in the very short term, these simple index
additions yielded non-trivial speedups even at small database sizes, in
my laptop testing.

* pds migration: fix index drops in 'down'
2023-02-09 21:01:05 -05:00
Paul Frazee
d252fd3e95
Fix knownValues for reasonType lexicon (#527) 2023-02-09 20:59:32 -05:00
bryan newbold
8b803e013c pds: run dotenv before any other imports
This fixes logging configuration via env vars set in .env file
2023-02-09 16:40:14 -08:00
bryan newbold
d545b3195e pds: add missing 'license' to package.json 2023-02-09 16:40:06 -08:00
bryan newbold
71d6ce8c5b pds and plc: example .env files for development 2023-02-09 16:22:20 -08:00
Daniel Holmgren
7d92eb0513
Feature branch: streaming repo sync (#542)
* Scaffold xrpc-stream

* Write and test stream frames

* Write and test stream server

* Test bad stream frame parsing cases

* Proof of concept streaming on xrpc-server

* Test bad streaming endpoint

* Make frame stream to fix buffering frame bytes together

* reorg

* some lex-cli things

* better codegen for subscriptions

* scaffolding repo subscriptions

* wip

* remove repo ops

* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* basic sequencer

* some refactoring

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* subscribeRepo outbox

* some cleanup

* wip

* wip

* bugfixin

* only send msgs after tx is committed

* better handle event-emitter -> generator

* max buffer size

* cleanup

* Sync-up xrpc subscriptions with current iteration of the spec

* Allow missing encoding for subscriptions

* track sequencedAt & eventType as well

* Resolve streaming type codes, streaming tests

* Fix interface for hooking into http server for streaming, fix stream routing

* Minor reorg how streaming is enabled in xrpc-server

* Server codegen for xrpc subscriptions, supporting types in xrpc-server

* fix up buffer overloading

* Rename data frame to message frame, rename subscription lex output to message

* Move body param first for streaming frames

* Tidy

* clean up evt types

* buff up tests

* missed merge conflict

* new schema

* blobs on subscriptions

* rm genned client subscription methods

* backfill limits

* testing subscription route & quick outbox bugfix

* fix up migration

* cascade on delete

* comments & naming

* fix dev env

* delete seqs on account deletion

* tidy

* fixing things up with db notify system for schemas

* fix duplicates in outbox

* tidy

* Break out of stream handler when socket closes

* fixing up some timing issues

* tidy

* terminate ws after using

* bump up timer on async reader

* fixing up NOTIFY in txns

* pr feedback

* pr bugfixes

* make order asc explicit

* bringing tests up to speed w atpagent

* bump up max listeners on sequencer

* increase timeouts for reading from generators

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-09 16:17:06 -06:00
Daniel Holmgren
7ac555bd41
Allow syncing a commit range (#535)
allow syncing a commit range
2023-02-09 11:27:44 -06:00
devin ivy
71d812abe6
Get profiles xrpc method (#532)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Move ArrayEl type helper into common
2023-02-09 11:11:46 -05:00
Paul Frazee
2242e8a313
Refactor @atproto/api to the AtpAgent interface (#529)
* Refactor @atproto/api to use the simplified AtpAgent API

* xrpc package: Export the defaultFetchHandler to reuse in api

* api package: Use the defaultFetchHandler defined in xrpc

* Update all usages of the api for the new AtpAgent

* Clear promise on thrown codepath

* Avoid updating the atpagent session until ready to return
2023-02-07 15:30:29 -06:00
devin ivy
b91e6e96b7
Incorporate blobs into moderation actions (#511)
* Lexicon updates for moderating blobs

* Add tracking of blob takedowns to moderation system

* Handle blobs in record takedown reversal

* Apply blob takedowns w/ image invalidation

* Test blob takedowns

* Implement cloudfront image invalidator

* Update lexicons for current moderation action on records and blobs

* Fix migration for postgres

* Enforce single current action per repo and blob, present current action on repo and blob

* Test single active mod action per repo, record, and blob

* Continue to resolve handles for moderated repos
2023-02-06 11:44:26 -05:00
Daniel Holmgren
64f5b4a022
Refactor repo indexing (#521)
* refactor repo indexing

* bit more
2023-02-03 17:11:39 -06:00
devin ivy
216e434c73
Handle client abort on image upload (#524) 2023-02-03 11:19:45 -05:00
Daniel Holmgren
55234b4b0d
Account deletion hotfix (#520)
* account deletion hotfix

* dont end pool

* set array val
2023-02-02 16:47:14 -06:00
Daniel Holmgren
396698f218
Filter out self & follows from suggestions (#498)
dont suggest self or follows
2023-02-02 16:48:41 -05:00
Daniel Holmgren
134920d872
Database notify/listen (#514)
* setup notify/listen for db

* end pool to fix hanging test

* small comment in test

* switch to event emitter

* reconnect on listener error

* rename notifyClient

* remove payload on channels

* pr feedback

* poor guy cant spell
2023-02-02 15:04:38 -06:00
Daniel Holmgren
773f9e304d
Account deletion (#488)
* wip

* fleshing out repo storage

* fleshing out sql storage

* cleaning things up

* fix up tests

* dumb bug - commit log reversed

* rm staging in favor of commiting diffs to blockstore

* clean up benches

* fixing up sql storage

* some caching for sql repo store

* pr feedback

* migration

* wip

* migraiton test

* unclear param

* sql repo storage tests

* rm unused code

* fix up some diff code

* pr feedback

* enum for action types

* missed some

* wip

* ripping out auth lib

* more auth cleanup

* another lurker

* wip better sync primitives

* wip

* improving diffs & sync

* tests working!

* actually implemented checkout lol

* simplify interface & improve error handling

* writing sql storage code

* fixing up tests

* testing & bugfixes

* checkouts return records instead of cids

* one last refactor lol

* missed one

* handle other cid codecs on incoming car verification

* tests + tricky bugs

* unneeded blockstore method

* trim mst on del instead of save

* cleanup comment

* dont resolve did for every commit

* use "commit" instead of "root"

* getRoot -> getHead

* pr feedback

* very silly bug fix

* improve sync output

* reorging + sync of particular records

* serve & verify proofs. also rename some ipld methods

* fix up sync issue in mst

* find reachable records form carfile

* getRecord xrpc method

* pr feedback

* better migration test

* check migraiton result

* fixing up a couple things for pg

* explicit migrateTo

* async exceptions

* ipld car mimetype + remove updateRepo

* Update module publish scripts (#478)

* Update pds package publishing scripts

* Update auth package publishing scripts

* Update crypto package publishing scripts

* Update did-resolver package publishing scripts

* Update handle package publishing scripts

* Update xrpc-server package publishing scripts

* Update common package publishing scripts

* Update plc package publishing scripts

* Update uri package publishing scripts

* Update repo package publishing scripts

* Sort "suggested follows" by number of posts (#477)

* return suggestions by post count

* pr feedback

* fix up PG pagination issue

* partiion commit-history & commit-blocks by user did

* some lexicons

reworking routes

request deletion flows

delete actor rows

migration for user-partitioned-cids

move creator to be on ipld_block

migration tests

* delete records & repos

* delete blobs

* hook it up in route

* pettier ignore email templates

* testing & bugfixes

* testing blobs & bugfixes

* pr feedback

* make deletion test more robust

* change out handle for did on account deletion

* small cleanup

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-02-02 12:34:00 -06:00
Paul Frazee
eb04cd333a
Revert the feed_thumbnail to fit:inside (#510)
* Revert the feed_thumbnail to fit:inside

* Update test snapshots for thumbnail sizing
2023-02-02 10:42:22 -06:00
dholms
dad13229c9 update pg schema name 2023-02-01 15:15:31 -06:00
dholms
62499d69ca testing backfill 2023-02-01 15:01:32 -06:00
dholms
54d51e83a1 unskip tests 2023-02-01 14:34:16 -06:00
dholms
0859119d93 Merge branch 'repo-sync-migration-reorg' into repo-sync-migration-reorg-pt2 2023-02-01 14:33:15 -06:00
dholms
572b38ae76 skipping tests 2023-02-01 14:33:02 -06:00