1610 Commits

Author SHA1 Message Date
devin ivy
30a3529c8c
PDS refresh token grace period (#657)
* Add grace period to pds refresh tokens

* Tidy

* Tidy
2023-03-14 13:53:17 -04:00
Daniel Holmgren
d8ec856865
Add hostname param to notifyOfUpdate (#638) 2023-03-14 11:17:22 -05:00
Daniel Holmgren
afa28c709f
Get blobs (#606)
* refactor repo routes

* basic blob route

* getBlob route

* tidy

* move getBlob to sync

* allow mimetype on getBlob

* creator on blob table

* migration

* migration

* handle deletes & check db on getBlob

* fix content type bug

* back to octet-stream

* Update packages/pds/src/api/com/atproto/sync/getBlob.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix up migrations

* pr feedback

* fixing up merge & migration

* patched up migration

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-03-13 19:11:47 -05:00
devin ivy
5b20053e36
In-pds app view feature branch (#637)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Setup table for tracking subscription state on app view

* Support ending app-view leader, clean disconnects from xrpc subscription consumer

* Frame out stateful app-view repo subscription

* Setup app-view lifecycle, run repo subscription

* Add logging to repo sub. Add jitter to leader polling.

* Support configuring app-view repo host

* Allow running pds/app-view dev env with an external repo host

* Fix subscription abort test

* Add test for app-view repo subscription

* Skip invalid repo sub messages w/ log, tidy

* Ensure app-view is permissive of reindexing

* Tidy app-view, renaming sync config, improve error logging

* Fix sync cursor todo, order migrations

* Map new ops field in repo subscription

* In-progress, support indexing updates

* Revert "In-progress, support indexing updates"

This reverts commit 4e082daf535c11c5d4cf69474e5d48a1a3fbe8aa.

* Support indexing record updates

* Prep notifications for post updates. Basic tests for indexing.

* Update indexing tests to better exercise indexes
2023-03-10 16:24:42 -05:00
Daniel Holmgren
9baf9b23d8
Repo v2 (#635)
* fanout to 4

* move meta to root

* combine repoRoot & commit

* finish up commits & tests

* cleanup

* tidy

* better diff

* fixed up mst tests for bytes & fanout 4

* added tests for allowable chars

* length limit & check on deserialize

* tidy

* better leading zeros fn

* pr feedback

* working on migration

* Setup app migration machinery

* Skip appMigration() tests on sqlite

* migration working

* make an app_migration table

* patch up test

* sanity check

* wrapped up

* working

* remove run on repov2

* delete seqs

* logs & static migration names

* cleanup

* missed a file

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 18:49:09 -05:00
Daniel Holmgren
937ab981a3
App migration setup (#641)
* Setup app migration machinery

* Skip appMigration() tests on sqlite

* make an app_migration table

* patch up test

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 10:09:04 -06:00
Daniel Holmgren
ea623ef392
Add some indices for the record table (#639)
add some indices for the record table
2023-03-09 09:55:17 -06:00
Daniel Holmgren
7cc4808907
MST util bug fixes for sync (#636)
couple mst util bug fixes for sync
2023-03-08 10:57:15 -06:00
Daniel Holmgren
ac30419b7d
Bugfix: resolving the root of an available domain (#634)
fix bug when resolving the root of an available domain
2023-03-07 16:22:03 -06:00
Daniel Holmgren
112e4f1610
Format invite codes with dashes (#633)
* format invite code with dashes

* fixed test
2023-03-07 16:21:52 -06:00
Daniel Holmgren
f4a029a897
Handle fixes (#632)
* no 500s on unresolvable handles

* dont resolve handle for service
2023-03-07 13:16:20 -06:00
Takumi Fujiwara
97dac2bbdc
Fix README of handle package (#630) 2023-03-05 23:16:30 -05:00
Daniel Holmgren
8dfcb4f996
Integrating PLC lib (#607)
* integrating new plc lib

* patching up did-resolver

* buffing up pds tests

* didResolver on ctx & plc in postgres

* bring dev env up to date

* re-add extension for linting

* tidy

* use current env vars

* pr feedback
2023-03-04 19:43:59 -06:00
Daniel Holmgren
3bd50b0e5f
Feed builder indices (#627)
* add indices on creators

* add orderby indices

* remove creator indices
2023-03-04 16:13:15 -05:00
devin ivy
f57063bebe
Feed and search query perf tuning (#628)
Switch unnecessary unions to union all in pds queries
2023-03-04 15:26:56 -05:00
devin ivy
50c892c835
Lengthen token lifetimes (#624)
* Lengthen refresh token lifetime

* Lengthen access token lifetime
2023-03-03 15:05:36 -05:00
Akiomi Kamakura
1c7e641ff7
Fix README.md (#622) 2023-03-03 10:48:05 -05:00
Daniel Holmgren
07d941639f
Repo ops on sync stream (#614)
* repo op schema

* repo ops on sync stream

* delete old seqs

* nullable cids

* pr feedback
2023-03-02 12:18:49 -06:00
dholms
4c346d2017 publish @atproto/common 0.1.1 2023-03-01 15:54:21 -08:00
Paul Frazee
5e5d3df615
Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
devin ivy
13ff521bf5
Ensure subscribeAllRepos syncs from cursor 0 (#611)
Ensure subscribeAllRepos allows sync from sequence 0
2023-03-01 16:25:17 -05:00
devin ivy
a9cab00ea1
XRPC subscription consumer (#603)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Yield info frames in byMessage()
2023-03-01 11:28:11 -05:00
Daniel Holmgren
6ee3a8cfda
Crawler schemas (#602)
* sketch of crawl schemas

* codgen

* hostname param

* Update lexicons/com/atproto/sync/requestCrawl.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* codegen

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-02-28 18:24:28 -06:00
devin ivy
5d2895f1ea
Running a task as a leader in the app view (#604)
Interface for picking a pds leader
2023-02-27 14:24:08 -05:00
Daniel Holmgren
bb05232720
Sync subscription tweaks (#599)
* tweaks to sync subscription

* fix test
2023-02-27 10:22:54 -06:00
devin ivy
8826ada589
Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold
627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
bnewbold
f08c08a675
MST review and tests (#556)
* mst: more test coverage

In particular, ports the older fanout=32 corner-case tests to fanout=16.

I generated new paths with correct level and ordering, but haven't
actually verified that the resulting structures have the shape of the
ASCII diagrams in text.

* mst: additional comments, including TODO questions

* mst: default new MST layer to 'null' not 0

Looks like all current call sites of this 'create()' function override
the layer already. This is just making it explicit that the default,
null, means the layer has not been computed yet, distinct from being
layer=0.

* mst: remove one TODO and reformat another

From review.
2023-02-23 10:47:16 -08:00
Daniel Holmgren
f29c08f0bb
Bump package versions (#578)
* bump pkgs

* fix package
2023-02-22 19:54:21 -06:00
devin ivy
ca87aeb93f
Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren
136b4b5933
Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
devin ivy
24fd3b17e2
Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren
c2c87c3226
Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
devin ivy
3a6d785ae0
Fix PDS build, updating main to dist in api package (#566) 2023-02-18 16:15:19 -05:00
Daniel Holmgren
66f731ecee
Repo sync semantic changes (#563)
* switch to cursor & only one event type

* quick bugfix

* no dangling async stuff

* didnt rebuild lex-cli
2023-02-17 17:33:31 -05:00
Daniel Holmgren
20a35b31c1
Add com.atproto.sync.getBlocks XRPC method (#564)
* add sync getBlocks method

* oops didnt build lex-cli
2023-02-17 17:18:13 -05:00
Devin Ivy
3f6482d0d2 Tidy snapshots 2023-02-16 16:32:17 -05:00
Daniel Holmgren
6c4f6a27f3
Bugfix: Flaky readFromGenerator test (#560)
fix flaky test
2023-02-16 14:34:26 -06:00
Daniel Holmgren
d4b1262f28
DNS handles (#555)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* basic support for dns resolution on handles

* handle resolution

* fix up account create

* tsconfig

* workshop handle constraints

* bring pds up to speed w handle changes

* change dns subdomain

* another lil test

* bugfix

* update dns record format

* typo
2023-02-16 13:59:56 -06:00
bnewbold
e3cd9c23a1
Small PDS and XRPC server fixes (#550)
* pds: fix logging of Bearer auth subject (DID)

* xrpc-server: handle missing and mangled Content-Type better

Eg, for blob uploads.

Also adds a test for incorrect mimetype supplied for a regular endpoint
(one not allowing "*/*").

* skip failing test

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-02-16 13:59:34 -06:00
devin ivy
9c119fb9c6
Allow unsetting profile images and description in updateProfile (#554)
* Add support for nullable on lex objects

* Support clearing fields in updateProfile lexicon

* Support clearing fields in updateProfile method
2023-02-16 14:32:25 -05:00
Paul Frazee
5737e653e3
Bump @atproto/pds@0.0.3, @atproto/api@0.1.2 (#558)
* @atproto/pds@0.0.3

* @atproto/api@0.1.2
2023-02-16 12:29:01 -06:00
devin ivy
850b54fad6
Add following state to profile/actor views, reorg views (#539)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Cull actor/profile view lexicons down to three variants

* Map viewer in profile view, mark todos, lex tweak

* Apply basic profile view to methods, lex fix

* Switch searchTypeahead to actor-with-info view

* Switch getVotes to actor-with-info view

* Switch getMutes to actor-with-info view

* Switch getFollows and getFollowers to actor-with-info view

* Switch getRepostedBy to actor-with-info view

* Switch feed and notification views to use updated actor-with-info view
2023-02-16 11:40:05 -05:00
Daniel Holmgren
57da2270b9
Update handles (#547)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* Fix did pkey down migration

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-15 14:50:29 -05:00
devin ivy
a24be34c27
Notify all ancestors of reply, optimize getPostThread method (#545)
* Initialize post_hierarchy index

* Build post_hierarchy index from posts

* Power getPostThread method with post hierarchy index

* Generate ancestor reply notifications, fix tests and down migration

* Generate notifications for all reply ancestors
2023-02-15 14:16:35 -05:00
Daniel Holmgren
ac48330fdb
User table fixes (#536)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing
2023-02-15 14:15:32 -05:00
devin ivy
807e2a783c
Apply lexicon defaults (#552)
* Thread values w/ applied defaults through validation, use in xrpc-server

* Adjust codegen to reflect xrpc defaults
2023-02-14 21:20:20 -05:00
devin ivy
751f16527d
Use DAG-CBOR for XRPC streaming (#546)
* Use canonical dag cbor for xrpc streaming

* Remove undefineds from subscribeAllRepos for dag-cbor
2023-02-13 15:08:26 -05:00
bnewbold
19c54fe55b
Merge pull request #537 from bluesky-social/bnewbold/dev-misc
some small low-priority patches
2023-02-12 20:23:05 -08:00
devin ivy
9a7b2c39e4
Mod action and report params as integers rather than floats (#544) 2023-02-10 16:40:32 -05:00