Commit Graph

1347 Commits

Author SHA1 Message Date
Daniel Holmgren 9af530e837 Add unspecced getPopular route (#687)
* added unspecced getPopular route

* remove dupe

* super simple test

* pr feedback
2023-03-17 17:36:34 -04:00
Daniel Holmgren 5a73bbdce8 Increase inv code length to 7 (#678)
increase inv code length to 7
2023-03-16 15:22:29 -05:00
Daniel Holmgren 197f40b891 List blobs method (#662)
* add list blobs method

* tests + small bugfix

* one more test
2023-03-15 11:15:01 -05:00
Daniel Holmgren fafd008816 Close open file handles (#660)
* close open file handles

* fix up
2023-03-14 14:07:27 -05:00
devin ivy 30a3529c8c PDS refresh token grace period (#657)
* Add grace period to pds refresh tokens

* Tidy

* Tidy
2023-03-14 13:53:17 -04:00
Daniel Holmgren d8ec856865 Add hostname param to notifyOfUpdate (#638) 2023-03-14 11:17:22 -05:00
Daniel Holmgren afa28c709f Get blobs (#606)
* refactor repo routes

* basic blob route

* getBlob route

* tidy

* move getBlob to sync

* allow mimetype on getBlob

* creator on blob table

* migration

* migration

* handle deletes & check db on getBlob

* fix content type bug

* back to octet-stream

* Update packages/pds/src/api/com/atproto/sync/getBlob.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix up migrations

* pr feedback

* fixing up merge & migration

* patched up migration

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-03-13 19:11:47 -05:00
devin ivy 5b20053e36 In-pds app view feature branch (#637)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Setup table for tracking subscription state on app view

* Support ending app-view leader, clean disconnects from xrpc subscription consumer

* Frame out stateful app-view repo subscription

* Setup app-view lifecycle, run repo subscription

* Add logging to repo sub. Add jitter to leader polling.

* Support configuring app-view repo host

* Allow running pds/app-view dev env with an external repo host

* Fix subscription abort test

* Add test for app-view repo subscription

* Skip invalid repo sub messages w/ log, tidy

* Ensure app-view is permissive of reindexing

* Tidy app-view, renaming sync config, improve error logging

* Fix sync cursor todo, order migrations

* Map new ops field in repo subscription

* In-progress, support indexing updates

* Revert "In-progress, support indexing updates"

This reverts commit 4e082daf535c11c5d4cf69474e5d48a1a3fbe8aa.

* Support indexing record updates

* Prep notifications for post updates. Basic tests for indexing.

* Update indexing tests to better exercise indexes
2023-03-10 16:24:42 -05:00
Daniel Holmgren 9baf9b23d8 Repo v2 (#635)
* fanout to 4

* move meta to root

* combine repoRoot & commit

* finish up commits & tests

* cleanup

* tidy

* better diff

* fixed up mst tests for bytes & fanout 4

* added tests for allowable chars

* length limit & check on deserialize

* tidy

* better leading zeros fn

* pr feedback

* working on migration

* Setup app migration machinery

* Skip appMigration() tests on sqlite

* migration working

* make an app_migration table

* patch up test

* sanity check

* wrapped up

* working

* remove run on repov2

* delete seqs

* logs & static migration names

* cleanup

* missed a file

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 18:49:09 -05:00
Daniel Holmgren 937ab981a3 App migration setup (#641)
* Setup app migration machinery

* Skip appMigration() tests on sqlite

* make an app_migration table

* patch up test

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 10:09:04 -06:00
Daniel Holmgren ea623ef392 Add some indices for the record table (#639)
add some indices for the record table
2023-03-09 09:55:17 -06:00
Daniel Holmgren ac30419b7d Bugfix: resolving the root of an available domain (#634)
fix bug when resolving the root of an available domain
2023-03-07 16:22:03 -06:00
Daniel Holmgren 112e4f1610 Format invite codes with dashes (#633)
* format invite code with dashes

* fixed test
2023-03-07 16:21:52 -06:00
Daniel Holmgren f4a029a897 Handle fixes (#632)
* no 500s on unresolvable handles

* dont resolve handle for service
2023-03-07 13:16:20 -06:00
Daniel Holmgren 8dfcb4f996 Integrating PLC lib (#607)
* integrating new plc lib

* patching up did-resolver

* buffing up pds tests

* didResolver on ctx & plc in postgres

* bring dev env up to date

* re-add extension for linting

* tidy

* use current env vars

* pr feedback
2023-03-04 19:43:59 -06:00
Daniel Holmgren 3bd50b0e5f Feed builder indices (#627)
* add indices on creators

* add orderby indices

* remove creator indices
2023-03-04 16:13:15 -05:00
devin ivy f57063bebe Feed and search query perf tuning (#628)
Switch unnecessary unions to union all in pds queries
2023-03-04 15:26:56 -05:00
devin ivy 50c892c835 Lengthen token lifetimes (#624)
* Lengthen refresh token lifetime

* Lengthen access token lifetime
2023-03-03 15:05:36 -05:00
Daniel Holmgren 07d941639f Repo ops on sync stream (#614)
* repo op schema

* repo ops on sync stream

* delete old seqs

* nullable cids

* pr feedback
2023-03-02 12:18:49 -06:00
Paul Frazee 5e5d3df615 Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
devin ivy 13ff521bf5 Ensure subscribeAllRepos syncs from cursor 0 (#611)
Ensure subscribeAllRepos allows sync from sequence 0
2023-03-01 16:25:17 -05:00
devin ivy a9cab00ea1 XRPC subscription consumer (#603)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Yield info frames in byMessage()
2023-03-01 11:28:11 -05:00
Daniel Holmgren 6ee3a8cfda Crawler schemas (#602)
* sketch of crawl schemas

* codgen

* hostname param

* Update lexicons/com/atproto/sync/requestCrawl.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* codegen

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-02-28 18:24:28 -06:00
devin ivy 5d2895f1ea Running a task as a leader in the app view (#604)
Interface for picking a pds leader
2023-02-27 14:24:08 -05:00
Daniel Holmgren bb05232720 Sync subscription tweaks (#599)
* tweaks to sync subscription

* fix test
2023-02-27 10:22:54 -06:00
devin ivy 8826ada589 Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold 627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
devin ivy ca87aeb93f Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren 136b4b5933 Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
devin ivy 24fd3b17e2 Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren c2c87c3226 Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
Daniel Holmgren 66f731ecee Repo sync semantic changes (#563)
* switch to cursor & only one event type

* quick bugfix

* no dangling async stuff

* didnt rebuild lex-cli
2023-02-17 17:33:31 -05:00
Daniel Holmgren 20a35b31c1 Add com.atproto.sync.getBlocks XRPC method (#564)
* add sync getBlocks method

* oops didnt build lex-cli
2023-02-17 17:18:13 -05:00
Devin Ivy 3f6482d0d2 Tidy snapshots 2023-02-16 16:32:17 -05:00
Daniel Holmgren d4b1262f28 DNS handles (#555)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* basic support for dns resolution on handles

* handle resolution

* fix up account create

* tsconfig

* workshop handle constraints

* bring pds up to speed w handle changes

* change dns subdomain

* another lil test

* bugfix

* update dns record format

* typo
2023-02-16 13:59:56 -06:00
bnewbold e3cd9c23a1 Small PDS and XRPC server fixes (#550)
* pds: fix logging of Bearer auth subject (DID)

* xrpc-server: handle missing and mangled Content-Type better

Eg, for blob uploads.

Also adds a test for incorrect mimetype supplied for a regular endpoint
(one not allowing "*/*").

* skip failing test

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-02-16 13:59:34 -06:00
devin ivy 9c119fb9c6 Allow unsetting profile images and description in updateProfile (#554)
* Add support for nullable on lex objects

* Support clearing fields in updateProfile lexicon

* Support clearing fields in updateProfile method
2023-02-16 14:32:25 -05:00
Paul Frazee 5737e653e3 Bump @atproto/pds@0.0.3, @atproto/api@0.1.2 (#558)
* @atproto/pds@0.0.3

* @atproto/api@0.1.2
2023-02-16 12:29:01 -06:00
devin ivy 850b54fad6 Add following state to profile/actor views, reorg views (#539)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Cull actor/profile view lexicons down to three variants

* Map viewer in profile view, mark todos, lex tweak

* Apply basic profile view to methods, lex fix

* Switch searchTypeahead to actor-with-info view

* Switch getVotes to actor-with-info view

* Switch getMutes to actor-with-info view

* Switch getFollows and getFollowers to actor-with-info view

* Switch getRepostedBy to actor-with-info view

* Switch feed and notification views to use updated actor-with-info view
2023-02-16 11:40:05 -05:00
Daniel Holmgren 57da2270b9 Update handles (#547)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* Fix did pkey down migration

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-15 14:50:29 -05:00
devin ivy a24be34c27 Notify all ancestors of reply, optimize getPostThread method (#545)
* Initialize post_hierarchy index

* Build post_hierarchy index from posts

* Power getPostThread method with post hierarchy index

* Generate ancestor reply notifications, fix tests and down migration

* Generate notifications for all reply ancestors
2023-02-15 14:16:35 -05:00
Daniel Holmgren ac48330fdb User table fixes (#536)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing
2023-02-15 14:15:32 -05:00
devin ivy 807e2a783c Apply lexicon defaults (#552)
* Thread values w/ applied defaults through validation, use in xrpc-server

* Adjust codegen to reflect xrpc defaults
2023-02-14 21:20:20 -05:00
devin ivy 751f16527d Use DAG-CBOR for XRPC streaming (#546)
* Use canonical dag cbor for xrpc streaming

* Remove undefineds from subscribeAllRepos for dag-cbor
2023-02-13 15:08:26 -05:00
bnewbold 19c54fe55b Merge pull request #537 from bluesky-social/bnewbold/dev-misc
some small low-priority patches
2023-02-12 20:23:05 -08:00
devin ivy 9a7b2c39e4 Mod action and report params as integers rather than floats (#544) 2023-02-10 16:40:32 -05:00
bnewbold 6ec452450a pds sql: add indices for feed and post aggregations (#538)
* pds sql: add indices for feed and post aggregations

Seems like aggregations/stats/counts are likely to get changed up in the
near term future, but in the very short term, these simple index
additions yielded non-trivial speedups even at small database sizes, in
my laptop testing.

* pds migration: fix index drops in 'down'
2023-02-09 21:01:05 -05:00
Paul Frazee d252fd3e95 Fix knownValues for reasonType lexicon (#527) 2023-02-09 20:59:32 -05:00
bryan newbold 8b803e013c pds: run dotenv before any other imports
This fixes logging configuration via env vars set in .env file
2023-02-09 16:40:14 -08:00
bryan newbold d545b3195e pds: add missing 'license' to package.json 2023-02-09 16:40:06 -08:00