2067 Commits

Author SHA1 Message Date
Daniel Holmgren
9af530e837
Add unspecced getPopular route (#687)
* added unspecced getPopular route

* remove dupe

* super simple test

* pr feedback
2023-03-17 17:36:34 -04:00
Daniel Holmgren
5a73bbdce8
Increase inv code length to 7 (#678)
increase inv code length to 7
2023-03-16 15:22:29 -05:00
Jake Gold
497a8988a5 Mirror containers on AWS 2023-03-16 10:41:26 -07:00
Daniel Holmgren
197f40b891
List blobs method (#662)
* add list blobs method

* tests + small bugfix

* one more test
2023-03-15 11:15:01 -05:00
Jake Gold
ba428a7c02
Add PDS container image mirror on AWS (#666)
* Add PDS container image mirror on AWS
2023-03-15 08:56:26 -07:00
Daniel Holmgren
fafd008816
Close open file handles (#660)
* close open file handles

* fix up
2023-03-14 14:07:27 -05:00
devin ivy
30a3529c8c
PDS refresh token grace period (#657)
* Add grace period to pds refresh tokens

* Tidy

* Tidy
2023-03-14 13:53:17 -04:00
Daniel Holmgren
d8ec856865
Add hostname param to notifyOfUpdate (#638) 2023-03-14 11:17:22 -05:00
Daniel Holmgren
afa28c709f
Get blobs (#606)
* refactor repo routes

* basic blob route

* getBlob route

* tidy

* move getBlob to sync

* allow mimetype on getBlob

* creator on blob table

* migration

* migration

* handle deletes & check db on getBlob

* fix content type bug

* back to octet-stream

* Update packages/pds/src/api/com/atproto/sync/getBlob.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix up migrations

* pr feedback

* fixing up merge & migration

* patched up migration

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-03-13 19:11:47 -05:00
devin ivy
5b20053e36
In-pds app view feature branch (#637)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Setup table for tracking subscription state on app view

* Support ending app-view leader, clean disconnects from xrpc subscription consumer

* Frame out stateful app-view repo subscription

* Setup app-view lifecycle, run repo subscription

* Add logging to repo sub. Add jitter to leader polling.

* Support configuring app-view repo host

* Allow running pds/app-view dev env with an external repo host

* Fix subscription abort test

* Add test for app-view repo subscription

* Skip invalid repo sub messages w/ log, tidy

* Ensure app-view is permissive of reindexing

* Tidy app-view, renaming sync config, improve error logging

* Fix sync cursor todo, order migrations

* Map new ops field in repo subscription

* In-progress, support indexing updates

* Revert "In-progress, support indexing updates"

This reverts commit 4e082daf535c11c5d4cf69474e5d48a1a3fbe8aa.

* Support indexing record updates

* Prep notifications for post updates. Basic tests for indexing.

* Update indexing tests to better exercise indexes
2023-03-10 16:24:42 -05:00
Daniel Holmgren
9baf9b23d8
Repo v2 (#635)
* fanout to 4

* move meta to root

* combine repoRoot & commit

* finish up commits & tests

* cleanup

* tidy

* better diff

* fixed up mst tests for bytes & fanout 4

* added tests for allowable chars

* length limit & check on deserialize

* tidy

* better leading zeros fn

* pr feedback

* working on migration

* Setup app migration machinery

* Skip appMigration() tests on sqlite

* migration working

* make an app_migration table

* patch up test

* sanity check

* wrapped up

* working

* remove run on repov2

* delete seqs

* logs & static migration names

* cleanup

* missed a file

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 18:49:09 -05:00
Daniel Holmgren
937ab981a3
App migration setup (#641)
* Setup app migration machinery

* Skip appMigration() tests on sqlite

* make an app_migration table

* patch up test

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 10:09:04 -06:00
Daniel Holmgren
ea623ef392
Add some indices for the record table (#639)
add some indices for the record table
2023-03-09 09:55:17 -06:00
Daniel Holmgren
7cc4808907
MST util bug fixes for sync (#636)
couple mst util bug fixes for sync
2023-03-08 10:57:15 -06:00
Daniel Holmgren
ac30419b7d
Bugfix: resolving the root of an available domain (#634)
fix bug when resolving the root of an available domain
2023-03-07 16:22:03 -06:00
Daniel Holmgren
112e4f1610
Format invite codes with dashes (#633)
* format invite code with dashes

* fixed test
2023-03-07 16:21:52 -06:00
Daniel Holmgren
f4a029a897
Handle fixes (#632)
* no 500s on unresolvable handles

* dont resolve handle for service
2023-03-07 13:16:20 -06:00
Takumi Fujiwara
97dac2bbdc
Fix README of handle package (#630) 2023-03-05 23:16:30 -05:00
Daniel Holmgren
8dfcb4f996
Integrating PLC lib (#607)
* integrating new plc lib

* patching up did-resolver

* buffing up pds tests

* didResolver on ctx & plc in postgres

* bring dev env up to date

* re-add extension for linting

* tidy

* use current env vars

* pr feedback
2023-03-04 19:43:59 -06:00
Daniel Holmgren
3bd50b0e5f
Feed builder indices (#627)
* add indices on creators

* add orderby indices

* remove creator indices
2023-03-04 16:13:15 -05:00
devin ivy
f57063bebe
Feed and search query perf tuning (#628)
Switch unnecessary unions to union all in pds queries
2023-03-04 15:26:56 -05:00
devin ivy
50c892c835
Lengthen token lifetimes (#624)
* Lengthen refresh token lifetime

* Lengthen access token lifetime
2023-03-03 15:05:36 -05:00
Akiomi Kamakura
1c7e641ff7
Fix README.md (#622) 2023-03-03 10:48:05 -05:00
Daniel Holmgren
07d941639f
Repo ops on sync stream (#614)
* repo op schema

* repo ops on sync stream

* delete old seqs

* nullable cids

* pr feedback
2023-03-02 12:18:49 -06:00
dholms
4c346d2017 publish @atproto/common 0.1.1 2023-03-01 15:54:21 -08:00
Paul Frazee
5e5d3df615
Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
mehdiMj
19a81fdcb5
Fix copyright year (#609)
The year wasn't correct, so I fix it :)
2023-03-01 15:53:08 -06:00
devin ivy
13ff521bf5
Ensure subscribeAllRepos syncs from cursor 0 (#611)
Ensure subscribeAllRepos allows sync from sequence 0
2023-03-01 16:25:17 -05:00
devin ivy
a9cab00ea1
XRPC subscription consumer (#603)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Yield info frames in byMessage()
2023-03-01 11:28:11 -05:00
Daniel Holmgren
6ee3a8cfda
Crawler schemas (#602)
* sketch of crawl schemas

* codgen

* hostname param

* Update lexicons/com/atproto/sync/requestCrawl.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* codegen

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-02-28 18:24:28 -06:00
devin ivy
5d2895f1ea
Running a task as a leader in the app view (#604)
Interface for picking a pds leader
2023-02-27 14:24:08 -05:00
Daniel Holmgren
bb05232720
Sync subscription tweaks (#599)
* tweaks to sync subscription

* fix test
2023-02-27 10:22:54 -06:00
devin ivy
8826ada589
Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold
fea90cb392
Merge pull request #601 from bluesky-social/jake/add-container-labels
Add standard container image labels
2023-02-23 16:24:57 -08:00
Jake Gold
627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
bnewbold
f08c08a675
MST review and tests (#556)
* mst: more test coverage

In particular, ports the older fanout=32 corner-case tests to fanout=16.

I generated new paths with correct level and ordering, but haven't
actually verified that the resulting structures have the shape of the
ASCII diagrams in text.

* mst: additional comments, including TODO questions

* mst: default new MST layer to 'null' not 0

Looks like all current call sites of this 'create()' function override
the layer already. This is just making it explicit that the default,
null, means the layer has not been computed yet, distinct from being
layer=0.

* mst: remove one TODO and reformat another

From review.
2023-02-23 10:47:16 -08:00
Daniel Holmgren
f29c08f0bb
Bump package versions (#578)
* bump pkgs

* fix package
2023-02-22 19:54:21 -06:00
devin ivy
ca87aeb93f
Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren
136b4b5933
Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
Jake Gold
93d54f49a6
Merge pull request #571 from bluesky-social/jake/add-pds-container-build
PDS container build-and-push action
2023-02-22 08:17:15 -08:00
devin ivy
24fd3b17e2
Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren
c2c87c3226
Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
devin ivy
3a6d785ae0
Fix PDS build, updating main to dist in api package (#566) 2023-02-18 16:15:19 -05:00
Daniel Holmgren
66f731ecee
Repo sync semantic changes (#563)
* switch to cursor & only one event type

* quick bugfix

* no dangling async stuff

* didnt rebuild lex-cli
2023-02-17 17:33:31 -05:00
Daniel Holmgren
20a35b31c1
Add com.atproto.sync.getBlocks XRPC method (#564)
* add sync getBlocks method

* oops didnt build lex-cli
2023-02-17 17:18:13 -05:00
Jake Gold
6357bc3509 add PDS build-and-push action 2023-02-16 15:06:11 -08:00
Devin Ivy
3f6482d0d2 Tidy snapshots 2023-02-16 16:32:17 -05:00
Daniel Holmgren
6c4f6a27f3
Bugfix: Flaky readFromGenerator test (#560)
fix flaky test
2023-02-16 14:34:26 -06:00
Daniel Holmgren
d4b1262f28
DNS handles (#555)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* basic support for dns resolution on handles

* handle resolution

* fix up account create

* tsconfig

* workshop handle constraints

* bring pds up to speed w handle changes

* change dns subdomain

* another lil test

* bugfix

* update dns record format

* typo
2023-02-16 13:59:56 -06:00
bnewbold
e3cd9c23a1
Small PDS and XRPC server fixes (#550)
* pds: fix logging of Bearer auth subject (DID)

* xrpc-server: handle missing and mangled Content-Type better

Eg, for blob uploads.

Also adds a test for incorrect mimetype supplied for a regular endpoint
(one not allowing "*/*").

* skip failing test

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-02-16 13:59:34 -06:00