1776 Commits

Author SHA1 Message Date
Devin Ivy
4cf9eeda7b Export app migrations from pds entrypoint 2023-03-28 15:42:22 -04:00
Daniel Holmgren
82e84043f1
Delete scenes app migration (#710)
* delete scenes app migration

* added log

* one more dummy check
2023-03-28 15:28:53 -04:00
Daniel Holmgren
3bfd27dd61
Drop scene tables (#709)
* drop scene tables

* Update packages/pds/src/db/migrations/20230323T162732466Z-remove-scenes.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-03-28 15:25:21 -04:00
Daniel Holmgren
4447228e37
Use swc for jest (#722)
* use swc for jest

* fix up deps

* cleanup
2023-03-27 19:01:44 -05:00
Safa Safari
b17d2a2e10
Fix example invite code (#701)
fix example invite code
2023-03-21 12:41:18 -05:00
Devin Ivy
d6935b42ef Lint 2023-03-21 09:38:34 -04:00
Shinya Kato / 加藤 真也
629bd46950
feat: added InvalidDidError (#696)
* feat: add `InvalidDidError`

* test: fixed to check `InvalidDidError`
2023-03-20 20:51:33 -05:00
Shinya Kato / 加藤 真也
2fc0715b3e
docs: fixed outdated usage (#694) 2023-03-20 10:28:11 -04:00
Shinya Kato / 加藤 真也
a86bb2891c
docs: fixed typo (#695) 2023-03-20 09:34:41 -04:00
Daniel Holmgren
565fd6afe0
Bugfix: don't emit events lost commits (#675)
* dont sequence lost commits

* added note
2023-03-19 18:48:05 -05:00
Daniel Holmgren
2d1a247918
bugfix: fixed pagination in getPopular (#689)
fixed up getPopular route
2023-03-17 19:05:30 -05:00
bnewbold
228431edf3
lexicon-level identifier validation helpers and test cases (#576)
* hardening: permissive tests for handles

* identifers: permissive tests for NSIDs

* identifiers: add DID validation

* identifiers: wrong about domain syntax (duh, 4chan.com)

In short, labels can start with digits except for the final TLD part.
I think that is all for "DNS domains as hostnames which are actually
used in the real world", though there might be other modern things.
Underscores are obviously allowed in DNS for things like SRV records,
but I don't think as "regular hostnames".

Not sure we want this for NSIDs, so not updating that code or tests yet.

* identifiers: ATURI test corner cases

* identifiers: remove TODOs (moved to formalism doc)

* identifiers: small comment typos and corrections

* move around & integrate indentifier hardneing

* fixed up some tests

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-03-17 17:16:31 -05:00
Daniel Holmgren
9af530e837
Add unspecced getPopular route (#687)
* added unspecced getPopular route

* remove dupe

* super simple test

* pr feedback
2023-03-17 17:36:34 -04:00
Daniel Holmgren
5a73bbdce8
Increase inv code length to 7 (#678)
increase inv code length to 7
2023-03-16 15:22:29 -05:00
Daniel Holmgren
197f40b891
List blobs method (#662)
* add list blobs method

* tests + small bugfix

* one more test
2023-03-15 11:15:01 -05:00
Daniel Holmgren
fafd008816
Close open file handles (#660)
* close open file handles

* fix up
2023-03-14 14:07:27 -05:00
devin ivy
30a3529c8c
PDS refresh token grace period (#657)
* Add grace period to pds refresh tokens

* Tidy

* Tidy
2023-03-14 13:53:17 -04:00
Daniel Holmgren
d8ec856865
Add hostname param to notifyOfUpdate (#638) 2023-03-14 11:17:22 -05:00
Daniel Holmgren
afa28c709f
Get blobs (#606)
* refactor repo routes

* basic blob route

* getBlob route

* tidy

* move getBlob to sync

* allow mimetype on getBlob

* creator on blob table

* migration

* migration

* handle deletes & check db on getBlob

* fix content type bug

* back to octet-stream

* Update packages/pds/src/api/com/atproto/sync/getBlob.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix up migrations

* pr feedback

* fixing up merge & migration

* patched up migration

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-03-13 19:11:47 -05:00
devin ivy
5b20053e36
In-pds app view feature branch (#637)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Setup table for tracking subscription state on app view

* Support ending app-view leader, clean disconnects from xrpc subscription consumer

* Frame out stateful app-view repo subscription

* Setup app-view lifecycle, run repo subscription

* Add logging to repo sub. Add jitter to leader polling.

* Support configuring app-view repo host

* Allow running pds/app-view dev env with an external repo host

* Fix subscription abort test

* Add test for app-view repo subscription

* Skip invalid repo sub messages w/ log, tidy

* Ensure app-view is permissive of reindexing

* Tidy app-view, renaming sync config, improve error logging

* Fix sync cursor todo, order migrations

* Map new ops field in repo subscription

* In-progress, support indexing updates

* Revert "In-progress, support indexing updates"

This reverts commit 4e082daf535c11c5d4cf69474e5d48a1a3fbe8aa.

* Support indexing record updates

* Prep notifications for post updates. Basic tests for indexing.

* Update indexing tests to better exercise indexes
2023-03-10 16:24:42 -05:00
Daniel Holmgren
9baf9b23d8
Repo v2 (#635)
* fanout to 4

* move meta to root

* combine repoRoot & commit

* finish up commits & tests

* cleanup

* tidy

* better diff

* fixed up mst tests for bytes & fanout 4

* added tests for allowable chars

* length limit & check on deserialize

* tidy

* better leading zeros fn

* pr feedback

* working on migration

* Setup app migration machinery

* Skip appMigration() tests on sqlite

* migration working

* make an app_migration table

* patch up test

* sanity check

* wrapped up

* working

* remove run on repov2

* delete seqs

* logs & static migration names

* cleanup

* missed a file

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 18:49:09 -05:00
Daniel Holmgren
937ab981a3
App migration setup (#641)
* Setup app migration machinery

* Skip appMigration() tests on sqlite

* make an app_migration table

* patch up test

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 10:09:04 -06:00
Daniel Holmgren
ea623ef392
Add some indices for the record table (#639)
add some indices for the record table
2023-03-09 09:55:17 -06:00
Daniel Holmgren
7cc4808907
MST util bug fixes for sync (#636)
couple mst util bug fixes for sync
2023-03-08 10:57:15 -06:00
Daniel Holmgren
ac30419b7d
Bugfix: resolving the root of an available domain (#634)
fix bug when resolving the root of an available domain
2023-03-07 16:22:03 -06:00
Daniel Holmgren
112e4f1610
Format invite codes with dashes (#633)
* format invite code with dashes

* fixed test
2023-03-07 16:21:52 -06:00
Daniel Holmgren
f4a029a897
Handle fixes (#632)
* no 500s on unresolvable handles

* dont resolve handle for service
2023-03-07 13:16:20 -06:00
Takumi Fujiwara
97dac2bbdc
Fix README of handle package (#630) 2023-03-05 23:16:30 -05:00
Daniel Holmgren
8dfcb4f996
Integrating PLC lib (#607)
* integrating new plc lib

* patching up did-resolver

* buffing up pds tests

* didResolver on ctx & plc in postgres

* bring dev env up to date

* re-add extension for linting

* tidy

* use current env vars

* pr feedback
2023-03-04 19:43:59 -06:00
Daniel Holmgren
3bd50b0e5f
Feed builder indices (#627)
* add indices on creators

* add orderby indices

* remove creator indices
2023-03-04 16:13:15 -05:00
devin ivy
f57063bebe
Feed and search query perf tuning (#628)
Switch unnecessary unions to union all in pds queries
2023-03-04 15:26:56 -05:00
devin ivy
50c892c835
Lengthen token lifetimes (#624)
* Lengthen refresh token lifetime

* Lengthen access token lifetime
2023-03-03 15:05:36 -05:00
Akiomi Kamakura
1c7e641ff7
Fix README.md (#622) 2023-03-03 10:48:05 -05:00
Daniel Holmgren
07d941639f
Repo ops on sync stream (#614)
* repo op schema

* repo ops on sync stream

* delete old seqs

* nullable cids

* pr feedback
2023-03-02 12:18:49 -06:00
dholms
4c346d2017 publish @atproto/common 0.1.1 2023-03-01 15:54:21 -08:00
Paul Frazee
5e5d3df615
Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
devin ivy
13ff521bf5
Ensure subscribeAllRepos syncs from cursor 0 (#611)
Ensure subscribeAllRepos allows sync from sequence 0
2023-03-01 16:25:17 -05:00
devin ivy
a9cab00ea1
XRPC subscription consumer (#603)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Yield info frames in byMessage()
2023-03-01 11:28:11 -05:00
Daniel Holmgren
6ee3a8cfda
Crawler schemas (#602)
* sketch of crawl schemas

* codgen

* hostname param

* Update lexicons/com/atproto/sync/requestCrawl.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* codegen

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-02-28 18:24:28 -06:00
devin ivy
5d2895f1ea
Running a task as a leader in the app view (#604)
Interface for picking a pds leader
2023-02-27 14:24:08 -05:00
Daniel Holmgren
bb05232720
Sync subscription tweaks (#599)
* tweaks to sync subscription

* fix test
2023-02-27 10:22:54 -06:00
devin ivy
8826ada589
Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold
627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
bnewbold
f08c08a675
MST review and tests (#556)
* mst: more test coverage

In particular, ports the older fanout=32 corner-case tests to fanout=16.

I generated new paths with correct level and ordering, but haven't
actually verified that the resulting structures have the shape of the
ASCII diagrams in text.

* mst: additional comments, including TODO questions

* mst: default new MST layer to 'null' not 0

Looks like all current call sites of this 'create()' function override
the layer already. This is just making it explicit that the default,
null, means the layer has not been computed yet, distinct from being
layer=0.

* mst: remove one TODO and reformat another

From review.
2023-02-23 10:47:16 -08:00
Daniel Holmgren
f29c08f0bb
Bump package versions (#578)
* bump pkgs

* fix package
2023-02-22 19:54:21 -06:00
devin ivy
ca87aeb93f
Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren
136b4b5933
Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
devin ivy
24fd3b17e2
Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren
c2c87c3226
Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
devin ivy
3a6d785ae0
Fix PDS build, updating main to dist in api package (#566) 2023-02-18 16:15:19 -05:00