Commit Graph

2872 Commits

Author SHA1 Message Date
Shinya Kato / 加藤 真也 2fc0715b3e docs: fixed outdated usage (#694) 2023-03-20 10:28:11 -04:00
Shinya Kato / 加藤 真也 a86bb2891c docs: fixed typo (#695) 2023-03-20 09:34:41 -04:00
Daniel Holmgren 565fd6afe0 Bugfix: don't emit events lost commits (#675)
* dont sequence lost commits

* added note
2023-03-19 18:48:05 -05:00
Daniel Holmgren 2d1a247918 bugfix: fixed pagination in getPopular (#689)
fixed up getPopular route
2023-03-17 19:05:30 -05:00
bnewbold 228431edf3 lexicon-level identifier validation helpers and test cases (#576)
* hardening: permissive tests for handles

* identifers: permissive tests for NSIDs

* identifiers: add DID validation

* identifiers: wrong about domain syntax (duh, 4chan.com)

In short, labels can start with digits except for the final TLD part.
I think that is all for "DNS domains as hostnames which are actually
used in the real world", though there might be other modern things.
Underscores are obviously allowed in DNS for things like SRV records,
but I don't think as "regular hostnames".

Not sure we want this for NSIDs, so not updating that code or tests yet.

* identifiers: ATURI test corner cases

* identifiers: remove TODOs (moved to formalism doc)

* identifiers: small comment typos and corrections

* move around & integrate indentifier hardneing

* fixed up some tests

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-03-17 17:16:31 -05:00
Daniel Holmgren 9af530e837 Add unspecced getPopular route (#687)
* added unspecced getPopular route

* remove dupe

* super simple test

* pr feedback
2023-03-17 17:36:34 -04:00
Daniel Holmgren 5a73bbdce8 Increase inv code length to 7 (#678)
increase inv code length to 7
2023-03-16 15:22:29 -05:00
Jake Gold 497a8988a5 Mirror containers on AWS 2023-03-16 10:41:26 -07:00
Daniel Holmgren 197f40b891 List blobs method (#662)
* add list blobs method

* tests + small bugfix

* one more test
2023-03-15 11:15:01 -05:00
Jake Gold ba428a7c02 Add PDS container image mirror on AWS (#666)
* Add PDS container image mirror on AWS
2023-03-15 08:56:26 -07:00
Daniel Holmgren fafd008816 Close open file handles (#660)
* close open file handles

* fix up
2023-03-14 14:07:27 -05:00
devin ivy 30a3529c8c PDS refresh token grace period (#657)
* Add grace period to pds refresh tokens

* Tidy

* Tidy
2023-03-14 13:53:17 -04:00
Daniel Holmgren d8ec856865 Add hostname param to notifyOfUpdate (#638) 2023-03-14 11:17:22 -05:00
Daniel Holmgren afa28c709f Get blobs (#606)
* refactor repo routes

* basic blob route

* getBlob route

* tidy

* move getBlob to sync

* allow mimetype on getBlob

* creator on blob table

* migration

* migration

* handle deletes & check db on getBlob

* fix content type bug

* back to octet-stream

* Update packages/pds/src/api/com/atproto/sync/getBlob.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix up migrations

* pr feedback

* fixing up merge & migration

* patched up migration

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-03-13 19:11:47 -05:00
devin ivy 5b20053e36 In-pds app view feature branch (#637)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Setup table for tracking subscription state on app view

* Support ending app-view leader, clean disconnects from xrpc subscription consumer

* Frame out stateful app-view repo subscription

* Setup app-view lifecycle, run repo subscription

* Add logging to repo sub. Add jitter to leader polling.

* Support configuring app-view repo host

* Allow running pds/app-view dev env with an external repo host

* Fix subscription abort test

* Add test for app-view repo subscription

* Skip invalid repo sub messages w/ log, tidy

* Ensure app-view is permissive of reindexing

* Tidy app-view, renaming sync config, improve error logging

* Fix sync cursor todo, order migrations

* Map new ops field in repo subscription

* In-progress, support indexing updates

* Revert "In-progress, support indexing updates"

This reverts commit 4e082daf535c11c5d4cf69474e5d48a1a3fbe8aa.

* Support indexing record updates

* Prep notifications for post updates. Basic tests for indexing.

* Update indexing tests to better exercise indexes
2023-03-10 16:24:42 -05:00
Daniel Holmgren 9baf9b23d8 Repo v2 (#635)
* fanout to 4

* move meta to root

* combine repoRoot & commit

* finish up commits & tests

* cleanup

* tidy

* better diff

* fixed up mst tests for bytes & fanout 4

* added tests for allowable chars

* length limit & check on deserialize

* tidy

* better leading zeros fn

* pr feedback

* working on migration

* Setup app migration machinery

* Skip appMigration() tests on sqlite

* migration working

* make an app_migration table

* patch up test

* sanity check

* wrapped up

* working

* remove run on repov2

* delete seqs

* logs & static migration names

* cleanup

* missed a file

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 18:49:09 -05:00
Daniel Holmgren 937ab981a3 App migration setup (#641)
* Setup app migration machinery

* Skip appMigration() tests on sqlite

* make an app_migration table

* patch up test

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-03-09 10:09:04 -06:00
Daniel Holmgren ea623ef392 Add some indices for the record table (#639)
add some indices for the record table
2023-03-09 09:55:17 -06:00
Daniel Holmgren 7cc4808907 MST util bug fixes for sync (#636)
couple mst util bug fixes for sync
2023-03-08 10:57:15 -06:00
Daniel Holmgren ac30419b7d Bugfix: resolving the root of an available domain (#634)
fix bug when resolving the root of an available domain
2023-03-07 16:22:03 -06:00
Daniel Holmgren 112e4f1610 Format invite codes with dashes (#633)
* format invite code with dashes

* fixed test
2023-03-07 16:21:52 -06:00
Daniel Holmgren f4a029a897 Handle fixes (#632)
* no 500s on unresolvable handles

* dont resolve handle for service
2023-03-07 13:16:20 -06:00
Takumi Fujiwara 97dac2bbdc Fix README of handle package (#630) 2023-03-05 23:16:30 -05:00
Daniel Holmgren 8dfcb4f996 Integrating PLC lib (#607)
* integrating new plc lib

* patching up did-resolver

* buffing up pds tests

* didResolver on ctx & plc in postgres

* bring dev env up to date

* re-add extension for linting

* tidy

* use current env vars

* pr feedback
2023-03-04 19:43:59 -06:00
Daniel Holmgren 3bd50b0e5f Feed builder indices (#627)
* add indices on creators

* add orderby indices

* remove creator indices
2023-03-04 16:13:15 -05:00
devin ivy f57063bebe Feed and search query perf tuning (#628)
Switch unnecessary unions to union all in pds queries
2023-03-04 15:26:56 -05:00
devin ivy 50c892c835 Lengthen token lifetimes (#624)
* Lengthen refresh token lifetime

* Lengthen access token lifetime
2023-03-03 15:05:36 -05:00
Akiomi Kamakura 1c7e641ff7 Fix README.md (#622) 2023-03-03 10:48:05 -05:00
Daniel Holmgren 07d941639f Repo ops on sync stream (#614)
* repo op schema

* repo ops on sync stream

* delete old seqs

* nullable cids

* pr feedback
2023-03-02 12:18:49 -06:00
dholms 4c346d2017 publish @atproto/common 0.1.1 2023-03-01 15:54:21 -08:00
Paul Frazee 5e5d3df615 Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
mehdiMj 19a81fdcb5 Fix copyright year (#609)
The year wasn't correct, so I fix it :)
2023-03-01 15:53:08 -06:00
devin ivy 13ff521bf5 Ensure subscribeAllRepos syncs from cursor 0 (#611)
Ensure subscribeAllRepos allows sync from sequence 0
2023-03-01 16:25:17 -05:00
devin ivy a9cab00ea1 XRPC subscription consumer (#603)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Yield info frames in byMessage()
2023-03-01 11:28:11 -05:00
Daniel Holmgren 6ee3a8cfda Crawler schemas (#602)
* sketch of crawl schemas

* codgen

* hostname param

* Update lexicons/com/atproto/sync/requestCrawl.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* codegen

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-02-28 18:24:28 -06:00
devin ivy 5d2895f1ea Running a task as a leader in the app view (#604)
Interface for picking a pds leader
2023-02-27 14:24:08 -05:00
Daniel Holmgren bb05232720 Sync subscription tweaks (#599)
* tweaks to sync subscription

* fix test
2023-02-27 10:22:54 -06:00
devin ivy 8826ada589 Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold fea90cb392 Merge pull request #601 from bluesky-social/jake/add-container-labels
Add standard container image labels
2023-02-23 16:24:57 -08:00
Jake Gold 627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
bnewbold f08c08a675 MST review and tests (#556)
* mst: more test coverage

In particular, ports the older fanout=32 corner-case tests to fanout=16.

I generated new paths with correct level and ordering, but haven't
actually verified that the resulting structures have the shape of the
ASCII diagrams in text.

* mst: additional comments, including TODO questions

* mst: default new MST layer to 'null' not 0

Looks like all current call sites of this 'create()' function override
the layer already. This is just making it explicit that the default,
null, means the layer has not been computed yet, distinct from being
layer=0.

* mst: remove one TODO and reformat another

From review.
2023-02-23 10:47:16 -08:00
Daniel Holmgren f29c08f0bb Bump package versions (#578)
* bump pkgs

* fix package
2023-02-22 19:54:21 -06:00
devin ivy ca87aeb93f Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren 136b4b5933 Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
Jake Gold 93d54f49a6 Merge pull request #571 from bluesky-social/jake/add-pds-container-build
PDS container build-and-push action
2023-02-22 08:17:15 -08:00
devin ivy 24fd3b17e2 Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren c2c87c3226 Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
devin ivy 3a6d785ae0 Fix PDS build, updating main to dist in api package (#566) 2023-02-18 16:15:19 -05:00
Daniel Holmgren 66f731ecee Repo sync semantic changes (#563)
* switch to cursor & only one event type

* quick bugfix

* no dangling async stuff

* didnt rebuild lex-cli
2023-02-17 17:33:31 -05:00
Daniel Holmgren 20a35b31c1 Add com.atproto.sync.getBlocks XRPC method (#564)
* add sync getBlocks method

* oops didnt build lex-cli
2023-02-17 17:18:13 -05:00