2251 Commits

Author SHA1 Message Date
Daniel Holmgren
f4a029a897
Handle fixes (#632)
* no 500s on unresolvable handles

* dont resolve handle for service
2023-03-07 13:16:20 -06:00
Takumi Fujiwara
97dac2bbdc
Fix README of handle package (#630) 2023-03-05 23:16:30 -05:00
Daniel Holmgren
8dfcb4f996
Integrating PLC lib (#607)
* integrating new plc lib

* patching up did-resolver

* buffing up pds tests

* didResolver on ctx & plc in postgres

* bring dev env up to date

* re-add extension for linting

* tidy

* use current env vars

* pr feedback
2023-03-04 19:43:59 -06:00
Daniel Holmgren
3bd50b0e5f
Feed builder indices (#627)
* add indices on creators

* add orderby indices

* remove creator indices
2023-03-04 16:13:15 -05:00
devin ivy
f57063bebe
Feed and search query perf tuning (#628)
Switch unnecessary unions to union all in pds queries
2023-03-04 15:26:56 -05:00
devin ivy
50c892c835
Lengthen token lifetimes (#624)
* Lengthen refresh token lifetime

* Lengthen access token lifetime
2023-03-03 15:05:36 -05:00
Akiomi Kamakura
1c7e641ff7
Fix README.md (#622) 2023-03-03 10:48:05 -05:00
Daniel Holmgren
07d941639f
Repo ops on sync stream (#614)
* repo op schema

* repo ops on sync stream

* delete old seqs

* nullable cids

* pr feedback
2023-03-02 12:18:49 -06:00
dholms
4c346d2017 publish @atproto/common 0.1.1 2023-03-01 15:54:21 -08:00
Paul Frazee
5e5d3df615
Bump @atproto/api@0.1.3, @atproto/pds@0.0.4 (#613)
* @atproto/api@0.1.3

* @atproto/pds@0.0.4
2023-03-01 17:04:38 -06:00
mehdiMj
19a81fdcb5
Fix copyright year (#609)
The year wasn't correct, so I fix it :)
2023-03-01 15:53:08 -06:00
devin ivy
13ff521bf5
Ensure subscribeAllRepos syncs from cursor 0 (#611)
Ensure subscribeAllRepos allows sync from sequence 0
2023-03-01 16:25:17 -05:00
devin ivy
a9cab00ea1
XRPC subscription consumer (#603)
* Add helper to consume subscription message frames

* Implement xrpc subscription consumer

* Simplify subscription consumer params

* Remove duped test

* Reconnect to subscription on initial setup

* Add reconnect hook on subscription, for logging

* Make subscription consumption cancelable

* Include error information in sub reconnect hook

* Yield info frames in byMessage()
2023-03-01 11:28:11 -05:00
Daniel Holmgren
6ee3a8cfda
Crawler schemas (#602)
* sketch of crawl schemas

* codgen

* hostname param

* Update lexicons/com/atproto/sync/requestCrawl.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* codegen

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-02-28 18:24:28 -06:00
devin ivy
5d2895f1ea
Running a task as a leader in the app view (#604)
Interface for picking a pds leader
2023-02-27 14:24:08 -05:00
Daniel Holmgren
bb05232720
Sync subscription tweaks (#599)
* tweaks to sync subscription

* fix test
2023-02-27 10:22:54 -06:00
devin ivy
8826ada589
Logical separation of app view (#572)
* Separate app.bsky methods out into app view

* Fix websocket issue while splitting-out app view

* Start organizing app-view tables out of pds

* Move feed service, getTimeline, vote table into app-view

* Move record processing/indexing logically out of pds and into app-view

* Pull actor service functionality into app-view

* Factor app-view user functionality out of pds, replace actor w/ account service

* Tidy
2023-02-24 17:22:51 -05:00
Jake Gold
fea90cb392
Merge pull request #601 from bluesky-social/jake/add-container-labels
Add standard container image labels
2023-02-23 16:24:57 -08:00
Jake Gold
627b3023f8 add standard container image labels 2023-02-23 13:42:03 -08:00
bnewbold
f08c08a675
MST review and tests (#556)
* mst: more test coverage

In particular, ports the older fanout=32 corner-case tests to fanout=16.

I generated new paths with correct level and ordering, but haven't
actually verified that the resulting structures have the shape of the
ASCII diagrams in text.

* mst: additional comments, including TODO questions

* mst: default new MST layer to 'null' not 0

Looks like all current call sites of this 'create()' function override
the layer already. This is just making it explicit that the default,
null, means the layer has not been computed yet, distinct from being
layer=0.

* mst: remove one TODO and reformat another

From review.
2023-02-23 10:47:16 -08:00
Daniel Holmgren
f29c08f0bb
Bump package versions (#578)
* bump pkgs

* fix package
2023-02-22 19:54:21 -06:00
devin ivy
ca87aeb93f
Post embeds (#553)
* Add lexicon for post embeds

* Model post embeds

* Handle and present post embeds

* Test and tidy post embeds

* Generalize post embeds to record embeds

* Tidy
2023-02-22 12:05:15 -05:00
Daniel Holmgren
136b4b5933
Temporarily disable PLC mutations (#575)
* temporarily disable all plc ops

* skip server tests for now

* dont skip create tests

* temp disable handle tests
2023-02-22 10:39:14 -06:00
Jake Gold
93d54f49a6
Merge pull request #571 from bluesky-social/jake/add-pds-container-build
PDS container build-and-push action
2023-02-22 08:17:15 -08:00
devin ivy
24fd3b17e2
Containerize PDS build (#561)
Add pds service w/ dockerfile
2023-02-22 10:59:09 -05:00
Daniel Holmgren
c2c87c3226
Fix content sniffing on streams (#570)
fix content sniffing on streamsx
2023-02-20 13:14:29 -06:00
devin ivy
3a6d785ae0
Fix PDS build, updating main to dist in api package (#566) 2023-02-18 16:15:19 -05:00
Daniel Holmgren
66f731ecee
Repo sync semantic changes (#563)
* switch to cursor & only one event type

* quick bugfix

* no dangling async stuff

* didnt rebuild lex-cli
2023-02-17 17:33:31 -05:00
Daniel Holmgren
20a35b31c1
Add com.atproto.sync.getBlocks XRPC method (#564)
* add sync getBlocks method

* oops didnt build lex-cli
2023-02-17 17:18:13 -05:00
Jake Gold
6357bc3509 add PDS build-and-push action 2023-02-16 15:06:11 -08:00
Devin Ivy
3f6482d0d2 Tidy snapshots 2023-02-16 16:32:17 -05:00
Daniel Holmgren
6c4f6a27f3
Bugfix: Flaky readFromGenerator test (#560)
fix flaky test
2023-02-16 14:34:26 -06:00
Daniel Holmgren
d4b1262f28
DNS handles (#555)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* basic support for dns resolution on handles

* handle resolution

* fix up account create

* tsconfig

* workshop handle constraints

* bring pds up to speed w handle changes

* change dns subdomain

* another lil test

* bugfix

* update dns record format

* typo
2023-02-16 13:59:56 -06:00
bnewbold
e3cd9c23a1
Small PDS and XRPC server fixes (#550)
* pds: fix logging of Bearer auth subject (DID)

* xrpc-server: handle missing and mangled Content-Type better

Eg, for blob uploads.

Also adds a test for incorrect mimetype supplied for a regular endpoint
(one not allowing "*/*").

* skip failing test

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-02-16 13:59:34 -06:00
devin ivy
9c119fb9c6
Allow unsetting profile images and description in updateProfile (#554)
* Add support for nullable on lex objects

* Support clearing fields in updateProfile lexicon

* Support clearing fields in updateProfile method
2023-02-16 14:32:25 -05:00
Paul Frazee
5737e653e3
Bump @atproto/pds@0.0.3, @atproto/api@0.1.2 (#558)
* @atproto/pds@0.0.3

* @atproto/api@0.1.2
2023-02-16 12:29:01 -06:00
devin ivy
850b54fad6
Add following state to profile/actor views, reorg views (#539)
* Add lexicon for getProfiles

* Split out presentation of profiles, implement getProfiles, test

* Cull actor/profile view lexicons down to three variants

* Map viewer in profile view, mark todos, lex tweak

* Apply basic profile view to methods, lex fix

* Switch searchTypeahead to actor-with-info view

* Switch getVotes to actor-with-info view

* Switch getMutes to actor-with-info view

* Switch getFollows and getFollowers to actor-with-info view

* Switch getRepostedBy to actor-with-info view

* Switch feed and notification views to use updated actor-with-info view
2023-02-16 11:40:05 -05:00
Daniel Holmgren
57da2270b9
Update handles (#547)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing

* some updating hanldes

scheams

* impl + passing test

* more handle tests

* tidy

* update did doc + some new tests

* one more test

* test handle casing

* Fix did pkey down migration

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-02-15 14:50:29 -05:00
devin ivy
a24be34c27
Notify all ancestors of reply, optimize getPostThread method (#545)
* Initialize post_hierarchy index

* Build post_hierarchy index from posts

* Power getPostThread method with post hierarchy index

* Generate ancestor reply notifications, fix tests and down migration

* Generate notifications for all reply ancestors
2023-02-15 14:16:35 -05:00
Daniel Holmgren
ac48330fdb
User table fixes (#536)
* change pkey on user table to did & rename to user_account

* migration

* tidy

* fixes suggested by bryn

* missed merge thing
2023-02-15 14:15:32 -05:00
devin ivy
807e2a783c
Apply lexicon defaults (#552)
* Thread values w/ applied defaults through validation, use in xrpc-server

* Adjust codegen to reflect xrpc defaults
2023-02-14 21:20:20 -05:00
devin ivy
751f16527d
Use DAG-CBOR for XRPC streaming (#546)
* Use canonical dag cbor for xrpc streaming

* Remove undefineds from subscribeAllRepos for dag-cbor
2023-02-13 15:08:26 -05:00
bnewbold
19c54fe55b
Merge pull request #537 from bluesky-social/bnewbold/dev-misc
some small low-priority patches
2023-02-12 20:23:05 -08:00
devin ivy
9a7b2c39e4
Mod action and report params as integers rather than floats (#544) 2023-02-10 16:40:32 -05:00
bnewbold
6ec452450a
pds sql: add indices for feed and post aggregations (#538)
* pds sql: add indices for feed and post aggregations

Seems like aggregations/stats/counts are likely to get changed up in the
near term future, but in the very short term, these simple index
additions yielded non-trivial speedups even at small database sizes, in
my laptop testing.

* pds migration: fix index drops in 'down'
2023-02-09 21:01:05 -05:00
Paul Frazee
d252fd3e95
Fix knownValues for reasonType lexicon (#527) 2023-02-09 20:59:32 -05:00
bnewbold
a3c1105aca
handles: reserve some top-100 handles; prevent new 2-char handle registrations (#531)
* handle: disallow new 2-char handle registration

This should not impact existing handles, only new registrations.

We discussed blocking these to prevent possible future URL path
collisions with, eg, ISO country or language codes.

* handle: reserve some more famous handles

This isn't the full list of thousands of twitter handles that have more
than 5 million or more than 1 million followers. Instead, it is only the
top 100 or so, plus a handful of additional top-50 instagram accounts (when
different), and a couple recent famous figures. Pretty subjective.

* handle: overall and segment lengths; enforce ASCII; test dashes

The 'address' package we are using to check domains doesn't seem to do
any overall length checks. The values and examples i'm testing here are
based on general domain limits. Technically a web domain could be 255,
it is email domains limited to 253 chars, but that seems like a
reasonable value to cut off at.
2023-02-09 20:58:42 -05:00
bryan newbold
7b5f8906d1 plc: run dotenv before any other imports
This fixes, eg, logging configuration via .env variables.
2023-02-09 16:40:28 -08:00
bryan newbold
8b803e013c pds: run dotenv before any other imports
This fixes logging configuration via env vars set in .env file
2023-02-09 16:40:14 -08:00
bryan newbold
d545b3195e pds: add missing 'license' to package.json 2023-02-09 16:40:06 -08:00