1225 Commits

Author SHA1 Message Date
devin ivy
b7210bf0a1
Fix strict_word_similarity_threshold configuration (#880)
* In-progress work on optimizing actor search

* Tidy-up approach to using trgm index on actor search, apply to bsky appview

* Set trigram on session rather than on db via migration
2023-04-24 13:18:08 -04:00
devin ivy
0c02804a73
Ensure indexes are used on actor search queries (#879)
* In-progress work on optimizing actor search

* Tidy-up approach to using trgm index on actor search, apply to bsky appview

* Fix comment typo
2023-04-24 11:17:30 -04:00
Daniel Holmgren
a360cc6004
Chunk getRepo query (#877)
chunk get repo query
2023-04-24 10:09:19 -05:00
Daniel Holmgren
553f59923c
Get posts method (#876)
* implement getPosts method

* tests

* bsky tests

* comment out test

* Handle stripping record-with-media viewer info in bsky tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-23 23:43:37 -05:00
Daniel Holmgren
f30887bc1c
Clean up Did-resolver (#842)
* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* couple helpers around cache invalidation

* fix expired check

* change cache semantics

* use map instead of record

* stale + expired

* tests

* clear entry method

* clear missing dids

* add missing type
2023-04-23 13:36:01 -05:00
Daniel Holmgren
01578bc5eb
Better handle races in createAccount (#868)
* better handle races in create account

* better locking

* cleanup

* tidy comment
2023-04-23 12:31:25 -05:00
Daniel Holmgren
2b5347b534
Remove profile labels (#866)
* moderation tweks

* fix: typo. (#818)

* Create invite codes for many accounts (#825)

* create invite codes for many accounts

* test

* Revamp dev env (#796)

* fix up a couple of tsc errors in app view merge

* wip

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* fixed missed merge conflict

* fix one more merge conflict

* fix popular test

* Sync test env with labeler changes (#836)

fix test-env

* Remove extraneous info on getRecord (#835)

* remove extraneous info on getRecord

* fix test-env

* Fix duplication of constants in the crypto package (#819)

* App passwords (#826)

* app password lex & auth chnages

* scrypt things

* implemented app password refresh tokens

* db tidy & migration

* revocation + bugfixin

* tests, listing passwords & cleanup

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* remove profile labels

* rm temp branch from workflow

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: S. Ota <1632335+susumuota@users.noreply.github.com>
Co-authored-by: Ilya Siamionau <ilya@marshal.dev>
2023-04-23 12:18:24 -05:00
devin ivy
d86cf03cd8
More forgiving config for empty db schema (#871)
* Add more config for pds db pool

* More forgiving handling of empty db schema config
2023-04-22 14:26:31 -04:00
devin ivy
92322af49a
Add more config for pds db pool (#867) 2023-04-21 22:13:41 -04:00
devin ivy
a832c54de7
PDS moderator credentials (#863)
* Setup config and auth verifiers for moderators

* Enforce admin vs. moderator access on PDS admin/server endpoints

* Tidy
2023-04-21 20:58:34 -04:00
Daniel Holmgren
89109ef32d
Format commits outside of tx (#860)
* format commits outside of tx

* fix tests

* fix bsky test

* use persistent cache

* reuse repo storage between attempts to get caching benefits

* better error handling

* tests

* rm failing sqlite test

* pr feedback
2023-04-21 18:13:04 -05:00
Daniel Holmgren
4f1ad109af
Increase getPopular likes to 8 (#862)
* increase get popular likes to 8

* use new like counts agg
2023-04-21 18:12:44 -05:00
devin ivy
a0f0301241
Serve materialized post and profile aggregations (#859)
* Add migration for post & profile aggs to pds

* Post/profile agg db tweaks

* Post/profile aggregation queries on pds

* Add facility for performing work on db commit in pds

* Add background queue to pds

* Perform aggregations during indexing on background queue

* Tidy pds background queue, incorporate into labeler

* Fix pds onCommit, remove unused indexing dep, fix zeros for aggs

* Finish background tasks for view tests in pds

* Update aggregations on account deletion, tidy

* Update profile and post views to serve materialized aggregations
2023-04-21 12:01:23 -04:00
devin ivy
587b24352d
Materialize post and profile aggregations on write (#858)
* Add migration for post & profile aggs to pds

* Post/profile agg db tweaks

* Post/profile aggregation queries on pds

* Add facility for performing work on db commit in pds

* Add background queue to pds

* Perform aggregations during indexing on background queue

* Tidy pds background queue, incorporate into labeler

* Fix pds onCommit, remove unused indexing dep, fix zeros for aggs

* Finish background tasks for view tests in pds

* Update aggregations on account deletion, tidy
2023-04-21 11:52:45 -04:00
Daniel Holmgren
24f7183896
Improve streaming sync routes (#848)
* improve streaming sync routes

* pr feedback
2023-04-20 10:26:55 -05:00
Daniel Holmgren
9a8d7529a3
Skip locked repos instead of waiting for lock (#847)
* skip locked repos instead of waiting for lock

* some tests

* rm racing tests on profile

* subscribe repos test
2023-04-19 21:57:06 -05:00
Daniel Holmgren
2c24cf6c15
Bump node-postgres version (#846)
* Bump node-postgres version

* fix package.json things
2023-04-19 19:06:56 -05:00
Daniel Holmgren
c71e351c38
Do not give notifs for a deleted subject (#843)
generally do not give notifs for a deleted subject
2023-04-19 16:24:18 -05:00
devin ivy
c70cb2b6e5
Notifications from the appview (#829)
* Update notif lexicons for stateless seenAt param

* In-progress work on appview notifs

* Fix-up bsky notification methods

* Add appview notification table table

* Process notifications in bsky appview, test notif indexing

* Test bsky appview notification methods

* Tidy bsky notification tests

* Explicitly don't support seenAt notif params on pds

* Tidy bsky notifs tests

* Sync bsky notif handling with pds

* Remove stale comment

* Switch bsky notifs tests to use testenv
2023-04-18 17:55:03 -04:00
Daniel Holmgren
6446e8d1d6
App passwords (#826)
* app password lex & auth chnages

* scrypt things

* implemented app password refresh tokens

* db tidy & migration

* revocation + bugfixin

* tests, listing passwords & cleanup

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/pds/src/db/scrypt.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-18 13:47:04 -05:00
Daniel Holmgren
6e2a3d3924
Remove extraneous info on getRecord (#835)
* remove extraneous info on getRecord

* fix test-env
2023-04-18 12:29:02 -05:00
Daniel Holmgren
9e1bb49957
Create invite codes for many accounts (#825)
* create invite codes for many accounts

* test
2023-04-18 10:44:34 -05:00
devin ivy
6a39ebf9f7
Stop sequencer listener in outbox when subscription aborts (#828)
* Update types to allow for abort signal in xrpc subscriptions

* Support abort signal in xrpc-server subscriptions

* Stop sequencer listener in outbox when subscription aborts
2023-04-17 14:22:03 -04:00
Daniel Holmgren
d8b50c73e4
Add admin.updateAccountEamil (#812)
* -add admin capability to update account email

* pr feedback
2023-04-13 14:43:53 -05:00
devin ivy
4aa9103fb5
Minor query optimization for getBlocksForCommits() (#811) 2023-04-13 14:39:49 -05:00
Daniel Holmgren
efb1cac2bf
Label reversal fix (#809)
* Reproduce issue with empty negateLabelVals

* normalize str arrays

* Reproduce issue with reversing labels on repo

* Use multiple labels for record labeling test

* join with space

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-12 23:48:47 -05:00
Daniel Holmgren
aa46ad1e1c
Admin labeling (#801)
* adding to moderation flows

* some bugfixing in labels

* hardcode profile nsid

* make labels off moderation action

* db updates

* wip

* report labels in admin views & reverse moderation actions

* Test admin get record and repo w/ labels

* update db

* fix

* exclude negs from labels

* exclude neg on moderation views as well

* Check-in missing lex

* Check-in missing lex

* In-progress admin label tests

* Test label creation/reversal via actions

* Admin label test snapshots (#808)

* new snapshots for label on user

* fix get moderation action snap

* fix dev-env

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-12 22:53:32 -05:00
bnewbold
6592fcd6eb
Lex: additional report types (#788)
* lex: more report types

* lex: basic definitions of report reasonTypes

* package/api codegen: report types

* packages/pds codegen: report types

* packages/bsky codegen: report types; also roll-up many other changes
2023-04-12 22:32:57 -05:00
Devin Ivy
3f632d8bdb v0.1.4 2023-04-12 18:09:48 -07:00
Daniel Holmgren
46e7ba986d
Update one label (#807) 2023-04-12 20:02:51 -05:00
Daniel Holmgren
f6aef2b50f
Fix race condition in labeling (#806)
* Fix race condition in labeling

* comment
2023-04-12 19:39:12 -05:00
Daniel Holmgren
d6ac47d76e
Label profiles in feeds (#805)
* Label profiles in feeds

* send empty array
2023-04-12 18:14:23 -05:00
Daniel Holmgren
521c1c1edb
Admin update handle (#803)
* admin route for updating accoutn handle on services domain

* tests

* Update lexicons/com/atproto/admin/updateAccountHandle.json

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-12 17:23:58 -05:00
Daniel Holmgren
0762e8b7f6
Label bugfixes (#802)
* some bugfixing in labels

* hardcode profile nsid
2023-04-12 17:23:32 -05:00
Daniel Holmgren
c41232b2ef
Fix DB schema in rebase tests (#799)
fix typoed db schema in test
2023-04-12 14:34:39 -05:00
Daniel Holmgren
8b1da9fed8
Labeling (#789)
* take notifs off of message queue

* fix order of notif handling

* table

* reintroducing message queue

* setting up user

* added labeler classes

* tidy + hook up labeler to config / service start

* hooking up p-queue to labeler

* rip out message queue

* drop mq tables

* rm stream consumers

* keyword labeler

* tidy + some tests

* work labels into views

* update snaps

* labeled images in dev-env

* snaps

* labeler tests

* more labels

* Update packages/pds/src/api/app/bsky/notification/listNotifications.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/dev-env/src/mock/index.ts

* Update packages/dev-env/src/mock/index.ts

* Update packages/pds/src/labeler/util.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/pds/tests/labeler/hive.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback + migration change

* fix lower case keywords issue

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-12 14:34:18 -05:00
Daniel Holmgren
d8b975d8dd
Normalize handles on resolve (#795)
* normalize handles on resolve

* tidy & test
2023-04-12 10:49:49 -07:00
Daniel Holmgren
42e9c69b0a
Fix bug in getPopular (#794)
fix bug in getPopular
2023-04-12 12:24:15 -05:00
Daniel Holmgren
286c6bbef5
Remove mutes & takedowns from getPopular (#793)
remove mutes & takedowns from getPopular
2023-04-11 22:58:06 -07:00
Daniel Holmgren
a9e725bb19
Adjustable db pool size (#792)
* add pool size env

* Update packages/pds/service/index.js

Co-authored-by: devin ivy <devinivy@gmail.com>

* part int

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-04-11 22:01:10 -07:00
devin ivy
153542ae81
Drop PDS event queue tables (#790)
* take notifs off of message queue

* fix order of notif handling

* rip out message queue

* drop mq tables

* rm stream consumers

* Tidy

* Temporarily remove drop message queue migration

* Revert "Temporarily remove drop message queue migration"

This reverts commit 95c02e128a650469543f5b14418c4ff32c088d20.

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-04-11 21:56:01 -07:00
Daniel Holmgren
95ac1808ac
No notifs for any records from muted accounts (#791) 2023-04-11 21:55:20 -07:00
Daniel Holmgren
15762260fd
Sync/listRepos (#784)
* add getHostedRepos lex & route

* tests

* tidy

* rename
2023-04-11 17:16:13 -07:00
Daniel Holmgren
fc4a34caf2
Fully deprecate handle on createSession (#761)
* fully deprecate handle on createSession

* fix tests
2023-04-11 17:10:35 -07:00
Daniel Holmgren
8d0cbb4c86
Create bulk invite codes (#783)
add bulk create invite code route
2023-04-11 17:09:53 -07:00
Daniel Holmgren
8ecc232796
Synchronous notifs (#785)
* take notifs off of message queue

* fix order of notif handling

* rip out message queue

* drop mq tables

* rm stream consumers

* Tidy

* Temporarily remove drop message queue migration

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-11 17:05:39 -07:00
bnewbold
aa20d20efb
More label lexicons (#672)
* com.atproto.label.label lexicon

For persisting un-signed label objects in repositories.

There will likely be a handful of XRPC query endpoints in the
com.atproto.label namespace in the near future.

* lexicons: com.atproto.label.subscribeLabels

* lexicons: com.atproto.label.query

* lexicons: label updates post lex-refactor

* lexicons: rename label.query to label.queryLabels

* lexicons: queryLabels larger batch sizes; improve description

* lexicons: add labels to profiles, posts, notifications

* lexicons: add negation ('neg') flag to labels

* lexicon: label vals on moderation actions

This is a mechanism to create (or negate) labels as part of a moderation
action. The fields are optional, so this is backwards compatible, and if
we wanted to embed entire labels in the future (instead of just "val"
strings), that would be possible using a different field name.

* lexicons: update label.label to label.defs#label

* codegen

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-04-11 12:44:19 -05:00
Devin Ivy
9697eaab89 Temporary, bump keepalive timeout on pds 2023-04-09 19:37:28 -07:00
devin ivy
6b55a95c88
Build timeline and author feeds from feed items index (#774)
* Index posts and reposts into feed_item table for building feeds

* Use feed_item table for building timeline and author feeds

* Apply feed item indexing to bsky app view

* Fix bsky appview tests, test getPopular

* Use feed item index to build feeds in bsky app view
2023-04-09 13:21:48 -07:00
devin ivy
2b8e1ac387
Index posts and reposts as feed items (#773)
* Index posts and reposts into feed_item table for building feeds

* Apply feed item indexing to bsky app view
2023-04-09 12:13:18 -07:00