705 Commits

Author SHA1 Message Date
Daniel Holmgren
ad1fcf1387
Repo history rewrite (#1479)
* logical changes in repo

* tests

* tweak commit data

* building pds

* patching up some tests

* tidy + more tests

* patch up bsky

* clean up db

* db migration

* small patches

* fix up another test

* reinclude prevs

* api & lex updates

* add back in deprecated routes

* backward compatibility for commit v2

* add upgrade repo version root

* move namespace

* migration test

* patch up a few more tests

* remove deprecated rebase routes

* tweak api

* sprinkle rev around a few more places

* getCurrent -> getLatestCommit

* fix test

* pr feedback & tidy

* fix up block pagination

* tidy again

* add in a tets

* add cursor to listBlobs

* getRepo rev -> since

* clean up proofs test

* dont change getHead

* tweak migrate route

* build branch

* hit index in block range query

* check for dupe record refs

* tidy

* set prev to null

* dont build branch
2023-08-29 19:07:21 -05:00
Daniel Holmgren
895a21fc25
Clean up block & mute handling for push notifs (#1520)
clean up block & mute handling for push notifs
2023-08-25 17:43:40 -05:00
Daniel Holmgren
46d7672476
Fix dependencies in dockerfiles (#1519)
* fix dependencies in dockerfiles

* fix hanging reference to uri
2023-08-25 10:49:17 -05:00
bnewbold
bd3966555f
@atproto/syntax: merge of @atproto/identifier, uri, and nsid (#1464)
* move nsid package in to identifier

* move uri package in to identifier

* update packages to pull from identifier, not nsid or uri

* rename @atproto/identifier to @atproto/syntax

* update all refs from @atproto/identifer to @atproto/syntax
2023-08-25 10:18:10 -05:00
devin ivy
d28c9ab3de
Speed-up a few hot queries on pds and bsky (#1518)
* hit index for notif unread count in appview

* fix index usage on pds account lookup by handle

* add index to blob tempkey column to support some hot queries
2023-08-25 11:09:15 -04:00
Paul Frazee
e1b69f37b2
Notifications improvements (#1512)
* Add collapse keys to notifications

* Ensure stop processing after a notification result has been added

* Simplify the collapse key to the notif reason

* Update test

* Fix tests

* build branch

* Tune notif rate limit to dramatically reduce engagement types but always deliver conversation types

* dont build branch

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-24 14:29:32 -05:00
Daniel Holmgren
0a2d1f5e1e
Partial reprocessing of repos (#1508)
* disable labels on reprocessing

* wait for reprocess

* avoid creating notifications on repo reindexing

* partial reindexes

* tidy

* bail out case for push notifs

* return -> continue

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-24 12:37:05 -05:00
Foysal Ahamed
d614678c49
Revert labels on takedown reversal (#1496)
*  Revert labels on takedown reversal

*  Add tests to cover label reversal

*  Add label reversal test to bsky package

*  Move xrpc call out of db transaction
2023-08-24 11:27:33 -04:00
Foysal Ahamed
82ef0ad6d7
Add self labels to getRecord and getRepo on com.atproto namespace (#1488)
*  Add self labels to getRecord and getRepo

*  Update snapshot to include self labelling
2023-08-24 11:25:16 -04:00
Ansh
eae682f3bf
Native Notifications (#1428)
* pseudocode for sending notification to user

* add notification push token table

* lexicon codegen

* `pds` and `api` codegen

* update lexicon

* add simple function to `putNotificationPushToken` into `notification_push_token` table

* reorgnaize imports

* add unspecced `putNotificationPushToken` to pds

* add `notification-push-tokens` table to PDS

* check if token exists before adding it to db

* add endpoint and appId to PDS table

* setup notification server

* fix logic for inserting token into db

* fix NotificationServer methods by making them static

* fix merge conflicts

* add comments on how sending ntoifications work

* remove dead code

* move notifServer to AppContext

* refactor code to handle notif sending in `Indexer` service

* add additional data when sending notifs

* clean up code

* move notif logic to indexer

* add `appId` and `endpoint` optional params to putNotificationPushToken

* clone notification code to AppView

* add endpoint to register push token with app view

* disable pds and enable app view notification server

* clean up code

* simply logic to check if a token already exists

* remove NotificationServer from PDS

* remove notification-push-token table from PDS

* remove `putNotificationPushToken` endpoint

* clean up code

* let `axios` throw error if `gorush` has an error

* let `kysely` throw error if notif cannot be registered by client

* rename `registerPushNotificationEndpoint` to `registerPushNotifications`

* delete `putNotificationPushToken` from AppView

* rename putPushNotificationToken to registerPushNotification

* remove dead notification code from pds

* remove sanitizeDisplayName from NotificationSever

* move `pushNotificationEndpoint` to config

* temp add `pushNotificationEndpoint` to dev-env setup

* remove example test from feed-generation.test.ts

* add test for registerPushNotification and clean up error handling

* move notifications test to its own file

* add test for NotificationServer to check if tokens are retrieved correctly

* remove unused functions from NotificationServer

* add additional tests for NotificationServer

* add return type to getNotificationDisplayAttributes function

* remove unnecessary comments

* remove dead NotificationServer code from PDS

* clean up code to prepareNotifsToSend

* put sending notifications as part of the backgroundQueue

* log instead of throwing error if notification attributes don't exist

* remove logs

* add more tests to `notification-server.test.ts`

* show replied with text for reply notifs

* better error handling when sending notifications via backgroundQueue

* add rate limit and batching to sending notifications

* add comments to NotificationServer

* merge with main

* use redis for rate limiting instead of normal rate limits

* move `notificationBatchSize` into config

* usePrimary db in test

* hoist push notif migration to present, update model to remove endpoint

* update push notif lexicon

* pare down lex for unregistering push

* helpers for working with service endpoints from did doc

* service-authed register/unregister push notifs

* add well-known endpoint to appview

* update bsky notif service tests

* fix to batching logic, misc tidy

* batch display notifications

* colocate all notification sending logic

* tidy tests

* remove unregister token for now

* fix registerPush lexicon, make a procedure

* fix registerPush impl, test pds proxy to notif service

* fix tests, make notif server optional when not configured

* fix notif server config for bsky app service

* move notif server rate limiting in-mem for now, add sending retry

* codegen tidy

* only push notifs on commit

* build

* fix notif rate limiter check

* send notifs from users w/o a profile

* remove build

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-23 18:37:14 -04:00
Daniel Holmgren
bbc6a2c0fd
Indexer request reprocess (#1504)
* allow requesting reprocessing a repo from an indexer

* partition id guardrailes

* test

* build branch

* dont set ports

* resupply port

* give port in correct spot

* dont build branch
2023-08-23 11:19:47 -05:00
devin ivy
831fe9fc29
Handle negative ISO year for indexing (#1505)
* handle negative iso year

* fix pkg main

* tidy
2023-08-22 21:07:07 -04:00
Eric Bailey
98cc8c5214 codegen 2023-08-22 11:16:56 -05:00
Eric Bailey
7364f17b33 Merge remote-tracking branch 'origin' into eric/app-796-likes-tab
* origin:
  Tweak rate limits (#1502)
  Fix auth on bsky author feed (#1501)
  Bugfix: Application ratelimits relative import (#1500)
  Application ratelimits (#1382)
  format
  do not notify users of own-actions
  Fix runtime flags open handles (#1497)
  make sure to await rejected promise expectations
  Misc fixes (#1492)
2023-08-22 11:15:20 -05:00
Eric Bailey
e5c96f596a format 2023-08-22 11:15:09 -05:00
Eric Bailey
4cdb43c7f3 view likes from muted accounts 2023-08-22 11:07:06 -05:00
Eric Bailey
2c7bc6535f
Merge pull request #1499 from bluesky-social/eric/app-1201-self-notifications
Don't notify user of self-likes and self-reposts
2023-08-22 10:54:40 -05:00
devin ivy
e938228db4
Fix auth on bsky author feed (#1501)
fix auth on bsky author feed
2023-08-21 18:06:24 -05:00
Daniel Holmgren
891bf3bb7a
Application ratelimits (#1382)
* initial setup

* lexgen

* tidying things up

* add in some rate limiting

* testing rate limits

* small bugfix

* fix build

* increase rate limit

* more limits

* config

* optional rate limiter

* tweak headers

* ratelimit headers

* multiple rate limits

* tests & bugfixes

* test bypass

* slight refactor

* fail open

* fail open

* right most xff ip

* setup redis for ratelimiting

* redis test

* more tests

* use new dev-infra package

* adjust limits

* temporarily remove repo write limits

* codegen

* redis scratch var name

* cfg var host -> address
2023-08-21 16:10:04 -05:00
Eric Bailey
896c5b0ad1 format 2023-08-21 16:08:29 -05:00
Eric Bailey
5316d3f65d do not notify users of own-actions 2023-08-21 16:08:09 -05:00
Eric Bailey
24a5414b64 make sure to await rejected promise expectations 2023-08-21 15:42:45 -05:00
Daniel Holmgren
b45c9aab6b
Misc fixes (#1492)
* proxy typeahead

* proxy post thread

* disable migration

* error handling on db

* fix case where replica explicitly configured w/ no tags

* appview build

* tidy

* block handling on some appview endpoints

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-21 15:30:34 -04:00
Eric Bailey
be794d2026 opaque errors 2023-08-21 10:02:15 -05:00
Eric Bailey
3576d10845 allow network to process 2023-08-21 09:59:47 -05:00
Eric Bailey
9b1f5f0124 only allow viewing your own likes 2023-08-18 16:25:07 -05:00
Eric Bailey
048cd16d0c Merge remote-tracking branch 'origin' into eric/app-796-likes-tab
* origin: (21 commits)
  styleguide
  Increase CI test matrix size (#1490)
  tweak author-feed rejects test syntax
  update test
  format
  add repost of reply, update snaps
  simplify query
  fix typos
  include reposted replies in posts_no_replies filter
  fix bad test syntax
  fix actor takedown tests
  use getActor, failing atm
  only include media posts by post creator
  Fix condition for viewing soft-deleted followers (#1485)
   Expose takendown profile, their follows and followers to mods (#1456)
  Handle revalidation (#1474)
  Handle db pool errors on appview (#1483)
  Handle db client errors on appview (#1481)
  v0.6.4
   Allow creating moderation action with a duration (#1431)
  ...
2023-08-18 11:10:04 -05:00
Eric Bailey
2b228a1e6f styleguide 2023-08-17 18:30:29 -05:00
Eric Bailey
9082a9ac78 Merge remote-tracking branch 'origin' into eric/fix-1197-reposted-replies
* origin:
  Increase CI test matrix size (#1490)
  Fix condition for viewing soft-deleted followers (#1485)
   Expose takendown profile, their follows and followers to mods (#1456)
  Handle revalidation (#1474)
  Handle db pool errors on appview (#1483)
  Handle db client errors on appview (#1481)
2023-08-17 18:25:32 -05:00
Daniel Holmgren
2413f48c93
Increase CI test matrix size (#1490)
* increase test matrix size

* tweak author feed tests
2023-08-17 18:24:42 -05:00
Eric Bailey
1c50428428 tweak author-feed rejects test syntax 2023-08-17 18:21:13 -05:00
Eric Bailey
88ea0b2693 update test 2023-08-17 17:16:21 -05:00
Eric Bailey
33685b8408 format 2023-08-17 15:50:12 -05:00
Eric Bailey
0ea353b21f add repost of reply, update snaps 2023-08-17 15:30:30 -05:00
Eric Bailey
2e90124f72 simplify query 2023-08-17 15:02:50 -05:00
Eric Bailey
377a9721b9 fix typos 2023-08-17 14:20:05 -05:00
Eric Bailey
d4b7d29c6b
Merge pull request #1486 from bluesky-social/eric/fix-1167-media-reposts
only include media posts by post creator
2023-08-17 14:18:15 -05:00
Eric Bailey
30cb6412ce include reposted replies in posts_no_replies filter
(cherry picked from commit 588cf2562cc61441c170fec4f2354eedee39c4f6)
2023-08-17 14:09:24 -05:00
Eric Bailey
40fde9585c fix bad test syntax 2023-08-17 14:02:56 -05:00
Eric Bailey
4347a21d92 fix actor takedown tests 2023-08-17 13:50:26 -05:00
Eric Bailey
ff0d2df6e3 use getActor, failing atm 2023-08-17 13:32:19 -05:00
Eric Bailey
1a5b422718 only include media posts by post creator 2023-08-17 12:10:36 -05:00
Foysal Ahamed
7c88b43fc5
Expose takendown profile, their follows and followers to mods (#1456)
*  Allow moderators to see takendown profiles

*  Allow moderators to see follows and followers of takendown account

* ♻️ Let auth check fail on optional verifier

* ♻️ Use role type to check moderator access
2023-08-17 09:47:34 -04:00
Daniel Holmgren
4241ee16ec
Handle revalidation (#1474)
* easier hanlde revalidation

* remove duplicate line

* backup handle nameservers on appview

* fix tests & add a couple
2023-08-16 18:34:19 -05:00
devin ivy
34b84131be
Handle db pool errors on appview (#1483)
handle db pool errors on bav
2023-08-16 18:22:42 -04:00
devin ivy
8eb930382c
Handle db client errors on appview (#1481)
handle db client errors on bav
2023-08-16 14:02:42 -04:00
Foysal Ahamed
7bc701e2e1
Allow creating moderation action with a duration (#1431)
*  Add action duration to action model

*  Add periodic moderation action reversal

*  Use actionDurationInHours and remove takedownExpiresAt

*  Destroy mod action reversal job

*  Add test for automatic moderation action reversal

* 🧹 Port over actionDuration to bsky package

* ♻️ Better naming for action duration and minor fixes

* ♻️ Optionally build expiresAt based on createdAt when logging action

* ♻️ Adjust constructor params

*  Copy over migration from pds to bsky package

* 🚨 Fix indentation

*  Propagate action reversal from app-view to pds

*  Revert locally only if pushAgent is not found

* build

* merge dbcoordinator work into periodic mod actions

* fix entrypoint for periodic mod reversals

* do not run periodic reversals on sequencer leader

* fix duration in action detail on bsky

* update env for pushing mod reversals

* allow zero-duration actions for testing

* remove build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-15 15:44:53 -04:00
Eric Bailey
e0e7b39b16 add block and mute tests 2023-08-15 13:46:58 -05:00
Eric Bailey
79dcad4cf7 handle mutes 2023-08-15 11:25:38 -05:00
Eric Bailey
d9849b6776 get it working 2023-08-14 18:31:15 -05:00