Commit Graph

2335 Commits

Author SHA1 Message Date
Daniel Holmgren b3046c348f Remove legacy repo sync impl (#1570)
remove legacy repo sync impl
2023-09-08 15:25:30 -05:00
Daniel Holmgren f8f2d04c1d Move fuzzy matcher to appview (#1566)
* impl on appview

* tests + clean up pds

* tidy cfg
2023-09-08 15:08:27 -05:00
Paul Frazee d1df1f3232 @atproto/api@0.6.11 2023-09-08 10:37:24 -07:00
Paul Frazee 44b721b6ae Add personal-details user preference with birth date (#1565)
* Add personal-details user preference with birth date

* Add personal details pref to settings union
2023-09-08 10:35:49 -07:00
Eric Bailey 15411f274b maintain feed order (#1559)
* maintain feed order

* clean up order, protect against undefined values

* format
2023-09-07 14:10:38 -05:00
Eric Bailey a7c42cfe39 Clean up prettier/eslint scripts (#1514)
* add prettierignore, hoist script

* upgrade prettier

* bump prettier eslint deps

* format all files

* hoist prettier check

* remove unused scripts, hoist lint:fix

* remove npm-run-all, unused

* hoist lint

* remove lint scripts

* improve lint scripts

* remove prettierignores

* downgrade prettier deps to fix codgen

* reformat all files

* update makefile, format

* fix locklife

* final format

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-09-06 19:27:50 -05:00
Daniel Holmgren f023cfb009 Abyss Integration (#1545)
* add blob scanner

* introduce auto moderator

* tidy

* tests

* more tests & tidy

* timeout & logs

* tweak cfg

* push takedowns to pds

* build branch

* update snaps

* add cloudfront invalidator to indexer

* tweak log

* dont build branch
2023-09-06 17:35:39 -05:00
Daniel Holmgren a948d53969 Only do read after write on own author feed (#1544)
only munge on own feed
2023-09-06 10:08:18 -05:00
David Buchanan 90b0a9d809 RFC: Support image aspect ratio metadata (#1306)
* Add aspectRatio field to embed image lexicon

* lexicon codegen

* pass aspectRatio through to imagesEmbed views

* simplify aspectRatio logic, port to bsky package

* re-run lexicon codegen
2023-09-06 10:08:06 -05:00
Eric Bailey 90e8325e5f pnpm setup (#1532)
* use pnpm

* fix dependency issues, replace yarn and lerna scripts

* remove the main/dist scripts

* update Dockerfiles

* use pnpm

* fix dependency issues, replace yarn and lerna scripts

* remove the main/dist scripts

* update Dockerfiles

* update bin script

* remove unused zod dep

* fix type errors in pds

* add types prop to packages

* remove unused, bump lock

* fix test running

* build before test

* fix pino types

* format

* pds depends on dev-env in test

* refer to src instead of built packages

* pds relies on bsky in test too

* remove yarn.lock

* add -r flag to root test

* test push to aws

* remove docker test

* add publishConfig to new package

* move services to top level dir

(cherry picked from commit f5012bec33435a4473e9960066807623334f3aff)

* update workflow paths

(cherry picked from commit 5c70f0176d381ca35d6be10cfa173e22373a5b5d)

* add main-to-dist script

* use script in all packages, remove old Dockerfiles

* remove old bsky service

* remove newline

* test container builds

* Revert "test container builds"

This reverts commit c228611f5e8e1624d4b124be4976c49590130f43.

* remove unused config

* test build containers

* pnpm in syntax

* bump dd-trace

* shamefully hoist

* even more shame

* hoist, externalize deps

* clean install for prod and smaller containers

* dont build branches

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-09-05 18:45:49 -05:00
Foysal Ahamed f7186f03a9 Improve matching in results from searchRepos (#1507)
*  Better results from searchRepos when matching by handle

*  Adjust tests for repo search result order

* 🧹 Cleanup

* ♻️ Refactor to re-use search query builder

* 🧹 Cleanup

*  Add test for pagination in search result
2023-09-05 10:56:19 -04:00
Foysal Ahamed 4fb32e1e06 Add missing labels that are available on the moderation tool (#1515)
* Add missing labels that are available on the moderation tool

* Remove unused label

* More cleanup

* 🧹 Get rid of ncii as well

*  Add new label in json instead of generated file 🤦🏽‍♂️
2023-09-05 10:49:55 -04:00
Eric Bailey 16662b220d @atproto/api@0.6.9 (#1546)
* @atproto/api@0.6.9

* @atproto/api@0.6.10
2023-09-01 10:43:30 -05:00
Eric Bailey f84027fe8e add getFeedSuggestions (#1542)
* add getFeedSuggestions lex

* move to feed namespace

* codegen

* add table and migration

* add endpoints

* remove comment

* add test

* fix order

* rename

* just remove proxy check
2023-08-31 17:19:34 -05:00
Daniel Holmgren 9a0fd91f5c Limit rev cache (#1543)
* disable rev cache

* disable rev cache

* use limit instead of disabling

* dont build branch
2023-08-31 15:35:47 -05:00
Daniel Holmgren f8c89e54ca Legacy checkouts (#1539)
* serve legacy repos & checkouts

* tidy

* check lock before formatting commit

* tidy locks

* clean up db test

* check error on failed commit format

* be more generous with new blocks

* Update packages/pds/src/db/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* force upgrade & ensure lock

* make it a bit smarter

* add log

* dont build branch

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-08-31 11:03:22 -05:00
Devin Ivy f9d6673da9 @atproto/repo@0.3.0 2023-08-30 00:14:06 -04:00
Paul Frazee 6d765f7f61 @atproto/api@0.6.8 2023-08-29 17:13:35 -07:00
Paul Frazee 8db9077c68 @atproto/api@0.6.7 2023-08-29 17:10:13 -07:00
Paul Frazee 2b8cbcee1f Add preferences helpers to SDK (#1537) 2023-08-29 17:09:46 -07:00
Daniel Holmgren ad1fcf1387 Repo history rewrite (#1479)
* logical changes in repo

* tests

* tweak commit data

* building pds

* patching up some tests

* tidy + more tests

* patch up bsky

* clean up db

* db migration

* small patches

* fix up another test

* reinclude prevs

* api & lex updates

* add back in deprecated routes

* backward compatibility for commit v2

* add upgrade repo version root

* move namespace

* migration test

* patch up a few more tests

* remove deprecated rebase routes

* tweak api

* sprinkle rev around a few more places

* getCurrent -> getLatestCommit

* fix test

* pr feedback & tidy

* fix up block pagination

* tidy again

* add in a tets

* add cursor to listBlobs

* getRepo rev -> since

* clean up proofs test

* dont change getHead

* tweak migrate route

* build branch

* hit index in block range query

* check for dupe record refs

* tidy

* set prev to null

* dont build branch
2023-08-29 19:07:21 -05:00
Daniel Holmgren 70c8159644 Fix ambiguous reference to repoRev (#1534)
fix ambiguous column
2023-08-29 14:30:26 -05:00
dholms 913c912bc4 v0.2.2 2023-08-28 19:10:53 -05:00
dholms d0a3d70700 v0.2.1 2023-08-28 19:04:30 -05:00
Daniel Holmgren 89917d7fc6 Bump pkg versions (#1525)
* v0.2.0

* bump xrpc
2023-08-28 15:00:24 -05:00
Daniel Holmgren cc231b5584 Fix broken sqlite migration (#1526)
fix broken sqlite migration
2023-08-28 14:56:37 -05:00
Foysal Ahamed 964e08ad82 🐛 Add labels caching as a config for pds and enable it when running dev-env (#1511)
* 🐛 Add labels caching as a config for pds and enable it when running dev-env

*  Move caching enable config to bin.ts
2023-08-28 09:39:57 -04:00
Daniel Holmgren 895a21fc25 Clean up block & mute handling for push notifs (#1520)
clean up block & mute handling for push notifs
2023-08-25 17:43:40 -05:00
devin ivy acafe8e986 Stop indexing post hierarchy on pds (#1358)
* serve pds post threads w/o post_hierarchy, continue to index post_hierarchy

* add missing files

* remove post-hierarchy table from pds

* uncomment migration

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-25 17:19:35 -04:00
Daniel Holmgren 46d7672476 Fix dependencies in dockerfiles (#1519)
* fix dependencies in dockerfiles

* fix hanging reference to uri
2023-08-25 10:49:17 -05:00
bnewbold c2f47b725d identity: parse support for newer DID document format (Multikey) (#1475)
* identity: parse support for newer DID document format (Multikey)

* identity: lint fixes

* identity: DID doc parsing allow full DID URL in service 'id'

* add parse/format multikey methods

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-25 10:19:27 -05:00
bnewbold bd3966555f @atproto/syntax: merge of @atproto/identifier, uri, and nsid (#1464)
* move nsid package in to identifier

* move uri package in to identifier

* update packages to pull from identifier, not nsid or uri

* rename @atproto/identifier to @atproto/syntax

* update all refs from @atproto/identifer to @atproto/syntax
2023-08-25 10:18:10 -05:00
devin ivy d28c9ab3de Speed-up a few hot queries on pds and bsky (#1518)
* hit index for notif unread count in appview

* fix index usage on pds account lookup by handle

* add index to blob tempkey column to support some hot queries
2023-08-25 11:09:15 -04:00
devin ivy 19d2bdc457 Stop using post hierarchy in pds (#1357)
* serve pds post threads w/o post_hierarchy, continue to index post_hierarchy

* add missing files

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-24 17:10:59 -05:00
Paul Frazee e1b69f37b2 Notifications improvements (#1512)
* Add collapse keys to notifications

* Ensure stop processing after a notification result has been added

* Simplify the collapse key to the notif reason

* Update test

* Fix tests

* build branch

* Tune notif rate limit to dramatically reduce engagement types but always deliver conversation types

* dont build branch

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-24 14:29:32 -05:00
Daniel Holmgren 0a2d1f5e1e Partial reprocessing of repos (#1508)
* disable labels on reprocessing

* wait for reprocess

* avoid creating notifications on repo reindexing

* partial reindexes

* tidy

* bail out case for push notifs

* return -> continue

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-24 12:37:05 -05:00
Foysal Ahamed d614678c49 Revert labels on takedown reversal (#1496)
*  Revert labels on takedown reversal

*  Add tests to cover label reversal

*  Add label reversal test to bsky package

*  Move xrpc call out of db transaction
2023-08-24 11:27:33 -04:00
Foysal Ahamed 82ef0ad6d7 Add self labels to getRecord and getRepo on com.atproto namespace (#1488)
*  Add self labels to getRecord and getRepo

*  Update snapshot to include self labelling
2023-08-24 11:25:16 -04:00
Paul Frazee 1dd0c46593 @atproto/api@0.6.6 2023-08-23 16:18:04 -07:00
Ansh eae682f3bf Native Notifications (#1428)
* pseudocode for sending notification to user

* add notification push token table

* lexicon codegen

* `pds` and `api` codegen

* update lexicon

* add simple function to `putNotificationPushToken` into `notification_push_token` table

* reorgnaize imports

* add unspecced `putNotificationPushToken` to pds

* add `notification-push-tokens` table to PDS

* check if token exists before adding it to db

* add endpoint and appId to PDS table

* setup notification server

* fix logic for inserting token into db

* fix NotificationServer methods by making them static

* fix merge conflicts

* add comments on how sending ntoifications work

* remove dead code

* move notifServer to AppContext

* refactor code to handle notif sending in `Indexer` service

* add additional data when sending notifs

* clean up code

* move notif logic to indexer

* add `appId` and `endpoint` optional params to putNotificationPushToken

* clone notification code to AppView

* add endpoint to register push token with app view

* disable pds and enable app view notification server

* clean up code

* simply logic to check if a token already exists

* remove NotificationServer from PDS

* remove notification-push-token table from PDS

* remove `putNotificationPushToken` endpoint

* clean up code

* let `axios` throw error if `gorush` has an error

* let `kysely` throw error if notif cannot be registered by client

* rename `registerPushNotificationEndpoint` to `registerPushNotifications`

* delete `putNotificationPushToken` from AppView

* rename putPushNotificationToken to registerPushNotification

* remove dead notification code from pds

* remove sanitizeDisplayName from NotificationSever

* move `pushNotificationEndpoint` to config

* temp add `pushNotificationEndpoint` to dev-env setup

* remove example test from feed-generation.test.ts

* add test for registerPushNotification and clean up error handling

* move notifications test to its own file

* add test for NotificationServer to check if tokens are retrieved correctly

* remove unused functions from NotificationServer

* add additional tests for NotificationServer

* add return type to getNotificationDisplayAttributes function

* remove unnecessary comments

* remove dead NotificationServer code from PDS

* clean up code to prepareNotifsToSend

* put sending notifications as part of the backgroundQueue

* log instead of throwing error if notification attributes don't exist

* remove logs

* add more tests to `notification-server.test.ts`

* show replied with text for reply notifs

* better error handling when sending notifications via backgroundQueue

* add rate limit and batching to sending notifications

* add comments to NotificationServer

* merge with main

* use redis for rate limiting instead of normal rate limits

* move `notificationBatchSize` into config

* usePrimary db in test

* hoist push notif migration to present, update model to remove endpoint

* update push notif lexicon

* pare down lex for unregistering push

* helpers for working with service endpoints from did doc

* service-authed register/unregister push notifs

* add well-known endpoint to appview

* update bsky notif service tests

* fix to batching logic, misc tidy

* batch display notifications

* colocate all notification sending logic

* tidy tests

* remove unregister token for now

* fix registerPush lexicon, make a procedure

* fix registerPush impl, test pds proxy to notif service

* fix tests, make notif server optional when not configured

* fix notif server config for bsky app service

* move notif server rate limiting in-mem for now, add sending retry

* codegen tidy

* only push notifs on commit

* build

* fix notif rate limiter check

* send notifs from users w/o a profile

* remove build

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-23 18:37:14 -04:00
Daniel Holmgren bbc6a2c0fd Indexer request reprocess (#1504)
* allow requesting reprocessing a repo from an indexer

* partition id guardrailes

* test

* build branch

* dont set ports

* resupply port

* give port in correct spot

* dont build branch
2023-08-23 11:19:47 -05:00
devin ivy 831fe9fc29 Handle negative ISO year for indexing (#1505)
* handle negative iso year

* fix pkg main

* tidy
2023-08-22 21:07:07 -04:00
Paul Frazee 7e4edb2ce1 @atproto/api@0.6.5 2023-08-22 12:13:26 -07:00
Eric Bailey c28013b685 remove unused 2023-08-22 11:21:26 -05:00
Eric Bailey 98cc8c5214 codegen 2023-08-22 11:16:56 -05:00
Eric Bailey 7364f17b33 Merge remote-tracking branch 'origin' into eric/app-796-likes-tab
* origin:
  Tweak rate limits (#1502)
  Fix auth on bsky author feed (#1501)
  Bugfix: Application ratelimits relative import (#1500)
  Application ratelimits (#1382)
  format
  do not notify users of own-actions
  Fix runtime flags open handles (#1497)
  make sure to await rejected promise expectations
  Misc fixes (#1492)
2023-08-22 11:15:20 -05:00
Eric Bailey e5c96f596a format 2023-08-22 11:15:09 -05:00
Eric Bailey 4cdb43c7f3 view likes from muted accounts 2023-08-22 11:07:06 -05:00
Eric Bailey 2c7bc6535f Merge pull request #1499 from bluesky-social/eric/app-1201-self-notifications
Don't notify user of self-likes and self-reposts
2023-08-22 10:54:40 -05:00
Daniel Holmgren ad25430691 Tweak rate limits (#1502)
* tweak rate limits

* tweak #

* fix test
2023-08-21 18:13:01 -05:00