440 Commits

Author SHA1 Message Date
Daniel Holmgren
4bf61c5f95
Enable updateEmail with no token & no app password (#1715)
* dont allow updateEmail with an app password

* re-enable email updates
2023-10-09 13:46:32 -05:00
Daniel Holmgren
d664b51c64
Finalize PDS in-process AppView removal (#1198)
* rm tables

* rm event-stream & proxied

* Remove appview services, move label service to pds

* only proxy appview stuff

* delete more tables

* Start removing message dispatched from pds

* more syncing-up removal of message dispatcher in pds

* merged

* remove feedgens from pds, remove getPopular

* remove unused image helper from pds

* fixing compiler errors

* clean up sharp

* rm label service

* first pass on cleaning up tests

* fix up a bunch of tests

* moderation view tests

* last admin tests

* got a lil overzealous in deletes

* clean up unused cfg

* clean up label table

* simplify admin repo search query/logic

* tidy pds entrypoint

* in-progress pds config changes

* cfg fiddling

* finish cleaning up cfg/ctx

* comments

* building

* pds prefix on env

* test env

* collapse pds migrations down into a single migration

* fix up dev-env

* tidy

* cleanup

* fix pds admin tests

* fix handle test

* fix pds proxy tests

* fix subscribe repos test

* fix sqlite config in pds tests

* add sqlite clause in sequencer-leader

* fix actor search w/ sqlite on pds

* fixes

* fix dev env build

* update pds service entrypoint

* simple env example

* make takedown ids opaque identifiers in the pds

* use pds routes for api tests

* update pds dockerfile with volume and correct port env var

* add a couple env vars to example

* add comments to env example

* @atproto/pds 0.2.0-beta.0

* @atproto/aws 0.0.1-beta.0

* appview did

* @atproto/aws 0.0.1

* enable logs by default

* update env example

* bugfixing sandbox issues

* consistency in pds env var name for appview url

* log on pds start and stop, configure version at runtime

* @atproto/pds 0.2.0-beta.1

* fix semver matching for pds beta version

* v0.2.0-beta.2

* default invites to being not required

* fix flaky test

* limit db connections in tests

* publish 0.2.0-beta.d3

* fix invite required parsing

* @atproto/pds 0.2.0-beta.5

* Proxy getPopularFeedGenerators on simplified pds (#1222)

proxy getPopularFeedGenerators on pds

Co-authored-by: dholms <dtholmgren@gmail.com>

* tidy migrations

* fix service entry

* bump version

* change auth order

* bump version

* bump version

* add upgradeRepoVersion & fallback url for cdn

* bump version

* merging

* merge pds

* building dev-env

* merging tests

* merge service entry

* test fixing

* tidy

* fix admin search

* tidy

* tidy

* add snap for getListFeed

* add backup nameserver cfg

* tidy + pr feedback

* tidy

* tidy env

* bit more

* re-add dotenv to root package.json

* fix dep

* build branch

* fix tests

* Refactor tests to make better use of dev-env (#1690)

* refactor pds tests to use dev env

* refactor bsky tests

* fix pds test

* tidy bsky tests

* build pds correctly

* fix entry point

* default logging to false (for now)

* format service entry

* Switch takedown ids back to ints on pds distribution (#1694)

* switch takedown ids back to ints, consistent with live pds

* tidy/fix migration

* update migration for sqlite

* export moderation action reversal

* takedown tests

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-10-02 13:27:45 -05:00
Daniel Holmgren
11e3b329e3
Clean up other email tokens (#1572)
* lexicons

* codegen

* email templates

* request routes

* impl

* migration

* tidy

* tests

* tidy & bugfixes

* format

* fix api test

* fix auth test

* impl

* update constraint name

* temporarily disable unconfirmed updates

* tidy

* fix some tests

* validate email syntax
2023-09-28 13:25:56 -05:00
Eric Bailey
0fa34e4819
apply basic email validation to createAccount (#1658)
* apply basic email validation to createAccount

* format
2023-09-28 13:17:36 -05:00
Eric Bailey
9b2ba28fff
improve tag validation (#1684)
* check faceted tags for slurs

* format

* fix baby bug
2023-09-28 11:45:29 -05:00
Daniel Holmgren
41ee177f5a
Email confirmation/update (#1568)
* lexicons

* codegen

* email templates

* request routes

* impl

* migration

* tidy

* tests

* tidy & bugfixes

* format

* fix api test

* fix auth test

* codegen

* add unique constraint

* Add email confirmed to AtpSessionData

* interop test files (#1529)

* initial interop-test-files

* crypto: switch signature-fixtures.json to a symlink

* syntax: test against interop files

* prettier

* Update interop-test-files/README.md

Co-authored-by: Eric Bailey <git@esb.lol>

* disable prettier on test vectors

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>

* add getSuggestedFollowsByActor (#1553)

* add getSuggestedFollowsByActor lex

* remove pagination

* codegen

* add pds route

* add app view route

* first pass at likes-based suggested actors, plus tests

* format

* backfill with suggested_follow table

* combine actors queries

* fall back to popular follows, handle backfill differently

* revert seed change, update test

* lower likes threshold

* cleanup

* remove todo

* format

* optimize queries

* cover mute lists

* clean up into pipeline steps

* add changeset

* List feeds (#1557)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* lex

* list purpose

* lex gen

* add route

* add proxy route

* seed client helpers

* tests

* mutes and blocks

* proxy test

* snapshot

* hoist actors out of composeThread()

* tidy

* tidy

* run ci on all prs

* format

* format

* fix snap name

* fix snapsh

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Improve xrpc server error handling (#1597)

improve xrpc server error handling

* Remove appview proxy runtime flags (#1590)

* remove appview proxy runtime flags

* clean up proxy tests

* getPopular hotfix (#1599)

dont pass all params

* Interaction Gating (#1561)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* initial lexicons for interaction-gating

* add interactions view to post views

* codegen

* model bad reply/interaction check state on posts

* initial impl for checking bad reply or interaction on write

* omit invalid interactions from post thread

* support not-found list in interaction view

* hydrate can-reply state on threads

* present interaction views on posts

* misc fixes, update snaps

* tidy/reorg

* tidy

* split interaction gating into separate record in lexicon

* switch interaction-gating impl to use separate record type

* allow checking reply gate w/ root post deletion

* fix

* initial gating tests

* tighten gated reply views, tests

* reply-gating list rule tests

* allow custom post rkeys within window

* hoist actors out of composeThread()

* tidy

* update thread gate lexicons, codegen

* lex fix

* rename gate to threadgate in bsky, update views

* lex fix

* improve terminology around reply validation

* fix down migration

* remove thread gates on actor unindexing

* add back .prettierignore

* tidy

* run ci on all prs

* syntax

* run ci on all prs

* format

* fix snap

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* order by `like.indexedAt` in app view (#1592)

* order by like.indexedAt

* use keyset for ordering

* simplify

* ok ok ok I get it now

* Update packages/bsky/src/api/app/bsky/feed/getActorLikes.ts

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* Remove default value for post table invalid attrs (#1601)

remove default value for post table attrs

* Version packages (#1602)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* update Bluesky PBLLC to PBC (Public Benefit Corporation) (#1600)

* Temporarily disable filtering `invalidReplyRoot`s (#1609)

temporarily disable invalidReplyRoot check

* fix syntax docs (#1611)

* Version packages (#1612)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Allow bypass on ratelimit ip (#1613)

allow bypass on ratelimit ip

* Write rate limits (#1578)

* get rate limit ip correctly

* add write rate-limits

* Tweak createSession rate limit key (#1614)

tweak create session rl key

* Filter preferences for app passwords (#1626)

filter preferences for app passwords

* Tweak rate limit setup for multi rate limit routes (#1627)

tweak rate limit setup for multi rate limit routes

* Remove zod from xrpc-server error handling (#1631)

remove zod from xrpc-server error handling check

* Enforce properties field on lexicon object schemas (#1628)

* add empty properites to thread gate schema fragments

* tweak lexicon type

* Add feed-vew and thread-view preferences (#1638)

* Add feed and thread preference lexicons

* Add feed-view and thread-view preference APIs

* Add changeset for new preferences  (#1639)

Add changeset

* Version packages (#1640)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable getAccountInviteCodes for app passwords (#1642)

disable getAccountInviteCodes for app passwords

* remove cruft packages (uri, nsid, identifier) (#1606)

* remove @atproto/nsid (previously moved to syntax)

* remove @atproto/uri (previously moved to syntax)

* remove @atproto/identifier (previously moved to syntax)

* bump lockfile to remove old packages

---------

Co-authored-by: Eric Bailey <git@esb.lol>

* api: update login/resumeSession examples in README (#1634)

* api: update login/resumeSession examples in README

* Update packages/api/README.md

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* small syntax lints (#1646)

* lint: remove unused imports and variables

* lint: prefix unused args with '_'

* eslint: skip no-explicit-any; ignore unused _var (prefix)

* eslint: explicitly mark ignores for tricky cases

* indicate that getPopular is deprecated (#1647)

* indicate that getPopular is deprecated

* codegen for deprecating getPopular

* tidy up package.json and READMEs (#1649)

* identity: README example and tidy

* tidy up package metadata (package.json files)

* updated README headers/stubs for several packages

* crypto: longer README, with usage

* syntax: tweak README

* Apply suggestions from code review

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

* Improve the types of the thread and feed preferences APIs (#1653)

* Improve the types of the thread and feed preferences APIs

* Remove unused import

* Add changeset

* Version packages (#1654)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable pds appview routes (#1644)

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch

* auto-moderator tweaks: pass along record URI, create report for takedown action (#1643)

* auto-moderator: include record URI in abyss requests

* auto-moderator: log attempt at hard takedown; create report as well

The motivation is to flag the event to mod team, and to make it easier
to confirm that takedown took place.

* auto-mod: typo fix

* auto-mod: bugfixes

* bsky: always create auto-mod report locally, not pushAgent (if possible)

* bsky: fix auto-mod build

* bsky: URL-encode scanBlob call

* Clear follow viewer state when blocking (#1659)

* clear follow viewer state when blocking

* tidy

* add `tags` to posts (#1637)

* add tags to post lex

* kiss

* add richtext facet and validation attrs

* add tag validation attrs to post

* codegen

* add maxLength for tags, add description

* validate post tags on write

* add test

* handle tags in indexer

* add tags to postView, codegen

* return tags on post thread view

* format

* revert formatting change to docs

* use establish validation pattern

* add changeset

(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)

* remove tags from postView, codegen

* remove tags from thread view

* revert unused changes

* Version packages (#1664)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* merge

* Reverse order of blocks from sync.getRepo (#1665)

* reverse order of blocks from sync.getRepo

* write to car while fetching next page

* Add hashtag detection to richtext (#1651)

* add tag detection to richtext

* fix duplicate tag index error

* add utils

* fix leading space index failures, test for them

* add changeset

* Version packages (#1669)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* proposed new search lexicons (#1594)

* proposed new search lexicons

* lexicons: lint

* lexicons: fix actors typo

* lexicons: camelCase bites again, ssssss

* lexicons: add 'q' and mark 'term' as deprecated for search endpoints

* codegen for search lexicon updates

* bsky: prefer 'q' over 'term' in existing search endpoints

* search: bugfix

* lexicons: make unspecced search endpoints return skeleton obj

* re-codegen for search skeleton obj

* Disable pds appview indexing (#1645)

* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build

* Refactor PDS appview routes (#1673)

move routes around

* Strip leading `#` from from detected tag facets (#1674)

ensure # is removed from facets

* Version packages (#1675)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Proxy search queries (#1676)

* proxy search

* tweak profile resp

* fix admin.searchRepos

* add mock mailer

* Fix to daniel's MOCKERY of a mock mailer

* Don't allow non-verified email updates until app feature is out (#1682)

stricter updating email until app feature is out

* changesets

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-27 16:08:58 -05:00
Daniel Holmgren
2fa9088639
Disable pds appview indexing (#1645)
* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build
2023-09-25 20:09:52 -05:00
Eric Bailey
d96f7d9b84
add tags to posts (#1637)
* add tags to post lex

* kiss

* add richtext facet and validation attrs

* add tag validation attrs to post

* codegen

* add maxLength for tags, add description

* validate post tags on write

* add test

* handle tags in indexer

* add tags to postView, codegen

* return tags on post thread view

* format

* revert formatting change to docs

* use establish validation pattern

* add changeset

(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)

* remove tags from postView, codegen

* remove tags from thread view

* revert unused changes
2023-09-25 11:14:57 -05:00
devin ivy
d703bc36e6
Clear follow viewer state when blocking (#1659)
* clear follow viewer state when blocking

* tidy
2023-09-25 10:27:48 +02:00
Daniel Holmgren
558981ead1
Disable pds appview routes (#1644)
* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch
2023-09-22 13:51:15 -05:00
bnewbold
7e36c48a77
small syntax lints (#1646)
* lint: remove unused imports and variables

* lint: prefix unused args with '_'

* eslint: skip no-explicit-any; ignore unused _var (prefix)

* eslint: explicitly mark ignores for tricky cases
2023-09-21 10:07:54 -07:00
Daniel Holmgren
36ba295333
Remove appview proxy runtime flags (#1590)
* remove appview proxy runtime flags

* clean up proxy tests
2023-09-14 13:55:12 -05:00
Daniel Holmgren
578757cb44
List feeds (#1557)
* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* lex

* list purpose

* lex gen

* add route

* add proxy route

* seed client helpers

* tests

* mutes and blocks

* proxy test

* snapshot

* hoist actors out of composeThread()

* tidy

* tidy

* run ci on all prs

* format

* format

* fix snap name

* fix snapsh

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-09-14 11:19:04 -05:00
Daniel Holmgren
07bb0da0cf
Remove temp.upgradeRepoVersion (#1588)
remove temp.upgradeRepoVersion
2023-09-12 12:37:57 -05:00
Daniel Holmgren
3cdc72a0a7
Fix getRepo since (#1579)
* fix schemas

* test

* format
2023-09-12 12:37:45 -05:00
devin ivy
6f419790ac
Feature: block lists (#1531)
* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* hoist actors out of composeThread()

* tidy

* run ci on all prs

* format

* build

* proxy graph.getListBlocks

* remove unneeded index

* build pds

* setup noop listblock indexer on pds

* remove build

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-09-12 12:07:31 -04:00
Daniel Holmgren
f8f2d04c1d
Move fuzzy matcher to appview (#1566)
* impl on appview

* tests + clean up pds

* tidy cfg
2023-09-08 15:08:27 -05:00
Daniel Holmgren
f023cfb009
Abyss Integration (#1545)
* add blob scanner

* introduce auto moderator

* tidy

* tests

* more tests & tidy

* timeout & logs

* tweak cfg

* push takedowns to pds

* build branch

* update snaps

* add cloudfront invalidator to indexer

* tweak log

* dont build branch
2023-09-06 17:35:39 -05:00
Daniel Holmgren
f8c89e54ca
Legacy checkouts (#1539)
* serve legacy repos & checkouts

* tidy

* check lock before formatting commit

* tidy locks

* clean up db test

* check error on failed commit format

* be more generous with new blocks

* Update packages/pds/src/db/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* force upgrade & ensure lock

* make it a bit smarter

* add log

* dont build branch

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-08-31 11:03:22 -05:00
Daniel Holmgren
ad1fcf1387
Repo history rewrite (#1479)
* logical changes in repo

* tests

* tweak commit data

* building pds

* patching up some tests

* tidy + more tests

* patch up bsky

* clean up db

* db migration

* small patches

* fix up another test

* reinclude prevs

* api & lex updates

* add back in deprecated routes

* backward compatibility for commit v2

* add upgrade repo version root

* move namespace

* migration test

* patch up a few more tests

* remove deprecated rebase routes

* tweak api

* sprinkle rev around a few more places

* getCurrent -> getLatestCommit

* fix test

* pr feedback & tidy

* fix up block pagination

* tidy again

* add in a tets

* add cursor to listBlobs

* getRepo rev -> since

* clean up proofs test

* dont change getHead

* tweak migrate route

* build branch

* hit index in block range query

* check for dupe record refs

* tidy

* set prev to null

* dont build branch
2023-08-29 19:07:21 -05:00
bnewbold
bd3966555f
@atproto/syntax: merge of @atproto/identifier, uri, and nsid (#1464)
* move nsid package in to identifier

* move uri package in to identifier

* update packages to pull from identifier, not nsid or uri

* rename @atproto/identifier to @atproto/syntax

* update all refs from @atproto/identifer to @atproto/syntax
2023-08-25 10:18:10 -05:00
devin ivy
19d2bdc457
Stop using post hierarchy in pds (#1357)
* serve pds post threads w/o post_hierarchy, continue to index post_hierarchy

* add missing files

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-24 17:10:59 -05:00
Foysal Ahamed
d614678c49
Revert labels on takedown reversal (#1496)
*  Revert labels on takedown reversal

*  Add tests to cover label reversal

*  Add label reversal test to bsky package

*  Move xrpc call out of db transaction
2023-08-24 11:27:33 -04:00
Foysal Ahamed
82ef0ad6d7
Add self labels to getRecord and getRepo on com.atproto namespace (#1488)
*  Add self labels to getRecord and getRepo

*  Update snapshot to include self labelling
2023-08-24 11:25:16 -04:00
Ansh
eae682f3bf
Native Notifications (#1428)
* pseudocode for sending notification to user

* add notification push token table

* lexicon codegen

* `pds` and `api` codegen

* update lexicon

* add simple function to `putNotificationPushToken` into `notification_push_token` table

* reorgnaize imports

* add unspecced `putNotificationPushToken` to pds

* add `notification-push-tokens` table to PDS

* check if token exists before adding it to db

* add endpoint and appId to PDS table

* setup notification server

* fix logic for inserting token into db

* fix NotificationServer methods by making them static

* fix merge conflicts

* add comments on how sending ntoifications work

* remove dead code

* move notifServer to AppContext

* refactor code to handle notif sending in `Indexer` service

* add additional data when sending notifs

* clean up code

* move notif logic to indexer

* add `appId` and `endpoint` optional params to putNotificationPushToken

* clone notification code to AppView

* add endpoint to register push token with app view

* disable pds and enable app view notification server

* clean up code

* simply logic to check if a token already exists

* remove NotificationServer from PDS

* remove notification-push-token table from PDS

* remove `putNotificationPushToken` endpoint

* clean up code

* let `axios` throw error if `gorush` has an error

* let `kysely` throw error if notif cannot be registered by client

* rename `registerPushNotificationEndpoint` to `registerPushNotifications`

* delete `putNotificationPushToken` from AppView

* rename putPushNotificationToken to registerPushNotification

* remove dead notification code from pds

* remove sanitizeDisplayName from NotificationSever

* move `pushNotificationEndpoint` to config

* temp add `pushNotificationEndpoint` to dev-env setup

* remove example test from feed-generation.test.ts

* add test for registerPushNotification and clean up error handling

* move notifications test to its own file

* add test for NotificationServer to check if tokens are retrieved correctly

* remove unused functions from NotificationServer

* add additional tests for NotificationServer

* add return type to getNotificationDisplayAttributes function

* remove unnecessary comments

* remove dead NotificationServer code from PDS

* clean up code to prepareNotifsToSend

* put sending notifications as part of the backgroundQueue

* log instead of throwing error if notification attributes don't exist

* remove logs

* add more tests to `notification-server.test.ts`

* show replied with text for reply notifs

* better error handling when sending notifications via backgroundQueue

* add rate limit and batching to sending notifications

* add comments to NotificationServer

* merge with main

* use redis for rate limiting instead of normal rate limits

* move `notificationBatchSize` into config

* usePrimary db in test

* hoist push notif migration to present, update model to remove endpoint

* update push notif lexicon

* pare down lex for unregistering push

* helpers for working with service endpoints from did doc

* service-authed register/unregister push notifs

* add well-known endpoint to appview

* update bsky notif service tests

* fix to batching logic, misc tidy

* batch display notifications

* colocate all notification sending logic

* tidy tests

* remove unregister token for now

* fix registerPush lexicon, make a procedure

* fix registerPush impl, test pds proxy to notif service

* fix tests, make notif server optional when not configured

* fix notif server config for bsky app service

* move notif server rate limiting in-mem for now, add sending retry

* codegen tidy

* only push notifs on commit

* build

* fix notif rate limiter check

* send notifs from users w/o a profile

* remove build

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-23 18:37:14 -04:00
Eric Bailey
7364f17b33 Merge remote-tracking branch 'origin' into eric/app-796-likes-tab
* origin:
  Tweak rate limits (#1502)
  Fix auth on bsky author feed (#1501)
  Bugfix: Application ratelimits relative import (#1500)
  Application ratelimits (#1382)
  format
  do not notify users of own-actions
  Fix runtime flags open handles (#1497)
  make sure to await rejected promise expectations
  Misc fixes (#1492)
2023-08-22 11:15:20 -05:00
Eric Bailey
4cdb43c7f3 view likes from muted accounts 2023-08-22 11:07:06 -05:00
Eric Bailey
2c7bc6535f
Merge pull request #1499 from bluesky-social/eric/app-1201-self-notifications
Don't notify user of self-likes and self-reposts
2023-08-22 10:54:40 -05:00
Daniel Holmgren
ad25430691
Tweak rate limits (#1502)
* tweak rate limits

* tweak #

* fix test
2023-08-21 18:13:01 -05:00
Daniel Holmgren
891bf3bb7a
Application ratelimits (#1382)
* initial setup

* lexgen

* tidying things up

* add in some rate limiting

* testing rate limits

* small bugfix

* fix build

* increase rate limit

* more limits

* config

* optional rate limiter

* tweak headers

* ratelimit headers

* multiple rate limits

* tests & bugfixes

* test bypass

* slight refactor

* fail open

* fail open

* right most xff ip

* setup redis for ratelimiting

* redis test

* more tests

* use new dev-infra package

* adjust limits

* temporarily remove repo write limits

* codegen

* redis scratch var name

* cfg var host -> address
2023-08-21 16:10:04 -05:00
Eric Bailey
896c5b0ad1 format 2023-08-21 16:08:29 -05:00
Eric Bailey
5316d3f65d do not notify users of own-actions 2023-08-21 16:08:09 -05:00
Eric Bailey
24a5414b64 make sure to await rejected promise expectations 2023-08-21 15:42:45 -05:00
Eric Bailey
be794d2026 opaque errors 2023-08-21 10:02:15 -05:00
Eric Bailey
9b1f5f0124 only allow viewing your own likes 2023-08-18 16:25:07 -05:00
Eric Bailey
048cd16d0c Merge remote-tracking branch 'origin' into eric/app-796-likes-tab
* origin: (21 commits)
  styleguide
  Increase CI test matrix size (#1490)
  tweak author-feed rejects test syntax
  update test
  format
  add repost of reply, update snaps
  simplify query
  fix typos
  include reposted replies in posts_no_replies filter
  fix bad test syntax
  fix actor takedown tests
  use getActor, failing atm
  only include media posts by post creator
  Fix condition for viewing soft-deleted followers (#1485)
   Expose takendown profile, their follows and followers to mods (#1456)
  Handle revalidation (#1474)
  Handle db pool errors on appview (#1483)
  Handle db client errors on appview (#1481)
  v0.6.4
   Allow creating moderation action with a duration (#1431)
  ...
2023-08-18 11:10:04 -05:00
Eric Bailey
9082a9ac78 Merge remote-tracking branch 'origin' into eric/fix-1197-reposted-replies
* origin:
  Increase CI test matrix size (#1490)
  Fix condition for viewing soft-deleted followers (#1485)
   Expose takendown profile, their follows and followers to mods (#1456)
  Handle revalidation (#1474)
  Handle db pool errors on appview (#1483)
  Handle db client errors on appview (#1481)
2023-08-17 18:25:32 -05:00
Daniel Holmgren
2413f48c93
Increase CI test matrix size (#1490)
* increase test matrix size

* tweak author feed tests
2023-08-17 18:24:42 -05:00
Eric Bailey
1c50428428 tweak author-feed rejects test syntax 2023-08-17 18:21:13 -05:00
Eric Bailey
0ea353b21f add repost of reply, update snaps 2023-08-17 15:30:30 -05:00
Eric Bailey
d4b7d29c6b
Merge pull request #1486 from bluesky-social/eric/fix-1167-media-reposts
only include media posts by post creator
2023-08-17 14:18:15 -05:00
Eric Bailey
30cb6412ce include reposted replies in posts_no_replies filter
(cherry picked from commit 588cf2562cc61441c170fec4f2354eedee39c4f6)
2023-08-17 14:09:24 -05:00
Eric Bailey
40fde9585c fix bad test syntax 2023-08-17 14:02:56 -05:00
Eric Bailey
4347a21d92 fix actor takedown tests 2023-08-17 13:50:26 -05:00
Eric Bailey
1a5b422718 only include media posts by post creator 2023-08-17 12:10:36 -05:00
Daniel Holmgren
4241ee16ec
Handle revalidation (#1474)
* easier hanlde revalidation

* remove duplicate line

* backup handle nameservers on appview

* fix tests & add a couple
2023-08-16 18:34:19 -05:00
Foysal Ahamed
7bc701e2e1
Allow creating moderation action with a duration (#1431)
*  Add action duration to action model

*  Add periodic moderation action reversal

*  Use actionDurationInHours and remove takedownExpiresAt

*  Destroy mod action reversal job

*  Add test for automatic moderation action reversal

* 🧹 Port over actionDuration to bsky package

* ♻️ Better naming for action duration and minor fixes

* ♻️ Optionally build expiresAt based on createdAt when logging action

* ♻️ Adjust constructor params

*  Copy over migration from pds to bsky package

* 🚨 Fix indentation

*  Propagate action reversal from app-view to pds

*  Revert locally only if pushAgent is not found

* build

* merge dbcoordinator work into periodic mod actions

* fix entrypoint for periodic mod reversals

* do not run periodic reversals on sequencer leader

* fix duration in action detail on bsky

* update env for pushing mod reversals

* allow zero-duration actions for testing

* remove build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-15 15:44:53 -04:00
Eric Bailey
e0e7b39b16 add block and mute tests 2023-08-15 13:46:58 -05:00
Eric Bailey
d9849b6776 get it working 2023-08-14 18:31:15 -05:00
Daniel Holmgren
73f64009a2
Appview database replicas (#1459)
* setup bsky so that writes always go to a db primary

* test bsky primary db functionality

* sprinkle db primary over moderation and view maintainer usage

* make bsky tests more strict on primary/secondary db, fixes

* maintain prev db config on appview, require primary db, make primary/replica explicit in config

* tidy

* sketch out db replicas

* wip

* support multiple tags per bsky replica, "any" tag, fallbacks and warns

* clarify tags

* use new coordinator api across routes & services

* wire-up replica tagging config

* tidy

* cleanup entroypoints and tests for db coordinator

* fix pds test

* re-hookup migrate db

* standardize env vars for db on bsky services

* tidy

* build

* fix bav api entrypoint

* support DB_POSTGRES_URL on ingester/indexer for backwards compat

* remove build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-14 18:38:00 -04:00