426 Commits

Author SHA1 Message Date
Paul Frazee
00a567daa0
Additional 3p labeler updates (#2286)
* Rename bsky_labeler_did

* Use labeldef default setting

* Add definedBy to interpretted label defs

* Improve dev-env mocks for labels

* Remove global label defs for dmca-violation, doxxing, and !no-promote

* Change nudity global label def to default to ignore and no longer be adult-only

* Remove old !no-promote tests

* Add mod authorities header and move bsky labeler into it

* Rename modsPref and addModService/removeModService to labelersPref and add/removeLabeler

* Add defaultSetting and adultOnly to custom label value definitions

* Rename InterprettedLabelValueDefinition to InterpretedLabelValueDefinition

* Update dev-env mock

* Move muted words and hidden posts into moderationPrefs

* Add muted word and hidden post handling to moderatePost

* Add mutewords tests

* Finish muteword tests

* Add mod-authority.test to dev-env

* Rename global label value def gore to graphic-media

* Fix typo

* Stop converting old label values

* Update label target=profile behaviors: dont filter content on hide, dont blur display names, but do show the info cards

* Update label target=account behaviors: dont blur media of content

* Add muteword moderation behaviors

* Fix mock label defs

* Implement quote-post moderation handling

* Add adult content test

* Handle sync legacy labels (#2291)

* Handle sync legacy labels

* Remap values on read

* Filter out double-written legacy label values

* Better naming, fix types

* Fix test

* Update moderation docs in sdk

* Update to new atproto-accept-labelers header behavior

* Add getLabelDefinitions() helper method

* Add proxy header support to agent

* Update mock moderation

* lint

* Implement moderation for userlists and feedgens

* Add another test label

* fix labeler in dev-env agents

* fix label hydration test

* fix lint error

* fix agent test

* fix takedown labels test

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>
2024-03-12 21:10:19 -07:00
Daniel Holmgren
2ea27e340a
Ozone lexicons (#2288)
* lexicons

* clean up old admin schemas

* codegen

* renaming all routes

* fix more

* fix snaps

* fix test

* format

* hook up ozone routes on pds

* publish next version

* build branch

* dont build branch
2024-03-12 18:42:11 -05:00
Daniel Holmgren
7df7d60735
Clean up PLC op submission (#2296)
* clean up plc op submission

* test
2024-03-12 16:19:51 -05:00
devin ivy
8dd67f5c81
Appview: labels on feed gens and list views (#2298)
* lexicon: add labels to feedgen and list views

* appview: hydrate labels onto lists, feedgens

* test hydration of labels on lists and feedgens

* update pds snap
2024-03-12 16:04:02 -05:00
Daniel Holmgren
71f9cc9d4f
Clean up role-based auth (#2255)
* tidy bsky auth

* hook up new auth verifier

* update auth throughout ozone

* handle mod signing keys

* add client proxy heads to pds

* hook up rest of routes

* simplify pipethrough & add some SSRF protection

* tests

* fix bad var

* remove basic auth in ozone

* wip

* fix key parsing in pds

* fix up all ozone tests

* fix admin auth test

* rename test

* fix ozone test

* clean up tokens in pds

* fix up pds tests

* fix up ozone tests

* add pipethrough to write routes

* reenable proxied admin test

* add moderator accounts to ozone in dev-env

* update did doc id values

* null creds string -> `none`

* fix fetchLabels auth check

*  Add a couple more proxied requests that we use in ozone ui

* Add runit to the services/bsky Dockerfile (#2254)

add runit to the services/bsky Dockerfile

* Improve tag detection (#2260)

* Allow tags to lead with and contain only numbers

* Break tags on other whitespace characters

* Export regexes from rich text detection

* Add test

* Add test

* Disallow number-only tags

* Avoid combining enclosing screen chars

* Allow full-width number sign

* Clarify tests

* Fix punctuation edge case

* Reorder

* Simplify, add another test

* Another test, comment

* Version packages (#2261)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* 🐛 Increment attempt count after each attempt to push ozone event (#2239)

* Ozone delegates email sending to actor's pds (#2272)

* ozone delegates email sending to user's pds

* lexicon: add content field to mod email event

* test email sending via mod event

* add dev dep for nodemailer in ozone

* fix auth verifier method

* build branch

* build branch

* fix url check

* better error handling for get account infos

* fix labeler service id

* fix iss on auth headers

* fix dev-env ozone did

* fix tests & another jwt issuer

* fix proxy auth

* ozone: fix ip check

* fix aud check on pds mod service auth

* tidy

* Update packages/pds/tests/proxied/admin.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix pipethrough of headers

* fix moderation status tests

* fix auth on ozone routes

* update iss on daemon

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-03-06 19:53:52 -06:00
Daniel Holmgren
2267f1ec48
Add client-set proxy headers to PDS (#2251)
* tidy bsky auth

* hook up new auth verifier

* update auth throughout ozone

* handle mod signing keys

* add client proxy heads to pds

* hook up rest of routes

* simplify pipethrough & add some SSRF protection

* tests

* fix bad var

* fix key parsing in pds

* fix admin auth test

* rename test

* add pipethrough to write routes

* update did doc id values

* null creds string -> `none`

* fix fetchLabels auth check

*  Add a couple more proxied requests that we use in ozone ui

* Add runit to the services/bsky Dockerfile (#2254)

add runit to the services/bsky Dockerfile

* Improve tag detection (#2260)

* Allow tags to lead with and contain only numbers

* Break tags on other whitespace characters

* Export regexes from rich text detection

* Add test

* Add test

* Disallow number-only tags

* Avoid combining enclosing screen chars

* Allow full-width number sign

* Clarify tests

* Fix punctuation edge case

* Reorder

* Simplify, add another test

* Another test, comment

* Version packages (#2261)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* 🐛 Increment attempt count after each attempt to push ozone event (#2239)

* Ozone delegates email sending to actor's pds (#2272)

* ozone delegates email sending to user's pds

* lexicon: add content field to mod email event

* test email sending via mod event

* fix auth verifier method

* build branch

* fix url check

* better error handling for get account infos

* fix labeler service id

* fix iss on auth headers

* fix dev-env ozone did

* fix tests & another jwt issuer

* ozone: fix ip check

* fix aud check on pds mod service auth

* tidy

* fix pipethrough of headers

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-03-06 19:38:21 -06:00
Daniel Holmgren
3988543258
Feature - 3rd party labelers (#2032)
* adding some basic views

* feed hydration, add blocks to post hydration

* pass over notification hydration

* tidy

* merge

* implement getProfile

* hydrate post aggregation and viewer state

* fix

* fix codegen

* get some tests passing!

* add takedowns & some like bugfixing

* all profile tests passing!

* likes test

* follow endpoints using data plane

* reorg follow block rules

* reposts

* post views!

* implement getList w/ dataplane caveat

* adjust dataplane getListMembers to return listitem uris

* implement getListMutes and -Blocks w/ dataplane

* suggestions

* timeline

* misc view fixes

* view fixes for mutes, self-mute/block

* author feed

* feed gen routes

* tidy

* misc block/mute fixes

* list feed & actor likes

* implement getLists, fix some empty cursors

* implement getMutes, empty profile description fix

* implement getBlocks, block application fix

* implement getSuggestedFollowsByActor, needs some fixes

* feed generation

* search routes

* threads

* tidy

* fix some snaps

* fix getSuggestedFollowsByActor

* implement listNotifications

* implement getUnreadCount w/ dataplane

* implement notifications.updateSeen w/ dataplane

* 3rd party blocking tests

* blocked profile viewer

* add search mocks

* refactor getFeed

* createPipeline -> createPipelineNew

* basic replygating functionality on dataplane w/o filtering violating replies

* hack threadgates into dataplane, apply gates

* deterministic thread orders in dataplane

* misc cleanup around dataplane

* upgrade typescript to v5.3

* update typescript linter deps

* sync bsky proto, codegen

* update dataplane, sync with bsky proto updates

* remove indexer, ingester, daemon, moderation services from appview

* convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane

* remove bsky services. db and indexing logic into mock dataplane.

* remove tests not needed by appview v2, misc reorg

* add basic in-mem repo subscription to dataplane mock

* fix dev-env, bsky tests, bsky build

* cull bsky service entrypoint

* add bsky service readme

* build

* tidy

* tidy, fix pds proxy tests

* fix

* fix bsky entrypoint deps

* support http2 grpc client

* build

* fix dataplane bad tls config/default

* support multiple dataplane urls, retry when unavailable

* build

* tidy/fix

* move dataplane mock tests into their own dir

* cover label hydration through timeline test

* bring back labels in appview tests

* remove unused db primary/replica/coordinator from bsky dataplane

* bsky proto add cids to contracts, buf codegen

* sync-up bsky data-plane w/ codegen updates

* start using dataplane interaction endpoints

* add file

* avoid overfetching from dataplane, plumb feed items and cids

* pass refs through for post viewer state

* Lexicons: Add labeler prefs, labeler declaration, and get labeler routes

* Add labelerViewBasic and update embed views

* Fix typo

* switch list feeds to use feed item in dataplane

* handle not found err on get-thread dataplane call

* support use of search service rather than dataplane methods

* mark some appview v2 todos

* tidy

* still use dataplane on search endpoints when search service is not configured

* fix pds test

* Switch to labelerViewDetailed

* Move label and report values to refs

* Add getActorLabelers

* lint

* fix up bsky tests & snaps

* small diff to open pr

* rm new line

* codegen schemas

* tidy migrations

* table + indexing

* protos

* rename lexicons

* views, hydration + rename lexicons

* rest of routes

* data plane routes

* parse labelers from req

* fix appview-v2 docker build

* Support label issuer tied to appview v2 (#2033)

support label issuer tied to appview

* hydrate context

* tidy header logic

* integrating into more routes

* more routes

* wrap up rest

* add mock labeler

* rework labelerlexicons

* tidy lexs

* codegen new lexicons

* integrate lexicon rework

* add proxy logic

* forward labeler headers through pds

* tweak label header parsing

* remove did from scheams

* update indexing for lexs

* tests for mod service views

* label hydration test

* Add 'associated' to profileViewDetailed

* Rename labelers to mods in preferences

* Change uri to did in mod preferences

* couple more

* syntax tweaks

* integrate updated lexicons

* update view snap

* handle mod service embeds

* tidy

* fix mock

* lint

* base default labels of config var

* fix label hydration

* Appview v2: handle empty cursor on list notifications (#2017)

handle empty cursor on appview listnotifs

* Update appview v2 to use author feed enum (#2047)

* update bsky protos with author feed enum, misc feed item changes

* support new author feed enums in dataplane

* fix build

* Appview v2: utilize sorted-at field in bsky protos (#2050)

utilize new sorted-at field in bsky protos

* remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts

* Appview v2, sync w/ changes to protos (#2071)

* sync bsky protos

* sync-up bsky implementation w/ proto changes

* Appview v2 initial implementation for getPopularFeedGenerators (#2072)

add an initial implementation for getPopularFeedGenerators on appview v2

* merge

* fixes

* fix feed tests

* fix bsync mock

* format

* remove unused config

* fix lockfile

* another lockfile fix

* fix duplicate type

* fix dupplicate test

* Appview v2 handling clearly bad cursors (#2092)

* make mock dataplane cursors different from v1 cursors

* fail open on clearly bad appview cursors

* fix pds appview proxy snaps

* Appview v2 no notifs seen behavior (#2096)

* alter behavior for presenting notifications w/ no last-seen time

* fix pds proxy tests

* Appview v2 dataplane retries based on client host (#2098)

choose dataplane client for retries based on host when possible/relevant

* don't apply negated labels

* display suspensions on actor profile in appview v2

* Appview v2 use dataplane for identity lookups (#2095)

* update bsky proto w/ identity methods

* setup identity endpoints on mock dataplane

* move from idresolver to dataplane for identity lookups on appview

* tidy

* Appview v2: apply safe takedown refs to records, actors (#2107)

apply safe takedown refs to records, actors

* Fix timing on appview v2 repo rev header (#2113)

fix timing on appview repo rev

* fix post thread responses

* Appview v2 don't apply 3p self blocks (#2112)

do not apply 3p self-blocks

* Appview v2 search for feed generators (#2118)

* add protos for feedgen search

* support feed search on getPopularFeedGenerators

* Appview v2 config tidy (#2117)

* remove mod and triage roles from appview

* rename cdn and search config

* remove custom feed harness from appview v2

* Appview v2: don't apply missing modlists (#2122)

* dont apply missing mod lists

* update mock dataplane

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* refactor & document a bit better

* fix up other routes

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* Appview v2 enforce post thread root boundary (#2120)

* enforce post thread root boundary

* test thread root boundary

* Appview v2 fix admin environment variable (#2137)

fix admin env in appview v2

* Remove re-pagination from getSuggestions (#2145)

* remove re-pagination from getSuggestions

* fix test

* Adjust wording for account suspension (#2153)

adjust wording for account suspension

* Appview v2: fix not-found and blocked uris in threads (#2201)

* fix uris of not-found and blocked posts in threads

* update snaps

*  Show author feed of takendown author to admins only (#2197)

* fold in cid, auth, tracing, node version changes

* remove dead config from bsky service entrypoint

* build

* remove ozone test codepaths for appview v2

* tidy, docs fix

* fix test

* add additional user counts

* add associated data to profiles

* update snaps

* update to is_mod_service

* format

* tidy

* 3p labeler sdk updates (#2176)

* Update sdk to support 3p labeler preferences

* Stick with intolerance instead of hate for the label group id

* wip expand labels and label groups

* Output moderationOpts (computed) and modsPref (unaltered)

* Add tests for enabling/disabling mod services

* Add atproto-labelers header config

* Expand labels and label groups in definitions

* Fix tests

* Tweaks to labels

* Remove label descriptions and improve output types on labels and label groups

* Add mocker to exported API

* Improve types of label and label group definitions

* Rework moderation prefs to continue using global labelgroup settings and only disable label groups per moderator

* Simplify encoding of the label preferences in definitions

* Add target constraints to labels

* Refactor the moderation sdk to derive more behaviors from the definition files

* Small cleanup

* Add hiding tool to modsdk

* Track filter causes

* Make mute state an alert

* Fix: dont blur profileview for blocks

* Prioritize causes by severity

* Add moderateNotification() and drop quote post moderation code

* Add mocker functions for notifications

* Improve mock data

* Lexicon: Add custom label definitions and remove modservice descriptions

* Lexicon: Update moderation prefs

* SDK updates: remove label groups, reduce builtin labels, update mod-preference apis

* Lexicon: Update global labels with new reduced set

* Lexicon: Remove moderation.getService and add detailed option to getServices

* Lexicons: add severity=none to custom label value defs

* Implement custom label-value definition tooling

* All custom labels are no-self

* Backend impl for labeler lexicon updates (#2258)

* codegen

* clean up impl

* fix up tests

* Lexicon: modservice -> labeler

* Remove x- prefix behavior; add label value syntax rules; add custom label precedence rules

* Lexicon: Remove the ability to choose a defaultSetting from custom labels

* Rework test suites

* Give behaviors to all labels regardless of target

* sync up backend with lex changes

* fix labelers in dev-env agent

* lint protos

* update protos & views

* small bugfix & update tests

* tweak protos

* fix build issue from merge

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-03-06 17:56:34 -06:00
Daniel Holmgren
582109cb03
Ozone ACLs (#2252)
* tidy bsky auth

* hook up new auth verifier

* update auth throughout ozone

* handle mod signing keys

* fix bad var

* fix key parsing in pds

* fix admin auth test

* rename test

* update did doc id values

* null creds string -> `none`

* fix fetchLabels auth check

*  Add a couple more proxied requests that we use in ozone ui

* Add runit to the services/bsky Dockerfile (#2254)

add runit to the services/bsky Dockerfile

* Improve tag detection (#2260)

* Allow tags to lead with and contain only numbers

* Break tags on other whitespace characters

* Export regexes from rich text detection

* Add test

* Add test

* Disallow number-only tags

* Avoid combining enclosing screen chars

* Allow full-width number sign

* Clarify tests

* Fix punctuation edge case

* Reorder

* Simplify, add another test

* Another test, comment

* Version packages (#2261)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* 🐛 Increment attempt count after each attempt to push ozone event (#2239)

* Ozone delegates email sending to actor's pds (#2272)

* ozone delegates email sending to user's pds

* lexicon: add content field to mod email event

* test email sending via mod event

* fix auth verifier method

* better error handling for get account infos

* fix labeler service id

* fix iss on auth headers

* fix dev-env ozone did

* fix tests & another jwt issuer

* ozone: fix ip check

* fix aud check on pds mod service auth

* tidy

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-03-05 19:04:57 -06:00
Daniel Holmgren
f65de89eed
Feature: Appview v2 (#1924)
* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* filler commit

* rm filler

* server boilerplate

* follows impl

* posts impl

* posts & likes impl

* repost impl

* profiles & handle null values

* list impl

* mutes impl

* blocks impl

* misc

* feed gen impl

* label impl

* notifs impl

* feeds impl

* threads impl

* early sketchwork

* wip

* stub out thick client

* in-progress work on hydrator

* tweak

* hydrate profile labels, detail lists

* feedgen hydration

* protobuf tweaks

* more protobuf tweaks

* wip

* snake case

* moar snake case

* tidy actor hydration

* tidy parsing

* type fixes, renaming, comments in hydrator

* hydrate list items and likes

* hydrate notifications

* feed hydration

* graph & label hydration

* more record protobufs

* pluralize

* tweak pbs

* use new methods

* Setup dataplane grpc client/mock server (#1921)

* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* add record getter mocks

* post view hydration

* fix up mock dataplane to match new protos

* missed one

* wire up dataplane in ctx & dev-env

* adding some basic views

* feed hydration, add blocks to post hydration

* pass over notification hydration

* tidy

* merge

* implement getProfile

* hydrate post aggregation and viewer state

* fix

* fix codegen

* get some tests passing!

* add takedowns & some like bugfixing

* all profile tests passing!

* likes test

* follow endpoints using data plane

* reorg follow block rules

* reposts

* post views!

* implement getList w/ dataplane caveat

* adjust dataplane getListMembers to return listitem uris

* implement getListMutes and -Blocks w/ dataplane

* suggestions

* timeline

* misc view fixes

* view fixes for mutes, self-mute/block

* author feed

* feed gen routes

* tidy

* misc block/mute fixes

* list feed & actor likes

* implement getLists, fix some empty cursors

* implement getMutes, empty profile description fix

* implement getBlocks, block application fix

* implement getSuggestedFollowsByActor, needs some fixes

* feed generation

* search routes

* threads

* tidy

* fix some snaps

* fix getSuggestedFollowsByActor

* implement listNotifications

* implement getUnreadCount w/ dataplane

* implement notifications.updateSeen w/ dataplane

* 3rd party blocking tests

* blocked profile viewer

* add search mocks

* refactor getFeed

* createPipeline -> createPipelineNew

* basic replygating functionality on dataplane w/o filtering violating replies

* hack threadgates into dataplane, apply gates

* deterministic thread orders in dataplane

* misc cleanup around dataplane

* upgrade typescript to v5.3

* update typescript linter deps

* sync bsky proto, codegen

* update dataplane, sync with bsky proto updates

* remove indexer, ingester, daemon, moderation services from appview

* convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane

* remove bsky services. db and indexing logic into mock dataplane.

* remove tests not needed by appview v2, misc reorg

* add basic in-mem repo subscription to dataplane mock

* fix dev-env, bsky tests, bsky build

* cull bsky service entrypoint

* add bsky service readme

* build

* tidy

* tidy, fix pds proxy tests

* fix

* fix bsky entrypoint deps

* support http2 grpc client

* build

* fix dataplane bad tls config/default

* support multiple dataplane urls, retry when unavailable

* build

* tidy/fix

* move dataplane mock tests into their own dir

* cover label hydration through timeline test

* bring back labels in appview tests

* remove unused db primary/replica/coordinator from bsky dataplane

* bsky proto add cids to contracts, buf codegen

* sync-up bsky data-plane w/ codegen updates

* start using dataplane interaction endpoints

* add file

* avoid overfetching from dataplane, plumb feed items and cids

* pass refs through for post viewer state

* switch list feeds to use feed item in dataplane

* handle not found err on get-thread dataplane call

* support use of search service rather than dataplane methods

* mark some appview v2 todos

* tidy

* still use dataplane on search endpoints when search service is not configured

* fix pds test

* fix up bsky tests & snaps

* tidy migrations

* fix appview-v2 docker build

* Support label issuer tied to appview v2 (#2033)

support label issuer tied to appview

* Appview v2: handle empty cursor on list notifications (#2017)

handle empty cursor on appview listnotifs

* Update appview v2 to use author feed enum (#2047)

* update bsky protos with author feed enum, misc feed item changes

* support new author feed enums in dataplane

* fix build

* Appview v2: utilize sorted-at field in bsky protos (#2050)

utilize new sorted-at field in bsky protos

* remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts

* Appview v2, sync w/ changes to protos (#2071)

* sync bsky protos

* sync-up bsky implementation w/ proto changes

* Appview v2 initial implementation for getPopularFeedGenerators (#2072)

add an initial implementation for getPopularFeedGenerators on appview v2

* merge

* fixes

* fix feed tests

* fix bsync mock

* format

* remove unused config

* fix lockfile

* another lockfile fix

* fix duplicate type

* fix dupplicate test

* Appview v2 handling clearly bad cursors (#2092)

* make mock dataplane cursors different from v1 cursors

* fail open on clearly bad appview cursors

* fix pds appview proxy snaps

* Appview v2 no notifs seen behavior (#2096)

* alter behavior for presenting notifications w/ no last-seen time

* fix pds proxy tests

* Appview v2 dataplane retries based on client host (#2098)

choose dataplane client for retries based on host when possible/relevant

* don't apply negated labels

* display suspensions on actor profile in appview v2

* Appview v2 use dataplane for identity lookups (#2095)

* update bsky proto w/ identity methods

* setup identity endpoints on mock dataplane

* move from idresolver to dataplane for identity lookups on appview

* tidy

* Appview v2: apply safe takedown refs to records, actors (#2107)

apply safe takedown refs to records, actors

* Fix timing on appview v2 repo rev header (#2113)

fix timing on appview repo rev

* fix post thread responses

* Appview v2 don't apply 3p self blocks (#2112)

do not apply 3p self-blocks

* Appview v2 search for feed generators (#2118)

* add protos for feedgen search

* support feed search on getPopularFeedGenerators

* Appview v2 config tidy (#2117)

* remove mod and triage roles from appview

* rename cdn and search config

* remove custom feed harness from appview v2

* Appview v2: don't apply missing modlists (#2122)

* dont apply missing mod lists

* update mock dataplane

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* refactor & document a bit better

* fix up other routes

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* Appview v2 enforce post thread root boundary (#2120)

* enforce post thread root boundary

* test thread root boundary

* Appview v2 fix admin environment variable (#2137)

fix admin env in appview v2

* Remove re-pagination from getSuggestions (#2145)

* remove re-pagination from getSuggestions

* fix test

* Adjust wording for account suspension (#2153)

adjust wording for account suspension

* Appview v2: fix not-found and blocked uris in threads (#2201)

* fix uris of not-found and blocked posts in threads

* update snaps

*  Show author feed of takendown author to admins only (#2197)

* fold in cid, auth, tracing, node version changes

* remove dead config from bsky service entrypoint

* build

* remove ozone test codepaths for appview v2

* tidy, docs fix

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-02-27 15:22:55 -05:00
Daniel Holmgren
87d59ff541
Admin: updateAccountPassword (#2212)
* add update account password route

* add test
2024-02-21 23:58:25 -06:00
devin ivy
9f90203f20
Optional service config in pds distribution (#2211)
* make appview and mod services optional on pds

* pds: allow configuring a reporting service optionally separate from an administrative mod service

* tidy
2024-02-21 20:14:45 -05:00
Daniel Holmgren
4e0271bddf
Firehose identity cleanup (#2210)
* missed a couple

* fix tests
2024-02-21 15:59:44 -06:00
Daniel Holmgren
6dfc899d99
Enable creation of unknown record types (#2171)
* start to allow 3p lexicons

* tests

* tidy

* tests + ensure no legacy blob ref

* increase the depth were willing to go when searching for blobs
2024-02-21 15:11:15 -06:00
Daniel Holmgren
1a12c7e34b
Firehose Identity event (#2208)
* add new identity event

* add note for tombstone

* send identity evts

* add emission of identity evts

* emit handle event on activate account & fix subscribeRepos

* add time to evt

* update indexer for tests

* rm logs
2024-02-21 15:10:12 -06:00
Daniel Holmgren
30b05a7d4b
Account migration (#2179)
* draft of account migration lexicons

* format

* clean up schemas

* codegen

* second pass on schemas

* small fix

* move around checkImportStatus

* re-codegen

* getServiceAuth

* getServiceAuth impl

* importRepo impl

* handle uploadBlob for import

* allow bringing your own did on createAccount

* working on test flow

* fleshing out flow

* fix up sendPlcOP

* small fixes

* activate/deactivate account

* full flow working!

* schema tweaks

* format

* update schemas

* moar codegen

* match impl to new schemas

* email flow for signed plc operation

* add email flow for plc operations

* impl plc op request email

* proxy to entryway

* tidy activate account

* integrating account deactivated state

* fix up tests

* friendly parse on optional did auth

* admin activate/deactivate routes

* proxy relevant requests to entryway

* remove admin activation routes

* do not proxy acitvaition to entryway

* cfg for disallowing imports

* buff up test

* refactor listMissingBlobs a bit

* add validDid & activated to accoutn status

* emit event on account activation

* test creating a post after migrating

* account deactivation tests

* test name

* tests on plc operations

* fix recommended did creds

* codegen

* turn off accepting imports on createAccount

* undo prev change

* increment version

* build branch

* pr feedback

* handle errs in p-queue

* handle blob upload outside of txn

* Clean old temp account migration lexicons (#2187)

* clean old temp lexicons

* rm old test

* fix agent session test

* fix bsky test

* dont build branch
2024-02-20 19:29:49 -06:00
Foysal Ahamed
d81573232b
Allow filtering moderation queue by language (#2161)
*  Store languages on subjects from record data

*  Fetch lang from author feed for repo subject

*  Regenerate lex

*  Add lang to profile subject

*  Move lang to flag

*  Add comment to flag event

*  Update pds test snapshot

* 🐛 Fix broken import

*  Rename flag to tag

*  Update snapshot for event by id

*  Bring back skipped test

*  Move tags to dedicated columns

*  Update test snapshots, use logger

*  Change lang:unknown to lang:und

*  Update test snapshots

* 🧹 Cleanup

*  Add filter params for tags on queryModerationEvents endpoint

*  Add tags property to subject status model

*  Update test code

*  Add filter to get subjects that do not have certain tags
2024-02-19 10:35:03 -05:00
devin ivy
d0be052e12
Support node v20 on pds distribution (#2157)
* upgrade node to v20, better-sqlite3 to v9

* fix tests, deps for node v20

* build

* add io_uring setting to pds dockerfile
2024-02-09 19:05:29 -05:00
Daniel Holmgren
f023494802
Remove appview feedgen (#2106)
* remove appview feedgen

* deprecate getTimelineSkeleton

* fix dev-env build

* rm unused test
2024-01-30 16:19:36 -06:00
Daniel Holmgren
bf8d718cf9
Namespace lexicon codegen (#2039)
* namespace codegen

* fix test that uses namespace prototype

* changeset
2024-01-18 15:24:38 -06:00
Daniel Holmgren
de2dbc2903
Split out moderation backend (#1970)
* mv appview

* copy

* finalize copy

* package names

* big WIP

* first pass at mod servce

* some tidy

* tidy & fix compiler errors

* rename to ozone, db migrations, add to dev-env & pds cfg

* getRecord & getRepo mostly working

* fix open handle

* get record tests all working

* moderation events working

* statuses working

* tidy test suite

* search repos

* server & db tests

* moderation tests

* wip daemon + push events

* pds fanout working

* fix db test

* fanning takedowns out to appview

* rm try/catch

* bsky moderation test

* introduce mod subject wrappers

* more tidy

* refactor event reversal

* tidy some db stuff

* tidy

* rename service to mod-service

* fix test

* tidy config

* refactor auth in bsky

* wip patching up auto-mod

* add label ingester in appview

* fix a couple build issues

* fix some timing bugs

* tidy polling logic

* fix up tests

* fix some pds tests

* eslint ignore

* fix ozone tests

* move seeds to dev-env

* move images around

* fix db schemas

* use service auth admin reqs

* fix remaining tests

* auth tests bsky

* another test

* random tidy

* fix up search

* clean up bsky mod service

* more tidy

* default attempts to 0

* tidy old test

* random tidy

* tidy package.json

* tidy logger

* takedownId -> takedownRef

* misc pr feedback

* split daemon out from ozone application

* fix blob takedown mgiration

* refactor ozone config

* do push event fanout on write instead of on read

* make suspend error work again

* add attempts check & add supporting index

* fix takedown test ref

* get tests working

* rm old test

* fix timing bug in event pusher tests

* attempt another fix for timing bug

* await req

* service files

* remove labelerDid cfg

* update snaps for labeler did + some cfg changes

* fix more snaps

* pnpm i

* build ozone images

* build

* make label provider optional

* fix build issues

* fix build

* fix build

* build pds

* build on ghcr

* fix syntax in entry

* another fix

* use correct import

* export logger

* remove event reverser

* adjust push event fanout

* push out multiple

* remove builds
2024-01-05 17:06:54 -06:00
devin ivy
16b31e3ce2
Proper validation of entryway op in account creation (#1975)
* ensure entryway plc rotation key

* update plc lib on pds, test
2023-12-18 12:02:17 -05:00
Foysal Ahamed
7d818b8f19
Allow moderators along with admins to toggle invite codes (#1928)
*  Allow moderators along with admins to toggle invite codes

*  Adjust test to use triage token
2023-12-14 17:29:13 -05:00
devin ivy
842e183925
Add labels to actor search typeahead results (#1940)
add labels to appview actor search typeahead
2023-12-06 19:21:06 -05:00
Daniel Holmgren
f9fd3e68ca
Feature branch: PDS v2 (#1789)
* cleanup repeat process all

* wip

* skip actor search test

* skip actor search test

* tweak processAll

* decrease wait to 1 sec

* repo_blob -> record_blob

* simplify backlink linkTo

* return repo_root to one row

* sequence before updating repo_root

* invite code forUser -> forAccount

* ipld_block -> repo_block

* use lru-cache fetchMethod

* move did_cache to own db

* better error handling on did cache

* drop did_handle

* fix sequencer wait time

* debug

* debug

* more debug

* check something

* fix bday paradox

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* user_account -> account

* remove inviteNote

* keypair per repo

* use an lru cache for keypairs as well

* clean up repo

* wip

* wrap up accoutn manager

* tidy

* tidy

* fix tests

* fix disabled codes

* fix appview tests

* add note

* set pragmas

* tidy account manager getDb

* rename pref transactor

* user pref -> account pref

* handle blob imports

* tidy imports

* add reserveSigningKey

* wip transferAccount

* clean up transferAccount

* tests

* tidy

* tidy

* configure entryway url on pds

* handle entryway in pds admin endpoints

* make importRepo temp

* fix imports

* make email optional on pds when using entryway

* handle diffs

* handle pds entryway usage for server, identity, admin endpoints

* pds support for credentials from entryway

* setup pds tests w/ entryway service

* tidy

* tidy

* update entryway version

* wip

* test handle updates w/ entryway

* split account table into two

* tidy

* tweak scripts

* tidy tests

* tidy

* better config for actorstore & dbs

* clean up cfg more

* reorg actorstore fs layout

* handle erros on actor db create

* pr tidy & fix accoutn deletion test

* pr feedback

* fix bad merge

* unskip test

* fix subscribe repos tests

* tidy repo root tables

* tidy

* fix tests

* tidy delete tokens

* tidy account getters

* tidy

* bulk deletesg

* increase chunk size

* handle racing refreshes

* wip

* fix auth test

* invert import flow

* clean up actor store on create account failure

* tweak sequencer

* prevent invite code races on createAccount

* rm note

* add back in race protection on getAccountInviteCodes

* start feature branch

* deleted app migration table

* patch up new auth test

* rm note

* g

* create accoutn delegated from entryway

* tidy

* fix test

* change plcOp type to unknown

* small fixes

* sync up w entryway branch

* Use proper error when authed account is not found (#1799)

provide proper error when account not found in access-takedown check

* build branch

* build on ghcr

* tweak service file

* tweak service file

* change where we save reserved keys

* no tmp dir in blobstore either

* fix blobstore temp location again

* handle repeat record_blobs

* create account before submitting plc op & undo if fail

* small tweak

* limit the number of local records

* push out empty commit on transfer

* fix issue with record_blob

* add push blob endpoint

* Set and validate token audiences on pds v2 (#1793)

set and validate token audience on pds v2

* merge

* include entryway did on tests

* build branch

* fix cache issue

* xrpc server blob limit

* put correct bytes

* add auth to routes

* handle quarantining/unquarantining a blob that does not exist

* tidy

* fix transfer tests

* fix email request routes for entryway

* PDS v2 entryway account deletion (#1819)

* add admin lexicon for account deletion

* implement admin account deletion endpoint

* fix entryway proxying on account email checks

* proxy to entryway for acct deletion

* read-after-write sanity check

* tweak

* wip

* finish refactor

* fix test schema

* application retry logic for busy

* pr feedback

* rm lru-cache

* fix test pg schema

* fix transfer test

* Sqlite instrumentation for pds v2 (#1838)

* sqlite instrumentation

* build

* remove build

* dont reimport blobs

* send ticks during import

* close on error

* catch handle validation error

* add log

* fix test

* return emailConfirmedAt on getAccountInfo

* Upgrade sharp on pds v2 (#1863)

upgrade sharp to 0.32.6

* read all bytes before parsing car

* Async car reader (#1867)

* asynchronously read in car

* dont buffer car

* tweak

* Gracefully handle indexing of invalid records (#1853)

* gracefully handle indexing of invalid records

* fix repo tests

* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)

fix role auth for access-or-role verifier, fix getBlob actor takedown check

* better cleanup of actor-stores

* add ability to not ensure leaves

* tidy

* allow did:web transfer

* Migration utility for actor-store (#1873)

beginnings of helper for migrating all actors

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* base case for findBlobRefs

* App-level retries for sqlite on pds (#1871)

* revamp retry helper to be more flexible re: backoff strategies

* sqlite timeout helper

* ensure sqlite wal on db creation/migration rather than every open

* layer retries for sqlite on writes outside transactions on pds

* tidy

* fix up lockfile

* tidy

* fix lex codegen

* fix timing bug in threadgate test

* No-op update handling (#1916)

do no produce commits on no-op updates

* Retry on all SQLITE_BUSY error codes (#1917)

retry on all sqlite_busy error codes

* Pds v2 ensure sqlite ready (#1918)

ensure sqlite is ready before making queries

* try something

* tidy

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-12-04 18:00:09 -06:00
Daniel Holmgren
a4e5abf5f4
Fix snapshots for list items (#1911)
fix snapshots for list items
2023-12-01 15:07:17 -06:00
intrnl
8d9b1f70cd
Attach record URI to listItemView (#1758)
* feat: attach record uri to listitemview

* chore: attempt to update test snapshots
2023-12-01 14:28:54 -06:00
bnewbold
3be9c741ce
helpers for rkey and tid syntax; validate rkey at record creation time (#1738)
* syntax: fix jest config displayName

* syntax: TID validation

* syntax: add recordkey validation

* pds: verify rkey syntax at record creation time

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-12-01 14:28:47 -06:00
bnewbold
c17971a2d8
harden datetime verification (#1702)
* syntax: add datetime validator (and interop tests)

* syntax: improve datetime normalization

* lexicon: stronger datetime validation (from syntax package)

* syntax: make datetime syntax norm test more flexible

* make fmt

* datetime: docs, normalize and always variant

* bsky replace toSimplifiedISOSafe with normalizeDatetimeAlways

* more rigorous datetime parsing on record creation

* handle negative dates

* syntax: disallow datetimes before year 0010

* syntax: datetime normalization functions validate output

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-11-30 18:19:06 -06:00
Mary
6da8182530
fix(pds): include aspectRatio on read-sticky posts (#1824)
* fix: include aspectRatio on read-sticky

* chore: add missing alt text test case
2023-11-30 17:51:07 -06:00
Foysal Ahamed
1f9040a44d
Evented architecture for moderation system (#1617)
* 🚧 WIP with proposed lexicons for event based mod architecture

* 🚧 Remove unnecessary moderation action lexicon

* 🚧 Working on event based actions

*  Add escalated subject status

* 🐛 Alright, fixed the error in lexicon

* 🚧 Working through reversal

*  Cleanup build errors

*  Add subject status endpoint

*  Add handler

*  get reports from mod actions table

* :rightwards_twisted_arrows: Merge with upstream

* 🚧 Builds but test network doesnt start

*  Tests passing on event based status change

*  Rename index

* ♻️ Rename takeModerationAction->emitModerationEvent

*  Implement label reversal

*  Auto-revert test working

* ♻️  Refactored to event types and tests are passing

*  Add takedown event sequence validation

*  Adds support for blobCid status

* 🧹 Cleanup unnecessary method:

*  Hydrate handles with status and events

*  Re-implement auto reversal

*  Add takendown and mute filters

*  Allow filtering events by type

*  Allow filtering events by creator did

*  Add subjectStatus to record and repoview

*  Add persistent note feature

*  Log send email event

* 🐛 Fix logging send email event

*  Better type

*  Adjust migration to create separate moderation_event table

* 🧹 Cleanup types

*  Adjust tests with mod event emitter

*  Fix more tests around takedowns

*  Get test suite to pass

*  Get test suite to pass for pds

*  Get test suite to pass for pds

*  Update snapshot for feedgen

*  Why are more snapshots updating?

* ♻️ Rename getModerationEvents -> queryModerationEvents

* ♻️ Rename getModerationStatuses -> queryModerationStatuses

* ♻️ Rename persistNote->sticky

* 🐛 Rename subject

* ♻️ Cleanup expiresAt for scheduled actions

*  Add more tests, allow fetching mod history for all content by a user

*  Fix repo and record tests

*  Migrate reports and actions to events

* 🐛 Fix escalated status overwrite

*  Implement direct sql query to create events from actions and reports

* 🚧 Adding keyset pagination for subject statuses

*  Add migration for lastReportedAt

*  Migrate blob cids

*  Fix pagination on mod subject list endpoint

* 🐛 Fix blob actions

*  All tests passing on bsky package

*  Bring back snapshots

*  Skipping timeline test temporarily

*  Skipping some more tests to isolate failing ones

*  Bring back list-feed test

*  Bring back timeline test

*  Fix label action in seeding

*  Enable timeline proxied test

*  Enable search actor proxied test

*  Enable feedgen tests

*  Fix test for admin/get-record

*  Move note to comment for subject status

*  Accept comments in mute event

*  Remap flag event to ack event

* 🐛 Add legacyRef in report union selection

* @atproto/api 0.6.24-next.0

* @atproto/api 0.6.24-next.1

*  Adjust migration export and add index for blobCids column

*  Maintin action ids when migrating

*  Paginate events using createdAt timestamp

*  Update snapshot for pds test with events cursor update

*  Use only events for snapshot testing

*  Use only events for snapshot in the remaining test

* relative paths to lexicons for build

* fix bsky periodic event reversal in service entrypoint

*  Allow comments in takedown and label

*  Only import reports on consecutive run of the migration script

*  Adjust moderation property of blob entries

* determine latest reports to migrate

*  Process new reports for subject status

*  Process unresolved reports on first migration run

* fix transaction error, process just unresolved reports, make reported-at updates safe for reruns

* tidy

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-11-30 17:53:56 +01:00
devin ivy
f232371934
Provide proper error for updating to an email that's already in use (#1770)
provide proper error for updating to an email already in use
2023-11-21 18:09:07 -05:00
Daniel Holmgren
74b7fdf754
Randomize suggestions (#1844)
* randomize suggestions

* fix snap

* cursor fix

* pr feedback
2023-11-14 18:14:08 -06:00
Daniel Holmgren
fcb19c9c51
Simplify PDS moderation (#1723)
* spec out new simple pds mod routes

* introduce new admin state endpoints

* wire up routes

* clean up pds

* revoke refresh tokens

* getUserAccountInfo

* pr tidy

* fixing some tests

* fixing up more tests

* fanout takedowns to pds

* fanout admin reqs to pds

* tidy

* more tidy & add more pds moderation tests

* getUserAccountInfo -> getAccountInfo

* dont hydrate pds info on searchRepos

* fix build

* port admin tests to bsky package

* clean up old snaps

* tests on fanout

* tweak naming

* missed a rename

* tidy renames

* fix lex name

* tidy & move snap

* fix build

* cleanup repeat process all

* skip actor search test

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* change takedownId col to takedownRef

* build branch

* fix bsky test

* add service key to indexer

* move signing key to api entry

* dont build
2023-10-30 16:56:17 -05:00
Daniel Holmgren
2af3f6c783
Fix read after write on threads when posting a media embed (#1749)
fix read after write on threads when posting a media embed
2023-10-19 16:11:03 -05:00
Daniel Holmgren
44ea5e80fa
Split apart auth grant & verification (#1743)
* transition to auth verifier

* tidy

* quick test fix
2023-10-13 11:43:44 -05:00
Daniel Holmgren
d425061179
Fix test seed & snap (#1733)
fix test seed & snap
2023-10-11 10:26:00 -05:00
Daniel Holmgren
4bf61c5f95
Enable updateEmail with no token & no app password (#1715)
* dont allow updateEmail with an app password

* re-enable email updates
2023-10-09 13:46:32 -05:00
Daniel Holmgren
d664b51c64
Finalize PDS in-process AppView removal (#1198)
* rm tables

* rm event-stream & proxied

* Remove appview services, move label service to pds

* only proxy appview stuff

* delete more tables

* Start removing message dispatched from pds

* more syncing-up removal of message dispatcher in pds

* merged

* remove feedgens from pds, remove getPopular

* remove unused image helper from pds

* fixing compiler errors

* clean up sharp

* rm label service

* first pass on cleaning up tests

* fix up a bunch of tests

* moderation view tests

* last admin tests

* got a lil overzealous in deletes

* clean up unused cfg

* clean up label table

* simplify admin repo search query/logic

* tidy pds entrypoint

* in-progress pds config changes

* cfg fiddling

* finish cleaning up cfg/ctx

* comments

* building

* pds prefix on env

* test env

* collapse pds migrations down into a single migration

* fix up dev-env

* tidy

* cleanup

* fix pds admin tests

* fix handle test

* fix pds proxy tests

* fix subscribe repos test

* fix sqlite config in pds tests

* add sqlite clause in sequencer-leader

* fix actor search w/ sqlite on pds

* fixes

* fix dev env build

* update pds service entrypoint

* simple env example

* make takedown ids opaque identifiers in the pds

* use pds routes for api tests

* update pds dockerfile with volume and correct port env var

* add a couple env vars to example

* add comments to env example

* @atproto/pds 0.2.0-beta.0

* @atproto/aws 0.0.1-beta.0

* appview did

* @atproto/aws 0.0.1

* enable logs by default

* update env example

* bugfixing sandbox issues

* consistency in pds env var name for appview url

* log on pds start and stop, configure version at runtime

* @atproto/pds 0.2.0-beta.1

* fix semver matching for pds beta version

* v0.2.0-beta.2

* default invites to being not required

* fix flaky test

* limit db connections in tests

* publish 0.2.0-beta.d3

* fix invite required parsing

* @atproto/pds 0.2.0-beta.5

* Proxy getPopularFeedGenerators on simplified pds (#1222)

proxy getPopularFeedGenerators on pds

Co-authored-by: dholms <dtholmgren@gmail.com>

* tidy migrations

* fix service entry

* bump version

* change auth order

* bump version

* bump version

* add upgradeRepoVersion & fallback url for cdn

* bump version

* merging

* merge pds

* building dev-env

* merging tests

* merge service entry

* test fixing

* tidy

* fix admin search

* tidy

* tidy

* add snap for getListFeed

* add backup nameserver cfg

* tidy + pr feedback

* tidy

* tidy env

* bit more

* re-add dotenv to root package.json

* fix dep

* build branch

* fix tests

* Refactor tests to make better use of dev-env (#1690)

* refactor pds tests to use dev env

* refactor bsky tests

* fix pds test

* tidy bsky tests

* build pds correctly

* fix entry point

* default logging to false (for now)

* format service entry

* Switch takedown ids back to ints on pds distribution (#1694)

* switch takedown ids back to ints, consistent with live pds

* tidy/fix migration

* update migration for sqlite

* export moderation action reversal

* takedown tests

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-10-02 13:27:45 -05:00
Daniel Holmgren
11e3b329e3
Clean up other email tokens (#1572)
* lexicons

* codegen

* email templates

* request routes

* impl

* migration

* tidy

* tests

* tidy & bugfixes

* format

* fix api test

* fix auth test

* impl

* update constraint name

* temporarily disable unconfirmed updates

* tidy

* fix some tests

* validate email syntax
2023-09-28 13:25:56 -05:00
Eric Bailey
0fa34e4819
apply basic email validation to createAccount (#1658)
* apply basic email validation to createAccount

* format
2023-09-28 13:17:36 -05:00
Eric Bailey
9b2ba28fff
improve tag validation (#1684)
* check faceted tags for slurs

* format

* fix baby bug
2023-09-28 11:45:29 -05:00
Daniel Holmgren
41ee177f5a
Email confirmation/update (#1568)
* lexicons

* codegen

* email templates

* request routes

* impl

* migration

* tidy

* tests

* tidy & bugfixes

* format

* fix api test

* fix auth test

* codegen

* add unique constraint

* Add email confirmed to AtpSessionData

* interop test files (#1529)

* initial interop-test-files

* crypto: switch signature-fixtures.json to a symlink

* syntax: test against interop files

* prettier

* Update interop-test-files/README.md

Co-authored-by: Eric Bailey <git@esb.lol>

* disable prettier on test vectors

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>

* add getSuggestedFollowsByActor (#1553)

* add getSuggestedFollowsByActor lex

* remove pagination

* codegen

* add pds route

* add app view route

* first pass at likes-based suggested actors, plus tests

* format

* backfill with suggested_follow table

* combine actors queries

* fall back to popular follows, handle backfill differently

* revert seed change, update test

* lower likes threshold

* cleanup

* remove todo

* format

* optimize queries

* cover mute lists

* clean up into pipeline steps

* add changeset

* List feeds (#1557)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* lex

* list purpose

* lex gen

* add route

* add proxy route

* seed client helpers

* tests

* mutes and blocks

* proxy test

* snapshot

* hoist actors out of composeThread()

* tidy

* tidy

* run ci on all prs

* format

* format

* fix snap name

* fix snapsh

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Improve xrpc server error handling (#1597)

improve xrpc server error handling

* Remove appview proxy runtime flags (#1590)

* remove appview proxy runtime flags

* clean up proxy tests

* getPopular hotfix (#1599)

dont pass all params

* Interaction Gating (#1561)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* initial lexicons for interaction-gating

* add interactions view to post views

* codegen

* model bad reply/interaction check state on posts

* initial impl for checking bad reply or interaction on write

* omit invalid interactions from post thread

* support not-found list in interaction view

* hydrate can-reply state on threads

* present interaction views on posts

* misc fixes, update snaps

* tidy/reorg

* tidy

* split interaction gating into separate record in lexicon

* switch interaction-gating impl to use separate record type

* allow checking reply gate w/ root post deletion

* fix

* initial gating tests

* tighten gated reply views, tests

* reply-gating list rule tests

* allow custom post rkeys within window

* hoist actors out of composeThread()

* tidy

* update thread gate lexicons, codegen

* lex fix

* rename gate to threadgate in bsky, update views

* lex fix

* improve terminology around reply validation

* fix down migration

* remove thread gates on actor unindexing

* add back .prettierignore

* tidy

* run ci on all prs

* syntax

* run ci on all prs

* format

* fix snap

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* order by `like.indexedAt` in app view (#1592)

* order by like.indexedAt

* use keyset for ordering

* simplify

* ok ok ok I get it now

* Update packages/bsky/src/api/app/bsky/feed/getActorLikes.ts

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* Remove default value for post table invalid attrs (#1601)

remove default value for post table attrs

* Version packages (#1602)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* update Bluesky PBLLC to PBC (Public Benefit Corporation) (#1600)

* Temporarily disable filtering `invalidReplyRoot`s (#1609)

temporarily disable invalidReplyRoot check

* fix syntax docs (#1611)

* Version packages (#1612)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Allow bypass on ratelimit ip (#1613)

allow bypass on ratelimit ip

* Write rate limits (#1578)

* get rate limit ip correctly

* add write rate-limits

* Tweak createSession rate limit key (#1614)

tweak create session rl key

* Filter preferences for app passwords (#1626)

filter preferences for app passwords

* Tweak rate limit setup for multi rate limit routes (#1627)

tweak rate limit setup for multi rate limit routes

* Remove zod from xrpc-server error handling (#1631)

remove zod from xrpc-server error handling check

* Enforce properties field on lexicon object schemas (#1628)

* add empty properites to thread gate schema fragments

* tweak lexicon type

* Add feed-vew and thread-view preferences (#1638)

* Add feed and thread preference lexicons

* Add feed-view and thread-view preference APIs

* Add changeset for new preferences  (#1639)

Add changeset

* Version packages (#1640)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable getAccountInviteCodes for app passwords (#1642)

disable getAccountInviteCodes for app passwords

* remove cruft packages (uri, nsid, identifier) (#1606)

* remove @atproto/nsid (previously moved to syntax)

* remove @atproto/uri (previously moved to syntax)

* remove @atproto/identifier (previously moved to syntax)

* bump lockfile to remove old packages

---------

Co-authored-by: Eric Bailey <git@esb.lol>

* api: update login/resumeSession examples in README (#1634)

* api: update login/resumeSession examples in README

* Update packages/api/README.md

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* small syntax lints (#1646)

* lint: remove unused imports and variables

* lint: prefix unused args with '_'

* eslint: skip no-explicit-any; ignore unused _var (prefix)

* eslint: explicitly mark ignores for tricky cases

* indicate that getPopular is deprecated (#1647)

* indicate that getPopular is deprecated

* codegen for deprecating getPopular

* tidy up package.json and READMEs (#1649)

* identity: README example and tidy

* tidy up package metadata (package.json files)

* updated README headers/stubs for several packages

* crypto: longer README, with usage

* syntax: tweak README

* Apply suggestions from code review

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

* Improve the types of the thread and feed preferences APIs (#1653)

* Improve the types of the thread and feed preferences APIs

* Remove unused import

* Add changeset

* Version packages (#1654)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable pds appview routes (#1644)

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch

* auto-moderator tweaks: pass along record URI, create report for takedown action (#1643)

* auto-moderator: include record URI in abyss requests

* auto-moderator: log attempt at hard takedown; create report as well

The motivation is to flag the event to mod team, and to make it easier
to confirm that takedown took place.

* auto-mod: typo fix

* auto-mod: bugfixes

* bsky: always create auto-mod report locally, not pushAgent (if possible)

* bsky: fix auto-mod build

* bsky: URL-encode scanBlob call

* Clear follow viewer state when blocking (#1659)

* clear follow viewer state when blocking

* tidy

* add `tags` to posts (#1637)

* add tags to post lex

* kiss

* add richtext facet and validation attrs

* add tag validation attrs to post

* codegen

* add maxLength for tags, add description

* validate post tags on write

* add test

* handle tags in indexer

* add tags to postView, codegen

* return tags on post thread view

* format

* revert formatting change to docs

* use establish validation pattern

* add changeset

(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)

* remove tags from postView, codegen

* remove tags from thread view

* revert unused changes

* Version packages (#1664)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* merge

* Reverse order of blocks from sync.getRepo (#1665)

* reverse order of blocks from sync.getRepo

* write to car while fetching next page

* Add hashtag detection to richtext (#1651)

* add tag detection to richtext

* fix duplicate tag index error

* add utils

* fix leading space index failures, test for them

* add changeset

* Version packages (#1669)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* proposed new search lexicons (#1594)

* proposed new search lexicons

* lexicons: lint

* lexicons: fix actors typo

* lexicons: camelCase bites again, ssssss

* lexicons: add 'q' and mark 'term' as deprecated for search endpoints

* codegen for search lexicon updates

* bsky: prefer 'q' over 'term' in existing search endpoints

* search: bugfix

* lexicons: make unspecced search endpoints return skeleton obj

* re-codegen for search skeleton obj

* Disable pds appview indexing (#1645)

* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build

* Refactor PDS appview routes (#1673)

move routes around

* Strip leading `#` from from detected tag facets (#1674)

ensure # is removed from facets

* Version packages (#1675)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Proxy search queries (#1676)

* proxy search

* tweak profile resp

* fix admin.searchRepos

* add mock mailer

* Fix to daniel's MOCKERY of a mock mailer

* Don't allow non-verified email updates until app feature is out (#1682)

stricter updating email until app feature is out

* changesets

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-27 16:08:58 -05:00
Daniel Holmgren
2fa9088639
Disable pds appview indexing (#1645)
* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build
2023-09-25 20:09:52 -05:00
Eric Bailey
d96f7d9b84
add tags to posts (#1637)
* add tags to post lex

* kiss

* add richtext facet and validation attrs

* add tag validation attrs to post

* codegen

* add maxLength for tags, add description

* validate post tags on write

* add test

* handle tags in indexer

* add tags to postView, codegen

* return tags on post thread view

* format

* revert formatting change to docs

* use establish validation pattern

* add changeset

(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)

* remove tags from postView, codegen

* remove tags from thread view

* revert unused changes
2023-09-25 11:14:57 -05:00
devin ivy
d703bc36e6
Clear follow viewer state when blocking (#1659)
* clear follow viewer state when blocking

* tidy
2023-09-25 10:27:48 +02:00
Daniel Holmgren
558981ead1
Disable pds appview routes (#1644)
* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch
2023-09-22 13:51:15 -05:00
bnewbold
7e36c48a77
small syntax lints (#1646)
* lint: remove unused imports and variables

* lint: prefix unused args with '_'

* eslint: skip no-explicit-any; ignore unused _var (prefix)

* eslint: explicitly mark ignores for tricky cases
2023-09-21 10:07:54 -07:00
Daniel Holmgren
36ba295333
Remove appview proxy runtime flags (#1590)
* remove appview proxy runtime flags

* clean up proxy tests
2023-09-14 13:55:12 -05:00
Daniel Holmgren
578757cb44
List feeds (#1557)
* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* lex

* list purpose

* lex gen

* add route

* add proxy route

* seed client helpers

* tests

* mutes and blocks

* proxy test

* snapshot

* hoist actors out of composeThread()

* tidy

* tidy

* run ci on all prs

* format

* format

* fix snap name

* fix snapsh

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-09-14 11:19:04 -05:00
Daniel Holmgren
07bb0da0cf
Remove temp.upgradeRepoVersion (#1588)
remove temp.upgradeRepoVersion
2023-09-12 12:37:57 -05:00