315 Commits

Author SHA1 Message Date
Christian De Angelis
5f9ff1f17f
Remove duplicate description in app.bsky.feed.post (#2183) 2024-02-13 23:01:08 -08:00
Devin Ivy
b400fae24e sanity check on did part 2024-02-13 11:28:19 -05:00
bnewbold
9579bec720
update descriptions in Lexicons (#2110)
* lexicons: update descriptions in com.atproto.*

* lexicons: update descriptions in app.bsky.*

* Apply suggestions from code review

Thanks Emily!

Co-authored-by: Emily Liu <emilyliu7321@gmail.com>

* codegen description-only Lexicon updates

---------

Co-authored-by: Emily Liu <emilyliu7321@gmail.com>
2024-02-11 16:06:58 -08:00
Foysal Ahamed
e4ec7af036
Add date range and comment filter to queryModerationEvents endpoint (#2124)
*  Add date range and comment filter to queryModerationEvents endpoint

*  Add report type and label filters and tests

* tidy tests

*  commentKeyword -> comment and make comment and hasComment independent

*  Fix multiple label filter query

* 📝 Add changesets

* codegen tidy

* add changeset

* remove unused changeset

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-02-05 19:10:44 -05:00
Daniel Holmgren
f023494802
Remove appview feedgen (#2106)
* remove appview feedgen

* deprecate getTimelineSkeleton

* fix dev-env build

* rm unused test
2024-01-30 16:19:36 -06:00
Daniel Holmgren
8994d3633a
Signup queueing - Client & PDS (#2090)
* schema + codegen

* port session refresh

* pds updates

* typo

* changeset
2024-01-25 14:16:30 -06:00
Eric Bailey
4171c04ad8
Add interest tags to preferences (#2086)
* Add interestsPref

* Codegen

* Update lex

* Add method to agent, test

* Codegen

* Format

* Remove console

* Update lex, codegen, update tests

* Update tests

* Format

* Add changeset

* Update property name
2024-01-25 12:05:10 -06:00
Daniel Holmgren
43eb4054a6
Fix typo in getRelationships (#2091)
fix typo in getRelationships + codegen
2024-01-24 19:23:01 -06:00
bnewbold
fb979b1cc5
lexicons: new app.bsky.graph.getRelationships endpoint (#2059)
* lexicons: new app.bsky.graph.getRelationships endpoint

* getRelationships: maxLength 30

* getRelationships: camelCase, consistent, simplify

* getRelationships: remove blocks; and handle in return

* lex: move graph relationships to defs file
2024-01-24 14:06:03 -06:00
Daniel Holmgren
dac5c9e202
Tagged suggestions (#2077)
* add tagged suggestions

* fix codegen

* rm unused err
2024-01-24 09:04:09 -06:00
Foysal Ahamed
15f38560b9
Manage communication templates for moderation purposes (#2045)
*  Add initial lexicons to manage communication templates

*  All 3 endpoints are functional

*  Add list and delete endpoints

*  Add tests for communication template CRUD

* 🔒 Allow only admins to create and update templates

* 🧹 Cleanup according to PR review

*  Make updatedBy and createdBy optional in lexicon

*  Typo

*  Allow string id and update lexicon language

*  Fix tests

*  content -> contentMarkdown

*  Change column name in db table

* add changeset

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-01-19 17:06:37 -05:00
Daniel Holmgren
e43396af09
Phone verification schemas (#2056)
* add phone verification schemas

* lint

* codegen

* changset
2024-01-18 15:17:57 -06:00
bnewbold
51fcba7a79
remove deprecated app.bsky.unspecced.getPopular (#1988)
* remove deprecated app.bsky.unspecced.getPopular lexicon

* codegen

* remove remaining references to getPopular

* changeset for getPopular removal
2024-01-09 15:29:07 -08:00
Daniel Holmgren
de2dbc2903
Split out moderation backend (#1970)
* mv appview

* copy

* finalize copy

* package names

* big WIP

* first pass at mod servce

* some tidy

* tidy & fix compiler errors

* rename to ozone, db migrations, add to dev-env & pds cfg

* getRecord & getRepo mostly working

* fix open handle

* get record tests all working

* moderation events working

* statuses working

* tidy test suite

* search repos

* server & db tests

* moderation tests

* wip daemon + push events

* pds fanout working

* fix db test

* fanning takedowns out to appview

* rm try/catch

* bsky moderation test

* introduce mod subject wrappers

* more tidy

* refactor event reversal

* tidy some db stuff

* tidy

* rename service to mod-service

* fix test

* tidy config

* refactor auth in bsky

* wip patching up auto-mod

* add label ingester in appview

* fix a couple build issues

* fix some timing bugs

* tidy polling logic

* fix up tests

* fix some pds tests

* eslint ignore

* fix ozone tests

* move seeds to dev-env

* move images around

* fix db schemas

* use service auth admin reqs

* fix remaining tests

* auth tests bsky

* another test

* random tidy

* fix up search

* clean up bsky mod service

* more tidy

* default attempts to 0

* tidy old test

* random tidy

* tidy package.json

* tidy logger

* takedownId -> takedownRef

* misc pr feedback

* split daemon out from ozone application

* fix blob takedown mgiration

* refactor ozone config

* do push event fanout on write instead of on read

* make suspend error work again

* add attempts check & add supporting index

* fix takedown test ref

* get tests working

* rm old test

* fix timing bug in event pusher tests

* attempt another fix for timing bug

* await req

* service files

* remove labelerDid cfg

* update snaps for labeler did + some cfg changes

* fix more snaps

* pnpm i

* build ozone images

* build

* make label provider optional

* fix build issues

* fix build

* fix build

* build pds

* build on ghcr

* fix syntax in entry

* another fix

* use correct import

* export logger

* remove event reverser

* adjust push event fanout

* push out multiple

* remove builds
2024-01-05 17:06:54 -06:00
Devin Ivy
5560b7a25a Revert "lexicons: more string limits (#1994)"
This reverts commit ad0d976188d1f07401b9675b5c6045c91e82a84e.
2024-01-03 16:41:07 -05:00
Foysal Ahamed
5e7b0136da
Allow appealing a moderator decision through special report type (#1969)
*  Allow appealing a moderator decision through special report type

*  Allow querying subjects by appealed status

*  Move to appealed boolean state column

*  Remove leftover

*  Move appealed status to new boolean column

*  Throw when non-author attempts to appeal a subject

* 🚨 Appease the linter gods

* build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-01-02 19:17:42 -05:00
bnewbold
ad0d976188
lexicons: more string limits (#1994)
* limit external embed strings sizes

* set a (very large) alt text length limit

The motivation is to have *some* size limit on every string in post
records, to maximize interoperation. For example, we currently have a
CBOR library rejecting some records because of too-long strings.

We don't want to limit the ability of folks to be very descriptive in
alt text, specifically, so chose what seems to be a very large limit. If
this is not large enough, based on feedback, we can bump it even higher.
As context this is the largest string length limit in all of our
lexicons.

* make thumbnail URL fields format=uri

This mostly results in checks against the string being empty, or
unlimited size.

* codegen: string limits

* add changeset for string format lex updates
2024-01-03 00:24:32 +01:00
Foysal Ahamed
905743de9f
Optionally allow comment when sending email to be added to moderation event (#1981) 2023-12-20 09:49:21 -05:00
Daniel Holmgren
0c54951f27
Add seenAt time to listNotifications (#1959) 2023-12-12 15:37:06 -06:00
Eric Bailey
ffe39aae83
Include own replies to own post roots via new filter posts_and_author_threads (#1776)
* Include own replies to own post roots in post_no_replies

* Updates

* Simplify

* snaps

* snaps

* Remove unused import

* Add new threads filter to getAuthorFeed

* Implement new filter

* Typo
2023-12-08 15:32:06 -06:00
Daniel Holmgren
f9fd3e68ca
Feature branch: PDS v2 (#1789)
* cleanup repeat process all

* wip

* skip actor search test

* skip actor search test

* tweak processAll

* decrease wait to 1 sec

* repo_blob -> record_blob

* simplify backlink linkTo

* return repo_root to one row

* sequence before updating repo_root

* invite code forUser -> forAccount

* ipld_block -> repo_block

* use lru-cache fetchMethod

* move did_cache to own db

* better error handling on did cache

* drop did_handle

* fix sequencer wait time

* debug

* debug

* more debug

* check something

* fix bday paradox

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* user_account -> account

* remove inviteNote

* keypair per repo

* use an lru cache for keypairs as well

* clean up repo

* wip

* wrap up accoutn manager

* tidy

* tidy

* fix tests

* fix disabled codes

* fix appview tests

* add note

* set pragmas

* tidy account manager getDb

* rename pref transactor

* user pref -> account pref

* handle blob imports

* tidy imports

* add reserveSigningKey

* wip transferAccount

* clean up transferAccount

* tests

* tidy

* tidy

* configure entryway url on pds

* handle entryway in pds admin endpoints

* make importRepo temp

* fix imports

* make email optional on pds when using entryway

* handle diffs

* handle pds entryway usage for server, identity, admin endpoints

* pds support for credentials from entryway

* setup pds tests w/ entryway service

* tidy

* tidy

* update entryway version

* wip

* test handle updates w/ entryway

* split account table into two

* tidy

* tweak scripts

* tidy tests

* tidy

* better config for actorstore & dbs

* clean up cfg more

* reorg actorstore fs layout

* handle erros on actor db create

* pr tidy & fix accoutn deletion test

* pr feedback

* fix bad merge

* unskip test

* fix subscribe repos tests

* tidy repo root tables

* tidy

* fix tests

* tidy delete tokens

* tidy account getters

* tidy

* bulk deletesg

* increase chunk size

* handle racing refreshes

* wip

* fix auth test

* invert import flow

* clean up actor store on create account failure

* tweak sequencer

* prevent invite code races on createAccount

* rm note

* add back in race protection on getAccountInviteCodes

* start feature branch

* deleted app migration table

* patch up new auth test

* rm note

* g

* create accoutn delegated from entryway

* tidy

* fix test

* change plcOp type to unknown

* small fixes

* sync up w entryway branch

* Use proper error when authed account is not found (#1799)

provide proper error when account not found in access-takedown check

* build branch

* build on ghcr

* tweak service file

* tweak service file

* change where we save reserved keys

* no tmp dir in blobstore either

* fix blobstore temp location again

* handle repeat record_blobs

* create account before submitting plc op & undo if fail

* small tweak

* limit the number of local records

* push out empty commit on transfer

* fix issue with record_blob

* add push blob endpoint

* Set and validate token audiences on pds v2 (#1793)

set and validate token audience on pds v2

* merge

* include entryway did on tests

* build branch

* fix cache issue

* xrpc server blob limit

* put correct bytes

* add auth to routes

* handle quarantining/unquarantining a blob that does not exist

* tidy

* fix transfer tests

* fix email request routes for entryway

* PDS v2 entryway account deletion (#1819)

* add admin lexicon for account deletion

* implement admin account deletion endpoint

* fix entryway proxying on account email checks

* proxy to entryway for acct deletion

* read-after-write sanity check

* tweak

* wip

* finish refactor

* fix test schema

* application retry logic for busy

* pr feedback

* rm lru-cache

* fix test pg schema

* fix transfer test

* Sqlite instrumentation for pds v2 (#1838)

* sqlite instrumentation

* build

* remove build

* dont reimport blobs

* send ticks during import

* close on error

* catch handle validation error

* add log

* fix test

* return emailConfirmedAt on getAccountInfo

* Upgrade sharp on pds v2 (#1863)

upgrade sharp to 0.32.6

* read all bytes before parsing car

* Async car reader (#1867)

* asynchronously read in car

* dont buffer car

* tweak

* Gracefully handle indexing of invalid records (#1853)

* gracefully handle indexing of invalid records

* fix repo tests

* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)

fix role auth for access-or-role verifier, fix getBlob actor takedown check

* better cleanup of actor-stores

* add ability to not ensure leaves

* tidy

* allow did:web transfer

* Migration utility for actor-store (#1873)

beginnings of helper for migrating all actors

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* base case for findBlobRefs

* App-level retries for sqlite on pds (#1871)

* revamp retry helper to be more flexible re: backoff strategies

* sqlite timeout helper

* ensure sqlite wal on db creation/migration rather than every open

* layer retries for sqlite on writes outside transactions on pds

* tidy

* fix up lockfile

* tidy

* fix lex codegen

* fix timing bug in threadgate test

* No-op update handling (#1916)

do no produce commits on no-op updates

* Retry on all SQLITE_BUSY error codes (#1917)

retry on all sqlite_busy error codes

* Pds v2 ensure sqlite ready (#1918)

ensure sqlite is ready before making queries

* try something

* tidy

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-12-04 18:00:09 -06:00
intrnl
8d9b1f70cd
Attach record URI to listItemView (#1758)
* feat: attach record uri to listitemview

* chore: attempt to update test snapshots
2023-12-01 14:28:54 -06:00
Foysal Ahamed
1f9040a44d
Evented architecture for moderation system (#1617)
* 🚧 WIP with proposed lexicons for event based mod architecture

* 🚧 Remove unnecessary moderation action lexicon

* 🚧 Working on event based actions

*  Add escalated subject status

* 🐛 Alright, fixed the error in lexicon

* 🚧 Working through reversal

*  Cleanup build errors

*  Add subject status endpoint

*  Add handler

*  get reports from mod actions table

* :rightwards_twisted_arrows: Merge with upstream

* 🚧 Builds but test network doesnt start

*  Tests passing on event based status change

*  Rename index

* ♻️ Rename takeModerationAction->emitModerationEvent

*  Implement label reversal

*  Auto-revert test working

* ♻️  Refactored to event types and tests are passing

*  Add takedown event sequence validation

*  Adds support for blobCid status

* 🧹 Cleanup unnecessary method:

*  Hydrate handles with status and events

*  Re-implement auto reversal

*  Add takendown and mute filters

*  Allow filtering events by type

*  Allow filtering events by creator did

*  Add subjectStatus to record and repoview

*  Add persistent note feature

*  Log send email event

* 🐛 Fix logging send email event

*  Better type

*  Adjust migration to create separate moderation_event table

* 🧹 Cleanup types

*  Adjust tests with mod event emitter

*  Fix more tests around takedowns

*  Get test suite to pass

*  Get test suite to pass for pds

*  Get test suite to pass for pds

*  Update snapshot for feedgen

*  Why are more snapshots updating?

* ♻️ Rename getModerationEvents -> queryModerationEvents

* ♻️ Rename getModerationStatuses -> queryModerationStatuses

* ♻️ Rename persistNote->sticky

* 🐛 Rename subject

* ♻️ Cleanup expiresAt for scheduled actions

*  Add more tests, allow fetching mod history for all content by a user

*  Fix repo and record tests

*  Migrate reports and actions to events

* 🐛 Fix escalated status overwrite

*  Implement direct sql query to create events from actions and reports

* 🚧 Adding keyset pagination for subject statuses

*  Add migration for lastReportedAt

*  Migrate blob cids

*  Fix pagination on mod subject list endpoint

* 🐛 Fix blob actions

*  All tests passing on bsky package

*  Bring back snapshots

*  Skipping timeline test temporarily

*  Skipping some more tests to isolate failing ones

*  Bring back list-feed test

*  Bring back timeline test

*  Fix label action in seeding

*  Enable timeline proxied test

*  Enable search actor proxied test

*  Enable feedgen tests

*  Fix test for admin/get-record

*  Move note to comment for subject status

*  Accept comments in mute event

*  Remap flag event to ack event

* 🐛 Add legacyRef in report union selection

* @atproto/api 0.6.24-next.0

* @atproto/api 0.6.24-next.1

*  Adjust migration export and add index for blobCids column

*  Maintin action ids when migrating

*  Paginate events using createdAt timestamp

*  Update snapshot for pds test with events cursor update

*  Use only events for snapshot testing

*  Use only events for snapshot in the remaining test

* relative paths to lexicons for build

* fix bsky periodic event reversal in service entrypoint

*  Allow comments in takedown and label

*  Only import reports on consecutive run of the migration script

*  Adjust moderation property of blob entries

* determine latest reports to migrate

*  Process new reports for subject status

*  Process unresolved reports on first migration run

* fix transaction error, process just unresolved reports, make reported-at updates safe for reruns

* tidy

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-11-30 17:53:56 +01:00
Daniel Holmgren
7edad62c12
Put canReply state on post viewer state instead of thread viewer state (#1882)
* switch canReply from thread to post view

* tweaks & fix up tests

* update snaps

* fix more snaps

* hydrate feed items for getPosts & searchPosts

* fix another snapshot

* getPosts test

* canReply -> blockedByGate & DRY up code

* blockedByGate -> excludedByGate

* replyDisabled
2023-11-27 20:14:20 -06:00
Emily Liu
59f70dbe80
Tweak lexicon descriptions (#1859)
* Tweak lexicon descriptions

* Tweak birthdate

* Update lexicons/app/bsky/actor/getProfiles.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Regenerate packages from lexicons

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-11-20 17:41:25 -05:00
Daniel Holmgren
b05130db68
Add temp.fetchLabels method (#1858)
* add temp.fetchLabels route

* update description
2023-11-16 11:07:52 -06:00
Foysal Ahamed
697f5d3627
Expose emailConfirmedAt field from admin getRepo (#1757)
*  Expose emailConfirmedAt field from admin getRepo

* ♻️ Fix typing for repo result

* 🧹 Cleanup unnecessary import

*  Adapt to the new pds based get account info method

* 🧹 Cleanup unused pds util
2023-11-08 00:26:31 +01:00
devin ivy
772736a010
Add did doc output to getSession for session resumption (#1806)
* add optional did doc output to getSession lexicon

* add did doc output to getSession on pds, update client to use it

* api test fixes

* api changeset

* tidy
2023-11-02 16:16:26 -04:00
Daniel Holmgren
cf848e87ab
Tweak schemas for entryway createAccount (#1797)
* tweak scheams

* require email & password
2023-11-01 14:56:57 -05:00
devin ivy
a161f815de
Make plc op unknown object rather than bytes (#1792)
make plc op unknown object rather than bytes
2023-11-01 11:54:00 -04:00
Daniel Holmgren
fcb19c9c51
Simplify PDS moderation (#1723)
* spec out new simple pds mod routes

* introduce new admin state endpoints

* wire up routes

* clean up pds

* revoke refresh tokens

* getUserAccountInfo

* pr tidy

* fixing some tests

* fixing up more tests

* fanout takedowns to pds

* fanout admin reqs to pds

* tidy

* more tidy & add more pds moderation tests

* getUserAccountInfo -> getAccountInfo

* dont hydrate pds info on searchRepos

* fix build

* port admin tests to bsky package

* clean up old snaps

* tests on fanout

* tweak naming

* missed a rename

* tidy renames

* fix lex name

* tidy & move snap

* fix build

* cleanup repeat process all

* skip actor search test

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* change takedownId col to takedownRef

* build branch

* fix bsky test

* add service key to indexer

* move signing key to api entry

* dont build
2023-10-30 16:56:17 -05:00
bnewbold
ec0dfdc8f5
lexicon: maximum report "reason" length of 1000 chars (graphemes) (#1171)
* lexicon: maximum report length of 500 chars (graphemes)

* lexicon: bump maximum report size to 1000 chars

* lexicon: bump max report size again to 2k graphemes

* make codegen
2023-10-30 09:28:42 -07:00
Paul Frazee
9c98a5baaf
Modlist updates: add "blockingByList" information and some utility functions to the sdk (#1779)
* Add blockingByList to app.bsky.actor.defs#viewerState

* Add blocking-by-list behaviors to moderation sdk

* Add modlist helper functions to bsky-agent

* codegen

* hydrate blockingByList in profile viewer state

* ignore self-mutes and self-blocks in read path

* format

* changeset

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-10-27 14:35:36 -04:00
devin ivy
46b108cb86
Facilitate authing w/ PDS based on DID doc (#1727)
* lexicon for did doc w/ auth credentials

* include did doc w/ session when configured. configure on dev-env.

* Add dynamic PDS URL adoption to the client

* remove usage of did doc field from getsession in client

* dry-up did doc type and validation

* remove explicit dep on zod by identity package

* move more did doc parsing to common-web

* go back to strings

* rollback breaking changes to identity package

* add changeset

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: dholms <dtholmgren@gmail.com>
2023-10-26 18:29:51 -04:00
devin ivy
696df1e488
Lexicons to support bring-your-own-did w/ per-repo signing keys (#1739)
* lexicons for per-repo signing keys

* fix

* rename lexicon

* fix filename

* rename getSigningKey to reserveSigningKey
2023-10-26 17:15:12 -04:00
Daniel Holmgren
8853bc174a
Add rev to sync.listRepos (#1709)
add rev to listRepos
2023-10-04 18:46:57 -05:00
Daniel Holmgren
41ee177f5a
Email confirmation/update (#1568)
* lexicons

* codegen

* email templates

* request routes

* impl

* migration

* tidy

* tests

* tidy & bugfixes

* format

* fix api test

* fix auth test

* codegen

* add unique constraint

* Add email confirmed to AtpSessionData

* interop test files (#1529)

* initial interop-test-files

* crypto: switch signature-fixtures.json to a symlink

* syntax: test against interop files

* prettier

* Update interop-test-files/README.md

Co-authored-by: Eric Bailey <git@esb.lol>

* disable prettier on test vectors

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>

* add getSuggestedFollowsByActor (#1553)

* add getSuggestedFollowsByActor lex

* remove pagination

* codegen

* add pds route

* add app view route

* first pass at likes-based suggested actors, plus tests

* format

* backfill with suggested_follow table

* combine actors queries

* fall back to popular follows, handle backfill differently

* revert seed change, update test

* lower likes threshold

* cleanup

* remove todo

* format

* optimize queries

* cover mute lists

* clean up into pipeline steps

* add changeset

* List feeds (#1557)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* lex

* list purpose

* lex gen

* add route

* add proxy route

* seed client helpers

* tests

* mutes and blocks

* proxy test

* snapshot

* hoist actors out of composeThread()

* tidy

* tidy

* run ci on all prs

* format

* format

* fix snap name

* fix snapsh

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Improve xrpc server error handling (#1597)

improve xrpc server error handling

* Remove appview proxy runtime flags (#1590)

* remove appview proxy runtime flags

* clean up proxy tests

* getPopular hotfix (#1599)

dont pass all params

* Interaction Gating (#1561)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* initial lexicons for interaction-gating

* add interactions view to post views

* codegen

* model bad reply/interaction check state on posts

* initial impl for checking bad reply or interaction on write

* omit invalid interactions from post thread

* support not-found list in interaction view

* hydrate can-reply state on threads

* present interaction views on posts

* misc fixes, update snaps

* tidy/reorg

* tidy

* split interaction gating into separate record in lexicon

* switch interaction-gating impl to use separate record type

* allow checking reply gate w/ root post deletion

* fix

* initial gating tests

* tighten gated reply views, tests

* reply-gating list rule tests

* allow custom post rkeys within window

* hoist actors out of composeThread()

* tidy

* update thread gate lexicons, codegen

* lex fix

* rename gate to threadgate in bsky, update views

* lex fix

* improve terminology around reply validation

* fix down migration

* remove thread gates on actor unindexing

* add back .prettierignore

* tidy

* run ci on all prs

* syntax

* run ci on all prs

* format

* fix snap

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* order by `like.indexedAt` in app view (#1592)

* order by like.indexedAt

* use keyset for ordering

* simplify

* ok ok ok I get it now

* Update packages/bsky/src/api/app/bsky/feed/getActorLikes.ts

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* Remove default value for post table invalid attrs (#1601)

remove default value for post table attrs

* Version packages (#1602)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* update Bluesky PBLLC to PBC (Public Benefit Corporation) (#1600)

* Temporarily disable filtering `invalidReplyRoot`s (#1609)

temporarily disable invalidReplyRoot check

* fix syntax docs (#1611)

* Version packages (#1612)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Allow bypass on ratelimit ip (#1613)

allow bypass on ratelimit ip

* Write rate limits (#1578)

* get rate limit ip correctly

* add write rate-limits

* Tweak createSession rate limit key (#1614)

tweak create session rl key

* Filter preferences for app passwords (#1626)

filter preferences for app passwords

* Tweak rate limit setup for multi rate limit routes (#1627)

tweak rate limit setup for multi rate limit routes

* Remove zod from xrpc-server error handling (#1631)

remove zod from xrpc-server error handling check

* Enforce properties field on lexicon object schemas (#1628)

* add empty properites to thread gate schema fragments

* tweak lexicon type

* Add feed-vew and thread-view preferences (#1638)

* Add feed and thread preference lexicons

* Add feed-view and thread-view preference APIs

* Add changeset for new preferences  (#1639)

Add changeset

* Version packages (#1640)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable getAccountInviteCodes for app passwords (#1642)

disable getAccountInviteCodes for app passwords

* remove cruft packages (uri, nsid, identifier) (#1606)

* remove @atproto/nsid (previously moved to syntax)

* remove @atproto/uri (previously moved to syntax)

* remove @atproto/identifier (previously moved to syntax)

* bump lockfile to remove old packages

---------

Co-authored-by: Eric Bailey <git@esb.lol>

* api: update login/resumeSession examples in README (#1634)

* api: update login/resumeSession examples in README

* Update packages/api/README.md

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* small syntax lints (#1646)

* lint: remove unused imports and variables

* lint: prefix unused args with '_'

* eslint: skip no-explicit-any; ignore unused _var (prefix)

* eslint: explicitly mark ignores for tricky cases

* indicate that getPopular is deprecated (#1647)

* indicate that getPopular is deprecated

* codegen for deprecating getPopular

* tidy up package.json and READMEs (#1649)

* identity: README example and tidy

* tidy up package metadata (package.json files)

* updated README headers/stubs for several packages

* crypto: longer README, with usage

* syntax: tweak README

* Apply suggestions from code review

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

* Improve the types of the thread and feed preferences APIs (#1653)

* Improve the types of the thread and feed preferences APIs

* Remove unused import

* Add changeset

* Version packages (#1654)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable pds appview routes (#1644)

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch

* auto-moderator tweaks: pass along record URI, create report for takedown action (#1643)

* auto-moderator: include record URI in abyss requests

* auto-moderator: log attempt at hard takedown; create report as well

The motivation is to flag the event to mod team, and to make it easier
to confirm that takedown took place.

* auto-mod: typo fix

* auto-mod: bugfixes

* bsky: always create auto-mod report locally, not pushAgent (if possible)

* bsky: fix auto-mod build

* bsky: URL-encode scanBlob call

* Clear follow viewer state when blocking (#1659)

* clear follow viewer state when blocking

* tidy

* add `tags` to posts (#1637)

* add tags to post lex

* kiss

* add richtext facet and validation attrs

* add tag validation attrs to post

* codegen

* add maxLength for tags, add description

* validate post tags on write

* add test

* handle tags in indexer

* add tags to postView, codegen

* return tags on post thread view

* format

* revert formatting change to docs

* use establish validation pattern

* add changeset

(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)

* remove tags from postView, codegen

* remove tags from thread view

* revert unused changes

* Version packages (#1664)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* merge

* Reverse order of blocks from sync.getRepo (#1665)

* reverse order of blocks from sync.getRepo

* write to car while fetching next page

* Add hashtag detection to richtext (#1651)

* add tag detection to richtext

* fix duplicate tag index error

* add utils

* fix leading space index failures, test for them

* add changeset

* Version packages (#1669)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* proposed new search lexicons (#1594)

* proposed new search lexicons

* lexicons: lint

* lexicons: fix actors typo

* lexicons: camelCase bites again, ssssss

* lexicons: add 'q' and mark 'term' as deprecated for search endpoints

* codegen for search lexicon updates

* bsky: prefer 'q' over 'term' in existing search endpoints

* search: bugfix

* lexicons: make unspecced search endpoints return skeleton obj

* re-codegen for search skeleton obj

* Disable pds appview indexing (#1645)

* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build

* Refactor PDS appview routes (#1673)

move routes around

* Strip leading `#` from from detected tag facets (#1674)

ensure # is removed from facets

* Version packages (#1675)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Proxy search queries (#1676)

* proxy search

* tweak profile resp

* fix admin.searchRepos

* add mock mailer

* Fix to daniel's MOCKERY of a mock mailer

* Don't allow non-verified email updates until app feature is out (#1682)

stricter updating email until app feature is out

* changesets

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-27 16:08:58 -05:00
Daniel Holmgren
2fa9088639
Disable pds appview indexing (#1645)
* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build
2023-09-25 20:09:52 -05:00
bnewbold
233a132c11
proposed new search lexicons (#1594)
* proposed new search lexicons

* lexicons: lint

* lexicons: fix actors typo

* lexicons: camelCase bites again, ssssss

* lexicons: add 'q' and mark 'term' as deprecated for search endpoints

* codegen for search lexicon updates

* bsky: prefer 'q' over 'term' in existing search endpoints

* search: bugfix

* lexicons: make unspecced search endpoints return skeleton obj

* re-codegen for search skeleton obj
2023-09-25 16:56:00 -07:00
Eric Bailey
d96f7d9b84
add tags to posts (#1637)
* add tags to post lex

* kiss

* add richtext facet and validation attrs

* add tag validation attrs to post

* codegen

* add maxLength for tags, add description

* validate post tags on write

* add test

* handle tags in indexer

* add tags to postView, codegen

* return tags on post thread view

* format

* revert formatting change to docs

* use establish validation pattern

* add changeset

(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)

* remove tags from postView, codegen

* remove tags from thread view

* revert unused changes
2023-09-25 11:14:57 -05:00
bnewbold
1487c9f1fe
indicate that getPopular is deprecated (#1647)
* indicate that getPopular is deprecated

* codegen for deprecating getPopular
2023-09-21 14:50:23 -07:00
Paul Frazee
061865c8bf
Add feed-vew and thread-view preferences (#1638)
* Add feed and thread preference lexicons

* Add feed-view and thread-view preference APIs
2023-09-20 10:22:27 -07:00
Daniel Holmgren
828dfa8a8a
Enforce properties field on lexicon object schemas (#1628)
* add empty properites to thread gate schema fragments

* tweak lexicon type
2023-09-19 10:51:26 -05:00
Daniel Holmgren
9879ca97b7
Interaction Gating (#1561)
* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* initial lexicons for interaction-gating

* add interactions view to post views

* codegen

* model bad reply/interaction check state on posts

* initial impl for checking bad reply or interaction on write

* omit invalid interactions from post thread

* support not-found list in interaction view

* hydrate can-reply state on threads

* present interaction views on posts

* misc fixes, update snaps

* tidy/reorg

* tidy

* split interaction gating into separate record in lexicon

* switch interaction-gating impl to use separate record type

* allow checking reply gate w/ root post deletion

* fix

* initial gating tests

* tighten gated reply views, tests

* reply-gating list rule tests

* allow custom post rkeys within window

* hoist actors out of composeThread()

* tidy

* update thread gate lexicons, codegen

* lex fix

* rename gate to threadgate in bsky, update views

* lex fix

* improve terminology around reply validation

* fix down migration

* remove thread gates on actor unindexing

* add back .prettierignore

* tidy

* run ci on all prs

* syntax

* run ci on all prs

* format

* fix snap

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-09-14 15:24:48 -05:00
Daniel Holmgren
578757cb44
List feeds (#1557)
* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* lex

* list purpose

* lex gen

* add route

* add proxy route

* seed client helpers

* tests

* mutes and blocks

* proxy test

* snapshot

* hoist actors out of composeThread()

* tidy

* tidy

* run ci on all prs

* format

* format

* fix snap name

* fix snapsh

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-09-14 11:19:04 -05:00
Eric Bailey
3877210e7f
add getSuggestedFollowsByActor (#1553)
* add getSuggestedFollowsByActor lex

* remove pagination

* codegen

* add pds route

* add app view route

* first pass at likes-based suggested actors, plus tests

* format

* backfill with suggested_follow table

* combine actors queries

* fall back to popular follows, handle backfill differently

* revert seed change, update test

* lower likes threshold

* cleanup

* remove todo

* format

* optimize queries

* cover mute lists

* clean up into pipeline steps

* add changeset
2023-09-13 15:56:12 -05:00
Daniel Holmgren
07bb0da0cf
Remove temp.upgradeRepoVersion (#1588)
remove temp.upgradeRepoVersion
2023-09-12 12:37:57 -05:00
Daniel Holmgren
3cdc72a0a7
Fix getRepo since (#1579)
* fix schemas

* test

* format
2023-09-12 12:37:45 -05:00
devin ivy
6f419790ac
Feature: block lists (#1531)
* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* hoist actors out of composeThread()

* tidy

* run ci on all prs

* format

* build

* proxy graph.getListBlocks

* remove unneeded index

* build pds

* setup noop listblock indexer on pds

* remove build

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-09-12 12:07:31 -04:00
Paul Frazee
44b721b6ae
Add personal-details user preference with birth date (#1565)
* Add personal-details user preference with birth date

* Add personal details pref to settings union
2023-09-08 10:35:49 -07:00