41 Commits

Author SHA1 Message Date
rafael
638f5a8312
Fix avatar path resolution in dev-env (#3266)
* Fix avatar path resolution in dev-env

* changeset

* extract dev-env assets to dedicated folder

* add comment

* fix fmt
2024-12-19 11:25:41 -03:00
rafael
2d9a2a8f47
Return empty cursor if feed echoes input cursor (#2989)
* Throw if custom feed echoes input cursor out

* Return empty cursor instead of throwing
2024-11-18 18:16:33 -05:00
Hailey
4ab2483547
Add quoteCount to embed view (#2735)
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-22 09:49:06 -07:00
Eric Bailey
aba664fbdf
Detached QPs and hidden replies (#2675)
* Add new postgate lex, hiddeReplies to threadgate, codegen

* Add protobufs

* Add to mock dataplane

* Add matching postgate method to feed hydration methods

* Add to getRecord

* Add to HydrationState

* Fix typo

* Add to mergeStates, fetch embeds in threads

* Integrate into embed views

* Add test for QPs in threads

* Add feed test

* Fix naming convention in protos

* Add #viewRemoved record view, rename postgate.json

* Integrate new view

* Filter hidden replies from feeds

* Filter out replies at the handler level, do not filter for author feeds

* Fix lint

* Move hidden reply check to view layer

* Reduce, reuse, recycle

* Rename to lowercase

* Rename layer vars

* Add quote gate props to postgate (#2693)

* Add quote gate props to postgate

* Consistent naming

* Fix record structure

* Codegen

* Show hidden replies in author feed

* Allow reposts of hidden replies

* Lex and codegen

* Add violates_quote_gate to proto

* Consistent naming, codegen

* Integrate violatesQuotegate and canQuotepost

* Remove rules, codegen

* Hydrate all postgates for all requested posts

* Match other impl

* Add test, need to split these out

* Format

* Hydrate first nested embeds too

* Add postgate test suite

* Add violatesQuoteGate to dataplane

* Ingest and set violatesQuoteGate, return on meta

* Return removed embed for quotes that violate gate

* Add test

* Dedupe URIs before fetching postgates

* Update snaps

* Snap

* Format

* Updating naming conventions for postgate-related attributes

* Correct naming

* Consistency

* Proto too

* Rename to viewDetached

* Codegen

* Rename everything

* Codegen

* Quotes that violate a quote gate can still be quoted themselves

* Couple more renames

* Snaps

* Ensure reply ref is tombstoned for hidden replies

* Split out hidden replies tests and create fresh fixture

* Hydrate threadgates for reply notifications, filter hidden replies

* Remove snap

* Add flaky test

* Rename violatesEmbeddingRules

* Fix flaky test

* Only write to db if violatesEmbeddingRules is true

* DRY up post uri -> gate uri logic

* isThreadgateListRule

* Don't share users object between tests

* No pascal

* Remove default params

* Find -> some

* canQuotepost -> canEmbed, remove optional

* Fix quoteee typo

* await follows

* Throw in post uri -> gate utils

* Ensure fetch threadgates for reply roots

* Don't hydrate threadgates twice

* DRY up uri -> did parsing

* Clean up parsePostgate logic

* Format

* Revert change

* Revert change

* Replace a couple more uri->did conversions

* Only filter replies from feeds if viewer hid them

* Revert, filter out replies that are hidden from feeds

* Remove old test

* Replace uri->did util

* Revert change to unused file

* Only validatePostEmbed and check postgates for post records

* Ensure notifications aren't generated down a hidden reply chain

* Changeset

* Cleanup

* Fix notification filtering logic

* Simplify

* Don't notify for invalid embeds

* Use new APIs

* Add hasPostGate and hasThreadGate flags from dataplane

* Only fetch postgates if post has one

* Only fetch threadgates if post has one or was deleted

* Remove notification filtering

* Don't hydrate threadgates for notifications

* Move hidden replies in feeds to match block handling

* Do no filtering of hidden replies in feeds

* Revert "Don't hydrate threadgates for notifications"

This reverts commit 1dcec0b239a7b9d6800427b26b8ba3e6a54210f9.

* Revert "Remove notification filtering"

This reverts commit 1e7069dfd809d1f18e9f05fd1d422e7399aa1bb0.

* Filter notifications for OP only

* Add additional check to hidden replies test

* Move noty filter logic into method handler

* Update .changeset/perfect-parrots-appear.md

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/bsky/tests/seed/postgates.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Another structuredClone

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Better comment

* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Regen protos to match dataplane

* Update quotes snap to include embeddingDisabled

* Clarify usage of post uri -> gate utils

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2024-08-21 14:36:51 -05:00
Hailey
2a0c088cc5
quote aggs, list quotes (#2658)
* add quote count to post_agg, add getPostQuotes

rework schema

rework schema

add getPostQuotes to api

use posts

use posts

codegen

use items instead of quotes

codegen

add getPostQuotes

add quoteCount to response

update lexicon for postview

increment post ags

add quote to post aggs

add quote interface

oops

add quote table migration

* update

* bufgen

* update params

* update to use v2

* logs

* rm comment

* pass cursor

* add index

* Update packages/bsky/src/data-plane/server/db/migrations/20240723T220703655Z-quotes.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* only if its a post

* tests

* Discard changes to packages/bsky/tests/views/posts.test.ts

* fix client call

* Include new quotes agg in test expectation

* Use new API for headers

* Update packages/bsky/src/data-plane/server/indexing/plugins/post.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* revert rm

* rm timeout

* cursor test

* Changeset

* Remove pds specific bump

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: Eric Bailey <git@esb.lol>
2024-08-21 12:33:05 -05:00
Hailey
ed58101790
Add starter packs to post hydration (#2613)
* add hydration state

* add view to post

* oops. `graph.defs` not `starterpack.defs`

* add tests, update snapshots

* update record lexicon

* codegen

* add changeset

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-07-02 17:52:28 -04:00
devin ivy
af7d3912a3
Starter packs (#2553)
* lexicon: initial starter pack record lexicons

* lexicon: first pass on views for starter packs

* lexicon: add starter pack query endpoints, misc fixes

* lexicon: add starter pack related fields to profile views

* lexicon: codegen

* api prerelease

* bsky: protos for starter packs

* appview: implement starter pack endpoints in dataplane, protos update

* appview: starter pack hydration and view, mock dataplane indexing

* appview: wire-up starter pack graph endpoints

* appview: test basic starter packs functionality, update dev-env

* fix snaps for profile createdAt

* appview: fix list item count for starter packs in dataplane

* fix ozone snaps, tidy

* appview: reorg list aggs for starter packs

* appview: starter pack lexicon and proto tweaks, move around aggregations

* appview: fix views and snaps

* api prerelease

* fix snaps

* appview: misc proto updates for starter packs, snaps

* appview: add starter pack counts to profile detail

* build

* api prerelease

* pds: starter pack slur detection

* lexicon: starter pack notifications and batch fetch

* appview: starter pack notification and batch fetch

* appview: add starterpack-joined notif to dataplane and test

* api prerelease

* ozone: fix snaps

* pds build

*  Proxy starterpack requests through ozone and include takedowns (#2594)

*  Proxy starterpack requests through ozone and include takedowns

* 🧹 Cleanup

*  Commit snapshots

*  Update snapshot

* lexicon: move starterpacks back to TIDs

* changeset

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-06-24 21:04:33 -04:00
Daniel Holmgren
7c1973841d
Thread mutes (#2558)
* schemas

* protos

* implementation

* add filtering on ingestion

* test

* add thread viewer state to post views

* update snaps

* changeset

* fix threadgating test
2024-06-13 15:30:49 -05:00
devin ivy
499fabe7b6
Appview: add chat details to profile views, chronological likes (#2471)
* update bsky protos

* appview: add associated chat deets to profile views, switch to sorted likes

* appview: test associated chat info on profile views

* tidy
2024-05-09 12:45:06 -04:00
Daniel Holmgren
e3063041a3
Feed context & sendInteractions impl (#2402)
* forward feedContext on getFeed

* add sendInteractions to PDS

* fix typeof

* add feedContext to snapshots

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-04-26 17:30:08 -04:00
devin ivy
84cdc5b65a
Appview: include like/repost/reply counts on post embed (#2395)
* lexicon: add post counts to embed#viewRecord

* appview: hydrate and present aggregations on post embeds

* update snaps for aggregations on post embeds
2024-04-09 21:24:47 -04:00
Daniel Holmgren
069d1432dc
Remove false neg on the labels that ozone emits (#2310)
* remove `neg` on the labels that ozone emits

* no negs in appview either

* no negs on self-labels

* no neg on self labels in ozone either

* fix query labels test
2024-03-13 17:33:50 -05:00
devin ivy
8dd67f5c81
Appview: labels on feed gens and list views (#2298)
* lexicon: add labels to feedgen and list views

* appview: hydrate labels onto lists, feedgens

* test hydration of labels on lists and feedgens

* update pds snap
2024-03-12 16:04:02 -05:00
Daniel Holmgren
f65de89eed
Feature: Appview v2 (#1924)
* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* filler commit

* rm filler

* server boilerplate

* follows impl

* posts impl

* posts & likes impl

* repost impl

* profiles & handle null values

* list impl

* mutes impl

* blocks impl

* misc

* feed gen impl

* label impl

* notifs impl

* feeds impl

* threads impl

* early sketchwork

* wip

* stub out thick client

* in-progress work on hydrator

* tweak

* hydrate profile labels, detail lists

* feedgen hydration

* protobuf tweaks

* more protobuf tweaks

* wip

* snake case

* moar snake case

* tidy actor hydration

* tidy parsing

* type fixes, renaming, comments in hydrator

* hydrate list items and likes

* hydrate notifications

* feed hydration

* graph & label hydration

* more record protobufs

* pluralize

* tweak pbs

* use new methods

* Setup dataplane grpc client/mock server (#1921)

* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* add record getter mocks

* post view hydration

* fix up mock dataplane to match new protos

* missed one

* wire up dataplane in ctx & dev-env

* adding some basic views

* feed hydration, add blocks to post hydration

* pass over notification hydration

* tidy

* merge

* implement getProfile

* hydrate post aggregation and viewer state

* fix

* fix codegen

* get some tests passing!

* add takedowns & some like bugfixing

* all profile tests passing!

* likes test

* follow endpoints using data plane

* reorg follow block rules

* reposts

* post views!

* implement getList w/ dataplane caveat

* adjust dataplane getListMembers to return listitem uris

* implement getListMutes and -Blocks w/ dataplane

* suggestions

* timeline

* misc view fixes

* view fixes for mutes, self-mute/block

* author feed

* feed gen routes

* tidy

* misc block/mute fixes

* list feed & actor likes

* implement getLists, fix some empty cursors

* implement getMutes, empty profile description fix

* implement getBlocks, block application fix

* implement getSuggestedFollowsByActor, needs some fixes

* feed generation

* search routes

* threads

* tidy

* fix some snaps

* fix getSuggestedFollowsByActor

* implement listNotifications

* implement getUnreadCount w/ dataplane

* implement notifications.updateSeen w/ dataplane

* 3rd party blocking tests

* blocked profile viewer

* add search mocks

* refactor getFeed

* createPipeline -> createPipelineNew

* basic replygating functionality on dataplane w/o filtering violating replies

* hack threadgates into dataplane, apply gates

* deterministic thread orders in dataplane

* misc cleanup around dataplane

* upgrade typescript to v5.3

* update typescript linter deps

* sync bsky proto, codegen

* update dataplane, sync with bsky proto updates

* remove indexer, ingester, daemon, moderation services from appview

* convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane

* remove bsky services. db and indexing logic into mock dataplane.

* remove tests not needed by appview v2, misc reorg

* add basic in-mem repo subscription to dataplane mock

* fix dev-env, bsky tests, bsky build

* cull bsky service entrypoint

* add bsky service readme

* build

* tidy

* tidy, fix pds proxy tests

* fix

* fix bsky entrypoint deps

* support http2 grpc client

* build

* fix dataplane bad tls config/default

* support multiple dataplane urls, retry when unavailable

* build

* tidy/fix

* move dataplane mock tests into their own dir

* cover label hydration through timeline test

* bring back labels in appview tests

* remove unused db primary/replica/coordinator from bsky dataplane

* bsky proto add cids to contracts, buf codegen

* sync-up bsky data-plane w/ codegen updates

* start using dataplane interaction endpoints

* add file

* avoid overfetching from dataplane, plumb feed items and cids

* pass refs through for post viewer state

* switch list feeds to use feed item in dataplane

* handle not found err on get-thread dataplane call

* support use of search service rather than dataplane methods

* mark some appview v2 todos

* tidy

* still use dataplane on search endpoints when search service is not configured

* fix pds test

* fix up bsky tests & snaps

* tidy migrations

* fix appview-v2 docker build

* Support label issuer tied to appview v2 (#2033)

support label issuer tied to appview

* Appview v2: handle empty cursor on list notifications (#2017)

handle empty cursor on appview listnotifs

* Update appview v2 to use author feed enum (#2047)

* update bsky protos with author feed enum, misc feed item changes

* support new author feed enums in dataplane

* fix build

* Appview v2: utilize sorted-at field in bsky protos (#2050)

utilize new sorted-at field in bsky protos

* remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts

* Appview v2, sync w/ changes to protos (#2071)

* sync bsky protos

* sync-up bsky implementation w/ proto changes

* Appview v2 initial implementation for getPopularFeedGenerators (#2072)

add an initial implementation for getPopularFeedGenerators on appview v2

* merge

* fixes

* fix feed tests

* fix bsync mock

* format

* remove unused config

* fix lockfile

* another lockfile fix

* fix duplicate type

* fix dupplicate test

* Appview v2 handling clearly bad cursors (#2092)

* make mock dataplane cursors different from v1 cursors

* fail open on clearly bad appview cursors

* fix pds appview proxy snaps

* Appview v2 no notifs seen behavior (#2096)

* alter behavior for presenting notifications w/ no last-seen time

* fix pds proxy tests

* Appview v2 dataplane retries based on client host (#2098)

choose dataplane client for retries based on host when possible/relevant

* don't apply negated labels

* display suspensions on actor profile in appview v2

* Appview v2 use dataplane for identity lookups (#2095)

* update bsky proto w/ identity methods

* setup identity endpoints on mock dataplane

* move from idresolver to dataplane for identity lookups on appview

* tidy

* Appview v2: apply safe takedown refs to records, actors (#2107)

apply safe takedown refs to records, actors

* Fix timing on appview v2 repo rev header (#2113)

fix timing on appview repo rev

* fix post thread responses

* Appview v2 don't apply 3p self blocks (#2112)

do not apply 3p self-blocks

* Appview v2 search for feed generators (#2118)

* add protos for feedgen search

* support feed search on getPopularFeedGenerators

* Appview v2 config tidy (#2117)

* remove mod and triage roles from appview

* rename cdn and search config

* remove custom feed harness from appview v2

* Appview v2: don't apply missing modlists (#2122)

* dont apply missing mod lists

* update mock dataplane

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* refactor & document a bit better

* fix up other routes

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* Appview v2 enforce post thread root boundary (#2120)

* enforce post thread root boundary

* test thread root boundary

* Appview v2 fix admin environment variable (#2137)

fix admin env in appview v2

* Remove re-pagination from getSuggestions (#2145)

* remove re-pagination from getSuggestions

* fix test

* Adjust wording for account suspension (#2153)

adjust wording for account suspension

* Appview v2: fix not-found and blocked uris in threads (#2201)

* fix uris of not-found and blocked posts in threads

* update snaps

*  Show author feed of takendown author to admins only (#2197)

* fold in cid, auth, tracing, node version changes

* remove dead config from bsky service entrypoint

* build

* remove ozone test codepaths for appview v2

* tidy, docs fix

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-02-27 15:22:55 -05:00
Daniel Holmgren
de2dbc2903
Split out moderation backend (#1970)
* mv appview

* copy

* finalize copy

* package names

* big WIP

* first pass at mod servce

* some tidy

* tidy & fix compiler errors

* rename to ozone, db migrations, add to dev-env & pds cfg

* getRecord & getRepo mostly working

* fix open handle

* get record tests all working

* moderation events working

* statuses working

* tidy test suite

* search repos

* server & db tests

* moderation tests

* wip daemon + push events

* pds fanout working

* fix db test

* fanning takedowns out to appview

* rm try/catch

* bsky moderation test

* introduce mod subject wrappers

* more tidy

* refactor event reversal

* tidy some db stuff

* tidy

* rename service to mod-service

* fix test

* tidy config

* refactor auth in bsky

* wip patching up auto-mod

* add label ingester in appview

* fix a couple build issues

* fix some timing bugs

* tidy polling logic

* fix up tests

* fix some pds tests

* eslint ignore

* fix ozone tests

* move seeds to dev-env

* move images around

* fix db schemas

* use service auth admin reqs

* fix remaining tests

* auth tests bsky

* another test

* random tidy

* fix up search

* clean up bsky mod service

* more tidy

* default attempts to 0

* tidy old test

* random tidy

* tidy package.json

* tidy logger

* takedownId -> takedownRef

* misc pr feedback

* split daemon out from ozone application

* fix blob takedown mgiration

* refactor ozone config

* do push event fanout on write instead of on read

* make suspend error work again

* add attempts check & add supporting index

* fix takedown test ref

* get tests working

* rm old test

* fix timing bug in event pusher tests

* attempt another fix for timing bug

* await req

* service files

* remove labelerDid cfg

* update snaps for labeler did + some cfg changes

* fix more snaps

* pnpm i

* build ozone images

* build

* make label provider optional

* fix build issues

* fix build

* fix build

* build pds

* build on ghcr

* fix syntax in entry

* another fix

* use correct import

* export logger

* remove event reverser

* adjust push event fanout

* push out multiple

* remove builds
2024-01-05 17:06:54 -06:00
Daniel Holmgren
7edad62c12
Put canReply state on post viewer state instead of thread viewer state (#1882)
* switch canReply from thread to post view

* tweaks & fix up tests

* update snaps

* fix more snaps

* hydrate feed items for getPosts & searchPosts

* fix another snapshot

* getPosts test

* canReply -> blockedByGate & DRY up code

* blockedByGate -> excludedByGate

* replyDisabled
2023-11-27 20:14:20 -06:00
devin ivy
95d33f7b11
Support unauthed usage of feeds (#1884)
* update local feedgens to not require a viewer where possible

* update getFeed to use optional auth

* test feeds w/ optional auth
2023-11-27 15:30:09 -05:00
Daniel Holmgren
fcb19c9c51
Simplify PDS moderation (#1723)
* spec out new simple pds mod routes

* introduce new admin state endpoints

* wire up routes

* clean up pds

* revoke refresh tokens

* getUserAccountInfo

* pr tidy

* fixing some tests

* fixing up more tests

* fanout takedowns to pds

* fanout admin reqs to pds

* tidy

* more tidy & add more pds moderation tests

* getUserAccountInfo -> getAccountInfo

* dont hydrate pds info on searchRepos

* fix build

* port admin tests to bsky package

* clean up old snaps

* tests on fanout

* tweak naming

* missed a rename

* tidy renames

* fix lex name

* tidy & move snap

* fix build

* cleanup repeat process all

* skip actor search test

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* change takedownId col to takedownRef

* build branch

* fix bsky test

* add service key to indexer

* move signing key to api entry

* dont build
2023-10-30 16:56:17 -05:00
Daniel Holmgren
d664b51c64
Finalize PDS in-process AppView removal (#1198)
* rm tables

* rm event-stream & proxied

* Remove appview services, move label service to pds

* only proxy appview stuff

* delete more tables

* Start removing message dispatched from pds

* more syncing-up removal of message dispatcher in pds

* merged

* remove feedgens from pds, remove getPopular

* remove unused image helper from pds

* fixing compiler errors

* clean up sharp

* rm label service

* first pass on cleaning up tests

* fix up a bunch of tests

* moderation view tests

* last admin tests

* got a lil overzealous in deletes

* clean up unused cfg

* clean up label table

* simplify admin repo search query/logic

* tidy pds entrypoint

* in-progress pds config changes

* cfg fiddling

* finish cleaning up cfg/ctx

* comments

* building

* pds prefix on env

* test env

* collapse pds migrations down into a single migration

* fix up dev-env

* tidy

* cleanup

* fix pds admin tests

* fix handle test

* fix pds proxy tests

* fix subscribe repos test

* fix sqlite config in pds tests

* add sqlite clause in sequencer-leader

* fix actor search w/ sqlite on pds

* fixes

* fix dev env build

* update pds service entrypoint

* simple env example

* make takedown ids opaque identifiers in the pds

* use pds routes for api tests

* update pds dockerfile with volume and correct port env var

* add a couple env vars to example

* add comments to env example

* @atproto/pds 0.2.0-beta.0

* @atproto/aws 0.0.1-beta.0

* appview did

* @atproto/aws 0.0.1

* enable logs by default

* update env example

* bugfixing sandbox issues

* consistency in pds env var name for appview url

* log on pds start and stop, configure version at runtime

* @atproto/pds 0.2.0-beta.1

* fix semver matching for pds beta version

* v0.2.0-beta.2

* default invites to being not required

* fix flaky test

* limit db connections in tests

* publish 0.2.0-beta.d3

* fix invite required parsing

* @atproto/pds 0.2.0-beta.5

* Proxy getPopularFeedGenerators on simplified pds (#1222)

proxy getPopularFeedGenerators on pds

Co-authored-by: dholms <dtholmgren@gmail.com>

* tidy migrations

* fix service entry

* bump version

* change auth order

* bump version

* bump version

* add upgradeRepoVersion & fallback url for cdn

* bump version

* merging

* merge pds

* building dev-env

* merging tests

* merge service entry

* test fixing

* tidy

* fix admin search

* tidy

* tidy

* add snap for getListFeed

* add backup nameserver cfg

* tidy + pr feedback

* tidy

* tidy env

* bit more

* re-add dotenv to root package.json

* fix dep

* build branch

* fix tests

* Refactor tests to make better use of dev-env (#1690)

* refactor pds tests to use dev env

* refactor bsky tests

* fix pds test

* tidy bsky tests

* build pds correctly

* fix entry point

* default logging to false (for now)

* format service entry

* Switch takedown ids back to ints on pds distribution (#1694)

* switch takedown ids back to ints, consistent with live pds

* tidy/fix migration

* update migration for sqlite

* export moderation action reversal

* takedown tests

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-10-02 13:27:45 -05:00
Daniel Holmgren
9879ca97b7
Interaction Gating (#1561)
* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* initial lexicons for interaction-gating

* add interactions view to post views

* codegen

* model bad reply/interaction check state on posts

* initial impl for checking bad reply or interaction on write

* omit invalid interactions from post thread

* support not-found list in interaction view

* hydrate can-reply state on threads

* present interaction views on posts

* misc fixes, update snaps

* tidy/reorg

* tidy

* split interaction gating into separate record in lexicon

* switch interaction-gating impl to use separate record type

* allow checking reply gate w/ root post deletion

* fix

* initial gating tests

* tighten gated reply views, tests

* reply-gating list rule tests

* allow custom post rkeys within window

* hoist actors out of composeThread()

* tidy

* update thread gate lexicons, codegen

* lex fix

* rename gate to threadgate in bsky, update views

* lex fix

* improve terminology around reply validation

* fix down migration

* remove thread gates on actor unindexing

* add back .prettierignore

* tidy

* run ci on all prs

* syntax

* run ci on all prs

* format

* fix snap

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-09-14 15:24:48 -05:00
Eric Bailey
f84027fe8e
add getFeedSuggestions (#1542)
* add getFeedSuggestions lex

* move to feed namespace

* codegen

* add table and migration

* add endpoints

* remove comment

* add test

* fix order

* rename

* just remove proxy check
2023-08-31 17:19:34 -05:00
Eric Bailey
0ea353b21f add repost of reply, update snaps 2023-08-17 15:30:30 -05:00
Daniel Holmgren
244bf46e74
Include limited info on blocked embeds (#1463)
* add block info to embeds

* fix codegen

* Correctly handle blocked embeds and add block-other cause

* update snaps

* Correctly identify blocking behavior in embeds

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-08-10 15:54:24 -05:00
Paul Frazee
ab50816461
Self-labeling (#1444)
* Add self-label schemas

* switch around array and union for self-labels

* apply self-labels to post and profile views

* test for self labels in pds

* apply self-labels to post and profile views in bsky appview

* test for self labels in bsky appview

* update pds proxied test snapshots

* Add support for self-labels to the mdoeration sdk

* Disable unknown-labeler filtering until 3P support is added

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-09 19:06:29 -04:00
devin ivy
203e72f1c1
Appview preset image URLs (#1248)
* update image signing logic to presets on bsky

* update bsky config and views for preset img urls

* update bsky image tests for preset urls

* update bsky snapshots for preset image urls

* update pds proxy tests for bsky preset image urls

* tweak image paths
2023-08-09 10:21:43 -04:00
devin ivy
3da0324873
Post record language tagging, lexicon language format (#1219)
* Add languages field to post record

* helper for parsing bcp47

* add language format to lexicon

* codegen for post record langs field

* re-export language parsing in api package

* tests and tidy for lexicon language format

* index post langs, in-progress

* update snapshots, fixes record-with-media embed issue

* index post langs on bsky appview

* don't bother indexing post langs in pds appview, tidy
2023-06-23 16:23:16 -04:00
devin ivy
e651ed67a4
Respect takedowns on feedgen records, sync routes (#1119)
* Don't serve high-level details about taken-down feedgens

* Takedown checks on sync routes

* Use simpler check for takedowns on sync routes

* ensure takendown feedgens are handled in bsky appview
2023-06-21 15:07:16 -04:00
Daniel Holmgren
8c4fa530d6
Pds proxy to appview based on headers (#1191)
* pds proxy to appview based on headers

* reorg appview routes forwarding mutes to appview

* tidy

* clean up ctx

* track notif state

* use notif state for notif routes

* add back in missing file

* pr feedback

* wip

* more views

* more views

* update snaps & suggestions test

* better choices for like & reposted

* note

* better block test

* Test pds-proxied mute and notif state

* Update bsky snaps, remove unnecessary #postView

* Update bsky notification tests to use updateSeen

* add getList getFeedGen tests

* Re-disconnect pds from in-process appview indexing for tests

* clean up forSnapshot

* tidy

* feed gen view tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 17:29:32 -05:00
Daniel Holmgren
08fbb33fdf
Appview blocks (#1187)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* hook up indexing

* Setup custom feeds on bsky

* graph service + getList routes

* migration

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* Test custom algos on bsky

* add mute state to snapshots

* table & indexing plugin

* getBlocks

* bail out of mute subquery when no viewer

* fleshing out services

* Test pds getFeed proxy

* tidy, fix build

* tidy

* Fix minor race in bsky mutelist tests

* blocks in feed views

* db migration, impl block logic & tests

* Update pds proxy snaps based on appview mutes

* missed some files on last commit

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

* apply blocks in getFeed

* update snaps

* upadate one last snap

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 11:05:22 -05:00
devin ivy
7981c5948f
Misc. syncing-up bsky appview with on-pds appview (#1189)
* Misc notification and optimization in bsky, tidy

* Handle notifs on message queue in bsky appview

* Sync-up profile labeling behavior in bsky

* No async work during actor unindexing in bsky, include additional indexes

* Prevent deeply nested notifs w/ hellthread exception in bsky appview

* Add indexes to bsky appview for more efficient actor deletion/unindexing

* Update pds proxy snaps for updated bsky profile labeling

* Return custom suggestions in bsky appview
2023-06-11 14:55:43 -04:00
Daniel Holmgren
4f2f3d8d17
Appview mutes (#1184)
* hook up indexing

* graph service + getList routes

* migration

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* add mute state to snapshots

* bail out of mute subquery when no viewer

* Fix minor race in bsky mutelist tests

* Update pds proxy snaps based on appview mutes

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-10 17:12:15 -04:00
devin ivy
283fb1d8d6
Custom feeds on bsky appview (#1185)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* Setup custom feeds on bsky

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* Test custom algos on bsky

* Test pds getFeed proxy

* tidy, fix build

* tidy
2023-06-09 11:48:17 -04:00
devin ivy
f46d95617d
Feed gen indexing and view endpoints on bsky appview (#1182)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* tidy, fix build

* tidy/fix bsky migration
2023-06-09 10:30:54 -04:00
Daniel Holmgren
8059e07d8a
Custom feeds (#1001)
* actor type in profiles

* skeleton lex

* more schemas

* first pass at feed composition

* cleanup services

* more tidying

* fix up tests

* bookmarking schemas

* make feeds records instead of actors

* some cleanup

* fleshing out feeds as records

* wip

* working into cfg & dev-env

* fix appview test util

* bookmarking + views

* tidy

* add displayName

* db migration

* wip test

* blob processing

* first test working!

* Sync-up snapshots

* wip

* Fix pds ctx type

* Sync lexicons on pds and api

* wip

* fully refactor

* Prep for open union in reply data in bsky appview, sync-up snapshots

* msgs & ports

* Make pds did cache sqlite-friendly

* Start scaffolding more feedgen tests

* revamp appview tests

* revamp pds tests

* merge in dev env utils

* ensure plc close

* add feed gen to dev-env

* second feed gen

* named feeds in mock

* fix issue with dev-env pds appview

* update bsky snaps

* Flesh-out some feed gen tests

* Fix feed gen reply test

* Test feed bookmarking

* update skeleton defs

* merge

* Sync-up pds with updated feed skeleton contract

* Fix feed bookmark viewer state

* add feed embeds

* Test inaccurate feed gen repost

* Tighten some error behaviors around feed gens

* generator view in record embed

* test

* bookmarked/subscribed -> saved

* updated snaps

* name & descript on alices feed

* add like counts

* proposed lex

* tweak

* fetch feed generator details

* add describeFeedGen to dev-env

* adds tests for getFeedGenerator

* Update lexicons/app/bsky/feed/getFeedGenerator.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* update describe

* add cid to feed gen view

* add alice feed gen avatar

* generator displayName required

* update snaps

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* cleanup lex

* mount route

* Starter bsky algos (#1046)

* implement some starter bsky algos

* add team dids

* tidy

* add quotepost check to whats hot classic

* tests

* carve out feedgen section of pds

* apply limits

* add describeFeedGenerator route

* Whats hot new (#1060)

* map out new whats hot

* only available in postgres

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* build algos branch

* remove relative imports to @atproto/api

* update hot classic name

* rm build

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>

* v0.3.6

* v0.3.7

* v0.1.10

* update name of skyline

* build branch

* aws build

* getFeedGenerators test

* For now only use new replyRef union for custom feeds, for compat

* Don't use replyRef union on new feeds yet

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-05-19 18:31:51 -04:00
devin ivy
e57c297a82
Handling deleted content on admin/moderation methods (#1045)
* Update lexicon for missing repo/record data in admin views

* Serve missing repo/record variant in admin views

* Test actioning and presenting unknown records/repos, fix on pds
2023-05-17 15:28:52 -04:00
devin ivy
977202fa2b
Fix indexed record datetimes (#1009)
* Normalize record datetimes used for sort to simplified iso

* Update tests and snapshots for alternate valid datetime formats

* Fix notifications race during record update
2023-05-09 12:53:33 -04:00
devin ivy
eb7667e4fc
Add labels onto record embeds (#854)
* Add lexicons for labeling embeds

* Hydrate labels onto record embeds in pds

* Support labels on embeds in bsky appview
2023-04-25 17:19:40 -04:00
devin ivy
47890d65b9
Add labeler functionality to the appview (#851)
* Add moderation and labeling model to bsky appview

* Carry over moderation service from pds

* Pass over bsky moderation service to get it working, missing blob support

* Carry over label service from pds to bsky

* Reorg bsky http auth, implement admin auth

* Carry over relevant admin endpoints pds to bsky

* Retrofit bsky admin routes from pds impls

* Implement resolve handle on appview

* Avoid loop in bsky handle resolution

* Add bsky appview to dev-env, opt-in

* Fix bsky searchRepos with empty term, tidy

* Include blobs on bsky admin views

* Stop resolving taken-down blobs on appview

* Tidy

* Carry over pds moderation tests to bsky

* Support image cache invalidation on bsky appview

* Add missing changes for bsky tests

* Test takedowns in bsky views

* Test takedowns on bsky notifs

* Separate blob resolution logic into standalone helper

* Carry over labeler from pds to bsky

* Update labelers for bsky usage

* Configure labeler on appview, tidy blob-resolver

* Add labels to views for posts and actors

* Update snapshots on bsky appview to include labels

* Carry over labeler tests from pds to bsky

* Tidy-up labeler tests for bsky appview

* Update bsky seeds and snapshots for keyword labeling

* Tidy

* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.

* Include labels on appview listNotifications, fix blob resolver content-type, tidy
2023-04-23 23:58:10 -04:00
devin ivy
05e6ebed63
Bsky appview admin and moderation endpoints (#840)
* Add moderation and labeling model to bsky appview

* Carry over moderation service from pds

* Pass over bsky moderation service to get it working, missing blob support

* Carry over label service from pds to bsky

* Reorg bsky http auth, implement admin auth

* Carry over relevant admin endpoints pds to bsky

* Retrofit bsky admin routes from pds impls

* Implement resolve handle on appview

* Avoid loop in bsky handle resolution

* Add bsky appview to dev-env, opt-in

* Fix bsky searchRepos with empty term, tidy

* Include blobs on bsky admin views

* Stop resolving taken-down blobs on appview

* Tidy

* Carry over pds moderation tests to bsky

* Support image cache invalidation on bsky appview

* Add missing changes for bsky tests

* Test takedowns in bsky views

* Test takedowns on bsky notifs

* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.
2023-04-23 23:43:08 -04:00
devin ivy
c70cb2b6e5
Notifications from the appview (#829)
* Update notif lexicons for stateless seenAt param

* In-progress work on appview notifs

* Fix-up bsky notification methods

* Add appview notification table table

* Process notifications in bsky appview, test notif indexing

* Test bsky appview notification methods

* Tidy bsky notification tests

* Explicitly don't support seenAt notif params on pds

* Tidy bsky notifs tests

* Sync bsky notif handling with pds

* Remove stale comment

* Switch bsky notifs tests to use testenv
2023-04-18 17:55:03 -04:00
devin ivy
4e83748545
Bsky app view (#716)
* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect

* Cull bsky config, services, auth, etc.

* Sweep app view xrpc methods, tidy deps, add storage back for img server

* Run repo subscription on bsky app view

* Collapse db migrations down for bsky app view

* Tidy app view bin

* Remove mute functionality from app view, delegate to pds

* Initial tidy/culling of bsky app view tests

* Passing bsky app view db, server, and repo subscription tests

* Passing bsky app view duplicate-records tests

* Bsky app view test tidy/cull

* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.

* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.

* Setup actor handles by did in bsky app view

* Passing indexing tests on bsky app view

* Passing image tests on bsky app view

* Fix bsky actor reindexing, support custom lock id for testing repo subs

* Sweep bsky view tests, misc tests, passing

* Tidy bsky deps

* Include did in resized image uris

* Update bsky image process server to use getBlob

* Update image server tests, misc fixes

* Implement bsky blob resolver

* Wire local image processing server to local blob resolver, test blob resolver

* Tidy

* Tidy

* Tidy

* Tidy app view init

* Fix handle resolution, tidy

* Add utils for partitioning indexing by did

* Update repo sub to parallelize work per repo

* Dep tidy

* Tidy bsky tests for updated repo sub destroy()

* Update thead indexing to handle out-of-order posts

* Sketch out strategy in bsky for handling too-big commits

* Set content-type on sync.getBlob

* Add logging for failed transmissions in bsky blob resolver

* Tidy

* Tidy bsky repo indexing and supporting repo interfaces

* Sort in app view based on combo of creation and indexing times

* Fix types

* Add retry utils to bsky

* Add retries to http requests made by bsky

* Test repo indexing

* Update bsky db/model for lex refactor

* Update bsky lexicons for lex refactor

* Update bsky actor service for lex refactor

* Update bsky feed service for lex refactor

* Update bsky indexing service for lex refactor

* Update bsky repo subscription for lex refactor

* Tidy bsky repo sub

* Add unspecced endpoints to bsky app view, update entrypoint

* Update bsky xrpc utils for lex refactor

* Update bsky xrpc methods for lex refactor

* Update bsky test seeds for lex refactor, tidy api entrypoint

* Update bsky non-view tests for lex refactor

* Update bsky likes view test for lex refactor, minor fix

* Update bsky author feed tests for lex refactor, minor test util fix

* Update bsky follow, profile, repost, search view tests for lex refactor

* Update bsky timeline view tests for lex refactor

* Replace bsky out-of-order thread indexing logic

* Update bsky thread view tests for lex refactor, general test tidying

* Handle rebases and too-big commits in repo subscription, tracking commit data cid

* Tidy

* Ensure did resolver reports "not found" only when positively not found

* Handle tombstones and handle updates in bsky

* Test indexing handle updates and did tombstones

* Support cors on bsky

* Allow app view to serve most routes unauthed

* Tests for bsky unauthed views

* Tidy bsky service entrypoint and dockerfile

* Remove unused storage interfaces from bsky

* Bsky entrypoint and dockerfile fixes, tidy

* Add workflow for bsky build to aws

* Use more standard db env variables, make migration creds optional

* Make bsky repo subscription optional

* Fix lex->json serialization in bsky

* Split bsky actor sync state into its own table

* Skip invalid records on indexing full repo, tidy

* Tidy

* Leader test timing

* Tidy/lint

* Fix bsky config overrides
2023-04-06 18:47:46 -04:00