13 Commits

Author SHA1 Message Date
Matthieu Sieben
72eba67af1
Drop axios dependency (#3177)
* Minor adaptation of VerifyCidTransform implementation

* refactor: factorize content-encoding negotiation into new lib

* bsky: Use undici to stream blob

* fixup! bsky: Use undici to stream blob

* disable ssrf bsky protection in dev-env

* remove http requests to self to host "/img/"

* drop axios from tests

* fixes

* fix tests

* reviex changes

* properly handle HEAD requests

* handle client disconnection

* fix tests

* drop unrelated change

* tidy

* tidy

* tidy

* remove axios from dev-env

* remove axios from identity package

* use undici 6

* remove axios dependency from ozone

* tidy

* remove axios from PDS package

* avoid killing bsky-pds connections

* improve debugging data

* Better handle invalid CID

* tidy

* tidy

* refactor "allFulfilled" util in @atproto/common

* tidy

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2025-01-06 18:34:11 +01:00
Daniel Holmgren
50c0ec176c
Service auth method binding (lxm) (#2663)
* add scopes to service auth impl

* add error to getServiceAuth

* send scoped tokens from pds

* clean up privileged access scopes & allow simple service auth tokens for app passwords

* integration into ozone

* fix up bsky tests

* cleanup xrpc-server tests

* fix up tests & types

* one more test

* fix read after write tests

* fix mod auth test

* convert scopes to be a single method name

* add scope check callback for auth verifier

* pds changes only

* fix feed generation tests

* use scope for ozone service profile

* dont verify scopes on pds yet

* tidy

* tidy imports

* changeset

* add tests

* tidy

* another changeset

* scope -> lxm

* tidy

* clean up scope references

* update nonce size

* pr feedback

* trim trailing slash

* nonce -> jti

* fix xrpc-server test

* allow service auth on uploadBlob

* fix build error

* changeset

* build, tidy

* xrpc-server: update lxm claim check error

* appview: temporarily permit labeler service calls to omit lxm claim

* xrpc-server: fix test

* changeset

* fix merged tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-08-18 15:46:07 -04:00
Daniel Holmgren
f65de89eed
Feature: Appview v2 (#1924)
* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* filler commit

* rm filler

* server boilerplate

* follows impl

* posts impl

* posts & likes impl

* repost impl

* profiles & handle null values

* list impl

* mutes impl

* blocks impl

* misc

* feed gen impl

* label impl

* notifs impl

* feeds impl

* threads impl

* early sketchwork

* wip

* stub out thick client

* in-progress work on hydrator

* tweak

* hydrate profile labels, detail lists

* feedgen hydration

* protobuf tweaks

* more protobuf tweaks

* wip

* snake case

* moar snake case

* tidy actor hydration

* tidy parsing

* type fixes, renaming, comments in hydrator

* hydrate list items and likes

* hydrate notifications

* feed hydration

* graph & label hydration

* more record protobufs

* pluralize

* tweak pbs

* use new methods

* Setup dataplane grpc client/mock server (#1921)

* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* add record getter mocks

* post view hydration

* fix up mock dataplane to match new protos

* missed one

* wire up dataplane in ctx & dev-env

* adding some basic views

* feed hydration, add blocks to post hydration

* pass over notification hydration

* tidy

* merge

* implement getProfile

* hydrate post aggregation and viewer state

* fix

* fix codegen

* get some tests passing!

* add takedowns & some like bugfixing

* all profile tests passing!

* likes test

* follow endpoints using data plane

* reorg follow block rules

* reposts

* post views!

* implement getList w/ dataplane caveat

* adjust dataplane getListMembers to return listitem uris

* implement getListMutes and -Blocks w/ dataplane

* suggestions

* timeline

* misc view fixes

* view fixes for mutes, self-mute/block

* author feed

* feed gen routes

* tidy

* misc block/mute fixes

* list feed & actor likes

* implement getLists, fix some empty cursors

* implement getMutes, empty profile description fix

* implement getBlocks, block application fix

* implement getSuggestedFollowsByActor, needs some fixes

* feed generation

* search routes

* threads

* tidy

* fix some snaps

* fix getSuggestedFollowsByActor

* implement listNotifications

* implement getUnreadCount w/ dataplane

* implement notifications.updateSeen w/ dataplane

* 3rd party blocking tests

* blocked profile viewer

* add search mocks

* refactor getFeed

* createPipeline -> createPipelineNew

* basic replygating functionality on dataplane w/o filtering violating replies

* hack threadgates into dataplane, apply gates

* deterministic thread orders in dataplane

* misc cleanup around dataplane

* upgrade typescript to v5.3

* update typescript linter deps

* sync bsky proto, codegen

* update dataplane, sync with bsky proto updates

* remove indexer, ingester, daemon, moderation services from appview

* convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane

* remove bsky services. db and indexing logic into mock dataplane.

* remove tests not needed by appview v2, misc reorg

* add basic in-mem repo subscription to dataplane mock

* fix dev-env, bsky tests, bsky build

* cull bsky service entrypoint

* add bsky service readme

* build

* tidy

* tidy, fix pds proxy tests

* fix

* fix bsky entrypoint deps

* support http2 grpc client

* build

* fix dataplane bad tls config/default

* support multiple dataplane urls, retry when unavailable

* build

* tidy/fix

* move dataplane mock tests into their own dir

* cover label hydration through timeline test

* bring back labels in appview tests

* remove unused db primary/replica/coordinator from bsky dataplane

* bsky proto add cids to contracts, buf codegen

* sync-up bsky data-plane w/ codegen updates

* start using dataplane interaction endpoints

* add file

* avoid overfetching from dataplane, plumb feed items and cids

* pass refs through for post viewer state

* switch list feeds to use feed item in dataplane

* handle not found err on get-thread dataplane call

* support use of search service rather than dataplane methods

* mark some appview v2 todos

* tidy

* still use dataplane on search endpoints when search service is not configured

* fix pds test

* fix up bsky tests & snaps

* tidy migrations

* fix appview-v2 docker build

* Support label issuer tied to appview v2 (#2033)

support label issuer tied to appview

* Appview v2: handle empty cursor on list notifications (#2017)

handle empty cursor on appview listnotifs

* Update appview v2 to use author feed enum (#2047)

* update bsky protos with author feed enum, misc feed item changes

* support new author feed enums in dataplane

* fix build

* Appview v2: utilize sorted-at field in bsky protos (#2050)

utilize new sorted-at field in bsky protos

* remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts

* Appview v2, sync w/ changes to protos (#2071)

* sync bsky protos

* sync-up bsky implementation w/ proto changes

* Appview v2 initial implementation for getPopularFeedGenerators (#2072)

add an initial implementation for getPopularFeedGenerators on appview v2

* merge

* fixes

* fix feed tests

* fix bsync mock

* format

* remove unused config

* fix lockfile

* another lockfile fix

* fix duplicate type

* fix dupplicate test

* Appview v2 handling clearly bad cursors (#2092)

* make mock dataplane cursors different from v1 cursors

* fail open on clearly bad appview cursors

* fix pds appview proxy snaps

* Appview v2 no notifs seen behavior (#2096)

* alter behavior for presenting notifications w/ no last-seen time

* fix pds proxy tests

* Appview v2 dataplane retries based on client host (#2098)

choose dataplane client for retries based on host when possible/relevant

* don't apply negated labels

* display suspensions on actor profile in appview v2

* Appview v2 use dataplane for identity lookups (#2095)

* update bsky proto w/ identity methods

* setup identity endpoints on mock dataplane

* move from idresolver to dataplane for identity lookups on appview

* tidy

* Appview v2: apply safe takedown refs to records, actors (#2107)

apply safe takedown refs to records, actors

* Fix timing on appview v2 repo rev header (#2113)

fix timing on appview repo rev

* fix post thread responses

* Appview v2 don't apply 3p self blocks (#2112)

do not apply 3p self-blocks

* Appview v2 search for feed generators (#2118)

* add protos for feedgen search

* support feed search on getPopularFeedGenerators

* Appview v2 config tidy (#2117)

* remove mod and triage roles from appview

* rename cdn and search config

* remove custom feed harness from appview v2

* Appview v2: don't apply missing modlists (#2122)

* dont apply missing mod lists

* update mock dataplane

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* refactor & document a bit better

* fix up other routes

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* Appview v2 enforce post thread root boundary (#2120)

* enforce post thread root boundary

* test thread root boundary

* Appview v2 fix admin environment variable (#2137)

fix admin env in appview v2

* Remove re-pagination from getSuggestions (#2145)

* remove re-pagination from getSuggestions

* fix test

* Adjust wording for account suspension (#2153)

adjust wording for account suspension

* Appview v2: fix not-found and blocked uris in threads (#2201)

* fix uris of not-found and blocked posts in threads

* update snaps

*  Show author feed of takendown author to admins only (#2197)

* fold in cid, auth, tracing, node version changes

* remove dead config from bsky service entrypoint

* build

* remove ozone test codepaths for appview v2

* tidy, docs fix

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-02-27 15:22:55 -05:00
Daniel Holmgren
de2dbc2903
Split out moderation backend (#1970)
* mv appview

* copy

* finalize copy

* package names

* big WIP

* first pass at mod servce

* some tidy

* tidy & fix compiler errors

* rename to ozone, db migrations, add to dev-env & pds cfg

* getRecord & getRepo mostly working

* fix open handle

* get record tests all working

* moderation events working

* statuses working

* tidy test suite

* search repos

* server & db tests

* moderation tests

* wip daemon + push events

* pds fanout working

* fix db test

* fanning takedowns out to appview

* rm try/catch

* bsky moderation test

* introduce mod subject wrappers

* more tidy

* refactor event reversal

* tidy some db stuff

* tidy

* rename service to mod-service

* fix test

* tidy config

* refactor auth in bsky

* wip patching up auto-mod

* add label ingester in appview

* fix a couple build issues

* fix some timing bugs

* tidy polling logic

* fix up tests

* fix some pds tests

* eslint ignore

* fix ozone tests

* move seeds to dev-env

* move images around

* fix db schemas

* use service auth admin reqs

* fix remaining tests

* auth tests bsky

* another test

* random tidy

* fix up search

* clean up bsky mod service

* more tidy

* default attempts to 0

* tidy old test

* random tidy

* tidy package.json

* tidy logger

* takedownId -> takedownRef

* misc pr feedback

* split daemon out from ozone application

* fix blob takedown mgiration

* refactor ozone config

* do push event fanout on write instead of on read

* make suspend error work again

* add attempts check & add supporting index

* fix takedown test ref

* get tests working

* rm old test

* fix timing bug in event pusher tests

* attempt another fix for timing bug

* await req

* service files

* remove labelerDid cfg

* update snaps for labeler did + some cfg changes

* fix more snaps

* pnpm i

* build ozone images

* build

* make label provider optional

* fix build issues

* fix build

* fix build

* build pds

* build on ghcr

* fix syntax in entry

* another fix

* use correct import

* export logger

* remove event reverser

* adjust push event fanout

* push out multiple

* remove builds
2024-01-05 17:06:54 -06:00
Daniel Holmgren
d664b51c64
Finalize PDS in-process AppView removal (#1198)
* rm tables

* rm event-stream & proxied

* Remove appview services, move label service to pds

* only proxy appview stuff

* delete more tables

* Start removing message dispatched from pds

* more syncing-up removal of message dispatcher in pds

* merged

* remove feedgens from pds, remove getPopular

* remove unused image helper from pds

* fixing compiler errors

* clean up sharp

* rm label service

* first pass on cleaning up tests

* fix up a bunch of tests

* moderation view tests

* last admin tests

* got a lil overzealous in deletes

* clean up unused cfg

* clean up label table

* simplify admin repo search query/logic

* tidy pds entrypoint

* in-progress pds config changes

* cfg fiddling

* finish cleaning up cfg/ctx

* comments

* building

* pds prefix on env

* test env

* collapse pds migrations down into a single migration

* fix up dev-env

* tidy

* cleanup

* fix pds admin tests

* fix handle test

* fix pds proxy tests

* fix subscribe repos test

* fix sqlite config in pds tests

* add sqlite clause in sequencer-leader

* fix actor search w/ sqlite on pds

* fixes

* fix dev env build

* update pds service entrypoint

* simple env example

* make takedown ids opaque identifiers in the pds

* use pds routes for api tests

* update pds dockerfile with volume and correct port env var

* add a couple env vars to example

* add comments to env example

* @atproto/pds 0.2.0-beta.0

* @atproto/aws 0.0.1-beta.0

* appview did

* @atproto/aws 0.0.1

* enable logs by default

* update env example

* bugfixing sandbox issues

* consistency in pds env var name for appview url

* log on pds start and stop, configure version at runtime

* @atproto/pds 0.2.0-beta.1

* fix semver matching for pds beta version

* v0.2.0-beta.2

* default invites to being not required

* fix flaky test

* limit db connections in tests

* publish 0.2.0-beta.d3

* fix invite required parsing

* @atproto/pds 0.2.0-beta.5

* Proxy getPopularFeedGenerators on simplified pds (#1222)

proxy getPopularFeedGenerators on pds

Co-authored-by: dholms <dtholmgren@gmail.com>

* tidy migrations

* fix service entry

* bump version

* change auth order

* bump version

* bump version

* add upgradeRepoVersion & fallback url for cdn

* bump version

* merging

* merge pds

* building dev-env

* merging tests

* merge service entry

* test fixing

* tidy

* fix admin search

* tidy

* tidy

* add snap for getListFeed

* add backup nameserver cfg

* tidy + pr feedback

* tidy

* tidy env

* bit more

* re-add dotenv to root package.json

* fix dep

* build branch

* fix tests

* Refactor tests to make better use of dev-env (#1690)

* refactor pds tests to use dev env

* refactor bsky tests

* fix pds test

* tidy bsky tests

* build pds correctly

* fix entry point

* default logging to false (for now)

* format service entry

* Switch takedown ids back to ints on pds distribution (#1694)

* switch takedown ids back to ints, consistent with live pds

* tidy/fix migration

* update migration for sqlite

* export moderation action reversal

* takedown tests

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-10-02 13:27:45 -05:00
Daniel Holmgren
73f64009a2
Appview database replicas (#1459)
* setup bsky so that writes always go to a db primary

* test bsky primary db functionality

* sprinkle db primary over moderation and view maintainer usage

* make bsky tests more strict on primary/secondary db, fixes

* maintain prev db config on appview, require primary db, make primary/replica explicit in config

* tidy

* sketch out db replicas

* wip

* support multiple tags per bsky replica, "any" tag, fallbacks and warns

* clarify tags

* use new coordinator api across routes & services

* wire-up replica tagging config

* tidy

* cleanup entroypoints and tests for db coordinator

* fix pds test

* re-hookup migrate db

* standardize env vars for db on bsky services

* tidy

* build

* fix bav api entrypoint

* support DB_POSTGRES_URL on ingester/indexer for backwards compat

* remove build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-14 18:38:00 -04:00
devin ivy
30b41029c1
Partitioning in bsky indexer (#1368)
* setup redis infra for appview indexer

* barebones bsky ingester

* add ioredis to bsky

* remove some indexer functionality from bsky api

* setup for bsky indexer

* tidy

* tidy, observe basic pipeline functioning

* process messages on bsky indexer pipeline, tidy tests and lifecycle

* trim partitions when moving cursor

* simplify config for partitions

* misc fixes for redis setup in bsky tests, add namespacing

* fix pds proxy tests

* remove cursor state from indexer partitions, simplify ingester state

* tidy

* utils for testing w/ multiple indexers, fix off-by-one xtrim

* test reingesting

* test indexer repartitioning

* add entrypoints for bsky ingester and indexer, fix db schema config, api entrypoint name, tidy

* setup and test bsky ingester backpressure, add config

* tidy

* add missing test file

* tidy redis calls, add redis sentinel config

* tidy/test some utils used in bsky pipeline

* tidy bsky pipeline tests, move helpers into dev-env

* fix pds crud test

* support redis host and password config

* better loggin/observability in ingester and indexer, make build

* add bsky ingester initial cursor config

* temporarily remove migrations from indexer/ingester

* allow ingester to batch

* packages/pg becomes packages/dev-infra with some cleanup (#1402)

* packages/dev-infra/

* Extract packages/dev-infra/_common.sh and use it

* Use period instead of source because of /bin/sh

* add docs for redis test script

* fix repartition test

* add logs to debug ci

* simplify repartitioning test, remove ci logs

---------

Co-authored-by: Jerry Chen <jerry@redelm.net>
2023-08-01 14:59:00 -04:00
Daniel Holmgren
9263ddc64d
Add compression in http res (#1022)
* add compression

* compress carfiles

* update package version

* add tests

* one more test

* remove compressible dep

* add gzip to appview

* fix package.json
2023-06-29 10:24:48 -05:00
Daniel Holmgren
47d30c1796
Revamp dev env (#1031)
* wip

* wip

* fully refactor

* msgs & ports

* revamp appview tests

* revamp pds tests

* ensure plc close

* ensure pds appview is hooked up when not running appview

* move service setup to dev env

* move admin auth headers to dev-env

* fix service headers

* missing port

* fixing ports

* start services in order

* startup plc immediately

* rm dead code

* fix up build

* rm log
2023-05-15 12:28:36 -05:00
Daniel Holmgren
68004c617f
Appview proxy (#827)
* fix up a couple of tsc errors in app view merge

* wip

* simple proxy

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* wip

* copy proxied tests & add mutes to getFollows/getFollowers

* tidy & add mutes to likes

* more routes + getAuthorFeed tests

* more testing

* tests for feed views

* thread testing

* finished tests for threads

* temporarily skip some tests

* cleaning up & updating test names

* separate db schema for appview

* rearrange

* typo

* add notifications

* re-enable notifs on pds appview

* update schemas

* updated some bsky snaps

* wip

* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* fix up & move to appCtx

* neat its working

* update bsky tests to new auth

* rm unused pds config var

* tidy

* check exp in seconds

* cache dids in postgres

* add migration & did-cache

* start tests

* couple helpers around cache invalidation

* fix expired check

* wip

* change cache semantics

* did cache testing

* do some cache revalidation in indexing

* fix config

* fix issue w did-resolver test-env prototype

* use map instead of record

* stale + expired

* tests

* clear entry method

* fix up build

* expired dids

* clear missing dids

* better verify payload

* bump test timeout

* fix notifs test

* fix up proxied actor search tests

* update snaps to include labels

* fix dev env

* fix up moderation route auth

* fix more auth headers

* fix auth on getPosts

* increase jest timeouts

* fix snaps
2023-04-25 15:26:14 -05:00
Daniel Holmgren
d4963e5003
Revamp dev env (#796)
* fix up a couple of tsc errors in app view merge

* wip

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* fixed missed merge conflict

* fix one more merge conflict

* fix popular test
2023-04-18 11:07:41 -05:00
Daniel Holmgren
8c71d0fe37
Cleanup appview merge (#772)
fix up a couple of tsc errors in app view merge
2023-04-07 10:44:00 -05:00
devin ivy
4e83748545
Bsky app view (#716)
* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect

* Cull bsky config, services, auth, etc.

* Sweep app view xrpc methods, tidy deps, add storage back for img server

* Run repo subscription on bsky app view

* Collapse db migrations down for bsky app view

* Tidy app view bin

* Remove mute functionality from app view, delegate to pds

* Initial tidy/culling of bsky app view tests

* Passing bsky app view db, server, and repo subscription tests

* Passing bsky app view duplicate-records tests

* Bsky app view test tidy/cull

* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.

* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.

* Setup actor handles by did in bsky app view

* Passing indexing tests on bsky app view

* Passing image tests on bsky app view

* Fix bsky actor reindexing, support custom lock id for testing repo subs

* Sweep bsky view tests, misc tests, passing

* Tidy bsky deps

* Include did in resized image uris

* Update bsky image process server to use getBlob

* Update image server tests, misc fixes

* Implement bsky blob resolver

* Wire local image processing server to local blob resolver, test blob resolver

* Tidy

* Tidy

* Tidy

* Tidy app view init

* Fix handle resolution, tidy

* Add utils for partitioning indexing by did

* Update repo sub to parallelize work per repo

* Dep tidy

* Tidy bsky tests for updated repo sub destroy()

* Update thead indexing to handle out-of-order posts

* Sketch out strategy in bsky for handling too-big commits

* Set content-type on sync.getBlob

* Add logging for failed transmissions in bsky blob resolver

* Tidy

* Tidy bsky repo indexing and supporting repo interfaces

* Sort in app view based on combo of creation and indexing times

* Fix types

* Add retry utils to bsky

* Add retries to http requests made by bsky

* Test repo indexing

* Update bsky db/model for lex refactor

* Update bsky lexicons for lex refactor

* Update bsky actor service for lex refactor

* Update bsky feed service for lex refactor

* Update bsky indexing service for lex refactor

* Update bsky repo subscription for lex refactor

* Tidy bsky repo sub

* Add unspecced endpoints to bsky app view, update entrypoint

* Update bsky xrpc utils for lex refactor

* Update bsky xrpc methods for lex refactor

* Update bsky test seeds for lex refactor, tidy api entrypoint

* Update bsky non-view tests for lex refactor

* Update bsky likes view test for lex refactor, minor fix

* Update bsky author feed tests for lex refactor, minor test util fix

* Update bsky follow, profile, repost, search view tests for lex refactor

* Update bsky timeline view tests for lex refactor

* Replace bsky out-of-order thread indexing logic

* Update bsky thread view tests for lex refactor, general test tidying

* Handle rebases and too-big commits in repo subscription, tracking commit data cid

* Tidy

* Ensure did resolver reports "not found" only when positively not found

* Handle tombstones and handle updates in bsky

* Test indexing handle updates and did tombstones

* Support cors on bsky

* Allow app view to serve most routes unauthed

* Tests for bsky unauthed views

* Tidy bsky service entrypoint and dockerfile

* Remove unused storage interfaces from bsky

* Bsky entrypoint and dockerfile fixes, tidy

* Add workflow for bsky build to aws

* Use more standard db env variables, make migration creds optional

* Make bsky repo subscription optional

* Fix lex->json serialization in bsky

* Split bsky actor sync state into its own table

* Skip invalid records on indexing full repo, tidy

* Tidy

* Leader test timing

* Tidy/lint

* Fix bsky config overrides
2023-04-06 18:47:46 -04:00