10 Commits
Author | SHA1 | Message | Date | |
---|---|---|---|---|
Matthieu Sieben
|
b934b396b1
|
Client SDK rework (#2483)
* feat(api): support creation of oauth based AtpAgents * oauth: misc fixes for confidential clients * fix(xprc): remove ReadableStream.from polyfill * OAuth docs tweaks (#2679) * OAuth: clarification about client_name being shown * OAuth: re-write handle resolution privacy concern * avoid relying on ReadableStream.from in xrpc-server tests * feat(oauth-types): expose "ALLOW_UNSECURE_ORIGINS" constant * feat(handle-resolver): expose "AtprotoIdentityDidMethods" type * fix(oauth-client): ensure that the oauth metadata document contains client_id_metadata_document_supported * fix(oauth-types): prevent unknown query string in loopback client id * fix(identity-resolver): check that handle is in did doc's "alsoKnownAs" * feat(oauth-client:oauth-resolver): allow logging in using either the PDS URL or Entryway URL * fix(oauth-client): return better error in case of invalid "oauth-protected-resource" status code * refactor(did): group atproto specific checks in own * feat(api): relax typing of "appLabelers" and "labelers" AtpClient properties * allow any did as labeller (for tests mainly) * fix(api): allow to override "atproto-proxy" on a per-request basis * remove release candidate versions from changelog * update changeset for api and xrpc packages * Add missing changeset * revert RC versions * Proper wording in OAUTH.md api example * remove "pre" changeset file * xrpc: restore original behavior of setHEader and unsetHeader * docs: add comment for XrpcClient 's constructor arg * feat(api): expose "schemas" publicly * feat(api): allow customizing the whatwg fetch function of the AtpAgent * docs(api): improve migration docs * docs: change reference to BskyAgent to AtpAgent * docs: mention the breaking change regarding setSessionPersistHandler * fix(api): better split AtpClient concerns * fix(xrpc): remove unused import * refactor(api): simplify class hierarchu by removeing AtpClient * fix(api): mock proper method for facets detection * restore ability to restore session asynchronously * feat(api): allow instantiating Agent with same argument as super class * docs(api): properly extend Agent class * style(xrpc): var name * docs(api): remove "async" to header getter --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> Co-authored-by: bnewbold <bnewbold@robocracy.org> Co-authored-by: Hailey <me@haileyok.com> |
||
Daniel Holmgren
|
f65de89eed
|
Feature: Appview v2 (#1924)
* add buf & connectrpc, codegen client * lint * prettier ignore * fix prettier ignore * tidy & add tests * filler commit * rm filler * server boilerplate * follows impl * posts impl * posts & likes impl * repost impl * profiles & handle null values * list impl * mutes impl * blocks impl * misc * feed gen impl * label impl * notifs impl * feeds impl * threads impl * early sketchwork * wip * stub out thick client * in-progress work on hydrator * tweak * hydrate profile labels, detail lists * feedgen hydration * protobuf tweaks * more protobuf tweaks * wip * snake case * moar snake case * tidy actor hydration * tidy parsing * type fixes, renaming, comments in hydrator * hydrate list items and likes * hydrate notifications * feed hydration * graph & label hydration * more record protobufs * pluralize * tweak pbs * use new methods * Setup dataplane grpc client/mock server (#1921) * add buf & connectrpc, codegen client * lint * prettier ignore * fix prettier ignore * tidy & add tests * add record getter mocks * post view hydration * fix up mock dataplane to match new protos * missed one * wire up dataplane in ctx & dev-env * adding some basic views * feed hydration, add blocks to post hydration * pass over notification hydration * tidy * merge * implement getProfile * hydrate post aggregation and viewer state * fix * fix codegen * get some tests passing! * add takedowns & some like bugfixing * all profile tests passing! * likes test * follow endpoints using data plane * reorg follow block rules * reposts * post views! * implement getList w/ dataplane caveat * adjust dataplane getListMembers to return listitem uris * implement getListMutes and -Blocks w/ dataplane * suggestions * timeline * misc view fixes * view fixes for mutes, self-mute/block * author feed * feed gen routes * tidy * misc block/mute fixes * list feed & actor likes * implement getLists, fix some empty cursors * implement getMutes, empty profile description fix * implement getBlocks, block application fix * implement getSuggestedFollowsByActor, needs some fixes * feed generation * search routes * threads * tidy * fix some snaps * fix getSuggestedFollowsByActor * implement listNotifications * implement getUnreadCount w/ dataplane * implement notifications.updateSeen w/ dataplane * 3rd party blocking tests * blocked profile viewer * add search mocks * refactor getFeed * createPipeline -> createPipelineNew * basic replygating functionality on dataplane w/o filtering violating replies * hack threadgates into dataplane, apply gates * deterministic thread orders in dataplane * misc cleanup around dataplane * upgrade typescript to v5.3 * update typescript linter deps * sync bsky proto, codegen * update dataplane, sync with bsky proto updates * remove indexer, ingester, daemon, moderation services from appview * convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane * remove bsky services. db and indexing logic into mock dataplane. * remove tests not needed by appview v2, misc reorg * add basic in-mem repo subscription to dataplane mock * fix dev-env, bsky tests, bsky build * cull bsky service entrypoint * add bsky service readme * build * tidy * tidy, fix pds proxy tests * fix * fix bsky entrypoint deps * support http2 grpc client * build * fix dataplane bad tls config/default * support multiple dataplane urls, retry when unavailable * build * tidy/fix * move dataplane mock tests into their own dir * cover label hydration through timeline test * bring back labels in appview tests * remove unused db primary/replica/coordinator from bsky dataplane * bsky proto add cids to contracts, buf codegen * sync-up bsky data-plane w/ codegen updates * start using dataplane interaction endpoints * add file * avoid overfetching from dataplane, plumb feed items and cids * pass refs through for post viewer state * switch list feeds to use feed item in dataplane * handle not found err on get-thread dataplane call * support use of search service rather than dataplane methods * mark some appview v2 todos * tidy * still use dataplane on search endpoints when search service is not configured * fix pds test * fix up bsky tests & snaps * tidy migrations * fix appview-v2 docker build * Support label issuer tied to appview v2 (#2033) support label issuer tied to appview * Appview v2: handle empty cursor on list notifications (#2017) handle empty cursor on appview listnotifs * Update appview v2 to use author feed enum (#2047) * update bsky protos with author feed enum, misc feed item changes * support new author feed enums in dataplane * fix build * Appview v2: utilize sorted-at field in bsky protos (#2050) utilize new sorted-at field in bsky protos * remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts * Appview v2, sync w/ changes to protos (#2071) * sync bsky protos * sync-up bsky implementation w/ proto changes * Appview v2 initial implementation for getPopularFeedGenerators (#2072) add an initial implementation for getPopularFeedGenerators on appview v2 * merge * fixes * fix feed tests * fix bsync mock * format * remove unused config * fix lockfile * another lockfile fix * fix duplicate type * fix dupplicate test * Appview v2 handling clearly bad cursors (#2092) * make mock dataplane cursors different from v1 cursors * fail open on clearly bad appview cursors * fix pds appview proxy snaps * Appview v2 no notifs seen behavior (#2096) * alter behavior for presenting notifications w/ no last-seen time * fix pds proxy tests * Appview v2 dataplane retries based on client host (#2098) choose dataplane client for retries based on host when possible/relevant * don't apply negated labels * display suspensions on actor profile in appview v2 * Appview v2 use dataplane for identity lookups (#2095) * update bsky proto w/ identity methods * setup identity endpoints on mock dataplane * move from idresolver to dataplane for identity lookups on appview * tidy * Appview v2: apply safe takedown refs to records, actors (#2107) apply safe takedown refs to records, actors * Fix timing on appview v2 repo rev header (#2113) fix timing on appview repo rev * fix post thread responses * Appview v2 don't apply 3p self blocks (#2112) do not apply 3p self-blocks * Appview v2 search for feed generators (#2118) * add protos for feedgen search * support feed search on getPopularFeedGenerators * Appview v2 config tidy (#2117) * remove mod and triage roles from appview * rename cdn and search config * remove custom feed harness from appview v2 * Appview v2: don't apply missing modlists (#2122) * dont apply missing mod lists * update mock dataplane * Update packages/bsky/src/hydration/hydrator.ts Co-authored-by: devin ivy <devinivy@gmail.com> * refactor & document a bit better * fix up other routes --------- Co-authored-by: devin ivy <devinivy@gmail.com> * Appview v2 enforce post thread root boundary (#2120) * enforce post thread root boundary * test thread root boundary * Appview v2 fix admin environment variable (#2137) fix admin env in appview v2 * Remove re-pagination from getSuggestions (#2145) * remove re-pagination from getSuggestions * fix test * Adjust wording for account suspension (#2153) adjust wording for account suspension * Appview v2: fix not-found and blocked uris in threads (#2201) * fix uris of not-found and blocked posts in threads * update snaps * ✨ Show author feed of takendown author to admins only (#2197) * fold in cid, auth, tracing, node version changes * remove dead config from bsky service entrypoint * build * remove ozone test codepaths for appview v2 * tidy, docs fix --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz> |
||
Daniel Holmgren
|
fcb19c9c51
|
Simplify PDS moderation (#1723)
* spec out new simple pds mod routes * introduce new admin state endpoints * wire up routes * clean up pds * revoke refresh tokens * getUserAccountInfo * pr tidy * fixing some tests * fixing up more tests * fanout takedowns to pds * fanout admin reqs to pds * tidy * more tidy & add more pds moderation tests * getUserAccountInfo -> getAccountInfo * dont hydrate pds info on searchRepos * fix build * port admin tests to bsky package * clean up old snaps * tests on fanout * tweak naming * missed a rename * tidy renames * fix lex name * tidy & move snap * fix build * cleanup repeat process all * skip actor search test * fix bday paradox * tidy up pds service auth * rm skipped test * retry http * tidy * improve fanout error handling * fix test * return signing key in did-web * more tests * tidy serivce auth checks * change takedownId col to takedownRef * build branch * fix bsky test * add service key to indexer * move signing key to api entry * dont build |
||
Daniel Holmgren
|
d664b51c64
|
Finalize PDS in-process AppView removal (#1198)
* rm tables * rm event-stream & proxied * Remove appview services, move label service to pds * only proxy appview stuff * delete more tables * Start removing message dispatched from pds * more syncing-up removal of message dispatcher in pds * merged * remove feedgens from pds, remove getPopular * remove unused image helper from pds * fixing compiler errors * clean up sharp * rm label service * first pass on cleaning up tests * fix up a bunch of tests * moderation view tests * last admin tests * got a lil overzealous in deletes * clean up unused cfg * clean up label table * simplify admin repo search query/logic * tidy pds entrypoint * in-progress pds config changes * cfg fiddling * finish cleaning up cfg/ctx * comments * building * pds prefix on env * test env * collapse pds migrations down into a single migration * fix up dev-env * tidy * cleanup * fix pds admin tests * fix handle test * fix pds proxy tests * fix subscribe repos test * fix sqlite config in pds tests * add sqlite clause in sequencer-leader * fix actor search w/ sqlite on pds * fixes * fix dev env build * update pds service entrypoint * simple env example * make takedown ids opaque identifiers in the pds * use pds routes for api tests * update pds dockerfile with volume and correct port env var * add a couple env vars to example * add comments to env example * @atproto/pds 0.2.0-beta.0 * @atproto/aws 0.0.1-beta.0 * appview did * @atproto/aws 0.0.1 * enable logs by default * update env example * bugfixing sandbox issues * consistency in pds env var name for appview url * log on pds start and stop, configure version at runtime * @atproto/pds 0.2.0-beta.1 * fix semver matching for pds beta version * v0.2.0-beta.2 * default invites to being not required * fix flaky test * limit db connections in tests * publish 0.2.0-beta.d3 * fix invite required parsing * @atproto/pds 0.2.0-beta.5 * Proxy getPopularFeedGenerators on simplified pds (#1222) proxy getPopularFeedGenerators on pds Co-authored-by: dholms <dtholmgren@gmail.com> * tidy migrations * fix service entry * bump version * change auth order * bump version * bump version * add upgradeRepoVersion & fallback url for cdn * bump version * merging * merge pds * building dev-env * merging tests * merge service entry * test fixing * tidy * fix admin search * tidy * tidy * add snap for getListFeed * add backup nameserver cfg * tidy + pr feedback * tidy * tidy env * bit more * re-add dotenv to root package.json * fix dep * build branch * fix tests * Refactor tests to make better use of dev-env (#1690) * refactor pds tests to use dev env * refactor bsky tests * fix pds test * tidy bsky tests * build pds correctly * fix entry point * default logging to false (for now) * format service entry * Switch takedown ids back to ints on pds distribution (#1694) * switch takedown ids back to ints, consistent with live pds * tidy/fix migration * update migration for sqlite * export moderation action reversal * takedown tests * dont build branch --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> |
||
Daniel Holmgren
|
578757cb44
|
List feeds (#1557)
* lexicons for block lists * reorg blockset functionality into graph service, impl block/mute filtering * apply filterBlocksAndMutes() throughout appview except feeds * update local feeds to pass through cleanFeedSkeleton(), offload block/mute application * impl for grabbing block/mute details by did pair * refactor getActorInfos away, use actor service * experiment with moving getFeedGenerators over to a pipeline * move getPostThread over to a pipeline * move feeds over to pipelines * move suggestions and likes over to pipelines * move reposted-by, follows, followers over to pipelines, tidy author feed and post thread * remove old block/mute checks * unify post presentation logic * move profiles endpoints over to pipelines * tidy * tidy * misc fixes * unify some profile hydration/presentation in appview * profile detail, split hydration and presentation, misc fixes * unify feed hydration w/ profile hydration * unify hydration step for embeds, tidy application of labels * setup indexing of list-blocks in bsky appview * apply list-blocks, impl getListBlocks, tidy getList, tests * tidy * update pds proxy snaps * update pds proxy snaps * fix snap * make algos return feed items, save work in getFeed * misc changes, tidy * tidy * fix aturi import * lex * list purpose * lex gen * add route * add proxy route * seed client helpers * tests * mutes and blocks * proxy test * snapshot * hoist actors out of composeThread() * tidy * tidy * run ci on all prs * format * format * fix snap name * fix snapsh --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> |
||
devin ivy
|
6f419790ac
|
Feature: block lists (#1531)
* lexicons for block lists * reorg blockset functionality into graph service, impl block/mute filtering * apply filterBlocksAndMutes() throughout appview except feeds * update local feeds to pass through cleanFeedSkeleton(), offload block/mute application * impl for grabbing block/mute details by did pair * refactor getActorInfos away, use actor service * experiment with moving getFeedGenerators over to a pipeline * move getPostThread over to a pipeline * move feeds over to pipelines * move suggestions and likes over to pipelines * move reposted-by, follows, followers over to pipelines, tidy author feed and post thread * remove old block/mute checks * unify post presentation logic * move profiles endpoints over to pipelines * tidy * tidy * misc fixes * unify some profile hydration/presentation in appview * profile detail, split hydration and presentation, misc fixes * unify feed hydration w/ profile hydration * unify hydration step for embeds, tidy application of labels * setup indexing of list-blocks in bsky appview * apply list-blocks, impl getListBlocks, tidy getList, tests * tidy * update pds proxy snaps * update pds proxy snaps * fix snap * make algos return feed items, save work in getFeed * misc changes, tidy * tidy * fix aturi import * hoist actors out of composeThread() * tidy * run ci on all prs * format * build * proxy graph.getListBlocks * remove unneeded index * build pds * setup noop listblock indexer on pds * remove build --------- Co-authored-by: dholms <dtholmgren@gmail.com> |
||
Daniel Holmgren
|
73f64009a2
|
Appview database replicas (#1459)
* setup bsky so that writes always go to a db primary * test bsky primary db functionality * sprinkle db primary over moderation and view maintainer usage * make bsky tests more strict on primary/secondary db, fixes * maintain prev db config on appview, require primary db, make primary/replica explicit in config * tidy * sketch out db replicas * wip * support multiple tags per bsky replica, "any" tag, fallbacks and warns * clarify tags * use new coordinator api across routes & services * wire-up replica tagging config * tidy * cleanup entroypoints and tests for db coordinator * fix pds test * re-hookup migrate db * standardize env vars for db on bsky services * tidy * build * fix bav api entrypoint * support DB_POSTGRES_URL on ingester/indexer for backwards compat * remove build --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> |
||
Daniel Holmgren
|
e7a0d27f1f
|
Proxy timeline skeleton construction (#1264)
proxy timeline skeleton construction to appview |
||
Daniel Holmgren
|
ee68a4037b
|
Appview - Invalid handles (#1244)
* run with-test-db in monorepo test * improve pg script * tidy * namespace bsky pg schemas * differentiate schemas * clean up script * first pass at invalid handles in appview * tests for handle invalidation * move mock to instance instead of prototype * change network mocks in general * fixing pagination on actor-search * fix snap & normalize handles on index * handling did pagination + update tests * one last update |
||
Daniel Holmgren
|
8c4fa530d6
|
Pds proxy to appview based on headers (#1191)
* pds proxy to appview based on headers * reorg appview routes forwarding mutes to appview * tidy * clean up ctx * track notif state * use notif state for notif routes * add back in missing file * pr feedback * wip * more views * more views * update snaps & suggestions test * better choices for like & reposted * note * better block test * Test pds-proxied mute and notif state * Update bsky snaps, remove unnecessary #postView * Update bsky notification tests to use updateSeen * add getList getFeedGen tests * Re-disconnect pds from in-process appview indexing for tests * clean up forSnapshot * tidy * feed gen view tests --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> |