15 Commits

Author SHA1 Message Date
devin ivy
0f7f53655b
Appview: timeout polo, fail open (#3007)
* appview: timeout getFollowsFollowing, fail open

* build
2024-11-19 10:52:22 -05:00
rafael
1982693e3e
Use node:crypto instead of noble/curves (#2936)
* Extract verifySignatureWithKey out of verifyJwt

* Accept optional verifySignatureWithKey as param

* Impl. verifySignatureWithKey with native crypto

* Test key validation

* changesets

* build

* build (fix)

* Move verifySig out

* Trigger Build

* Move test

* Remove redundant check

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-11-07 13:29:42 -03:00
Daniel Holmgren
6c1ec149cf
PDS proxy to appview performance (#2773)
* accept entryway session tokens

* extra check + tests

* build

* build

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-09-06 18:56:37 -05:00
Daniel Holmgren
c46dc9144a
Fix bsky build (#2790)
* test build

* see logs

* copy sync pakcage

* tidy
2024-09-04 20:46:56 -05:00
Daniel Holmgren
50c0ec176c
Service auth method binding (lxm) (#2663)
* add scopes to service auth impl

* add error to getServiceAuth

* send scoped tokens from pds

* clean up privileged access scopes & allow simple service auth tokens for app passwords

* integration into ozone

* fix up bsky tests

* cleanup xrpc-server tests

* fix up tests & types

* one more test

* fix read after write tests

* fix mod auth test

* convert scopes to be a single method name

* add scope check callback for auth verifier

* pds changes only

* fix feed generation tests

* use scope for ozone service profile

* dont verify scopes on pds yet

* tidy

* tidy imports

* changeset

* add tests

* tidy

* another changeset

* scope -> lxm

* tidy

* clean up scope references

* update nonce size

* pr feedback

* trim trailing slash

* nonce -> jti

* fix xrpc-server test

* allow service auth on uploadBlob

* fix build error

* changeset

* build, tidy

* xrpc-server: update lxm claim check error

* appview: temporarily permit labeler service calls to omit lxm claim

* xrpc-server: fix test

* changeset

* fix merged tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-08-18 15:46:07 -04:00
devin ivy
7e88f6fc1d
Appview: enable insight into full thread context (#2651)
* appview: enable mods viewing full thread context

* build
2024-07-18 19:06:26 -07:00
devin ivy
f05539db4c
Appview: apply hosting status in getRecord (#2620)
appview: apply hosting status in getRecord
2024-07-02 17:43:21 -04:00
devin ivy
af7d3912a3
Starter packs (#2553)
* lexicon: initial starter pack record lexicons

* lexicon: first pass on views for starter packs

* lexicon: add starter pack query endpoints, misc fixes

* lexicon: add starter pack related fields to profile views

* lexicon: codegen

* api prerelease

* bsky: protos for starter packs

* appview: implement starter pack endpoints in dataplane, protos update

* appview: starter pack hydration and view, mock dataplane indexing

* appview: wire-up starter pack graph endpoints

* appview: test basic starter packs functionality, update dev-env

* fix snaps for profile createdAt

* appview: fix list item count for starter packs in dataplane

* fix ozone snaps, tidy

* appview: reorg list aggs for starter packs

* appview: starter pack lexicon and proto tweaks, move around aggregations

* appview: fix views and snaps

* api prerelease

* fix snaps

* appview: misc proto updates for starter packs, snaps

* appview: add starter pack counts to profile detail

* build

* api prerelease

* pds: starter pack slur detection

* lexicon: starter pack notifications and batch fetch

* appview: starter pack notification and batch fetch

* appview: add starterpack-joined notif to dataplane and test

* api prerelease

* ozone: fix snaps

* pds build

*  Proxy starterpack requests through ozone and include takedowns (#2594)

*  Proxy starterpack requests through ozone and include takedowns

* 🧹 Cleanup

*  Commit snapshots

*  Update snapshot

* lexicon: move starterpacks back to TIDs

* changeset

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-06-24 21:04:33 -04:00
Daniel Holmgren
71f9cc9d4f
Clean up role-based auth (#2255)
* tidy bsky auth

* hook up new auth verifier

* update auth throughout ozone

* handle mod signing keys

* add client proxy heads to pds

* hook up rest of routes

* simplify pipethrough & add some SSRF protection

* tests

* fix bad var

* remove basic auth in ozone

* wip

* fix key parsing in pds

* fix up all ozone tests

* fix admin auth test

* rename test

* fix ozone test

* clean up tokens in pds

* fix up pds tests

* fix up ozone tests

* add pipethrough to write routes

* reenable proxied admin test

* add moderator accounts to ozone in dev-env

* update did doc id values

* null creds string -> `none`

* fix fetchLabels auth check

*  Add a couple more proxied requests that we use in ozone ui

* Add runit to the services/bsky Dockerfile (#2254)

add runit to the services/bsky Dockerfile

* Improve tag detection (#2260)

* Allow tags to lead with and contain only numbers

* Break tags on other whitespace characters

* Export regexes from rich text detection

* Add test

* Add test

* Disallow number-only tags

* Avoid combining enclosing screen chars

* Allow full-width number sign

* Clarify tests

* Fix punctuation edge case

* Reorder

* Simplify, add another test

* Another test, comment

* Version packages (#2261)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* 🐛 Increment attempt count after each attempt to push ozone event (#2239)

* Ozone delegates email sending to actor's pds (#2272)

* ozone delegates email sending to user's pds

* lexicon: add content field to mod email event

* test email sending via mod event

* add dev dep for nodemailer in ozone

* fix auth verifier method

* build branch

* build branch

* fix url check

* better error handling for get account infos

* fix labeler service id

* fix iss on auth headers

* fix dev-env ozone did

* fix tests & another jwt issuer

* fix proxy auth

* ozone: fix ip check

* fix aud check on pds mod service auth

* tidy

* Update packages/pds/tests/proxied/admin.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix pipethrough of headers

* fix moderation status tests

* fix auth on ozone routes

* update iss on daemon

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-03-06 19:53:52 -06:00
Daniel Holmgren
2267f1ec48
Add client-set proxy headers to PDS (#2251)
* tidy bsky auth

* hook up new auth verifier

* update auth throughout ozone

* handle mod signing keys

* add client proxy heads to pds

* hook up rest of routes

* simplify pipethrough & add some SSRF protection

* tests

* fix bad var

* fix key parsing in pds

* fix admin auth test

* rename test

* add pipethrough to write routes

* update did doc id values

* null creds string -> `none`

* fix fetchLabels auth check

*  Add a couple more proxied requests that we use in ozone ui

* Add runit to the services/bsky Dockerfile (#2254)

add runit to the services/bsky Dockerfile

* Improve tag detection (#2260)

* Allow tags to lead with and contain only numbers

* Break tags on other whitespace characters

* Export regexes from rich text detection

* Add test

* Add test

* Disallow number-only tags

* Avoid combining enclosing screen chars

* Allow full-width number sign

* Clarify tests

* Fix punctuation edge case

* Reorder

* Simplify, add another test

* Another test, comment

* Version packages (#2261)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* 🐛 Increment attempt count after each attempt to push ozone event (#2239)

* Ozone delegates email sending to actor's pds (#2272)

* ozone delegates email sending to user's pds

* lexicon: add content field to mod email event

* test email sending via mod event

* fix auth verifier method

* build branch

* fix url check

* better error handling for get account infos

* fix labeler service id

* fix iss on auth headers

* fix dev-env ozone did

* fix tests & another jwt issuer

* ozone: fix ip check

* fix aud check on pds mod service auth

* tidy

* fix pipethrough of headers

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-03-06 19:38:21 -06:00
Daniel Holmgren
f65de89eed
Feature: Appview v2 (#1924)
* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* filler commit

* rm filler

* server boilerplate

* follows impl

* posts impl

* posts & likes impl

* repost impl

* profiles & handle null values

* list impl

* mutes impl

* blocks impl

* misc

* feed gen impl

* label impl

* notifs impl

* feeds impl

* threads impl

* early sketchwork

* wip

* stub out thick client

* in-progress work on hydrator

* tweak

* hydrate profile labels, detail lists

* feedgen hydration

* protobuf tweaks

* more protobuf tweaks

* wip

* snake case

* moar snake case

* tidy actor hydration

* tidy parsing

* type fixes, renaming, comments in hydrator

* hydrate list items and likes

* hydrate notifications

* feed hydration

* graph & label hydration

* more record protobufs

* pluralize

* tweak pbs

* use new methods

* Setup dataplane grpc client/mock server (#1921)

* add buf & connectrpc, codegen client

* lint

* prettier ignore

* fix prettier ignore

* tidy & add tests

* add record getter mocks

* post view hydration

* fix up mock dataplane to match new protos

* missed one

* wire up dataplane in ctx & dev-env

* adding some basic views

* feed hydration, add blocks to post hydration

* pass over notification hydration

* tidy

* merge

* implement getProfile

* hydrate post aggregation and viewer state

* fix

* fix codegen

* get some tests passing!

* add takedowns & some like bugfixing

* all profile tests passing!

* likes test

* follow endpoints using data plane

* reorg follow block rules

* reposts

* post views!

* implement getList w/ dataplane caveat

* adjust dataplane getListMembers to return listitem uris

* implement getListMutes and -Blocks w/ dataplane

* suggestions

* timeline

* misc view fixes

* view fixes for mutes, self-mute/block

* author feed

* feed gen routes

* tidy

* misc block/mute fixes

* list feed & actor likes

* implement getLists, fix some empty cursors

* implement getMutes, empty profile description fix

* implement getBlocks, block application fix

* implement getSuggestedFollowsByActor, needs some fixes

* feed generation

* search routes

* threads

* tidy

* fix some snaps

* fix getSuggestedFollowsByActor

* implement listNotifications

* implement getUnreadCount w/ dataplane

* implement notifications.updateSeen w/ dataplane

* 3rd party blocking tests

* blocked profile viewer

* add search mocks

* refactor getFeed

* createPipeline -> createPipelineNew

* basic replygating functionality on dataplane w/o filtering violating replies

* hack threadgates into dataplane, apply gates

* deterministic thread orders in dataplane

* misc cleanup around dataplane

* upgrade typescript to v5.3

* update typescript linter deps

* sync bsky proto, codegen

* update dataplane, sync with bsky proto updates

* remove indexer, ingester, daemon, moderation services from appview

* convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane

* remove bsky services. db and indexing logic into mock dataplane.

* remove tests not needed by appview v2, misc reorg

* add basic in-mem repo subscription to dataplane mock

* fix dev-env, bsky tests, bsky build

* cull bsky service entrypoint

* add bsky service readme

* build

* tidy

* tidy, fix pds proxy tests

* fix

* fix bsky entrypoint deps

* support http2 grpc client

* build

* fix dataplane bad tls config/default

* support multiple dataplane urls, retry when unavailable

* build

* tidy/fix

* move dataplane mock tests into their own dir

* cover label hydration through timeline test

* bring back labels in appview tests

* remove unused db primary/replica/coordinator from bsky dataplane

* bsky proto add cids to contracts, buf codegen

* sync-up bsky data-plane w/ codegen updates

* start using dataplane interaction endpoints

* add file

* avoid overfetching from dataplane, plumb feed items and cids

* pass refs through for post viewer state

* switch list feeds to use feed item in dataplane

* handle not found err on get-thread dataplane call

* support use of search service rather than dataplane methods

* mark some appview v2 todos

* tidy

* still use dataplane on search endpoints when search service is not configured

* fix pds test

* fix up bsky tests & snaps

* tidy migrations

* fix appview-v2 docker build

* Support label issuer tied to appview v2 (#2033)

support label issuer tied to appview

* Appview v2: handle empty cursor on list notifications (#2017)

handle empty cursor on appview listnotifs

* Update appview v2 to use author feed enum (#2047)

* update bsky protos with author feed enum, misc feed item changes

* support new author feed enums in dataplane

* fix build

* Appview v2: utilize sorted-at field in bsky protos (#2050)

utilize new sorted-at field in bsky protos

* remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts

* Appview v2, sync w/ changes to protos (#2071)

* sync bsky protos

* sync-up bsky implementation w/ proto changes

* Appview v2 initial implementation for getPopularFeedGenerators (#2072)

add an initial implementation for getPopularFeedGenerators on appview v2

* merge

* fixes

* fix feed tests

* fix bsync mock

* format

* remove unused config

* fix lockfile

* another lockfile fix

* fix duplicate type

* fix dupplicate test

* Appview v2 handling clearly bad cursors (#2092)

* make mock dataplane cursors different from v1 cursors

* fail open on clearly bad appview cursors

* fix pds appview proxy snaps

* Appview v2 no notifs seen behavior (#2096)

* alter behavior for presenting notifications w/ no last-seen time

* fix pds proxy tests

* Appview v2 dataplane retries based on client host (#2098)

choose dataplane client for retries based on host when possible/relevant

* don't apply negated labels

* display suspensions on actor profile in appview v2

* Appview v2 use dataplane for identity lookups (#2095)

* update bsky proto w/ identity methods

* setup identity endpoints on mock dataplane

* move from idresolver to dataplane for identity lookups on appview

* tidy

* Appview v2: apply safe takedown refs to records, actors (#2107)

apply safe takedown refs to records, actors

* Fix timing on appview v2 repo rev header (#2113)

fix timing on appview repo rev

* fix post thread responses

* Appview v2 don't apply 3p self blocks (#2112)

do not apply 3p self-blocks

* Appview v2 search for feed generators (#2118)

* add protos for feedgen search

* support feed search on getPopularFeedGenerators

* Appview v2 config tidy (#2117)

* remove mod and triage roles from appview

* rename cdn and search config

* remove custom feed harness from appview v2

* Appview v2: don't apply missing modlists (#2122)

* dont apply missing mod lists

* update mock dataplane

* Update packages/bsky/src/hydration/hydrator.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* refactor & document a bit better

* fix up other routes

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* Appview v2 enforce post thread root boundary (#2120)

* enforce post thread root boundary

* test thread root boundary

* Appview v2 fix admin environment variable (#2137)

fix admin env in appview v2

* Remove re-pagination from getSuggestions (#2145)

* remove re-pagination from getSuggestions

* fix test

* Adjust wording for account suspension (#2153)

adjust wording for account suspension

* Appview v2: fix not-found and blocked uris in threads (#2201)

* fix uris of not-found and blocked posts in threads

* update snaps

*  Show author feed of takendown author to admins only (#2197)

* fold in cid, auth, tracing, node version changes

* remove dead config from bsky service entrypoint

* build

* remove ozone test codepaths for appview v2

* tidy, docs fix

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
2024-02-27 15:22:55 -05:00
Eric Bailey
90e8325e5f
pnpm setup (#1532)
* use pnpm

* fix dependency issues, replace yarn and lerna scripts

* remove the main/dist scripts

* update Dockerfiles

* use pnpm

* fix dependency issues, replace yarn and lerna scripts

* remove the main/dist scripts

* update Dockerfiles

* update bin script

* remove unused zod dep

* fix type errors in pds

* add types prop to packages

* remove unused, bump lock

* fix test running

* build before test

* fix pino types

* format

* pds depends on dev-env in test

* refer to src instead of built packages

* pds relies on bsky in test too

* remove yarn.lock

* add -r flag to root test

* test push to aws

* remove docker test

* add publishConfig to new package

* move services to top level dir

(cherry picked from commit f5012bec33435a4473e9960066807623334f3aff)

* update workflow paths

(cherry picked from commit 5c70f0176d381ca35d6be10cfa173e22373a5b5d)

* add main-to-dist script

* use script in all packages, remove old Dockerfiles

* remove old bsky service

* remove newline

* test container builds

* Revert "test container builds"

This reverts commit c228611f5e8e1624d4b124be4976c49590130f43.

* remove unused config

* test build containers

* pnpm in syntax

* bump dd-trace

* shamefully hoist

* even more shame

* hoist, externalize deps

* clean install for prod and smaller containers

* dont build branches

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-09-05 18:45:49 -05:00
Jerry Chen
cfb6835664
Require bluesky-social/atproto for .github/workflows/build-and-push-* (#1460) 2023-08-11 15:46:42 -05:00
Miles Zimmerman
bba1ba739e
chore: update docker/setup-buildx-action github action to v2 (#975)
chore: update docker/setup-buildx-action to v2
2023-05-12 19:39:25 -05:00
bnewbold
af9f11fbde
'bsky' (appview) dev helpers (#888)
* remove commited .env files; exclude in gitignore

* bsky: distinct postgresql database in dev mode

* github CI: build and upload 'bsky' container to GHCR

This isn't how we deploy, but is useful for public access and things
like integration tests.

* Makefile: run-dev-appview
2023-04-24 16:27:34 -07:00