* tidy bsky auth
* hook up new auth verifier
* update auth throughout ozone
* handle mod signing keys
* add client proxy heads to pds
* hook up rest of routes
* simplify pipethrough & add some SSRF protection
* tests
* fix bad var
* fix key parsing in pds
* fix admin auth test
* rename test
* add pipethrough to write routes
* update did doc id values
* null creds string -> `none`
* fix fetchLabels auth check
* ✨ Add a couple more proxied requests that we use in ozone ui
* Add runit to the services/bsky Dockerfile (#2254)
add runit to the services/bsky Dockerfile
* Improve tag detection (#2260)
* Allow tags to lead with and contain only numbers
* Break tags on other whitespace characters
* Export regexes from rich text detection
* Add test
* Add test
* Disallow number-only tags
* Avoid combining enclosing screen chars
* Allow full-width number sign
* Clarify tests
* Fix punctuation edge case
* Reorder
* Simplify, add another test
* Another test, comment
* Version packages (#2261)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* 🐛 Increment attempt count after each attempt to push ozone event (#2239)
* Ozone delegates email sending to actor's pds (#2272)
* ozone delegates email sending to user's pds
* lexicon: add content field to mod email event
* test email sending via mod event
* fix auth verifier method
* build branch
* fix url check
* better error handling for get account infos
* fix labeler service id
* fix iss on auth headers
* fix dev-env ozone did
* fix tests & another jwt issuer
* ozone: fix ip check
* fix aud check on pds mod service auth
* tidy
* fix pipethrough of headers
---------
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
* adding some basic views
* feed hydration, add blocks to post hydration
* pass over notification hydration
* tidy
* merge
* implement getProfile
* hydrate post aggregation and viewer state
* fix
* fix codegen
* get some tests passing!
* add takedowns & some like bugfixing
* all profile tests passing!
* likes test
* follow endpoints using data plane
* reorg follow block rules
* reposts
* post views!
* implement getList w/ dataplane caveat
* adjust dataplane getListMembers to return listitem uris
* implement getListMutes and -Blocks w/ dataplane
* suggestions
* timeline
* misc view fixes
* view fixes for mutes, self-mute/block
* author feed
* feed gen routes
* tidy
* misc block/mute fixes
* list feed & actor likes
* implement getLists, fix some empty cursors
* implement getMutes, empty profile description fix
* implement getBlocks, block application fix
* implement getSuggestedFollowsByActor, needs some fixes
* feed generation
* search routes
* threads
* tidy
* fix some snaps
* fix getSuggestedFollowsByActor
* implement listNotifications
* implement getUnreadCount w/ dataplane
* implement notifications.updateSeen w/ dataplane
* 3rd party blocking tests
* blocked profile viewer
* add search mocks
* refactor getFeed
* createPipeline -> createPipelineNew
* basic replygating functionality on dataplane w/o filtering violating replies
* hack threadgates into dataplane, apply gates
* deterministic thread orders in dataplane
* misc cleanup around dataplane
* upgrade typescript to v5.3
* update typescript linter deps
* sync bsky proto, codegen
* update dataplane, sync with bsky proto updates
* remove indexer, ingester, daemon, moderation services from appview
* convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane
* remove bsky services. db and indexing logic into mock dataplane.
* remove tests not needed by appview v2, misc reorg
* add basic in-mem repo subscription to dataplane mock
* fix dev-env, bsky tests, bsky build
* cull bsky service entrypoint
* add bsky service readme
* build
* tidy
* tidy, fix pds proxy tests
* fix
* fix bsky entrypoint deps
* support http2 grpc client
* build
* fix dataplane bad tls config/default
* support multiple dataplane urls, retry when unavailable
* build
* tidy/fix
* move dataplane mock tests into their own dir
* cover label hydration through timeline test
* bring back labels in appview tests
* remove unused db primary/replica/coordinator from bsky dataplane
* bsky proto add cids to contracts, buf codegen
* sync-up bsky data-plane w/ codegen updates
* start using dataplane interaction endpoints
* add file
* avoid overfetching from dataplane, plumb feed items and cids
* pass refs through for post viewer state
* Lexicons: Add labeler prefs, labeler declaration, and get labeler routes
* Add labelerViewBasic and update embed views
* Fix typo
* switch list feeds to use feed item in dataplane
* handle not found err on get-thread dataplane call
* support use of search service rather than dataplane methods
* mark some appview v2 todos
* tidy
* still use dataplane on search endpoints when search service is not configured
* fix pds test
* Switch to labelerViewDetailed
* Move label and report values to refs
* Add getActorLabelers
* lint
* fix up bsky tests & snaps
* small diff to open pr
* rm new line
* codegen schemas
* tidy migrations
* table + indexing
* protos
* rename lexicons
* views, hydration + rename lexicons
* rest of routes
* data plane routes
* parse labelers from req
* fix appview-v2 docker build
* Support label issuer tied to appview v2 (#2033)
support label issuer tied to appview
* hydrate context
* tidy header logic
* integrating into more routes
* more routes
* wrap up rest
* add mock labeler
* rework labelerlexicons
* tidy lexs
* codegen new lexicons
* integrate lexicon rework
* add proxy logic
* forward labeler headers through pds
* tweak label header parsing
* remove did from scheams
* update indexing for lexs
* tests for mod service views
* label hydration test
* Add 'associated' to profileViewDetailed
* Rename labelers to mods in preferences
* Change uri to did in mod preferences
* couple more
* syntax tweaks
* integrate updated lexicons
* update view snap
* handle mod service embeds
* tidy
* fix mock
* lint
* base default labels of config var
* fix label hydration
* Appview v2: handle empty cursor on list notifications (#2017)
handle empty cursor on appview listnotifs
* Update appview v2 to use author feed enum (#2047)
* update bsky protos with author feed enum, misc feed item changes
* support new author feed enums in dataplane
* fix build
* Appview v2: utilize sorted-at field in bsky protos (#2050)
utilize new sorted-at field in bsky protos
* remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts
* Appview v2, sync w/ changes to protos (#2071)
* sync bsky protos
* sync-up bsky implementation w/ proto changes
* Appview v2 initial implementation for getPopularFeedGenerators (#2072)
add an initial implementation for getPopularFeedGenerators on appview v2
* merge
* fixes
* fix feed tests
* fix bsync mock
* format
* remove unused config
* fix lockfile
* another lockfile fix
* fix duplicate type
* fix dupplicate test
* Appview v2 handling clearly bad cursors (#2092)
* make mock dataplane cursors different from v1 cursors
* fail open on clearly bad appview cursors
* fix pds appview proxy snaps
* Appview v2 no notifs seen behavior (#2096)
* alter behavior for presenting notifications w/ no last-seen time
* fix pds proxy tests
* Appview v2 dataplane retries based on client host (#2098)
choose dataplane client for retries based on host when possible/relevant
* don't apply negated labels
* display suspensions on actor profile in appview v2
* Appview v2 use dataplane for identity lookups (#2095)
* update bsky proto w/ identity methods
* setup identity endpoints on mock dataplane
* move from idresolver to dataplane for identity lookups on appview
* tidy
* Appview v2: apply safe takedown refs to records, actors (#2107)
apply safe takedown refs to records, actors
* Fix timing on appview v2 repo rev header (#2113)
fix timing on appview repo rev
* fix post thread responses
* Appview v2 don't apply 3p self blocks (#2112)
do not apply 3p self-blocks
* Appview v2 search for feed generators (#2118)
* add protos for feedgen search
* support feed search on getPopularFeedGenerators
* Appview v2 config tidy (#2117)
* remove mod and triage roles from appview
* rename cdn and search config
* remove custom feed harness from appview v2
* Appview v2: don't apply missing modlists (#2122)
* dont apply missing mod lists
* update mock dataplane
* Update packages/bsky/src/hydration/hydrator.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* refactor & document a bit better
* fix up other routes
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
* Appview v2 enforce post thread root boundary (#2120)
* enforce post thread root boundary
* test thread root boundary
* Appview v2 fix admin environment variable (#2137)
fix admin env in appview v2
* Remove re-pagination from getSuggestions (#2145)
* remove re-pagination from getSuggestions
* fix test
* Adjust wording for account suspension (#2153)
adjust wording for account suspension
* Appview v2: fix not-found and blocked uris in threads (#2201)
* fix uris of not-found and blocked posts in threads
* update snaps
* ✨ Show author feed of takendown author to admins only (#2197)
* fold in cid, auth, tracing, node version changes
* remove dead config from bsky service entrypoint
* build
* remove ozone test codepaths for appview v2
* tidy, docs fix
* fix test
* add additional user counts
* add associated data to profiles
* update snaps
* update to is_mod_service
* format
* tidy
* 3p labeler sdk updates (#2176)
* Update sdk to support 3p labeler preferences
* Stick with intolerance instead of hate for the label group id
* wip expand labels and label groups
* Output moderationOpts (computed) and modsPref (unaltered)
* Add tests for enabling/disabling mod services
* Add atproto-labelers header config
* Expand labels and label groups in definitions
* Fix tests
* Tweaks to labels
* Remove label descriptions and improve output types on labels and label groups
* Add mocker to exported API
* Improve types of label and label group definitions
* Rework moderation prefs to continue using global labelgroup settings and only disable label groups per moderator
* Simplify encoding of the label preferences in definitions
* Add target constraints to labels
* Refactor the moderation sdk to derive more behaviors from the definition files
* Small cleanup
* Add hiding tool to modsdk
* Track filter causes
* Make mute state an alert
* Fix: dont blur profileview for blocks
* Prioritize causes by severity
* Add moderateNotification() and drop quote post moderation code
* Add mocker functions for notifications
* Improve mock data
* Lexicon: Add custom label definitions and remove modservice descriptions
* Lexicon: Update moderation prefs
* SDK updates: remove label groups, reduce builtin labels, update mod-preference apis
* Lexicon: Update global labels with new reduced set
* Lexicon: Remove moderation.getService and add detailed option to getServices
* Lexicons: add severity=none to custom label value defs
* Implement custom label-value definition tooling
* All custom labels are no-self
* Backend impl for labeler lexicon updates (#2258)
* codegen
* clean up impl
* fix up tests
* Lexicon: modservice -> labeler
* Remove x- prefix behavior; add label value syntax rules; add custom label precedence rules
* Lexicon: Remove the ability to choose a defaultSetting from custom labels
* Rework test suites
* Give behaviors to all labels regardless of target
* sync up backend with lex changes
* fix labelers in dev-env agent
* lint protos
* update protos & views
* small bugfix & update tests
* tweak protos
* fix build issue from merge
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
* make appview and mod services optional on pds
* pds: allow configuring a reporting service optionally separate from an administrative mod service
* tidy
* ✨ Store languages on subjects from record data
* ✨ Fetch lang from author feed for repo subject
* ✨ Regenerate lex
* ✨ Add lang to profile subject
* ✨ Move lang to flag
* ✨ Add comment to flag event
* ✅ Update pds test snapshot
* 🐛 Fix broken import
* ✨ Rename flag to tag
* ✅ Update snapshot for event by id
* ✅ Bring back skipped test
* ✨ Move tags to dedicated columns
* ✅ Update test snapshots, use logger
* ✨ Change lang:unknown to lang:und
* ✅ Update test snapshots
* 🧹 Cleanup
* ✨ Add filter params for tags on queryModerationEvents endpoint
* ✨ Add tags property to subject status model
* ✅ Update test code
* ✨ Add filter to get subjects that do not have certain tags
* mv appview
* copy
* finalize copy
* package names
* big WIP
* first pass at mod servce
* some tidy
* tidy & fix compiler errors
* rename to ozone, db migrations, add to dev-env & pds cfg
* getRecord & getRepo mostly working
* fix open handle
* get record tests all working
* moderation events working
* statuses working
* tidy test suite
* search repos
* server & db tests
* moderation tests
* wip daemon + push events
* pds fanout working
* fix db test
* fanning takedowns out to appview
* rm try/catch
* bsky moderation test
* introduce mod subject wrappers
* more tidy
* refactor event reversal
* tidy some db stuff
* tidy
* rename service to mod-service
* fix test
* tidy config
* refactor auth in bsky
* wip patching up auto-mod
* add label ingester in appview
* fix a couple build issues
* fix some timing bugs
* tidy polling logic
* fix up tests
* fix some pds tests
* eslint ignore
* fix ozone tests
* move seeds to dev-env
* move images around
* fix db schemas
* use service auth admin reqs
* fix remaining tests
* auth tests bsky
* another test
* random tidy
* fix up search
* clean up bsky mod service
* more tidy
* default attempts to 0
* tidy old test
* random tidy
* tidy package.json
* tidy logger
* takedownId -> takedownRef
* misc pr feedback
* split daemon out from ozone application
* fix blob takedown mgiration
* refactor ozone config
* do push event fanout on write instead of on read
* make suspend error work again
* add attempts check & add supporting index
* fix takedown test ref
* get tests working
* rm old test
* fix timing bug in event pusher tests
* attempt another fix for timing bug
* await req
* service files
* remove labelerDid cfg
* update snaps for labeler did + some cfg changes
* fix more snaps
* pnpm i
* build ozone images
* build
* make label provider optional
* fix build issues
* fix build
* fix build
* build pds
* build on ghcr
* fix syntax in entry
* another fix
* use correct import
* export logger
* remove event reverser
* adjust push event fanout
* push out multiple
* remove builds
* cleanup repeat process all
* wip
* skip actor search test
* skip actor search test
* tweak processAll
* decrease wait to 1 sec
* repo_blob -> record_blob
* simplify backlink linkTo
* return repo_root to one row
* sequence before updating repo_root
* invite code forUser -> forAccount
* ipld_block -> repo_block
* use lru-cache fetchMethod
* move did_cache to own db
* better error handling on did cache
* drop did_handle
* fix sequencer wait time
* debug
* debug
* more debug
* check something
* fix bday paradox
* fix bday paradox
* tidy up pds service auth
* rm skipped test
* retry http
* tidy
* improve fanout error handling
* fix test
* return signing key in did-web
* more tests
* tidy serivce auth checks
* user_account -> account
* remove inviteNote
* keypair per repo
* use an lru cache for keypairs as well
* clean up repo
* wip
* wrap up accoutn manager
* tidy
* tidy
* fix tests
* fix disabled codes
* fix appview tests
* add note
* set pragmas
* tidy account manager getDb
* rename pref transactor
* user pref -> account pref
* handle blob imports
* tidy imports
* add reserveSigningKey
* wip transferAccount
* clean up transferAccount
* tests
* tidy
* tidy
* configure entryway url on pds
* handle entryway in pds admin endpoints
* make importRepo temp
* fix imports
* make email optional on pds when using entryway
* handle diffs
* handle pds entryway usage for server, identity, admin endpoints
* pds support for credentials from entryway
* setup pds tests w/ entryway service
* tidy
* tidy
* update entryway version
* wip
* test handle updates w/ entryway
* split account table into two
* tidy
* tweak scripts
* tidy tests
* tidy
* better config for actorstore & dbs
* clean up cfg more
* reorg actorstore fs layout
* handle erros on actor db create
* pr tidy & fix accoutn deletion test
* pr feedback
* fix bad merge
* unskip test
* fix subscribe repos tests
* tidy repo root tables
* tidy
* fix tests
* tidy delete tokens
* tidy account getters
* tidy
* bulk deletesg
* increase chunk size
* handle racing refreshes
* wip
* fix auth test
* invert import flow
* clean up actor store on create account failure
* tweak sequencer
* prevent invite code races on createAccount
* rm note
* add back in race protection on getAccountInviteCodes
* start feature branch
* deleted app migration table
* patch up new auth test
* rm note
* g
* create accoutn delegated from entryway
* tidy
* fix test
* change plcOp type to unknown
* small fixes
* sync up w entryway branch
* Use proper error when authed account is not found (#1799)
provide proper error when account not found in access-takedown check
* build branch
* build on ghcr
* tweak service file
* tweak service file
* change where we save reserved keys
* no tmp dir in blobstore either
* fix blobstore temp location again
* handle repeat record_blobs
* create account before submitting plc op & undo if fail
* small tweak
* limit the number of local records
* push out empty commit on transfer
* fix issue with record_blob
* add push blob endpoint
* Set and validate token audiences on pds v2 (#1793)
set and validate token audience on pds v2
* merge
* include entryway did on tests
* build branch
* fix cache issue
* xrpc server blob limit
* put correct bytes
* add auth to routes
* handle quarantining/unquarantining a blob that does not exist
* tidy
* fix transfer tests
* fix email request routes for entryway
* PDS v2 entryway account deletion (#1819)
* add admin lexicon for account deletion
* implement admin account deletion endpoint
* fix entryway proxying on account email checks
* proxy to entryway for acct deletion
* read-after-write sanity check
* tweak
* wip
* finish refactor
* fix test schema
* application retry logic for busy
* pr feedback
* rm lru-cache
* fix test pg schema
* fix transfer test
* Sqlite instrumentation for pds v2 (#1838)
* sqlite instrumentation
* build
* remove build
* dont reimport blobs
* send ticks during import
* close on error
* catch handle validation error
* add log
* fix test
* return emailConfirmedAt on getAccountInfo
* Upgrade sharp on pds v2 (#1863)
upgrade sharp to 0.32.6
* read all bytes before parsing car
* Async car reader (#1867)
* asynchronously read in car
* dont buffer car
* tweak
* Gracefully handle indexing of invalid records (#1853)
* gracefully handle indexing of invalid records
* fix repo tests
* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)
fix role auth for access-or-role verifier, fix getBlob actor takedown check
* better cleanup of actor-stores
* add ability to not ensure leaves
* tidy
* allow did:web transfer
* Migration utility for actor-store (#1873)
beginnings of helper for migrating all actors
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* base case for findBlobRefs
* App-level retries for sqlite on pds (#1871)
* revamp retry helper to be more flexible re: backoff strategies
* sqlite timeout helper
* ensure sqlite wal on db creation/migration rather than every open
* layer retries for sqlite on writes outside transactions on pds
* tidy
* fix up lockfile
* tidy
* fix lex codegen
* fix timing bug in threadgate test
* No-op update handling (#1916)
do no produce commits on no-op updates
* Retry on all SQLITE_BUSY error codes (#1917)
retry on all sqlite_busy error codes
* Pds v2 ensure sqlite ready (#1918)
ensure sqlite is ready before making queries
* try something
* tidy
* dont build branch
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* 🚧 WIP with proposed lexicons for event based mod architecture
* 🚧 Remove unnecessary moderation action lexicon
* 🚧 Working on event based actions
* ✨ Add escalated subject status
* 🐛 Alright, fixed the error in lexicon
* 🚧 Working through reversal
* ✨ Cleanup build errors
* ✨ Add subject status endpoint
* ✨ Add handler
* ✨ get reports from mod actions table
* :rightwards_twisted_arrows: Merge with upstream
* 🚧 Builds but test network doesnt start
* ✨ Tests passing on event based status change
* ✨ Rename index
* ♻️ Rename takeModerationAction->emitModerationEvent
* ✨ Implement label reversal
* ✅ Auto-revert test working
* ♻️✅ Refactored to event types and tests are passing
* ✨ Add takedown event sequence validation
* ✨ Adds support for blobCid status
* 🧹 Cleanup unnecessary method:
* ✨ Hydrate handles with status and events
* ✨ Re-implement auto reversal
* ✨ Add takendown and mute filters
* ✨ Allow filtering events by type
* ✨ Allow filtering events by creator did
* ✨ Add subjectStatus to record and repoview
* ✨ Add persistent note feature
* ✨ Log send email event
* 🐛 Fix logging send email event
* ✨ Better type
* ✨ Adjust migration to create separate moderation_event table
* 🧹 Cleanup types
* ✅ Adjust tests with mod event emitter
* ✨ Fix more tests around takedowns
* ✅ Get test suite to pass
* ✅ Get test suite to pass for pds
* ✅ Get test suite to pass for pds
* ✅ Update snapshot for feedgen
* ✅ Why are more snapshots updating?
* ♻️ Rename getModerationEvents -> queryModerationEvents
* ♻️ Rename getModerationStatuses -> queryModerationStatuses
* ♻️ Rename persistNote->sticky
* 🐛 Rename subject
* ♻️ Cleanup expiresAt for scheduled actions
* ✨ Add more tests, allow fetching mod history for all content by a user
* ✅ Fix repo and record tests
* ✨ Migrate reports and actions to events
* 🐛 Fix escalated status overwrite
* ✨ Implement direct sql query to create events from actions and reports
* 🚧 Adding keyset pagination for subject statuses
* ✨ Add migration for lastReportedAt
* ✨ Migrate blob cids
* ✨ Fix pagination on mod subject list endpoint
* 🐛 Fix blob actions
* ✅ All tests passing on bsky package
* ✅ Bring back snapshots
* ✅ Skipping timeline test temporarily
* ✅ Skipping some more tests to isolate failing ones
* ✅ Bring back list-feed test
* ✅ Bring back timeline test
* ✅ Fix label action in seeding
* ✅ Enable timeline proxied test
* ✅ Enable search actor proxied test
* ✅ Enable feedgen tests
* ✅ Fix test for admin/get-record
* ✨ Move note to comment for subject status
* ✨ Accept comments in mute event
* ✨ Remap flag event to ack event
* 🐛 Add legacyRef in report union selection
* @atproto/api 0.6.24-next.0
* @atproto/api 0.6.24-next.1
* ✨ Adjust migration export and add index for blobCids column
* ✨ Maintin action ids when migrating
* ✨ Paginate events using createdAt timestamp
* ✅ Update snapshot for pds test with events cursor update
* ✅ Use only events for snapshot testing
* ✅ Use only events for snapshot in the remaining test
* relative paths to lexicons for build
* fix bsky periodic event reversal in service entrypoint
* ✨ Allow comments in takedown and label
* ✨ Only import reports on consecutive run of the migration script
* ✨ Adjust moderation property of blob entries
* determine latest reports to migrate
* ✨ Process new reports for subject status
* ✨ Process unresolved reports on first migration run
* fix transaction error, process just unresolved reports, make reported-at updates safe for reruns
* tidy
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* spec out new simple pds mod routes
* introduce new admin state endpoints
* wire up routes
* clean up pds
* revoke refresh tokens
* getUserAccountInfo
* pr tidy
* fixing some tests
* fixing up more tests
* fanout takedowns to pds
* fanout admin reqs to pds
* tidy
* more tidy & add more pds moderation tests
* getUserAccountInfo -> getAccountInfo
* dont hydrate pds info on searchRepos
* fix build
* port admin tests to bsky package
* clean up old snaps
* tests on fanout
* tweak naming
* missed a rename
* tidy renames
* fix lex name
* tidy & move snap
* fix build
* cleanup repeat process all
* skip actor search test
* fix bday paradox
* tidy up pds service auth
* rm skipped test
* retry http
* tidy
* improve fanout error handling
* fix test
* return signing key in did-web
* more tests
* tidy serivce auth checks
* change takedownId col to takedownRef
* build branch
* fix bsky test
* add service key to indexer
* move signing key to api entry
* dont build
* rm tables
* rm event-stream & proxied
* Remove appview services, move label service to pds
* only proxy appview stuff
* delete more tables
* Start removing message dispatched from pds
* more syncing-up removal of message dispatcher in pds
* merged
* remove feedgens from pds, remove getPopular
* remove unused image helper from pds
* fixing compiler errors
* clean up sharp
* rm label service
* first pass on cleaning up tests
* fix up a bunch of tests
* moderation view tests
* last admin tests
* got a lil overzealous in deletes
* clean up unused cfg
* clean up label table
* simplify admin repo search query/logic
* tidy pds entrypoint
* in-progress pds config changes
* cfg fiddling
* finish cleaning up cfg/ctx
* comments
* building
* pds prefix on env
* test env
* collapse pds migrations down into a single migration
* fix up dev-env
* tidy
* cleanup
* fix pds admin tests
* fix handle test
* fix pds proxy tests
* fix subscribe repos test
* fix sqlite config in pds tests
* add sqlite clause in sequencer-leader
* fix actor search w/ sqlite on pds
* fixes
* fix dev env build
* update pds service entrypoint
* simple env example
* make takedown ids opaque identifiers in the pds
* use pds routes for api tests
* update pds dockerfile with volume and correct port env var
* add a couple env vars to example
* add comments to env example
* @atproto/pds 0.2.0-beta.0
* @atproto/aws 0.0.1-beta.0
* appview did
* @atproto/aws 0.0.1
* enable logs by default
* update env example
* bugfixing sandbox issues
* consistency in pds env var name for appview url
* log on pds start and stop, configure version at runtime
* @atproto/pds 0.2.0-beta.1
* fix semver matching for pds beta version
* v0.2.0-beta.2
* default invites to being not required
* fix flaky test
* limit db connections in tests
* publish 0.2.0-beta.d3
* fix invite required parsing
* @atproto/pds 0.2.0-beta.5
* Proxy getPopularFeedGenerators on simplified pds (#1222)
proxy getPopularFeedGenerators on pds
Co-authored-by: dholms <dtholmgren@gmail.com>
* tidy migrations
* fix service entry
* bump version
* change auth order
* bump version
* bump version
* add upgradeRepoVersion & fallback url for cdn
* bump version
* merging
* merge pds
* building dev-env
* merging tests
* merge service entry
* test fixing
* tidy
* fix admin search
* tidy
* tidy
* add snap for getListFeed
* add backup nameserver cfg
* tidy + pr feedback
* tidy
* tidy env
* bit more
* re-add dotenv to root package.json
* fix dep
* build branch
* fix tests
* Refactor tests to make better use of dev-env (#1690)
* refactor pds tests to use dev env
* refactor bsky tests
* fix pds test
* tidy bsky tests
* build pds correctly
* fix entry point
* default logging to false (for now)
* format service entry
* Switch takedown ids back to ints on pds distribution (#1694)
* switch takedown ids back to ints, consistent with live pds
* tidy/fix migration
* update migration for sqlite
* export moderation action reversal
* takedown tests
* dont build branch
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* lex
* list purpose
* lex gen
* add route
* add proxy route
* seed client helpers
* tests
* mutes and blocks
* proxy test
* snapshot
* hoist actors out of composeThread()
* tidy
* tidy
* run ci on all prs
* format
* format
* fix snap name
* fix snapsh
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* lexicons for block lists
* reorg blockset functionality into graph service, impl block/mute filtering
* apply filterBlocksAndMutes() throughout appview except feeds
* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application
* impl for grabbing block/mute details by did pair
* refactor getActorInfos away, use actor service
* experiment with moving getFeedGenerators over to a pipeline
* move getPostThread over to a pipeline
* move feeds over to pipelines
* move suggestions and likes over to pipelines
* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread
* remove old block/mute checks
* unify post presentation logic
* move profiles endpoints over to pipelines
* tidy
* tidy
* misc fixes
* unify some profile hydration/presentation in appview
* profile detail, split hydration and presentation, misc fixes
* unify feed hydration w/ profile hydration
* unify hydration step for embeds, tidy application of labels
* setup indexing of list-blocks in bsky appview
* apply list-blocks, impl getListBlocks, tidy getList, tests
* tidy
* update pds proxy snaps
* update pds proxy snaps
* fix snap
* make algos return feed items, save work in getFeed
* misc changes, tidy
* tidy
* fix aturi import
* hoist actors out of composeThread()
* tidy
* run ci on all prs
* format
* build
* proxy graph.getListBlocks
* remove unneeded index
* build pds
* setup noop listblock indexer on pds
* remove build
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* pseudocode for sending notification to user
* add notification push token table
* lexicon codegen
* `pds` and `api` codegen
* update lexicon
* add simple function to `putNotificationPushToken` into `notification_push_token` table
* reorgnaize imports
* add unspecced `putNotificationPushToken` to pds
* add `notification-push-tokens` table to PDS
* check if token exists before adding it to db
* add endpoint and appId to PDS table
* setup notification server
* fix logic for inserting token into db
* fix NotificationServer methods by making them static
* fix merge conflicts
* add comments on how sending ntoifications work
* remove dead code
* move notifServer to AppContext
* refactor code to handle notif sending in `Indexer` service
* add additional data when sending notifs
* clean up code
* move notif logic to indexer
* add `appId` and `endpoint` optional params to putNotificationPushToken
* clone notification code to AppView
* add endpoint to register push token with app view
* disable pds and enable app view notification server
* clean up code
* simply logic to check if a token already exists
* remove NotificationServer from PDS
* remove notification-push-token table from PDS
* remove `putNotificationPushToken` endpoint
* clean up code
* let `axios` throw error if `gorush` has an error
* let `kysely` throw error if notif cannot be registered by client
* rename `registerPushNotificationEndpoint` to `registerPushNotifications`
* delete `putNotificationPushToken` from AppView
* rename putPushNotificationToken to registerPushNotification
* remove dead notification code from pds
* remove sanitizeDisplayName from NotificationSever
* move `pushNotificationEndpoint` to config
* temp add `pushNotificationEndpoint` to dev-env setup
* remove example test from feed-generation.test.ts
* add test for registerPushNotification and clean up error handling
* move notifications test to its own file
* add test for NotificationServer to check if tokens are retrieved correctly
* remove unused functions from NotificationServer
* add additional tests for NotificationServer
* add return type to getNotificationDisplayAttributes function
* remove unnecessary comments
* remove dead NotificationServer code from PDS
* clean up code to prepareNotifsToSend
* put sending notifications as part of the backgroundQueue
* log instead of throwing error if notification attributes don't exist
* remove logs
* add more tests to `notification-server.test.ts`
* show replied with text for reply notifs
* better error handling when sending notifications via backgroundQueue
* add rate limit and batching to sending notifications
* add comments to NotificationServer
* merge with main
* use redis for rate limiting instead of normal rate limits
* move `notificationBatchSize` into config
* usePrimary db in test
* hoist push notif migration to present, update model to remove endpoint
* update push notif lexicon
* pare down lex for unregistering push
* helpers for working with service endpoints from did doc
* service-authed register/unregister push notifs
* add well-known endpoint to appview
* update bsky notif service tests
* fix to batching logic, misc tidy
* batch display notifications
* colocate all notification sending logic
* tidy tests
* remove unregister token for now
* fix registerPush lexicon, make a procedure
* fix registerPush impl, test pds proxy to notif service
* fix tests, make notif server optional when not configured
* fix notif server config for bsky app service
* move notif server rate limiting in-mem for now, add sending retry
* codegen tidy
* only push notifs on commit
* build
* fix notif rate limiter check
* send notifs from users w/o a profile
* remove build
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* setup bsky so that writes always go to a db primary
* test bsky primary db functionality
* sprinkle db primary over moderation and view maintainer usage
* make bsky tests more strict on primary/secondary db, fixes
* maintain prev db config on appview, require primary db, make primary/replica explicit in config
* tidy
* sketch out db replicas
* wip
* support multiple tags per bsky replica, "any" tag, fallbacks and warns
* clarify tags
* use new coordinator api across routes & services
* wire-up replica tagging config
* tidy
* cleanup entroypoints and tests for db coordinator
* fix pds test
* re-hookup migrate db
* standardize env vars for db on bsky services
* tidy
* build
* fix bav api entrypoint
* support DB_POSTGRES_URL on ingester/indexer for backwards compat
* remove build
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* first pass on profiles
* quick test
* wip
* wip
* test post thread
* record embeds
* get author feed profiles
* wip timeline
* fix get timeline
* switch from counter to tid
* tidy into a service
* quick tid test
* pr feedback
* clock -> rev
* update image formatting
* disable migration & build branch
* add recent posts to getAuthorFeed & handle post thread not found errors
* refactor for lag header
* tidy
* rm collections check
* tidy test
* pr feedback
* fix small bug
* build branch
* get migrations into system
* enable migrations
* Add self-label schemas
* switch around array and union for self-labels
* apply self-labels to post and profile views
* test for self labels in pds
* apply self-labels to post and profile views in bsky appview
* test for self labels in bsky appview
* update pds proxied test snapshots
* Add support for self-labels to the mdoeration sdk
* Disable unknown-labeler filtering until 3P support is added
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* proxy timeline skeleton construction to appview
* add getFeedSkeleton to appview
* mount route
* smart proxy feed skeletons
* tests
* proper error code
* only proxy specific feeds
* build branch
* update proxyable feed logic, should use feed publisher rather than generator
* fix feed proxy tests, configure feed publisher (in addition to generator)
* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)
* prevent user-supplied rkey on posts with createRecord
* allow empty-string rkey parameter
Co-authored-by: devin ivy <devinivy@gmail.com>
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
* add slurs to reserved words (#1318)
* add slurs to reserved words (#1314)
* Update reserved.ts
Add slurs to reserved words
* Update reserved.ts
fix typo
* Update reserved.ts to clean up the slur list
* linting
* pluralise
---------
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
* identifier: tweaks and additions to slur list (#1319)
* Refactor appview repo subscription for memleak (#1308)
* refactor to remove closure in loop
* move consecutive item out of p-queue
* Handle validation improvements (#1336)
* Handle matches and false positives for unacceptable words in handles
* move handle validation logic to pds
* missed merge
* add cfg flag
* encode lists
* fix build issues
* move words to cfg
* tidy
---------
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
* Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
* ✨ Allow searching reports by moderator did (#1283)
* ✨ Allow searching reports by moderator did
* ✅ Remove .only flags on tests
* ✅ Update snapshot
* ✅ Add checks for did match in actions
* v0.4.1
* Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds
* tidy
* use 127.0.0.1 in with-test-db.sh for colima (#1297)
So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).
Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts. See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)
To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.
(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)
* Subscription util tests (#1295)
* consecutive list tests
* flesh out subscription util tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Content reporting on record fields (#1351)
* content reporting on record fields
* fix test
* tests
* tidy
* Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
* ✨ Added new procedure for sending admin email (#1312)
* 🚧 Added new lexicon for sending admin email
* ✨ Add moderation mailer
* ✨ Switch to text email content from html
* 🧹 Cleanup some early implementation code and reflect PR reivew
* ✨ Use smtp host instead of gmail service config
* ✨ Move to using single smtp url
* v0.4.2
* Patch up a couple sqlite tests (#1355)
patch up a couple sqlite tests
* enable feeds & build branch
* disable branch building & enable without proxy header
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
* limit backsearch to 1 day instead of 3
* lower like count threshold
* bump to 6
* disable like count check
* disable with friends
* preemptively cache last commit
* inline list mutes
* actor service
* label cache
* placehodler on popular with friends
* bulk sequence
* no limit but chunk
* bump chunk to 5k
* try 10k
* fix notify
* tweaking
* syntax
* one more fix
* increase backfill allowance
* full refresh label cache
* limit 1 on mute list
* reserve aclu handle
* clean up testing with label cache
* note on with-friends
* rm defer from label cache
* label cache error handling
* rm branch build
* run with-test-db in monorepo test
* improve pg script
* tidy
* namespace bsky pg schemas
* differentiate schemas
* clean up script
* first pass at invalid handles in appview
* tests for handle invalidation
* move mock to instance instead of prototype
* change network mocks in general
* fixing pagination on actor-search
* fix snap & normalize handles on index
* handling did pagination + update tests
* one last update
* add cid to ListView and ListViewBasic
* add test for mute list embed
* add support for list embeds views
* test
* port to appview
* update missing snap
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* Add languages field to post record
* helper for parsing bcp47
* add language format to lexicon
* codegen for post record langs field
* re-export language parsing in api package
* tests and tidy for lexicon language format
* index post langs, in-progress
* update snapshots, fixes record-with-media embed issue
* index post langs on bsky appview
* don't bother indexing post langs in pds appview, tidy
* pds proxy to appview based on headers
* reorg appview routes forwarding mutes to appview
* tidy
* clean up ctx
* track notif state
* use notif state for notif routes
* add back in missing file
* pr feedback
* wip
* more views
* more views
* update snaps & suggestions test
* better choices for like & reposted
* note
* better block test
* Test pds-proxied mute and notif state
* Update bsky snaps, remove unnecessary #postView
* Update bsky notification tests to use updateSeen
* add getList getFeedGen tests
* Re-disconnect pds from in-process appview indexing for tests
* clean up forSnapshot
* tidy
* feed gen view tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Model post and profile aggs in bsky
* Add background, onCommit, db and background stats
* Add aggregations for post and profile
* Use background queue in bsky labeler, misc plumbing
* Process background throughout bsky tests
* bsky tidy and fixes
* Build views using agg tables in bsky
* Views of feedgens on bsky
* hook up indexing
* Setup custom feeds on bsky
* graph service + getList routes
* migration
* Feed views refactor for bsky, implement getFeed w/ auth passthrough
* mute lists
* add getListMutes
* mute list tests
* add mutes table & mute/unmuteActor routes
* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests
* add getMutes
* implement mute functionality
* remove unspecced
* mute tests
* mute impl tests
* remove proxy tests
* Test custom algos on bsky
* add mute state to snapshots
* table & indexing plugin
* getBlocks
* bail out of mute subquery when no viewer
* fleshing out services
* Test pds getFeed proxy
* tidy, fix build
* tidy
* Fix minor race in bsky mutelist tests
* blocks in feed views
* db migration, impl block logic & tests
* Update pds proxy snaps based on appview mutes
* missed some files on last commit
* ensure process after unmute
* update snap
* move adminAuth to seedclient param
* apply blocks in getFeed
* update snaps
* upadate one last snap
* pr feedback
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Misc notification and optimization in bsky, tidy
* Handle notifs on message queue in bsky appview
* Sync-up profile labeling behavior in bsky
* No async work during actor unindexing in bsky, include additional indexes
* Prevent deeply nested notifs w/ hellthread exception in bsky appview
* Add indexes to bsky appview for more efficient actor deletion/unindexing
* Update pds proxy snaps for updated bsky profile labeling
* Return custom suggestions in bsky appview
* Model post and profile aggs in bsky
* Add background, onCommit, db and background stats
* Add aggregations for post and profile
* Use background queue in bsky labeler, misc plumbing
* Process background throughout bsky tests
* bsky tidy and fixes
* Build views using agg tables in bsky
* Views of feedgens on bsky
* Setup custom feeds on bsky
* Feed views refactor for bsky, implement getFeed w/ auth passthrough
* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests
* Test custom algos on bsky
* Test pds getFeed proxy
* tidy, fix build
* tidy
* actor type in profiles
* skeleton lex
* more schemas
* first pass at feed composition
* cleanup services
* more tidying
* fix up tests
* bookmarking schemas
* make feeds records instead of actors
* some cleanup
* fleshing out feeds as records
* wip
* working into cfg & dev-env
* fix appview test util
* bookmarking + views
* tidy
* add displayName
* db migration
* wip test
* blob processing
* first test working!
* Sync-up snapshots
* wip
* Fix pds ctx type
* Sync lexicons on pds and api
* wip
* fully refactor
* Prep for open union in reply data in bsky appview, sync-up snapshots
* msgs & ports
* Make pds did cache sqlite-friendly
* Start scaffolding more feedgen tests
* revamp appview tests
* revamp pds tests
* merge in dev env utils
* ensure plc close
* add feed gen to dev-env
* second feed gen
* named feeds in mock
* fix issue with dev-env pds appview
* update bsky snaps
* Flesh-out some feed gen tests
* Fix feed gen reply test
* Test feed bookmarking
* update skeleton defs
* merge
* Sync-up pds with updated feed skeleton contract
* Fix feed bookmark viewer state
* add feed embeds
* Test inaccurate feed gen repost
* Tighten some error behaviors around feed gens
* generator view in record embed
* test
* bookmarked/subscribed -> saved
* updated snaps
* name & descript on alices feed
* add like counts
* proposed lex
* tweak
* fetch feed generator details
* add describeFeedGen to dev-env
* adds tests for getFeedGenerator
* Update lexicons/app/bsky/feed/getFeedGenerator.json
Co-authored-by: devin ivy <devinivy@gmail.com>
* update describe
* add cid to feed gen view
* add alice feed gen avatar
* generator displayName required
* update snaps
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* cleanup lex
* mount route
* Starter bsky algos (#1046)
* implement some starter bsky algos
* add team dids
* tidy
* add quotepost check to whats hot classic
* tests
* carve out feedgen section of pds
* apply limits
* add describeFeedGenerator route
* Whats hot new (#1060)
* map out new whats hot
* only available in postgres
* Fix dev env public url (#1058)
* Fix the public url used by dev-env
* move publicUrl location
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add defs for feed gen prefs & decrease displayName length
* build branch
* updated how saved feeds works
* build algos branch
* remove relative imports to @atproto/api
* update hot classic name
* rm build
---------
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* v0.3.6
* v0.3.7
* v0.1.10
* update name of skyline
* build branch
* aws build
* getFeedGenerators test
* For now only use new replyRef union for custom feeds, for compat
* Don't use replyRef union on new feeds yet
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Daniel Holmgren
devin ivy
Foysal Ahamed
intrnl
Mary
Ansh
Eric Bailey
Paul Frazee