* tighten fuzzy actor search for larger dataset
* add indexes to support post deletion on feed_item, listing user invites on invite_code
* fix migration whitespace
* temp comment migrations and build
* Revert "temp comment migrations and build"
This reverts commit 225d2c00cce1885f2d2cffc4724843eda4b01cee.
* First pass on label definitions and reference-doc generation for them
* Tune labels
* Drop the 'csam' label in favor of using !hide
CSAM is an extremely delicate issue. Moderators must be careful to
remove it from the system without calling direct attention to it.
Using the generic !hide achieves the same effect while obscuring
the reason for the removal. Server and AppView bans will then be
employed (along with any other needed mechanisms) to strip the
content from the network.
There is also a 'false accusation' concern with using the csam
label. Fingerprinting techniques and AI are used to scan for csam.
These are capable of producing false positives which will then be
reverted after human review. The reputational damage caused by a
false positive could be severe, so we want to be careful about any
system which might publicize a false positive.
* Add label definition code generation
* Implement all type signatures for moderation
* Complete a first pass on the labeling sdk
* Add post-moderation test suite
* Add post moderation behavior documentation
* Add self-post behaviors
* Improve post moderation doc
* Tune up the post moderation doc
* Tune up the post moderation doc
* Tune up the post moderation doc
* Simplify the post moderation behavior descriptions
* More behavior description tuneup
* Add profile moderation final behaviors and tests
* Improve generated post-moderation-behaviors doc
* Add profile moderation behaviors doc
* Test muted-by-list specifically
* Fixes to label descriptions
* Fix to muted-by-list behavior
* Dont blur account in moderateProfile() when muting
* Tune label copy
* Apply post embed blurring when account is marked with blurmedia label
* Fix output signature
* Fixes to blocking behavior
* Rename LabelDefinitionPreference to LabelPreference
* Update docs
* Fix test
* Fix: self-harm should blur media only
* Fixes to tests
* Improve label copy
* Remove all labels that do not have a specific policy
Communicating moderation policies with active users is important.
This PR originally included labels which were proposed but did not
yet have policies. While we didn't intend to use them until a
policy was established, I decided it's better to hold off putting
them in the code until we're sure about them. They can be found in
backup files prefixed with "proposed-".
* add lexicon for unspecced applyLabels procedure
* implement label push to pds via unspecced.applyLabels
* add hive retry to bsky appview
* build
* update applyLabels to work with raw label data
* update bsky hive labeler
* remove build
* Allow setting a note when enabling/disabling invite codes
* Add inviteNote to repoView
* ✅ Add test for inviteNote
* ✨ Set to null when note is empty
* ✨ Change note from text to varchar
* 📝 Better description
* setup redis infra for appview indexer
* barebones bsky ingester
* add ioredis to bsky
* remove some indexer functionality from bsky api
* setup for bsky indexer
* tidy
* tidy, observe basic pipeline functioning
* process messages on bsky indexer pipeline, tidy tests and lifecycle
* trim partitions when moving cursor
* simplify config for partitions
* misc fixes for redis setup in bsky tests, add namespacing
* fix pds proxy tests
* remove cursor state from indexer partitions, simplify ingester state
* tidy
* utils for testing w/ multiple indexers, fix off-by-one xtrim
* test reingesting
* test indexer repartitioning
* add entrypoints for bsky ingester and indexer, fix db schema config, api entrypoint name, tidy
* setup and test bsky ingester backpressure, add config
* tidy
* add missing test file
* tidy redis calls, add redis sentinel config
* tidy/test some utils used in bsky pipeline
* tidy bsky pipeline tests, move helpers into dev-env
* fix pds crud test
* support redis host and password config
* better loggin/observability in ingester and indexer, make build
* add bsky ingester initial cursor config
* temporarily remove migrations from indexer/ingester
* allow ingester to batch
* packages/pg becomes packages/dev-infra with some cleanup (#1402)
* packages/dev-infra/
* Extract packages/dev-infra/_common.sh and use it
* Use period instead of source because of /bin/sh
* add docs for redis test script
* fix repartition test
* add logs to debug ci
* simplify repartitioning test, remove ci logs
---------
Co-authored-by: Jerry Chen <jerry@redelm.net>
* fix up bug in actor service
* add in hydrate methods
* build fix
* couple more fixes
* tidy
* port to appview
* dont build branch
* fix issue with items in list
* tidy
* dont build branch
* quick fix
tlds have not been appropriately checked since 6b51ecb due to what appears to be a typo/oversight.
this commit fixes it so that tlds are once again validated
* add index to support listRepos
* disable contents of user-account-cursor-idx migration, build
* ready for merge
---------
Co-authored-by: dholms <dtholmgren@gmail.com>
* add idx to post table & tweak feed construction
* comment out migration & build branch
* split out queries
* couple tweaks
* bugfix in with-friends
* lower feed data threshold
* return to subquery
* log injection on tracing
* Tweak timeline migraiton (#1369)
* v0.4.3
* tweak timeline migraiton
* misspelling
* move mutes & blocks to pds
* moar tweaks
* joins with 1 day cutoff
* no blocks/mutes on with friends
* attempt forcing custom plan
* increase feed date threshold
* trying something out
* trying something more
* tidy
* lower timeline date threshold
* tidy
* apply block rules for third parties on replies and embeds
* tidy terminology around blocking and actor relationships
* apply third-party blocking functionality to appview
* tidy, bsky tests
* refactor appview services
* tidy types
* Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
* ✨ Allow searching reports by moderator did (#1283)
* ✨ Allow searching reports by moderator did
* ✅ Remove .only flags on tests
* ✅ Update snapshot
* ✅ Add checks for did match in actions
* v0.4.1
* Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds
* tidy
* use 127.0.0.1 in with-test-db.sh for colima (#1297)
So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).
Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts. See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)
To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.
(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)
* Subscription util tests (#1295)
* consecutive list tests
* flesh out subscription util tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Content reporting on record fields (#1351)
* content reporting on record fields
* fix test
* tests
* tidy
* Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
* ✨ Added new procedure for sending admin email (#1312)
* 🚧 Added new lexicon for sending admin email
* ✨ Add moderation mailer
* ✨ Switch to text email content from html
* 🧹 Cleanup some early implementation code and reflect PR reivew
* ✨ Use smtp host instead of gmail service config
* ✨ Move to using single smtp url
* v0.4.2
* Patch up a couple sqlite tests (#1355)
patch up a couple sqlite tests
* Increase appview keepalive (#1360)
increase keepalive to 90s
* Appview - serve feed skeletons (#1265)
* proxy timeline skeleton construction to appview
* add getFeedSkeleton to appview
* mount route
* smart proxy feed skeletons
* tests
* proper error code
* only proxy specific feeds
* build branch
* update proxyable feed logic, should use feed publisher rather than generator
* fix feed proxy tests, configure feed publisher (in addition to generator)
* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)
* prevent user-supplied rkey on posts with createRecord
* allow empty-string rkey parameter
Co-authored-by: devin ivy <devinivy@gmail.com>
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
* add slurs to reserved words (#1318)
* add slurs to reserved words (#1314)
* Update reserved.ts
Add slurs to reserved words
* Update reserved.ts
fix typo
* Update reserved.ts to clean up the slur list
* linting
* pluralise
---------
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
* identifier: tweaks and additions to slur list (#1319)
* Refactor appview repo subscription for memleak (#1308)
* refactor to remove closure in loop
* move consecutive item out of p-queue
* Handle validation improvements (#1336)
* Handle matches and false positives for unacceptable words in handles
* move handle validation logic to pds
* missed merge
* add cfg flag
* encode lists
* fix build issues
* move words to cfg
* tidy
---------
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
* Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
* ✨ Allow searching reports by moderator did (#1283)
* ✨ Allow searching reports by moderator did
* ✅ Remove .only flags on tests
* ✅ Update snapshot
* ✅ Add checks for did match in actions
* v0.4.1
* Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds
* tidy
* use 127.0.0.1 in with-test-db.sh for colima (#1297)
So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).
Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts. See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)
To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.
(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)
* Subscription util tests (#1295)
* consecutive list tests
* flesh out subscription util tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Content reporting on record fields (#1351)
* content reporting on record fields
* fix test
* tests
* tidy
* Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
* ✨ Added new procedure for sending admin email (#1312)
* 🚧 Added new lexicon for sending admin email
* ✨ Add moderation mailer
* ✨ Switch to text email content from html
* 🧹 Cleanup some early implementation code and reflect PR reivew
* ✨ Use smtp host instead of gmail service config
* ✨ Move to using single smtp url
* v0.4.2
* Patch up a couple sqlite tests (#1355)
patch up a couple sqlite tests
* enable feeds & build branch
* disable branch building & enable without proxy header
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
* add optional ilike query to getPopularFeedGenerators
* fix lint
* fix lint
* handle sqlite too in pds only
* fix lint
* v0.4.3
* Include takendown posts for admins (feature branch) (#1361)
* 🚧 WIP including takendown posts on author feed
* ✨ Add takedown id on posts when including taken down posts
* 🧹 Cleanup the auth verifier and other bsky package code
* ✅ Add test for admin getAuthorFeed
* 🧹 Cleanup lexicon and exclude takedownId
* more explicit plumbing for post hydration w/o requester or with takedown info
* pass along flag for soft-deleted actors
* cleanup getAuthorFeed w/ auth
* reorg getAuthorFeed logic around role/access-based auth
---------
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
* ✨ Disable signing up with invite code from takendown account (#1350)
* fix viewer data
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Eric Bailey <git@esb.lol>
* 🚧 WIP including takendown posts on author feed
* ✨ Add takedown id on posts when including taken down posts
* 🧹 Cleanup the auth verifier and other bsky package code
* ✅ Add test for admin getAuthorFeed
* 🧹 Cleanup lexicon and exclude takedownId
* more explicit plumbing for post hydration w/o requester or with takedown info
* pass along flag for soft-deleted actors
* cleanup getAuthorFeed w/ auth
* reorg getAuthorFeed logic around role/access-based auth
---------
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
* proxy timeline skeleton construction to appview
* add getFeedSkeleton to appview
* mount route
* smart proxy feed skeletons
* tests
* proper error code
* only proxy specific feeds
* build branch
* update proxyable feed logic, should use feed publisher rather than generator
* fix feed proxy tests, configure feed publisher (in addition to generator)
* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)
* prevent user-supplied rkey on posts with createRecord
* allow empty-string rkey parameter
Co-authored-by: devin ivy <devinivy@gmail.com>
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
* add slurs to reserved words (#1318)
* add slurs to reserved words (#1314)
* Update reserved.ts
Add slurs to reserved words
* Update reserved.ts
fix typo
* Update reserved.ts to clean up the slur list
* linting
* pluralise
---------
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
* identifier: tweaks and additions to slur list (#1319)
* Refactor appview repo subscription for memleak (#1308)
* refactor to remove closure in loop
* move consecutive item out of p-queue
* Handle validation improvements (#1336)
* Handle matches and false positives for unacceptable words in handles
* move handle validation logic to pds
* missed merge
* add cfg flag
* encode lists
* fix build issues
* move words to cfg
* tidy
---------
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
* Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
* ✨ Allow searching reports by moderator did (#1283)
* ✨ Allow searching reports by moderator did
* ✅ Remove .only flags on tests
* ✅ Update snapshot
* ✅ Add checks for did match in actions
* v0.4.1
* Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds
* tidy
* use 127.0.0.1 in with-test-db.sh for colima (#1297)
So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).
Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts. See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)
To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.
(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)
* Subscription util tests (#1295)
* consecutive list tests
* flesh out subscription util tests
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Content reporting on record fields (#1351)
* content reporting on record fields
* fix test
* tests
* tidy
* Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
* ✨ Added new procedure for sending admin email (#1312)
* 🚧 Added new lexicon for sending admin email
* ✨ Add moderation mailer
* ✨ Switch to text email content from html
* 🧹 Cleanup some early implementation code and reflect PR reivew
* ✨ Use smtp host instead of gmail service config
* ✨ Move to using single smtp url
* v0.4.2
* Patch up a couple sqlite tests (#1355)
patch up a couple sqlite tests
* enable feeds & build branch
* disable branch building & enable without proxy header
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
* 🚧 Added new lexicon for sending admin email
* ✨ Add moderation mailer
* ✨ Switch to text email content from html
* 🧹 Cleanup some early implementation code and reflect PR reivew
* ✨ Use smtp host instead of gmail service config
* ✨ Move to using single smtp url
