Commit Graph

1347 Commits

Author SHA1 Message Date
Daniel Holmgren e1d10d21c0 Increase backfill page size (#1432)
increase backfill page size
2023-08-08 19:16:00 -05:00
bnewbold 3ed6dceafb remove unused/unspecified label (#1426)
* labeler: remove redundant 'underwear' label

This commit contains no change to mod policy, app behavior, etc.

We have been creating redundant "underwear" labels along with "sexual"
for many months. "sexual" is the actual label that gets actioned in mod
preferences, etc. The un-specified "underwear" labels are confusing and
take up resources on-disk and in API responses.

* reflect in appview

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-08 17:56:38 -05:00
Daniel Holmgren 4879ccb365 Appview label cache (#1451)
* add label cache to appview

* add to dev env

* fix test
2023-08-08 17:56:21 -05:00
Eric Bailey 55f33234e2 fix naming 2023-08-08 16:40:55 -05:00
Eric Bailey b70238476f fix select cause 2023-08-08 16:38:39 -05:00
Eric Bailey 5f2fc6c1ae format 2023-08-08 16:14:27 -05:00
Eric Bailey 08ea9790d9 use where exists subquery 2023-08-08 16:13:47 -05:00
Eric Bailey 399dfdbcc5 fix tests 2023-08-08 16:03:44 -05:00
Eric Bailey 92f5979d44 update naming based on feedback 2023-08-08 15:41:48 -05:00
Eric Bailey 975b0f40ed fix lint 2023-08-08 12:22:22 -05:00
Eric Bailey 5083705a37 use more specific naming 2023-08-08 12:20:08 -05:00
Eric Bailey 9e2b14765a add query and test to pds 2023-08-08 12:14:40 -05:00
Eric Bailey 59c435648d codegen again 2023-08-08 12:02:34 -05:00
Eric Bailey 7f84fb215e codegen 2023-08-08 12:02:34 -05:00
Daniel Holmgren 5cc57d5ebc Hotfix: remove relative imports from mod routes (#1446)
remove relative imports from mod methods
2023-08-07 19:26:07 -05:00
Daniel Holmgren bb4b4344bd Get post hotfix (#1445)
ensure post is defined
2023-08-07 18:54:33 -05:00
Daniel Holmgren 556e438ccc Filter out blocked posts on getPosts (#1442)
filter out blocked posts on getPosts
2023-08-07 18:52:10 -05:00
devin ivy b5569121f8 Misc perf improvements on actor search, post deletion, actor invite codes (#1436)
* tighten fuzzy actor search for larger dataset

* add indexes to support post deletion on feed_item, listing user invites on invite_code

* fix migration whitespace

* temp comment migrations and build

* Revert "temp comment migrations and build"

This reverts commit 225d2c00cce1885f2d2cffc4724843eda4b01cee.
2023-08-07 19:38:04 -04:00
devin ivy 6321e7839f Include pds account info on mod report and action details (#1441)
hydrate pds repo state onto mod report and action subject details
2023-08-07 17:20:26 -04:00
devin ivy 3f3ae4cfd5 Support proxying moderation through to appview (#1233)
* in-progress work on proxying moderation to appview

* tidy

* proxy reports pds to appview, misc tidy

* test proxying of moderation endpoints

* remove report action fkeys from pds for appview sync

* test appview/pds moderation synchronization

* tidy

* tidy

* fix admin proxy tests, build

* temp disable migration

* rm relative @atproto/api imports

* fix a couple more relative imports

* tidy

* reenable migration, comment contents temporarily

* fully enable migration, remove build, misc test fixes

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-08-04 16:06:04 -04:00
Daniel Holmgren 6d04bf76ef Hive retry logic & logging (#1434)
* better logging & retry logic for hive

* tidy

* fix test

* bump timeout

* tidy pds labeler

* fix test

* build branch

* fix types

* fix types

* rm branch builds

* log hive response

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-03 17:42:05 -05:00
devin ivy e5e24d510e Push labels to PDS (#1423)
* add lexicon for unspecced applyLabels procedure

* implement label push to pds via unspecced.applyLabels

* add hive retry to bsky appview

* build

* update applyLabels to work with raw label data

* update bsky hive labeler

* remove build
2023-08-03 12:25:56 -04:00
Daniel Holmgren 450dff7fa3 Fix blocks in follow views (#1427)
* fix blocks in follow views

* check blocks between follow subjects & creators

* test
2023-08-03 10:43:30 -05:00
Foysal Ahamed 13b16b33f2 Allow setting a note when enabling/disabling invite codes (#1363)
* Allow setting a note when enabling/disabling invite codes

* Add inviteNote to repoView

*  Add test for inviteNote

*  Set to null when note is empty

*  Change note from text to varchar

* 📝 Better description
2023-08-01 17:43:21 -05:00
devin ivy 30b41029c1 Partitioning in bsky indexer (#1368)
* setup redis infra for appview indexer

* barebones bsky ingester

* add ioredis to bsky

* remove some indexer functionality from bsky api

* setup for bsky indexer

* tidy

* tidy, observe basic pipeline functioning

* process messages on bsky indexer pipeline, tidy tests and lifecycle

* trim partitions when moving cursor

* simplify config for partitions

* misc fixes for redis setup in bsky tests, add namespacing

* fix pds proxy tests

* remove cursor state from indexer partitions, simplify ingester state

* tidy

* utils for testing w/ multiple indexers, fix off-by-one xtrim

* test reingesting

* test indexer repartitioning

* add entrypoints for bsky ingester and indexer, fix db schema config, api entrypoint name, tidy

* setup and test bsky ingester backpressure, add config

* tidy

* add missing test file

* tidy redis calls, add redis sentinel config

* tidy/test some utils used in bsky pipeline

* tidy bsky pipeline tests, move helpers into dev-env

* fix pds crud test

* support redis host and password config

* better loggin/observability in ingester and indexer, make build

* add bsky ingester initial cursor config

* temporarily remove migrations from indexer/ingester

* allow ingester to batch

* packages/pg becomes packages/dev-infra with some cleanup (#1402)

* packages/dev-infra/

* Extract packages/dev-infra/_common.sh and use it

* Use period instead of source because of /bin/sh

* add docs for redis test script

* fix repartition test

* add logs to debug ci

* simplify repartitioning test, remove ci logs

---------

Co-authored-by: Jerry Chen <jerry@redelm.net>
2023-08-01 14:59:00 -04:00
Daniel Holmgren 1ad6274202 Actor service fix (#1414)
* fix up bug in actor service

* add in hydrate methods

* build fix

* couple more fixes

* tidy

* port to appview

* dont build branch

* fix issue with items in list

* tidy

* dont build branch

* quick fix
2023-08-01 12:38:14 -05:00
bnewbold 3befcfe35e pds: set default postgresql URL to localhost (#1415) 2023-07-31 21:53:45 -07:00
Daniel Holmgren 1b0a5716ce Reflect takedowns in actor service (#1412)
reflect takedowns in actor service
2023-07-31 18:43:04 -05:00
dumbmoron 850265d7bd pds: actually validate tlds (#1408)
tlds have not been appropriately checked since 6b51ecb due to what appears to be a typo/oversight.
this commit fixes it so that tlds are once again validated
2023-07-31 17:39:53 -05:00
devin ivy c060db24bd Optimize com.atproto.sync.listRepos (#1391)
* add index to support listRepos

* disable contents of user-account-cursor-idx migration, build

* ready for merge

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-07-28 16:46:32 -05:00
Daniel Holmgren 7ef73467eb Remove local whats hot from pds (#1393)
* put behind proxy header

* toggle for everyone

* tidy

* tidy

* fix
2023-07-27 19:05:12 -05:00
Daniel Holmgren 030be22f60 Adjust bsky team feed (#1394)
* adjust bsky team feed

* tweak
2023-07-27 18:49:09 -05:00
Daniel Holmgren 7c20b01272 Proxy writes to appview (#1392)
* proxy writes to appview

* dont build branch
2023-07-27 13:10:14 -05:00
Daniel Holmgren ca7b891ece Log injection (#1384)
datadog inject logs
2023-07-25 22:43:15 -05:00
Patrick Linnane acd5185984 Fix various typos (#1373) 2023-07-25 22:43:05 -05:00
Daniel Holmgren a2655a4001 Move rkey check to prepareCreate (#1385)
* move rkey check to prepareCreate

* allow updates on applyWrites

* gen rkey in prepareCreate

* fix up tests
2023-07-25 22:41:10 -05:00
devin ivy d9f73769e7 Batch outgoing sequencing (#1298)
* batch outgoing sequencing for postgres

* small reorg of sequence consumer to repoll w/o pg notify saturation

* patch up a couple sqlite tests

* use a postgres sequence for repo-seq seqs

* increase sequence gap

* build branch w no migration

* Sequencer benching (#1359)

* benchmark sequencer

* perf improvements & refactor

* bugfix

* no optimistic backfill pagination

* tidy

* ready for merge

* add back in logging

* missing async

* tidy

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-07-24 19:53:06 -05:00
devin ivy 353ff19f99 Apply block rules for third parties on replies and embeds (#1378)
* apply block rules for third parties on replies and embeds

* tidy terminology around blocking and actor relationships

* apply third-party blocking functionality to appview

* tidy, bsky tests
2023-07-24 16:35:04 -04:00
Foysal Ahamed 4619e2b24a Disable signing up with invite code from takendown account (#1350) 2023-07-21 12:50:41 -04:00
devin ivy 8457800a3e Include takendown posts for admins (feature branch) (#1361)
* 🚧 WIP including takendown posts on author feed

*  Add takedown id on posts when including taken down posts

* 🧹 Cleanup the auth verifier and other bsky package code

*  Add test for admin getAuthorFeed

* 🧹 Cleanup lexicon and exclude takedownId

* more explicit plumbing for post hydration w/o requester or with takedown info

* pass along flag for soft-deleted actors

* cleanup getAuthorFeed w/ auth

* reorg getAuthorFeed logic around role/access-based auth

---------

Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
2023-07-21 12:47:03 -04:00
Eric Bailey b2ef3865bc Merge pull request #1365 from bluesky-social/eric/search-getPopularFeedGenerators
add optional `ILIKE` query to `getPopularFeedGenerators`
2023-07-20 11:00:51 -05:00
Eric Bailey 4a9a53866c fix lint 2023-07-20 10:46:00 -05:00
Eric Bailey 2e128869a4 handle sqlite too in pds only 2023-07-20 09:42:55 -05:00
Daniel Holmgren 2836ee38bb Appview - serve feed skeletons (#1265)
* proxy timeline skeleton construction to appview

* add getFeedSkeleton to appview

* mount route

* smart proxy feed skeletons

* tests

* proper error code

* only proxy specific feeds

* build branch

* update proxyable feed logic, should use feed publisher rather than generator

* fix feed proxy tests, configure feed publisher (in addition to generator)

* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)

* prevent user-supplied rkey on posts with createRecord

* allow empty-string rkey parameter

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* add slurs to reserved words (#1318)

* add slurs to reserved words (#1314)

* Update reserved.ts

Add slurs to reserved words

* Update reserved.ts

fix typo

* Update reserved.ts to clean up the slur list

* linting

* pluralise

---------

Co-authored-by: jess <jesslarkdev+gh@gmail.com>

* identifier: tweaks and additions to slur list (#1319)

* Refactor appview repo subscription for memleak (#1308)

* refactor to remove closure in loop

* move consecutive item out of p-queue

* Handle validation improvements (#1336)

* Handle matches and false positives for unacceptable words in handles

* move handle validation logic to pds

* missed merge

* add cfg flag

* encode lists

* fix build issues

* move words to cfg

* tidy

---------

Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>

* Allow moderators to take and reverse actor takedowns (#1330)

allow moderators to take and reverse actor takedowns

*  Allow searching reports by moderator did (#1283)

*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions

* v0.4.1

* Make sequencer leader behavior optional on pds (#1250)

* make sequencer leader behavior optional on pds

* tidy

* use 127.0.0.1 in with-test-db.sh for colima (#1297)

So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).

Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts.  See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)

To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.

(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)

* Subscription util tests (#1295)

* consecutive list tests

* flesh out subscription util tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Content reporting on record fields (#1351)

* content reporting on record fields

* fix test

* tests

* tidy

* Check rkey contents just for non-tids (#1353)

check rkey content for non-tids

*  Added new procedure for sending admin email (#1312)

* 🚧 Added new lexicon for sending admin email

*  Add moderation mailer

*  Switch to text email content from html

* 🧹 Cleanup some early implementation code and reflect PR reivew

*  Use smtp host instead of gmail service config

*  Move to using single smtp url

* v0.4.2

* Patch up a couple sqlite tests (#1355)

patch up a couple sqlite tests

* enable feeds & build branch

* disable branch building & enable without proxy header

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
2023-07-19 17:38:43 -05:00
Eric Bailey 77274b31a0 fix lint 2023-07-19 17:37:06 -05:00
Eric Bailey 64137c7e5c add optional ilike query to getPopularFeedGenerators 2023-07-19 17:25:49 -05:00
Daniel Holmgren bb607ccc55 Patch up a couple sqlite tests (#1355)
patch up a couple sqlite tests
2023-07-18 13:17:20 -05:00
Foysal Ahamed 775944e84a Added new procedure for sending admin email (#1312)
* 🚧 Added new lexicon for sending admin email

*  Add moderation mailer

*  Switch to text email content from html

* 🧹 Cleanup some early implementation code and reflect PR reivew

*  Use smtp host instead of gmail service config

*  Move to using single smtp url
2023-07-17 19:06:44 -04:00
devin ivy 47bf80646f Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
2023-07-17 18:38:05 -04:00
Daniel Holmgren c9acdd39a2 Content reporting on record fields (#1351)
* content reporting on record fields

* fix test

* tests

* tidy
2023-07-17 16:39:01 -05:00