2067 Commits

Author SHA1 Message Date
Daniel Holmgren
1ad6274202
Actor service fix (#1414)
* fix up bug in actor service

* add in hydrate methods

* build fix

* couple more fixes

* tidy

* port to appview

* dont build branch

* fix issue with items in list

* tidy

* dont build branch

* quick fix
2023-08-01 12:38:14 -05:00
bnewbold
3befcfe35e
pds: set default postgresql URL to localhost (#1415) 2023-07-31 21:53:45 -07:00
Emily Liu
43d6bcd77b
Merge pull request #1416 from bluesky-social/update-readme
Update readme
2023-07-31 21:06:32 -07:00
Emily Liu
d960aa66df Update readme 2023-07-31 21:04:57 -07:00
Daniel Holmgren
1b0a5716ce
Reflect takedowns in actor service (#1412)
reflect takedowns in actor service
2023-07-31 18:43:04 -05:00
Alona Enraght-Moony
74274c597d
lexicons: Remove empty array to make roundtriping work (#1407) 2023-07-31 16:35:05 -07:00
dumbmoron
850265d7bd
pds: actually validate tlds (#1408)
tlds have not been appropriately checked since 6b51ecb due to what appears to be a typo/oversight.
this commit fixes it so that tlds are once again validated
2023-07-31 17:39:53 -05:00
devin ivy
c060db24bd
Optimize com.atproto.sync.listRepos (#1391)
* add index to support listRepos

* disable contents of user-account-cursor-idx migration, build

* ready for merge

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-07-28 16:46:32 -05:00
Daniel Holmgren
7ef73467eb
Remove local whats hot from pds (#1393)
* put behind proxy header

* toggle for everyone

* tidy

* tidy

* fix
2023-07-27 19:05:12 -05:00
Daniel Holmgren
f293366e6a
Tweak timeline construction (#1367)
* add idx to post table & tweak feed construction

* comment out migration & build branch

* split out queries

* couple tweaks

* bugfix in with-friends

* lower feed data threshold

* return to subquery

* log injection on tracing

* Tweak timeline migraiton (#1369)

* v0.4.3

* tweak timeline migraiton

* misspelling

* move mutes & blocks to pds

* moar tweaks

* joins with 1 day cutoff

* no blocks/mutes on with friends

* attempt forcing custom plan

* increase feed date threshold

* trying something out

* trying something more

* tidy

* lower timeline date threshold

* tidy
2023-07-27 19:03:06 -05:00
Daniel Holmgren
030be22f60
Adjust bsky team feed (#1394)
* adjust bsky team feed

* tweak
2023-07-27 18:49:09 -05:00
Daniel Holmgren
7c20b01272
Proxy writes to appview (#1392)
* proxy writes to appview

* dont build branch
2023-07-27 13:10:14 -05:00
Daniel Holmgren
ca7b891ece
Log injection (#1384)
datadog inject logs
2023-07-25 22:43:15 -05:00
Patrick Linnane
acd5185984
Fix various typos (#1373) 2023-07-25 22:43:05 -05:00
Daniel Holmgren
a2655a4001
Move rkey check to prepareCreate (#1385)
* move rkey check to prepareCreate

* allow updates on applyWrites

* gen rkey in prepareCreate

* fix up tests
2023-07-25 22:41:10 -05:00
Daniel Holmgren
46beab62ee
Fix flaky partition queue test (#1386)
fix flaky partition queue test
2023-07-25 18:54:18 -05:00
devin ivy
d9f73769e7
Batch outgoing sequencing (#1298)
* batch outgoing sequencing for postgres

* small reorg of sequence consumer to repoll w/o pg notify saturation

* patch up a couple sqlite tests

* use a postgres sequence for repo-seq seqs

* increase sequence gap

* build branch w no migration

* Sequencer benching (#1359)

* benchmark sequencer

* perf improvements & refactor

* bugfix

* no optimistic backfill pagination

* tidy

* ready for merge

* add back in logging

* missing async

* tidy

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-07-24 19:53:06 -05:00
devin ivy
353ff19f99
Apply block rules for third parties on replies and embeds (#1378)
* apply block rules for third parties on replies and embeds

* tidy terminology around blocking and actor relationships

* apply third-party blocking functionality to appview

* tidy, bsky tests
2023-07-24 16:35:04 -04:00
Daniel Holmgren
23d701e723
Refactor appview media embeds (#1343)
* refactor appview services

* tidy types

* Allow moderators to take and reverse actor takedowns (#1330)

allow moderators to take and reverse actor takedowns

*  Allow searching reports by moderator did (#1283)

*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions

* v0.4.1

* Make sequencer leader behavior optional on pds (#1250)

* make sequencer leader behavior optional on pds

* tidy

* use 127.0.0.1 in with-test-db.sh for colima (#1297)

So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).

Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts.  See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)

To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.

(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)

* Subscription util tests (#1295)

* consecutive list tests

* flesh out subscription util tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Content reporting on record fields (#1351)

* content reporting on record fields

* fix test

* tests

* tidy

* Check rkey contents just for non-tids (#1353)

check rkey content for non-tids

*  Added new procedure for sending admin email (#1312)

* 🚧 Added new lexicon for sending admin email

*  Add moderation mailer

*  Switch to text email content from html

* 🧹 Cleanup some early implementation code and reflect PR reivew

*  Use smtp host instead of gmail service config

*  Move to using single smtp url

* v0.4.2

* Patch up a couple sqlite tests (#1355)

patch up a couple sqlite tests

* Increase appview keepalive (#1360)

increase keepalive to 90s

* Appview - serve feed skeletons (#1265)

* proxy timeline skeleton construction to appview

* add getFeedSkeleton to appview

* mount route

* smart proxy feed skeletons

* tests

* proper error code

* only proxy specific feeds

* build branch

* update proxyable feed logic, should use feed publisher rather than generator

* fix feed proxy tests, configure feed publisher (in addition to generator)

* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)

* prevent user-supplied rkey on posts with createRecord

* allow empty-string rkey parameter

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* add slurs to reserved words (#1318)

* add slurs to reserved words (#1314)

* Update reserved.ts

Add slurs to reserved words

* Update reserved.ts

fix typo

* Update reserved.ts to clean up the slur list

* linting

* pluralise

---------

Co-authored-by: jess <jesslarkdev+gh@gmail.com>

* identifier: tweaks and additions to slur list (#1319)

* Refactor appview repo subscription for memleak (#1308)

* refactor to remove closure in loop

* move consecutive item out of p-queue

* Handle validation improvements (#1336)

* Handle matches and false positives for unacceptable words in handles

* move handle validation logic to pds

* missed merge

* add cfg flag

* encode lists

* fix build issues

* move words to cfg

* tidy

---------

Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>

* Allow moderators to take and reverse actor takedowns (#1330)

allow moderators to take and reverse actor takedowns

*  Allow searching reports by moderator did (#1283)

*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions

* v0.4.1

* Make sequencer leader behavior optional on pds (#1250)

* make sequencer leader behavior optional on pds

* tidy

* use 127.0.0.1 in with-test-db.sh for colima (#1297)

So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).

Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts.  See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)

To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.

(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)

* Subscription util tests (#1295)

* consecutive list tests

* flesh out subscription util tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Content reporting on record fields (#1351)

* content reporting on record fields

* fix test

* tests

* tidy

* Check rkey contents just for non-tids (#1353)

check rkey content for non-tids

*  Added new procedure for sending admin email (#1312)

* 🚧 Added new lexicon for sending admin email

*  Add moderation mailer

*  Switch to text email content from html

* 🧹 Cleanup some early implementation code and reflect PR reivew

*  Use smtp host instead of gmail service config

*  Move to using single smtp url

* v0.4.2

* Patch up a couple sqlite tests (#1355)

patch up a couple sqlite tests

* enable feeds & build branch

* disable branch building & enable without proxy header

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>

* add optional ilike query to getPopularFeedGenerators

* fix lint

* fix lint

* handle sqlite too in pds only

* fix lint

* v0.4.3

* Include takendown posts for admins (feature branch) (#1361)

* 🚧 WIP including takendown posts on author feed

*  Add takedown id on posts when including taken down posts

* 🧹 Cleanup the auth verifier and other bsky package code

*  Add test for admin getAuthorFeed

* 🧹 Cleanup lexicon and exclude takedownId

* more explicit plumbing for post hydration w/o requester or with takedown info

* pass along flag for soft-deleted actors

* cleanup getAuthorFeed w/ auth

* reorg getAuthorFeed logic around role/access-based auth

---------

Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>

*  Disable signing up with invite code from takendown account (#1350)

* fix viewer data

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Eric Bailey <git@esb.lol>
2023-07-24 13:56:31 -05:00
Foysal Ahamed
4619e2b24a
Disable signing up with invite code from takendown account (#1350) 2023-07-21 12:50:41 -04:00
devin ivy
8457800a3e
Include takendown posts for admins (feature branch) (#1361)
* 🚧 WIP including takendown posts on author feed

*  Add takedown id on posts when including taken down posts

* 🧹 Cleanup the auth verifier and other bsky package code

*  Add test for admin getAuthorFeed

* 🧹 Cleanup lexicon and exclude takedownId

* more explicit plumbing for post hydration w/o requester or with takedown info

* pass along flag for soft-deleted actors

* cleanup getAuthorFeed w/ auth

* reorg getAuthorFeed logic around role/access-based auth

---------

Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
2023-07-21 12:47:03 -04:00
dholms
354e05225b v0.4.3 2023-07-20 12:35:42 -05:00
Eric Bailey
b2ef3865bc
Merge pull request #1365 from bluesky-social/eric/search-getPopularFeedGenerators
add optional `ILIKE` query to `getPopularFeedGenerators`
2023-07-20 11:00:51 -05:00
Eric Bailey
4a9a53866c fix lint 2023-07-20 10:46:00 -05:00
Eric Bailey
2e128869a4 handle sqlite too in pds only 2023-07-20 09:42:55 -05:00
Eric Bailey
0b33d30140 fix lint 2023-07-19 17:47:32 -05:00
Daniel Holmgren
2836ee38bb
Appview - serve feed skeletons (#1265)
* proxy timeline skeleton construction to appview

* add getFeedSkeleton to appview

* mount route

* smart proxy feed skeletons

* tests

* proper error code

* only proxy specific feeds

* build branch

* update proxyable feed logic, should use feed publisher rather than generator

* fix feed proxy tests, configure feed publisher (in addition to generator)

* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)

* prevent user-supplied rkey on posts with createRecord

* allow empty-string rkey parameter

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* add slurs to reserved words (#1318)

* add slurs to reserved words (#1314)

* Update reserved.ts

Add slurs to reserved words

* Update reserved.ts

fix typo

* Update reserved.ts to clean up the slur list

* linting

* pluralise

---------

Co-authored-by: jess <jesslarkdev+gh@gmail.com>

* identifier: tweaks and additions to slur list (#1319)

* Refactor appview repo subscription for memleak (#1308)

* refactor to remove closure in loop

* move consecutive item out of p-queue

* Handle validation improvements (#1336)

* Handle matches and false positives for unacceptable words in handles

* move handle validation logic to pds

* missed merge

* add cfg flag

* encode lists

* fix build issues

* move words to cfg

* tidy

---------

Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>

* Allow moderators to take and reverse actor takedowns (#1330)

allow moderators to take and reverse actor takedowns

*  Allow searching reports by moderator did (#1283)

*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions

* v0.4.1

* Make sequencer leader behavior optional on pds (#1250)

* make sequencer leader behavior optional on pds

* tidy

* use 127.0.0.1 in with-test-db.sh for colima (#1297)

So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).

Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts.  See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)

To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.

(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)

* Subscription util tests (#1295)

* consecutive list tests

* flesh out subscription util tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Content reporting on record fields (#1351)

* content reporting on record fields

* fix test

* tests

* tidy

* Check rkey contents just for non-tids (#1353)

check rkey content for non-tids

*  Added new procedure for sending admin email (#1312)

* 🚧 Added new lexicon for sending admin email

*  Add moderation mailer

*  Switch to text email content from html

* 🧹 Cleanup some early implementation code and reflect PR reivew

*  Use smtp host instead of gmail service config

*  Move to using single smtp url

* v0.4.2

* Patch up a couple sqlite tests (#1355)

patch up a couple sqlite tests

* enable feeds & build branch

* disable branch building & enable without proxy header

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
2023-07-19 17:38:43 -05:00
Eric Bailey
77274b31a0 fix lint 2023-07-19 17:37:06 -05:00
Eric Bailey
64137c7e5c add optional ilike query to getPopularFeedGenerators 2023-07-19 17:25:49 -05:00
Daniel Holmgren
8c32ed363b
Increase appview keepalive (#1360)
increase keepalive to 90s
2023-07-19 10:03:01 -04:00
Daniel Holmgren
bb607ccc55
Patch up a couple sqlite tests (#1355)
patch up a couple sqlite tests
2023-07-18 13:17:20 -05:00
Devin Ivy
754ad7f4f9 v0.4.2 2023-07-17 19:10:42 -04:00
Foysal Ahamed
775944e84a
Added new procedure for sending admin email (#1312)
* 🚧 Added new lexicon for sending admin email

*  Add moderation mailer

*  Switch to text email content from html

* 🧹 Cleanup some early implementation code and reflect PR reivew

*  Use smtp host instead of gmail service config

*  Move to using single smtp url
2023-07-17 19:06:44 -04:00
devin ivy
47bf80646f
Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
2023-07-17 18:38:05 -04:00
Daniel Holmgren
c9acdd39a2
Content reporting on record fields (#1351)
* content reporting on record fields

* fix test

* tests

* tidy
2023-07-17 16:39:01 -05:00
Daniel Holmgren
8c9becc063
Subscription util tests (#1295)
* consecutive list tests

* flesh out subscription util tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-07-17 15:07:35 -04:00
Jeff Hodges
73a9cd50be
use 127.0.0.1 in with-test-db.sh for colima (#1297)
So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).

Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts.  See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)

To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.

(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)
2023-07-17 14:51:28 -04:00
devin ivy
1ebda12b9d
Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds

* tidy
2023-07-17 10:53:22 -04:00
Devin Ivy
670daf5edd v0.4.1 2023-07-15 10:32:45 -07:00
Foysal Ahamed
b9ca76f012
Allow searching reports by moderator did (#1283)
*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions
2023-07-14 16:05:56 -07:00
devin ivy
53c4956146
Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
2023-07-14 14:37:32 -07:00
Daniel Holmgren
6b51ecbbb2
Handle validation improvements (#1336)
* Handle matches and false positives for unacceptable words in handles

* move handle validation logic to pds

* missed merge

* add cfg flag

* encode lists

* fix build issues

* move words to cfg

* tidy

---------

Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
2023-07-13 22:29:01 -05:00
Daniel Holmgren
89f21e9571
Refactor appview repo subscription for memleak (#1308)
* refactor to remove closure in loop

* move consecutive item out of p-queue
2023-07-13 13:35:29 -05:00
bnewbold
0bac4e7752
identifier: tweaks and additions to slur list (#1319) 2023-07-12 22:37:00 -07:00
Daniel Holmgren
3d47eed592
add slurs to reserved words (#1318)
* add slurs to reserved words (#1314)

* Update reserved.ts

Add slurs to reserved words

* Update reserved.ts

fix typo

* Update reserved.ts to clean up the slur list

* linting

* pluralise

---------

Co-authored-by: jess <jesslarkdev+gh@gmail.com>
2023-07-12 23:38:15 -05:00
David Buchanan
c78d30f363
hotfix: prevent user-supplied rkey on posts with createRecord (#1313)
* prevent user-supplied rkey on posts with createRecord

* allow empty-string rkey parameter

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-07-12 01:00:26 -05:00
Daniel Holmgren
5e34a8bfc6
Add websocket heartbeat (#1294)
* refactor & add heartbeat to websockets

* comment

* testing

* tidy

* pr feedback

* pass ws ref to startHeartbeat

* tidy

* end interval on ws close

* rm only

* build branch

* dont build branch
2023-07-07 13:52:29 -05:00
devin ivy
2e52f382a1
Perform pessimistic handle check when updating handle (#1300)
perform pessimistic handle check when updating handle
2023-07-07 13:49:47 -05:00
Ansh
08dc2b725d
Pagination on getPopularFeedGenerators (#1293)
* add temporary page-based pagination

* use cursor & limit for pagination

* tests

* add to appview

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-07-07 13:49:02 -05:00
devin ivy
b92288cfca
Sync-up appview feed date threshold w/ pds (#1286)
sync-up appview feed date threshold w/ pds
2023-07-06 23:31:11 -04:00