1610 Commits

Author SHA1 Message Date
devin ivy
47bf80646f
Check rkey contents just for non-tids (#1353)
check rkey content for non-tids
2023-07-17 18:38:05 -04:00
Daniel Holmgren
c9acdd39a2
Content reporting on record fields (#1351)
* content reporting on record fields

* fix test

* tests

* tidy
2023-07-17 16:39:01 -05:00
Daniel Holmgren
8c9becc063
Subscription util tests (#1295)
* consecutive list tests

* flesh out subscription util tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-07-17 15:07:35 -04:00
Jeff Hodges
73a9cd50be
use 127.0.0.1 in with-test-db.sh for colima (#1297)
So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).

Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts.  See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)

To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.

(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)
2023-07-17 14:51:28 -04:00
devin ivy
1ebda12b9d
Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds

* tidy
2023-07-17 10:53:22 -04:00
Devin Ivy
670daf5edd v0.4.1 2023-07-15 10:32:45 -07:00
Foysal Ahamed
b9ca76f012
Allow searching reports by moderator did (#1283)
*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions
2023-07-14 16:05:56 -07:00
devin ivy
53c4956146
Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
2023-07-14 14:37:32 -07:00
Daniel Holmgren
6b51ecbbb2
Handle validation improvements (#1336)
* Handle matches and false positives for unacceptable words in handles

* move handle validation logic to pds

* missed merge

* add cfg flag

* encode lists

* fix build issues

* move words to cfg

* tidy

---------

Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
2023-07-13 22:29:01 -05:00
Daniel Holmgren
89f21e9571
Refactor appview repo subscription for memleak (#1308)
* refactor to remove closure in loop

* move consecutive item out of p-queue
2023-07-13 13:35:29 -05:00
bnewbold
0bac4e7752
identifier: tweaks and additions to slur list (#1319) 2023-07-12 22:37:00 -07:00
Daniel Holmgren
3d47eed592
add slurs to reserved words (#1318)
* add slurs to reserved words (#1314)

* Update reserved.ts

Add slurs to reserved words

* Update reserved.ts

fix typo

* Update reserved.ts to clean up the slur list

* linting

* pluralise

---------

Co-authored-by: jess <jesslarkdev+gh@gmail.com>
2023-07-12 23:38:15 -05:00
David Buchanan
c78d30f363
hotfix: prevent user-supplied rkey on posts with createRecord (#1313)
* prevent user-supplied rkey on posts with createRecord

* allow empty-string rkey parameter

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-07-12 01:00:26 -05:00
Daniel Holmgren
5e34a8bfc6
Add websocket heartbeat (#1294)
* refactor & add heartbeat to websockets

* comment

* testing

* tidy

* pr feedback

* pass ws ref to startHeartbeat

* tidy

* end interval on ws close

* rm only

* build branch

* dont build branch
2023-07-07 13:52:29 -05:00
devin ivy
2e52f382a1
Perform pessimistic handle check when updating handle (#1300)
perform pessimistic handle check when updating handle
2023-07-07 13:49:47 -05:00
Ansh
08dc2b725d
Pagination on getPopularFeedGenerators (#1293)
* add temporary page-based pagination

* use cursor & limit for pagination

* tests

* add to appview

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-07-07 13:49:02 -05:00
devin ivy
b92288cfca
Sync-up appview feed date threshold w/ pds (#1286)
sync-up appview feed date threshold w/ pds
2023-07-06 23:31:11 -04:00
devin ivy
d7f87418a5
Tidy repo sub usage of indexing service (#1296)
* use single indexing service in repo sub

* tidy
2023-07-06 19:44:52 -05:00
devin ivy
0d3a5559cf
Do not sequence handle updates in slow path (#1292)
* do not sequence handle updates in slow path

* build

* bail early if invite code doesn't exist
2023-07-06 15:52:16 -04:00
devin ivy
3ea892bc57
Add administrative triage role, update moderator role (#1216)
* setup triage user on pds, update moderator username (invalidates old token)

* initial pass on triage access on pds, limit access to email addrs

* apply moderator vs triage rules on taking and reversing mod actions for pds

* update pds tests for triage auth role

* setup moderator and triage roles on bsky appview

* apply mod and triage access rules to bsky admin endpoints

* reframe admin auth as role-based auth, tidy auth apis

* tidy

* build

* revert change to basic auth username for role-based auth
2023-07-06 15:18:52 -04:00
Daniel Holmgren
60e8284b15
Fix flaky db test (#1287)
fix flaky db test
2023-07-06 09:57:41 -04:00
Daniel Holmgren
7e0201d4d6
Bump package versions (#1288)
* v0.3.0

* bump common-web to 0.2.0

* v0.2.0

* v0.2.0

* v0.1.0

* v0.1.0

* v0.1.0

* v0.2.0

* v0.4.0

* v0.1.0

* v0.2.0

* v0.2.0

* v0.3.0

* v0.2.0
2023-07-05 20:54:26 -05:00
Daniel Holmgren
fde94736bc
Desanctify hellthread (#1289)
* remove blessed hellthread

* tweak hot classic labels
2023-07-05 20:40:49 -05:00
Daniel Holmgren
05a7c46192
Add migrations for handmade indices (#1266)
* add indices

* add record index to bsky

* sync-up bsky index migration, remove duplicate index

* backpressure on bsky backfill indexing (#1268)

* backpressure on bsky backfill indexing

* skip image resolution for text labeler

* increase background queue concurrency for backfill

* tidy

* Proxy timeline skeleton construction (#1264)

proxy timeline skeleton construction to appview

* Only pass through known params on timeline skeleton (#1270)

only pass through own params

* Require headers on getRecord proxy (#1271)

require headers on getRecord proxy

* Add boolean for enabling generic appview proxying (#1273)

* add boolean config for enabling generic proxying

* tweak

* tweak cfg var name

* tweak

* 🐛 Only ignore reports for specific at-uri when ignoreSubject contains at-uri (#1251)

* Move timeline construction to appview (#1274)

full switch timeline construction to appview

* Better propagate errors on repo streams (#1276)

better propgate errors on repo streams

* Log pds sequencer leader stats (#1277)

log pds sequencer leader stats

* Explicit dns servers (#1281)

* add ability to setup explicit dns servers

* cleanup

* fix

* reorder

* pr feedback

* Thread through id-resolver cfg (#1282)

thread through id-resolver-cfg

* Bsky log commit error (#1275)

* don't bail on bad record index

* add build

* temporarily disable check, full reindex on rabase

* don't bail on bad record index during rebase, track last commit on rebase

* log bsky repo subscription stats

* add running and waiting count to repo sub stats

* re-enable fast path for happy rebases

* only hold onto seq in cursor consecutivelist, don't hold onto whole completed messages

* Misc scaling (#1284)

* limit backsearch to 1 day instead of 3

* lower like count threshold

* bump to 6

* disable like count check

* disable with friends

* preemptively cache last commit

* inline list mutes

* actor service

* label cache

* placehodler on popular with friends

* bulk sequence

* no limit but chunk

* bump chunk to 5k

* try 10k

* fix notify

* tweaking

* syntax

* one more fix

* increase backfill allowance

* full refresh label cache

* limit 1 on mute list

* reserve aclu handle

* clean up testing with label cache

* note on with-friends

* rm defer from label cache

* label cache error handling

* rm branch build

* build appview

* update indices

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
2023-07-05 17:22:53 -05:00
Daniel Holmgren
4d1f8d3289
Misc scaling (#1284)
* limit backsearch to 1 day instead of 3

* lower like count threshold

* bump to 6

* disable like count check

* disable with friends

* preemptively cache last commit

* inline list mutes

* actor service

* label cache

* placehodler on popular with friends

* bulk sequence

* no limit but chunk

* bump chunk to 5k

* try 10k

* fix notify

* tweaking

* syntax

* one more fix

* increase backfill allowance

* full refresh label cache

* limit 1 on mute list

* reserve aclu handle

* clean up testing with label cache

* note on with-friends

* rm defer from label cache

* label cache error handling

* rm branch build
2023-07-05 16:41:08 -05:00
devin ivy
bb2848e7f5
Bsky log commit error (#1275)
* don't bail on bad record index

* add build

* temporarily disable check, full reindex on rabase

* don't bail on bad record index during rebase, track last commit on rebase

* log bsky repo subscription stats

* add running and waiting count to repo sub stats

* re-enable fast path for happy rebases

* only hold onto seq in cursor consecutivelist, don't hold onto whole completed messages
2023-07-05 15:01:13 -04:00
Daniel Holmgren
4f7fd8b118
Thread through id-resolver cfg (#1282)
thread through id-resolver-cfg
2023-07-04 20:58:06 -05:00
Daniel Holmgren
0ceed96b11
Explicit dns servers (#1281)
* add ability to setup explicit dns servers

* cleanup

* fix

* reorder

* pr feedback
2023-07-04 18:40:43 -05:00
devin ivy
dd4d91ce6b
Log pds sequencer leader stats (#1277)
log pds sequencer leader stats
2023-07-04 00:15:13 -04:00
Daniel Holmgren
24be348dfb
Better propagate errors on repo streams (#1276)
better propgate errors on repo streams
2023-07-03 20:07:20 -05:00
Daniel Holmgren
8815efd86e
Move timeline construction to appview (#1274)
full switch timeline construction to appview
2023-07-03 15:03:04 -05:00
Foysal Ahamed
03200c1d8b
🐛 Only ignore reports for specific at-uri when ignoreSubject contains at-uri (#1251) 2023-07-03 15:21:16 -04:00
Daniel Holmgren
c793ff9103
Add boolean for enabling generic appview proxying (#1273)
* add boolean config for enabling generic proxying

* tweak

* tweak cfg var name

* tweak
2023-07-03 14:14:49 -05:00
Daniel Holmgren
9610ba061c
Require headers on getRecord proxy (#1271)
require headers on getRecord proxy
2023-07-03 13:19:56 -05:00
Daniel Holmgren
6695e6c9e4
Only pass through known params on timeline skeleton (#1270)
only pass through own params
2023-07-03 12:38:33 -05:00
Daniel Holmgren
e7a0d27f1f
Proxy timeline skeleton construction (#1264)
proxy timeline skeleton construction to appview
2023-07-03 11:28:39 -05:00
devin ivy
7637fdbf10
backpressure on bsky backfill indexing (#1268)
* backpressure on bsky backfill indexing

* skip image resolution for text labeler

* increase background queue concurrency for backfill

* tidy
2023-07-03 09:48:27 -04:00
Daniel Holmgren
4db7aeca57
Rework record embeds (#1262)
* wip

wip

wip

fix tests

tidy

* small fix

* fix flaky proxy test
2023-07-02 20:37:27 -05:00
Daniel Holmgren
e5dc790eda
Ensure not to serve repo endpoints for taken-down actor (#1263)
ensure not to serve repo endpoints for taken-down actor

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-07-02 20:22:35 -05:00
devin ivy
2e4a114379
Remove post hierarchy indexing from bsky appview (#1257)
* remove post_hierarchy from db model and indexing in bsky appview

* update bsky appview getPostThread to use recursive query to build thread

* add covering index to speed-up descendents query

* tidy post/notification processing w/o post_hierarchy

* tidy, disallow infinitely following reply cycles
2023-06-30 18:18:04 -04:00
Daniel Holmgren
ee68a4037b
Appview - Invalid handles (#1244)
* run with-test-db in monorepo test

* improve pg script

* tidy

* namespace bsky pg schemas

* differentiate schemas

* clean up script

* first pass at invalid handles in appview

* tests for handle invalidation

* move mock to instance instead of prototype

* change network mocks in general

* fixing pagination on actor-search

* fix snap & normalize handles on index

* handling did pagination + update tests

* one last update
2023-06-30 16:02:47 -05:00
Daniel Holmgren
8f8f2c4b73
Prevent duplicate blocks in checkout (#1256)
* prevent duplicate cids in checkout

* tidy
2023-06-29 19:33:04 -05:00
Daniel Holmgren
e81ff25f02
Throw concurrent write error on prev mismatch (#1254)
* throw concurrent write error

* build branch

* remove branch build
2023-06-29 15:44:19 -05:00
Daniel Holmgren
7cb8c62b54
Repo update advisory lock (#1230)
* use for no key update

* use a tx advisory lock for repo updates

* skip tests for sqlite

* move check of commit swap on rebase

* do lock before formatting rebase

* hash schema in for lock id

* no tx lock in sqlite

* move rebase formatting to tx

* move dialect check

* rm log

* make the lock ids a bit safer

* change how we do lock id

* refactor id generator
2023-06-29 13:42:10 -05:00
Daniel Holmgren
9263ddc64d
Add compression in http res (#1022)
* add compression

* compress carfiles

* update package version

* add tests

* one more test

* remove compressible dep

* add gzip to appview

* fix package.json
2023-06-29 10:24:48 -05:00
Daniel Holmgren
ab6621e017
Ensure rebase recovers repos in a broken state (#1247)
rebase recovers from repos in a broken state
2023-06-29 10:24:13 -05:00
Daniel Holmgren
207d44796d
Fix dev env package (#1245)
* v0.2.1

* fix dev-env publish

* v0.2.2

* fix package.json
2023-06-28 16:38:26 -05:00
Daniel Holmgren
ba870d3dbe
Run test DB for full monorepo tests (#1225)
* run with-test-db in monorepo test

* improve pg script

* tidy

* namespace bsky pg schemas

* differentiate schemas

* clean up script

* tweaking script
2023-06-28 15:12:37 -05:00
dholms
b852d2e3b4 publish bsky 2023-06-27 19:45:43 -05:00
dholms
09385bbb55 v0.2.0 2023-06-27 19:35:26 -05:00