218 Commits

Author SHA1 Message Date
devin ivy
977202fa2b
Fix indexed record datetimes (#1009)
* Normalize record datetimes used for sort to simplified iso

* Update tests and snapshots for alternate valid datetime formats

* Fix notifications race during record update
2023-05-09 12:53:33 -04:00
Devin Ivy
3d1b3b3675 Add csp and nosniff when serving user uploads 2023-04-29 11:47:50 -04:00
Devin Ivy
01501f0d49 Fix snapshots 2023-04-25 17:33:29 -04:00
devin ivy
eb7667e4fc
Add labels onto record embeds (#854)
* Add lexicons for labeling embeds

* Hydrate labels onto record embeds in pds

* Support labels on embeds in bsky appview
2023-04-25 17:19:40 -04:00
Daniel Holmgren
68004c617f
Appview proxy (#827)
* fix up a couple of tsc errors in app view merge

* wip

* simple proxy

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* wip

* copy proxied tests & add mutes to getFollows/getFollowers

* tidy & add mutes to likes

* more routes + getAuthorFeed tests

* more testing

* tests for feed views

* thread testing

* finished tests for threads

* temporarily skip some tests

* cleaning up & updating test names

* separate db schema for appview

* rearrange

* typo

* add notifications

* re-enable notifs on pds appview

* update schemas

* updated some bsky snaps

* wip

* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* fix up & move to appCtx

* neat its working

* update bsky tests to new auth

* rm unused pds config var

* tidy

* check exp in seconds

* cache dids in postgres

* add migration & did-cache

* start tests

* couple helpers around cache invalidation

* fix expired check

* wip

* change cache semantics

* did cache testing

* do some cache revalidation in indexing

* fix config

* fix issue w did-resolver test-env prototype

* use map instead of record

* stale + expired

* tests

* clear entry method

* fix up build

* expired dids

* clear missing dids

* better verify payload

* bump test timeout

* fix notifs test

* fix up proxied actor search tests

* update snaps to include labels

* fix dev env

* fix up moderation route auth

* fix more auth headers

* fix auth on getPosts

* increase jest timeouts

* fix snaps
2023-04-25 15:26:14 -05:00
Daniel Holmgren
cae6779916
Fix leaky database transactions (#890)
* fix leaky txs

* tidy

* end tx in finally
2023-04-25 10:51:59 -05:00
devin ivy
0c02804a73
Ensure indexes are used on actor search queries (#879)
* In-progress work on optimizing actor search

* Tidy-up approach to using trgm index on actor search, apply to bsky appview

* Fix comment typo
2023-04-24 11:17:30 -04:00
Devin Ivy
92995859dc Fix bsky getPosts 2023-04-24 09:03:39 -04:00
Daniel Holmgren
553f59923c
Get posts method (#876)
* implement getPosts method

* tests

* bsky tests

* comment out test

* Handle stripping record-with-media viewer info in bsky tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-23 23:43:37 -05:00
devin ivy
47890d65b9
Add labeler functionality to the appview (#851)
* Add moderation and labeling model to bsky appview

* Carry over moderation service from pds

* Pass over bsky moderation service to get it working, missing blob support

* Carry over label service from pds to bsky

* Reorg bsky http auth, implement admin auth

* Carry over relevant admin endpoints pds to bsky

* Retrofit bsky admin routes from pds impls

* Implement resolve handle on appview

* Avoid loop in bsky handle resolution

* Add bsky appview to dev-env, opt-in

* Fix bsky searchRepos with empty term, tidy

* Include blobs on bsky admin views

* Stop resolving taken-down blobs on appview

* Tidy

* Carry over pds moderation tests to bsky

* Support image cache invalidation on bsky appview

* Add missing changes for bsky tests

* Test takedowns in bsky views

* Test takedowns on bsky notifs

* Separate blob resolution logic into standalone helper

* Carry over labeler from pds to bsky

* Update labelers for bsky usage

* Configure labeler on appview, tidy blob-resolver

* Add labels to views for posts and actors

* Update snapshots on bsky appview to include labels

* Carry over labeler tests from pds to bsky

* Tidy-up labeler tests for bsky appview

* Update bsky seeds and snapshots for keyword labeling

* Tidy

* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.

* Include labels on appview listNotifications, fix blob resolver content-type, tidy
2023-04-23 23:58:10 -04:00
devin ivy
05e6ebed63
Bsky appview admin and moderation endpoints (#840)
* Add moderation and labeling model to bsky appview

* Carry over moderation service from pds

* Pass over bsky moderation service to get it working, missing blob support

* Carry over label service from pds to bsky

* Reorg bsky http auth, implement admin auth

* Carry over relevant admin endpoints pds to bsky

* Retrofit bsky admin routes from pds impls

* Implement resolve handle on appview

* Avoid loop in bsky handle resolution

* Add bsky appview to dev-env, opt-in

* Fix bsky searchRepos with empty term, tidy

* Include blobs on bsky admin views

* Stop resolving taken-down blobs on appview

* Tidy

* Carry over pds moderation tests to bsky

* Support image cache invalidation on bsky appview

* Add missing changes for bsky tests

* Test takedowns in bsky views

* Test takedowns on bsky notifs

* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.
2023-04-23 23:43:08 -04:00
Daniel Holmgren
89109ef32d
Format commits outside of tx (#860)
* format commits outside of tx

* fix tests

* fix bsky test

* use persistent cache

* reuse repo storage between attempts to get caching benefits

* better error handling

* tests

* rm failing sqlite test

* pr feedback
2023-04-21 18:13:04 -05:00
Daniel Holmgren
c71e351c38
Do not give notifs for a deleted subject (#843)
generally do not give notifs for a deleted subject
2023-04-19 16:24:18 -05:00
devin ivy
c70cb2b6e5
Notifications from the appview (#829)
* Update notif lexicons for stateless seenAt param

* In-progress work on appview notifs

* Fix-up bsky notification methods

* Add appview notification table table

* Process notifications in bsky appview, test notif indexing

* Test bsky appview notification methods

* Tidy bsky notification tests

* Explicitly don't support seenAt notif params on pds

* Tidy bsky notifs tests

* Sync bsky notif handling with pds

* Remove stale comment

* Switch bsky notifs tests to use testenv
2023-04-18 17:55:03 -04:00
Daniel Holmgren
d4963e5003
Revamp dev env (#796)
* fix up a couple of tsc errors in app view merge

* wip

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* fixed missed merge conflict

* fix one more merge conflict

* fix popular test
2023-04-18 11:07:41 -05:00
devin ivy
6b55a95c88
Build timeline and author feeds from feed items index (#774)
* Index posts and reposts into feed_item table for building feeds

* Use feed_item table for building timeline and author feeds

* Apply feed item indexing to bsky app view

* Fix bsky appview tests, test getPopular

* Use feed item index to build feeds in bsky app view
2023-04-09 13:21:48 -07:00
Daniel Holmgren
8c71d0fe37
Cleanup appview merge (#772)
fix up a couple of tsc errors in app view merge
2023-04-07 10:44:00 -05:00
devin ivy
4e83748545
Bsky app view (#716)
* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect

* Cull bsky config, services, auth, etc.

* Sweep app view xrpc methods, tidy deps, add storage back for img server

* Run repo subscription on bsky app view

* Collapse db migrations down for bsky app view

* Tidy app view bin

* Remove mute functionality from app view, delegate to pds

* Initial tidy/culling of bsky app view tests

* Passing bsky app view db, server, and repo subscription tests

* Passing bsky app view duplicate-records tests

* Bsky app view test tidy/cull

* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.

* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.

* Setup actor handles by did in bsky app view

* Passing indexing tests on bsky app view

* Passing image tests on bsky app view

* Fix bsky actor reindexing, support custom lock id for testing repo subs

* Sweep bsky view tests, misc tests, passing

* Tidy bsky deps

* Include did in resized image uris

* Update bsky image process server to use getBlob

* Update image server tests, misc fixes

* Implement bsky blob resolver

* Wire local image processing server to local blob resolver, test blob resolver

* Tidy

* Tidy

* Tidy

* Tidy app view init

* Fix handle resolution, tidy

* Add utils for partitioning indexing by did

* Update repo sub to parallelize work per repo

* Dep tidy

* Tidy bsky tests for updated repo sub destroy()

* Update thead indexing to handle out-of-order posts

* Sketch out strategy in bsky for handling too-big commits

* Set content-type on sync.getBlob

* Add logging for failed transmissions in bsky blob resolver

* Tidy

* Tidy bsky repo indexing and supporting repo interfaces

* Sort in app view based on combo of creation and indexing times

* Fix types

* Add retry utils to bsky

* Add retries to http requests made by bsky

* Test repo indexing

* Update bsky db/model for lex refactor

* Update bsky lexicons for lex refactor

* Update bsky actor service for lex refactor

* Update bsky feed service for lex refactor

* Update bsky indexing service for lex refactor

* Update bsky repo subscription for lex refactor

* Tidy bsky repo sub

* Add unspecced endpoints to bsky app view, update entrypoint

* Update bsky xrpc utils for lex refactor

* Update bsky xrpc methods for lex refactor

* Update bsky test seeds for lex refactor, tidy api entrypoint

* Update bsky non-view tests for lex refactor

* Update bsky likes view test for lex refactor, minor fix

* Update bsky author feed tests for lex refactor, minor test util fix

* Update bsky follow, profile, repost, search view tests for lex refactor

* Update bsky timeline view tests for lex refactor

* Replace bsky out-of-order thread indexing logic

* Update bsky thread view tests for lex refactor, general test tidying

* Handle rebases and too-big commits in repo subscription, tracking commit data cid

* Tidy

* Ensure did resolver reports "not found" only when positively not found

* Handle tombstones and handle updates in bsky

* Test indexing handle updates and did tombstones

* Support cors on bsky

* Allow app view to serve most routes unauthed

* Tests for bsky unauthed views

* Tidy bsky service entrypoint and dockerfile

* Remove unused storage interfaces from bsky

* Bsky entrypoint and dockerfile fixes, tidy

* Add workflow for bsky build to aws

* Use more standard db env variables, make migration creds optional

* Make bsky repo subscription optional

* Fix lex->json serialization in bsky

* Split bsky actor sync state into its own table

* Skip invalid records on indexing full repo, tidy

* Tidy

* Leader test timing

* Tidy/lint

* Fix bsky config overrides
2023-04-06 18:47:46 -04:00