2067 Commits

Author SHA1 Message Date
Max Graey
7329c28564
Some improvements for tid.ts (#936)
* Some improvements for tid.ts

* refactor dedash helper

* tripple equal

* run prettify
2023-05-02 10:53:19 -05:00
bnewbold
5fd71c099b
appview: tweak repo error log line to not clobber 'service' (#891)
Somehow the output of this log line has been ending up in datadog under
the wrong 'service' (with the 'service' being an "wss://"-prefixed URL,
not "pds" or "bav" or whatever expected).

Tweaks the 'service' key, and some others that might have conflict.
2023-05-01 20:53:38 -05:00
Wes Bos
958c9ef6b3
add repository field to package.json (#895) 2023-05-01 20:43:32 -05:00
Roj
3370eac6ca
Improve readability in instanceof assertions (#909)
* Improve readability in `instanceof` assertions

* Remove unused import
2023-05-01 20:43:09 -05:00
Daniel Holmgren
12c6d886eb
Deterministic DNS handle resolution (#883)
* enforce constraint for one handle per domain

* tweak + tests
2023-05-01 20:27:12 -05:00
Daniel Holmgren
1e24d3e820
Include nsfw flag for getPopular (#949)
add includeNsfw flag for whats hot
2023-05-01 15:50:21 -05:00
Daniel Holmgren
bff96544d1
Tweak whats hot (#945)
* tweak whats hot

* couple fixes & test
2023-05-01 14:10:48 -05:00
bnewbold
0b70f50eb4
pds: iterated Hive label parsing (#941)
Tune "porn", "sexual", and "nudity" label categorization, including
re-structuring of code paths.

This comes out of some image testing as well as review of Hive API docs.
2023-05-01 11:20:50 -07:00
devin ivy
9366734f6d
More aggressive backpressure on websocket writer (#929)
* Implement more aggressive backpressure on websocket writer, may be temporary

* Add heap snapshot signal to pds
2023-05-01 08:48:45 -04:00
Devin Ivy
6a4258c973 Merge remote-tracking branch 'origin/blob-csp' 2023-04-29 17:24:51 -04:00
devin ivy
37df00d1e2
Add @ianklatzco to contributors file (#923)
Update CONTRIBUTORS.md

@devinivy

Co-authored-by: Ian Klatzco <iklatzco@gmail.com>
2023-04-29 12:33:41 -04:00
Devin Ivy
3d1b3b3675 Add csp and nosniff when serving user uploads 2023-04-29 11:47:50 -04:00
Paul Frazee
01b5971907
api@0.2.11, pds@0.1.5 (#925)
* @atproto/api@0.2.11

* @atproto/pds@0.1.5
2023-04-28 19:41:01 -05:00
dholms
87cc1267fd Remove temporary CI on blocks branch 2023-04-28 19:37:01 -05:00
Daniel Holmgren
db1042cf0d
Blocks (#922)
* block lexicons

* table schema

* block processing

* Actor-block migration

* Add blocked post lexicon to getPostThread

* getBlocks routes

* Add missing lexicons to bsky

* Add block state to actor views

* throw on blocked accounts in getAuthorFeed

* blcok state in profile views

* Update getPostThread to support block views, update existing test

* strip out blocks out oof timeline & popular

* more routes

* Test blocked posts in threads

* Support blocks on record embeds

* notifs

* Update pds test snapshots w/ block state

* fix search

* Update pds actor search snapshots w/ blocks

* test

* tests

* update lex

* fix up from returning blocking as record

* ci for branch

* fix build issue in bsky

* more blocks tests

* fix actor search test

* hook getBlocks up to router

* bugfix & actually test lol

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-28 19:36:02 -05:00
Daniel Holmgren
63bef856cc
Return on subscription abort (#921)
return on subscription abort
2023-04-27 23:34:03 -05:00
devin ivy
6b3d5c133e
Misc notification and suggestion optimizations (#920)
* Background notification creation/deletion

* Add todo

* Update tests for backgrounded notifs

* Add index to notifications supporting record deletion

* Use profile_agg to power getSuggestions

* Fetch record blocks separately from getting notification resultset

* Update background queue to drain pending tasks
2023-04-27 19:10:21 -04:00
devin ivy
da6ee3908a
Background notification creation and deletion (#919)
* Background notification creation/deletion

* Add todo

* Update tests for backgrounded notifs

* Undo backgrounding of notification creation

* Revert "Undo backgrounding of notification creation"

This reverts commit 507353a9802da75a232fc245ede30d47dab74e5f.
2023-04-27 17:12:18 -04:00
devin ivy
5249e52f36
Lengthen pass reset and invite codes (#912)
* Lengthen pass reset and invite codes, invalidation on reset

* Tidy

* Adjust token format for reset and deletion

* Update invite codes to use same token gen as pass reset and acct deletion
2023-04-27 14:26:31 -04:00
devin ivy
7d1243f53a
Limit concurrency of pds background tasks (#917) 2023-04-27 14:07:46 -04:00
Daniel Holmgren
384e739a3b
Allow admins to update handles to reserved handles (#916)
* allow admins to update handles to reserved handles

* tidy
2023-04-27 12:02:47 -05:00
devin ivy
4b70b80bd0
Log db and background queue stats (#908)
* Log db stats

* Log background queue stats
2023-04-26 19:09:24 -04:00
Daniel Holmgren
df7cbae4f2
Lower seq ranges on sequencer backfill (#907)
lower seq ranges on sequencer backfill
2023-04-26 18:24:37 -04:00
Paul Frazee
d2f1207b20
Publish @atproto/api@0.2.10 (#905)
@atproto/api@0.2.10
2023-04-26 15:46:46 -05:00
Paul Frazee
480a186c57 Merge branch 'thinkverse-add-shortcuts-to-bskyagent' into main 2023-04-26 15:44:41 -05:00
Paul Frazee
1656f6c5a8 Apply code formatting 2023-04-26 15:44:24 -05:00
Paul Frazee
23cf788216 Merge branch 'add-shortcuts-to-bskyagent' of https://github.com/thinkverse/atproto into thinkverse-add-shortcuts-to-bskyagent 2023-04-26 15:41:59 -05:00
Daniel Holmgren
ece4b61d50
Fix typo in appview (#899)
fix typo
2023-04-25 19:24:23 -05:00
devin ivy
e2d256404c
Finalize support for new moderation report reasons (#896) 2023-04-25 19:12:29 -05:00
Daniel Holmgren
4b7f7a5294
Appview proxying bugfixes (#897)
* fix some bugs in proxying reqs to appview

* update snap
2023-04-25 17:24:46 -05:00
Devin Ivy
01501f0d49 Fix snapshots 2023-04-25 17:33:29 -04:00
devin ivy
eb7667e4fc
Add labels onto record embeds (#854)
* Add lexicons for labeling embeds

* Hydrate labels onto record embeds in pds

* Support labels on embeds in bsky appview
2023-04-25 17:19:40 -04:00
Daniel Holmgren
68004c617f
Appview proxy (#827)
* fix up a couple of tsc errors in app view merge

* wip

* simple proxy

* use dev-env for appview tests

* process all in blob resolver

* another test fix

* wip

* copy proxied tests & add mutes to getFollows/getFollowers

* tidy & add mutes to likes

* more routes + getAuthorFeed tests

* more testing

* tests for feed views

* thread testing

* finished tests for threads

* temporarily skip some tests

* cleaning up & updating test names

* separate db schema for appview

* rearrange

* typo

* add notifications

* re-enable notifs on pds appview

* update schemas

* updated some bsky snaps

* wip

* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* fix up & move to appCtx

* neat its working

* update bsky tests to new auth

* rm unused pds config var

* tidy

* check exp in seconds

* cache dids in postgres

* add migration & did-cache

* start tests

* couple helpers around cache invalidation

* fix expired check

* wip

* change cache semantics

* did cache testing

* do some cache revalidation in indexing

* fix config

* fix issue w did-resolver test-env prototype

* use map instead of record

* stale + expired

* tests

* clear entry method

* fix up build

* expired dids

* clear missing dids

* better verify payload

* bump test timeout

* fix notifs test

* fix up proxied actor search tests

* update snaps to include labels

* fix dev env

* fix up moderation route auth

* fix more auth headers

* fix auth on getPosts

* increase jest timeouts

* fix snaps
2023-04-25 15:26:14 -05:00
Daniel Holmgren
cae6779916
Fix leaky database transactions (#890)
* fix leaky txs

* tidy

* end tx in finally
2023-04-25 10:51:59 -05:00
Devin Ivy
a8fa9b2b14 Merge remote-tracking branch 'origin/bsky-app-view' 2023-04-25 11:08:26 -04:00
bnewbold
af9f11fbde
'bsky' (appview) dev helpers (#888)
* remove commited .env files; exclude in gitignore

* bsky: distinct postgresql database in dev mode

* github CI: build and upload 'bsky' container to GHCR

This isn't how we deploy, but is useful for public access and things
like integration tests.

* Makefile: run-dev-appview
2023-04-24 16:27:34 -07:00
Kim Hallberg
8256df303f
Add getPosts shortcut 2023-04-25 00:29:13 +02:00
Kim Hallberg
3852b857d5
Add getSuggestions shortcut 2023-04-25 00:27:15 +02:00
Devin Ivy
0cc211b98a Fix appview dockerfile 2023-04-24 18:07:09 -04:00
Paul Frazee
02dcf7d891
@atproto/api@0.2.9 (#886) 2023-04-24 16:37:58 -05:00
dholms
b4dc8ab2b4 make sure we dont accept non-empty string 2023-04-24 14:22:41 -05:00
bnewbold
5137b427de
add BR and JP specific handles to reserved list (#878)
Co-authored-by: dholms <dtholmgren@gmail.com>
2023-04-24 12:39:35 -05:00
devin ivy
b7210bf0a1
Fix strict_word_similarity_threshold configuration (#880)
* In-progress work on optimizing actor search

* Tidy-up approach to using trgm index on actor search, apply to bsky appview

* Set trigram on session rather than on db via migration
2023-04-24 13:18:08 -04:00
devin ivy
0c02804a73
Ensure indexes are used on actor search queries (#879)
* In-progress work on optimizing actor search

* Tidy-up approach to using trgm index on actor search, apply to bsky appview

* Fix comment typo
2023-04-24 11:17:30 -04:00
Daniel Holmgren
a360cc6004
Chunk getRepo query (#877)
chunk get repo query
2023-04-24 10:09:19 -05:00
Devin Ivy
92995859dc Fix bsky getPosts 2023-04-24 09:03:39 -04:00
Daniel Holmgren
553f59923c
Get posts method (#876)
* implement getPosts method

* tests

* bsky tests

* comment out test

* Handle stripping record-with-media viewer info in bsky tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-23 23:43:37 -05:00
devin ivy
47890d65b9
Add labeler functionality to the appview (#851)
* Add moderation and labeling model to bsky appview

* Carry over moderation service from pds

* Pass over bsky moderation service to get it working, missing blob support

* Carry over label service from pds to bsky

* Reorg bsky http auth, implement admin auth

* Carry over relevant admin endpoints pds to bsky

* Retrofit bsky admin routes from pds impls

* Implement resolve handle on appview

* Avoid loop in bsky handle resolution

* Add bsky appview to dev-env, opt-in

* Fix bsky searchRepos with empty term, tidy

* Include blobs on bsky admin views

* Stop resolving taken-down blobs on appview

* Tidy

* Carry over pds moderation tests to bsky

* Support image cache invalidation on bsky appview

* Add missing changes for bsky tests

* Test takedowns in bsky views

* Test takedowns on bsky notifs

* Separate blob resolution logic into standalone helper

* Carry over labeler from pds to bsky

* Update labelers for bsky usage

* Configure labeler on appview, tidy blob-resolver

* Add labels to views for posts and actors

* Update snapshots on bsky appview to include labels

* Carry over labeler tests from pds to bsky

* Tidy-up labeler tests for bsky appview

* Update bsky seeds and snapshots for keyword labeling

* Tidy

* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.

* Include labels on appview listNotifications, fix blob resolver content-type, tidy
2023-04-23 23:58:10 -04:00
devin ivy
05e6ebed63
Bsky appview admin and moderation endpoints (#840)
* Add moderation and labeling model to bsky appview

* Carry over moderation service from pds

* Pass over bsky moderation service to get it working, missing blob support

* Carry over label service from pds to bsky

* Reorg bsky http auth, implement admin auth

* Carry over relevant admin endpoints pds to bsky

* Retrofit bsky admin routes from pds impls

* Implement resolve handle on appview

* Avoid loop in bsky handle resolution

* Add bsky appview to dev-env, opt-in

* Fix bsky searchRepos with empty term, tidy

* Include blobs on bsky admin views

* Stop resolving taken-down blobs on appview

* Tidy

* Carry over pds moderation tests to bsky

* Support image cache invalidation on bsky appview

* Add missing changes for bsky tests

* Test takedowns in bsky views

* Test takedowns on bsky notifs

* In bsky appview ensure label.neg is modeled as a boolean, not an integer bit. Add select column when checking blob takedown.
2023-04-23 23:43:08 -04:00
Daniel Holmgren
f30887bc1c
Clean up Did-resolver (#842)
* refactor did-resolver

* clean up deps

* some fixups + caching utilities

* couple helpers around cache invalidation

* fix expired check

* change cache semantics

* use map instead of record

* stale + expired

* tests

* clear entry method

* clear missing dids

* add missing type
2023-04-23 13:36:01 -05:00