1610 Commits

Author SHA1 Message Date
devin ivy
c2d0578b5a
Bsky appview sync service (#2031)
* init bsky-sync

* add bsync models and config

* rename bsky-sync to bsync

* protos and gen for bsync service

* start roughing-out bsync routes

* adjust bsync model, validation

* bsync auth, context, notify

* implement bsync scan mute ops, listen for mute op event

* setup basic bsync tests, misc fixes

* rename some files

* reorg bsync server routes

* reorg bsync server routes

* tests

* test input validation to addMuteOperation

* add db stats to bsync

* add bsync service

* redact bsync auth header from logs

* upgrade typescript to v5.3

* prettier on codegened bsync files
2024-01-09 15:40:35 -05:00
Daniel Holmgren
e41a25fa17
Automod: create reports using emitModerationAction (#2025)
* create reports using emitModerationAction

* fix input

* dont build
2024-01-08 17:49:31 -06:00
Daniel Holmgren
de2dbc2903
Split out moderation backend (#1970)
* mv appview

* copy

* finalize copy

* package names

* big WIP

* first pass at mod servce

* some tidy

* tidy & fix compiler errors

* rename to ozone, db migrations, add to dev-env & pds cfg

* getRecord & getRepo mostly working

* fix open handle

* get record tests all working

* moderation events working

* statuses working

* tidy test suite

* search repos

* server & db tests

* moderation tests

* wip daemon + push events

* pds fanout working

* fix db test

* fanning takedowns out to appview

* rm try/catch

* bsky moderation test

* introduce mod subject wrappers

* more tidy

* refactor event reversal

* tidy some db stuff

* tidy

* rename service to mod-service

* fix test

* tidy config

* refactor auth in bsky

* wip patching up auto-mod

* add label ingester in appview

* fix a couple build issues

* fix some timing bugs

* tidy polling logic

* fix up tests

* fix some pds tests

* eslint ignore

* fix ozone tests

* move seeds to dev-env

* move images around

* fix db schemas

* use service auth admin reqs

* fix remaining tests

* auth tests bsky

* another test

* random tidy

* fix up search

* clean up bsky mod service

* more tidy

* default attempts to 0

* tidy old test

* random tidy

* tidy package.json

* tidy logger

* takedownId -> takedownRef

* misc pr feedback

* split daemon out from ozone application

* fix blob takedown mgiration

* refactor ozone config

* do push event fanout on write instead of on read

* make suspend error work again

* add attempts check & add supporting index

* fix takedown test ref

* get tests working

* rm old test

* fix timing bug in event pusher tests

* attempt another fix for timing bug

* await req

* service files

* remove labelerDid cfg

* update snaps for labeler did + some cfg changes

* fix more snaps

* pnpm i

* build ozone images

* build

* make label provider optional

* fix build issues

* fix build

* fix build

* build pds

* build on ghcr

* fix syntax in entry

* another fix

* use correct import

* export logger

* remove event reverser

* adjust push event fanout

* push out multiple

* remove builds
2024-01-05 17:06:54 -06:00
github-actions[bot]
65254ab148
Version packages (#2021)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-01-04 19:55:23 -06:00
Eric Bailey
140677335f
Improve handling of resumeSession failures (#2010)
* Better handling of resumeSession errors

* Update test

* Format

* Pare back to only necessary

* Update handling for 500s

* Should really be update

* Update logic from feedback

* Update tests

* Feedback

* Revert debug change

* Changeset

* Bump minor
2024-01-04 19:50:59 -06:00
Devin Ivy
5560b7a25a Revert "lexicons: more string limits (#1994)"
This reverts commit ad0d976188d1f07401b9675b5c6045c91e82a84e.
2024-01-03 16:41:07 -05:00
Foysal Ahamed
5e7b0136da
Allow appealing a moderator decision through special report type (#1969)
*  Allow appealing a moderator decision through special report type

*  Allow querying subjects by appealed status

*  Move to appealed boolean state column

*  Remove leftover

*  Move appealed status to new boolean column

*  Throw when non-author attempts to appeal a subject

* 🚨 Appease the linter gods

* build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2024-01-02 19:17:42 -05:00
bnewbold
ad0d976188
lexicons: more string limits (#1994)
* limit external embed strings sizes

* set a (very large) alt text length limit

The motivation is to have *some* size limit on every string in post
records, to maximize interoperation. For example, we currently have a
CBOR library rejecting some records because of too-long strings.

We don't want to limit the ability of folks to be very descriptive in
alt text, specifically, so chose what seems to be a very large limit. If
this is not large enough, based on feedback, we can bump it even higher.
As context this is the largest string length limit in all of our
lexicons.

* make thumbnail URL fields format=uri

This mostly results in checks against the string being empty, or
unlimited size.

* codegen: string limits

* add changeset for string format lex updates
2024-01-03 00:24:32 +01:00
bnewbold
50f209e650
bsky: add top-level and robots.txt endpoints (#1979)
These provide helpful context (instead of just a generic 404).

The robots.txt in particular will show up for public.api.bsky.app
2023-12-21 18:01:21 +01:00
Foysal Ahamed
905743de9f
Optionally allow comment when sending email to be added to moderation event (#1981) 2023-12-20 09:49:21 -05:00
Foysal Ahamed
f365dc54dd
Create a mod event when automod applies labels to a record (#1927) 2023-12-20 09:47:51 -05:00
Daniel Holmgren
aadd35e1bd
Better handle 3rd party blocks in feeds (#1677)
* better handle 3rd party blocks inf eeds

* show anchor post in threads

* test
2023-12-19 11:43:16 -06:00
devin ivy
16b31e3ce2
Proper validation of entryway op in account creation (#1975)
* ensure entryway plc rotation key

* update plc lib on pds, test
2023-12-18 12:02:17 -05:00
devin ivy
80161e3c62
Optimize timeline for limit=1 (#1971)
* optimize timeline skeleton w/ limit=1

* build
2023-12-15 15:45:31 -05:00
Foysal Ahamed
7d818b8f19
Allow moderators along with admins to toggle invite codes (#1928)
*  Allow moderators along with admins to toggle invite codes

*  Adjust test to use triage token
2023-12-14 17:29:13 -05:00
github-actions[bot]
2012796c3b
Version packages (#1967)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-13 18:17:15 -08:00
Paul Frazee
8f3f43cb40
Fix: elevate !no-unauthenticated label to affect the entire account when on the profile (#1966)
* Fix: elevate !no-unauthenticated label to affect the entire account when on the profile

* lint

* Add changeset

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-12-13 18:13:19 -08:00
github-actions[bot]
8d4a87c0a8
Version packages (#1963)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-12 15:35:52 -08:00
Daniel Holmgren
89b4d3d354
Report separate error on getProfile for account suspension (#1960)
* report separate error on getProfile for account suspension

* Update packages/bsky/src/services/moderation/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-12-12 15:40:38 -06:00
Daniel Holmgren
0c54951f27
Add seenAt time to listNotifications (#1959) 2023-12-12 15:37:06 -06:00
github-actions[bot]
f3c1272e0e
Version packages (#1946)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-08 15:35:01 -06:00
Eric Bailey
ffe39aae83
Include own replies to own post roots via new filter posts_and_author_threads (#1776)
* Include own replies to own post roots in post_no_replies

* Updates

* Simplify

* snaps

* snaps

* Remove unused import

* Add new threads filter to getAuthorFeed

* Implement new filter

* Typo
2023-12-08 15:32:06 -06:00
github-actions[bot]
8f84207a75
Version packages (#1945)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-08 14:14:34 -06:00
Eric Bailey
60deea1762
Strip trailing colons from link detection, add test (#1944) 2023-12-08 11:44:43 -06:00
devin ivy
842e183925
Add labels to actor search typeahead results (#1940)
add labels to appview actor search typeahead
2023-12-06 19:21:06 -05:00
github-actions[bot]
cbc94aca13
Version packages (#1938)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-06 12:23:41 -08:00
Paul Frazee
cea5189c0c
Add !no-unauthenticated imperative label (#1926)
* Add !no-logged-out imperative label

* Add profile behavior docs and tests

* Tweak copy

* Prettier pass

* Rename no-logged-out label to no-authenticated
2023-12-06 12:15:37 -08:00
Daniel Holmgren
5038b50428
Fail open on did cache (#1934)
fail open on did cache
2023-12-05 18:08:41 -06:00
Daniel Holmgren
10f934b6c5
Appview rate limits parse cidr block (#1932)
appview rate limits parse cidr block
2023-12-05 17:18:24 -06:00
Daniel Holmgren
90564eeed2
Tweak rate limit cfg (#1931)
tweak rate limit cfg
2023-12-05 16:52:15 -06:00
Daniel Holmgren
e8473cb6be
Set trust proxy on appview for xff headers (#1930)
set trust proxy on appview
2023-12-05 16:37:47 -06:00
Daniel Holmgren
cf53d2025c
Transfer with-friends & best-of-follows to feed generator (#1919)
* transfer feeds

* remove unused feeds
2023-12-05 16:33:48 -06:00
Daniel Holmgren
af3e33a68b
Add basic application rate limits to appview (#1902)
* cache did docs in redis

* drop table

* expire from redis

* fix tests

* add cache class

* update api

* refactor

* filter negative labels

* fix up dev-env

* refactor did cache to use new redis cache class

* tidy

* ensure caching negatives

* redis cache tests

* remove timeout on did cache

* fix ns in test

* rename driver

* add timeout & fail open

* add test for timeout & fail open

* add basic rate limits

* tidy

* small pr feedback

* refactor caches

* bugfixg

* test for caching negative values

* little more to cache

* wire up cache cfg

* switch from redis scratch to redis

* fix build issues

* use different redis clients for tests

* fix test

* fix flaky test

* fix build issue

* use separate db for redis cache
2023-12-05 16:33:28 -06:00
Daniel Holmgren
3110a2a704
Cache tweaks (#1929)
* cache tweaks

* tweak

* another fix

* tweaks to expiry

* handle empty setMulti

* dont build
2023-12-05 15:57:05 -06:00
Daniel Holmgren
e566bef1cc
Cache labels in Redis (#1897)
* cache did docs in redis

* drop table

* expire from redis

* fix tests

* add cache class

* update api

* refactor

* filter negative labels

* fix up dev-env

* refactor did cache to use new redis cache class

* tidy

* ensure caching negatives

* redis cache tests

* remove timeout on did cache

* fix ns in test

* rename driver

* add timeout & fail open

* add test for timeout & fail open

* small pr feedback

* refactor caches

* bugfixg

* test for caching negative values

* little more to cache

* wire up cache cfg

* switch from redis scratch to redis

* fix build issues

* use different redis clients for tests

* fix test

* fix flaky test

* use separate db for redis cache
2023-12-05 13:53:05 -06:00
Foysal Ahamed
7e74f00a28
♻️ Cleanup linter warnings (#1907) 2023-12-05 19:42:29 +01:00
Matthieu Sieben
9cb81426dd
pre-compile handlebar templates at build time (#1833)
feat(pds): pre-compile handlebar templates at build time
2023-12-05 11:48:22 -05:00
Matthieu Sieben
d54a7e25e5
feat: infer definition type from "types" argument in getDefOrThrow (#1812)
feat(lex-cli): infer definition type from "types" argument in getDefOrThrow
2023-12-05 11:43:20 -05:00
devin ivy
49e7f98fdb
Config to start notifications daemon from a specific did (#1922)
config to start notifications daemon from a specific did
2023-12-05 10:39:57 -05:00
Daniel Holmgren
f9fd3e68ca
Feature branch: PDS v2 (#1789)
* cleanup repeat process all

* wip

* skip actor search test

* skip actor search test

* tweak processAll

* decrease wait to 1 sec

* repo_blob -> record_blob

* simplify backlink linkTo

* return repo_root to one row

* sequence before updating repo_root

* invite code forUser -> forAccount

* ipld_block -> repo_block

* use lru-cache fetchMethod

* move did_cache to own db

* better error handling on did cache

* drop did_handle

* fix sequencer wait time

* debug

* debug

* more debug

* check something

* fix bday paradox

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* user_account -> account

* remove inviteNote

* keypair per repo

* use an lru cache for keypairs as well

* clean up repo

* wip

* wrap up accoutn manager

* tidy

* tidy

* fix tests

* fix disabled codes

* fix appview tests

* add note

* set pragmas

* tidy account manager getDb

* rename pref transactor

* user pref -> account pref

* handle blob imports

* tidy imports

* add reserveSigningKey

* wip transferAccount

* clean up transferAccount

* tests

* tidy

* tidy

* configure entryway url on pds

* handle entryway in pds admin endpoints

* make importRepo temp

* fix imports

* make email optional on pds when using entryway

* handle diffs

* handle pds entryway usage for server, identity, admin endpoints

* pds support for credentials from entryway

* setup pds tests w/ entryway service

* tidy

* tidy

* update entryway version

* wip

* test handle updates w/ entryway

* split account table into two

* tidy

* tweak scripts

* tidy tests

* tidy

* better config for actorstore & dbs

* clean up cfg more

* reorg actorstore fs layout

* handle erros on actor db create

* pr tidy & fix accoutn deletion test

* pr feedback

* fix bad merge

* unskip test

* fix subscribe repos tests

* tidy repo root tables

* tidy

* fix tests

* tidy delete tokens

* tidy account getters

* tidy

* bulk deletesg

* increase chunk size

* handle racing refreshes

* wip

* fix auth test

* invert import flow

* clean up actor store on create account failure

* tweak sequencer

* prevent invite code races on createAccount

* rm note

* add back in race protection on getAccountInviteCodes

* start feature branch

* deleted app migration table

* patch up new auth test

* rm note

* g

* create accoutn delegated from entryway

* tidy

* fix test

* change plcOp type to unknown

* small fixes

* sync up w entryway branch

* Use proper error when authed account is not found (#1799)

provide proper error when account not found in access-takedown check

* build branch

* build on ghcr

* tweak service file

* tweak service file

* change where we save reserved keys

* no tmp dir in blobstore either

* fix blobstore temp location again

* handle repeat record_blobs

* create account before submitting plc op & undo if fail

* small tweak

* limit the number of local records

* push out empty commit on transfer

* fix issue with record_blob

* add push blob endpoint

* Set and validate token audiences on pds v2 (#1793)

set and validate token audience on pds v2

* merge

* include entryway did on tests

* build branch

* fix cache issue

* xrpc server blob limit

* put correct bytes

* add auth to routes

* handle quarantining/unquarantining a blob that does not exist

* tidy

* fix transfer tests

* fix email request routes for entryway

* PDS v2 entryway account deletion (#1819)

* add admin lexicon for account deletion

* implement admin account deletion endpoint

* fix entryway proxying on account email checks

* proxy to entryway for acct deletion

* read-after-write sanity check

* tweak

* wip

* finish refactor

* fix test schema

* application retry logic for busy

* pr feedback

* rm lru-cache

* fix test pg schema

* fix transfer test

* Sqlite instrumentation for pds v2 (#1838)

* sqlite instrumentation

* build

* remove build

* dont reimport blobs

* send ticks during import

* close on error

* catch handle validation error

* add log

* fix test

* return emailConfirmedAt on getAccountInfo

* Upgrade sharp on pds v2 (#1863)

upgrade sharp to 0.32.6

* read all bytes before parsing car

* Async car reader (#1867)

* asynchronously read in car

* dont buffer car

* tweak

* Gracefully handle indexing of invalid records (#1853)

* gracefully handle indexing of invalid records

* fix repo tests

* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)

fix role auth for access-or-role verifier, fix getBlob actor takedown check

* better cleanup of actor-stores

* add ability to not ensure leaves

* tidy

* allow did:web transfer

* Migration utility for actor-store (#1873)

beginnings of helper for migrating all actors

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* base case for findBlobRefs

* App-level retries for sqlite on pds (#1871)

* revamp retry helper to be more flexible re: backoff strategies

* sqlite timeout helper

* ensure sqlite wal on db creation/migration rather than every open

* layer retries for sqlite on writes outside transactions on pds

* tidy

* fix up lockfile

* tidy

* fix lex codegen

* fix timing bug in threadgate test

* No-op update handling (#1916)

do no produce commits on no-op updates

* Retry on all SQLITE_BUSY error codes (#1917)

retry on all sqlite_busy error codes

* Pds v2 ensure sqlite ready (#1918)

ensure sqlite is ready before making queries

* try something

* tidy

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-12-04 18:00:09 -06:00
devin ivy
cad30a7cc8
Cleanup outdated notifications in appview, add daemon for similar tasks (#1893)
* initial notification tidy logic

* helper for maintenance across all appview users

* tiny reorg

* add bsky daemon to tidy notifications

* tidy, add bsky daemon service entrypoint

* test notifs tidy daemon, add stats

* tidy

* crash failed notification daemon

* fix notification tidy constants
2023-12-01 19:01:31 -05:00
devin ivy
9cec13ee46
Do not generate notifs when post violates threadgate (#1901)
* do not generate notifs when post violates threadgate

* don't count threadgate-violating replies, style
2023-12-01 18:15:27 -05:00
github-actions[bot]
1f3fad2829
Version packages (#1909)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-01 18:14:01 -05:00
Daniel Holmgren
a4e5abf5f4
Fix snapshots for list items (#1911)
fix snapshots for list items
2023-12-01 15:07:17 -06:00
intrnl
8d9b1f70cd
Attach record URI to listItemView (#1758)
* feat: attach record uri to listitemview

* chore: attempt to update test snapshots
2023-12-01 14:28:54 -06:00
bnewbold
3be9c741ce
helpers for rkey and tid syntax; validate rkey at record creation time (#1738)
* syntax: fix jest config displayName

* syntax: TID validation

* syntax: add recordkey validation

* pds: verify rkey syntax at record creation time

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-12-01 14:28:47 -06:00
bnewbold
c17971a2d8
harden datetime verification (#1702)
* syntax: add datetime validator (and interop tests)

* syntax: improve datetime normalization

* lexicon: stronger datetime validation (from syntax package)

* syntax: make datetime syntax norm test more flexible

* make fmt

* datetime: docs, normalize and always variant

* bsky replace toSimplifiedISOSafe with normalizeDatetimeAlways

* more rigorous datetime parsing on record creation

* handle negative dates

* syntax: disallow datetimes before year 0010

* syntax: datetime normalization functions validate output

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-11-30 18:19:06 -06:00
Matthieu Sieben
691511b9b9
fix(debug): properly type debugCatch wrapper result (#1817) 2023-11-30 17:55:20 -06:00
Matthieu Sieben
2513bd1b26
style(xrpc-server): avoid un-neccessary "if" statement (#1826)
fix(xrpc-server): avoid un-neccessary "if" statement
2023-11-30 17:53:23 -06:00
Matthieu Sieben
2fc6ca54c1
perf(bsky): avoid re-creating auth functions on every request (#1822)
perf(bsky): avoid re-creating auth utilities on every request

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
2023-11-30 17:53:16 -06:00