615 Commits

Author SHA1 Message Date
Daniel Holmgren
89b4d3d354
Report separate error on getProfile for account suspension (#1960)
* report separate error on getProfile for account suspension

* Update packages/bsky/src/services/moderation/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-12-12 15:40:38 -06:00
Daniel Holmgren
0c54951f27
Add seenAt time to listNotifications (#1959) 2023-12-12 15:37:06 -06:00
github-actions[bot]
f3c1272e0e
Version packages (#1946)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-08 15:35:01 -06:00
Eric Bailey
ffe39aae83
Include own replies to own post roots via new filter posts_and_author_threads (#1776)
* Include own replies to own post roots in post_no_replies

* Updates

* Simplify

* snaps

* snaps

* Remove unused import

* Add new threads filter to getAuthorFeed

* Implement new filter

* Typo
2023-12-08 15:32:06 -06:00
github-actions[bot]
8f84207a75
Version packages (#1945)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-08 14:14:34 -06:00
devin ivy
842e183925
Add labels to actor search typeahead results (#1940)
add labels to appview actor search typeahead
2023-12-06 19:21:06 -05:00
github-actions[bot]
cbc94aca13
Version packages (#1938)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-06 12:23:41 -08:00
Daniel Holmgren
5038b50428
Fail open on did cache (#1934)
fail open on did cache
2023-12-05 18:08:41 -06:00
Daniel Holmgren
10f934b6c5
Appview rate limits parse cidr block (#1932)
appview rate limits parse cidr block
2023-12-05 17:18:24 -06:00
Daniel Holmgren
90564eeed2
Tweak rate limit cfg (#1931)
tweak rate limit cfg
2023-12-05 16:52:15 -06:00
Daniel Holmgren
e8473cb6be
Set trust proxy on appview for xff headers (#1930)
set trust proxy on appview
2023-12-05 16:37:47 -06:00
Daniel Holmgren
cf53d2025c
Transfer with-friends & best-of-follows to feed generator (#1919)
* transfer feeds

* remove unused feeds
2023-12-05 16:33:48 -06:00
Daniel Holmgren
af3e33a68b
Add basic application rate limits to appview (#1902)
* cache did docs in redis

* drop table

* expire from redis

* fix tests

* add cache class

* update api

* refactor

* filter negative labels

* fix up dev-env

* refactor did cache to use new redis cache class

* tidy

* ensure caching negatives

* redis cache tests

* remove timeout on did cache

* fix ns in test

* rename driver

* add timeout & fail open

* add test for timeout & fail open

* add basic rate limits

* tidy

* small pr feedback

* refactor caches

* bugfixg

* test for caching negative values

* little more to cache

* wire up cache cfg

* switch from redis scratch to redis

* fix build issues

* use different redis clients for tests

* fix test

* fix flaky test

* fix build issue

* use separate db for redis cache
2023-12-05 16:33:28 -06:00
Daniel Holmgren
3110a2a704
Cache tweaks (#1929)
* cache tweaks

* tweak

* another fix

* tweaks to expiry

* handle empty setMulti

* dont build
2023-12-05 15:57:05 -06:00
Daniel Holmgren
e566bef1cc
Cache labels in Redis (#1897)
* cache did docs in redis

* drop table

* expire from redis

* fix tests

* add cache class

* update api

* refactor

* filter negative labels

* fix up dev-env

* refactor did cache to use new redis cache class

* tidy

* ensure caching negatives

* redis cache tests

* remove timeout on did cache

* fix ns in test

* rename driver

* add timeout & fail open

* add test for timeout & fail open

* small pr feedback

* refactor caches

* bugfixg

* test for caching negative values

* little more to cache

* wire up cache cfg

* switch from redis scratch to redis

* fix build issues

* use different redis clients for tests

* fix test

* fix flaky test

* use separate db for redis cache
2023-12-05 13:53:05 -06:00
Foysal Ahamed
7e74f00a28
♻️ Cleanup linter warnings (#1907) 2023-12-05 19:42:29 +01:00
devin ivy
49e7f98fdb
Config to start notifications daemon from a specific did (#1922)
config to start notifications daemon from a specific did
2023-12-05 10:39:57 -05:00
Daniel Holmgren
f9fd3e68ca
Feature branch: PDS v2 (#1789)
* cleanup repeat process all

* wip

* skip actor search test

* skip actor search test

* tweak processAll

* decrease wait to 1 sec

* repo_blob -> record_blob

* simplify backlink linkTo

* return repo_root to one row

* sequence before updating repo_root

* invite code forUser -> forAccount

* ipld_block -> repo_block

* use lru-cache fetchMethod

* move did_cache to own db

* better error handling on did cache

* drop did_handle

* fix sequencer wait time

* debug

* debug

* more debug

* check something

* fix bday paradox

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* user_account -> account

* remove inviteNote

* keypair per repo

* use an lru cache for keypairs as well

* clean up repo

* wip

* wrap up accoutn manager

* tidy

* tidy

* fix tests

* fix disabled codes

* fix appview tests

* add note

* set pragmas

* tidy account manager getDb

* rename pref transactor

* user pref -> account pref

* handle blob imports

* tidy imports

* add reserveSigningKey

* wip transferAccount

* clean up transferAccount

* tests

* tidy

* tidy

* configure entryway url on pds

* handle entryway in pds admin endpoints

* make importRepo temp

* fix imports

* make email optional on pds when using entryway

* handle diffs

* handle pds entryway usage for server, identity, admin endpoints

* pds support for credentials from entryway

* setup pds tests w/ entryway service

* tidy

* tidy

* update entryway version

* wip

* test handle updates w/ entryway

* split account table into two

* tidy

* tweak scripts

* tidy tests

* tidy

* better config for actorstore & dbs

* clean up cfg more

* reorg actorstore fs layout

* handle erros on actor db create

* pr tidy & fix accoutn deletion test

* pr feedback

* fix bad merge

* unskip test

* fix subscribe repos tests

* tidy repo root tables

* tidy

* fix tests

* tidy delete tokens

* tidy account getters

* tidy

* bulk deletesg

* increase chunk size

* handle racing refreshes

* wip

* fix auth test

* invert import flow

* clean up actor store on create account failure

* tweak sequencer

* prevent invite code races on createAccount

* rm note

* add back in race protection on getAccountInviteCodes

* start feature branch

* deleted app migration table

* patch up new auth test

* rm note

* g

* create accoutn delegated from entryway

* tidy

* fix test

* change plcOp type to unknown

* small fixes

* sync up w entryway branch

* Use proper error when authed account is not found (#1799)

provide proper error when account not found in access-takedown check

* build branch

* build on ghcr

* tweak service file

* tweak service file

* change where we save reserved keys

* no tmp dir in blobstore either

* fix blobstore temp location again

* handle repeat record_blobs

* create account before submitting plc op & undo if fail

* small tweak

* limit the number of local records

* push out empty commit on transfer

* fix issue with record_blob

* add push blob endpoint

* Set and validate token audiences on pds v2 (#1793)

set and validate token audience on pds v2

* merge

* include entryway did on tests

* build branch

* fix cache issue

* xrpc server blob limit

* put correct bytes

* add auth to routes

* handle quarantining/unquarantining a blob that does not exist

* tidy

* fix transfer tests

* fix email request routes for entryway

* PDS v2 entryway account deletion (#1819)

* add admin lexicon for account deletion

* implement admin account deletion endpoint

* fix entryway proxying on account email checks

* proxy to entryway for acct deletion

* read-after-write sanity check

* tweak

* wip

* finish refactor

* fix test schema

* application retry logic for busy

* pr feedback

* rm lru-cache

* fix test pg schema

* fix transfer test

* Sqlite instrumentation for pds v2 (#1838)

* sqlite instrumentation

* build

* remove build

* dont reimport blobs

* send ticks during import

* close on error

* catch handle validation error

* add log

* fix test

* return emailConfirmedAt on getAccountInfo

* Upgrade sharp on pds v2 (#1863)

upgrade sharp to 0.32.6

* read all bytes before parsing car

* Async car reader (#1867)

* asynchronously read in car

* dont buffer car

* tweak

* Gracefully handle indexing of invalid records (#1853)

* gracefully handle indexing of invalid records

* fix repo tests

* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)

fix role auth for access-or-role verifier, fix getBlob actor takedown check

* better cleanup of actor-stores

* add ability to not ensure leaves

* tidy

* allow did:web transfer

* Migration utility for actor-store (#1873)

beginnings of helper for migrating all actors

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* base case for findBlobRefs

* App-level retries for sqlite on pds (#1871)

* revamp retry helper to be more flexible re: backoff strategies

* sqlite timeout helper

* ensure sqlite wal on db creation/migration rather than every open

* layer retries for sqlite on writes outside transactions on pds

* tidy

* fix up lockfile

* tidy

* fix lex codegen

* fix timing bug in threadgate test

* No-op update handling (#1916)

do no produce commits on no-op updates

* Retry on all SQLITE_BUSY error codes (#1917)

retry on all sqlite_busy error codes

* Pds v2 ensure sqlite ready (#1918)

ensure sqlite is ready before making queries

* try something

* tidy

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-12-04 18:00:09 -06:00
devin ivy
cad30a7cc8
Cleanup outdated notifications in appview, add daemon for similar tasks (#1893)
* initial notification tidy logic

* helper for maintenance across all appview users

* tiny reorg

* add bsky daemon to tidy notifications

* tidy, add bsky daemon service entrypoint

* test notifs tidy daemon, add stats

* tidy

* crash failed notification daemon

* fix notification tidy constants
2023-12-01 19:01:31 -05:00
devin ivy
9cec13ee46
Do not generate notifs when post violates threadgate (#1901)
* do not generate notifs when post violates threadgate

* don't count threadgate-violating replies, style
2023-12-01 18:15:27 -05:00
github-actions[bot]
1f3fad2829
Version packages (#1909)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-01 18:14:01 -05:00
Daniel Holmgren
a4e5abf5f4
Fix snapshots for list items (#1911)
fix snapshots for list items
2023-12-01 15:07:17 -06:00
intrnl
8d9b1f70cd
Attach record URI to listItemView (#1758)
* feat: attach record uri to listitemview

* chore: attempt to update test snapshots
2023-12-01 14:28:54 -06:00
bnewbold
c17971a2d8
harden datetime verification (#1702)
* syntax: add datetime validator (and interop tests)

* syntax: improve datetime normalization

* lexicon: stronger datetime validation (from syntax package)

* syntax: make datetime syntax norm test more flexible

* make fmt

* datetime: docs, normalize and always variant

* bsky replace toSimplifiedISOSafe with normalizeDatetimeAlways

* more rigorous datetime parsing on record creation

* handle negative dates

* syntax: disallow datetimes before year 0010

* syntax: datetime normalization functions validate output

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-11-30 18:19:06 -06:00
Matthieu Sieben
691511b9b9
fix(debug): properly type debugCatch wrapper result (#1817) 2023-11-30 17:55:20 -06:00
Matthieu Sieben
2fc6ca54c1
perf(bsky): avoid re-creating auth functions on every request (#1822)
perf(bsky): avoid re-creating auth utilities on every request

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
2023-11-30 17:53:16 -06:00
Daniel Holmgren
401538a933
Handle missing creator on lists and feed generators (#1906)
handle missing creator on lists and feed generators
2023-11-30 15:42:00 -06:00
Foysal Ahamed
4a5b65893d
Expose labels attached with legacy actions when events are queried and fix email event builder (#1905)
*  Expose labels attached with legacy actions when events are queried

* 🐛 Fix property name for modEventEmail

*  Only attach label vals for legacy actions if they exist

* 🐛 Set empty string as default subjectLine for email event

* 🧹 Cleaning up
2023-11-30 20:00:39 +01:00
Foysal Ahamed
1f9040a44d
Evented architecture for moderation system (#1617)
* 🚧 WIP with proposed lexicons for event based mod architecture

* 🚧 Remove unnecessary moderation action lexicon

* 🚧 Working on event based actions

*  Add escalated subject status

* 🐛 Alright, fixed the error in lexicon

* 🚧 Working through reversal

*  Cleanup build errors

*  Add subject status endpoint

*  Add handler

*  get reports from mod actions table

* :rightwards_twisted_arrows: Merge with upstream

* 🚧 Builds but test network doesnt start

*  Tests passing on event based status change

*  Rename index

* ♻️ Rename takeModerationAction->emitModerationEvent

*  Implement label reversal

*  Auto-revert test working

* ♻️  Refactored to event types and tests are passing

*  Add takedown event sequence validation

*  Adds support for blobCid status

* 🧹 Cleanup unnecessary method:

*  Hydrate handles with status and events

*  Re-implement auto reversal

*  Add takendown and mute filters

*  Allow filtering events by type

*  Allow filtering events by creator did

*  Add subjectStatus to record and repoview

*  Add persistent note feature

*  Log send email event

* 🐛 Fix logging send email event

*  Better type

*  Adjust migration to create separate moderation_event table

* 🧹 Cleanup types

*  Adjust tests with mod event emitter

*  Fix more tests around takedowns

*  Get test suite to pass

*  Get test suite to pass for pds

*  Get test suite to pass for pds

*  Update snapshot for feedgen

*  Why are more snapshots updating?

* ♻️ Rename getModerationEvents -> queryModerationEvents

* ♻️ Rename getModerationStatuses -> queryModerationStatuses

* ♻️ Rename persistNote->sticky

* 🐛 Rename subject

* ♻️ Cleanup expiresAt for scheduled actions

*  Add more tests, allow fetching mod history for all content by a user

*  Fix repo and record tests

*  Migrate reports and actions to events

* 🐛 Fix escalated status overwrite

*  Implement direct sql query to create events from actions and reports

* 🚧 Adding keyset pagination for subject statuses

*  Add migration for lastReportedAt

*  Migrate blob cids

*  Fix pagination on mod subject list endpoint

* 🐛 Fix blob actions

*  All tests passing on bsky package

*  Bring back snapshots

*  Skipping timeline test temporarily

*  Skipping some more tests to isolate failing ones

*  Bring back list-feed test

*  Bring back timeline test

*  Fix label action in seeding

*  Enable timeline proxied test

*  Enable search actor proxied test

*  Enable feedgen tests

*  Fix test for admin/get-record

*  Move note to comment for subject status

*  Accept comments in mute event

*  Remap flag event to ack event

* 🐛 Add legacyRef in report union selection

* @atproto/api 0.6.24-next.0

* @atproto/api 0.6.24-next.1

*  Adjust migration export and add index for blobCids column

*  Maintin action ids when migrating

*  Paginate events using createdAt timestamp

*  Update snapshot for pds test with events cursor update

*  Use only events for snapshot testing

*  Use only events for snapshot in the remaining test

* relative paths to lexicons for build

* fix bsky periodic event reversal in service entrypoint

*  Allow comments in takedown and label

*  Only import reports on consecutive run of the migration script

*  Adjust moderation property of blob entries

* determine latest reports to migrate

*  Process new reports for subject status

*  Process unresolved reports on first migration run

* fix transaction error, process just unresolved reports, make reported-at updates safe for reruns

* tidy

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-11-30 17:53:56 +01:00
Daniel Holmgren
7edad62c12
Put canReply state on post viewer state instead of thread viewer state (#1882)
* switch canReply from thread to post view

* tweaks & fix up tests

* update snaps

* fix more snaps

* hydrate feed items for getPosts & searchPosts

* fix another snapshot

* getPosts test

* canReply -> blockedByGate & DRY up code

* blockedByGate -> excludedByGate

* replyDisabled
2023-11-27 20:14:20 -06:00
devin ivy
95d33f7b11
Support unauthed usage of feeds (#1884)
* update local feedgens to not require a viewer where possible

* update getFeed to use optional auth

* test feeds w/ optional auth
2023-11-27 15:30:09 -05:00
github-actions[bot]
b532c502af
Version packages (#1877)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-11-21 15:47:43 -06:00
Emily Liu
59f70dbe80
Tweak lexicon descriptions (#1859)
* Tweak lexicon descriptions

* Tweak birthdate

* Update lexicons/app/bsky/actor/getProfiles.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* Regenerate packages from lexicons

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-11-20 17:41:25 -05:00
devin ivy
c28754f681
Upgrade sharp on pds and appview (#1864)
upgrade sharp to 0.32.6
2023-11-16 18:21:57 -05:00
Daniel Holmgren
e6fb39dc84
Bugfix: fetchLabels (#1862)
* fix fetch labels

* tidy

* update cast to parseInt
2023-11-16 12:27:46 -06:00
Daniel Holmgren
b05130db68
Add temp.fetchLabels method (#1858)
* add temp.fetchLabels route

* update description
2023-11-16 11:07:52 -06:00
Daniel Holmgren
1ef6da191b
Pass along hitsTotal and cursor from searchPostsSkeleton (#1857)
pass along hitsTotal and cursor from searchPostsSkeleton
2023-11-15 15:31:14 -06:00
Daniel Holmgren
599cb449b5
Bugfix: don't pass the viewer as a param to searchPosts skeleton (#1855)
dont pass the viewer as a param to searchPosts skeleton
2023-11-14 22:18:47 -06:00
Daniel Holmgren
a434d586dd
Add searchPosts api to appview (#1845)
* proxy search posts to search service

* add search posts proxy

* tidy

* add type annotations
2023-11-14 18:14:22 -06:00
Daniel Holmgren
74b7fdf754
Randomize suggestions (#1844)
* randomize suggestions

* fix snap

* cursor fix

* pr feedback
2023-11-14 18:14:08 -06:00
Foysal Ahamed
697f5d3627
Expose emailConfirmedAt field from admin getRepo (#1757)
*  Expose emailConfirmedAt field from admin getRepo

* ♻️ Fix typing for repo result

* 🧹 Cleanup unnecessary import

*  Adapt to the new pds based get account info method

* 🧹 Cleanup unused pds util
2023-11-08 00:26:31 +01:00
devin ivy
bebc4bac8c
Simplify moderation fanout to configured url (#1804)
* simplify moderation fanout to configured url

* fix bsky mod fanout tests
2023-11-05 00:33:14 -04:00
Daniel Holmgren
a3d81dd911
Prevent thread loops (#1813)
* prevent thread loops

* include original uri

* tidy

* tweak

* last tweak

* last tweak i swear

* wording
2023-11-03 16:55:50 -05:00
github-actions[bot]
b90f2c4ef8
Version packages (#1808)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-11-02 16:30:44 -04:00
devin ivy
772736a010
Add did doc output to getSession for session resumption (#1806)
* add optional did doc output to getSession lexicon

* add did doc output to getSession on pds, update client to use it

* api test fixes

* api changeset

* tidy
2023-11-02 16:16:26 -04:00
github-actions[bot]
3598898c03
Version packages (#1802)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-11-02 19:58:57 +00:00
bnewbold
84e2d4d2b6
Update atproto source license to MIT/Apache2 (#1788)
* update LICENSE files to MIT/Apache2

* README: update top-level license

* update package-level READMEs with license

* changeset: license update
2023-11-02 00:45:13 -07:00
Daniel Holmgren
cf848e87ab
Tweak schemas for entryway createAccount (#1797)
* tweak scheams

* require email & password
2023-11-01 14:56:57 -05:00
devin ivy
a161f815de
Make plc op unknown object rather than bytes (#1792)
make plc op unknown object rather than bytes
2023-11-01 11:54:00 -04:00
devin ivy
8637c367fe
Respect updated service auth keys (#1765)
* bust key cache when verifying service auth

* unit tests for xrpc auth

* fix

* support option for verifying non-low-s signatures

* fix verifyJwt tests
2023-10-31 18:09:02 -04:00