27 Commits

Author SHA1 Message Date
rafael
638f5a8312
Fix avatar path resolution in dev-env (#3266)
* Fix avatar path resolution in dev-env

* changeset

* extract dev-env assets to dedicated folder

* add comment

* fix fmt
2024-12-19 11:25:41 -03:00
Matthieu Sieben
b934b396b1
Client SDK rework (#2483)
* feat(api): support creation of oauth based AtpAgents

* oauth: misc fixes for confidential clients

* fix(xprc): remove ReadableStream.from polyfill

* OAuth docs tweaks (#2679)

* OAuth: clarification about client_name being shown

* OAuth: re-write handle resolution privacy concern

* avoid relying on ReadableStream.from in xrpc-server tests

* feat(oauth-types): expose "ALLOW_UNSECURE_ORIGINS" constant

* feat(handle-resolver): expose "AtprotoIdentityDidMethods" type

* fix(oauth-client): ensure that the oauth metadata document contains client_id_metadata_document_supported

* fix(oauth-types): prevent unknown query string in loopback client id

* fix(identity-resolver): check that handle is in did doc's "alsoKnownAs"

* feat(oauth-client:oauth-resolver): allow logging in using either the PDS URL or Entryway URL

* fix(oauth-client): return better error in case of invalid "oauth-protected-resource" status code

* refactor(did): group atproto specific checks in own

* feat(api): relax typing of "appLabelers" and "labelers" AtpClient properties

* allow any did as labeller (for tests mainly)

* fix(api): allow to override "atproto-proxy" on a per-request basis

* remove release candidate versions from changelog

* update changeset for api and xrpc packages

* Add missing changeset

* revert RC versions

* Proper wording in OAUTH.md api example

* remove "pre" changeset file

* xrpc: restore original behavior of setHEader and unsetHeader

* docs: add comment for XrpcClient 's constructor arg

* feat(api): expose "schemas" publicly

* feat(api): allow customizing the whatwg fetch function of the AtpAgent

* docs(api): improve migration docs

* docs: change reference to BskyAgent to AtpAgent

* docs: mention the breaking change regarding setSessionPersistHandler

* fix(api): better split AtpClient concerns

* fix(xrpc): remove unused import

* refactor(api): simplify class hierarchu by removeing AtpClient

* fix(api): mock proper method for facets detection

* restore ability to restore session asynchronously

* feat(api): allow instantiating Agent with same argument as super class

* docs(api): properly extend Agent class

* style(xrpc): var name

* docs(api): remove "async" to header getter

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Hailey <me@haileyok.com>
2024-08-12 19:57:21 +02:00
Daniel Holmgren
5643fa46b5
Account event (#2269)
* subscribe repos lex: new #account event type

* lex: getAccountStatus endpoint

* lex: add account status errors to sync methods

* tweak type of token union

* fix getAccountStatus parameter name

* codegen

* Account -> Repo

* re-codegen

* update errors in sync methods

* add getRepoStatus route

* add account events to sequencer

* emit account evts

* fix test + small bugfixin

* handle evt on bsky side

* codegen

* loggable message

* schema tweaks

* build errors & tidy

* tidy account deactivation tests

* more subscribe repos tests

* identity evt tests + tidy

* return optional did doc on identity evts

* update identity evt

* update impl for identity evt handles

* add description to handle field

* status on listRepos & notate deprecated firehose events

* return status on listRepos

* pr feedback

* tests

* fix account deletion test

* fix more tests

* fix type error in repo.test

---------

Co-authored-by: bryan newbold <bnewbold@robocracy.org>
2024-05-30 11:37:52 -05:00
Daniel Holmgren
71f9cc9d4f
Clean up role-based auth (#2255)
* tidy bsky auth

* hook up new auth verifier

* update auth throughout ozone

* handle mod signing keys

* add client proxy heads to pds

* hook up rest of routes

* simplify pipethrough & add some SSRF protection

* tests

* fix bad var

* remove basic auth in ozone

* wip

* fix key parsing in pds

* fix up all ozone tests

* fix admin auth test

* rename test

* fix ozone test

* clean up tokens in pds

* fix up pds tests

* fix up ozone tests

* add pipethrough to write routes

* reenable proxied admin test

* add moderator accounts to ozone in dev-env

* update did doc id values

* null creds string -> `none`

* fix fetchLabels auth check

*  Add a couple more proxied requests that we use in ozone ui

* Add runit to the services/bsky Dockerfile (#2254)

add runit to the services/bsky Dockerfile

* Improve tag detection (#2260)

* Allow tags to lead with and contain only numbers

* Break tags on other whitespace characters

* Export regexes from rich text detection

* Add test

* Add test

* Disallow number-only tags

* Avoid combining enclosing screen chars

* Allow full-width number sign

* Clarify tests

* Fix punctuation edge case

* Reorder

* Simplify, add another test

* Another test, comment

* Version packages (#2261)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* 🐛 Increment attempt count after each attempt to push ozone event (#2239)

* Ozone delegates email sending to actor's pds (#2272)

* ozone delegates email sending to user's pds

* lexicon: add content field to mod email event

* test email sending via mod event

* add dev dep for nodemailer in ozone

* fix auth verifier method

* build branch

* build branch

* fix url check

* better error handling for get account infos

* fix labeler service id

* fix iss on auth headers

* fix dev-env ozone did

* fix tests & another jwt issuer

* fix proxy auth

* ozone: fix ip check

* fix aud check on pds mod service auth

* tidy

* Update packages/pds/tests/proxied/admin.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix pipethrough of headers

* fix moderation status tests

* fix auth on ozone routes

* update iss on daemon

---------

Co-authored-by: Foysal Ahamed <foysal@blueskyweb.xyz>
Co-authored-by: Jake Gold <52801504+Jacob2161@users.noreply.github.com>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin ivy <devinivy@gmail.com>
2024-03-06 19:53:52 -06:00
Daniel Holmgren
30b05a7d4b
Account migration (#2179)
* draft of account migration lexicons

* format

* clean up schemas

* codegen

* second pass on schemas

* small fix

* move around checkImportStatus

* re-codegen

* getServiceAuth

* getServiceAuth impl

* importRepo impl

* handle uploadBlob for import

* allow bringing your own did on createAccount

* working on test flow

* fleshing out flow

* fix up sendPlcOP

* small fixes

* activate/deactivate account

* full flow working!

* schema tweaks

* format

* update schemas

* moar codegen

* match impl to new schemas

* email flow for signed plc operation

* add email flow for plc operations

* impl plc op request email

* proxy to entryway

* tidy activate account

* integrating account deactivated state

* fix up tests

* friendly parse on optional did auth

* admin activate/deactivate routes

* proxy relevant requests to entryway

* remove admin activation routes

* do not proxy acitvaition to entryway

* cfg for disallowing imports

* buff up test

* refactor listMissingBlobs a bit

* add validDid & activated to accoutn status

* emit event on account activation

* test creating a post after migrating

* account deactivation tests

* test name

* tests on plc operations

* fix recommended did creds

* codegen

* turn off accepting imports on createAccount

* undo prev change

* increment version

* build branch

* pr feedback

* handle errs in p-queue

* handle blob upload outside of txn

* Clean old temp account migration lexicons (#2187)

* clean old temp lexicons

* rm old test

* fix agent session test

* fix bsky test

* dont build branch
2024-02-20 19:29:49 -06:00
Daniel Holmgren
de2dbc2903
Split out moderation backend (#1970)
* mv appview

* copy

* finalize copy

* package names

* big WIP

* first pass at mod servce

* some tidy

* tidy & fix compiler errors

* rename to ozone, db migrations, add to dev-env & pds cfg

* getRecord & getRepo mostly working

* fix open handle

* get record tests all working

* moderation events working

* statuses working

* tidy test suite

* search repos

* server & db tests

* moderation tests

* wip daemon + push events

* pds fanout working

* fix db test

* fanning takedowns out to appview

* rm try/catch

* bsky moderation test

* introduce mod subject wrappers

* more tidy

* refactor event reversal

* tidy some db stuff

* tidy

* rename service to mod-service

* fix test

* tidy config

* refactor auth in bsky

* wip patching up auto-mod

* add label ingester in appview

* fix a couple build issues

* fix some timing bugs

* tidy polling logic

* fix up tests

* fix some pds tests

* eslint ignore

* fix ozone tests

* move seeds to dev-env

* move images around

* fix db schemas

* use service auth admin reqs

* fix remaining tests

* auth tests bsky

* another test

* random tidy

* fix up search

* clean up bsky mod service

* more tidy

* default attempts to 0

* tidy old test

* random tidy

* tidy package.json

* tidy logger

* takedownId -> takedownRef

* misc pr feedback

* split daemon out from ozone application

* fix blob takedown mgiration

* refactor ozone config

* do push event fanout on write instead of on read

* make suspend error work again

* add attempts check & add supporting index

* fix takedown test ref

* get tests working

* rm old test

* fix timing bug in event pusher tests

* attempt another fix for timing bug

* await req

* service files

* remove labelerDid cfg

* update snaps for labeler did + some cfg changes

* fix more snaps

* pnpm i

* build ozone images

* build

* make label provider optional

* fix build issues

* fix build

* fix build

* build pds

* build on ghcr

* fix syntax in entry

* another fix

* use correct import

* export logger

* remove event reverser

* adjust push event fanout

* push out multiple

* remove builds
2024-01-05 17:06:54 -06:00
Daniel Holmgren
f9fd3e68ca
Feature branch: PDS v2 (#1789)
* cleanup repeat process all

* wip

* skip actor search test

* skip actor search test

* tweak processAll

* decrease wait to 1 sec

* repo_blob -> record_blob

* simplify backlink linkTo

* return repo_root to one row

* sequence before updating repo_root

* invite code forUser -> forAccount

* ipld_block -> repo_block

* use lru-cache fetchMethod

* move did_cache to own db

* better error handling on did cache

* drop did_handle

* fix sequencer wait time

* debug

* debug

* more debug

* check something

* fix bday paradox

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* user_account -> account

* remove inviteNote

* keypair per repo

* use an lru cache for keypairs as well

* clean up repo

* wip

* wrap up accoutn manager

* tidy

* tidy

* fix tests

* fix disabled codes

* fix appview tests

* add note

* set pragmas

* tidy account manager getDb

* rename pref transactor

* user pref -> account pref

* handle blob imports

* tidy imports

* add reserveSigningKey

* wip transferAccount

* clean up transferAccount

* tests

* tidy

* tidy

* configure entryway url on pds

* handle entryway in pds admin endpoints

* make importRepo temp

* fix imports

* make email optional on pds when using entryway

* handle diffs

* handle pds entryway usage for server, identity, admin endpoints

* pds support for credentials from entryway

* setup pds tests w/ entryway service

* tidy

* tidy

* update entryway version

* wip

* test handle updates w/ entryway

* split account table into two

* tidy

* tweak scripts

* tidy tests

* tidy

* better config for actorstore & dbs

* clean up cfg more

* reorg actorstore fs layout

* handle erros on actor db create

* pr tidy & fix accoutn deletion test

* pr feedback

* fix bad merge

* unskip test

* fix subscribe repos tests

* tidy repo root tables

* tidy

* fix tests

* tidy delete tokens

* tidy account getters

* tidy

* bulk deletesg

* increase chunk size

* handle racing refreshes

* wip

* fix auth test

* invert import flow

* clean up actor store on create account failure

* tweak sequencer

* prevent invite code races on createAccount

* rm note

* add back in race protection on getAccountInviteCodes

* start feature branch

* deleted app migration table

* patch up new auth test

* rm note

* g

* create accoutn delegated from entryway

* tidy

* fix test

* change plcOp type to unknown

* small fixes

* sync up w entryway branch

* Use proper error when authed account is not found (#1799)

provide proper error when account not found in access-takedown check

* build branch

* build on ghcr

* tweak service file

* tweak service file

* change where we save reserved keys

* no tmp dir in blobstore either

* fix blobstore temp location again

* handle repeat record_blobs

* create account before submitting plc op & undo if fail

* small tweak

* limit the number of local records

* push out empty commit on transfer

* fix issue with record_blob

* add push blob endpoint

* Set and validate token audiences on pds v2 (#1793)

set and validate token audience on pds v2

* merge

* include entryway did on tests

* build branch

* fix cache issue

* xrpc server blob limit

* put correct bytes

* add auth to routes

* handle quarantining/unquarantining a blob that does not exist

* tidy

* fix transfer tests

* fix email request routes for entryway

* PDS v2 entryway account deletion (#1819)

* add admin lexicon for account deletion

* implement admin account deletion endpoint

* fix entryway proxying on account email checks

* proxy to entryway for acct deletion

* read-after-write sanity check

* tweak

* wip

* finish refactor

* fix test schema

* application retry logic for busy

* pr feedback

* rm lru-cache

* fix test pg schema

* fix transfer test

* Sqlite instrumentation for pds v2 (#1838)

* sqlite instrumentation

* build

* remove build

* dont reimport blobs

* send ticks during import

* close on error

* catch handle validation error

* add log

* fix test

* return emailConfirmedAt on getAccountInfo

* Upgrade sharp on pds v2 (#1863)

upgrade sharp to 0.32.6

* read all bytes before parsing car

* Async car reader (#1867)

* asynchronously read in car

* dont buffer car

* tweak

* Gracefully handle indexing of invalid records (#1853)

* gracefully handle indexing of invalid records

* fix repo tests

* Fix role auth for access-or-role verifier, getBlob check on actor takedowns (#1869)

fix role auth for access-or-role verifier, fix getBlob actor takedown check

* better cleanup of actor-stores

* add ability to not ensure leaves

* tidy

* allow did:web transfer

* Migration utility for actor-store (#1873)

beginnings of helper for migrating all actors

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* base case for findBlobRefs

* App-level retries for sqlite on pds (#1871)

* revamp retry helper to be more flexible re: backoff strategies

* sqlite timeout helper

* ensure sqlite wal on db creation/migration rather than every open

* layer retries for sqlite on writes outside transactions on pds

* tidy

* fix up lockfile

* tidy

* fix lex codegen

* fix timing bug in threadgate test

* No-op update handling (#1916)

do no produce commits on no-op updates

* Retry on all SQLITE_BUSY error codes (#1917)

retry on all sqlite_busy error codes

* Pds v2 ensure sqlite ready (#1918)

ensure sqlite is ready before making queries

* try something

* tidy

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-12-04 18:00:09 -06:00
Daniel Holmgren
fcb19c9c51
Simplify PDS moderation (#1723)
* spec out new simple pds mod routes

* introduce new admin state endpoints

* wire up routes

* clean up pds

* revoke refresh tokens

* getUserAccountInfo

* pr tidy

* fixing some tests

* fixing up more tests

* fanout takedowns to pds

* fanout admin reqs to pds

* tidy

* more tidy & add more pds moderation tests

* getUserAccountInfo -> getAccountInfo

* dont hydrate pds info on searchRepos

* fix build

* port admin tests to bsky package

* clean up old snaps

* tests on fanout

* tweak naming

* missed a rename

* tidy renames

* fix lex name

* tidy & move snap

* fix build

* cleanup repeat process all

* skip actor search test

* fix bday paradox

* tidy up pds service auth

* rm skipped test

* retry http

* tidy

* improve fanout error handling

* fix test

* return signing key in did-web

* more tests

* tidy serivce auth checks

* change takedownId col to takedownRef

* build branch

* fix bsky test

* add service key to indexer

* move signing key to api entry

* dont build
2023-10-30 16:56:17 -05:00
Daniel Holmgren
d664b51c64
Finalize PDS in-process AppView removal (#1198)
* rm tables

* rm event-stream & proxied

* Remove appview services, move label service to pds

* only proxy appview stuff

* delete more tables

* Start removing message dispatched from pds

* more syncing-up removal of message dispatcher in pds

* merged

* remove feedgens from pds, remove getPopular

* remove unused image helper from pds

* fixing compiler errors

* clean up sharp

* rm label service

* first pass on cleaning up tests

* fix up a bunch of tests

* moderation view tests

* last admin tests

* got a lil overzealous in deletes

* clean up unused cfg

* clean up label table

* simplify admin repo search query/logic

* tidy pds entrypoint

* in-progress pds config changes

* cfg fiddling

* finish cleaning up cfg/ctx

* comments

* building

* pds prefix on env

* test env

* collapse pds migrations down into a single migration

* fix up dev-env

* tidy

* cleanup

* fix pds admin tests

* fix handle test

* fix pds proxy tests

* fix subscribe repos test

* fix sqlite config in pds tests

* add sqlite clause in sequencer-leader

* fix actor search w/ sqlite on pds

* fixes

* fix dev env build

* update pds service entrypoint

* simple env example

* make takedown ids opaque identifiers in the pds

* use pds routes for api tests

* update pds dockerfile with volume and correct port env var

* add a couple env vars to example

* add comments to env example

* @atproto/pds 0.2.0-beta.0

* @atproto/aws 0.0.1-beta.0

* appview did

* @atproto/aws 0.0.1

* enable logs by default

* update env example

* bugfixing sandbox issues

* consistency in pds env var name for appview url

* log on pds start and stop, configure version at runtime

* @atproto/pds 0.2.0-beta.1

* fix semver matching for pds beta version

* v0.2.0-beta.2

* default invites to being not required

* fix flaky test

* limit db connections in tests

* publish 0.2.0-beta.d3

* fix invite required parsing

* @atproto/pds 0.2.0-beta.5

* Proxy getPopularFeedGenerators on simplified pds (#1222)

proxy getPopularFeedGenerators on pds

Co-authored-by: dholms <dtholmgren@gmail.com>

* tidy migrations

* fix service entry

* bump version

* change auth order

* bump version

* bump version

* add upgradeRepoVersion & fallback url for cdn

* bump version

* merging

* merge pds

* building dev-env

* merging tests

* merge service entry

* test fixing

* tidy

* fix admin search

* tidy

* tidy

* add snap for getListFeed

* add backup nameserver cfg

* tidy + pr feedback

* tidy

* tidy env

* bit more

* re-add dotenv to root package.json

* fix dep

* build branch

* fix tests

* Refactor tests to make better use of dev-env (#1690)

* refactor pds tests to use dev env

* refactor bsky tests

* fix pds test

* tidy bsky tests

* build pds correctly

* fix entry point

* default logging to false (for now)

* format service entry

* Switch takedown ids back to ints on pds distribution (#1694)

* switch takedown ids back to ints, consistent with live pds

* tidy/fix migration

* update migration for sqlite

* export moderation action reversal

* takedown tests

* dont build branch

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-10-02 13:27:45 -05:00
Daniel Holmgren
11e3b329e3
Clean up other email tokens (#1572)
* lexicons

* codegen

* email templates

* request routes

* impl

* migration

* tidy

* tests

* tidy & bugfixes

* format

* fix api test

* fix auth test

* impl

* update constraint name

* temporarily disable unconfirmed updates

* tidy

* fix some tests

* validate email syntax
2023-09-28 13:25:56 -05:00
Daniel Holmgren
2fa9088639
Disable pds appview indexing (#1645)
* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build
2023-09-25 20:09:52 -05:00
Daniel Holmgren
558981ead1
Disable pds appview routes (#1644)
* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch
2023-09-22 13:51:15 -05:00
bnewbold
bd3966555f
@atproto/syntax: merge of @atproto/identifier, uri, and nsid (#1464)
* move nsid package in to identifier

* move uri package in to identifier

* update packages to pull from identifier, not nsid or uri

* rename @atproto/identifier to @atproto/syntax

* update all refs from @atproto/identifer to @atproto/syntax
2023-08-25 10:18:10 -05:00
Foysal Ahamed
d614678c49
Revert labels on takedown reversal (#1496)
*  Revert labels on takedown reversal

*  Add tests to cover label reversal

*  Add label reversal test to bsky package

*  Move xrpc call out of db transaction
2023-08-24 11:27:33 -04:00
Foysal Ahamed
7bc701e2e1
Allow creating moderation action with a duration (#1431)
*  Add action duration to action model

*  Add periodic moderation action reversal

*  Use actionDurationInHours and remove takedownExpiresAt

*  Destroy mod action reversal job

*  Add test for automatic moderation action reversal

* 🧹 Port over actionDuration to bsky package

* ♻️ Better naming for action duration and minor fixes

* ♻️ Optionally build expiresAt based on createdAt when logging action

* ♻️ Adjust constructor params

*  Copy over migration from pds to bsky package

* 🚨 Fix indentation

*  Propagate action reversal from app-view to pds

*  Revert locally only if pushAgent is not found

* build

* merge dbcoordinator work into periodic mod actions

* fix entrypoint for periodic mod reversals

* do not run periodic reversals on sequencer leader

* fix duration in action detail on bsky

* update env for pushing mod reversals

* allow zero-duration actions for testing

* remove build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-15 15:44:53 -04:00
devin ivy
53c4956146
Allow moderators to take and reverse actor takedowns (#1330)
allow moderators to take and reverse actor takedowns
2023-07-14 14:37:32 -07:00
devin ivy
3ea892bc57
Add administrative triage role, update moderator role (#1216)
* setup triage user on pds, update moderator username (invalidates old token)

* initial pass on triage access on pds, limit access to email addrs

* apply moderator vs triage rules on taking and reversing mod actions for pds

* update pds tests for triage auth role

* setup moderator and triage roles on bsky appview

* apply mod and triage access rules to bsky admin endpoints

* reframe admin auth as role-based auth, tidy auth apis

* tidy

* build

* revert change to basic auth username for role-based auth
2023-07-06 15:18:52 -04:00
Daniel Holmgren
4d1f8d3289
Misc scaling (#1284)
* limit backsearch to 1 day instead of 3

* lower like count threshold

* bump to 6

* disable like count check

* disable with friends

* preemptively cache last commit

* inline list mutes

* actor service

* label cache

* placehodler on popular with friends

* bulk sequence

* no limit but chunk

* bump chunk to 5k

* try 10k

* fix notify

* tweaking

* syntax

* one more fix

* increase backfill allowance

* full refresh label cache

* limit 1 on mute list

* reserve aclu handle

* clean up testing with label cache

* note on with-friends

* rm defer from label cache

* label cache error handling

* rm branch build
2023-07-05 16:41:08 -05:00
devin ivy
e57c297a82
Handling deleted content on admin/moderation methods (#1045)
* Update lexicon for missing repo/record data in admin views

* Serve missing repo/record variant in admin views

* Test actioning and presenting unknown records/repos, fix on pds
2023-05-17 15:28:52 -04:00
devin ivy
a832c54de7
PDS moderator credentials (#863)
* Setup config and auth verifiers for moderators

* Enforce admin vs. moderator access on PDS admin/server endpoints

* Tidy
2023-04-21 20:58:34 -04:00
Daniel Holmgren
efb1cac2bf
Label reversal fix (#809)
* Reproduce issue with empty negateLabelVals

* normalize str arrays

* Reproduce issue with reversing labels on repo

* Use multiple labels for record labeling test

* join with space

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-12 23:48:47 -05:00
Daniel Holmgren
aa46ad1e1c
Admin labeling (#801)
* adding to moderation flows

* some bugfixing in labels

* hardcode profile nsid

* make labels off moderation action

* db updates

* wip

* report labels in admin views & reverse moderation actions

* Test admin get record and repo w/ labels

* update db

* fix

* exclude negs from labels

* exclude neg on moderation views as well

* Check-in missing lex

* Check-in missing lex

* In-progress admin label tests

* Test label creation/reversal via actions

* Admin label test snapshots (#808)

* new snapshots for label on user

* fix get moderation action snap

* fix dev-env

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-04-12 22:53:32 -05:00
Daniel Holmgren
7f008c05a0
Lexicon refactor (#658)
* remove return in test

* couple of fixups in other pacakges

* Add dummy checks to declaration and follow app migrations, remove paranoid join

* update db nsid migration

* Ensure there are writes in follow app migration

* Add dumy check to votes-to-likes app migration, tidy

* Ensure there are writes in vote-to-like app migration

* update migration name

* pr feedback

* count utf8 & grapheme length

* add maxUtf8

* siwtch max semantics

* plural

* update post schema

* added bytes & cid refs

* add ipld<>json

* fixin up a could tings

* Add app.bsky.richtext.facet, replace post entities with facets

* plural actors

* wip

* Setup backlinks table on pds

* wip

* send & recieve cids/bytes with xrpc

* Track backlinks when indexing records on pds

* handle ipld vals in xrpc server

* added cids & bytes to codegen

* In createRecord, add deletions to avoid duplicate likes/follows/reposts

* Tests and fixes for prevention of dupe follows, likes, reposts

* Backlink migration tidy

* cleanup dag json parser

* Fix dupe backlink inserts

* Tidy

* blob refs + codegen

* Make profile displayName optional

* Test view and updateProfile for empty display name

* working into pds

* Make aggregate counts optional on post and profile views

* Make viewer state optional on post view for consistency

* Remove deprecated myState field on profile view

* Tidy repo method descriptions

* tests & types & fixes

* Implementation and tests for putRecord

* Remove updateProfile method

* Update repo service so that head can be taken for update externally

* Lex updates for compare-and-swap records/commits

* Add error to lex for bad repo compare-and-swaps

* Improve update-at-head thru repo service

* common package

* Implement and test compare-and-swaps on repo write methods

* Use lex discriminator for applyWrites

* Remove post entity/facet index

* Update lex descriptions to clarify repo write semantics

* Make deleteRecord idempotent w/ tests

* cleanup

* fix things up

* adding more formats

* tests

* updating schema

* Only generate tid rkeys on pds, support literal rkeys on client

* Add backlink indexes

* Update format of post embed views, fix external uri validation

* fixing up tests

* Include embeds on record embeds

* cleanup

* Notify users when they are quoted

* Remove determineRkey indirection

* fix api tests

* support concatenated cbor

* integrating to server

* re-enable tests

* fix up tests

* Thread compare-and-swaps down into repo service rather than use pinned storage

* Tidy

* Update packages/common/tests/ipld-multi.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/lexicon/src/validators/formats.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

* pr feedback

* Add postgres-specific migration path for missing profile display names

* Tidy/clarify deep embeds

* Tidy

* rm unused escape

* decrease crud race count

* update subscribeRepos lexicon

* Fix applyWrite lexicon re: collection fields

* sign post event type

* update cids & bytes json encoding

* update lex blob & cid-link types

* updated codegen & pds

* number -> float

* missed a couple

* remove old image constraints

* pr feedback + descripts

* no hardcoded port numbers

* remove separate tooLarge evt

* fix dumb build error

* fixin gup lex + xrpc server

* better parsing of message types

* dont mutate body in subscription

* bugfix in subscription

* rm commented out code

* init feature branch

* undo

* Remove old lexicons

* Remove creator from profile view

* wip

* rework seqs

* fixed up tests

* bug fixing

* sequence handles & notify in dbTxn

* tidy

* update lex to include times

* test syncing handle changes

* one more fix

* handle too big evts

* dont thread sequencer through everything

* Split common into server vs web-friendly versions

* Make lexicon, identifier web-safe using common-web

* Switch api package to be a browser build, fix identifier package for browser bundling

* Fix pds and repo for lexicon package changes, tidy

* Make common-web a browser build, tidy

* fixing up deps

* fix up test

* turn off caching in actions

* Standardize repo write interfaces around repo input

* Update repo write endpoints for repo input field

* Remove scene follows during app migration

* API package updates (#712)

* Add bsky agent and various sugars to the api package

* Add richtext library to api package

* Update richtext to use facets and deprecate entities

* Update richtext to use utf8 indices

* Richtext converts deprecated entity indices from utf16 locations to utf8 locations

* Add note about encodings in the lexicon

* Add RichText facet detection

* Remove dead code

* Add deprecation notices to lexicons

* Usability improvements to RichText

* Update the api package readme

* Add RichText#detectFacetsWithoutResolution

* Add upsertProfile to bsky-agent

* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

* fix flaky timing streaming tests

* simplify emptyPromise

* fixed up open handles

* fix missed repo syntax

* fix error in test from fkey constraint

* fix another api agent bug

* Embed consistency, add complex record embed

* Tidy embed lex descriptions

* rename pg schemas

* use swc for jest

* fix up deps

* cleanup

* Update pds indexing, views, tests for complex record embeds

* fixing up profile view semantics

* wip

* update snaps

* Rename embed.complexRecord to embed.recordWithMedia

* Tidy aroud record w/ media embeds

* Add grapheme utilities to api RichText (#720)

Co-authored-by: dholms <dtholmgren@gmail.com>

* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)

* Update bskyagent to use repo param

* Minor typing fix

* Add exports to api package: blobref & lex/json converters (#727)

* Add exports to api package: BlobRef & lex/json converters

* Add an example react-native fetch handler

* Switch all lingering references of recordRef to strongRef

* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice

* Implement multi-feature richtext facets on pds

* Update api package to use updated richtext facets

* Minor fixes to admin repo/record views

* Fix app migration exports, remove old app migration

* Fix: sort richtext facets so they can render correctly

* Disable app migration dummy checks that don't work on live deploy

* Optimize lex de/serialization using simple checks

* Tidy comment typos

* App migration to cleanup notifications for likes, follows, old scene notifs

* Fix notification reason for change from vote to like

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-03-31 13:34:51 -04:00
Daniel Holmgren
afa28c709f
Get blobs (#606)
* refactor repo routes

* basic blob route

* getBlob route

* tidy

* move getBlob to sync

* allow mimetype on getBlob

* creator on blob table

* migration

* migration

* handle deletes & check db on getBlob

* fix content type bug

* back to octet-stream

* Update packages/pds/src/api/com/atproto/sync/getBlob.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* fix up migrations

* pr feedback

* fixing up merge & migration

* patched up migration

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-03-13 19:11:47 -05:00
Paul Frazee
2242e8a313
Refactor @atproto/api to the AtpAgent interface (#529)
* Refactor @atproto/api to use the simplified AtpAgent API

* xrpc package: Export the defaultFetchHandler to reuse in api

* api package: Use the defaultFetchHandler defined in xrpc

* Update all usages of the api for the new AtpAgent

* Clear promise on thrown codepath

* Avoid updating the atpagent session until ready to return
2023-02-07 15:30:29 -06:00
devin ivy
b91e6e96b7
Incorporate blobs into moderation actions (#511)
* Lexicon updates for moderating blobs

* Add tracking of blob takedowns to moderation system

* Handle blobs in record takedown reversal

* Apply blob takedowns w/ image invalidation

* Test blob takedowns

* Implement cloudfront image invalidator

* Update lexicons for current moderation action on records and blobs

* Fix migration for postgres

* Enforce single current action per repo and blob, present current action on repo and blob

* Test single active mod action per repo, record, and blob

* Continue to resolve handles for moderated repos
2023-02-06 11:44:26 -05:00
devin ivy
ef487900f6
Server moderation action/resolution and reporting (#458)
* Model mod actions for actor takedowns

* Add administration lex schemas for moderation actions

* Initial implementation of takedown mod action and reversal

* Refactor admin actions into a service

* Check for account takedown when creating new session or refreshing session

* Sweep usage of getUser() and getUserByEmail() for takedowns

* Begin sweep for account takedowns

* Tidy takedown checks

* Handle post author takedowns

* Fix tests and migrations for mod takedown

* Fix xrpc-server delayed read of blob payloads

* Test takedown mod action, misc fixes

* Remodel lex schemas for admin moderation actions

* Remove delcaration cid from actor takedowns, lex schema tidy

* Add notes

* Move moderation methods to com.atproto, based around repo rather than actor

* Moderation reports of repos and records

* Implement moderation report resolution method

* Tidy

* Place moderation-init migration after mute-init

* Allow services to reference each other

* Model record takedowns in db

* Lex schemas for record takedowns

* Service/methods for record takedowns, fix report view subject

* Apply record takedown to posts in feeds

* Test record takedown effects, fix reversal

* Support server moderation flag and acknowledge actions, fix action/report resolution for records

* Track repo takedowns on repo_root rather than did_handle

* Move repo/record reporting under com.atproto.report

* Colocate all moderation functionality into moderation service, combine from admin and repo services

* Tidy

* Tidy

* Unify mod report and action methods with new repo/record ref schemas
2023-01-06 12:59:47 -05:00