* Init pulling bsky app view from pds package into its own package, remove sqlite db dialect
* Cull bsky config, services, auth, etc.
* Sweep app view xrpc methods, tidy deps, add storage back for img server
* Run repo subscription on bsky app view
* Collapse db migrations down for bsky app view
* Tidy app view bin
* Remove mute functionality from app view, delegate to pds
* Initial tidy/culling of bsky app view tests
* Passing bsky app view db, server, and repo subscription tests
* Passing bsky app view duplicate-records tests
* Bsky app view test tidy/cull
* In bsky app view replace repo_root, ipld_block, did_handle with actor and record tables. Remove assertions/confirmations.
* Update bsky impl for simpler actor and record tables, removed asserion/confirmations. Skip indexing unknown collections.
* Setup actor handles by did in bsky app view
* Passing indexing tests on bsky app view
* Passing image tests on bsky app view
* Fix bsky actor reindexing, support custom lock id for testing repo subs
* Sweep bsky view tests, misc tests, passing
* Tidy bsky deps
* Include did in resized image uris
* Update bsky image process server to use getBlob
* Update image server tests, misc fixes
* Implement bsky blob resolver
* Wire local image processing server to local blob resolver, test blob resolver
* Tidy
* Tidy
* Tidy
* Tidy app view init
* Fix handle resolution, tidy
* Add utils for partitioning indexing by did
* Update repo sub to parallelize work per repo
* Dep tidy
* Tidy bsky tests for updated repo sub destroy()
* Update thead indexing to handle out-of-order posts
* Sketch out strategy in bsky for handling too-big commits
* Set content-type on sync.getBlob
* Add logging for failed transmissions in bsky blob resolver
* Tidy
* Tidy bsky repo indexing and supporting repo interfaces
* Sort in app view based on combo of creation and indexing times
* Fix types
* Add retry utils to bsky
* Add retries to http requests made by bsky
* Test repo indexing
* Update bsky db/model for lex refactor
* Update bsky lexicons for lex refactor
* Update bsky actor service for lex refactor
* Update bsky feed service for lex refactor
* Update bsky indexing service for lex refactor
* Update bsky repo subscription for lex refactor
* Tidy bsky repo sub
* Add unspecced endpoints to bsky app view, update entrypoint
* Update bsky xrpc utils for lex refactor
* Update bsky xrpc methods for lex refactor
* Update bsky test seeds for lex refactor, tidy api entrypoint
* Update bsky non-view tests for lex refactor
* Update bsky likes view test for lex refactor, minor fix
* Update bsky author feed tests for lex refactor, minor test util fix
* Update bsky follow, profile, repost, search view tests for lex refactor
* Update bsky timeline view tests for lex refactor
* Replace bsky out-of-order thread indexing logic
* Update bsky thread view tests for lex refactor, general test tidying
* Handle rebases and too-big commits in repo subscription, tracking commit data cid
* Tidy
* Ensure did resolver reports "not found" only when positively not found
* Handle tombstones and handle updates in bsky
* Test indexing handle updates and did tombstones
* Support cors on bsky
* Allow app view to serve most routes unauthed
* Tests for bsky unauthed views
* Tidy bsky service entrypoint and dockerfile
* Remove unused storage interfaces from bsky
* Bsky entrypoint and dockerfile fixes, tidy
* Add workflow for bsky build to aws
* Use more standard db env variables, make migration creds optional
* Make bsky repo subscription optional
* Fix lex->json serialization in bsky
* Split bsky actor sync state into its own table
* Skip invalid records on indexing full repo, tidy
* Tidy
* Leader test timing
* Tidy/lint
* Fix bsky config overrides
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* setting up rebase in repo & storage
* repo tests
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* integrate into services & sequencer
* more tests
* Update api package to use updated richtext facets
* one more test
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* Fix notification reason for change from vote to like
* Update packages/pds/src/sequencer/events.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* handle rebased blobs properly
---------
Co-authored-by: devin ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* remove return in test
* couple of fixups in other pacakges
* Add dummy checks to declaration and follow app migrations, remove paranoid join
* update db nsid migration
* Ensure there are writes in follow app migration
* Add dumy check to votes-to-likes app migration, tidy
* Ensure there are writes in vote-to-like app migration
* update migration name
* pr feedback
* count utf8 & grapheme length
* add maxUtf8
* siwtch max semantics
* plural
* update post schema
* added bytes & cid refs
* add ipld<>json
* fixin up a could tings
* Add app.bsky.richtext.facet, replace post entities with facets
* plural actors
* wip
* Setup backlinks table on pds
* wip
* send & recieve cids/bytes with xrpc
* Track backlinks when indexing records on pds
* handle ipld vals in xrpc server
* added cids & bytes to codegen
* In createRecord, add deletions to avoid duplicate likes/follows/reposts
* Tests and fixes for prevention of dupe follows, likes, reposts
* Backlink migration tidy
* cleanup dag json parser
* Fix dupe backlink inserts
* Tidy
* blob refs + codegen
* Make profile displayName optional
* Test view and updateProfile for empty display name
* working into pds
* Make aggregate counts optional on post and profile views
* Make viewer state optional on post view for consistency
* Remove deprecated myState field on profile view
* Tidy repo method descriptions
* tests & types & fixes
* Implementation and tests for putRecord
* Remove updateProfile method
* Update repo service so that head can be taken for update externally
* Lex updates for compare-and-swap records/commits
* Add error to lex for bad repo compare-and-swaps
* Improve update-at-head thru repo service
* common package
* Implement and test compare-and-swaps on repo write methods
* Use lex discriminator for applyWrites
* Remove post entity/facet index
* Update lex descriptions to clarify repo write semantics
* Make deleteRecord idempotent w/ tests
* cleanup
* fix things up
* adding more formats
* tests
* updating schema
* Only generate tid rkeys on pds, support literal rkeys on client
* Add backlink indexes
* Update format of post embed views, fix external uri validation
* fixing up tests
* Include embeds on record embeds
* cleanup
* Notify users when they are quoted
* Remove determineRkey indirection
* fix api tests
* support concatenated cbor
* integrating to server
* re-enable tests
* fix up tests
* Thread compare-and-swaps down into repo service rather than use pinned storage
* Tidy
* Update packages/common/tests/ipld-multi.test.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* Update packages/lexicon/src/validators/formats.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* pr feedback
* Add postgres-specific migration path for missing profile display names
* Tidy/clarify deep embeds
* Tidy
* rm unused escape
* decrease crud race count
* update subscribeRepos lexicon
* Fix applyWrite lexicon re: collection fields
* sign post event type
* update cids & bytes json encoding
* update lex blob & cid-link types
* updated codegen & pds
* number -> float
* missed a couple
* remove old image constraints
* pr feedback + descripts
* no hardcoded port numbers
* remove separate tooLarge evt
* fix dumb build error
* fixin gup lex + xrpc server
* better parsing of message types
* dont mutate body in subscription
* bugfix in subscription
* rm commented out code
* init feature branch
* undo
* Remove old lexicons
* Remove creator from profile view
* wip
* rework seqs
* fixed up tests
* bug fixing
* sequence handles & notify in dbTxn
* tidy
* update lex to include times
* test syncing handle changes
* one more fix
* handle too big evts
* dont thread sequencer through everything
* Split common into server vs web-friendly versions
* Make lexicon, identifier web-safe using common-web
* Switch api package to be a browser build, fix identifier package for browser bundling
* Fix pds and repo for lexicon package changes, tidy
* Make common-web a browser build, tidy
* fixing up deps
* fix up test
* turn off caching in actions
* Standardize repo write interfaces around repo input
* Update repo write endpoints for repo input field
* Remove scene follows during app migration
* API package updates (#712)
* Add bsky agent and various sugars to the api package
* Add richtext library to api package
* Update richtext to use facets and deprecate entities
* Update richtext to use utf8 indices
* Richtext converts deprecated entity indices from utf16 locations to utf8 locations
* Add note about encodings in the lexicon
* Add RichText facet detection
* Remove dead code
* Add deprecation notices to lexicons
* Usability improvements to RichText
* Update the api package readme
* Add RichText#detectFacetsWithoutResolution
* Add upsertProfile to bsky-agent
* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts
Co-authored-by: devin ivy <devinivy@gmail.com>
* pr feedback
* fix flaky timing streaming tests
* simplify emptyPromise
* fixed up open handles
* fix missed repo syntax
* fix error in test from fkey constraint
* fix another api agent bug
* Embed consistency, add complex record embed
* Tidy embed lex descriptions
* rename pg schemas
* use swc for jest
* fix up deps
* cleanup
* Update pds indexing, views, tests for complex record embeds
* fixing up profile view semantics
* wip
* update snaps
* Rename embed.complexRecord to embed.recordWithMedia
* Tidy aroud record w/ media embeds
* Add grapheme utilities to api RichText (#720)
Co-authored-by: dholms <dtholmgren@gmail.com>
* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)
* Update bskyagent to use repo param
* Minor typing fix
* Add exports to api package: blobref & lex/json converters (#727)
* Add exports to api package: BlobRef & lex/json converters
* Add an example react-native fetch handler
* Switch all lingering references of recordRef to strongRef
* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice
* Implement multi-feature richtext facets on pds
* Update api package to use updated richtext facets
* Minor fixes to admin repo/record views
* Fix app migration exports, remove old app migration
* Fix: sort richtext facets so they can render correctly
* Disable app migration dummy checks that don't work on live deploy
* Optimize lex de/serialization using simple checks
* Tidy comment typos
* App migration to cleanup notifications for likes, follows, old scene notifs
* Fix notification reason for change from vote to like
---------
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
* mst: more test coverage
In particular, ports the older fanout=32 corner-case tests to fanout=16.
I generated new paths with correct level and ordering, but haven't
actually verified that the resulting structures have the shape of the
ASCII diagrams in text.
* mst: additional comments, including TODO questions
* mst: default new MST layer to 'null' not 0
Looks like all current call sites of this 'create()' function override
the layer already. This is just making it explicit that the default,
null, means the layer has not been computed yet, distinct from being
layer=0.
* mst: remove one TODO and reformat another
From review.
* Lexicon updates for moderating blobs
* Add tracking of blob takedowns to moderation system
* Handle blobs in record takedown reversal
* Apply blob takedowns w/ image invalidation
* Test blob takedowns
* Implement cloudfront image invalidator
* Update lexicons for current moderation action on records and blobs
* Fix migration for postgres
* Enforce single current action per repo and blob, present current action on repo and blob
* Test single active mod action per repo, record, and blob
* Continue to resolve handles for moderated repos
* Fix typo
* Add failing test for attaching image to a post
* Tidy failing test
* Fix
* Expand the available image resize fit modes
* Add proper validation of glob-prefixed mimes (eg image/*)
* Detect and handle blobs in post embeds
* evert "Expand the available image resize fit modes"
This reverts commit 3390fbe4d091a04f9b67a2885eae19d025747fc4.
* Use fit: inside for post media
* Update tests
* Make the thumbnail optional in external embeds
Co-authored-by: Devin Ivy <devinivy@gmail.com>
* Enable all modules to be built shallowly, externalizing their deps
* Setup shallow builds to modify package.json main
* Hoist update-main-to-dist to separate call, fix lex-cli and plc shallow builds
* Generate sourcemaps during build
* Tidy
* Hoist new build deps up to root
* immutable repo
* wrap mutable repo around immutable
* fixing up repo & structure
* added back in safeCommit
* fix up server
* return rebase cid on commit callback
* getReocrd on structure + a bit of cleanup
* pr cleanup
* Rename ADX to ATP
* Remove old docs (now in atproto website repo)
* Update readme
* Update README.md
Co-authored-by: devin ivy <devinivy@gmail.com>
* A couple of @ -> AT
Co-authored-by: devin ivy <devinivy@gmail.com>
* update schemas for keyed
* codegen dict
* codgen record ids
* determine key type from schema
* allow non-tid keys
* rm tids from batch write schema
* use schema ids record in db layouts
* standardize on rkey
* doc update
* crud test
* repo tests for custom rkeys
* record type for accepted badge
* getBadgeMembers view
* updateProfile procedure
* better return type on updateProfile & gen client
* add badges to getProfile view
* testing & bug fixes
* tests for updating profile & linking badges
* errors
* badgeOffer record type
* getProfile view ensures badges are both offered & accepted
* getBadgeMembers test
* PR feedback
* cid check
* parsing dids to something we understand
* revamping did-sdk with did-resolver
* testing
* fixing some thigns & finishing tests
* fix type error & remove old test
* wip
* create did on server & add email
* check username available & store email
* better test utils
* remove did:test
* allow authstore to sign for controlled dids
* plc ucan integration wip
* wip
* finished auth lib refactor
* integrating to repo & server
* couple fix ups
* couple more fixups
* disable dev-env test
* comments
* validate the signatures
add the id to the did doc
pid from sha256 of oragin doc
added the tick to diffs extractor
added the key authorisation
add lots of types
add routes for calling the aic
refactor to jest based repo structure
* WIP
* migrate the tests
* more test fixes
* test.
* replace snake case with camel case
* moved to use isolated server for aic
* remove the aic test from the PDS
* clean up some changes outside the aic folder
* fix CaS for saving ticks
* rearange the folder
* missed one
* cleaned up aic package.json
* broken build
* test passing
* prettier
* pid tests refactor
* fix eslint/prettier setup, add eslint-config-prettier
* misc linting and style tweaks in aic
* minor touchups to aic sign/verify
* tidy error-handling in aic
* tidy-up crypto init in aic
* add aic route for getting a did doc, update /tick routes
* knex -> typeorm
* refactor crypto
* error handling & dependencies
* switching op model & some other cleanup
* change db to use operations
* add cid references to ops & ensure no branching in DB tx
* cleaning up
* rotues
* working on client & tests
* document & oepration tests
* server tests
* cleanup
* remove lobby page & static
* formatting did documents
* aic/pch -> plc
* fixing up some doc semantics
* tooling/test server
* added recovery window
* tx mutex
* test on document formatting
* take validate op out of write tx
* error handling
* readme
* couple bug fixes
* quick readme edit
Co-authored-by: Aaron Goldman <aaron@aarons-mbp.lan>
Co-authored-by: Aaron D Goldman <aaron@blueskyweb.xyz>
Co-authored-by: Devin Ivy <devin@bigroomstudios.com>
* Add xrpc-cli package; implement 'new' and 'gen-md' commands
* Add gen-ts-obj to xrpc-cli
* Add gen-api command to xrpc-cli
* Add gen-server to xrpc-cli
* Update codegen from xrpc-cli to run prettier and include a banner comment
* Use 'void' instead of 'undefined' for return type
* Skip invalid schemas when generating code
* Rewrite schemas package to use new simplified schema
* Output type definitions in nested folders
* Add record schema code-generation
* Change xrcp client call() signature to be easier to use
* Update client codegen to match new xrpc call signature
* Fixes and improvements to generated record APIs
* Update generated api to use per-service instances
* Add xrpc-cli readme
* (WIP) Rewrite server to use xrpc
* Add missing references to server/tsconfig.json
* Replace api implementation with xrpc generated client
* Add /schemas folder with todo.adx
* Add /schemas/todo.social
* Update api package to include todo.social xrpc and adx schemas
* Update server package to include todo.social xrpc and adx schemas
* Fix some schema definitions
* (WIP) Work toward xrpc update of server
* Update auth package to use nsids for collections
* Update repo package to use nsids for collections
* Update AdxUri to use nsid collection IDs
* Bring server CRUD tests to passing state
* Move API to simpler xrpc call signature
* Apply fixes and improvements to record apis
* Update to new generated api with per-service instances
* Rename the todo.social.* view methods
* Rewrite the views system to use xrpc
* Remove paths accidentally added to tsconfig
* Remove microblog package
* Add simple API readme
* todo about deduping ucan prfs
* increase postThread default depth to fix failing test. & fix USE_TEST_SERVER config
Co-authored-by: dholms <dtholmgren@gmail.com>
