Daniel Holmgren 7f008c05a0
Lexicon refactor (#658)
* remove return in test

* couple of fixups in other pacakges

* Add dummy checks to declaration and follow app migrations, remove paranoid join

* update db nsid migration

* Ensure there are writes in follow app migration

* Add dumy check to votes-to-likes app migration, tidy

* Ensure there are writes in vote-to-like app migration

* update migration name

* pr feedback

* count utf8 & grapheme length

* add maxUtf8

* siwtch max semantics

* plural

* update post schema

* added bytes & cid refs

* add ipld<>json

* fixin up a could tings

* Add app.bsky.richtext.facet, replace post entities with facets

* plural actors

* wip

* Setup backlinks table on pds

* wip

* send & recieve cids/bytes with xrpc

* Track backlinks when indexing records on pds

* handle ipld vals in xrpc server

* added cids & bytes to codegen

* In createRecord, add deletions to avoid duplicate likes/follows/reposts

* Tests and fixes for prevention of dupe follows, likes, reposts

* Backlink migration tidy

* cleanup dag json parser

* Fix dupe backlink inserts

* Tidy

* blob refs + codegen

* Make profile displayName optional

* Test view and updateProfile for empty display name

* working into pds

* Make aggregate counts optional on post and profile views

* Make viewer state optional on post view for consistency

* Remove deprecated myState field on profile view

* Tidy repo method descriptions

* tests & types & fixes

* Implementation and tests for putRecord

* Remove updateProfile method

* Update repo service so that head can be taken for update externally

* Lex updates for compare-and-swap records/commits

* Add error to lex for bad repo compare-and-swaps

* Improve update-at-head thru repo service

* common package

* Implement and test compare-and-swaps on repo write methods

* Use lex discriminator for applyWrites

* Remove post entity/facet index

* Update lex descriptions to clarify repo write semantics

* Make deleteRecord idempotent w/ tests

* cleanup

* fix things up

* adding more formats

* tests

* updating schema

* Only generate tid rkeys on pds, support literal rkeys on client

* Add backlink indexes

* Update format of post embed views, fix external uri validation

* fixing up tests

* Include embeds on record embeds

* cleanup

* Notify users when they are quoted

* Remove determineRkey indirection

* fix api tests

* support concatenated cbor

* integrating to server

* re-enable tests

* fix up tests

* Thread compare-and-swaps down into repo service rather than use pinned storage

* Tidy

* Update packages/common/tests/ipld-multi.test.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* Update packages/lexicon/src/validators/formats.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

* pr feedback

* Add postgres-specific migration path for missing profile display names

* Tidy/clarify deep embeds

* Tidy

* rm unused escape

* decrease crud race count

* update subscribeRepos lexicon

* Fix applyWrite lexicon re: collection fields

* sign post event type

* update cids & bytes json encoding

* update lex blob & cid-link types

* updated codegen & pds

* number -> float

* missed a couple

* remove old image constraints

* pr feedback + descripts

* no hardcoded port numbers

* remove separate tooLarge evt

* fix dumb build error

* fixin gup lex + xrpc server

* better parsing of message types

* dont mutate body in subscription

* bugfix in subscription

* rm commented out code

* init feature branch

* undo

* Remove old lexicons

* Remove creator from profile view

* wip

* rework seqs

* fixed up tests

* bug fixing

* sequence handles & notify in dbTxn

* tidy

* update lex to include times

* test syncing handle changes

* one more fix

* handle too big evts

* dont thread sequencer through everything

* Split common into server vs web-friendly versions

* Make lexicon, identifier web-safe using common-web

* Switch api package to be a browser build, fix identifier package for browser bundling

* Fix pds and repo for lexicon package changes, tidy

* Make common-web a browser build, tidy

* fixing up deps

* fix up test

* turn off caching in actions

* Standardize repo write interfaces around repo input

* Update repo write endpoints for repo input field

* Remove scene follows during app migration

* API package updates (#712)

* Add bsky agent and various sugars to the api package

* Add richtext library to api package

* Update richtext to use facets and deprecate entities

* Update richtext to use utf8 indices

* Richtext converts deprecated entity indices from utf16 locations to utf8 locations

* Add note about encodings in the lexicon

* Add RichText facet detection

* Remove dead code

* Add deprecation notices to lexicons

* Usability improvements to RichText

* Update the api package readme

* Add RichText#detectFacetsWithoutResolution

* Add upsertProfile to bsky-agent

* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* pr feedback

* fix flaky timing streaming tests

* simplify emptyPromise

* fixed up open handles

* fix missed repo syntax

* fix error in test from fkey constraint

* fix another api agent bug

* Embed consistency, add complex record embed

* Tidy embed lex descriptions

* rename pg schemas

* use swc for jest

* fix up deps

* cleanup

* Update pds indexing, views, tests for complex record embeds

* fixing up profile view semantics

* wip

* update snaps

* Rename embed.complexRecord to embed.recordWithMedia

* Tidy aroud record w/ media embeds

* Add grapheme utilities to api RichText (#720)

Co-authored-by: dholms <dtholmgren@gmail.com>

* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (#726)

* Update bskyagent to use repo param

* Minor typing fix

* Add exports to api package: blobref & lex/json converters (#727)

* Add exports to api package: BlobRef & lex/json converters

* Add an example react-native fetch handler

* Switch all lingering references of recordRef to strongRef

* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice

* Implement multi-feature richtext facets on pds

* Update api package to use updated richtext facets

* Minor fixes to admin repo/record views

* Fix app migration exports, remove old app migration

* Fix: sort richtext facets so they can render correctly

* Disable app migration dummy checks that don't work on live deploy

* Optimize lex de/serialization using simple checks

* Tidy comment typos

* App migration to cleanup notifications for likes, follows, old scene notifs

* Fix notification reason for change from vote to like

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-03-31 13:34:51 -04:00

138 lines
4.2 KiB
TypeScript

import * as cborx from 'cbor-x'
import * as uint8arrays from 'uint8arrays'
import { MessageFrame, ErrorFrame, Frame, FrameType } from '../src'
describe('Frames', () => {
it('creates and parses message frame.', async () => {
const messageFrame = new MessageFrame(
{ a: 'b', c: [1, 2, 3] },
{ type: '#d' },
)
expect(messageFrame.header).toEqual({
op: FrameType.Message,
t: '#d',
})
expect(messageFrame.op).toEqual(FrameType.Message)
expect(messageFrame.type).toEqual('#d')
expect(messageFrame.body).toEqual({ a: 'b', c: [1, 2, 3] })
const bytes = messageFrame.toBytes()
expect(
uint8arrays.equals(
bytes,
new Uint8Array([
/*header*/ 162, 97, 116, 98, 35, 100, 98, 111, 112, 1, /*body*/ 162,
97, 97, 97, 98, 97, 99, 131, 1, 2, 3,
]),
),
).toEqual(true)
const parsedFrame = Frame.fromBytes(bytes)
if (!(parsedFrame instanceof MessageFrame)) {
throw new Error('Did not parse as message frame')
}
expect(parsedFrame.header).toEqual(messageFrame.header)
expect(parsedFrame.op).toEqual(messageFrame.op)
expect(parsedFrame.type).toEqual(messageFrame.type)
expect(parsedFrame.body).toEqual(messageFrame.body)
})
it('creates and parses error frame.', async () => {
const errorFrame = new ErrorFrame({
error: 'BigOops',
message: 'Something went awry',
})
expect(errorFrame.header).toEqual({ op: FrameType.Error })
expect(errorFrame.op).toEqual(FrameType.Error)
expect(errorFrame.code).toEqual('BigOops')
expect(errorFrame.message).toEqual('Something went awry')
expect(errorFrame.body).toEqual({
error: 'BigOops',
message: 'Something went awry',
})
const bytes = errorFrame.toBytes()
expect(
uint8arrays.equals(
bytes,
new Uint8Array([
/*header*/ 161, 98, 111, 112, 32, /*body*/ 162, 101, 101, 114, 114,
111, 114, 103, 66, 105, 103, 79, 111, 112, 115, 103, 109, 101, 115,
115, 97, 103, 101, 115, 83, 111, 109, 101, 116, 104, 105, 110, 103,
32, 119, 101, 110, 116, 32, 97, 119, 114, 121,
]),
),
).toEqual(true)
const parsedFrame = Frame.fromBytes(bytes)
if (!(parsedFrame instanceof ErrorFrame)) {
throw new Error('Did not parse as error frame')
}
expect(parsedFrame.header).toEqual(errorFrame.header)
expect(parsedFrame.op).toEqual(errorFrame.op)
expect(parsedFrame.code).toEqual(errorFrame.code)
expect(parsedFrame.message).toEqual(errorFrame.message)
expect(parsedFrame.body).toEqual(errorFrame.body)
})
it('parsing fails when frame is not CBOR.', async () => {
const bytes = Buffer.from('some utf8 bytes')
const emptyBytes = Buffer.from('')
expect(() => Frame.fromBytes(bytes)).toThrow('Unexpected end of CBOR data')
expect(() => Frame.fromBytes(emptyBytes)).toThrow(
'Unexpected end of CBOR data',
)
})
it('parsing fails when frame header is malformed.', async () => {
const bytes = uint8arrays.concat([
cborx.encode({ op: -2 }), // Unknown op
cborx.encode({ a: 'b', c: [1, 2, 3] }),
])
expect(() => Frame.fromBytes(bytes)).toThrow('Invalid frame header:')
})
it('parsing fails when frame is missing body.', async () => {
const messageFrame = new MessageFrame(
{ a: 'b', c: [1, 2, 3] },
{ type: '#d' },
)
const headerBytes = cborx.encode(messageFrame.header)
expect(() => Frame.fromBytes(headerBytes)).toThrow('Missing frame body')
})
it('parsing fails when frame has too many data items.', async () => {
const messageFrame = new MessageFrame(
{ a: 'b', c: [1, 2, 3] },
{ type: '#d' },
)
const bytes = uint8arrays.concat([
messageFrame.toBytes(),
cborx.encode({ d: 'e', f: [4, 5, 6] }),
])
expect(() => Frame.fromBytes(bytes)).toThrow(
'Too many CBOR data items in frame',
)
})
it('parsing fails when error frame has invalid body.', async () => {
const errorFrame = new ErrorFrame({ error: 'BadOops' })
const bytes = uint8arrays.concat([
cborx.encode(errorFrame.header),
cborx.encode({ blah: 1 }),
])
expect(() => Frame.fromBytes(bytes)).toThrow('Invalid error frame body:')
})
})