427 Commits

Author SHA1 Message Date
github-actions[bot]
527fc58e6d
Version packages (#1685)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-27 16:18:27 -05:00
Daniel Holmgren
41ee177f5a
Email confirmation/update (#1568)
* lexicons

* codegen

* email templates

* request routes

* impl

* migration

* tidy

* tests

* tidy & bugfixes

* format

* fix api test

* fix auth test

* codegen

* add unique constraint

* Add email confirmed to AtpSessionData

* interop test files (#1529)

* initial interop-test-files

* crypto: switch signature-fixtures.json to a symlink

* syntax: test against interop files

* prettier

* Update interop-test-files/README.md

Co-authored-by: Eric Bailey <git@esb.lol>

* disable prettier on test vectors

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: dholms <dtholmgren@gmail.com>

* add getSuggestedFollowsByActor (#1553)

* add getSuggestedFollowsByActor lex

* remove pagination

* codegen

* add pds route

* add app view route

* first pass at likes-based suggested actors, plus tests

* format

* backfill with suggested_follow table

* combine actors queries

* fall back to popular follows, handle backfill differently

* revert seed change, update test

* lower likes threshold

* cleanup

* remove todo

* format

* optimize queries

* cover mute lists

* clean up into pipeline steps

* add changeset

* List feeds (#1557)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* lex

* list purpose

* lex gen

* add route

* add proxy route

* seed client helpers

* tests

* mutes and blocks

* proxy test

* snapshot

* hoist actors out of composeThread()

* tidy

* tidy

* run ci on all prs

* format

* format

* fix snap name

* fix snapsh

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Improve xrpc server error handling (#1597)

improve xrpc server error handling

* Remove appview proxy runtime flags (#1590)

* remove appview proxy runtime flags

* clean up proxy tests

* getPopular hotfix (#1599)

dont pass all params

* Interaction Gating (#1561)

* lexicons for block lists

* reorg blockset functionality into graph service, impl block/mute filtering

* apply filterBlocksAndMutes() throughout appview except feeds

* update local feeds to pass through cleanFeedSkeleton(), offload block/mute application

* impl for grabbing block/mute details by did pair

* refactor getActorInfos away, use actor service

* experiment with moving getFeedGenerators over to a pipeline

* move getPostThread over to a pipeline

* move feeds over to pipelines

* move suggestions and likes over to pipelines

* move reposted-by, follows, followers over to pipelines, tidy author feed and post thread

* remove old block/mute checks

* unify post presentation logic

* move profiles endpoints over to pipelines

* tidy

* tidy

* misc fixes

* unify some profile hydration/presentation in appview

* profile detail, split hydration and presentation, misc fixes

* unify feed hydration w/ profile hydration

* unify hydration step for embeds, tidy application of labels

* setup indexing of list-blocks in bsky appview

* apply list-blocks, impl getListBlocks, tidy getList, tests

* tidy

* update pds proxy snaps

* update pds proxy snaps

* fix snap

* make algos return feed items, save work in getFeed

* misc changes, tidy

* tidy

* fix aturi import

* initial lexicons for interaction-gating

* add interactions view to post views

* codegen

* model bad reply/interaction check state on posts

* initial impl for checking bad reply or interaction on write

* omit invalid interactions from post thread

* support not-found list in interaction view

* hydrate can-reply state on threads

* present interaction views on posts

* misc fixes, update snaps

* tidy/reorg

* tidy

* split interaction gating into separate record in lexicon

* switch interaction-gating impl to use separate record type

* allow checking reply gate w/ root post deletion

* fix

* initial gating tests

* tighten gated reply views, tests

* reply-gating list rule tests

* allow custom post rkeys within window

* hoist actors out of composeThread()

* tidy

* update thread gate lexicons, codegen

* lex fix

* rename gate to threadgate in bsky, update views

* lex fix

* improve terminology around reply validation

* fix down migration

* remove thread gates on actor unindexing

* add back .prettierignore

* tidy

* run ci on all prs

* syntax

* run ci on all prs

* format

* fix snap

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* order by `like.indexedAt` in app view (#1592)

* order by like.indexedAt

* use keyset for ordering

* simplify

* ok ok ok I get it now

* Update packages/bsky/src/api/app/bsky/feed/getActorLikes.ts

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* Remove default value for post table invalid attrs (#1601)

remove default value for post table attrs

* Version packages (#1602)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* update Bluesky PBLLC to PBC (Public Benefit Corporation) (#1600)

* Temporarily disable filtering `invalidReplyRoot`s (#1609)

temporarily disable invalidReplyRoot check

* fix syntax docs (#1611)

* Version packages (#1612)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Allow bypass on ratelimit ip (#1613)

allow bypass on ratelimit ip

* Write rate limits (#1578)

* get rate limit ip correctly

* add write rate-limits

* Tweak createSession rate limit key (#1614)

tweak create session rl key

* Filter preferences for app passwords (#1626)

filter preferences for app passwords

* Tweak rate limit setup for multi rate limit routes (#1627)

tweak rate limit setup for multi rate limit routes

* Remove zod from xrpc-server error handling (#1631)

remove zod from xrpc-server error handling check

* Enforce properties field on lexicon object schemas (#1628)

* add empty properites to thread gate schema fragments

* tweak lexicon type

* Add feed-vew and thread-view preferences (#1638)

* Add feed and thread preference lexicons

* Add feed-view and thread-view preference APIs

* Add changeset for new preferences  (#1639)

Add changeset

* Version packages (#1640)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable getAccountInviteCodes for app passwords (#1642)

disable getAccountInviteCodes for app passwords

* remove cruft packages (uri, nsid, identifier) (#1606)

* remove @atproto/nsid (previously moved to syntax)

* remove @atproto/uri (previously moved to syntax)

* remove @atproto/identifier (previously moved to syntax)

* bump lockfile to remove old packages

---------

Co-authored-by: Eric Bailey <git@esb.lol>

* api: update login/resumeSession examples in README (#1634)

* api: update login/resumeSession examples in README

* Update packages/api/README.md

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>

* small syntax lints (#1646)

* lint: remove unused imports and variables

* lint: prefix unused args with '_'

* eslint: skip no-explicit-any; ignore unused _var (prefix)

* eslint: explicitly mark ignores for tricky cases

* indicate that getPopular is deprecated (#1647)

* indicate that getPopular is deprecated

* codegen for deprecating getPopular

* tidy up package.json and READMEs (#1649)

* identity: README example and tidy

* tidy up package metadata (package.json files)

* updated README headers/stubs for several packages

* crypto: longer README, with usage

* syntax: tweak README

* Apply suggestions from code review

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

* Improve the types of the thread and feed preferences APIs (#1653)

* Improve the types of the thread and feed preferences APIs

* Remove unused import

* Add changeset

* Version packages (#1654)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Disable pds appview routes (#1644)

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch

* auto-moderator tweaks: pass along record URI, create report for takedown action (#1643)

* auto-moderator: include record URI in abyss requests

* auto-moderator: log attempt at hard takedown; create report as well

The motivation is to flag the event to mod team, and to make it easier
to confirm that takedown took place.

* auto-mod: typo fix

* auto-mod: bugfixes

* bsky: always create auto-mod report locally, not pushAgent (if possible)

* bsky: fix auto-mod build

* bsky: URL-encode scanBlob call

* Clear follow viewer state when blocking (#1659)

* clear follow viewer state when blocking

* tidy

* add `tags` to posts (#1637)

* add tags to post lex

* kiss

* add richtext facet and validation attrs

* add tag validation attrs to post

* codegen

* add maxLength for tags, add description

* validate post tags on write

* add test

* handle tags in indexer

* add tags to postView, codegen

* return tags on post thread view

* format

* revert formatting change to docs

* use establish validation pattern

* add changeset

(cherry picked from commit fcb6fe7c26144662f791c7900afcd84c7bf1be6b)

* remove tags from postView, codegen

* remove tags from thread view

* revert unused changes

* Version packages (#1664)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* merge

* Reverse order of blocks from sync.getRepo (#1665)

* reverse order of blocks from sync.getRepo

* write to car while fetching next page

* Add hashtag detection to richtext (#1651)

* add tag detection to richtext

* fix duplicate tag index error

* add utils

* fix leading space index failures, test for them

* add changeset

* Version packages (#1669)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* proposed new search lexicons (#1594)

* proposed new search lexicons

* lexicons: lint

* lexicons: fix actors typo

* lexicons: camelCase bites again, ssssss

* lexicons: add 'q' and mark 'term' as deprecated for search endpoints

* codegen for search lexicon updates

* bsky: prefer 'q' over 'term' in existing search endpoints

* search: bugfix

* lexicons: make unspecced search endpoints return skeleton obj

* re-codegen for search skeleton obj

* Disable pds appview indexing (#1645)

* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build

* Refactor PDS appview routes (#1673)

move routes around

* Strip leading `#` from from detected tag facets (#1674)

ensure # is removed from facets

* Version packages (#1675)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Proxy search queries (#1676)

* proxy search

* tweak profile resp

* fix admin.searchRepos

* add mock mailer

* Fix to daniel's MOCKERY of a mock mailer

* Don't allow non-verified email updates until app feature is out (#1682)

stricter updating email until app feature is out

* changesets

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-27 16:08:58 -05:00
github-actions[bot]
7dacb9d6f5
Version packages (#1675)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-26 11:56:02 -05:00
Daniel Holmgren
2fa9088639
Disable pds appview indexing (#1645)
* rm indexing service

* remove message queue & refactor background queue

* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* wip - removing labeler

* fix service file

* remove kysely tables

* re-enable getPopular

* format

* cleaning up tests

* rm unused sharp code

* rm pds build

* clean up tests

* fix build

* fix build

* migration

* tidy

* build branch

* tidy

* build branch

* small tidy

* dont build
2023-09-25 20:09:52 -05:00
github-actions[bot]
4a64944783
Version packages (#1669)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-25 17:48:05 -05:00
github-actions[bot]
3ea11a1a79
Version packages (#1664)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-25 11:17:11 -05:00
Daniel Holmgren
558981ead1
Disable pds appview routes (#1644)
* wip

* remove all canProxyReadc

* finish cleanup

* clean up tests

* fix up tests

* fix api tests

* fix build

* fix compression test

* update image tests

* fix dev envs

* build branch

* fix service file

* re-enable getPopular

* format

* rm unused sharp code

* dont build branch
2023-09-22 13:51:15 -05:00
github-actions[bot]
86bbf0860a
Version packages (#1654)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-21 18:19:28 -07:00
bnewbold
584dea52c4
tidy up package.json and READMEs (#1649)
* identity: README example and tidy

* tidy up package metadata (package.json files)

* updated README headers/stubs for several packages

* crypto: longer README, with usage

* syntax: tweak README

* Apply suggestions from code review

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: Eric Bailey <git@esb.lol>
Co-authored-by: devin ivy <devinivy@gmail.com>
2023-09-21 18:07:33 -07:00
github-actions[bot]
49278430e0
Version packages (#1640)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-20 11:12:04 -07:00
github-actions[bot]
38f8f44c41
Version packages (#1612)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-15 15:49:31 -05:00
github-actions[bot]
d7d7a043de
Version packages (#1602)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-09-14 17:53:14 -05:00
Daniel Holmgren
36ba295333
Remove appview proxy runtime flags (#1590)
* remove appview proxy runtime flags

* clean up proxy tests
2023-09-14 13:55:12 -05:00
Daniel Holmgren
fdf6a4642f
Enable appview proxy in dev-env full network (#1580)
proxy runtime flags
2023-09-11 19:32:04 -05:00
Daniel Holmgren
f8f2d04c1d
Move fuzzy matcher to appview (#1566)
* impl on appview

* tests + clean up pds

* tidy cfg
2023-09-08 15:08:27 -05:00
Eric Bailey
a7c42cfe39
Clean up prettier/eslint scripts (#1514)
* add prettierignore, hoist script

* upgrade prettier

* bump prettier eslint deps

* format all files

* hoist prettier check

* remove unused scripts, hoist lint:fix

* remove npm-run-all, unused

* hoist lint

* remove lint scripts

* improve lint scripts

* remove prettierignores

* downgrade prettier deps to fix codgen

* reformat all files

* update makefile, format

* fix locklife

* final format

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-09-06 19:27:50 -05:00
Daniel Holmgren
f023cfb009
Abyss Integration (#1545)
* add blob scanner

* introduce auto moderator

* tidy

* tests

* more tests & tidy

* timeout & logs

* tweak cfg

* push takedowns to pds

* build branch

* update snaps

* add cloudfront invalidator to indexer

* tweak log

* dont build branch
2023-09-06 17:35:39 -05:00
Eric Bailey
90e8325e5f
pnpm setup (#1532)
* use pnpm

* fix dependency issues, replace yarn and lerna scripts

* remove the main/dist scripts

* update Dockerfiles

* use pnpm

* fix dependency issues, replace yarn and lerna scripts

* remove the main/dist scripts

* update Dockerfiles

* update bin script

* remove unused zod dep

* fix type errors in pds

* add types prop to packages

* remove unused, bump lock

* fix test running

* build before test

* fix pino types

* format

* pds depends on dev-env in test

* refer to src instead of built packages

* pds relies on bsky in test too

* remove yarn.lock

* add -r flag to root test

* test push to aws

* remove docker test

* add publishConfig to new package

* move services to top level dir

(cherry picked from commit f5012bec33435a4473e9960066807623334f3aff)

* update workflow paths

(cherry picked from commit 5c70f0176d381ca35d6be10cfa173e22373a5b5d)

* add main-to-dist script

* use script in all packages, remove old Dockerfiles

* remove old bsky service

* remove newline

* test container builds

* Revert "test container builds"

This reverts commit c228611f5e8e1624d4b124be4976c49590130f43.

* remove unused config

* test build containers

* pnpm in syntax

* bump dd-trace

* shamefully hoist

* even more shame

* hoist, externalize deps

* clean install for prod and smaller containers

* dont build branches

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-09-05 18:45:49 -05:00
Foysal Ahamed
964e08ad82
🐛 Add labels caching as a config for pds and enable it when running dev-env (#1511)
* 🐛 Add labels caching as a config for pds and enable it when running dev-env

*  Move caching enable config to bin.ts
2023-08-28 09:39:57 -04:00
bnewbold
bd3966555f
@atproto/syntax: merge of @atproto/identifier, uri, and nsid (#1464)
* move nsid package in to identifier

* move uri package in to identifier

* update packages to pull from identifier, not nsid or uri

* rename @atproto/identifier to @atproto/syntax

* update all refs from @atproto/identifer to @atproto/syntax
2023-08-25 10:18:10 -05:00
Ansh
eae682f3bf
Native Notifications (#1428)
* pseudocode for sending notification to user

* add notification push token table

* lexicon codegen

* `pds` and `api` codegen

* update lexicon

* add simple function to `putNotificationPushToken` into `notification_push_token` table

* reorgnaize imports

* add unspecced `putNotificationPushToken` to pds

* add `notification-push-tokens` table to PDS

* check if token exists before adding it to db

* add endpoint and appId to PDS table

* setup notification server

* fix logic for inserting token into db

* fix NotificationServer methods by making them static

* fix merge conflicts

* add comments on how sending ntoifications work

* remove dead code

* move notifServer to AppContext

* refactor code to handle notif sending in `Indexer` service

* add additional data when sending notifs

* clean up code

* move notif logic to indexer

* add `appId` and `endpoint` optional params to putNotificationPushToken

* clone notification code to AppView

* add endpoint to register push token with app view

* disable pds and enable app view notification server

* clean up code

* simply logic to check if a token already exists

* remove NotificationServer from PDS

* remove notification-push-token table from PDS

* remove `putNotificationPushToken` endpoint

* clean up code

* let `axios` throw error if `gorush` has an error

* let `kysely` throw error if notif cannot be registered by client

* rename `registerPushNotificationEndpoint` to `registerPushNotifications`

* delete `putNotificationPushToken` from AppView

* rename putPushNotificationToken to registerPushNotification

* remove dead notification code from pds

* remove sanitizeDisplayName from NotificationSever

* move `pushNotificationEndpoint` to config

* temp add `pushNotificationEndpoint` to dev-env setup

* remove example test from feed-generation.test.ts

* add test for registerPushNotification and clean up error handling

* move notifications test to its own file

* add test for NotificationServer to check if tokens are retrieved correctly

* remove unused functions from NotificationServer

* add additional tests for NotificationServer

* add return type to getNotificationDisplayAttributes function

* remove unnecessary comments

* remove dead NotificationServer code from PDS

* clean up code to prepareNotifsToSend

* put sending notifications as part of the backgroundQueue

* log instead of throwing error if notification attributes don't exist

* remove logs

* add more tests to `notification-server.test.ts`

* show replied with text for reply notifs

* better error handling when sending notifications via backgroundQueue

* add rate limit and batching to sending notifications

* add comments to NotificationServer

* merge with main

* use redis for rate limiting instead of normal rate limits

* move `notificationBatchSize` into config

* usePrimary db in test

* hoist push notif migration to present, update model to remove endpoint

* update push notif lexicon

* pare down lex for unregistering push

* helpers for working with service endpoints from did doc

* service-authed register/unregister push notifs

* add well-known endpoint to appview

* update bsky notif service tests

* fix to batching logic, misc tidy

* batch display notifications

* colocate all notification sending logic

* tidy tests

* remove unregister token for now

* fix registerPush lexicon, make a procedure

* fix registerPush impl, test pds proxy to notif service

* fix tests, make notif server optional when not configured

* fix notif server config for bsky app service

* move notif server rate limiting in-mem for now, add sending retry

* codegen tidy

* only push notifs on commit

* build

* fix notif rate limiter check

* send notifs from users w/o a profile

* remove build

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-23 18:37:14 -04:00
Daniel Holmgren
bbc6a2c0fd
Indexer request reprocess (#1504)
* allow requesting reprocessing a repo from an indexer

* partition id guardrailes

* test

* build branch

* dont set ports

* resupply port

* give port in correct spot

* dont build branch
2023-08-23 11:19:47 -05:00
Daniel Holmgren
891bf3bb7a
Application ratelimits (#1382)
* initial setup

* lexgen

* tidying things up

* add in some rate limiting

* testing rate limits

* small bugfix

* fix build

* increase rate limit

* more limits

* config

* optional rate limiter

* tweak headers

* ratelimit headers

* multiple rate limits

* tests & bugfixes

* test bypass

* slight refactor

* fail open

* fail open

* right most xff ip

* setup redis for ratelimiting

* redis test

* more tests

* use new dev-infra package

* adjust limits

* temporarily remove repo write limits

* codegen

* redis scratch var name

* cfg var host -> address
2023-08-21 16:10:04 -05:00
Daniel Holmgren
73f64009a2
Appview database replicas (#1459)
* setup bsky so that writes always go to a db primary

* test bsky primary db functionality

* sprinkle db primary over moderation and view maintainer usage

* make bsky tests more strict on primary/secondary db, fixes

* maintain prev db config on appview, require primary db, make primary/replica explicit in config

* tidy

* sketch out db replicas

* wip

* support multiple tags per bsky replica, "any" tag, fallbacks and warns

* clarify tags

* use new coordinator api across routes & services

* wire-up replica tagging config

* tidy

* cleanup entroypoints and tests for db coordinator

* fix pds test

* re-hookup migrate db

* standardize env vars for db on bsky services

* tidy

* build

* fix bav api entrypoint

* support DB_POSTGRES_URL on ingester/indexer for backwards compat

* remove build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-08-14 18:38:00 -04:00
Daniel Holmgren
3f0e1b2563
Read after write (#1398)
* first pass on profiles

* quick test

* wip

* wip

* test post thread

* record embeds

* get author feed profiles

* wip timeline

* fix get timeline

* switch from counter to tid

* tidy into a service

* quick tid test

* pr feedback

* clock -> rev

* update image formatting

* disable migration & build branch

* add recent posts to getAuthorFeed & handle post thread not found errors

* refactor for lag header

* tidy

* rm collections check

* tidy test

* pr feedback

* fix small bug

* build branch

* get migrations into system

* enable migrations
2023-08-10 13:42:31 -05:00
Devin Ivy
38a4b25c27 v0.2.3 2023-08-09 19:11:15 -04:00
Daniel Holmgren
0bf75fef60
Add bin for launching dev-env with an appview (#1454)
script for launching a full dev-env network
2023-08-09 13:42:03 -05:00
devin ivy
203e72f1c1
Appview preset image URLs (#1248)
* update image signing logic to presets on bsky

* update bsky config and views for preset img urls

* update bsky image tests for preset urls

* update bsky snapshots for preset image urls

* update pds proxy tests for bsky preset image urls

* tweak image paths
2023-08-09 10:21:43 -04:00
Daniel Holmgren
4879ccb365
Appview label cache (#1451)
* add label cache to appview

* add to dev env

* fix test
2023-08-08 17:56:21 -05:00
Daniel Holmgren
25458ade66
Fix open handles in appview tests (#1452)
fix open handles in appview tests
2023-08-08 15:30:42 -05:00
devin ivy
30b41029c1
Partitioning in bsky indexer (#1368)
* setup redis infra for appview indexer

* barebones bsky ingester

* add ioredis to bsky

* remove some indexer functionality from bsky api

* setup for bsky indexer

* tidy

* tidy, observe basic pipeline functioning

* process messages on bsky indexer pipeline, tidy tests and lifecycle

* trim partitions when moving cursor

* simplify config for partitions

* misc fixes for redis setup in bsky tests, add namespacing

* fix pds proxy tests

* remove cursor state from indexer partitions, simplify ingester state

* tidy

* utils for testing w/ multiple indexers, fix off-by-one xtrim

* test reingesting

* test indexer repartitioning

* add entrypoints for bsky ingester and indexer, fix db schema config, api entrypoint name, tidy

* setup and test bsky ingester backpressure, add config

* tidy

* add missing test file

* tidy redis calls, add redis sentinel config

* tidy/test some utils used in bsky pipeline

* tidy bsky pipeline tests, move helpers into dev-env

* fix pds crud test

* support redis host and password config

* better loggin/observability in ingester and indexer, make build

* add bsky ingester initial cursor config

* temporarily remove migrations from indexer/ingester

* allow ingester to batch

* packages/pg becomes packages/dev-infra with some cleanup (#1402)

* packages/dev-infra/

* Extract packages/dev-infra/_common.sh and use it

* Use period instead of source because of /bin/sh

* add docs for redis test script

* fix repartition test

* add logs to debug ci

* simplify repartitioning test, remove ci logs

---------

Co-authored-by: Jerry Chen <jerry@redelm.net>
2023-08-01 14:59:00 -04:00
Daniel Holmgren
2836ee38bb
Appview - serve feed skeletons (#1265)
* proxy timeline skeleton construction to appview

* add getFeedSkeleton to appview

* mount route

* smart proxy feed skeletons

* tests

* proper error code

* only proxy specific feeds

* build branch

* update proxyable feed logic, should use feed publisher rather than generator

* fix feed proxy tests, configure feed publisher (in addition to generator)

* hotfix: prevent user-supplied rkey on posts with createRecord (#1313)

* prevent user-supplied rkey on posts with createRecord

* allow empty-string rkey parameter

Co-authored-by: devin ivy <devinivy@gmail.com>

---------

Co-authored-by: devin ivy <devinivy@gmail.com>

* add slurs to reserved words (#1318)

* add slurs to reserved words (#1314)

* Update reserved.ts

Add slurs to reserved words

* Update reserved.ts

fix typo

* Update reserved.ts to clean up the slur list

* linting

* pluralise

---------

Co-authored-by: jess <jesslarkdev+gh@gmail.com>

* identifier: tweaks and additions to slur list (#1319)

* Refactor appview repo subscription for memleak (#1308)

* refactor to remove closure in loop

* move consecutive item out of p-queue

* Handle validation improvements (#1336)

* Handle matches and false positives for unacceptable words in handles

* move handle validation logic to pds

* missed merge

* add cfg flag

* encode lists

* fix build issues

* move words to cfg

* tidy

---------

Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>

* Allow moderators to take and reverse actor takedowns (#1330)

allow moderators to take and reverse actor takedowns

*  Allow searching reports by moderator did (#1283)

*  Allow searching reports by moderator did

*  Remove .only flags on tests

*  Update snapshot

*  Add checks for did match in actions

* v0.4.1

* Make sequencer leader behavior optional on pds (#1250)

* make sequencer leader behavior optional on pds

* tidy

* use 127.0.0.1 in with-test-db.sh for colima (#1297)

So, since Docker Desktop has licensing issues, some folks use colima for
running containers on their macOS machines (The licensing exempted
CLI-only version of Docker only exists on Linux).

Unfortunately, colima binds host ports only on the IPv4 localhost
address (`127.0.0.1`) while the atproto postgres clients will attempt to
connect to the IPv6 localhost address (`::1`) that macOS sets in
/etc/hosts.  See https://github.com/abiosoft/colima/issues/583 and
https://github.com/lima-vm/lima/issues/1330 for the tickets against
colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses
and so doesn't have this issue.)

To workaround this silly issue, we can use `localhost` within the docker
containers and docker-compose, but need to set the `DB_POSTGRES_URL` env
var to use the IPv4 localhost explicitly.

(Asking folks to edit /etc/hosts causes other tools to break and will be
overridden on each OS upgrade.)

* Subscription util tests (#1295)

* consecutive list tests

* flesh out subscription util tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>

* Content reporting on record fields (#1351)

* content reporting on record fields

* fix test

* tests

* tidy

* Check rkey contents just for non-tids (#1353)

check rkey content for non-tids

*  Added new procedure for sending admin email (#1312)

* 🚧 Added new lexicon for sending admin email

*  Add moderation mailer

*  Switch to text email content from html

* 🧹 Cleanup some early implementation code and reflect PR reivew

*  Use smtp host instead of gmail service config

*  Move to using single smtp url

* v0.4.2

* Patch up a couple sqlite tests (#1355)

patch up a couple sqlite tests

* enable feeds & build branch

* disable branch building & enable without proxy header

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: David Buchanan <d@vidbuchanan.co.uk>
Co-authored-by: jess <jesslarkdev+gh@gmail.com>
Co-authored-by: bnewbold <bnewbold@robocracy.org>
Co-authored-by: Jaz Volpert <ericvolp12@gmail.com>
Co-authored-by: Foysal Ahamed <cfaion341@gmail.com>
Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com>
2023-07-19 17:38:43 -05:00
devin ivy
1ebda12b9d
Make sequencer leader behavior optional on pds (#1250)
* make sequencer leader behavior optional on pds

* tidy
2023-07-17 10:53:22 -04:00
devin ivy
3ea892bc57
Add administrative triage role, update moderator role (#1216)
* setup triage user on pds, update moderator username (invalidates old token)

* initial pass on triage access on pds, limit access to email addrs

* apply moderator vs triage rules on taking and reversing mod actions for pds

* update pds tests for triage auth role

* setup moderator and triage roles on bsky appview

* apply mod and triage access rules to bsky admin endpoints

* reframe admin auth as role-based auth, tidy auth apis

* tidy

* build

* revert change to basic auth username for role-based auth
2023-07-06 15:18:52 -04:00
Daniel Holmgren
4d1f8d3289
Misc scaling (#1284)
* limit backsearch to 1 day instead of 3

* lower like count threshold

* bump to 6

* disable like count check

* disable with friends

* preemptively cache last commit

* inline list mutes

* actor service

* label cache

* placehodler on popular with friends

* bulk sequence

* no limit but chunk

* bump chunk to 5k

* try 10k

* fix notify

* tweaking

* syntax

* one more fix

* increase backfill allowance

* full refresh label cache

* limit 1 on mute list

* reserve aclu handle

* clean up testing with label cache

* note on with-friends

* rm defer from label cache

* label cache error handling

* rm branch build
2023-07-05 16:41:08 -05:00
Daniel Holmgren
c793ff9103
Add boolean for enabling generic appview proxying (#1273)
* add boolean config for enabling generic proxying

* tweak

* tweak cfg var name

* tweak
2023-07-03 14:14:49 -05:00
Daniel Holmgren
e7a0d27f1f
Proxy timeline skeleton construction (#1264)
proxy timeline skeleton construction to appview
2023-07-03 11:28:39 -05:00
Daniel Holmgren
ee68a4037b
Appview - Invalid handles (#1244)
* run with-test-db in monorepo test

* improve pg script

* tidy

* namespace bsky pg schemas

* differentiate schemas

* clean up script

* first pass at invalid handles in appview

* tests for handle invalidation

* move mock to instance instead of prototype

* change network mocks in general

* fixing pagination on actor-search

* fix snap & normalize handles on index

* handling did pagination + update tests

* one last update
2023-06-30 16:02:47 -05:00
Daniel Holmgren
7cb8c62b54
Repo update advisory lock (#1230)
* use for no key update

* use a tx advisory lock for repo updates

* skip tests for sqlite

* move check of commit swap on rebase

* do lock before formatting rebase

* hash schema in for lock id

* no tx lock in sqlite

* move rebase formatting to tx

* move dialect check

* rm log

* make the lock ids a bit safer

* change how we do lock id

* refactor id generator
2023-06-29 13:42:10 -05:00
Daniel Holmgren
207d44796d
Fix dev env package (#1245)
* v0.2.1

* fix dev-env publish

* v0.2.2

* fix package.json
2023-06-28 16:38:26 -05:00
dholms
09385bbb55 v0.2.0 2023-06-27 19:35:26 -05:00
Daniel Holmgren
5d85188509
Invite epochs (#1205)
* add invite epoch

* reorg test

* fix dev env

* tidy
2023-06-13 17:49:47 -05:00
Daniel Holmgren
8c4fa530d6
Pds proxy to appview based on headers (#1191)
* pds proxy to appview based on headers

* reorg appview routes forwarding mutes to appview

* tidy

* clean up ctx

* track notif state

* use notif state for notif routes

* add back in missing file

* pr feedback

* wip

* more views

* more views

* update snaps & suggestions test

* better choices for like & reposted

* note

* better block test

* Test pds-proxied mute and notif state

* Update bsky snaps, remove unnecessary #postView

* Update bsky notification tests to use updateSeen

* add getList getFeedGen tests

* Re-disconnect pds from in-process appview indexing for tests

* clean up forSnapshot

* tidy

* feed gen view tests

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 17:29:32 -05:00
Daniel Holmgren
08fbb33fdf
Appview blocks (#1187)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* hook up indexing

* Setup custom feeds on bsky

* graph service + getList routes

* migration

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* mute lists

* add getListMutes

* mute list tests

* add mutes table & mute/unmuteActor routes

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* add getMutes

* implement mute functionality

* remove unspecced

* mute tests

* mute impl tests

* remove proxy tests

* Test custom algos on bsky

* add mute state to snapshots

* table & indexing plugin

* getBlocks

* bail out of mute subquery when no viewer

* fleshing out services

* Test pds getFeed proxy

* tidy, fix build

* tidy

* Fix minor race in bsky mutelist tests

* blocks in feed views

* db migration, impl block logic & tests

* Update pds proxy snaps based on appview mutes

* missed some files on last commit

* ensure process after unmute

* update snap

* move adminAuth to seedclient param

* apply blocks in getFeed

* update snaps

* upadate one last snap

* pr feedback

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-06-12 11:05:22 -05:00
devin ivy
283fb1d8d6
Custom feeds on bsky appview (#1185)
* Model post and profile aggs in bsky

* Add background, onCommit, db and background stats

* Add aggregations for post and profile

* Use background queue in bsky labeler, misc plumbing

* Process background throughout bsky tests

* bsky tidy and fixes

* Build views using agg tables in bsky

* Views of feedgens on bsky

* Setup custom feeds on bsky

* Feed views refactor for bsky, implement getFeed w/ auth passthrough

* Passthrough auth from bsky appview to feedgen, fixes/tidy/tests

* Test custom algos on bsky

* Test pds getFeed proxy

* tidy, fix build

* tidy
2023-06-09 11:48:17 -04:00
Daniel Holmgren
6fb9a8faea
Fix dropped events in sync.subscribeRepos (#1167)
* wip

* first pass done

* chnage around how channels function

* bugfixing

* fix up migration & tests

* fix dev-env

* add prep migration for stream fixing

* application code changes to use new repo_event table

* dont send evts

* add outgoing_repo_seq table + sqlite migrations

* patch up down migration

* patching up a couple things

* update test

* patchup

* re-enable subscription route

* add outgoingSeq column

* updating

* accidental add

* temporarily disable stream

* resolve merge conflict

* add migration

* migration for seq

* fix test

* id -> seq

* get schema lock ids

* tidy

* fix server test

* fix up

* use bsky uniqueLockId approach

* tidy

* gh actions build

* build branch

* build branch

* tweak test

* rm constraint rename

* small bugfix

* check for seq not null

* fix up issues

* one more small change

* tidy

* no build branches
2023-06-06 19:18:44 -05:00
Foysal Ahamed
46e1f051e1
Allow searching repo by DID (#1084)
*  Allow searching repo by DID

*  Port over did search from pds

*  Add tests for bsky implementation of admin searchRepos

* 🧹 Cleanup according to PR review
2023-06-06 00:12:14 -04:00
Daniel Holmgren
743eaf1cf0
Move handle resolution to .well-known (#1048)
* move handle resolution to .well-known

* required handle on resolveHandle

* rm test

* tidy

* tidy

* fix up appview

* missing await

* atproto-handle -> atproto-did

* shift did & handle resolution to new identity package

* fix up network mocks

* fix up another test

* one more

* drop lex comment

* rm handle param

* Update packages/identity/src/handle/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* still temporarily support xrpc handle resolution

* typo

* ensure return value is a string

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-05-25 19:04:10 -05:00
Daniel Holmgren
5df2903061
Feed publishing tweaks (#1087)
* add feed gen did to cfg

* allow puts for generator records

* add a quick test

* quick test for updates

* fix up test
2023-05-24 17:38:01 -05:00
Daniel Holmgren
8059e07d8a
Custom feeds (#1001)
* actor type in profiles

* skeleton lex

* more schemas

* first pass at feed composition

* cleanup services

* more tidying

* fix up tests

* bookmarking schemas

* make feeds records instead of actors

* some cleanup

* fleshing out feeds as records

* wip

* working into cfg & dev-env

* fix appview test util

* bookmarking + views

* tidy

* add displayName

* db migration

* wip test

* blob processing

* first test working!

* Sync-up snapshots

* wip

* Fix pds ctx type

* Sync lexicons on pds and api

* wip

* fully refactor

* Prep for open union in reply data in bsky appview, sync-up snapshots

* msgs & ports

* Make pds did cache sqlite-friendly

* Start scaffolding more feedgen tests

* revamp appview tests

* revamp pds tests

* merge in dev env utils

* ensure plc close

* add feed gen to dev-env

* second feed gen

* named feeds in mock

* fix issue with dev-env pds appview

* update bsky snaps

* Flesh-out some feed gen tests

* Fix feed gen reply test

* Test feed bookmarking

* update skeleton defs

* merge

* Sync-up pds with updated feed skeleton contract

* Fix feed bookmark viewer state

* add feed embeds

* Test inaccurate feed gen repost

* Tighten some error behaviors around feed gens

* generator view in record embed

* test

* bookmarked/subscribed -> saved

* updated snaps

* name & descript on alices feed

* add like counts

* proposed lex

* tweak

* fetch feed generator details

* add describeFeedGen to dev-env

* adds tests for getFeedGenerator

* Update lexicons/app/bsky/feed/getFeedGenerator.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* update describe

* add cid to feed gen view

* add alice feed gen avatar

* generator displayName required

* update snaps

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* cleanup lex

* mount route

* Starter bsky algos (#1046)

* implement some starter bsky algos

* add team dids

* tidy

* add quotepost check to whats hot classic

* tests

* carve out feedgen section of pds

* apply limits

* add describeFeedGenerator route

* Whats hot new (#1060)

* map out new whats hot

* only available in postgres

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* build algos branch

* remove relative imports to @atproto/api

* update hot classic name

* rm build

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>

* v0.3.6

* v0.3.7

* v0.1.10

* update name of skyline

* build branch

* aws build

* getFeedGenerators test

* For now only use new replyRef union for custom feeds, for compat

* Don't use replyRef union on new feeds yet

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-05-19 18:31:51 -04:00