2067 Commits

Author SHA1 Message Date
devin ivy
6ee810c6d8
Minor perf tweaks for getTimeline and getRepo (#1111)
* Avoid long scans for users w/ sparse feeds on getTimeline

* Increase getRepo commit chunk size
2023-05-29 11:33:46 -04:00
Devin Ivy
e0d3c58113 v0.3.9 2023-05-29 00:54:16 -04:00
Foysal Ahamed
3414fcedbf
Allow ignoring moderation reports by subject (#1091)
*  Allow ignoring moderation reports by subject

* 🔧 Remove unwanted package.json change

*  Remove subject format specifier

*  Port over ignore subject implementation to bsky package

* Revert " Port over ignore subject implementation to bsky package"

This reverts commit 3c782c1805548d1ebdbbd08b68b6e92ac97fdee5.

* 🚧 Port over ignore subject implementation to bsky package

*  Generate lexicons for bsky
2023-05-29 00:39:03 -04:00
devin ivy
237bb97e20
Log backgrounded error when refreshing did cache (#1110)
* Log backgrounded error when refreshing did cache

* remove branch build
2023-05-29 00:23:17 -04:00
Daniel Holmgren
693c703d78
Skip popular with friends test (#1109)
skip test
2023-05-26 18:48:35 -05:00
Daniel Holmgren
ec15af5de8
Algo tweaks (#1108)
* Temporarily skip online/valid checks for feedgens while external feedgens catch-up

* tweak algos

* hook up did resolver with cache

* Re-enable whats-hot w/ materialization

* pulling a fast one on the query optimizer

* clean up

* disable branch build

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
2023-05-26 18:27:23 -05:00
devin ivy
d00e3112d0
Update pool size for view maintainer (#1107)
Update pool size for view maintainer, prettier format
2023-05-26 19:19:58 -04:00
Daniel Holmgren
43e83a8681
Improving mutuals again (#1106)
* another go at it

* one more go at it

* add self to best of follows
2023-05-26 17:34:15 -05:00
Daniel Holmgren
a9c70050fd
Improve mutuals feed (#1105)
fix
2023-05-26 16:25:07 -05:00
Daniel Holmgren
12dc71931a
Best of follows feed (#1102)
add best of follows feed
2023-05-26 15:02:36 -05:00
devin ivy
212a3a7456
Improve logging around materialized view refresh (#1104) 2023-05-26 16:02:01 -04:00
Daniel Holmgren
9489194841
Mutuals feed (#1101)
add mutuals feed
2023-05-26 15:01:18 -05:00
devin ivy
9eb817d488
Materialize candidates for whats-hot algo (#1086)
* Setup whats-hot materialized view w/ params, run view maintainer in service entrypoint

* Update whats-hot to use new materialized view, tidy

* Update migration to create with no data

* Revert whats-hot migration change for no data

* Bump refresh rate for view down to 1min
2023-05-26 15:05:53 -04:00
devin ivy
24d50e8f33
Standalone migration for whats-hot algo materialized view (#1100) 2023-05-26 14:20:39 -04:00
devin ivy
549688a67a
Implement quicker mute checks on pds (#1098) 2023-05-26 12:49:43 -05:00
Daniel Holmgren
03a74b4259
Catch missed txn rollbacks (#1094)
* catch missed rollbacks

* change approach

* revert back to simple way

* test

* add to bsky package as well

* improve handler & tests
2023-05-26 11:04:26 -05:00
devin ivy
7a7c9c75af
Improved custom algo debugging (#1096)
* Better feedgen errors in getFeed and getFeedGenerator

* Support handler output headers in codegen

* Support handler output headers in xrpc-server

* Implement server-timing helpers and apply them to getFeed

* fix codegen issue

---------

Co-authored-by: dholms <dtholmgren@gmail.com>
2023-05-26 11:59:05 -04:00
Miles Zimmerman
5ca6ce31e1
Increase flexibility of paramaters accepted by DB_CREDS_JSON env var (#974) 2023-05-25 21:02:20 -05:00
Daniel Holmgren
e53fb27e32
Bugfix for well-known handle resolution (#1097)
bugfix for well-known handle resolution
2023-05-25 20:26:09 -05:00
Patryk
18c9924d83
Make lexicon validation strict (#1088)
This makes the whole type of bugs like #1080 impossible,
Also fixes `app.bsky.feed.defs` in the bsky package as it was now throwing errors

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
2023-05-25 19:23:59 -05:00
Daniel Holmgren
41330ca6b0
Rebase rework (#1095)
* rework rebases a bit

* include did in err

* move formatting outside of txn

* reorg after rebase processing
2023-05-25 19:23:47 -05:00
David Buchanan
c62964b7b4
Limit getPostThread recursion into parents (#1062)
* limit getPostThread recursion into parents

* add parentHeight parameter

* Lexicons: specify default/min/max values for depth and parentHeight, for getPostThread

* prettier formatting

* re-run lexicon codegen

* do the rest of lexicon codegen (oops)

* add 'shallow ancestors' fetch test
2023-05-25 19:22:05 -05:00
Daniel Holmgren
0307340c5a
Label after write (#1067)
* label imgs after write completes

* in on commit fn

* dont do in commit hook
2023-05-25 19:14:12 -05:00
Daniel Holmgren
743eaf1cf0
Move handle resolution to .well-known (#1048)
* move handle resolution to .well-known

* required handle on resolveHandle

* rm test

* tidy

* tidy

* fix up appview

* missing await

* atproto-handle -> atproto-did

* shift did & handle resolution to new identity package

* fix up network mocks

* fix up another test

* one more

* drop lex comment

* rm handle param

* Update packages/identity/src/handle/index.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* still temporarily support xrpc handle resolution

* typo

* ensure return value is a string

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-05-25 19:04:10 -05:00
Patryk
5f93577aaf
Fix incorrect union validation (#1093)
* Fix incorrect union validation

Fixes lexicon validation so incorrect types like in #1089 don't validate

* Add lexUri hash segment count check
2023-05-25 19:03:54 -05:00
dholms
85bcd18a7b v0.1.11 2023-05-24 17:43:02 -05:00
dholms
2248218059 v0.3.8 2023-05-24 17:41:48 -05:00
Daniel Holmgren
5df2903061
Feed publishing tweaks (#1087)
* add feed gen did to cfg

* allow puts for generator records

* add a quick test

* quick test for updates

* fix up test
2023-05-24 17:38:01 -05:00
devin ivy
53044ae48e
Get popular feedgens (#1090)
* Lexicon for unspecced popular feed gen method

* Endpoint impl for popular feed gens
2023-05-24 17:37:53 -05:00
devin ivy
f7c502b145
Improve performance of whats-hot custom algo (#1078)
Improve whats-hot algo perf by reorganizing query, excluding some unpopular results more proactively
2023-05-23 23:01:51 -04:00
Ilya Siamionau
a22d56aa21
Fix lexicon. "format" instead of "ref" in type "string" (#1080)
* Fix lexicon

* codegen types
2023-05-23 16:06:03 -04:00
devin ivy
7e22719967
Tweaks to with-friends algo (#1082)
* Iteration on with-friends algo

* Fix with-friends tests
2023-05-23 15:50:45 -04:00
Patryk
d1cb392241
Fix definitions in bsky (#1081)
Lexicon required properties check wasn't rebased onto main
which introduced invalid lexicon(creator field was not defined despite being in the required fields list)
2023-05-23 10:19:43 -04:00
Patryk
36d4e5bb00
Normalize http headers before sending xrpc requests (#1039)
* Normalize http headers before sending xrpc requests

This isn't fully upto HTTP Spec, but it solves #1029.
The `normalizeHeaders` function deduplicates HTTP headers,
but it doesn't take into account the rules from the rfc*

When using `AtpAgent` the `authorization` header supplied
to the xrpc call will take precedence over the internal one

Mentioned Rfc: https://www.rfc-editor.org/rfc/rfc9110#appendix-A

* Fix xrpc call headers double normalization
2023-05-23 01:15:28 -04:00
Patryk
d661a60357
Add lexicon LexXrpcParameters, Object required properties check (#1040)
* Add lexicon LexXrpcParameters, Object required properties check

Now schemas like the one below won't validate
```
...
{

  "type": "object",
  "required": ["foo"],
  "properties": {}
}
...
````

* Fix lexicons with missing required fields

* Improve lexicon user type error messages and document why it needs to be `z.custom`
2023-05-23 01:02:36 -04:00
Daniel Holmgren
8059e07d8a
Custom feeds (#1001)
* actor type in profiles

* skeleton lex

* more schemas

* first pass at feed composition

* cleanup services

* more tidying

* fix up tests

* bookmarking schemas

* make feeds records instead of actors

* some cleanup

* fleshing out feeds as records

* wip

* working into cfg & dev-env

* fix appview test util

* bookmarking + views

* tidy

* add displayName

* db migration

* wip test

* blob processing

* first test working!

* Sync-up snapshots

* wip

* Fix pds ctx type

* Sync lexicons on pds and api

* wip

* fully refactor

* Prep for open union in reply data in bsky appview, sync-up snapshots

* msgs & ports

* Make pds did cache sqlite-friendly

* Start scaffolding more feedgen tests

* revamp appview tests

* revamp pds tests

* merge in dev env utils

* ensure plc close

* add feed gen to dev-env

* second feed gen

* named feeds in mock

* fix issue with dev-env pds appview

* update bsky snaps

* Flesh-out some feed gen tests

* Fix feed gen reply test

* Test feed bookmarking

* update skeleton defs

* merge

* Sync-up pds with updated feed skeleton contract

* Fix feed bookmark viewer state

* add feed embeds

* Test inaccurate feed gen repost

* Tighten some error behaviors around feed gens

* generator view in record embed

* test

* bookmarked/subscribed -> saved

* updated snaps

* name & descript on alices feed

* add like counts

* proposed lex

* tweak

* fetch feed generator details

* add describeFeedGen to dev-env

* adds tests for getFeedGenerator

* Update lexicons/app/bsky/feed/getFeedGenerator.json

Co-authored-by: devin ivy <devinivy@gmail.com>

* update describe

* add cid to feed gen view

* add alice feed gen avatar

* generator displayName required

* update snaps

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* cleanup lex

* mount route

* Starter bsky algos (#1046)

* implement some starter bsky algos

* add team dids

* tidy

* add quotepost check to whats hot classic

* tests

* carve out feedgen section of pds

* apply limits

* add describeFeedGenerator route

* Whats hot new (#1060)

* map out new whats hot

* only available in postgres

* Fix dev env public url (#1058)

* Fix the public url used by dev-env

* move publicUrl location

---------

Co-authored-by: dholms <dtholmgren@gmail.com>

* add defs for feed gen prefs & decrease displayName length

* build branch

* updated how saved feeds works

* build algos branch

* remove relative imports to @atproto/api

* update hot classic name

* rm build

---------

Co-authored-by: Paul Frazee <pfrazee@gmail.com>

* v0.3.6

* v0.3.7

* v0.1.10

* update name of skyline

* build branch

* aws build

* getFeedGenerators test

* For now only use new replyRef union for custom feeds, for compat

* Don't use replyRef union on new feeds yet

---------

Co-authored-by: Devin Ivy <devinivy@gmail.com>
Co-authored-by: Paul Frazee <pfrazee@gmail.com>
2023-05-19 18:31:51 -04:00
devin ivy
0899510e6f
Db migration to prep for custom feeds (#1074) 2023-05-19 16:40:47 -04:00
Daniel Holmgren
bfd0670d38
Update README + issue templates (#1061)
* add contribution guidelines to readme

* add issue templates

* edit
2023-05-18 19:39:00 -05:00
bnewbold
d8785649c6
auto-label: less sensitive for some classes (#1071) 2023-05-18 16:31:37 -07:00
Daniel Holmgren
cf36b363b4
Rkey len (#1066)
* set a lex limit on rkey length

* enforce in repo package

* rm checks in repo pkg

* lift constraint on update
2023-05-18 17:57:59 -05:00
Devin Ivy
37af595a73 v0.3.4 2023-05-17 17:02:57 -04:00
Elijah
bb4dd6e8a8
Count graphemes without Intl (#1044)
* count graphemes without intl

* fix main file

* run prettier

* use graphemer

* fix packages

* use yarn

* fix main script

* reresolve packages
2023-05-17 14:32:29 -05:00
devin ivy
e57c297a82
Handling deleted content on admin/moderation methods (#1045)
* Update lexicon for missing repo/record data in admin views

* Serve missing repo/record variant in admin views

* Test actioning and presenting unknown records/repos, fix on pds
2023-05-17 15:28:52 -04:00
Patryk
1de53e99ab
Profiling (#1015)
* Add bench/profiling docs

* Fix repo benches

We probably want to create a shared config for benchmarks,
this fix isn't universal across packages

* Add benchs to API

This should help with #915 and any other future API performance investivations

* Make repo `bench:profile` wait for debugger

* Update packages/README.md

---------

Co-authored-by: Daniel Holmgren <dtholmgren@gmail.com>
2023-05-17 10:43:42 -05:00
bnewbold
2fc058046b
more labeling tweaks (#1053)
* labeling: include 'sex toy' as 'sexually suggestive'

* labeling: label 'hanging' as 'corpse' (for now)

* labeling: include 'underwear' as 'sexual'
2023-05-16 17:32:46 -07:00
Paul Frazee
61c946501b
Add to CONTRIBUTORS (#1047) 2023-05-15 21:00:28 -05:00
Daniel Holmgren
6ad16b2fa4
Ensure sequencing is the last query in each tx (#1036)
* ensure that inserting repo_seq is the last thing to happen in every tx

* another
2023-05-15 14:09:27 -05:00
dholms
7dddc861ae Tidy logs 2023-05-15 12:31:09 -05:00
Daniel Holmgren
47d30c1796
Revamp dev env (#1031)
* wip

* wip

* fully refactor

* msgs & ports

* revamp appview tests

* revamp pds tests

* ensure plc close

* ensure pds appview is hooked up when not running appview

* move service setup to dev env

* move admin auth headers to dev-env

* fix service headers

* missing port

* fixing ports

* start services in order

* startup plc immediately

* rm dead code

* fix up build

* rm log
2023-05-15 12:28:36 -05:00
Daniel Holmgren
231729db72
Return email in mod views according to lexicon (#885)
* return email in mod views according to lexicon

* Update packages/pds/src/services/moderation/views.ts

Co-authored-by: devin ivy <devinivy@gmail.com>

* updated snaps

* tidy

* Fix snapshot

---------

Co-authored-by: devin ivy <devinivy@gmail.com>
2023-05-15 11:43:06 -04:00