submitted by /u/Stukwan [link] [comments]

submitted by /u/Stukwan [link] [comments]

Article URL: https://github.com/asalsali/covenant-framework-community Comments URL: https://news.ycombinator.com/item?id=49569003 Points: 2 # Comments: 0
We needed a framework to write our evals in and easily update them, run A/B tests, set all kinds of constraints (ex: timeouts, number of turns), and configure the execution environment (ex: sandboxes, dependencies, how to handle AskQuestion). We also have an agent judge. Itβs all in a YAML file now. We seem to be moving toward a world where companies rely more and more on evals to decide what to ship, so this should be super helpful for normalizing evals across teams. Comments URL: https://news.ycombinator.com/item?id=49568876 Points: 2 # Comments: 0