At AbanteAI we’ve been working on various AI developer tools such as mentat and rawdog. Today we’re announcing for closed beta our github bot, Mentat Butler, which can automatically review and write pull requests. Apply for access here. It’s not perfect but I’ve already found it surprisingly useful. Here are its contributions to its own repo:

Currently my workflow is:

1. Write an Issue.

2. Ask Butler to make a PR for it.

3. Check it out locally and finish it up.

That last step isn’t even always necessary. Butler wrote this first try:

In response to this issue:

Github actions are of course easy mode because they don’t require much project context and are mostly boilerplate. But often Butler surprises me with what it can do. For instance after this interaction Butler made a pretty good first try:

And then after these two requests made almost a complete correct PR:

Note the thumbs up on my request. It’s a little touch but it makes it a lot more fun to talk to the bot.

You can see an open source contribution here.

Why a github interface?

Given the bot can’t do everything and often makes mistakes that need to be corrected, why have it be a github bot at all instead of run locally? There are a lot of benefits:

1. Github issues are more durable than prompts. Say I’m about to start something that I’m pretty sure an LLM could do. I could write a bunch about my problem into chatgpt and get a response. But if I don’t like the response then the prompt is sort of wasted. If instead I wrote a github issue I have a record of something I wanted to do. If after seeing the LLM’s attempt I decide it is unlikely to succeed I’ll have it for later or I can pass it on to someone else to do.

2. Similarly pull requests are more durable than LLM responses. If I see the LLM’s response and like parts of it but think it’ll take some work to get to a finished state it’s easy to just lose it in my chat history if I decide to work on something else. But if it’s a pull request it’s easier to leave and return to later.

3. It’s very fun to get automatic feedback on your PR. Presently the bot is a bit too positive which is part of this. But still it feels good to feel like someone is looking at your work.

Beta sign up

If this workflow/bot sounds exciting to you, request access to the closed beta here.