---
title: "Subprocess IO in Lua 5.1"
author: Bob Rubbens
publish_timestamp: 2026-08-23T15:52:31+02:00
state: published
template: template/post.mako
id: 3f90ac56-e09f-4daf-9a18-845229415400
---

I wanted to have a keypress in neovim format a python script with [`black`](https://black.readthedocs.io/en/stable/). Yes, this is supported through LSP and so on, but I don't want to turn every one-off python script I write into a full-blown python *project*, just to trigger the right code path in neovim to have my code formatted. Conversely, formatting every single Python file I edit with Black is also undesirable. An opt-in low-friction keybinding is the perfect middle ground for my use case.

Also, it's just a subprocess call, how much work can it be? Famous last words!

Maybe it actually *is* easy! I have no clue. I have a hard time navigating neovim's docs. The way I got it to work certainly doesn't look intuitive to me. Clearly I've been coddled by Python's `subprocess.run` for too long. In the hope to make some future person's life easier, I'm documenting my solution here.

If you just need some code to paste that solves the problem, consider [this cute reusable Lua function](https://stackoverflow.com/a/42644964/2078414) from David Qin. 

Moving on, these were the hurdles to be taken:

- How to use both stdin and stdout when doing an [`io.popen`](https://www.lua.org/manual/5.1/manual.html#pdf-io.popen) call?
- How to get the return code of said call?

# Using both stdin and stdout

Lua 5.1 only allows opening a process in read or write mode, meaning that you can't read the output of a process that you've supplied with data. Or at least, not without jumping through hoops.

[As Edgar Toernig explains](http://lua-users.org/lists/lua-l/2007-10/msg00189.html), `io.popen` doesn't support this natively because getting this right with inputs and outputs both being buffered is difficult. If you're not careful, you risk a deadlock. Summarizing, if you write some bytes to a process' stdin, there's no guarantee you'll get some bytes back out from its stdout, due to buffering inbetween.

As a workaround, we are advised to either pipe the output into a file and read that after the process completes, or pipe the output into another Lua script. As I was writing a neovim keybinding, the latter seemed like a bad idea, so I figured I'd give the former a shot:

```lua { .numberLines }
fp = io.popen("black --quiet - 1>/tmp/black_formatted_output", "w")
fp:write(buf)
fp:close()

fp = io.open("/tmp/black_formatted_output", "r")
formatted_buf = fp:read("*a")
fp:close()
```

Here, `buf` holds the contents of the currently active buffer. The trick is that you can just pass arbitrary `sh` shell commands to `io.popen`, so redirecting streams, piping stuff, or setting variables: it's all fair game.

Now for the second problem: how to get the return code of the call, so I can detect if formatting failed?

# How to get the return code of a process?

As far as I can tell, this is actually not possible via the `io` API. [People claim that](https://stackoverflow.com/a/14031974/2078414), in Lua 5.2, `io.close` will return the return code of the process. The reference manual doesn't mention this, and even if it's actually the case, neovim is stuck on Lua 5.1. Luckily, inspecting the stderr stream for any output is good enough of an approximation for my uses. This can be achieved by also redirecting the stderr stream in the `io.popen` call:

```lua { .numberLines }
fp = io.popen("black --quiet - 1>/tmp/black_formatted_output 2>/tmp/black_err", "w")
fp:write(buf)
fp:close()

fp = io.open("/tmp/black_err", "r")
black_err = fp:read("*a")
fp:close()

if black_err:len() > 0 then
  vim.api.nvim_err_writeln("There was an error formatting the buffer with black. See /tmp/black_err for more info") 
  return
end

-- Process stdout as normal
```

In hindsight, the return code also could've been printed to a file using the following `io.popen` command:

```
black --quiet - 1>/tmp/stdout 2>/tmp/stderr; echo $? > /tmp/returncode
```

Once you realize `io.popen` is just dispatching to `sh`, a world of frightening possibility opens up!