Unit Propagation - Subprocess IO in Lua 5.1

I wanted to have a keypress in neovim format a python script with black. Yes, this is supported through LSP and so on, but I don’t want to turn every one-off python script I write into a full-blown python project, just to trigger the right code path in neovim to have my code formatted. Conversely, formatting every single Python file I edit with Black is also undesirable. An opt-in low-friction keybinding is the perfect middle ground for my use case.

Also, it’s just a subprocess call, how much work can it be? Famous last words!

Maybe it actually is easy! I have no clue. I have a hard time navigating neovim’s docs. The way I got it to work certainly doesn’t look intuitive to me. Clearly I’ve been coddled by Python’s subprocess.run for too long. In the hope to make some future person’s life easier, I’m documenting my solution here.

Using both stdin and stdout

Lua 5.1 only allows opening a process in read or write mode, meaning that you can’t read the output of a process that you’ve supplied with data. Or at least, not without jumping through hoops.

As Edgar Toernig explains, io.popen doesn’t support this natively because getting this right with inputs and outputs both being buffered is difficult. If you’re not careful, you risk a deadlock. Summarizing, if you write some bytes to a process’ stdin, there’s no guarantee you’ll get some bytes back out from its stdout, due to buffering inbetween.

As a workaround, we are advised to either pipe the output into a file and read that after the process completes, or pipe the output into another Lua script. As I was writing a neovim keybinding, the latter seemed like a bad idea, so I figured I’d give the former a shot:

fp = io.popen("black --quiet - 1>/tmp/black_formatted_output", "w")
fp:write(buf)
fp:close()

fp = io.open("/tmp/black_formatted_output", "r")
formatted_buf = fp:read("*a")
fp:close()

Here, buf holds the contents of the currently active buffer. The trick is that you can just pass arbitrary sh shell commands to io.popen, so redirecting streams, piping stuff, or setting variables: it’s all fair game.

Now for the second problem: how to get the return code of the call, so I can detect if formatting failed?

How to get the return code of a process?

As far as I can tell, this is actually not possible via the io API. People claim that, in Lua 5.2, io.close will return the return code of the process. The reference manual doesn’t mention this, and even if it’s actually the case, neovim is stuck on Lua 5.1. Luckily, inspecting the stderr stream for any output is good enough of an approximation for my uses. This can be achieved by also redirecting the stderr stream in the io.popen call:

fp = io.popen("black --quiet - 1>/tmp/black_formatted_output 2>/tmp/black_err", "w")
fp:write(buf)
fp:close()

fp = io.open("/tmp/black_err", "r")
black_err = fp:read("*a")
fp:close()

if black_err:len() > 0 then
  vim.api.nvim_err_writeln("There was an error formatting the buffer with black. See /tmp/black_err for more info") 
  return
end

-- Process stdout as normal

In hindsight, the return code also could’ve been printed to a file using the following io.popen command:

Once you realize io.popen is just dispatching to sh, a world of frightening possibility opens up!