Unable to use coroutine.wrap in libraries

[daurnimator]

Trying to use coroutine.wrap inside of libraries loaded with require throws an error.

As an example, take this example that creates+uses an iterator:
Contents of wut.lua:

local t = {}
for i in coroutine.wrap(function()
        local x 
	for i=1, 100 do
		coroutine.yield(i)
	end
end) do
	t[i] = i
end
return t

Contents of access.lua (set as access_by_lua_file):

require "wut"

Running that in openresty results in:

2018/03/28 17:29:25 [error] 16118#0: *1 lua entry thread aborted: runtime error: attempt to yield across C-call boundary
stack traceback:
coroutine 0:
	[C]: in function 'require'
	./access.lua:1: in function <./access.lua:1>, client: 127.0.0.1, server: , request: "GET /get HTTP/1.1", host: "mynode"

The snippet works fine inside of e.g. access.lua itself.

[daurnimator]

Btw, I don't think this is a duplicate of #208, as it can be replicated without the for loop.
e.g. if wut.lua contains:

local the_answer = coroutine.wrap(function() coroutine.yield(42) end)()

We get the same result.

[agentzh]

@spacewander Will you have a quick look at this? Thanks!

[spacewander]

@daurnimator
To make the coroutine API be compatible with OpenResty's own scheduler, lua-nginx-module redefine the coroutine API with C function.
https://github.com/openresty/lua-nginx-module/blob/master/src/ngx_http_lua_coroutine.c#L273

So the coroutine.* called in OpenResty is not the same functions called in LuaJIT only environment.

Although it's resumable to call LuaJIT's coroutine API in require, it is not resumable to call a C function like ngx_http_lua_coroutine_yield.

[agentzh]

@spacewander Thanks for digging it up! It all makes sense now :)

@daurnimator Better wrap your code inside a Lua function of your wut Lua module, like function init(), and let the caller call it like this:

require "wut".init()

Then you can dodge the "cannot yield across C-function boundary" limitation of LuaJIT's require() builtin.

[daurnimator]

@daurnimator Better wrap your code inside a Lua function of your wut Lua module, like function init(), and let the caller call it like this:

The issue is that coroutine.wrap is used by (existing) lua libraries. Ones that don't require an .init call to start working (and that's a terribly ugly API!).

Looking at the code at

lua-nginx-module/src/ngx_http_lua_coroutine.c

Line 301 in 91a0ad2

"if ctx ~= 0x020 and ctx ~= 0x040 then\n"

I think we could add an extra condition that if the current coroutine is not the top level, then use std instead of ours.

[spacewander]

@daurnimator
I am afraid whether this hack could work, since it bypass the OpenResty's coroutine scheduler.
The std coroutine is used in header_filter_by_lua* and body_filter_by_lua* because there are not yield in these phases.

Anyway, you could still call the origin coroutine.xx API via coroutine._xx, if you need.

[daurnimator]

I am afraid whether this hack could work, since it bypass the OpenResty's coroutine scheduler.

Wouldn't this effctively be modifying the coroutine scheduler rather than bypassing it?

The other option is: what if we do change the coroutine scheduler to make this a bit less messy?
Though that's going to be a larger project.

The std coroutine is used in header_filter_by_lua* and body_filter_by_lua* because there are not yield in these phases.

That's even weirder.... it means that such a library works if I first require it in header_filter_by_lua, but not if I first require it in content_by_lua...

[spacewander]

That's even weirder.... it means that such a library works if I first require it in header_filter_by_lua, but not if I first require it in content_by_lua...

As I have mentioned, you could still use the original coroutine API if you want.
The phase limitation resultes from how Nginx handles the request in different phases. It is not quite strange that different phases have different behaviors. Note that the require function of your module has side effect, so it is not can't not require a module in some phases, it is can't create some side effect in some phases to require a module.

[daurnimator]

Note that the require function of your module has side effect, so it is not can't not require a module in some phases, it is can't create some side effect in some phases to require a module.

What is the side effect in:

local the_answer = coroutine.wrap(function() coroutine.yield(42) end)()
return {
    the_answer = the_answer
}

[spacewander]

@daurnimator
The snippet creates a coroutine when people try to require it, so it could not be required if creating a coroutine is restricted.

[daurnimator]

But in the example, the coroutine should never yield to outside of the module: the creation, use and discarding of the coroutine is within the library.

I think we should rework the coroutine overrides to not yield back to the main loop when they don't have to, which will fix this error of trying to yield through require.

[DarkWiiPlayer]

I'm gonna risk sounding stupid and ask: Doesn't CoCo exist precisely so this problem doesn't happen? I'm not very familiar with the inner workings of openresty, but from what I understand, the only reason for re-implementing coroutines is to make openrestys non-blocking API transparent to the user, or am I mistaken there?