Skip to content

Random segfaults with req_raw_header #1313

Closed
@ghost

Description

Hello

We are seeing occasional segfaults using req_raw_header.

Here's how to reproduce:

local h = ngx.req.raw_header()

backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0817073f in ngx_http_lua_ngx_req_raw_header ()
(gdb) bt full
#0 0x0817073f in ngx_http_lua_ngx_req_raw_header ()
No symbol table info available.
#1 0xf76b23c5 in lj_BC_FUNCC () from /usr/local/openresty/luajit/lib/libluajit-5.1.so.2
No symbol table info available.
#2 0xf76c9f25 in lua_pcall (L=0xf69f41c0, nargs=0, nresults=1, errfunc=1) at lj_api.c:1129
g = 0xf69f41f0
oldh = 0 '\000'
ef =
status =
PRETTY_FUNCTION = "lua_pcall"
#3 0x0818c00c in ngx_http_lua_header_filter_by_chunk ()
No symbol table info available.
#4 0x0818c2b2 in ngx_http_lua_header_filter_file ()
No symbol table info available.
#5 0x0818bc83 in ngx_http_lua_header_filter ()
No symbol table info available.
#6 0x081b3da8 in ngx_http_headers_more_filter ()
No symbol table info available.
#7 0x082323b9 in ngx_http_subs_header_filter ()
No symbol table info available.
#8 0x081091ad in ngx_http_not_modified_header_filter ()
No symbol table info available.
#9 0x080cbbe3 in ngx_http_send_header ()
No symbol table info available.
#10 0x080cdc45 in ngx_http_send_special_response.isra ()
---Type to continue, or q to quit---
No symbol table info available.
#11 0x080ce013 in ngx_http_special_response_handler ()
No symbol table info available.
#12 0x080d1f93 in ngx_http_finalize_request ()
No symbol table info available.
#13 0x080d3aa7 in ngx_http_process_request_line ()
No symbol table info available.
#14 0x080a993b in ngx_event_process_posted ()
No symbol table info available.
#15 0x080a922b in ngx_process_events_and_timers ()
No symbol table info available.
#16 0x080b3468 in ngx_worker_process_cycle ()
No symbol table info available.
#17 0x080b1a81 in ngx_spawn_process ()
No symbol table info available.
#18 0x080b4f74 in ngx_master_process_cycle ()
No symbol table info available.
#19 0x0807f450 in main ()
No symbol table info available.

nginx version: openresty/1.13.6.1
built with OpenSSL 1.0.1t 3 May 2016

on debian jessie

I'm not sure which requests are causing this and if it's possible to find

Activity

ghost

ghost commented on May 2, 2018

@ghost

Note that i'm using header_filter_by_lua_file in the http context to call the code

ghost

ghost commented on May 2, 2018

@ghost

My full configure line

configure arguments: --prefix=/usr/local/openresty/nginx --with-debug --with-cc-opt='-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2' --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.05 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.31 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.07 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.11 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.18 --add-module=../redis2-nginx-module-0.14 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.08 --add-module=../ngx_stream_lua-0.0.3 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/lib' --with-http_sub_module --with-threads --with-file-aio --with-http_gunzip_module --with-http_ssl_module --with-ipv6 --with-http_stub_status_module --with-http_geoip_module --with-http_realip_module --error-log-path=/usr/local/nginx/logs/error.log --http-log-path=/usr/local/nginx/logs/access.log --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_perl_module --with-stream --with-stream_ssl_module

I also tried the latest lua-nginx-module with the same result

p0pr0ck5

p0pr0ck5 commented on May 2, 2018

@p0pr0ck5
Contributor

This sounds vaguely similar to an issue reported several years ago involving the ngx.req.raw_header API, also specifically on Debian Jessie (and using the system's PCRE): p0pr0ck5/lua-resty-waf#24

@Vladimir000 might you consider building OpenResty with PCRE statically, as a test?

ghost

ghost commented on May 2, 2018

@ghost

@p0pr0ck5 I'm doing tests and i found out it happens with requests with an invalid request line (first line), eg A AAA AA instead of GET / ...
Even if i pass "false" to ngx.req.raw_header.

I will try what you proposed

ghost

ghost commented on May 2, 2018

@ghost

Of course i move the code to the server context, no problem, as the request line is valid.

But in the doc it says

syntax: str = ngx.req.raw_header(no_request_line?)
context: set_by_lua*, rewrite_by_lua*, access_by_lua*, content_by_lua*, header_filter_by_lua*

and

header_filter_by_lua_file
syntax: header_filter_by_lua_file
context: http, server, location, location if

Possibly broken in the http context?

p0pr0ck5

p0pr0ck5 commented on May 2, 2018

@p0pr0ck5
Contributor

Do you have an example request/packet capture that can reliably cause a segfault, as well as a minimal Nginx config?

agentzh

agentzh commented on May 2, 2018

@agentzh
Member

@Vladimir000 It seems like you are trying to call this Lua API function in an error_page handler for 400 responses for invalid requests. DO NOT DO THAT! A segfault would almost always be guaranteed, even when using standard nginx modules in such contexts. This is because nginx does not even fully initialize the request data structure for invalid client requests and when you are trying to do something complex with the request, you'll surely run into troubles and memory issues (actually segfault is the best thing you can get).

ghost

ghost commented on May 2, 2018

@ghost

@agentzh can you be more precise, does it mean all ngx.req.* shouldn't be used in the http context?

agentzh

agentzh commented on May 3, 2018

@agentzh
Member

@Vladimir000 You should not do anything fancy, not even the standard proxy_pass or fastcgi_pass directives, inside the location serving the 400 error pages (not to mention any of those *_by_lua* directives).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @agentzh@p0pr0ck5

        Issue actions

          Random segfaults with req_raw_header · Issue #1313 · openresty/lua-nginx-module