Massive result size even when limit=1 during trace query which hangs browser tab

[trtg]

I've included the curl request from chrome's network tab for the API call that kicks off when searching for traces with results limited to 1. What ends up happening is I see the call below transferring around 65MB+ of data and then causing the tab to hang. So I tried just running the curl call below in a terminal and sending the output to a file. After prettifying the json with jq or python -mjson.tool, the resulting json file is 151MB!! I'm not sure where/how to begin debugging this. The "processes" section of the JSON has something like 40 entries whereas a normal trace only has 4, so I'm not sure if that's a clue.
Also looking at the spans section of the JSON, spans that should only appear once per trace appear 5140 times (all referencing the same trace ID. It's as if somehow the traceID is not changing and a ton of traces got merged together?

curl 'http://jaeger.internal:16686/api/traces?end=1505521805205000&limit=1&lookback=1h&maxDuration&minDuration&operation=calling_assetselection&service=kiip_service&start=1505518205205000' -H 'Accept: /' -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Referer: http://jaeger.internal:16686/search?end=1505521805205000&limit=1&lookback=1h&maxDuration&minDuration&operation=calling_assetselection&service=kiip_service&start=1505518205205000' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36' --compressed

[yurishkuro]

I doubt this is related to any issue with the UI, more likely it's a bug in the instrumentation that causes an enormous trace. How many spans does the trace have in the DB?

[trtg]

I can't tell, trying to run this query:
select count(*) from jaeger_v1_datacenter1.traces where trace_id in (0x000000000000000001d233dd238cc036) ;
results in

ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

I've tried raising timeouts to no avail, but i assume it must be a huge number of spans, since running that same style of query on traces that have say 19 spans works fine.

[yurishkuro]

what if you do

select trace_id, span_id from jaeger_v1_datacenter1.traces where trace_id = 0x000000000000000001d233dd238cc036 LIMIT 10;

and keep increasing the limit?

[trtg]

I was able to confirm that there are traces being generated with thousands of spans. I tried blowing away all the data in cassandra and starting over from scratch, but now the traces table and service_name_index table are getting populated, but the service_names table does not, so there is no service to select in the dropdown.

There appears to be something very bizarre going on, because I did an apt-get purge of cassandra and confirmed that all its data files were gone, then setup cassandra from scratch again and I'm getting another huge trace with the exact same trace_id that i pasted 2 days ago 0x000000000000000001d233dd238cc036
how is this possible?

[trtg]

Just replying to myself here in case it's useful to anyone else- the issue appeared to be something along these lines: we had a large traffic spike which seemed to somehow corrupt our cassandra cluster (i was unable to run a nodetool repair) cassandra being down in turn meant the collector had nothing to write to and I'm guessing due to some issue in jaeger-client-python we wound up with one ec2 instance that was stuck emitting tons of spans with the same trace ID. Restarting uwsgi on that box resolved the issue.

[tiffon]

Closing as this sounds like a case where the data was inherently large rather than the query or the UI were acting up. @trtg Please re-open if I've gotten my wires crossed, here.