Handling of expected error codes coming from grpc storage plugins

[sergiimk]

Requirement - what kind of business use case are you trying to solve?

We are implementing a custom gRPC-based storage plugin as per this doc.

Problem - what in Jaeger blocks you from solving the requirement?

When searching in jaeger UI for a non-existing trace ID the GetTrace function in the storage plugin is expected to return spanstore.ErrTraceNotFound. However the error handling on the Jaeger side doesn't seem to handle this error correctly and treats it as a generic internal error:

{"level":"error","ts":1566246187.758402,"caller":"app/http_handler.go:409","msg":"HTTP handler, Internal Server Error","error":"grpc stream error: rpc error: code = Unknown desc = trace was not found", ...

UI displays:

HTTP Error: grpc stream error: rpc error: code = Unknown desc = trace was not found

You can easily reproduce this issue by running Jaeger with the example in-memory storage gRPC plugin and searching for a trace.

Impact: cosmetic in this case, but other expected error types returned by gRPC plugin may be affected as well.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Any open questions to address

[Abhi-khandelwal]

Hi @yurishkuro , Can I work on this issue ?

[yurishkuro]

@Abhi-khandelwal by all means

[chandresh-pancholi]

@sergiimk I will be picking this up as there is no activity on this thread.

[chandresh-pancholi]

@sergiimk Could you please help me out here with the reproduction of this issue?

[sergiimk]

@chandresh-pancholi you can reproduce this by compiling the memstore-plugin and running Jaeger with it as described here. Searching for a random traceID in the UI will then return an ugly-looking HTTP error.

The sequence of commands should look roughly like this:

$ go build -o memstore-plugin examples/memstore-plugin/main.go
$ SPAN_STORAGE_TYPE="grpc-plugin" \
	go run -tags ui ./cmd/all-in-one/main.go \
	--grpc-storage-plugin.binary=`pwd`/memstore-plugin

[chandresh-pancholi]

@sergiimk Thanks, I am able to reproduce it.
Please correct me if I am wrong.
Currently getting:

{
  "data": null,
  "total": 0,
  "limit": 0,
  "offset": 0,
  "errors": [
    {
      "code": 500,
      "msg": "grpc stream error: rpc error: code = Unknown desc = trace was not found"
    }
  ]
}

Expecting:

{
  "data": null,
  "total": 0,
  "limit": 0,
  "offset": 0,
  "errors": [
    {
      "code": 500,
      "msg": "trace was not found"
    }
  ]
}

[chandresh-pancholi]

PR Raised. Please review.