[BUG]: Error on startup when using devproxy with Azure Functions Isolated Worker (HTTP/2 Connection)

[shueseman]

Description

I am trying to use devproxy with my Azure Functions (isolated worker) to simulate errors with Graph. However, after running devproxy, then launching my function in debug mode, I get the following error:

Grpc.Core.RpcException: 'Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while unable to establish HTTP/2 connection.", DebugException="System.Net.Http.HttpRequestException: Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while unable to establish HTTP/2 connection.")'

It looks like a similar issue was reported before with Fiddler (another proxy), so it's something to do with the proxy itself:

https://stackoverflow.com/questions/68724527/fiddler-with-dotnet-isolated-az-function-error-starting-grpc-call

I looked over the devproxy config options and other documentation, and don't see anything that address this. Any ideas?

Expected behaviour

Able to startup Azure Function Isolated Worker in debug mode while devproxy is running.

Actual behaviour

Error above is thrown during HostBuilder.RunAsync

Steps to reproduce

Start devproxy
In Visual Studio 2022, launch debug mode on an Azure Function Isolated Worker project.

Dev Proxy Version

0.19.1

Operating system (environment)

Windows

Shell

cmd

Configuration file

{
"$schema": "https://raw.githubusercontent.com/microsoft/dev-proxy/main/schemas/v0.19.1/rc.schema.json",
"plugins": [
{
"name": "RetryAfterPlugin",
"enabled": true,
"pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll"
},
{
"name": "GenericRandomErrorPlugin",
"enabled": true,
"pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
"configSection": "genericRandomErrorPlugin"
},
{
"name": "GraphRandomErrorPlugin",
"enabled": true,
"pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
"configSection": "graphRandomErrorPlugin",
"urlsToWatch": [
"https://graph.microsoft.com/v1.0/",
"https://graph.microsoft.com/beta/",
"https://graph.microsoft.us/v1.0/",
"https://graph.microsoft.us/beta/",
"https://dod-graph.microsoft.us/v1.0/",
"https://dod-graph.microsoft.us/beta/",
"https://microsoftgraph.chinacloudapi.cn/v1.0/",
"https://microsoftgraph.chinacloudapi.cn/beta/"
]
}
],
"urlsToWatch": [
"https://jsonplaceholder.typicode.com/",
"https://graph.microsoft.com/",
"https://graph.microsoft.us/*"
],
"genericRandomErrorPlugin": {
"errorsFile": "devproxy-errors.json"
},
"rate": 50,
"logLevel": "information",
"newVersionNotification": "stable"
}

Additional Info

No response

[waldekmastykarz]

Thanks for bringing this up @shueseman. I've seen this before, but so far, I haven't been able to figure out how to fix it. As a workaround, you can start the Azure Function first, and then start proxy. Sorry for the trouble. Let's keep the issue open in case we have a solution for it.

@svrooij does this seem familiar by any chance?

[svrooij]

Azure Functions Isolated work as following

1. You connect to the function process
2. The function process acts like a reverse proxy to the "isolated" process
3. Isolated process responds
4. Function process returns the response to the respective output

This proxy process uses grpc (streaming) over http/2

Dev Proxy issue

If you start the dev proxy as system proxy, it will also try to proxy between functions process and isolated process, which is http/2 (default disabled, experimental, no streaming support).

Maybe it helps to not start it as system proxy and then configure the proxy only for the isolated process.

By setting the environment variable HTTPS_PROXY for the functions process.

[shueseman]

Thanks for bringing this up @shueseman. I've seen this before, but so far, I haven't been able to figure out how to fix it. As a workaround, you can start the Azure Function first, and then start proxy. Sorry for the trouble. Let's keep the issue open in case we have a solution for it.

@svrooij does this seem familiar by any chance?

I did try starting the function first, then starting the proxy, but none my graph calls from the function were going through the proxy.

I will try what svrooij suggested, and let you know.

[shueseman]

I ran: devproxy --as-system-proxy false
Then added an environment variable HTTPS_PROXY with value https://127.0.0.1:8000

And now on startup, it's failing during HostBuilder.Build with an exception:

AggregateException: Retry failed after 3 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)

[svrooij]

@shueseman it seems that the function runtime also uses the dotnet http client. Can you also add the env variable NO_PROXY with the value localhost,127.0.0.1. That should disable the https proxy for all those hosts.

[shueseman]

I am getting the same error.

Here is my launchSettings.json, maybe I am not entering something correctly? When I remove the environmentVariables entries, it launches fine (but of course not using the devproxy then).

{
"profiles": {
"MyFunctionApp": {
"commandName": "Project",
"commandLineArgs": "--port 7221",
"environmentVariables": {
"HTTPS_PROXY": "https://127.0.0.1:8000",
"NO_PROXY": "localhost,127.0.0.1"
}
}
}
}

[svrooij]

What is you loose the https:// from the proxy url? According to the docs:

The proxy server may be a hostname or IP address, optionally followed by a colon and port number, or it may be an http URL, optionally including a username and password for proxy authentication. The URL must be start with http, not https, and cannot include any text after the hostname, IP, or port.

If the dev proxy is messing with the functions runtime, what does it say? I think we have to say that setting those environment variables also messes with the Functions runtime.

The alternative way to set a proxy for all outbound http requests from the proxy is to configure a proxy in the Program.cs file.

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults(functionBuilder =>
    {
        functionBuilder.RegisterAuthorizationMiddleware();
        functionBuilder.ConfigureTablesExtension();
    })
    .ConfigureServices((context, services) =>
    {
      services.AddHttpClient().ConfigureHttpClientDefaults(builder =>
      {
         builder.ConfigurePrimaryHttpMessageHandler(() =>
         {
          return new HttpClientHandler
          {
            Proxy = new WebProxy("http://localhost:8000"),
            ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => true
          };
         });
    })
    .Build();

host.Run();

[shueseman]

What is you loose the https:// from the proxy url? According to the docs:

That did it! Now the function app starts up, and devproxy is returning random errors on my graph calls.

Thanks!

Update, just to recap, for using devproxy with Isolated Worker Azure functions:

1. run devproxy --as-system-proxy false
2. Add environment variables:
   "HTTPS_PROXY": "127.0.0.1:8000",
   "NO_PROXY": "localhost,127.0.0.1"

[waldekmastykarz]

For HTTPS_PROXY you could also use http://127.0.0.1:8000. HTTPS_PROXY means a proxy for HTTPS request, but it can still point to a proxy running on HTTP, and Dev Proxy runs on http://127.0.0.1:8000 rather than https. Awesome to hear you're unblocked!

We've got some guidance about using Dev Proxy with Azure Functions, but perhaps we should extend it with some more information to ensure others won't run into the same issue.

[martinlingstuyl]

Awesome @svrooij, this unblocked me today as well!

[waldekmastykarz]

We've just submitted an update to our docs and the changes should be reflected shortly. Thanks again all!