Bedrock Runtime Nova Sonic bidirectional streaming on EC2 instance - no output event stream

[oleg-boulanov]

Describe the bug

Hello.

I'm building an app using bedrock-runtime, with AWS SDK C++ (built with AWS CRT).
Reached the point of unit tests succeeding on my laptop, but failing on EC2 instance.
Basically, there is no event stream coming to the instance from Sonic.
VPC does not have bedrock service endpoints.

I wonder if that could be the issue?
What are the options to resolve it, if any?

AWS SDK C++ built on Windows with:

-G "Visual Studio 17 2022" 
-A Win32 
-DBUILD_ONLY="bedrock-runtime" 
-DTARGET_ARCH=Windows 
-DBUILD_SHARED_LIBS=ON 
-DUSE_OPENSSL=OFF 
-DUSE_CRT_HTTP_CLIENT=ON

Thanks,
Oleg

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Sonic output event stream coming from service endpoint to EC2 instance.

Current Behavior

There is no Sonic output event stream, if program (unit tests) is running on EC2 instance,
but streaming is successful if running on developer laptop outside of AWS.

Reproduction Steps

I'm not sure if this is a bug

Possible Solution

Overriding HTTP endpoint with public address?

Additional Information/Context

No response

AWS CPP SDK version used

1.11.705

Compiler and Version used

MS Visual Studio/Build Tools

Operating System and version

Windows 11 (10.0.26100), Amazon Windows 2022 AMI

[sbaluja]

I'm not sure I fully understand the issue here, is your claim that bedrock-runtime unit tests fail on an EC2 instance?

[oleg-boulanov]

No, not those.
The one I wrote for my program (I stress this to avoid blaming it on configuration and setup), which uses bedrock-runtime bidirectional streaming.
They run fine on development computer outside of AWS, but fail to receive the stream from bedrock, if running on EC2 instance.

I don't have much experience with HTTP/2 bidirectional streaming, so I wonder if the EC2 instance needs to be reachable from bedrock-runtime service (like, for example, for Chime Voice Connector)?

If it is a requirement, then I think the problem (as logs show) is that Bedrock Client uses private network IP, instead of public (elastic)

[oleg-boulanov]

On standalone computer, in corporate network:

[DEBUG] socket [57612] id=0093D830 handle=0000041C: initializing with domain 0 and type 0
[DEBUG] socket [57612] id=0093D830 handle=0000041C: setting socket options to: keep-alive 1, keep idle 0, keep-alive interval 30, max failed probe count 0
[DEBUG] socket [57612] id=0093D830 handle=0000041C: beginning connect.
[DEBUG] socket [57612] id=0093D830 handle=0000041C: connecting to endpoint 54.145.148.192:443.
[DEBUG] socket [57612] id=0093D830 handle=0000041C: setting socket options to: keep-alive 1, keep idle 0, keep-alive interval 30, max failed probe count 0
[DEBUG] socket [57612] id=0093D830 handle=0000041C: connection success
[DEBUG] socket [57612] id=0093D830 handle=0000041C: local endpoint 192.168.68.51:62288
[DEBUG] channel-bootstrap [57612] id=008CC918: client connection on socket 0093D830 completed with error 0 : AWS_ERROR_SUCCESS
[DEBUG] channel [57612] id=008ECE00: Beginning creation and setup of new channel.
[DEBUG] channel [57612] id=008ECE00: setup complete, notifying caller.
[DEBUG] channel [57612] id=008ECE00: no message pool is currently stored in the event-loop local storage, adding 008E5748 with max message size 16384, message count 4, with 4 small blocks of 128 bytes.
[DEBUG] channel-bootstrap [57612] id=008CC918: channel 008ECE00 setup succeeded: bootstrapping.
[DEBUG] socket-handler [57612] id=00906E70: Socket handler created with max_read_size of 16384
[DEBUG] tls-handler [57612] Checking Windows Version: running windows build 20348 or later
[DEBUG] tls-handler [57612] id=00958F40: Setting SNI to bedrock-runtime.us-east-1.amazonaws.com
[DEBUG] tls-handler [57612] id=00958F40: Kicking off TLS negotiation
[DEBUG] tls-handler [57612] static: This library was built with Windows 8.1 or later, probing OS to see what we're actually running on.
[DEBUG] tls-handler [57612] static: We're running on Windows 8.1 or later. ALPN is available.
[DEBUG] tls-handler [57612] id=00958F40: Setting ALPN data as h2;http/1.1
[DEBUG] tls-handler [57612] 
[DEBUG] tls-handler [57612] Setting ALPN extension with string h2;http/1.1.
[DEBUG] tls-handler [57612] static: This library was built with Windows 8.1 or later, probing OS to see what we're actually running on.
[DEBUG] tls-handler [57612] static: We're running on Windows 8.1 or later. ALPN is available.
[DEBUG] tls-handler [57612] id=00958F40: Negotiated protocol h2
[DEBUG] tls-handler [57612] id=00958F40: TLS handshake completed successfully.
[DEBUG] channel-bootstrap [57612] id=008CC918: tls negotiation result 0 on channel 008ECE00
[INFO] http-connection [57612] id=0096DE80: HTTP/2 client connection established.
[DEBUG] connection-manager [57612] id=0091B5A8: Received new connection (id=0096DE80) from http layer
[DEBUG] connection-manager [57612] id=0091B5A8: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=1, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1
[DEBUG] connection-manager [57612] id=0091B5A8: HTTP/2 connection (id=0096DE80) completed initial settings
[DEBUG] connection-manager [57612] id=0091B5A8: Grabbing pooled connection (0096DE80)
[DEBUG] connection-manager [57612] id=0091B5A8: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1
[DEBUG] connection-manager [57612] id=0091B5A8: Successfully completed connection acquisition with connection id=0096DE80
[DEBUG] CRTHttpClient [57612] Obtained connection handle 00931370
[DEBUG] http-stream [57612] id=0 connection=0096DE80 state=IDLE: Created HTTP/2 request stream

USER: what's the next day after october thirty first
ASST: The next day after October 31st is November 1st, which is All Saints' Day!

StopStreaming(100 ms): inputAudioBytes:290816, outputAudioBytes:172800, outputEvents:25, usageEvents:23
SonicEngine: inputEventCount: 574, outputEventCount: 293

and inbound stream of events from Bedrock follows...

while on EC2 instance, virtually identical setup:

[DEBUG] socket [3160] id=007CC460 handle=000003D0: initializing with domain 0 and type 0
[DEBUG] socket [3160] id=007CC460 handle=000003D0: setting socket options to: keep-alive 1, keep idle 0, keep-alive interval 30, max failed probe count 0
[DEBUG] socket [3160] id=007CC460 handle=000003D0: beginning connect.
[DEBUG] socket [3160] id=007CC460 handle=000003D0: connecting to endpoint 13.216.100.148:443.
[DEBUG] socket [3160] id=007CC460 handle=000003D0: setting socket options to: keep-alive 1, keep idle 0, keep-alive interval 30, max failed probe count 0
[DEBUG] socket [3160] id=007CC460 handle=000003D0: connection success
[DEBUG] socket [3160] id=007CC460 handle=000003D0: local endpoint 10.174.16.164:55734
[DEBUG] channel-bootstrap [3160] id=0072A2E8: client connection on socket 007CC460 completed with error 0 : AWS_ERROR_SUCCESS
[DEBUG] channel [3160] id=007CC040: Beginning creation and setup of new channel.
[DEBUG] channel [3160] id=007CC040: setup complete, notifying caller.
[DEBUG] channel [3160] id=007CC040: message pool 0072F0C8 found in event-loop local storage: using it.
[DEBUG] channel-bootstrap [3160] id=0072A2E8: channel 007CC040 setup succeeded: bootstrapping.
[DEBUG] socket-handler [3160] id=006E9238: Socket handler created with max_read_size of 16384
[DEBUG] tls-handler [3160] Checking Windows Version: running windows build 20348 or later
[DEBUG] tls-handler [3160] id=036D2458: Setting SNI to bedrock-runtime.us-east-1.amazonaws.com
[DEBUG] tls-handler [3160] id=036D2458: Kicking off TLS negotiation
[DEBUG] tls-handler [3160] static: This library was built with Windows 8.1 or later, probing OS to see what we're actually running on.
[DEBUG] tls-handler [3160] static: We're running on Windows 8.1 or later. ALPN is available.
[DEBUG] tls-handler [3160] id=036D2458: Setting ALPN data as h2;http/1.1
[DEBUG] tls-handler [3160] 
[DEBUG] tls-handler [3160] Setting ALPN extension with string h2;http/1.1.
[DEBUG] tls-handler [3160] static: This library was built with Windows 8.1 or later, probing OS to see what we're actually running on.
[DEBUG] tls-handler [3160] static: We're running on Windows 8.1 or later. ALPN is available.
[DEBUG] tls-handler [3160] id=036D2458: Negotiated protocol h2
[DEBUG] tls-handler [3160] id=036D2458: TLS handshake completed successfully.
[DEBUG] channel-bootstrap [3160] id=0072A2E8: tls negotiation result 0 on channel 007CC040
[INFO] http-connection [3160] id=036E7870: HTTP/2 client connection established.
[DEBUG] connection-manager [3160] id=007BFDC8: Received new connection (id=036E7870) from http layer
[DEBUG] connection-manager [3160] id=007BFDC8: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=1, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1
[DEBUG] connection-manager [3160] id=007BFDC8: HTTP/2 connection (id=036E7870) completed initial settings
[DEBUG] connection-manager [3160] id=007BFDC8: Grabbing pooled connection (036E7870)
[DEBUG] connection-manager [3160] id=007BFDC8: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1
[DEBUG] connection-manager [3160] id=007BFDC8: Successfully completed connection acquisition with connection id=036E7870
[DEBUG] CRTHttpClient [3160] Obtained connection handle 036E41C0
[DEBUG] http-stream [3160] id=0 connection=036E7870 state=IDLE: Created HTTP/2 request stream

StopStreaming(100 ms): inputAudioBytes:290816, outputAudioBytes:0, outputEvents:0, usageEvents:0
SonicEngine: inputEventCount: 573, outputEventCount: 0

... and no inbound (model output events) stream

So I wonder - what am I missing here?