Skip to content

Always compress responses when the client compression is on #1286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 8, 2024

Conversation

zhkvia
Copy link
Contributor

@zhkvia zhkvia commented Apr 27, 2024

Summary

  • It resolves An obscure error in the HTTP protocol when using compression #1150 again, since the fix was broken at some time (body after return is empty since it hasn't been read yet)
  • The previous fix covered only the 403 code, but this one covers all non-200
  • As a bonus, with this fix the error doesn't state that it failed to read the response (which is technically not true since a response is successfully read)
    Now it looks like:
clickhouse [execute]:: 403 code: Code: 516. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name.
<and the rest>

@CLAassistant
Copy link

CLAassistant commented Apr 27, 2024

CLA assistant check
All committers have signed the CLA.

@zhkvia zhkvia force-pushed the dont-decompress-a-non-200-response branch from 7a1f322 to a1b3d5d Compare May 2, 2024 22:50
@zhkvia
Copy link
Contributor Author

zhkvia commented May 2, 2024

Actually after applying my fix and running it for some time in production I noticed that errors returned from a failed DESCRIBE don't get decompressed:

clickhouse [execute]:: 404 code: ~k���"x�
�h��`�������Code: 60. DB::Exception: Table logs.���_shard does not exist. Maybe you meant0�0_trace?. (UNKNOWN_TABLE) (version 23.8.12.13 (official build))

So I looked at the source code more closely and noticed the difference between DESCRIBE and INSERT queries:

  1. responses from DESCRIBE queries are always compressed:
    1. https://github.com/ClickHouse/clickhouse-go/blob/main/conn_http_batch.go#L54
    2. https://github.com/ClickHouse/clickhouse-go/blob/main/conn_http_query.go#L39
  2. while responses from INSERT queries are not:
    1. https://github.com/ClickHouse/clickhouse-go/blob/main/conn_http_batch.go#L52
    2. https://github.com/ClickHouse/clickhouse-go/blob/main/conn_http_batch.go#L108
    3. https://github.com/ClickHouse/clickhouse-go/blob/main/conn_http_batch.go#L201

But responses are always decompressed no matter what a query it was: https://github.com/ClickHouse/clickhouse-go/blob/main/conn_http.go#L432

So in order to fix this, I explicitly set compress option for INSERT queries as well


More than that, I noticed that in older versions of the library EOF was treated explicitly because it was wrapped with other errors several times in a row (e.g. read next block: header: EOF):
https://github.com/ClickHouse/clickhouse-go/blob/v2.14.0/conn_http.go#L453

But now this error is not checked and ReadAll can't handle it by itself since it doesn't look inside wrapped errors:
https://github.com/ClickHouse/clickhouse-go/blob/main/conn_http.go#L436

So I fixed this too

@zhkvia zhkvia changed the title Dont decompress a non-200 response Always compress responses when the client compression is on May 2, 2024
@mshustov mshustov requested a review from jkaflik May 4, 2024 06:52
@zhkvia
Copy link
Contributor Author

zhkvia commented May 7, 2024

run-tests / single-node (1.21, 24.1) (pull_request) Failing after 1m

Well, I couldn't reproduce the same results locally:

 λ  CLICKHOUSE_VERSION=24.1 go test .
ok      github.com/ClickHouse/clickhouse-go/v2  0.392s
 λ  cd tests/issues
 λ  CLICKHOUSE_VERSION=24.1 go test .
ok      github.com/ClickHouse/clickhouse-go/v2/tests/issues     0.565s
 λ  go version
go version go1.21.9 darwin/amd64

Also

 λ  CLICKHOUSE_VERSION=24.1 make test

runs fine

@jkaflik jkaflik merged commit c3f7a77 into ClickHouse:main May 8, 2024
codeboten referenced this pull request in open-telemetry/opentelemetry-collector-contrib Jul 10, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[github.com/ClickHouse/clickhouse-go/v2](https://togithub.com/ClickHouse/clickhouse-go)
| `v2.23.0` -> `v2.26.0` |
[![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fClickHouse%2fclickhouse-go%2fv2/v2.26.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fClickHouse%2fclickhouse-go%2fv2/v2.26.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fClickHouse%2fclickhouse-go%2fv2/v2.23.0/v2.26.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fClickHouse%2fclickhouse-go%2fv2/v2.23.0/v2.26.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.

---

### Release Notes

<details>
<summary>ClickHouse/clickhouse-go
(github.com/ClickHouse/clickhouse-go/v2)</summary>

###
[`v2.26.0`](https://togithub.com/ClickHouse/clickhouse-go/blob/HEAD/CHANGELOG.md#v2260-2024-06-25----Release-notes-generated-using-configuration-in-githubreleaseyml-at-main---)

[Compare
Source](https://togithub.com/ClickHouse/clickhouse-go/compare/v2.25.0...v2.26.0)

#### What's Changed

##### Enhancements 🎉

- Reintroduce the random connection strategy by
[@&#8203;larry-cdn77](https://togithub.com/larry-cdn77) in
[https://github.com/ClickHouse/clickhouse-go/pull/1313](https://togithub.com/ClickHouse/clickhouse-go/pull/1313)
- Make custom debug log function on-par with the built-in one by
[@&#8203;vespian](https://togithub.com/vespian) in
[https://github.com/ClickHouse/clickhouse-go/pull/1317](https://togithub.com/ClickHouse/clickhouse-go/pull/1317)
- Remove date overflow check as it's normalised by ClickHouse server by
[@&#8203;gogingersnap777](https://togithub.com/gogingersnap777) in
[https://github.com/ClickHouse/clickhouse-go/pull/1315](https://togithub.com/ClickHouse/clickhouse-go/pull/1315)
- Batch: impl `Columns() []column.Interface` method by
[@&#8203;egsam98](https://togithub.com/egsam98) in
[https://github.com/ClickHouse/clickhouse-go/pull/1277](https://togithub.com/ClickHouse/clickhouse-go/pull/1277)

##### Fixes 🐛

- Fix rows.Close do not return too early by
[@&#8203;yujiarista](https://togithub.com/yujiarista) in
[https://github.com/ClickHouse/clickhouse-go/pull/1314](https://togithub.com/ClickHouse/clickhouse-go/pull/1314)
- Setting `X-Clickhouse-SSL-Certificate-Auth` header correctly given
`X-ClickHouse-Key` by
[@&#8203;gogingersnap777](https://togithub.com/gogingersnap777) in
[https://github.com/ClickHouse/clickhouse-go/pull/1316](https://togithub.com/ClickHouse/clickhouse-go/pull/1316)
- Retry on network errors and fix retries on async inserts with
`database/sql` interface by
[@&#8203;tommyzli](https://togithub.com/tommyzli) in
[https://github.com/ClickHouse/clickhouse-go/pull/1330](https://togithub.com/ClickHouse/clickhouse-go/pull/1330)
- BatchInsert parentheses issue fix by
[@&#8203;ramzes642](https://togithub.com/ramzes642) in
[https://github.com/ClickHouse/clickhouse-go/pull/1327](https://togithub.com/ClickHouse/clickhouse-go/pull/1327)

##### Other Changes 🛠

- ClickHouse 24.5 by [@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1319](https://togithub.com/ClickHouse/clickhouse-go/pull/1319)
- Align `allow_suspicious_low_cardinality_types` and `
allow_suspicious_low_cardinality_types ` settings in tests due to
ClickHouse Cloud incompatibility by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1331](https://togithub.com/ClickHouse/clickhouse-go/pull/1331)
- Use HTTPs scheme in std connection failover tests by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1332](https://togithub.com/ClickHouse/clickhouse-go/pull/1332)

#### New Contributors

- [@&#8203;larry-cdn77](https://togithub.com/larry-cdn77) made their
first contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1313](https://togithub.com/ClickHouse/clickhouse-go/pull/1313)
- [@&#8203;vespian](https://togithub.com/vespian) made their first
contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1317](https://togithub.com/ClickHouse/clickhouse-go/pull/1317)
- [@&#8203;gogingersnap777](https://togithub.com/gogingersnap777) made
their first contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1315](https://togithub.com/ClickHouse/clickhouse-go/pull/1315)
- [@&#8203;yujiarista](https://togithub.com/yujiarista) made their first
contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1314](https://togithub.com/ClickHouse/clickhouse-go/pull/1314)
- [@&#8203;egsam98](https://togithub.com/egsam98) made their first
contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1277](https://togithub.com/ClickHouse/clickhouse-go/pull/1277)
- [@&#8203;tommyzli](https://togithub.com/tommyzli) made their first
contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1330](https://togithub.com/ClickHouse/clickhouse-go/pull/1330)
- [@&#8203;ramzes642](https://togithub.com/ramzes642) made their first
contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1327](https://togithub.com/ClickHouse/clickhouse-go/pull/1327)

**Full Changelog**:
ClickHouse/clickhouse-go@v2.25.0...v2.26.0

###
[`v2.25.0`](https://togithub.com/ClickHouse/clickhouse-go/blob/HEAD/CHANGELOG.md#v2250-2024-05-28----Release-notes-generated-using-configuration-in-githubreleaseyml-at-main---)

[Compare
Source](https://togithub.com/ClickHouse/clickhouse-go/compare/v2.24.0...v2.25.0)

#### What's Changed

##### Fixes 🐛

- Add a compatibility layer for a database/sql driver to work with
sql.NullString and ClickHouse nullable column by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1306](https://togithub.com/ClickHouse/clickhouse-go/pull/1306)

##### Other Changes 🛠

- Use Go 1.22 in head tests by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1305](https://togithub.com/ClickHouse/clickhouse-go/pull/1305)
- Skip flaky 1127 test by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1307](https://togithub.com/ClickHouse/clickhouse-go/pull/1307)

**Full Changelog**:
ClickHouse/clickhouse-go@v2.24.0...v2.25.0

###
[`v2.24.0`](https://togithub.com/ClickHouse/clickhouse-go/blob/HEAD/CHANGELOG.md#v2240-2024-05-08----Release-notes-generated-using-configuration-in-githubreleaseyml-at-main---)

[Compare
Source](https://togithub.com/ClickHouse/clickhouse-go/compare/v2.23.2...v2.24.0)

#### What's Changed

##### Enhancements 🎉

- Always compress responses when the client compression is on by
[@&#8203;zhkvia](https://togithub.com/zhkvia) in
[https://github.com/ClickHouse/clickhouse-go/pull/1286](https://togithub.com/ClickHouse/clickhouse-go/pull/1286)
- Optional flag to close query with flush by
[@&#8203;hongker](https://togithub.com/hongker) in
[https://github.com/ClickHouse/clickhouse-go/pull/1276](https://togithub.com/ClickHouse/clickhouse-go/pull/1276)

##### Fixes 🐛

- Fix prepare batch does not break on `values` substring in table name
by [@&#8203;Wang](https://togithub.com/Wang) in
[https://github.com/ClickHouse/clickhouse-go/pull/1290](https://togithub.com/ClickHouse/clickhouse-go/pull/1290)
- Fix nil checks when appending slices of pointers by
[@&#8203;markandrus](https://togithub.com/markandrus) in
[https://github.com/ClickHouse/clickhouse-go/pull/1283](https://togithub.com/ClickHouse/clickhouse-go/pull/1283)

##### Other Changes 🛠

- Don't recreate keys from LC columns from direct stream by
[@&#8203;genzgd](https://togithub.com/genzgd) in
[https://github.com/ClickHouse/clickhouse-go/pull/1291](https://togithub.com/ClickHouse/clickhouse-go/pull/1291)

#### New Contributors

- [@&#8203;zhkvia](https://togithub.com/zhkvia) made their first
contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1286](https://togithub.com/ClickHouse/clickhouse-go/pull/1286)

**Full Changelog**:
ClickHouse/clickhouse-go@v2.23.2...v2.24.0

###
[`v2.23.2`](https://togithub.com/ClickHouse/clickhouse-go/blob/HEAD/CHANGELOG.md#v2232-2024-04-25----Release-notes-generated-using-configuration-in-githubreleaseyml-at-main---)

[Compare
Source](https://togithub.com/ClickHouse/clickhouse-go/compare/v2.23.1...v2.23.2)

#### What's Changed

##### Fixes 🐛

- Fixed panic on concurrent context key map write by
[@&#8203;Wang](https://togithub.com/Wang) in
[https://github.com/ClickHouse/clickhouse-go/pull/1284](https://togithub.com/ClickHouse/clickhouse-go/pull/1284)

##### Other Changes 🛠

- Fix ClickHouse Terraform provider version by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1285](https://togithub.com/ClickHouse/clickhouse-go/pull/1285)

#### New Contributors

- [@&#8203;Wang](https://togithub.com/Wang) made their first
contribution in
[https://github.com/ClickHouse/clickhouse-go/pull/1284](https://togithub.com/ClickHouse/clickhouse-go/pull/1284)

**Full Changelog**:
ClickHouse/clickhouse-go@v2.23.1...v2.23.2

###
[`v2.23.1`](https://togithub.com/ClickHouse/clickhouse-go/blob/HEAD/CHANGELOG.md#v2231-2024-04-15----Release-notes-generated-using-configuration-in-githubreleaseyml-at-main---)

[Compare
Source](https://togithub.com/ClickHouse/clickhouse-go/compare/v2.23.0...v2.23.1)

#### What's Changed

##### Fixes 🐛

- Zero-value timestamp to be formatted as toDateTime(0) in bind by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1260](https://togithub.com/ClickHouse/clickhouse-go/pull/1260)

##### Other Changes 🛠

- Update
[#&#8203;1127](https://togithub.com/ClickHouse/clickhouse-go/issues/1127)
test case to reproduce a progress handle when exception is thrown by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1259](https://togithub.com/ClickHouse/clickhouse-go/pull/1259)
- Set max parallel for GH jobs by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1261](https://togithub.com/ClickHouse/clickhouse-go/pull/1261)
- Ensure test container termination by
[@&#8203;jkaflik](https://togithub.com/jkaflik) in
[https://github.com/ClickHouse/clickhouse-go/pull/1274](https://togithub.com/ClickHouse/clickhouse-go/pull/1274)

**Full Changelog**:
ClickHouse/clickhouse-go@v2.23.0...v2.23.1

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MTMuMiIsInVwZGF0ZWRJblZlciI6IjM3LjQyNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZWJvdCJdfQ==-->

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <[email protected]>
Co-authored-by: Yang Song <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

An obscure error in the HTTP protocol when using compression
3 participants