ClickHouse can produce blocks bigger than 1 mil rows.

[UnamedRus]

ClickHouse can produce blocks bigger than 1 mil rows, in case you have GROUP BY in your query.

clickhouse-go/lib/proto/block.go

Line 144 in e9d1875

Err: errors.New("more then 1 000 000 rows in block"),

[alexey-milovidov]

ClickHouse itself has 64 bit block size, it allows blocks with more than 4 billion records, although it is a rare case.
Every client library should not have any artificial limitations on block size.

[UnamedRus]

But imo, it make sense to limit possible max_block_size outside of ClickHouse, because it will allow to lower max memory usage for client library when it will parse columnar block.

[variantf]

@alexey-milovidov @UnamedRus Same here! Is there any workaround for now? Looks like the connection will broken after server returning block large than 1 million rows.

My query doesn't have groub by clause but have order by

[alexey-milovidov]

The library maintainer stopped to respond. No workaround for now, we are going to address it.

[variantf]

I vendor-ed this project and just remove the lines for errors after 1m rows. it works fine for me now.

[gingerwizard]

I'm considering moving to this to a configuration value (possibly with the original default) but allow users to override on either client or at query time. Any other suggestions @variantf @UnamedRus @alexey-milovidov

[UnamedRus]

I'm curious why this limit was introduced at first?

[gingerwizard]

Presumably to limit memory consumption but the author is not contactable to ask.

[alexey-milovidov]

Ok from my side, but we can also set it by default to some large value (like 1 billion) if it is only a protection and does not affect memory consumption in regular cases.

[alexey-milovidov]

And if we do, don't need to add any configuration (no one will use it).

[variantf]

I prefer to set some client side settings to force a small chunk size, but I have no idea if these kind of settings exists? @alexey-milovidov any ideas?

I don't think we will have this large-chunk-detection anywhere. since if we have ever encounter this error, we have nothing to do with. We will never do a success query especially when the client doesn't have permission to modify server configurations.

[gingerwizard]

@variantf you can set max_block_size e.g.

ctx := clickhouse.Context(context.Background(), clickhouse.WithSettings(clickhouse.Settings{
"max_block_size": 10000,
}),

While this will force a smaller block size on most queries, there are circumstances where it cannot be enforced - resulting in large blocks. Think of it as a largely obeyed hint. that we plan to adhere to more strictly in the future.

I will increase this to 1b - 1m seems a little low.