StarRocks Roadmap 2024

[Dshadowzh]

Refer to roadmap 2023 2022

Shared-data & StarOS

• Align with all functionalities to shared-nothing
  • Sync materialized view
  • Generated column
  • Partial update with column mode
  • Optimize table and manual compaction
• Better cache system
  • Multi-layer cache
  • Global cache
  • Cache warmup Cache warmup
  • Cache blacklist/whitelist
  • Refine evict algorithm
• StarOS internal optimization
  • Multi-replicas for shard management
  • Shard schedule optimization for large scale (more than 10M shards)
  • Local storage for StarOS
• Decoupled storage for FE (Finished design)
• Open API for StarRocks table format (sink and source)
• Time Travel
• Backup support Snapshot for shared-data #53999

Performance

• Full columnar Json index Flat json
• Cost model with primary key and foreign key constrains
• Arm optimization for codecs
• Adaptive DOP and adaptive query engine
• Global dictionary encoding
• Enhance IO schedule framework
• JIT / Codegen
• Fine granularity Fe lock(from db level to table level)

Easy to use

• Online optimize table
• List partition optimization
• Arrow flight interface Support for apache arrow flight SQL #22944
• Improve files table function
  • Improve schema inference
  • CSV and json format support
  • Other format: Avro, Arrow, Protobuf
  • Better performance for read, predicates pushdown
• Insert statement improvement (on duplicate key, insert properties)
• Unified data ingestion with Pipe
  • Pipe for continuous ingestion from Kafka
  • Read from external stream table(Kafka)
  • Continues data ingestion from SQS with Pipe
• Out-of-the-box parameters

Data lake analytics

• Better file format support
  • Parquet reader tuning
  • ORC reader tuning
• Better table format support

Lake	Query	Insert	DDL	Update/Delete/Merge into	MV
Hive	1.18	3.2			2.5
Iceberg	2.1	3.1			3.0
Hudi	2.2				3.0
Paimon	3.0				3.2
Delta lake	3.0				3.2

• Iceberg metadata optimization (Iceberg metadata super optimization #43460 3.3)
• Materialized view improvement
  • Improve partition mapping (list partition, expression partition) (3.4 unified all partition method)
  • Task scheduler DAG & Lineage
  • Better query rewrite
• JDBC catalog improvement
• Enhance JNI reader and implement JNI writer
• Text File format support (basic csv format 3.3)
• Presto/Trino/Spark/Hive SQL compatibility
• Presto/Trino/Spark/Hive UDF compatibility
• Automatic cooldown to lake format

Data warehousing(batch and streaming)

Batch processing & ETL improvement

• Enable spilling to GA (3.3)
• Multi-statement transaction Support Transaction Statement #53978 (3.5)
• Temporary table (3.3)
• Group execution [Feature] support Group execution  #42352 (3.3)
• Task auto retry

Streaming processing & real-time update

• Schemaless partial update
• Merge into statement
• Binlog to flink and spark streaming
• Transaction level incremental refresh in materialized view (Aggregation, Join, functions)
• Incremental refresh for iceberg/Hudi/Paimon materialized view

All-in-one scenarios

• Search: Optimize full text inverted index [inverted_index] )(https://docs.starrocks.io/docs/table_design/indexes/inverted_index/)
• Row store: Optimize row store for high concurrent point lookup Hybrid row-column store
• Time series db: Asof join, high concurrent ingestion
• Vector database: vector index [Feature] Support vector index and ANNS. #46678 (3.4)

Release

• Release plan 3.3 #40907
• Release plan 3.4 #52573

[arsenalzp]

Hello,
Any chance to have good-first-issue feature among those tasks?

[Dshadowzh]

Hello, Any chance to have good-first-issue feature among those tasks?

Welcome, you can check this #13300 first, we'll update more good-first-issues in 2024 later. Particularly regarding external catalog and connectors.

[Zhangg7723]

How about incremental refresh materialized view for external table like Iceberg or Hudi? I think this feature can reduce the cost of refresh mv

[Dshadowzh]

How about incremental refresh materialized view for external table like Iceberg or Hudi? I think this feature can reduce the cost of refresh mv

Yes. We are considering about it, there is a Incremental refresh for iceberg/Hudi/Paimon materialized view. By the way, Iceberg and Hudi, which do you perfer?

[Zhangg7723]

How about incremental refresh materialized view for external table like Iceberg or Hudi? I think this feature can reduce the cost of refresh mv

Yes. We are considering about it, there is a Incremental refresh for iceberg/Hudi/Paimon materialized view. By the way, Iceberg and Hudi, which do you perfer?

We prefer Iceberg, for better interface design and less bugs. incremental snapshot refresh is useful for non-partition table.

[MatthewH00]

On yesterday 2024 roadmap meeting，mention that will support tag on BE in shared nothing mode，it like multi warehouse mechanism like in shared data mode，could split into load data warehouse\adhoc query warehouse\ETL warehouse...? And when will release?

[Dshadowzh]

On yesterday 2024 roadmap meeting，mention that will support tag on BE in shared nothing mode，it like multi warehouse mechanism like in shared data mode，could split into load data warehouse\adhoc query warehouse\ETL warehouse...? And when will release?

#38833 It has finished already, will be published in the next version.

[trikker]

On yesterday 2024 roadmap meeting，mention that will support tag on BE in shared nothing mode，it like multi warehouse mechanism like in shared data mode，could split into load data warehouse\adhoc query warehouse\ETL warehouse...? And when will release?

#38833 It has finished already, will be published in the next version.

I think there are different issues. Multi-warehouse enables different users to see different machines so as to get resource isolation at machine level. #38833 is about the replica location. In share-nothing deployment the data is all on HDFS/S3, we don't have replicas but we still need multi-warehouse cabability to isolate different machines to different resource group.

[trikker]

Thanks, the following are our wanted features and improvements based on my tests on StarRocks and my company's business scenarios:

1. Datalake Query
   (1) support more types of catalogs, like oracle, tidb, oceanbase and greenplum(don't know if it is fully compatible with postgresql)
   (2) support catalog metadata cache for JDBC MySQL, JDBC postgresql and the above catalogs
   (3) support automatic sample and histogram statistic collection for Hive, Iceberg

2. Materialized View
   (1) support rewrite for SQLs with UNION, ORDER BY and LIMIT
   (2) support rewrite for nested-aggregation SQLs, aggregation-then-join SQLs and related-subquery SQLs with one MV
   (3) enable view based mv rewrite to rewrite a query without needing user to query the view
   (4) support incremental refresh of MV(like flink stream computing), rather than refresh a whole table or partition
   (5) support materiailized view recommendation

3. Query Plan
   (1) support query plan cache and plan binding for SQLs, like Oracle

4. Resource Management
   (1) support CPU real hard limit for resource group, currently it is actually a soft limit;
   (2) suport multi-warehouse for users or resource group, different users can only see only different part of the machines when executing queries

5. Stability
   (1) memory spill still don't work for some SQLs when enable_spill is true and spill_mode is force, see issue: TPC-DS query04 still reports "Mem usage has exceed the limit of single query" when query_mem_limit is 5GB and enable_spill is true and spill_mode is force #40936

[chengyi3192]

Expected to support deletion queries in the Parquet format of Iceberg

[Dshadowzh]

Expected to support deletion queries in the Parquet format of Iceberg

We plan implementing it in v3.3

[Dshadowzh]

Thanks, the following are our wanted features and improvements based on my tests on StarRocks and my company's business scenarios:

Thanks a lot for the extensive feedback:

1. We want to improve JDBC catalog for more databases. It's a community driven project, It's welcome if you want to participate and we'll list some detail about this project later.
2. New statistics collection framework for hive and iceberg is WIP
3. View-based mv rewrite is already finished(v3.3).You can create some independent issue for UNION, ORDER BY, LIMIT and nested-aggregation SQLs rewrite. We have some work arounds for these cases, if you have some specific scenarios, we'd like to prioritize these issues.
4. Query plan cache is a good idea, if you have some basic design, we'd like to discuss in details.
5. Hard limit for resource group and multi-warehouse are in our roadmap.

[trikker]

Thanks, the following are our wanted features and improvements based on my tests on StarRocks and my company's business scenarios:

Thanks a lot for the extensive feedback:

1. We want to improve JDBC catalog for more databases. It's a community driven project, It's welcome if you want to participate and we'll list some detail about this project later.
2. New statistics collection framework for hive and iceberg is WIP
3. View-based mv rewrite is already finished(v3.3).You can create some independent issue for UNION, ORDER BY, LIMIT and nested-aggregation SQLs rewrite. We have some work arounds for these cases, if you have some specific scenarios, we'd like to prioritize these issues.
4. Query plan cache is a good idea, if you have some basic design, we'd like to discuss in details.
5. Hard limit for resource group and multi-warehouse are in our roadmap.

OK, hope we have a detailed communication later. Happy Spring Festival! ^_^