Anonymous View
Skip to content

feat(engine): Add StarRocks engine support#5832

Merged
StuffbyYuki merged 20 commits into
SQLMesh:mainfrom
jukiewiczm-marketer:feat/starrocks_support
Jun 9, 2026
Merged

feat(engine): Add StarRocks engine support#5832
StuffbyYuki merged 20 commits into
SQLMesh:mainfrom
jukiewiczm-marketer:feat/starrocks_support

Conversation

@jukiewiczm-marketer

@jukiewiczm-marketer jukiewiczm-marketer commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Description

This PR is based on a great work of @jaogoy and his original PR with StarRocks support implementation. Given his permission, I took over the implementation and added the following:

  • updated the starrocks in tests from 3.5 to 4.1
  • fixed all the merge conflicts
  • fixed all the unit tests (and added new ones)
  • addressed all the concerns that I raised in the original PR [1], [2], [3]
  • updated the documentation accordingly

Calling the sqlmesh maintainers that were called/commented on the original pr @erindru @StuffbyYuki

Test Plan

I tested it locally in various scenarios (especially related to AMVs) + there are plenty of unit tests for the engine.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

jaogoy and others added 20 commits June 8, 2026 12:31
@jaogoy @jukiewiczm-marketer
Feat: Add StarRocks engine support
14efc90 
- **Add StarRocks engine support to SQLMesh** via StarRocks’
MySQL-compatible protocol.
- Ship **engine adapter + docs + real integration tests** to ensure
generated SQL works on StarRocks.

- **User demand / adoption**: StarRocks is a common OLAP choice; SQLMesh
users want to run the same model lifecycle (build, incremental
maintenance, views/MVs) on StarRocks without bespoke SQL.
- **Engine-specific semantics**: StarRocks differs from vanilla MySQL in
DDL/DML constraints (e.g., key types, delete behavior, rename caveats).
An adapter is needed to produce correct and predictable SQL.
- **Confidence & maintainability**: Documenting config patterns +
codifying behavior with integration tests prevents regressions and makes
support “real” (not just “it parses”).

- **Connectivity**: Connect through MySQL protocol (e.g., `pymysql`).
- **Table creation / DDL**:
- Key table types via `physical_properties`: **DUPLICATE KEY
(default)**, **PRIMARY KEY (recommended for incremental)**, **UNIQUE
KEY**
- **Partitioning**: simple `partitioned_by` and advanced
`partition_by` (complex expression partitioning) + optional initial
`partitions`
- **Distribution**: `distributed_by` structured form or string
fallback (HASH / RANDOM; buckets required)
  - **Ordering**: `order_by` / `clustered_by`
  - **Generic PROPERTIES passthrough** (string key/value)
- **Views**:
  - Regular views
- **Materialized views** via `kind VIEW(materialized true)` with
StarRocks-specific notes/constraints
- **DML / maintenance**:
  - Insert/select/update basics
- Delete behavior handled with StarRocks compatibility constraints
(PRIMARY KEY tables recommended for robust deletes)

- **Engine adapter**: `sqlmesh/core/engine_adapter/starrocks.py`
- **Docs**: `docs/integrations/engines/starrocks.md`
- **Integration tests**:
`tests/core/engine_adapter/integration/test_integration_starrocks.py`,
and `tests/core/engine_adapter/test_starrocks.py`

- **Integration tests require a running StarRocks** instance.
- Ran:
  - set `STARROCKS_HOST/PORT/USER/PASSWORD`
- `pytest -m "starrocks and docker"
tests/core/engine_adapter/integration/test_integration_starrocks.py`

- **No sync MV support (currently)**
- **No tuple IN**: `(c1, c2) IN ((v1, v2), ...)`
- **No `SELECT ... FOR UPDATE`**
- **RENAME caveat**: rename target can’t be qualified with a database
name

- **Changes are StarRocks-scoped** (adapter/docs/tests) and should not
impact other engines.

Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
refine code style
17a8bcf 
And optimize some test cases.

Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
optimize doc-test
2e22e87 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
remove some debug log
5aa926a 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
optimize doc
3c85ada 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
starrocks: add some test cases for partition
f3758f4 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
optimize docs for View and MV
bf8af5b 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
optimize code style
8fb9246 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
fix docker compose usage
41474ca 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
modity test cases according to the sqlglot change
79924fa 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
modify code according some small change of sqlglot
63135e7 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
modify test about materialized_properteis
a57f721 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jaogoy @jukiewiczm-marketer
set default replication num to 1
42492d1 
Signed-off-by: jaogoy <jaogoy@gmail.com>
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jukiewiczm-marketer
add starrocks to the CI matrix in the GitHub Actions workflow
55b3bb5 
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jukiewiczm-marketer
Fix make style errors (mypy, ruff)
a5c258d 
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jukiewiczm-marketer
Update StarRocks Docker images to version 4.1 in the integration comp…
9dec6d9 
…ose file

Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jukiewiczm-marketer
* Refactoring
f258b5e 
* Making optional but failing requirements mandatory
* Fixing unit tests
* Documentation updates

Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jukiewiczm-marketer
Make StarRocks AMVs not recreate themselves during sqlmesh run
33ed712 
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jukiewiczm-marketer
Docs update + StarRocks AMV with audits gets refreshed immediately
10378c9 
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@jukiewiczm-marketer
Physical properties map to physical table names based on configuration
091a215 
Signed-off-by: Mateusz Jukiewicz <mateusz@marketer.tech>
@StuffbyYuki StuffbyYuki mentioned this pull request Jun 9, 2026
Closed
@StuffbyYuki StuffbyYuki merged commit ba4054a into SQLMesh:main Jun 9, 2026
32 checks passed
@StuffbyYuki

StuffbyYuki commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

@jukiewiczm-marketer thanks for this PR! I just merged this and since it's a new adapter, let's continue monitoring issues and improving it as things come up. Again, thanks for your work on this.

@StuffbyYuki StuffbyYuki mentioned this pull request Jun 9, 2026
Closed
@jukiewiczm-marketer

jukiewiczm-marketer commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@StuffbyYuki sounds great, thanks.
How does the SQLMesh release schedule work? I'd ideally use the official version rather than a commit-pinned dependency in my repo :)

@StuffbyYuki

StuffbyYuki commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

@jukiewiczm-marketer It looks like we just manually push it. I might suggest to the core team that we do this weekly or something going forward!

@StuffbyYuki

StuffbyYuki commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

@jukiewiczm-marketer this is available in the newst release. But for some reason the starrocks docs don't show up on the website. Let me know if you have any good idea, I'll continue figuring out why that might be

@jukiewiczm-marketer

jukiewiczm-marketer commented Jun 13, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the update @StuffbyYuki 🙏. I'll check it out tomorrow, I only edited the docs that were created in the original PR so never investigated if there are any issues with them showing up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jukiewiczm-marketer @StuffbyYuki @jaogoy