This was announced by Databricks at their Data & AI summit.

It’s interesting to see these older data warehouses starting to implement features that newer projects have proven over the past few years.

This method of incremental materialisation for streaming ingestion is the default in ClickHouse, and there’s all sorts of so-called “streaming databases” that have popped up that exclusively do this e.g. Materialize, Rising Wave, etc.

I don’t know what Databricks’ implementation is like, I’m not a customer of theirs, so I’m interesting to see how successful it is. I have been using ClickHouse for some time now and it’s a super powerful feature to get away from batch & schedules.