Options to migrate from Synapse for Spark Data Ingestion Jobs

Question

Options to migrate from Synapse for Spark Data Ingestion Jobs

Arjun Kedlaya 0 Microsoft Employee

Hi All,

My team is currently trying to migrate from Synapse for our Spark Data Ingestion Job where we ingest data from Kusto clusters via Service Principal and store since the recommendation is to migrate to newer solutions which are actively being developed.

Which is the best solution for this use case - Microsoft Fabric or Azure Machine Learning or any other solution?

I am seeing limitations with both solutions compared to Synapse, namely

With Fabric, I don't see support for connecting to Kusto clusters via Service Principal auth. Fabric Eventhouse does have support for Service Principal Auth but only via Client Key and Secret which is considered a security risk in our organization.
With Azure ML, I don't see any support for Kusto connections at all to ingest data

Is there any other service worth exploring other than the above two solutions?

Pilladi Padma Sai Manisha 7,710 Reputation points Microsoft External Staff Moderator

2026-04-30T12:09:24.92+00:00

Hi Arjun Kedlaya,

You’re absolutely right in your assessment,today neither Microsoft Fabric nor Azure Machine Learning fully supports your scenario of Spark-based ingestion from Azure Data Explorer using secure Service Principal patterns without relying on client secrets.

For your use case, the most practical and production-ready option right now is Azure Databricks. It provides native Spark support along with the Azure Data Explorer connector and supports Service Principal authentication, including more secure approaches such as Managed Identity or Key Vault-backed secrets. This makes it a closer match to your existing Synapse setup while aligning with enterprise security expectations.

You can refer to the official documentation here: https://learn.microsoft.com/azure/data-explorer/spark-connector https://learn.microsoft.com/azure/databricks/connect/data/azure-data-explorer https://learn.microsoft.com/azure/databricks/security/authentication/

If you prefer a more orchestrated approach, Azure Data Factory can be used to trigger Databricks jobs or, in cases where Spark is not strictly required, directly ingest from Azure Data Explorer using its native connector: https://learn.microsoft.com/azure/data-factory/connector-azure-data-explorer

In short, Azure Databricks is currently the best-fit replacement for Synapse in this scenario, while Fabric is still evolving and Azure ML is not designed for ingestion workloads
Arjun Kedlaya 0 Reputation points Microsoft Employee

2026-04-30T13:55:40.38+00:00

@Pilladi Padma Sai Manisha , even Azure Data Factory is deprecated by my organization which is why we decided to migrate from Synapse in the first place. Spark is a hard requirement for us. Is there any other solution?
Pilladi Padma Sai Manisha 7,710 Reputation points Microsoft External Staff Moderator

2026-05-01T11:21:30.4433333+00:00

Arjun Kedlaya Spark is mandatory and both Azure Synapse Analytics and Azure Data Factory are not options,there isn’t another Microsoft service today that fully supports Spark-based ingestion from Azure Data Explorer with secure auth patterns.

The most viable and supported option remains Azure Databricks. It provides native Spark, supports the Kusto connector, and allows secure authentication approaches like Managed Identity or Key Vault-backed secrets.

Microsoft Fabric is still evolving for this scenario, and Azure Machine Learning isn’t designed for ingestion workloads.

In short, Azure Databricks is currently the only practical replacement that meets your requirements.