Spark 3.5 - Cosmos OLAP read fails with "Session isn't active" while working in Spark 3.4

Question

Spark 3.5 - Cosmos OLAP read fails with "Session isn't active" while working in Spark 3.4

Victor Buzy 25

🔹 Description

We are currently migrating workloads from Azure Synapse Spark 3.4 to Spark 3.5.

We are experiencing an issue when reading data from Azure Cosmos DB using Synapse Link (cosmos.olap).

The exact same code works perfectly in Spark 3.4, but consistently fails in Spark 3.5.

🔹 Minimal Repro Code

df = spark.read \

🔹 Observed Behavior

Spark session starts successfully:
- Apache Spark session started... Running
- Immediately after executing the first action:

InvalidHttpRequest:

🔹 Expected Behavior

The notebook should successfully read data from Cosmos DB as it does in Spark 3.4.
df.limit(1).show() should return data.

🔹 Additional Context

The issue occurs even with minimal code (no transformations).
The issue seems specific to cosmos.olap (Synapse Link).
SQL-based notebooks work correctly in Spark 3.5.
The same Cosmos notebook works without any issue in Spark 3.4.
Spark session appears to start but becomes inactive immediately afterward.
🔹 Description
We are currently migrating workloads from Azure Synapse Spark 3.4 to Spark 3.5. We are experiencing an issue when reading data from Azure Cosmos DB using Synapse Link (cosmos.olap). The exact same code works perfectly in Spark 3.4, but consistently fails in Spark 3.5.
🔹 Minimal Repro Code
```
  df
```
🔹 Observed Behavior
- Spark session starts successfully:
- Apache Spark session started... Running
Immediately after executing the first action:

  InvalidHttpRequest:

## 🔹 Expected Behavior
```  -  The notebook should successfully read data from Cosmos DB as it does in Spark 3.4. 

- `df.limit(1).show()` should return data. 

    ## 🔹 Additional Context

  -  The issue occurs even with minimal code (no transformations). 
  
  -  The issue seems specific to `cosmos.olap` (Synapse Link). 
  
  -  SQL-based notebooks work correctly in Spark 3.5. 
  
  -  The same Cosmos notebook works without any issue in Spark 3.4. 
  
  -  Spark session appears to start but becomes inactive immediately afterward.
  
## 🔹 Error Trace ID


```dockerfile
e9c2162f-93e2-46a0-ac70-91bc47fde292

🔹 Environment Details

Spark Version: 3.5
Node Size: (ex: Medium 8 vCores)
Autoscale: *(ex: 1–10 nodes)
*See picture attached

🔹 Impact

This issue is blocking migration of Cosmos-based workloads to Spark 3.5.

To recap: To clarify the issue, here is the clean minimal repro and context.

Description:

We are migrating workloads from Azure Synapse Spark 3.4 to Spark 3.5.

We are experiencing an issue when reading data from Azure Cosmos DB using Synapse Link (cosmos.olap).

The exact same code works correctly in Spark 3.4, but consistently fails in Spark 3.5.

Minimal repro code:

df = spark.read \

.format("cosmos.olap") \

.option("spark.synapse.linkedService", "DATABASE") \

.option("spark.cosmos.container", "TABLE") \

.load()
```df.limit(1).show()

Observed behavior:

- Spark session starts successfully and shows as Running

- Immediately after executing the first action, the notebook fails with:

InvalidHttpRequest: Submission failed due to error content =["requirement failed: Session isn't active."]

HTTP status code: 400

Trace ID: e9c2162f-93e2-46a0-ac70-91bc47fde292

Expected behavior:

- The notebook should successfully read data from Cosmos DB in Spark 3.5, as it does in Spark 3.4

- df.limit(1).show() should return data

Additional context:

- The issue occurs even with minimal code and no transformations

- The issue seems specific to cosmos.olap (Synapse Link)

- SQL-based notebooks work correctly in Spark 3.5

- The same Cosmos notebook works without any issue in Spark 3.4

- Spark session appears to start, then becomes inactive immediately afterward

Environment:

- Spark version: 3.5

- Node size: Medium (8 vCores / 64 GB)

- Autoscale: Enabled (3 to 10 nodes)

- Dynamic allocation: Enabled

- Default executors: 1 to 4

- Python version: 3.11

- Delta Lake version: 3.2

Impact:

This issue is blocking migration of Cosmos-based workloads to Spark 3.5.

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Pilladi Padma Sai Manisha 7,390 Reputation points Microsoft External Staff Moderator

2026-04-01T20:11:24.5033333+00:00

Hi Victor Buzy,
Based on what you’ve described, this behavior appears to be related to a compatibility gap between Spark 3.5 and the Cosmos DB OLAP (cosmos.olap) connector used through Synapse Link. The key observation is that the same code executes successfully in Spark 3.4 but fails in Spark 3.5 with the “Session isn’t active” error as soon as an action is triggered. This typically indicates that the Spark session is being marked inactive during execution, which points to a runtime or connector-level issue rather than anything in your code or configuration.

Given this, the most reliable approach at the moment is to continue running Cosmos OLAP workloads on Spark 3.4, where the connector integration is stable and functioning as expected. This ensures continuity while avoiding the session instability observed in Spark 3.5.

For additional context on how Synapse Link integrates with Spark and Cosmos DB, you can refer to the following documentation:

https://learn.microsoft.com/azure/cosmos-db/synapse-link
https://learn.microsoft.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark https://learn.microsoft.com/azure/synapse-analytics/spark/apache-spark-version-support

Your implementation and repro steps look correct, and the behavior you’re seeing aligns with a version-specific limitation rather than a usage issue. Please feel free to share if you notice any changes or need help validating alternative approaches.
Pilladi Padma Sai Manisha 7,390 Reputation points Microsoft External Staff Moderator

2026-04-03T21:53:00.7766667+00:00

Hi Victor Buzy,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Victor Buzy 25 Reputation points

2026-04-06T13:14:38.18+00:00
Hi @Anonymous

Thank you for your response.

However, this situation is blocking our migration strategy.

On one hand, Spark 3.4 is reaching end of support and we are encouraged to migrate to Spark 3.5.

On the other hand, Cosmos OLAP workloads are currently not working on Spark 3.5 and you are advised to remain on Spark 3.4.

This creates a contradiction, as we cannot fully migrate our platform while Cosmos OLAP is not supported or stable in Spark 3.5.

Could you please clarify:

Is this a known limitation or a confirmed bug in Spark 3.5 with cosmos.olap?

Is there an official ETA or roadmap for a fix or support in Spark 3.5?

Are there any recommended workarounds or alternative approaches for Cosmos workloads in Spark 3.5?

Should we expect an upcoming patch or runtime update?

This issue is critical for us as it directly blocks the migration of Cosmos-based workloads.

Thank you for your support.
Pilladi Padma Sai Manisha 7,390 Reputation points Microsoft External Staff Moderator

2026-04-09T13:20:27.87+00:00

Hi Victor Buzy,
Thanks for calling this out , your concern makes complete sense, and I agree this creates a difficult situation from a migration standpoint.

Based on what you’ve shared and the behavior observed, this is best treated as a current limitation with Spark 3.5 when used with the Cosmos OLAP (cosmos.olap) connector via Synapse Link, rather than an issue in your code or configuration. The fact that the same workload works in Spark 3.4 and fails consistently in Spark 3.5 right at execution strongly points to a gap in compatibility at the runtime/connector level.

At the moment, there isn’t any public documentation that confirms full support of Cosmos OLAP workloads on Spark 3.5, and there’s no officially published ETA for a fix yet. These kinds of issues are typically addressed through runtime updates, so it’s reasonable to expect improvements over time, but there isn’t a committed timeline available right now.

In terms of how to move forward, the most practical approach today is to continue running Cosmos OLAP workloads on Spark 3.4 while moving other workloads to Spark 3.5 where possible. I understand that’s not ideal, but it does allow you to keep progressing your migration instead of blocking everything on this dependency.

If you need to explore options within Spark 3.5, you could look at alternative access patterns depending on your use case, for example, using Cosmos DB through transactional APIs or leveraging SQL-based paths that are already working in your environment. This won’t be a like-for-like replacement for Synapse Link OLAP scenarios, but in some cases it can help bridge the gap temporarily.

For background and current support details, these docs may be helpful: https://learn.microsoft.com/azure/cosmos-db/synapse-link https://learn.microsoft.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark https://learn.microsoft.com/azure/synapse-analytics/spark/apache-spark-version-support

I know this isn’t the smooth migration path you were expecting, and your feedback here is important. If you want to walk through your workload and see what can safely move to 3.5 versus what should stay on 3.4 for now, I’m happy to help with that.
Pilladi Padma Sai Manisha 7,390 Reputation points Microsoft External Staff Moderator

2026-04-14T18:13:20.1366667+00:00

Hi Victor Buzy,
Thank you for taking the time to explain the situation so clearly, I completely understand why this feels like a blocker, and your concern is very valid.

From what we’re seeing, this is best treated as a current limitation with Spark 3.5 when used with the Cosmos OLAP (cosmos.olap) connector via Synapse Link, rather than an issue with your implementation. The consistent behavior — working in Spark 3.4 and failing in Spark 3.5 at the point of execution — points to a compatibility gap at the runtime/connector level.

To your specific questions:

At this point, it is not formally documented as a known issue in public docs, but based on the reproducibility and version-specific behavior, it aligns with a limitation or regression in the current Spark 3.5 runtime with the OLAP connector.

There isn’t an officially published ETA or roadmap confirming when full support or a fix will be available for Spark 3.5 in this scenario. Updates of this nature are typically delivered through runtime or connector improvements, but no committed timeline has been shared publicly yet.

In terms of workarounds, the most reliable option today is to continue running Cosmos OLAP workloads on Spark 3.4 while moving other workloads to Spark 3.5 where possible. I understand this isn’t ideal, but it allows you to keep progressing your migration instead of being completely blocked. For Spark 3.5 specifically, some teams explore alternative patterns such as using Cosmos DB through transactional access (OLTP connector) or leveraging SQL-based querying paths where feasible, depending on workload requirements.

Regarding Spark 3.4 lifecycle, while it is moving toward end of support, it is not immediately removed. Azure Synapse typically maintains older runtimes for a transition period to allow customers to migrate safely. The exact retirement timeline isn’t fixed publicly, so it’s best to keep an eye on the official support page for updates: https://learn.microsoft.com/azure/synapse-analytics/spark/apache-spark-version-support

For additional context on how Synapse Link and Cosmos OLAP are expected to work with Spark, these references may help: https://learn.microsoft.com/azure/cosmos-db/synapse-link https://learn.microsoft.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark

I agree this creates a temporary gap between platform guidance and workload readiness, and your feedback highlights that clearly. If helpful, I’m happy to work with you on identifying a phased migration approach so that progress can continue while this limitation is being addressed.

4 answers

Your answer

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Pilladi Padma Sai Manisha 7,390 Reputation points Microsoft External Staff Moderator

2026-04-01T20:11:24.5033333+00:00

Hi Victor Buzy,
Based on what you’ve described, this behavior appears to be related to a compatibility gap between Spark 3.5 and the Cosmos DB OLAP (cosmos.olap) connector used through Synapse Link. The key observation is that the same code executes successfully in Spark 3.4 but fails in Spark 3.5 with the “Session isn’t active” error as soon as an action is triggered. This typically indicates that the Spark session is being marked inactive during execution, which points to a runtime or connector-level issue rather than anything in your code or configuration.

Given this, the most reliable approach at the moment is to continue running Cosmos OLAP workloads on Spark 3.4, where the connector integration is stable and functioning as expected. This ensures continuity while avoiding the session instability observed in Spark 3.5.

For additional context on how Synapse Link integrates with Spark and Cosmos DB, you can refer to the following documentation:

https://learn.microsoft.com/azure/cosmos-db/synapse-link
https://learn.microsoft.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark https://learn.microsoft.com/azure/synapse-analytics/spark/apache-spark-version-support

Your implementation and repro steps look correct, and the behavior you’re seeing aligns with a version-specific limitation rather than a usage issue. Please feel free to share if you notice any changes or need help validating alternative approaches.
Pilladi Padma Sai Manisha 7,390 Reputation points Microsoft External Staff Moderator

2026-04-03T21:53:00.7766667+00:00

Hi Victor Buzy,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Victor Buzy 25 Reputation points

2026-04-06T13:14:38.18+00:00

Hi @Anonymous

Thank you for your response.

However, this situation is blocking our migration strategy.

On one hand, Spark 3.4 is reaching end of support and we are encouraged to migrate to Spark 3.5.

On the other hand, Cosmos OLAP workloads are currently not working on Spark 3.5 and you are advised to remain on Spark 3.4.

This creates a contradiction, as we cannot fully migrate our platform while Cosmos OLAP is not supported or stable in Spark 3.5.

Could you please clarify:

Is this a known limitation or a confirmed bug in Spark 3.5 with cosmos.olap?

Is there an official ETA or roadmap for a fix or support in Spark 3.5?

Are there any recommended workarounds or alternative approaches for Cosmos workloads in Spark 3.5?

Should we expect an upcoming patch or runtime update?

This issue is critical for us as it directly blocks the migration of Cosmos-based workloads.

Thank you for your support.

Answer 1

Hi @Victor Buzy
I checked a similar case internally where the same error (“Session isn’t active”) was seen after moving from Spark 3.4 to 3.5 when using cosmos.olap.

In that case, the issue was related to Java module access changes in Spark 3.5, and it was resolved by adding the below configuration at the notebook level:

%%configure -f
{
  "conf": {
    "spark.driver.extraJavaOptions": "--add-opens java.base/sun.security.util=ALL-UNNAMED",
    "spark.executor.extraJavaOptions": "--add-opens java.base/sun.security.util=ALL-UNNAMED"
  }
}

Could you please try this once and see if it makes any difference?

Let me know how it goes after trying this.

Answer 2

Victor Buzy 25

Hi,

Thank you for the detailed clarification.

I understand that this is currently treated as a limitation or compatibility gap between Spark 3.5 and the Cosmos OLAP connector via Synapse Link. However, given the impact on our platform, I would like to clarify and escalate this further.

At the moment, this situation creates a blocking dependency for us:

Spark 3.4 is approaching end of support and we are expected to migrate

Cosmos OLAP workloads (critical for us) are not functional on Spark 3.5

There is no documented limitation, no ETA, and no clear migration path

From our perspective, this behaves like a regression:

Same code

Same configuration

Works in 3.4

Fails deterministically in 3.5 at execution time with session invalidation

Could you please help with the following:

Can this be formally confirmed as a known product limitation or a regression/bug?

Can this issue be escalated to the product/engineering team for investigation?

Is there an internal tracking ID / bug ID we can follow?

Are there any planned runtime or connector updates that target Spark 3.5 compatibility?

Is there any official guidance for customers relying on Synapse Link OLAP during Spark migration?

At this stage, a split strategy (3.4 for Cosmos / 3.5 for others) is operationally complex and not sustainable long term, so we need more visibility to plan accordingly.

We are happy to provide additional logs, run tests, or validate fixes if needed.

Thanks again for your support — we would really appreciate help in getting this in front of the relevant engineering team.

Best regards,
Victor

Pilladi Padma Sai Manisha 7,390 Reputation points Microsoft External Staff Moderator

2026-04-15T12:46:50.79+00:00

Hi Victor Buzy,
Thank you for the thoughtful follow-up, I completely understand why this is a blocker for your migration, and I appreciate how clearly you’ve laid out the impact.

From everything you’ve shared and the consistent repro, this does behave like a regression introduced with the Spark 3.5 runtime when used with the Cosmos OLAP (cosmos.olap) connector via Synapse Link. While it isn’t currently called out as a documented limitation in public docs, the pattern (same code and config working in 3.4 and failing deterministically in 3.5 at execution) strongly indicates a runtime/connector compatibility issue rather than expected behavior.

To your questions directly:

This can reasonably be treated as a product gap/regression scenario rather than an intended limitation, based on reproducibility and version-specific failure.

The issue can be escalated internally to the engineering team for investigation. While I don’t have a public-facing bug ID or tracking number to share at this point, cases like this are typically tracked internally once validated, especially when there is a clear minimal repro and business impact as in your case.

At the moment, there is no publicly communicated ETA or confirmed roadmap item specifically addressing Spark 3.5 compatibility for Cosmos OLAP. Updates in this area are generally delivered through runtime or connector improvements, but nothing has been formally announced yet.

There isn’t any official guidance published specifically for this exact migration gap. In practice, the interim approach has been to run Cosmos OLAP workloads on Spark 3.4 while gradually moving other workloads to 3.5, although I fully understand your point that this split model introduces operational overhead and isn’t a sustainable long-term solution.

Regarding next steps, what would help most from your side is exactly what you’ve already offered — the minimal repro, trace IDs, and confirmation of behavior across versions. That significantly strengthens the case for engineering review and prioritization.

For completeness, I’m including the relevant documentation that reflects current supported patterns and runtime alignment: https://learn.microsoft.com/azure/cosmos-db/synapse-link https://learn.microsoft.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark https://learn.microsoft.com/azure/synapse-analytics/spark/apache-spark-version-support

I agree with your assessment that the current state creates a gap between upgrade guidance and workload readiness. I’ll continue to treat this as a high-impact scenario and push for visibility with the engineering team. If there are any updates, fixes, or validation opportunities, I’ll make sure to share them with you right away.

Really appreciate your collaboration on this, and I’m here to help as we move this forward.

Answer 3

Hi @Anonymous

Thank you for your response.

However, this situation is blocking our migration strategy.

On one hand, Spark 3.4 is reaching end of support and we are encouraged to migrate to Spark 3.5.

On the other hand, Cosmos OLAP workloads are currently not working on Spark 3.5 and you are advised to remain on Spark 3.4.

This creates a contradiction, as we cannot fully migrate our platform while Cosmos OLAP is not supported or stable in Spark 3.5.

Could you please clarify:

Is this a known limitation or a confirmed bug in Spark 3.5 with cosmos.olap?
Is there an official ETA or roadmap for a fix or support in Spark 3.5?
Are there any recommended workarounds or alternative approaches for Cosmos workloads in Spark 3.5?
Should we expect an upcoming patch or runtime update?
What is the expected sunset date of spark 3.4, will it be removed from the cloud at what point of time?

This issue is critical for us as it directly blocks the migration of Cosmos-based workloads.

Thank you for your support.

Answer 4

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Share via

Spark 3.5 - Cosmos OLAP read fails with "Session isn't active" while working in Spark 3.4

🔹 Description

🔹 Minimal Repro Code

🔹 Observed Behavior

🔹 Expected Behavior

🔹 Additional Context

🔹 Description

🔹 Minimal Repro Code

🔹 Observed Behavior

🔹 Environment Details

🔹 Impact

4 answers

Your answer