PROBLEM
PySpark crashes with Python 3.12+ on Windows
Description
PySpark crashes when calling createDataFrame() in Spark local mode on Windows with Spark 3.5+ and Python 3.12+. This crash prevents running many of the code samples in the ArcGIS GeoAnalytics Engine documentation, though it will occur whether or not Geoanalytics Engine is installed. This bug will not reproduce when using non-Windows operating systems or when running Spark in cluster mode.
PySpark crashes result in a traceback that includes "org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)"; however, if your environment does not match the description above, the crash is most likely unrelated to this issue.
Cause
The root cause is a bug in the core PySpark API that may or may not be resolved in a future release. For more information on the status of this bug see https://issues.apache.org/jira/browse/SPARK-53759 .
Solution or Workaround
To resolve this issue, make any of the following changes to your environment:
- Run Spark in cluster mode.
- Use a non-Windows operating system.
- Downgrade Python to version 3.11.13 or lower.
- Downgrade Spark to version 3.4 or lower.
Article ID: 000039267
- ArcGIS GeoAnalytics for Microsoft Fabric
Get support with AI
Resolve your issue quickly with the Esri Support AI Chatbot.
Related Information
Discover more on this topic
Search for related information
Find training related to this topic
Explore ideas and give feedback
Get help from ArcGIS experts
Start chatting now