![]() If a node has 32 GB of memory, then setting this property to 2 GB and executor memory to 13 GB will allow 2 executors to run on this node. The value for + should not be more than what a YARN container can support. Recommendation: Memory cannot be shared across nodes. The value of the property can be changed at any time and will be respected in the next build.įor this property, the Spark applications should be running on YARN, which means Spark must be deployed in cluster mode. This means when the underlying build engine is set as Spark. This property comes into effect only when is set as Spark. NOTE: If the property is set on a dataset and a dataset is built, then the value will override the connection level value for that dataset build job. Dataset: If the property is set on a dataset then the value will override cube level value for that cube’s dataset build job.Cube: If the property is set on a cube, then the value will override the connection level value for that cube’s build job. ![]() Connection: If the property is set at the connection then the property value is applicable for all dataset build, cube build, or data profile jobs launched using Spark.If you enter 5g, Kyvos picks it as 5 GB.ĭefault value: 2048 - 2 GB will be allocated to each Spark executor as the off-heap memory. By default, the values are in MB, to enter the value in GB, add a g at the end, and Kyvos will pick the value in GB. For example, if you enter the value as 5120, Kyvos picks it as 5120 MB. Values and behavior: Any positive integer over 384. Description: This property specifies the amount of off-heap memory per executor when jobs are executed using Spark. Kyvos delegates this property value to the same-named property of Spark when Kyvos submits a dataset build, cube build, or data profile job to Spark.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |