Question : repartition in spark scala
Answered by : expensive-eagle-4ub623zm7847
rdd1.saveAsTextFile("/tmp/partition")
//Writes 6 part files, one for each partition
Partition 1 : 0 1 2
Partition 2 : 3 4 5
Partition 3 : 6 7 8 9
Partition 4 : 10 11 12
Partition 5 : 13 14 15
Partition 6 : 16 17 18 19Source : https://sparkbyexamples.com/spark/spark-repartition-vs-coalesce/ | Last Update : Wed, 14 Jul 21
Question : repartition in spark scala
Answered by : expensive-eagle-4ub623zm7847
val spark:SparkSession = SparkSession.builder() .master("local[5]") .appName("SparkByExamples.com") .getOrCreate() val df = spark.range(0,20) println(df.rdd.partitions.length) df.write.mode(SaveMode.Overwrite)csv("partition.csv")Source : https://sparkbyexamples.com/spark/spark-repartition-vs-coalesce/ | Last Update : Wed, 14 Jul 21