Question : foreach batch spark scala
Answered by : ogun-ozalp
streamingDF.writeStream.foreachBatch { (batchDF: DataFrame, batchId: Long) => batchDF.persist() batchDF.write.format(...).save(...) // location 1 batchDF.write.format(...).save(...) // location 2 batchDF.unpersist()
}Source : https://docs.databricks.com/spark/latest/structured-streaming/foreach.html | Last Update : Tue, 24 May 22