I have a dataframe which I stream in using Spark Structured Streaming .readStream():
| Id | json_data |
|---|---|
| 123 | {color: “red”, value: “#f00”} |
| 125 | {color: “blue”, value: “#f45”} |
I want to send each json_data from each row to a rest API, as a json payload. What is the best way of doing this?
I’m aware of a dataframe writer (https://docs.databricks.com/en/structured-streaming/foreach.html) for Databricks however unclear how I’d do this.
Do I need to write the column into a python dictionary?
Bit confused how this script would work, the streaming data comes in and appends onto the dataframe, but I need the json_data column (which is stored as a string) to be used as a payload.