I am tasked to design system that should receive both files and via api and perform ETL functions. the end result is put into RDBMS.
for the sake of example, imagine a system that handles billing for electricity companies. some of the customers provide the usage as files on periodical basis, while others hook up the consumer meters directly and the meters send usage data every few minutes.
the processing of both input is same.
so my system answers to both batch and data streaming use cases. one of my colleagues suggested to design the system for data streaming and when file comes in, read the file and feed the incoming data streaming with lines from the file. I do not like this design but am struggling to find reasons why.
I searched this site and found this relevant question but still cannot tell how my system should be designed