I am new to Kafka Connect. I am using File pulse source connector. My use case is to skip first line in a file, use second line as headers, and values from third line onwards.
Example file: testFile
sampleFileabc
key1|key2
val1|val2
Now I want to convert it as below by ignoring line 1.
{
key1: val1,
key2: val2
}
How can I achieve this using File pulse source connector ?
When I use skip.headers=1
in source connector properties, it does not work.
These are the properties
filters.ParseCSVLine.extract.column.name=headers
filters.ParseCSVLine.separator=\|
filters.ParseCSVLine.trim.column=true
filters.ParseCSVLine.type=io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter
filters.SubstituteFilter.field=$key
filters.SubstituteFilter.type=io.streamthoughts.kafka.connect.filepulse.filter.AppendFilter
filters.SubstituteFilter.value=$metadata.name
filters=ParseCSVLine,SubstituteFilter
skip.footers=0
skip.headers=1
tasks.reader.class=filepulse.fs.reader.LocalRowFileInputReader
New contributor
Shivani is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.