I am trying to get summarized workflow metrics to write to a table. I decided to use a (python) script transform that looks like this
The code is as follows (as seen in the screenshot)
import re
import json
metrics_regex = r'dddd/dd/ddsdd:dd:dds-s.*s-sFinished processings(I=(d+),sw=d+,sw=d+,sw=(d+),sw=d+,sw=(d+))'
matches = re.findall(metrics_regex, rowMeta.getString(row, 6))
ingest_nr = 'total_records_ingested'
written_nr = 'total_records_written'
error_nr = 'total_errored_records'
metrics = {'hop_job': rowMeta.getString(row, 1)}
for match in matches:
r_in, r_out, r_err = int(match[0]), int(match[1]), int(match[2])
if len(metrics.keys()) == 1:
metrics[ingest_nr] = r_in
metrics[written_nr] = r_out
metrics[error_nr] = r_err
else:
metrics[ingest_nr] += r_in
metrics[written_nr] += r_out
metrics[error_nr] += r_err
metrics = json.dumps(metrics)
However, I cannot get the required field written to the table. It outputs as NULL to the destination table. Everything else in the pipeline works as intended. This is the only issue I’m having. How can I go about resolving this to ensure that I get the relevant metrics data into the destination table?
I have tested this script locally on a sample string and it works.
To clarify: The code works, this is code that is part of the script transform in an Apache Hop pipeline. The core issue is that the metrics field is written as NULL to the destination table as opposed to a non-null JSON value
2
Initialize metrics if they are not already present before:
for match in matches:
like :
metrics = {'hop_job': rowMeta.getString(row, 1)}
if len(metrics) == 1:
metrics[ingest_nr] = 0
metrics[written_nr] = 0
metrics[error_nr] = 0
for match in matches:
.
.
.
There are some examples in the Apache Hop Script transform docs.
Alternatively, you might want to look at the Pipeline Log and Workflow Log metadata types. These allow you to specify a pipeline to process logging data, which allows you to write to database tables and all of the other functionality pipelines offer without the need to write code.