I am trying to store data in Apache hive using filter operation where MYVARIABLE is a parameter sent.
Filtering using the variable sends nullpointer exception .This happens when I try to store data in hive or use DUMP function.
MY_TABLE = FILTER MY_TABLE BY (value1 == ‘${VALUE1}’ and
(LOWER(‘${MYVARIABLE}’) ==’true’));
Error Is:
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule PartitionFilterOptimizer. Try -t PartitionFilterOptimizer
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:125)
at ....
Caused by: java.lang.NullPointerException
at org.apache.hive.hcatalog.data.schema.HCatSchema.get(HCatSchema.java:105)
at org.apache.hive.hcatalog.pig.HCatLoader.getHCatComparisonString(HCatLoader.java:317)
at org.apache.hive.hcatalog.pig.HCatLoader.setPartitionFilter(HCatLoader.java:249) ....
Have tried this too
MY_TABLE = FILTER MY_TABLE BY (LOWER(‘${MYVARIABLE}’) == ‘false’ ?
false : true);
This works when storing in csv.
So basically I want to filter data based on the value of MYVARIABLE is it possible in pig and store in hive? The code works fine without the filter line.