I’m tryin to uplaod csv file to bigquery using spring – boot java
I;m getting 400 invaid data
public void uploadToBigQuery(File csvFile, String table, boolean overwrite, List<Pair<String, String>> columns) {
log.info(“Upload CSV file [{}] to BigQuery table [{}/{}]”, csvFile.getPath(), dataSetName, table);
try {
// Fields
List<Field> fields = columns.stream()
.map(c -> Field.of(c.getFirst(), StandardSQLTypeName.valueOf(c.getSecond())))
.collect(Collectors.toList());
TableId tableId = TableId.of(dataSetName, table);
WriteChannelConfiguration writeChannelConfiguration =
WriteChannelConfiguration
.newBuilder(tableId)
.setFormatOptions(FormatOptions.csv())
.setCreateDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED)
.setWriteDisposition(overwrite ? JobInfo.WriteDisposition.WRITE_TRUNCATE : JobInfo.WriteDisposition.WRITE_APPEND)
.setSchema(Schema.of(fields))
.build();
// The location must be specified; other fields can be auto-detected.
JobId jobId = JobId.newBuilder().setLocation(DATASET_LOCATION).build();
TableDataWriteChannel writer = bigQuery.writer(jobId, writeChannelConfiguration);
// Write data to writer
try (OutputStream stream = Channels.newOutputStream(writer)) {
Files.copy(csvFile.toPath(), stream);
}
// Get load job
Job job = writer.getJob();
job = job.waitFor();
System.err.println("BigQuery job error: " + job.getStatus().getError().toString());
System.err.println("Full error: " + job.getStatus());
JobStatistics.LoadStatistics stats = job.getStatistics();
log.info("Wrote {} records to BigQuery table [{}/{}]", stats.getOutputRows(), dataSetName, table);
} catch (IOException | InterruptedException | BigQueryException ex) {
log.error("Error occured during writing of file to BigQuery", ex);
}
}
but got below error
2024-07-04T23:28:06.389+05:30 INFO 32704 --- [ main] c.google.api.client.http.HttpTransport : { "error": { "code": 400, "message": "Error: 3848323", "errors": [ { "message": "Error: 3848323", "domain": "global", "reason": "invalid" } ], "status": "INVALID_ARGUMENT" } }
Why this error and how ro trouble shoot
Lakshitha Nalindanath is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.