So I understand that there is a 16MB limit when loading XML files into variant column within Snowflake.
Data looks like
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE note SYSTEM "Note.dtd">
<output>
<response>
<person>
<id>123</id>
<name>John</name>
</person>
<person>
<id>124</id>
<name>Ava</name>
</person>
</response>
</output>
I am only interested in the person data and flattening the attributes into a table.
I am trying to load XML file that is larger than 16MB. is it possible to strip further outer elements, or another workaround within Snowflake itself?
Or is the only option to split the file before loading into Snowflake with another tool?
2
Or is the only option to split the file before loading into Snowflake with another tool?
An alternative is to put “raw” XML file into Snowflake stage using preferred method:
- PUT
- Snowsight UI
- ETL Tool
Then process staged XML file with Snowflake function or stored procedure(Python, Java, Scala) to a desired form and save results into a table.
More at:
- Reading files
- Processing unstructured data with UDF and procedure handlers