I’ve been messing around in unity and have begun learning about data serialization. My first tries have been to save the splatmaps/alphamaps of my terrain tiles as they change overtime.
The splatmaps/alphamaps are of size 5125123 = 786432. Over half a million float datapoints.
When I call SerializeObjects on my struct, it takes 50 seconds to serialize 16 splatmaps/alphamaps. The overall size of the file is 246mb. As this is my first time doing anything like this, I’m unsure if the time it takes is off by an order of magnitude or if it is within reasonable expectations of the time it would take to perform this operation.
Can you please help me figure out if there is an issue with the methodology ?
I’m using Newtonsoft and custom converters.
I’m having trouble understanding what the general benchmarks are and how to know if there’s an issue with the code itself. I’ve checked the output and it seems fine.
Code snippet where the serializing takes place
private void SaveTerrainCellData(string SaveName)
{
string saveName = $"{SaveName} - TerrainCellData";
string filePath = (Application.persistentDataPath + "/" + saveName + ".json");
string lines = null;
//Debug.Log(Application.persistentDataPath);
// Use custom converters
var settings = new JsonSerializerSettings
{
Converters = new JsonConverter[] { new Vector3Converter(), new StringEnumConverter() },
Formatting = Formatting.Indented
};
foreach (TerrainCellData data in this.terrainCellsData)
{
lines += JsonConvert.SerializeObject(data, settings);
}
File.WriteAllText(filePath, lines);
//await Task.Yield();
}
The data structure
public struct TerrainCellData
{
[SerializeField] public string TerrainName;
[Header("--- Weather ---")]
public Weather Weather;
public float MinutesSinceWeatherChange;
[Header("--- Simulation ---")]
public Biome Biome;
public float WaterLevel;
public float SunLevel;
public int DirtPatchCount;
public int GrassPatchCount;
public List<Vector3> DirtAlphaMapGridID;
public List<Vector3> GrassAlphaMapGridID;
public Vector3 TrailingGrassPatch;
public Vector3 TrailingDirtPatch;
public Dictionary<ObjectName, Vector3> TrailingPlants;
public int TrailingGrassPlantIndex;
public float[,,] OriginalSplatmapData;
public Dictionary<Vector3, float[,,]> AlphaMapGrid;
public float[,,] AlphaMap;
public void WriteData(string TerrainName,
Weather Weather,
float MinutesSinceWeatherChange,
Biome Biome,
float WaterLevel,
float SunLevel,
int DirtPatchCount,
int GrassPatchCount,
List<Vector3> DirtAlphaMapGridID,
List<Vector3> GrassAlphaMapGridID,
Vector3 TrailingGrassPatch,
Vector3 TrailingDirtPatch,
Dictionary<ObjectName, Vector3> TrailingPlants,
int TrailingGrassPlantIndex,
float[,,] OriginalSplatmapData,
Dictionary<Vector3, float[,,]> AlphaMapGrid,
float[,,] AlphaMap)
{
this.TerrainName = TerrainName;
this.Weather = Weather;
this.MinutesSinceWeatherChange = MinutesSinceWeatherChange;
this.Biome = Biome;
this.WaterLevel = WaterLevel;
this.SunLevel = SunLevel;
this.DirtPatchCount = DirtPatchCount;
this.GrassPatchCount = GrassPatchCount;
this.DirtAlphaMapGridID = DirtAlphaMapGridID;
this.GrassAlphaMapGridID = GrassAlphaMapGridID;
this.TrailingGrassPatch = TrailingGrassPatch;
this.TrailingDirtPatch = TrailingDirtPatch;
this.TrailingPlants = TrailingPlants;
this.TrailingGrassPlantIndex = TrailingGrassPlantIndex;
this.OriginalSplatmapData = OriginalSplatmapData;
this.AlphaMapGrid = AlphaMapGrid;
this.AlphaMap = AlphaMap;
}
}
Thank you all very much!
MrSig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
4
Json is not great for serializing large amounts of binary data, like large arrays. It will have significantly lower performance than just writing the data directly to a file.
A fairly common approach is to have one json file with all metadata, and a separate file with the actual raw binary data. You seem to have multiple arrays, so would need multiple files. A slightly more advanced design is to put all the files in a zip-archive with some custom extension, so you still have a single file to manage. You can use BinaryWriter.Write(ReadOnlySpan<byte>) to write an array directly to a stream, after casting the span for the array to the correct type.
A note about multi dimensional arrays. These can be a bit more difficult to use when serializing, since some methods only work with 1D arrays. There can also be issues with very large volumes, since there is a max size per array. Some of these issues can be worked around with unsafe code, but not all.
In my experience a better approach is to split your volume into separate 2D-slices, where each slice uses a plain 1D array as storage, and calculate the 2D-index yourself ([y * width + x]). This provides a decent tradeoff between allocation sizes, number of indirection, etc. But this will require a fair bit of more code, and is probably not necessary for smaller volumes. If your arrays are not actually volumetric data, and instead contains some kind of structure data, like RGB or vectors, you probably want to use an Array of structs to ensure type safety.