I am trying to create some sample datasets that I will pass to my main function from the Test class.
One of the samples looks like this:
<code> val expected_ds_t1 = Seq(
driveAggMonthly(1, 1),
driveAggMonthly(2, 2),
driveAggMonthly(3, 1)
).toDS().as[driveAggMonthly]
</code>
<code> val expected_ds_t1 = Seq(
driveAggMonthly(1, 1),
driveAggMonthly(2, 2),
driveAggMonthly(3, 1)
).toDS().as[driveAggMonthly]
</code>
val expected_ds_t1 = Seq(
driveAggMonthly(1, 1),
driveAggMonthly(2, 2),
driveAggMonthly(3, 1)
).toDS().as[driveAggMonthly]
Where the driveAggMonthly class looks like:
<code>case class driveAggMonthly(Month: Int,
numberOfDrives: BigInt)
</code>
<code>case class driveAggMonthly(Month: Int,
numberOfDrives: BigInt)
</code>
case class driveAggMonthly(Month: Int,
numberOfDrives: BigInt)
Now, when I run printSchema() on expected_ds_t1, it looks like :
<code>root
|-- Month: integer (nullable = false)
|-- numberOfDrives: decimal(38,0) (nullable = true)
</code>
<code>root
|-- Month: integer (nullable = false)
|-- numberOfDrives: decimal(38,0) (nullable = true)
</code>
root
|-- Month: integer (nullable = false)
|-- numberOfDrives: decimal(38,0) (nullable = true)
This is an issue for me because I later compare this result with another, which indeed has a BigInt cast for numberOfDrives.
I don’t understand why the schema of the case class is not preserved.
I know I can cast this as a Dataframe and change the column type and then recast it back but I would like to avoid doing this.
New contributor
Vanessa Hydoski is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.