I have an excerpt of code below which pushes data into a sql database table.
I am utilising SQL within a Visual Studio C# Console App in .NET 4.8.
The data that is being pushed is called via API, parsed from Json and populates a data table I have created within C#. The next step is to populate a table in SQL with this output.
The first part runs very quickly, in less than half a second, however the second part, below causes the integration to take several minutes to run.
I have used a similar method in other integrations I’ve built (either create/update/delete or alternatively merge) and the results have usually been quicker.
Any advice or suggestions are very welcome on what I could be doing wrong here, thank you!
C
//sql staging connection
SqlConnection connection = ConnectionStrings.SQLStagingArea();
SqlCommand delEmployeeUpdate = new SqlCommand("DELETE sample.data.employeestoUpdate;", connection);
string mergeEmployees = @"DELETE sample.data.employees
WHERE NOT EXISTS(SELECT * FROM sample.data.employees_toUpdate WHERE employees_toUpdate.Empid = employees_toUpdate.EmpId)
UPDATE sample.data.employees
SET serviceNumber = employees_toUpdate.serviceNumber,
firstName = employees_toUpdate.firstName,
lastName = employees_toUpdate.lastName,
homePhone = employees_toUpdate.homePhone,
workPhone = employees_toUpdate.workPhone,
mobilePhone = employees_toUpdate.mobilePhone,
workMobilePhone = employees_toUpdate.workMobilePhone
FROM sample.data.employees_toUpdate
WHERE employees.Empid = employees_toUpdate.Empid
INSERT INTO sample.data.employees
SELECT DISTINCT
empId,
serviceNumber,
firstName,
lastName,
homePhone,
workPhone,
mobilePhone,
workMobilePhone
FROM sample.data.employees_toUpdate
WHERE
NOT EXISTS(
SELECT * FROM sample.data.employees
WHERE
employees_toUpdate.empId = sample.data.employees.empId)";
SqlCommand mergeCmdEmployees = new SqlCommand(mergeEmployees, connection);
using (connection)
{
connection.Open();
delEmployeeUpdate.ExecuteNonQuery();
using (var bulkCopy = new SqlBulkCopy(connection))
{
foreach (DataColumn col in employeesDT.Columns)
{
bulkCopy.ColumnMappings.Add(col.ColumnName, col.ColumnName);
}
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.DestinationTableName = "sample.data.employees_toUpdate";
try
{
bulkCopy.WriteToServer(employeesDT);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.ReadLine();
}
}
mergeCmdEmployees.ExecuteNonQuery();
connection.Close();
}
My expectation is that the code should run within a couple of seconds.
To optimize this I have attempted to simplify the code to see where the problem areas are – the major cause of delay appears to be the following command.
mergeCmdEmployees.ExecuteNonQuery();