I have a dataset representing coders’ scores, with a final column representing the final score from coder discussion:

df <- data.frame(
  Coder1 = c(1, NA, 0, 1, NA),
  Coder2 = c(1, 1, NA, 4, 1),
  Coder3 = c(NA, 0, 1, NA, 0),
  Resolved_score = c(1, 1, 0, 1, 0)
)

I would like to get the kappas of the whole dataset by getting the kappa of each pair of coders + the final score in each row ignoring the NA value of the non-coder.

I tried using kappam.fleiss(df, na.rm=T) as my professor suggested, but that doesn’t seem to work.

I was wondering if a different irr function or an alternative package would be able to calculate the overall kappa of the dataset (each combination of coders in each row + the final score) while ignoring the non-coder in each row.