I read some related posts here, and success using do the row majored matrixes multiplication with cuBLAS:

A*B (column majored) = B*A (row majored)

I write a wrapper to do this so that I can pass row majored matrixes A and B and return results accordingly.

cublasStatus_t gemm_float(cuDataF &out, cublasHandle_t &handle, const cuDataF &in1, int row1, int col1, cublasOperation_t trans1, const cuDataF &in2, int row2, int col2, cublasOperation_t trans2, float alpha, float beta) {
        /*if (trans1 == CUBLAS_OP_N && trans2 == CUBLAS_OP_T) {
            
        }*/

        return cublasSgemm(handle, trans1, trans2,
                            col2, row1, col1,
                            &alpha,
                            in2.data(), col2,
                            in1.data(), col1,
                            &beta,
                            out.data(), col2);
  }

But now my question is, if I want A*transpose(B) in row majored, how to do this?

I tried to derivation mathematics by hand but the answer is wrong.

Like:

A(2x3) = {1,2,3,4,5,6} (row majored) 
B(2x3) = {7,8,9,10,11,12} (row majored)
A*transpose(B)?