Thiết kế website giá rẻ

Question

I want to perform the XOR operations between two 2D numpy.ndarray by saving the memory usage. For each row of u_values, I want to perform the XOR operation to every row of v_values and so on. For example, if u has dimension (8,10) and v has dimension (6,10), the result will end up with a dimension (48, 10).

Here is an example:

import numpy as np

u_values = np.array([[True, True, True, True, True, True, False, True, True, True],
                     [True, True, True, True, True, True, True, False, False, True],
                     [True, True, True, True, False, True, False, True, True, True],
                     [True, True, True, True, False, True, True, False, False, True],
                     [True, True, True, True, True, False, False, False, False, True],
                     [True, True, True, False, False, False, False, False, False, True],
                     [True, False, False, False, False, False, False, False, False, True],
                     [True, True, False, True, False, True, False, True, True, True]])

v_values = np.array([[True, True, True, True, True, True, False, True, True, True],
                     [True, False, True, False, True, True, True, False, False, True],
                     [True, True, True, True, False, True, False, True, True, True],
                     [True, False, True, True, False, False, True, False, False, True],
                     [True, True, False, True, True, False, False, False, False, True],
                     [True, True, True, False, False, True, False, False, False, True]])

#What I have tried so far:
u_reshaped = u_values[:, None, :]
v_reshaped = v_values[None, :, :]
xor_result = u_reshaped ^ v_reshaped
xor_results = xor_result.reshape((-1, xor_result.shape[2]))

This is what I get a result with dimension (48, 10)

[[False False False False False False False False False False]
 [False  True False  True False False  True  True  True False]
 [False False False False  True False False False False False]
 [False  True False False  True  True  True  True  True False]
 [False False  True False False  True False  True  True False]
 [False False False  True  True False False  True  True False]
 [False False False False False False  True  True  True False]
 [False  True False  True False False False False False False]
 [False False False False  True False  True  True  True False]
 [False  True False False  True  True False False False False]
 [False False  True False False  True  True False False False]
 [False False False  True  True False  True False False False]
 [False False False False  True False False False False False]
 [False  True False  True  True False  True  True  True False]
 [False False False False False False False False False False]
 [False  True False False False  True  True  True  True False]
 [False False  True False  True  True False  True  True False]
 [False False False  True False False False  True  True False]
 [False False False False  True False  True  True  True False]
 [False  True False  True  True False False False False False]
 [False False False False False False  True  True  True False]
 [False  True False False False  True False False False False]
 [False False  True False  True  True  True False False False]
 [False False False  True False False  True False False False]
 [False False False False False  True False  True  True False]
 [False  True False  True False  True  True False False False]
 [False False False False  True  True False  True  True False]
 [False  True False False  True False  True False False False]
 [False False  True False False False False False False False]
 [False False False  True  True  True False False False False]
 [False False False  True  True  True False  True  True False]
 [False  True False False  True  True  True False False False]
 [False False False  True False  True False  True  True False]
 [False  True False  True False False  True False False False]
 [False False  True  True  True False False False False False]
 [False False False False False  True False False False False]
 [False  True  True  True  True  True False  True  True False]
 [False False  True False  True  True  True False False False]
 [False  True  True  True False  True False  True  True False]
 [False False  True  True False False  True False False False]
 [False  True False  True  True False False False False False]
 [False  True  True False False  True False False False False]
 [False False  True False  True False False False False False]
 [False  True  True  True  True False  True  True  True False]
 [False False  True False False False False False False False]
 [False  True  True False False  True  True  True  True False]
 [False False False False  True  True False  True  True False]
 [False False  True  True False False False  True  True False]]

The final result that I want is a list of rows which each row contains the index of the True value

final_results = [np.nonzero(row)[0] + 1 for row in xor_results]

It worked pretty fine with a small example. However, for my real case, I encounter a very huge 2D numpy.ndarray with u_values with the size of (2788, 203769) and v_values with the size of (1813, 203769)

I ran out of memory for the XOR operations:

numpy.core._exceptions.MemoryError: Unable to allocate 959. GiB for an array with shape (2788, 1813, 203769) and data type uint8

I was wondering if there is a more straight forward and faster way to speed up this process handle this kind of problem in order to get the list of the desired result as mentioned above (i.e. final_results) without computationally so expensive?

Thiết kế website giá rẻ

Danh mục

Most efficient way to perform XOR operations between two matrices by using less memory