I’m encountering a MemoryError while working on a Python program that involves processing an array (made with 311 observations, i.e. numbers). Specifically, I’m attempting to calculate the signed area of intersected rectangles, but I keep running into memory allocation issues.
Here’s the code:
<code>import pandas as pd
import matplotlib.pyplot as plt
# Define the step function
return np.where((x >= 0) & (x < 1), 1, 0)
f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
df = pd.read_excel(r'C:UsersMefile.xlsx')
a = df['EUIGDP'].to_numpy()
import matplotlib.pyplot as plt
# vectorized version of psi
return ((x>=0)&(x<1))*(2*(x<0.5)-1)
return 2**(j/2) * psi(2**j * x - n)
import matplotlib.pyplot as plt
# Define the x-values for plotting
x = np.arange(0, sample_size + 1, step)
j_values = [x for x in range(-J, 0)] # values of j (the abstract Fourier coeffcients are different from zero only if j<0)
n_values = [x for x in range(0, N+1)] # values of n (the abstract Fourier coeffcients are different from zero only if n>=0))
matrix_psi_j_n = np.zeros((len(j_values), len(n_values)))
# Code without resorting to integrals but only to the calculation of the area of rectangles
# Calculate the signed area of the intersected rectangles
area=(psi_j_n(x,j,n)*step_function(x, a)).sum()*step
matrix_psi_j_n[-min(j_values)+j, -min(n_values) + n ] = area
print(f"The Fourier coefficient for j={j}, n={n} is: {area}")
# Set the threshold value
threshold = 0.00001 # Example threshold value, please adjust according to your needs
# Replace values below the threshold with 0
matrix_psi_j_n[np.abs(matrix_psi_j_n) < threshold] = 0
<code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# Define the step function
def step_function(x, a):
def rect(x):
return np.where((x >= 0) & (x < 1), 1, 0)
f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
return f
# Read the XLSX file
df = pd.read_excel(r'C:UsersMefile.xlsx')
a = df['EUIGDP'].to_numpy()
import numpy as np
import matplotlib.pyplot as plt
# vectorized version of psi
def psi(x):
return ((x>=0)&(x<1))*(2*(x<0.5)-1)
def psi_j_n(x, j, n):
return 2**(j/2) * psi(2**j * x - n)
import numpy as np
import matplotlib.pyplot as plt
# Define the step
step = 0.00001
# Define the x-values for plotting
x = np.arange(0, sample_size + 1, step)
J=20
N=20
j_values = [x for x in range(-J, 0)] # values of j (the abstract Fourier coeffcients are different from zero only if j<0)
n_values = [x for x in range(0, N+1)] # values of n (the abstract Fourier coeffcients are different from zero only if n>=0))
matrix_psi_j_n = np.zeros((len(j_values), len(n_values)))
# Code without resorting to integrals but only to the calculation of the area of rectangles
for j in j_values:
for n in n_values:
# Calculate the signed area of the intersected rectangles
area=(psi_j_n(x,j,n)*step_function(x, a)).sum()*step
matrix_psi_j_n[-min(j_values)+j, -min(n_values) + n ] = area
print(f"The Fourier coefficient for j={j}, n={n} is: {area}")
# Set the threshold value
threshold = 0.00001 # Example threshold value, please adjust according to your needs
# Replace values below the threshold with 0
matrix_psi_j_n[np.abs(matrix_psi_j_n) < threshold] = 0
</code>
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# Define the step function
def step_function(x, a):
def rect(x):
return np.where((x >= 0) & (x < 1), 1, 0)
f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
return f
# Read the XLSX file
df = pd.read_excel(r'C:UsersMefile.xlsx')
a = df['EUIGDP'].to_numpy()
import numpy as np
import matplotlib.pyplot as plt
# vectorized version of psi
def psi(x):
return ((x>=0)&(x<1))*(2*(x<0.5)-1)
def psi_j_n(x, j, n):
return 2**(j/2) * psi(2**j * x - n)
import numpy as np
import matplotlib.pyplot as plt
# Define the step
step = 0.00001
# Define the x-values for plotting
x = np.arange(0, sample_size + 1, step)
J=20
N=20
j_values = [x for x in range(-J, 0)] # values of j (the abstract Fourier coeffcients are different from zero only if j<0)
n_values = [x for x in range(0, N+1)] # values of n (the abstract Fourier coeffcients are different from zero only if n>=0))
matrix_psi_j_n = np.zeros((len(j_values), len(n_values)))
# Code without resorting to integrals but only to the calculation of the area of rectangles
for j in j_values:
for n in n_values:
# Calculate the signed area of the intersected rectangles
area=(psi_j_n(x,j,n)*step_function(x, a)).sum()*step
matrix_psi_j_n[-min(j_values)+j, -min(n_values) + n ] = area
print(f"The Fourier coefficient for j={j}, n={n} is: {area}")
# Set the threshold value
threshold = 0.00001 # Example threshold value, please adjust according to your needs
# Replace values below the threshold with 0
matrix_psi_j_n[np.abs(matrix_psi_j_n) < threshold] = 0
Here’s the error message I’m getting:
<code>---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
22 # Calculate the signed area of the intersected rectangles
---> 23 area=(psi_j_n(x,j,n)*step_function(x, a)).sum()*step
24 matrix_psi_j_n[-min(j_values)+j, -min(n_values) + n ] = area
25 print(f"The Fourier coefficient for j={j}, n={n} is: {area}")
Cell In[1], line 10, in step_function(x, a)
8 return np.where((x >= 0) & (x < 1), 1, 0)
---> 10 f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
Cell In[1], line 10, in <listcomp>(.0)
8 return np.where((x >= 0) & (x < 1), 1, 0)
---> 10 f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
Cell In[1], line 8, in step_function.<locals>.rect(x)
----> 8 return np.where((x >= 0) & (x < 1), 1, 0)
File <__array_function__ internals>:200, in where(*args, **kwargs)
MemoryError: Unable to allocate 119. MiB for an array with shape (31100000,) and data type int32
<code>---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
Cell In[3], line 23
20 for j in j_values:
21 for n in n_values:
22 # Calculate the signed area of the intersected rectangles
---> 23 area=(psi_j_n(x,j,n)*step_function(x, a)).sum()*step
24 matrix_psi_j_n[-min(j_values)+j, -min(n_values) + n ] = area
25 print(f"The Fourier coefficient for j={j}, n={n} is: {area}")
Cell In[1], line 10, in step_function(x, a)
7 def rect(x):
8 return np.where((x >= 0) & (x < 1), 1, 0)
---> 10 f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
11 return f
Cell In[1], line 10, in <listcomp>(.0)
7 def rect(x):
8 return np.where((x >= 0) & (x < 1), 1, 0)
---> 10 f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
11 return f
Cell In[1], line 8, in step_function.<locals>.rect(x)
7 def rect(x):
----> 8 return np.where((x >= 0) & (x < 1), 1, 0)
File <__array_function__ internals>:200, in where(*args, **kwargs)
MemoryError: Unable to allocate 119. MiB for an array with shape (31100000,) and data type int32
</code>
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
Cell In[3], line 23
20 for j in j_values:
21 for n in n_values:
22 # Calculate the signed area of the intersected rectangles
---> 23 area=(psi_j_n(x,j,n)*step_function(x, a)).sum()*step
24 matrix_psi_j_n[-min(j_values)+j, -min(n_values) + n ] = area
25 print(f"The Fourier coefficient for j={j}, n={n} is: {area}")
Cell In[1], line 10, in step_function(x, a)
7 def rect(x):
8 return np.where((x >= 0) & (x < 1), 1, 0)
---> 10 f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
11 return f
Cell In[1], line 10, in <listcomp>(.0)
7 def rect(x):
8 return np.where((x >= 0) & (x < 1), 1, 0)
---> 10 f = np.sum([a[k-1] * rect(x - k) for k in range(1, len(a) + 1)], axis=0)
11 return f
Cell In[1], line 8, in step_function.<locals>.rect(x)
7 def rect(x):
----> 8 return np.where((x >= 0) & (x < 1), 1, 0)
File <__array_function__ internals>:200, in where(*args, **kwargs)
MemoryError: Unable to allocate 119. MiB for an array with shape (31100000,) and data type int32
It seems like the issue arises when trying to create an array with approximately 31100000 elements. My current implementation probably utilizes a large amount of memory, which leads to this error (but I do not know why).
I’d appreciate any advice or suggestions on how to tackle this problem more efficiently.
Thank you in advance for your help!