115,289 questions
0
votes
4
answers
241
views
How can I compare two pandas DataFrames with object-type columns, with a numeric tolerance?
I have two pandas dataframes: One assembled manually in Python, the other imported from a dashboard's .csv output.
All columns in both dataframes are objects, and look like this:
2020
2021
2022
2023
0....
5
votes
1
answer
303
views
Why is the Panda's apply function so slow when iterating over an entire row, rather than a specific column? [duplicate]
My intuition when using Pandas is that, if you have to use df.apply, it would be more optimal to group all the apply operations into one call. This was further reinforced by me learning that NumPy ...
2
votes
0
answers
168
views
TensorFlow/Keras model accumulates system and GPU RAM during training
I am training a model using TensorFlow/Keras using TensorFlow 2.19.0/Keras 3.10.0. During training, I monitor nvidia-smi and top, and the system RAM and the GPU RAM increase during the training period....
0
votes
1
answer
90
views
how to create uniform nested lists and convert them in to multi dimensional arrays?
i would like to create a nested list of arbitrary depth (containing numerical values specifically) with uniform arbitrary lengths at each level and then compress it to a NumPy array of minimum ...
3
votes
1
answer
208
views
Lambda container - Pyarrow and numpy
I have difficulties from this: (aws-lambda-python-alpha): Failed to install numpy 2.3.0 with Python 3.11 or lower
My Dockerfile:
FROM public.ecr.aws/lambda/python:3.11
# Install
RUN pip install '...
13
votes
2
answers
2k
views
What's wrong with this Python assignment on signal processing - mostly Fourier series and transform
In this assignment I created the basic rect signal a[n] such that over the domain [-1000, 1000] it's 1 only at |n|<100, meaning it's an array (complex one with zero in the imaginary part) that ...
1
vote
1
answer
124
views
How to create a numpy dtype object array from a python list without copying data?
As the numpy docs describe for the object dtype, arrays created with the object dtype are simply references to an underlying data store like a python list. The tobytes() method on such an object ...
3
votes
1
answer
240
views
How to shuffle inner dimensions /axis of an array without disturbing the internal structure/data?
It seems like the standard numpy.random.shuffle function shuffles (in place) only the first dimension / axis. I want a similar functionality for inner dimensions.
Note that numpy.random.default_rng()....
1
vote
2
answers
106
views
Issues with boundary layers in step propagation
I am trying to create a function which receives the transversal profile of a beam and outputs the transversal profile propagated by some distance. To exemplify the task, here is the profile of a ...
6
votes
1
answer
133
views
What is wrong with my block GMRES implementation?
I'm trying to implement a block GMRES procedure (i.e., GMRES to solve Ax=b, but with b that is not a vector but a n x r matrix, where r << n). My goal is to have a first implementation in Python ...
1
vote
1
answer
132
views
Numpy Bug in computing business days?
I am trying to compute business days using Numpy.
I how ever found an inconsistency.
import numpy as np
# Count from 1st Jan 2023 to 31st Jan 2023
print("Jan: {0}".format(np.busday_count('...
2
votes
1
answer
104
views
Is there a clean pythonic way of querying multiple points on a bivariate spline in the B-spline basis? [closed]
I need to efficiently evaluate a bivariate spline on a B-spline basis. I have already calculated the knot positions and spline coefficients (independently of scipy classes/methods such as ...
3
votes
3
answers
124
views
How do element assignments work with numpy arrays using an array for indexing?
I'm very puzzled by the following behaviour of NumPy when assigning elements to an array using an array as indices. Here is a minimal working example:
import numpy as np
i = np.arange(2,4)
a = np....
1
vote
1
answer
86
views
Is there a simpler way to select 2D vectors bounded by a box from a NumPy array?
This Python script:
import numpy as np
a = np.arange(12).reshape(6, 2)
inf = np.array([2, 2])
sup = np.array([9, 9])
b = (inf < a) & (a < sup)
r = a[b[:, 0] & b[:, 1]]
creates a ...
0
votes
2
answers
159
views
Nested for-loop alternative
I have a python code that calculates a function F(X), where both F and X are arrays of the same shape. F(X) uses another function called from a package that only accepts a scalar as an argument, but I ...
0
votes
1
answer
98
views
Difference between Tensorflow/Keras Dense Layer output and matmul operation with weights with NumPy
I was trying to mimic the result of a simple Tensorflow/Keras Dense layer with NumPy (forward pass only) and I was surprised not to have the exact same result.
A dense layer output is just the product ...
3
votes
2
answers
221
views
Why does `np.random.default_rng()` prevent seed change compared to `numpy.random.seed`?
I recently learned about the downsides of using numpy.random.seed(seed). Some internal file can modify the seed.I read an online article that suggested I use np.random.default_rng(). But I am not sure ...
2
votes
3
answers
94
views
Unfolding a cartesian binned dataset into polar coordinates
I have a dataset of binned events, corresponding to a cartesian coordinate set
[[ 0. 0. 0. 0. 0. 0. 2. 5. 2. 3. 3. 0. 0. 0. 0. 0. 0. 0.]
[ 0. 0. 0. 0. 0. 4. 10. 9. 7. 10. 6. ...
0
votes
0
answers
75
views
numpy TypeError with importing GPflow in ipython
I installed GPflow and just tried to import it with ipython. But, I got an error like below.
---------------------------------------------------------------------------
TypeError ...
9
votes
1
answer
314
views
Generating blue noise with values sampled from a log normal distribution
Aim
I am trying to generate random signals with the following two properties:
The values should be approximately log-normally distributed (any long-tailed distribution bounded form below with non-...
1
vote
1
answer
174
views
Calculate Exponential decay rate constant from graph
I would like to calculate the lifetime of the particles from the extrapolated curve below, I know that lifetime is 1/decay constant but I get values for the decay constant that dont make sense.
import ...
4
votes
3
answers
168
views
NumPy broadcasting error when trying to add 2D array and 1D array
I'm trying to perform an element-wise addition between a 2D NumPy array and a 1D array (like adding a row vector to each row of a matrix), but I'm running into a broadcasting error.
import numpy as np
...
0
votes
0
answers
99
views
Why is my numpy-based custom data loader extremely slow and unstable when iterating over large tick data
I'm currently working on a model similar to DeepLOB, using high-frequency tick-level financial data. Due to the massive volume and the need to structure the data into time series format, it's ...
0
votes
1
answer
114
views
In Python, How to run two statistical tests on all numeric columns [closed]
I have a dataframe df, I want to do the following:
run two stats tests on all the numeric columns (column_1 to column_84) to compare if there is a statistical difference between Types X, Y and Z
The ...
-3
votes
1
answer
78
views
How do I Convert data type in Python? [closed]
I have written this code:
df.loc[df['Profile Name'] == 'karladdo201', 'Duration'].astype('timedelta64[s]').sum()
and get this value:
Timedelta('8 days 12:02:45')
How do I get a values similar or in ...
3
votes
1
answer
59
views
Using a class with numpy does not work correctly
I define an array of 4 corners
import numpy as np
class Corner():
centres = np.zeros((3,2))
id=0
corners=np.empty((4)).astype(Corner)
for i in range(4):
corner=Corner()
corner.id = ...
0
votes
2
answers
281
views
How to access documentation of libraries like numpy, scipy etc. via pydoc offline on my workstation
Everytime, I want to access the documentation of libraries that I need for my project, I have to go to their website. Python has the excellent utility called pydoc, but I can only access the ...
1
vote
1
answer
108
views
ValueError: Eigenvalues did not converge
I am seeing below error with numpy v2.0.2.
numpy being used is locally built with openblas v0.3.29, but during a test I am seeing error:
ValueError: Eigenvalues did not converge
I am unable to share ...
0
votes
1
answer
146
views
Numba cfunc factory with numpy arrays
I want to have a factory method that calls a cfunc using numpy arrays. I am trying to pass the numpy arrays by using a ctype pointer.
Since my original code is rather complicated I have made a simple ...
3
votes
1
answer
122
views
Neural Network built from scratch using numpy isn't learning
I'm building a neural network from scratch using only Python and numpy, It's meant for classifying the MNIST data set, I got everything to work but the network isn't really learning, at epoch 0 it's ...
1
vote
1
answer
108
views
in kdb how do you save a 2d matrix into a binary format such that it can be read in python using numpy.fromfile or numpy.memmap?
My goal is to save a huge 2d matrix as binary in kdb q, so that my python code can import it as numpy matrix. I played with numpy.fromfile and numpy.memmap but I don't think kdb is writing it into a ...
2
votes
1
answer
86
views
np.random() is getting treated as module [closed]
I am trying to create a random numpy array using np.random() but for some reason, instead of taking it as a function, google collab is taking it as a module, I checked the documentation, but I would ...
0
votes
1
answer
68
views
softmax functions always outputing garbage numbers that don't add up to one [closed]
I am creating a simple NN from scratch that can classify MNIST digits, It only has 1 hidden layer:-
Loading the data:
import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import ...
3
votes
2
answers
167
views
How does one compare the contents of two lists of non-hashable objects in python without caring about order?
I have two lists of numpy arrays, and I want to check if the two lists have the same set of numpy arrays. If they have the same arrays in a different order, I still want it to return true.
The numpy ...
1
vote
1
answer
77
views
3D array to 2D then back again
I have the array
import numpy as np
a = np.array([[[11,12,13], [14,15,16]],
[[21,22,23], [24,25,26]],
[[31,32,33], [34,35,36]]])
# array([[[11, 12, 13],
# [14, 15, ...
1
vote
1
answer
81
views
Outer Product and Partial Trace of large Vectors exceeds Memory
I'm trying to find the outer product of a large complex-valued vector (of size 91204) to later on find the it's partial trace using np.einsum. However I get the following error:
numpy._core....
4
votes
3
answers
163
views
Get precise polygon coordinates for the outline of a solid shape, in order
I want to create a map (via the HTML "map" and "area" tags) with several unusually-shaped areas. The shapes are detailed enough that I don't want to write out all the coordinates ...
5
votes
1
answer
161
views
is there a way to assign a numpy array into pandas without a copy?
I'm in a performance critical field, where we store our results in pandas dataframes - issue is we are doing most of computations in numpy and then assigning to pd later - but this forces a copy on ...
0
votes
0
answers
55
views
When running the Tformer model training function, all label predictions evaluated are 0, while during the trainning phase,normal predictions are made
I tried to run the code in the Tformer paper, but when I evaluated the model in the test section of the train function, it predicted that pred_np (labels) would all be 0. My expected pred_np would be ...
1
vote
0
answers
48
views
How to send/recv from a specific index in mpi4py
Here is a translation of an C+MPI example in Python+Numpy+mpi4py. The goal of this example was to show that the message received is put in memory and that memory is in one dimension
from mpi4py import ...
5
votes
3
answers
101
views
I have an np array of a number single entry lists and I want to add 1 to each single entry lists
I have created the following array, called X:
array([[6.575],
[6.421],
[7.185],
[6.998],
[6.43 ],
[6.012],
[6.172],
[5.631],
[6.004],
[6.377],
[6.03 ]])
and I would like to create
array([[6.575, 1],
[...
1
vote
1
answer
131
views
Why does keyword argument 'weights' not work when calling NumPy histogram in Numba?
This Python 3.13.5 script with numpy 2.2.6 and numba 0.61.2:
import numpy as np, numba as nb
@nb.njit(fastmath=True)
def f(a, b):
return np.histogram(a, 10, weights=b)
a = np.random.randint(0, 256,...
3
votes
4
answers
343
views
How to clip to max an integer overflow using numpy or opencv
I have an array of the form a = np.array([1], dtype='uint8').
Now if I add 255 to a it will overflow and be np.array([0]).
Is there a built-in way to "clip" to value to 255?
NumPy has the ...
3
votes
1
answer
129
views
Why does the last bin in a NumPy histogram have an unusually high count?
This Python 3.12.7 script with numpy 2.2.4:
import numpy as np
a = np.random.randint(0, 256, (500, 500)).astype(np.uint8)
counts, bins = np.histogram(a, range(0, 255, 25))
print(np.column_stack((...
0
votes
0
answers
66
views
Building NN from scratch, why does my NN not memorize a small sample size of training data? It ends up being a class distribution
No matter which input I give it after training, it still spits the class distribution.. whereas if I just remove the hidden layer and use a single layer nn, it works much better.
I know the proper ...
2
votes
3
answers
78
views
An unusual resorting of a 3d numpy array
Consider the 3d numpy array arr:
[[[ 0, 9, 0, 2], [ 6, 8, 0, 2]],
[[ 0, 9, 1, 5], [ 4, 8, 1, 5]],
[[ 1, 0, 1, 2], [ 6, 6, 1, 2]],
[[ 0, 9, 1, 5], [ 5, 5, 1, 5]],
[[ 0, 9, ...
0
votes
2
answers
75
views
How to repeatedly sample so that the sampling result doesn't change?
I have an array from which I want to select a number of rows, so that the rows selected don't vary if I repeat the sampling process.
In my following code, this is not the case; each time I run the ...
4
votes
1
answer
75
views
How can I vectorize a function that returns eigenvalues and eigenvectors of a matrix in python?
I'm working with a function in Python that constructs a 4×4 matrix based on inputs (x1, y1, x2, y2), and computes its eigenvalues and eigenvectors using np.linalg.eigh.
Here is a simplified version of ...
2
votes
0
answers
77
views
numpy performance issues -- old macbook orders of magnitude faster than new thinkpad
I've written a tool in python that makes heavy use of numpy and pandas. Surprisingly, the tool is much slower on a brand new, high-end thinkpad as opposed to a ten year old Mac.
To illustrate the ...
3
votes
2
answers
146
views
Efficiently compute all multi-dimensional traces for all offsets and store in matrix
I have a $N\times N \times N$ array $a$ and would like to implement the formula $$ b_{ij} = \sum_k a_{i+k,j+k,k} $$ efficiently.
Right now, I'm doing this via
b = np.zeros((N, N))
for i in range(N):
...