Python: How to keep variable memory location across assignment?

Question

I'm still pretty new to Python, so forgive me this question if it's stupid. I could not find an answer through Google...

I am using PyFFTW in my code, which has a planning stage, where you pass it the two variables (source/destination), that your transforming from/to. It will then, when you call the FFT operate on the exact memory space, where those variables were located during the planning stage. Thus any operations done on the variables will need to be done, so that the location in memory of these two variables do not change. I found the operators *=, +=, etc., which do this for standard mathematical operators. However in my program I need to apply a function to the variable, which should return it to the same memory location. How to do this?!

I initially used slicing in the following way:

a[:] = func(a)[:]

However I just realized, that this is extremely slow (my code is about 10% slower). So does anybody know how to go about this?

Any help is very appreciated. Thanks in advance!

In short, you have absolutely no power over memory details in Python. Even slice assignment can easily trigger a re-allocation, which moves the items. Some implementations might stop the world and compact the whole heap just for the heck of it in the middle of your loop. — user395760
– user395760, Commented Mar 10, 2011 at 16:52
Hi. I am very surprised by this notion as that would render the idea of the way PyFFTW works completely useless, would it not?! — packoman
– packoman, Commented Mar 10, 2011 at 18:48
Yes, that's why I was to surprised hearing about it. You're propably fine if you stick to CPython (since PyFFTW is a C library, it's limited to that anyway) and don't resize the list. It's only more fancy garbage collectors and growing the list beyond the already allocated size (most data structures feature a bit of over-allocation, so you can a few items before re-allocating) that can cause nasty nasty problems (from segfaults over silently changing unused memory to silent data corruption). — user395760
– user395760, Commented Mar 10, 2011 at 18:52
Ok. That's pretty disappointing, because I would have to look for a completely different way of doing this, as my algorithm is doing literally millions of FFTs and inverse FFTs... the whole algorithm is based around this! — packoman
– packoman, Commented Mar 10, 2011 at 18:55
A list (as well as every other collections) only allocates so much memory. When you add more items, you need to reallocate (i.e. get a new, completely unrelated chunk of memory from the OS and copy your stuff there) because you can't resize an existing memory chunk at your whim. See the source. The same problem exist in every other language. But in C you know when you re-allocate because nobody does memory management for you. You can propably still do what you want, you just need to make a list of appropriate size beforehand. — user395760
– user395760, Commented Mar 10, 2011 at 19:00

John Flatness · Accepted Answer · 2011-03-11 02:05:57Z

1

Your variable is a mutable type, so your function can just operate on it directly.

You still won't be able to use functions and operators that are going to create copies and/or make new assignments (the same ones you couldn't use already), but direct mutations of the argument of your function will be visible outside the function.

edited Mar 11, 2011 at 2:05

answered Mar 10, 2011 at 16:50

John Flatness

34.1k5 gold badges81 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

packoman Over a year ago

Thats a very good point. I hadn't considered that - I guess because come from a MATLAB background. I will try that as I guess it also fits object oriented programming much better... up to now I alway returned 'self' from my methods, which I suppose is not very pretty style-wise.

packoman Over a year ago

Hi. I just realized I can't really do, what you suggested, because in my code I use the functions fftshift/ifftshift from Numpy, so I would have something like: a=fftshift(a) Now I don't want mess with the inner workings of fftshift, so I can't do direct mutation. So do you know of any way of doing the assignment without changing the memory location of 'a'? I'd be thankful!

JungJik Lee · Accepted Answer · 2011-03-11 04:50:00Z

1

how about using local value and referring it to the global value. I guess it could make faster...

global a
a = []
def test(): 
    global a
    b = [1,2,3,4]
    a = b
....

um.. I also would have to test it with cProfiler

answered Mar 11, 2011 at 4:50

JungJik Lee

1488 bronze badges

Comments

Wai Yip Tung · Accepted Answer · 2011-03-10 18:16:46Z

0

Is this something you are trying to do?

def time10(a):
  """ multiple the elements by 10 in-place """
  for i in xrange(len(a)):
    a[i] *= 10
  # returning a is optional. I think it maybe useful for use to chain the operation
  return a

>>> a = range(10)
>>> time10(a)
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
>>> a
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
>>> b = range(10)
>>> time10(time10(b))
[0, 100, 200, 300, 400, 500, 600, 700, 800, 900]
>>> b
[0, 100, 200, 300, 400, 500, 600, 700, 800, 900]
>>>

Your original code copy the array after it returns. It is usually not a useful practice and contribute to your slower runtime.

answered Mar 10, 2011 at 18:16

Wai Yip Tung

18.9k10 gold badges46 silver badges49 bronze badges

1 Comment

packoman Over a year ago

Hi, thanks for the reply. Actually what I am trying to do is related, yes. If I would do a*=b instead of a=a*b it gives me the same result, but a*=b will not change the memory location of 'a' whereas the latter will. So the question is, how I would do something like a=ifftshift(a) without changing the memory location of 'a'. 'ifftshift' is a built-in function I don't want to reimplement so direct mutations as zerocrates suggested will not work. Any ideas?

packoman · Accepted Answer · 2013-12-31 13:31:57Z

0

So in the end I could not find a satisfying solution to the problem. I initially ended up using the solution proposed by delnan above

a[:] = func(a)[:]

and preallocating the arrays of a certain size. Sorry for adding this as my answer, as I couldn't figure out how to/if it's possible accept delnans comments as the answer...

edited Dec 31, 2013 at 13:31

answered Dec 31, 2013 at 13:22

packoman

1,2921 gold badge20 silver badges40 bronze badges

Collectives™ on Stack Overflow

Python: How to keep variable memory location across assignment?

4 Answers 4

2 Comments

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related