Python multiprocessing shared memory error on close

Question

I am using Python multiprocessing shared memory.

I see the following error when I try to close a SharedMemory object: BufferError: memoryview has 1 exported buffer

Can someone please let me know what this error means?

Here is how to reproduce: I am using Pyarrow

import pyarrow as pa
import pandas as pd
import numpy as np
from multiprocessing import shared_memory

a = pd.DataFrame(np.random.normal(size=(1000,1000)))
batch = pa.RecordBatch.from_pandas(a)

mock_sink = pa.MockOutputStream()
with pa.RecordBatchStreamWriter(mock_sink, batch.schema) as stream_writer:
    stream_writer.write_batch(batch)

data_size = mock_sink.size()
print(data_size)
shm_a = shared_memory.SharedMemory(create=True, size=data_size)

buf = pa.py_buffer(shm_a.buf)
stream = pa.FixedSizeBufferWriter(buf)
with pa.RecordBatchStreamWriter(stream, batch.schema) as stream_writer:
    stream_writer.write_batch(batch)
print(shm_a.name)
shm_a.close()

It's very hard to say anything without any code / minimal working example - please provide some more details. — rammelmueller
– rammelmueller, Commented Apr 1, 2022 at 5:52
I tried to run this, but data_size is undefined - please fix this. — rammelmueller
– rammelmueller, Commented Apr 1, 2022 at 12:44
I just fixed it. I have no idea why that line failed to copy — bumpbump
– bumpbump, Commented Apr 1, 2022 at 15:16

makiko_fly · Accepted Answer · 2022-05-16 06:57:50Z

2

I have encountered the same problem. After some digging, I found some clue:

From the code above, you can see that once we create a pa.py_buffer object from share memory's buf, shm.buf can't be released. After we delete that py_buffer object, it can be released successfully without throwing the BufferError exception.

So the solution I came up to solve my problem is as follows:

This may not be a perfect solution. If anybody comes up with better ones, please post it.

Also note that the BufferError still seem to arise when we read the shared memory from multiple processes. I am still working on it.

edited May 16, 2022 at 6:57

answered May 15, 2022 at 8:34

makiko_fly

6066 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

bumpbump Over a year ago

I don't believe it solved my problem...

makiko_fly Over a year ago

sorry, I made a mistake, just edited the answer. The solution is temporary, if you found better ones, please post it, thanks.

hst01 · Accepted Answer · 2023-06-15 11:10:06Z

0

That resolved my problem but now, that I'm trying to put dataframes of pyarrow types into a shared memory I'm getting this problem again. If you learn something more about this please visit my question here: BufferError: memoryview has 1 exported buffer trying to close a shared memory where I put a dataframe of pyarrow data type values

answered Jun 15, 2023 at 11:10

hst01

31 gold badge1 silver badge3 bronze badges

1 Comment

Community Over a year ago

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

Python multiprocessing shared memory error on close

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related