How to use io to generate in memory data streams as file like objects?

Question

I want to generate a in-memory (temp file) data stream in Python. One thread is filling the stream with data, and a other one consumes it.

After checking the io - Core tools for working with streams , it seems to me that the io module is the best choice for it.

So I made a simple example:

#!/usr/local/bin/python3
# encoding: utf-8

import io

if __name__ == '__main__':
    a = io.BytesIO()
    a.write("hello".encode())
    txt = a.read(100)
    txt = txt.decode("utf-8")
    print(txt)

My example does not work. "hello" is not written to a and can not be read after. So where is my error? How do I have to alter my code to get a file like object in memory?

Stefan Jaritz · Accepted Answer · 2018-09-10 09:20:23Z

4

@dhilmathy and @ShadowRanger mentioned that io.BytesIO() do not have separate read and write pointer.

I overcome this problem with creating a simple class that implements a read pointer and remembers the amount of bytes written. When the amount of read bytes is equal the amount of written bytes the file is shrink to save memory.

My solution so far:

#!/usr/local/bin/python3
# encoding: utf-8

import io

class memoryStreamIO(io.BytesIO):
    """
    memoryStreamIO

    a in memory file like stream object 
    """

    def __init__(self):
        super().__init__()
        self._wIndex = 0
        self._rIndex = 0
        self._mutex = threading.Lock()

    def write(self, d : bytearray):
        self._mutex.acquire()
        r = super().write(d)
        self._wIndex += len(d)
        self._mutex.release()
        return r

    def read(self, n : int):
        self._mutex.acquire()
        super().seek(self._rIndex)
        r = super().read(n)
        self._rIndex += len(r)
        # now we are checking if we can
        if self._rIndex == self._wIndex:
            super().truncate(0)
            super().seek(0)
            self._rIndex = 0
            self._wIndex = 0
        self._mutex.release()
        return r

    def seek(self, n):
        self._mutex.acquire()
        self._rIndex = n
        r = super().seek(n)
        self._mutex.release()
        return r


if __name__ == '__main__':
    a = streamIO()

    a.write("hello".encode())
    txt = (a.read(100)).decode()
    print(txt)

    a.write("abc".encode())
    txt = (a.read(100)).decode()
    print(txt)

edited Sep 10, 2018 at 9:20

answered Sep 5, 2018 at 12:02

Stefan Jaritz

2,4218 gold badges46 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user202729 Over a year ago

I don't think this is how the file interface is supposed to work. What you want is a pipe --- two ends, one end readable, the other end writable.

dhilmathy · Accepted Answer · 2018-09-05 10:51:08Z

3

Actually it's written; but reading is the problem. You should be referring to class io.BytesIO. You can get the value using getvalue(). Like,

import io

a = io.BytesIO()
a.write("hello".encode())
txt = a.getvalue()
txt = txt.decode("utf-8")
print(txt)

answered Sep 5, 2018 at 10:51

dhilmathy

2,8882 gold badges23 silver badges33 bronze badges

3 Comments

Stefan Jaritz Over a year ago

merci - I like to handle my io instance like a file - only write and read. Seems that I have to use a other io class

ShadowRanger Over a year ago

@StefanJaritz: If you want to use it in a file-like way, a.seek(0) before the a.read(100) would also work; the write moves the file pointer, so you need to move it back to the beginning of the "file" to have read pick up what you just wrote.

Stefan Jaritz Over a year ago

is there a io class having separate read & write file pointers? Like a fifo?

Collectives™ on Stack Overflow

How to use io to generate in memory data streams as file like objects?

2 Answers 2

1 Comment

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related