in kdb how do you save a 2d matrix into a binary format such that it can be read in python using numpy.fromfile or numpy.memmap?

Question

My goal is to save a huge 2d matrix as binary in kdb q, so that my python code can import it as numpy matrix. I played with numpy.fromfile and numpy.memmap but I don't think kdb is writing it into a binary format that numpy can understand. I also tried using the save / set / 0x0 sv commands in kdb but nothing seems to be working. Is there anyway to do this at all?

For example, in kdb:

m:(3 4)#12?1.0
`:/home/davidmoss/matrix set raze m

Then in python:

import numpy as np
matrix = np.fromfile('/home/davidmoss/matrix', dtype=np.float64)
matrix = np.memmap('/home/davidmoss/matrix', dtype='float64', mode='r', shape=(3,4))

both of these don't work, what am I missing?

np.save/load writes the array data buffer as a binary copy, with a preface block with shape and dtyoe info. fromfile expects just the data, loading it as a 1d array. You specify the dtype, and reshape as desired. — hpaulj
– hpaulj, Commented Jul 2 at 0:37

rianoc · Accepted Answer · 2025-07-01 21:48:13Z

set writes using kdb+ specific on disk format including a header so NumPy will not be able to read it easily.

You can use vs to get byte representation and 1: to save raw binary data to disk:

q)m:(3 4)#12?1.0
q)m
0.3927524 0.5170911 0.5159796 0.4066642
0.1780839 0.3017723 0.785033  0.5347096
0.7111716 0.411597  0.4931835 0.5785203
q)`:mybin 1:raze over 0x0 vs''m
`:mybin

When reading from NumPy you must use the > prefixed datatype to denote the contents as Big Endian:

>>> import numpy as np
>>> np.fromfile('mybin', dtype='>f8')
array([0.39275238, 0.51709112, 0.51597965, 0.40666419, 0.17808386,
       0.30177225, 0.78503301, 0.53470961, 0.71117165, 0.41159698,
       0.49318348, 0.57852028], dtype='>f8')

If you have access to PyKX you can save more directly in a .npy file:

q)\l pykx.q
q)m:(3 4)#12?1.0
q)np:.pykx.import`numpy
q)np[`:save][`myarray;.pykx.tonp m]

>>> import numpy as np
>>> matrix = np.load('myarray.npy', allow_pickle=True)
>>> matrix
array([array([0.39275238, 0.51709112, 0.51597965, 0.40666419]),
       array([0.17808386, 0.30177225, 0.78503301, 0.53470961]),
       array([0.71117165, 0.41159698, 0.49318348, 0.57852028])],
      dtype=object)

Collectives™ on Stack Overflow

in kdb how do you save a 2d matrix into a binary format such that it can be read in python using numpy.fromfile or numpy.memmap?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related