0

say a = 580991.3636

When I b = np.float32(a)

b 580991.4

I'm after b 580991.3636

I'm hoping this is an easy one, any ideas? I'm also assuming it is my misunderstanding of floats causing this. I know python is dynamic when it comes to dtypes, but I need it to be in float32 for a specific library (python-pcl).

1
  • 2
    The closest float to 580991.3636 has exact value 580991.375 which would print as 580991.4 Commented Nov 25, 2018 at 16:47

1 Answer 1

3

Float32 can only offer between 6 and 9 digits of precision.

Float32 works by storing the sign in 1 bit, the exponent in 8 bits and the value in 23 bits (also called the fraction). So you can represent a wide range of real numbers at the cost of precision.

If you need it to be more precise it has to be a double or use the python's library decimal.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.