373 questions
-2
votes
0
answers
279
views
Rounding madness - No matter the code I keep on getting rounding issues. I don't know what else to do
I got stuck on an exercise. I really need your help guys!
Exercise:
Instructions
Create 3 functions which work like the .filter, .map and .reduce array methods, but for the entries in the grocery ...
- 17
0
votes
0
answers
49
views
Is there a known issue with Spark version 3 behaving differently and inconsistent to Spark version 2 when rounding decimal points?
My project is having a problem with decimal rounding errors in Spark v3 compared to Spark v2, where rounding to second decimal place works fine in Spark v2 but Spark v3 rounds the values up or down (...
0
votes
2
answers
64
views
Why isn't the signif function working for Banker's even rounding for 0.6745? [duplicate]
For this example, I am working with the following data:
Z_Value <- c(-0.6745, 0.6745)
I'm using the following function and want to use 3 sig figs. (If I have results that are 999 or higher, I want ...
- 59
0
votes
2
answers
66
views
Round to n or less decimal places [duplicate]
I'm trying to round the output of some calculations to 3 or less decimal places, so they can stored compactly in a comma-delimited text file. That is 2 should be stored as 2 not 2.000, 3.14 should be ...
- 1,132
0
votes
1
answer
75
views
How do I fix the floating point problem in this case? [duplicate]
I'm trying to make a code for a simple calculator for a programming problem, and I can't figure out how the rounding works in this case.
1 + 0 + 0.0005 returns 1.000 but 2 + 1 + 0.0005 returns 3.001.
...
- 1
1
vote
2
answers
146
views
Example of the utility of the `G9` format specifier in C# for round-tripping floating-point numbers?
I'm trying to understand the practical utility of the G9 format specifier in C# when dealing with round-tripping floating-point numbers.
The book C#12 in a nutshell mentions that G9 is helpful to ...
3
votes
2
answers
206
views
Truncating floats to int on x86 vs x86_64 has different rounding error
I stumbled upon a failing unit-test, that involves converting doubles to integers.
The actual number that is converted is 1.234 * 1000., and the code basically boils down to:
#include <iostream>
...
- 32.2k
2
votes
1
answer
116
views
Collision detection for concave polygons with SAT, round/off error optimization
Basically I want to create my own tools for developing a 2D top-down game and when it comes to collision detection I'm trying to implement the SAT method, but somehow my calculations seem to be way ...
1
vote
1
answer
291
views
HLSL: Force order of floating point operations to prevent rounding error
When writing shaders in HLSL, sometimes you can run into very large floating point numbers, and with large floating point numbers, they struggle to maintain precision. When I wrote the blow code, I ...
- 1,021
2
votes
1
answer
336
views
Using MTLPixelFormat.rgba16Float results in random round-off errors
I'm using a Metal compute pipeline to render into a CAMetalLayer's drawable and keep running into problems with non-integer pixel formats resulting in a slightly noisy image, due to some apparent ...
- 152
0
votes
2
answers
75
views
why does the default round function in python fix double rounding errors?
There's something wrong with my understanding of either the round function in python or how doubles are represented in python. I find a very perplexing behavior when rounding numbers that have a ...
- 101
0
votes
3
answers
167
views
MariaDB inconsistent comparison results between DOUBLE(10,2) and string
I'm searching a table for an exact match on an amount column DOUBLE(10,2). Due to the database driver I'm using, I have to bind my amount as a string parameter. In general this works well, hoever this ...
- 29
-1
votes
1
answer
75
views
Can't use double, can't use int, what do I use to stop the rounding? (Java) [duplicate]
I am incredibly new to java and this is the description for a lab I currently have:
Statistics are often calculated with varying amounts of input data. Write a program that takes any number of non-...
0
votes
1
answer
73
views
R: Allocating N discrete elements between population cohorts by probability
I want to model demographic changes within a population, using historical data going from 1986 until the present. My data covers a population divided into age cohorts of a single year: 0,1,2 ... 106+.
...
- 59
-1
votes
1
answer
90
views
C++ rounding errors [closed]
Given an array of integers, calculate the ratios of its elements that are positive, negative, and zero. Print the decimal value of each fraction on a new line with places after the decimal.
Note: ...
-1
votes
1
answer
118
views
Swift NumberFormatter gives incorrect values with lots of extra zeros [duplicate]
I found that for some strings the NumberFormatter produces incorrect results, as in the example below.
let input = "579.988"
let formatter = NumberFormatter()
formatter....
1
vote
2
answers
76
views
Precision Error while converting dollars/gallon to dollars/liter and back
The following code has a precision error but I'm at a loss as to how to resolve it. Given an initial value set in dollar/gal of $3.95, when change the ChoiceBox to "dollars/liter" I get $1....
- 21
0
votes
0
answers
26
views
Alpha blending on int colors exceeds byte ranges
I have implemented an algorithm for alpha blending. It worked for some time, but soon I have noticed that I'm receiving some weird results in some specific cases.
I have investigated and it turned out ...
- 26.2k
2
votes
1
answer
372
views
How to handle rounding errors in solidity code? Avoid precision loss or keep track of precision losses?
How to handle rounding errors in solidity code? Avoid precision loss or keep track of precision losses.
Example, if i have some calculated(includes a division) factors stored in a variable when a user ...
- 31
0
votes
1
answer
400
views
Phone number formatting with a dot after the country code
My domain registrar asks for a phone number in the following format: +358.401234567 (where 358 is the country code, 40 is the area code and the rest is the local phone number).
To me personally this ...
- 578
2
votes
1
answer
144
views
When does using floating point operations to calculate ceiling-integer-log2 fail?
I'm curious what the first input is which differentiates these two functions:
from math import *
def ilog2_ceil_alt(i: int) -> int:
# DO NOT USE THIS FUNCTION IT'S WRONG
return ceil(log2(i)...
- 1,117
-1
votes
2
answers
444
views
C# Math.Round (1.785m, 2) not rounded correctly [closed]
I’m experiencing a small issue when I’m trying to round the number 1.785m to two digits. The number can probably not be represented correctly using the decimal data type, since it’s still a floating ...
- 529
0
votes
0
answers
75
views
I am having rounding off problem while working with certain data in Excel sheet
I have thousands of data in an sheet which are actually head wise monthly revenue generated by my organisation.
The data are all numeric and in unit figure. I have to generate report in thousand fig ...
1
vote
2
answers
187
views
Powershell Midpoint Rounding
I am building an API and the system that I am using just uses midpoint rounding in PowerShell. I've come to find out that certain numbers still disobey this rule and my research as to why has proven ...
- 11
2
votes
2
answers
200
views
How do I round the numbers in a df column correctly in Python?
The following is my df: https://www.dropbox.com/s/nbez3esbo8fedmf/aapl.csv?dl=0
date ticker open high low close adjClose
2019-07-08 AAPL 50.2025 50.35 49.6025 50.005 48.516
...
- 775
5
votes
0
answers
176
views
What is the causing the discrepancies in rounding error between different machines?
Consider the following code
func main() {
var (
negativeTwoThirds = -2.0 / 3.0
negativeSix = -6.0
negativeThree = -3.0
)
for _, v := range []float64{...
- 18.4k
0
votes
0
answers
23
views
Python rounding grades [duplicate]
Trying to round student grades to nearest multiple of five. If grade is two or less closer to the round then that grade gets used. What is wrong with my current code
Grade = input(23)
Round_grade = (((...
- 1
5
votes
1
answer
219
views
Python : Rounding errors between C-contiguous and F-contiguous arrays for matrix multiplication
Problem :
I stumbled across a problem that has deep (and very impactful) consequences in the industry, and does not seem to be documented anywhere :
It seems that in python, Matrix Multiplication (...
- 1,497
1
vote
1
answer
766
views
How can I implement Pytorch Linear layer forward calculation precisely?
I am trying to implement F.linear function from scratch using Pytorch, but it returns a slightly different value from Pytorch library function. In my case, the difference is about 1e-5 to 1e-9. 1e-5 ...
- 101
0
votes
1
answer
91
views
Rounding error in density functions of R-programming language
Using R I am trying to compute the likelihoods of a vector of values. Some of these values are way off into the tails of the distributions. Rounding error appears to be causing the results to be ...
- 3
0
votes
0
answers
63
views
How can I solve a rounding error using update_all in ruby on rails?
Assume there is a table named somethings with a column value of type float.
The schema is shown below.
# == Schema Information
#
# Table name: somethings
#
# id :bigint not null, primary key
...
-1
votes
1
answer
75
views
how to output exact value when i use "floating point" type variable
#include <stdio.h>
int main(void) {
float interest_rate, amount_of_loan, monthly_payment, remain_payment;
printf("Enter amount of loan :");
scanf("%f", &...
- 21
0
votes
1
answer
342
views
Minimizing rounding errors in a sum of floats
Column A contains floats that are rounded to two decimals. Column B contains float multipliers that should sum to ~1.00. Cn is calculated as ATOTAL * Bn - An. Column C contains floats rounded to two ...
- 1
1
vote
4
answers
198
views
Float rounding error in time-critical C++ loop, looking for an efficient solution
As a premise, I am aware that this problem has been addressed already, but never in this specific scenario, from what I could find searching.
In a time-critical piece of code, I have a loop where a ...
- 325
3
votes
2
answers
992
views
Floating point multiplication results in infinity even though rounding to nearest
Consider the following C++ code:
#include <fenv.h>
#include <iostream>
using namespace std;
int main(){
fesetround(FE_TONEAREST);
double a = 0x1.efc7f0001p+376;
double b = -...
- 389
3
votes
0
answers
69
views
C++ rounding double with maximum possible precision [duplicate]
Surely rounding is (one of) the mostly asked and resolved questions. With any pre-defined precision after decimal point e.g. 10 ^ N, one could easily do
double round(double input) {
static ...
- 3,167
1
vote
0
answers
106
views
Javascript Integer Point [X,Y] is in or on Polygon [[X,Y],[X,Y],...]
When looking around for a way to determine if a point [X,Y] is within or on a polygon [[X,Y],[X,Y],...] I am running into the issue where the upper bounds (maxX and maxY) are being ignored.
Example:
...
- 11
1
vote
1
answer
139
views
How is Half-even rounding off for number 6.325 to two-decimal places is 6.33 in JAVA. Can somebody explain this? [duplicate]
According to official java doc,
RoundingMode HALF_EVEN:
Rounding mode to round towards the nearest neighbor unless both neighbors are equidistant, in which case, round towards the even neighbor.
So ...
0
votes
1
answer
58
views
Almost correct output, but not quite right. Math help in Python
I am trying to match an expected output of "13031.157014219536" exactly, and I have attempted 3 times to get the value with different methods, detailed below, and come extremely close to the ...
0
votes
1
answer
50
views
conditional rounding function in R negated by dataframe
I am trying to produce a table for a manuscript. For this table, there are NA values, large values above 1,000, and values that are below 10.
I'd like the numbers less than 10 to be rounded to 1 ...
- 17
5
votes
1
answer
1k
views
Classification of x86 instructions according to floating point rounding mode sensitivity?
I'm implementing randomized rounding mode for evaluating numerical stability of floating point (FP) computations in C99 code. This is the idea of this computational chemistry work, but updated to ...
0
votes
0
answers
268
views
Rounding in R with print statement
Q4 <-flights %>%
filter(origin=="JFK") %>%
group_by(month,day) %>%
summarise(mean=round(mean(distance),digits=2)) %>%
arrange(desc(mean)) %>%
head(5)
print(Q4)...
0
votes
2
answers
116
views
Is there a double number better than 1.0 for representing the real number 1 + 1e-16?
It is well known floating-point computation is inexact. For example:
In [64]: 1+1e-16
Out[64]: 1.0
In this case, I would like to know if there is a double number that is better than 1.0 for ...
- 10.4k
5
votes
1
answer
579
views
NLopt minimum working example fails with "RoundoffLimited"
I was experimenting with NLopt and created the following minimum working example, which consistently fails with RoundoffLimited: NLopt roundoff-limited:
import numpy as np
import nlopt
dim = 1
def ...
- 51
0
votes
0
answers
56
views
Is there a way to know when will a floating-point computation be accurate/inaccurate without actually doing the computation?
I understand that floating-point computation is inherently inaccurate due to rounding errors. But how can I understand that it is sometimes accurate, sometimes inaccurate, like the following:
In [58]: ...
- 10.4k
-2
votes
2
answers
696
views
How to round float numbers in Python?
I know there are different questions like this; and I searched my question on stackoverflow and many other webs, and basically the easiest way to do this is using the round/2 function, but the round ...
- 3
1
vote
1
answer
344
views
How many rows will SAS check to determine if the input data contains decimal?
Recently I found a problem when uploading excel file to snow flake using SAS.
My excel include 2 columns 1400 rows, in the first column only 11 rows have 2 decimal place, in the second column, most of ...
- 267
3
votes
0
answers
50
views
Rounding in java shows different result [duplicate]
I stumbled across following scenario where how rounding works does not make sense to me.
new BigDecimal(9999.005).setScale(2, RoundingMode.HALF_UP) -- 9999.0
new BigDecimal(999.005).setScale(2, ...
- 31
0
votes
1
answer
337
views
Rounding error when converting Double to Real in fortran
i'm trying to convert a double into a real for some operations but i noticed an error.
i have a macro for reading purposes, short is 4 bytes, and long is 8 bytes.
this is the execution code :
print*, ...
- 67
3
votes
1
answer
90
views
Rust: Strange state-based rounding behaviour on f32
When computing the dot-product of two nalgebra::Vector3 structs using specific values, I get the following behaviour (link to playground):
use nalgebra::{Point3, Vector3}; // 0.31.0
fn dot(v1: ...
- 129