JUEL rounding issues
Locked
JUEL rounding issues
 
|
Out of curiosity today I ran a couple of simple tests using JUEL's
numeric expressions and the results are a bit worrying, here's some example code: <set field="value1" value="0.0001" type="BigDecimal"/> <set field="value2" value="10" type="BigDecimal"/> <set field="resultValue" value="${value1 / value2}" type="BigDecimal"/> <log level="always" message="${resultValue}"></log> and here are some example results: 0.0001 / 10 = 0 10 / 3 = 3 10.89879 + 6.8978336 = 17.797 10.89879 * 6.8978336 = 75.178 10 - 2.5001 = 7.5 I don't have time right this minute to fix it so just reporting here so people are aware. Adrian, I'm not asking you to fix it but if you have any pointers or suggestions they'd be most welcome. Regards Scott HotWax Media http://www.hotwaxmedia.com |
smime.p7s (3K)
Re: JUEL rounding issues
|
|
Re: JUEL rounding issues
|
|
That is one solution but it would be nice to fix JUEL. By the way I
just had quick look at the calculate method and if you don't set a decimal-scale then 2 is used which seems a bit of a worry as well. Regards Scott On 10/09/2009, at 7:34 PM, Vince Clark wrote: > I don't know much about UEL but did run into a rounding issue. > > I chose to instead use <calculate> so I could control precision. > > Vince Clark > www.globalera.com > [hidden email] > (303) 493-6723 office > (303) 523-4843 cell > > > ----- Original Message ----- > From: "Scott Gray" <[hidden email]> > To: [hidden email] > Sent: Thursday, September 10, 2009 1:27:08 AM GMT -07:00 US/Canada > Mountain > Subject: JUEL rounding issues > > Out of curiosity today I ran a couple of simple tests using JUEL's > numeric expressions and the results are a bit worrying, here's some > example code: > <set field="value1" value="0.0001" type="BigDecimal"/> > <set field="value2" value="10" type="BigDecimal"/> > <set field="resultValue" value="${value1 / value2}" > type="BigDecimal"/> > <log level="always" message="${resultValue}"></log> > > and here are some example results: > 0.0001 / 10 = 0 > 10 / 3 = 3 > 10.89879 + 6.8978336 = 17.797 > 10.89879 * 6.8978336 = 75.178 > 10 - 2.5001 = 7.5 > > I don't have time right this minute to fix it so just reporting here > so people are aware. Adrian, I'm not asking you to fix it but if you > have any pointers or suggestions they'd be most welcome. > > Regards > Scott > > HotWax Media > http://www.hotwaxmedia.com > |
Re: JUEL rounding issues
|
|
Re: JUEL rounding issues
|
|
I would prefer it if no rounding occurs anywhere, there's really only
two places it needs to occur: 1. For division (even then only when the result would cause infinite precision e.g. 10 / 3) 2. When you're ready to arrive at the final result in a series of calculations Regards Scott On 10/09/2009, at 7:47 PM, Vince Clark wrote: > Yes the default is too small if precision is an issue. But at least > you have control over it. > > > > Vince Clark > www.globalera.com > [hidden email] > (303) 493-6723 office > (303) 523-4843 cell > > > ----- Original Message ----- > From: "Scott Gray" <[hidden email]> > To: [hidden email] > Sent: Thursday, September 10, 2009 1:41:58 AM GMT -07:00 US/Canada > Mountain > Subject: Re: JUEL rounding issues > > That is one solution but it would be nice to fix JUEL. By the way I > just had quick look at the calculate method and if you don't set a > decimal-scale then 2 is used which seems a bit of a worry as well. > > Regards > Scott > > On 10/09/2009, at 7:34 PM, Vince Clark wrote: > >> I don't know much about UEL but did run into a rounding issue. >> >> I chose to instead use <calculate> so I could control precision. >> >> Vince Clark >> www.globalera.com >> [hidden email] >> (303) 493-6723 office >> (303) 523-4843 cell >> >> >> ----- Original Message ----- >> From: "Scott Gray" <[hidden email]> >> To: [hidden email] >> Sent: Thursday, September 10, 2009 1:27:08 AM GMT -07:00 US/Canada >> Mountain >> Subject: JUEL rounding issues >> >> Out of curiosity today I ran a couple of simple tests using JUEL's >> numeric expressions and the results are a bit worrying, here's some >> example code: >> <set field="value1" value="0.0001" type="BigDecimal"/> >> <set field="value2" value="10" type="BigDecimal"/> >> <set field="resultValue" value="${value1 / value2}" >> type="BigDecimal"/> >> <log level="always" message="${resultValue}"></log> >> >> and here are some example results: >> 0.0001 / 10 = 0 >> 10 / 3 = 3 >> 10.89879 + 6.8978336 = 17.797 >> 10.89879 * 6.8978336 = 75.178 >> 10 - 2.5001 = 7.5 >> >> I don't have time right this minute to fix it so just reporting here >> so people are aware. Adrian, I'm not asking you to fix it but if you >> have any pointers or suggestions they'd be most welcome. >> >> Regards >> Scott >> >> HotWax Media >> http://www.hotwaxmedia.com >> > |
Re: JUEL rounding issues
Administrator
|
Re: JUEL rounding issues
|
|
Re: JUEL rounding issues
|
|
Crazy numbers are only a problem for final values though right? Take
a basic tax calc for example: quantity * price = interim value (no rounding) interim value * tax rate = final value (rounded 3dp, displayed to user and stored in db) Regards Scott On 10/09/2009, at 9:35 PM, David E Jones wrote: > > There is one other very general case I can think of: When you do a > floating point multiply or divide you can get some pretty crazy > numbers if you don't round. The solution is to do fixed point math, > but even then for currency you can do a multiplication and get more > digits that the 2 normally desired (unless you are doing certain > things like tax calculations where you want 3 digits for the interim > values and round to 2 digits after the interim values are added > together). > > -David > > > On Sep 10, 2009, at 1:56 AM, Scott Gray wrote: > >> I would prefer it if no rounding occurs anywhere, there's really >> only two places it needs to occur: >> 1. For division (even then only when the result would cause >> infinite precision e.g. 10 / 3) >> 2. When you're ready to arrive at the final result in a series of >> calculations >> >> Regards >> Scott >> >> On 10/09/2009, at 7:47 PM, Vince Clark wrote: >> >>> Yes the default is too small if precision is an issue. But at >>> least you have control over it. >>> >>> >>> >>> Vince Clark >>> www.globalera.com >>> [hidden email] >>> (303) 493-6723 office >>> (303) 523-4843 cell >>> >>> >>> ----- Original Message ----- >>> From: "Scott Gray" <[hidden email]> >>> To: [hidden email] >>> Sent: Thursday, September 10, 2009 1:41:58 AM GMT -07:00 US/Canada >>> Mountain >>> Subject: Re: JUEL rounding issues >>> >>> That is one solution but it would be nice to fix JUEL. By the way I >>> just had quick look at the calculate method and if you don't set a >>> decimal-scale then 2 is used which seems a bit of a worry as well. >>> >>> Regards >>> Scott >>> >>> On 10/09/2009, at 7:34 PM, Vince Clark wrote: >>> >>>> I don't know much about UEL but did run into a rounding issue. >>>> >>>> I chose to instead use <calculate> so I could control precision. >>>> >>>> Vince Clark >>>> www.globalera.com >>>> [hidden email] >>>> (303) 493-6723 office >>>> (303) 523-4843 cell >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Scott Gray" <[hidden email]> >>>> To: [hidden email] >>>> Sent: Thursday, September 10, 2009 1:27:08 AM GMT -07:00 US/Canada >>>> Mountain >>>> Subject: JUEL rounding issues >>>> >>>> Out of curiosity today I ran a couple of simple tests using JUEL's >>>> numeric expressions and the results are a bit worrying, here's some >>>> example code: >>>> <set field="value1" value="0.0001" type="BigDecimal"/> >>>> <set field="value2" value="10" type="BigDecimal"/> >>>> <set field="resultValue" value="${value1 / value2}" >>>> type="BigDecimal"/> >>>> <log level="always" message="${resultValue}"></log> >>>> >>>> and here are some example results: >>>> 0.0001 / 10 = 0 >>>> 10 / 3 = 3 >>>> 10.89879 + 6.8978336 = 17.797 >>>> 10.89879 * 6.8978336 = 75.178 >>>> 10 - 2.5001 = 7.5 >>>> >>>> I don't have time right this minute to fix it so just reporting >>>> here >>>> so people are aware. Adrian, I'm not asking you to fix it but if >>>> you >>>> have any pointers or suggestions they'd be most welcome. >>>> >>>> Regards >>>> Scott >>>> >>>> HotWax Media >>>> http://www.hotwaxmedia.com >>>> >>> >> > |
Re: JUEL rounding issues
|
|
Re: JUEL rounding issues
|
|
| Free forum by Nabble | Edit this page |