Entity Field Type Checking and Move BigDecimal versus Double Stuff

[David E Jones]

I added some stuff to make the warnings from entity field type checks  
a bit more prominent, and include a stack trace to make them easier to  
track down. I tried throwing exceptions instead of the warning log  
messages but that pretty much breaks everything, especially because we  
have lots of BigDecimal/Double problems...

On the BigDecimal versus Double stuff, one of the main type change  
problems comes from this old issue. To help resolve this I'd like to  
change the java-type of currency-amount and currency-precise to  
BigDecimal instead of Double.

It doesn't make sense to change the "floating-point" type's java-type  
to BigDecimal, because it really is a floating point like a Double.  
However, we also have many fields that need decimal precision but  
really should be fixed point and not floating point... but they really  
aren't "currency". For these I propose to add a "fixed-point" field  
type. This would have a BigDecimal java-type and in the database  
should have a type that is also fixed point (ie not DOUBLE or FLOAT),  
unless the database has no fixed point types (which is a big minus for  
those databases). Also, the floating-point type should have a Double  
java-type and a floating point SQL type.

One side effect to changing the java-type on these field types is that  
it changes services definitions that are based on entity definitions,  
which causes the service engine to throw exceptions for type checks...  
and it does throw exceptions instead of just printing warning  
messages, so it does break stuff when doing that.

This is a change that we can't switch back and forth on. Either all of  
these services take a Double typed parameters, or BigDecimal typed  
ones... and never some of each.

Because of this I'm actually tempted to do a branch and get the  
changes going. I'm paying with a few common processes with this in  
place right now.

Any thoughts related to this would be appreciated.

Checking types... what a can of worms!

-David

[Scott Gray]

+1 for making the changes and I'm happy to help with the mundane stuff.

Regards
Scott

2008/10/22 David E Jones <[hidden email]>:

>
> I added some stuff to make the warnings from entity field type checks a bit
> more prominent, and include a stack trace to make them easier to track down.
> I tried throwing exceptions instead of the warning log messages but that
> pretty much breaks everything, especially because we have lots of
> BigDecimal/Double problems...
>
> On the BigDecimal versus Double stuff, one of the main type change problems
> comes from this old issue. To help resolve this I'd like to change the
> java-type of currency-amount and currency-precise to BigDecimal instead of
> Double.
>
> It doesn't make sense to change the "floating-point" type's java-type to
> BigDecimal, because it really is a floating point like a Double. However, we
> also have many fields that need decimal precision but really should be fixed
> point and not floating point... but they really aren't "currency". For these
> I propose to add a "fixed-point" field type. This would have a BigDecimal
> java-type and in the database should have a type that is also fixed point
> (ie not DOUBLE or FLOAT), unless the database has no fixed point types
> (which is a big minus for those databases). Also, the floating-point type
> should have a Double java-type and a floating point SQL type.
>
> One side effect to changing the java-type on these field types is that it
> changes services definitions that are based on entity definitions, which
> causes the service engine to throw exceptions for type checks... and it does
> throw exceptions instead of just printing warning messages, so it does break
> stuff when doing that.
>
> This is a change that we can't switch back and forth on. Either all of these
> services take a Double typed parameters, or BigDecimal typed ones... and
> never some of each.
>
> Because of this I'm actually tempted to do a branch and get the changes
> going. I'm paying with a few common processes with this in place right now.
>
> Any thoughts related to this would be appreciated.
>
> Checking types... what a can of worms!
>
> -David
>
>
>

[David E Jones]

I've created a branch, now available at:

https://svn.apache.org/repos/asf/ofbiz/branches/typecheckcleanup200810

The branch split was revision 706867, so we can merge from the trunk  
at that rev going forward, and eventually merge back into the trunk  
once we're comfortable that enough things have been tested by running,  
perhaps using a bunch of the automated tests too (not sure how well  
those run in the trunk right now though, ie I haven't run them in  
quite some time!).

Anyway, the main initial process that I targeted was ecommerce  
browsing, add to cart, and checkout/order. Still need to test basic  
things like order fulfillment, purchasing and receiving, and tons of  
other stuff, but hopefully most other things won't be impacted as much.

Any and all help is appreciated!

-David

NOTE: I'm still checking out the branch and applying my patch with the  
work so far, so it'll be a few minutes before that's in.

On Oct 21, 2008, at 10:20 PM, Scott Gray wrote:

> +1 for making the changes and I'm happy to help with the mundane  
> stuff.
>
> Regards
> Scott
>
> 2008/10/22 David E Jones <[hidden email]>:
>>
>> I added some stuff to make the warnings from entity field type  
>> checks a bit
>> more prominent, and include a stack trace to make them easier to  
>> track down.
>> I tried throwing exceptions instead of the warning log messages but  
>> that
>> pretty much breaks everything, especially because we have lots of
>> BigDecimal/Double problems...
>>
>> On the BigDecimal versus Double stuff, one of the main type change  
>> problems
>> comes from this old issue. To help resolve this I'd like to change  
>> the
>> java-type of currency-amount and currency-precise to BigDecimal  
>> instead of
>> Double.
>>
>> It doesn't make sense to change the "floating-point" type's java-
>> type to
>> BigDecimal, because it really is a floating point like a Double.  
>> However, we
>> also have many fields that need decimal precision but really should  
>> be fixed
>> point and not floating point... but they really aren't "currency".  
>> For these
>> I propose to add a "fixed-point" field type. This would have a  
>> BigDecimal
>> java-type and in the database should have a type that is also fixed  
>> point
>> (ie not DOUBLE or FLOAT), unless the database has no fixed point  
>> types
>> (which is a big minus for those databases). Also, the floating-
>> point type
>> should have a Double java-type and a floating point SQL type.
>>
>> One side effect to changing the java-type on these field types is  
>> that it
>> changes services definitions that are based on entity definitions,  
>> which
>> causes the service engine to throw exceptions for type checks...  
>> and it does
>> throw exceptions instead of just printing warning messages, so it  
>> does break
>> stuff when doing that.
>>
>> This is a change that we can't switch back and forth on. Either all  
>> of these
>> services take a Double typed parameters, or BigDecimal typed  
>> ones... and
>> never some of each.
>>
>> Because of this I'm actually tempted to do a branch and get the  
>> changes
>> going. I'm paying with a few common processes with this in place  
>> right now.
>>
>> Any thoughts related to this would be appreciated.
>>
>> Checking types... what a can of worms!
>>
>> -David
>>
>>
>>

[Scott Gray]

Hi David, everyone

I thought I'd set about changing pretty much all the doubles in the
accounting component to BigDecimal and I've got a couple of questions:
1.  Do we need to follow the standard path we've taken in the past
where we deprecate any methods taking or returning doubles and add Bd
to the end of a new BigDecimal version?  I guess we do but it's going
to take a longer so I figure no harm in asking.

2.  Slightly OT but I can't see the point of the scaled and rounded
ZEROs in the java files, could someone explain the advantage of them
over BigDecimal.ZERO?

3.  Could someone give me an example of when to use
floating-point/double over fixed-point/BigDecimal?  I'm not 100% sure
of that one.

Thanks
Scott

2008/10/22 David E Jones <[hidden email]>:

>
> I've created a branch, now available at:
>
> https://svn.apache.org/repos/asf/ofbiz/branches/typecheckcleanup200810
>
> The branch split was revision 706867, so we can merge from the trunk at that
> rev going forward, and eventually merge back into the trunk once we're
> comfortable that enough things have been tested by running, perhaps using a
> bunch of the automated tests too (not sure how well those run in the trunk
> right now though, ie I haven't run them in quite some time!).
>
> Anyway, the main initial process that I targeted was ecommerce browsing, add
> to cart, and checkout/order. Still need to test basic things like order
> fulfillment, purchasing and receiving, and tons of other stuff, but
> hopefully most other things won't be impacted as much.
>
> Any and all help is appreciated!
>
> -David
>
> NOTE: I'm still checking out the branch and applying my patch with the work
> so far, so it'll be a few minutes before that's in.
>
> On Oct 21, 2008, at 10:20 PM, Scott Gray wrote:
>
>> +1 for making the changes and I'm happy to help with the mundane stuff.
>>
>> Regards
>> Scott
>>
>> 2008/10/22 David E Jones <[hidden email]>:
>>>
>>> I added some stuff to make the warnings from entity field type checks a
>>> bit
>>> more prominent, and include a stack trace to make them easier to track
>>> down.
>>> I tried throwing exceptions instead of the warning log messages but that
>>> pretty much breaks everything, especially because we have lots of
>>> BigDecimal/Double problems...
>>>
>>> On the BigDecimal versus Double stuff, one of the main type change
>>> problems
>>> comes from this old issue. To help resolve this I'd like to change the
>>> java-type of currency-amount and currency-precise to BigDecimal instead
>>> of
>>> Double.
>>>
>>> It doesn't make sense to change the "floating-point" type's java-type to
>>> BigDecimal, because it really is a floating point like a Double. However,
>>> we
>>> also have many fields that need decimal precision but really should be
>>> fixed
>>> point and not floating point... but they really aren't "currency". For
>>> these
>>> I propose to add a "fixed-point" field type. This would have a BigDecimal
>>> java-type and in the database should have a type that is also fixed point
>>> (ie not DOUBLE or FLOAT), unless the database has no fixed point types
>>> (which is a big minus for those databases). Also, the floating-point type
>>> should have a Double java-type and a floating point SQL type.
>>>
>>> One side effect to changing the java-type on these field types is that it
>>> changes services definitions that are based on entity definitions, which
>>> causes the service engine to throw exceptions for type checks... and it
>>> does
>>> throw exceptions instead of just printing warning messages, so it does
>>> break
>>> stuff when doing that.
>>>
>>> This is a change that we can't switch back and forth on. Either all of
>>> these
>>> services take a Double typed parameters, or BigDecimal typed ones... and
>>> never some of each.
>>>
>>> Because of this I'm actually tempted to do a branch and get the changes
>>> going. I'm paying with a few common processes with this in place right
>>> now.
>>>
>>> Any thoughts related to this would be appreciated.
>>>
>>> Checking types... what a can of worms!
>>>
>>> -David
>>>
>>>
>>>
>
>

[David E Jones]

On Oct 23, 2008, at 3:15 AM, Scott Gray wrote:

> Hi David, everyone
>
> I thought I'd set about changing pretty much all the doubles in the
> accounting component to BigDecimal and I've got a couple of questions:
> 1.  Do we need to follow the standard path we've taken in the past
> where we deprecate any methods taking or returning doubles and add Bd
> to the end of a new BigDecimal version?  I guess we do but it's going
> to take a longer so I figure no harm in asking.

I'm not sure where these got started, but they seem only really useful  
if you want a BigDecimal version of a function AND a Double version of  
it.

I'd say no, there is no reason to continue following this pattern. It  
may break backward compatibility in esoteric areas, but really this  
should be considered a bug and all code affected should be moved to  
use BigDecimal instead of Double anyway.

> 2.  Slightly OT but I can't see the point of the scaled and rounded
> ZEROs in the java files, could someone explain the advantage of them
> over BigDecimal.ZERO?

In older versions of Java there was no BigDecimal.ZERO, so this is  
probably from older code and should updated.

> 3.  Could someone give me an example of when to use
> floating-point/double over fixed-point/BigDecimal?  I'm not 100% sure
> of that one.

Since we're doing business applications most things should be fixed  
point. The problem with floating point is that certain decimal numbers  
are not represented accurately and so if used without rounding in a  
calculation you'll get weird results, usually something like a bunch  
of 9s or 1s at the end of a decimal part of the number. This is  
because a binary number representation for non-integer numbers  
basically look at 1/2 and 1/2 of 1/2 and 1/2 of 1/2 of 1/2, and not  
all fractions split up evenly like that, so even with a 64 bit  
floating point number when converted back into a base-10 number you  
get something weird.

So when to use a floating point number? When you are dealing with  
extreme numbers where precision beyond a certain number of significant  
digits doesn't matter so much. For financial things ALL digits are  
significant, so a number representation that may cause you to lose  
some is a BAD thing. For scientific stuff sometimes you need numbers  
that are on the order to 10E30 or 10E-30 or the like that don't work  
so well for fixed point things, but they only really care about, say,  
5 of the 30 or more possible significant digits.

-David

>
> 2008/10/22 David E Jones <[hidden email]>:
>>
>> I've created a branch, now available at:
>>
>> https://svn.apache.org/repos/asf/ofbiz/branches/ 
>> typecheckcleanup200810
>>
>> The branch split was revision 706867, so we can merge from the  
>> trunk at that
>> rev going forward, and eventually merge back into the trunk once  
>> we're
>> comfortable that enough things have been tested by running, perhaps  
>> using a
>> bunch of the automated tests too (not sure how well those run in  
>> the trunk
>> right now though, ie I haven't run them in quite some time!).
>>
>> Anyway, the main initial process that I targeted was ecommerce  
>> browsing, add
>> to cart, and checkout/order. Still need to test basic things like  
>> order
>> fulfillment, purchasing and receiving, and tons of other stuff, but
>> hopefully most other things won't be impacted as much.
>>
>> Any and all help is appreciated!
>>
>> -David
>>
>> NOTE: I'm still checking out the branch and applying my patch with  
>> the work
>> so far, so it'll be a few minutes before that's in.
>>
>> On Oct 21, 2008, at 10:20 PM, Scott Gray wrote:
>>
>>> +1 for making the changes and I'm happy to help with the mundane  
>>> stuff.
>>>
>>> Regards
>>> Scott
>>>
>>> 2008/10/22 David E Jones <[hidden email]>:
>>>>
>>>> I added some stuff to make the warnings from entity field type  
>>>> checks a
>>>> bit
>>>> more prominent, and include a stack trace to make them easier to  
>>>> track
>>>> down.
>>>> I tried throwing exceptions instead of the warning log messages  
>>>> but that
>>>> pretty much breaks everything, especially because we have lots of
>>>> BigDecimal/Double problems...
>>>>
>>>> On the BigDecimal versus Double stuff, one of the main type change
>>>> problems
>>>> comes from this old issue. To help resolve this I'd like to  
>>>> change the
>>>> java-type of currency-amount and currency-precise to BigDecimal  
>>>> instead
>>>> of
>>>> Double.
>>>>
>>>> It doesn't make sense to change the "floating-point" type's java-
>>>> type to
>>>> BigDecimal, because it really is a floating point like a Double.  
>>>> However,
>>>> we
>>>> also have many fields that need decimal precision but really  
>>>> should be
>>>> fixed
>>>> point and not floating point... but they really aren't  
>>>> "currency". For
>>>> these
>>>> I propose to add a "fixed-point" field type. This would have a  
>>>> BigDecimal
>>>> java-type and in the database should have a type that is also  
>>>> fixed point
>>>> (ie not DOUBLE or FLOAT), unless the database has no fixed point  
>>>> types
>>>> (which is a big minus for those databases). Also, the floating-
>>>> point type
>>>> should have a Double java-type and a floating point SQL type.
>>>>
>>>> One side effect to changing the java-type on these field types is  
>>>> that it
>>>> changes services definitions that are based on entity  
>>>> definitions, which
>>>> causes the service engine to throw exceptions for type checks...  
>>>> and it
>>>> does
>>>> throw exceptions instead of just printing warning messages, so it  
>>>> does
>>>> break
>>>> stuff when doing that.
>>>>
>>>> This is a change that we can't switch back and forth on. Either  
>>>> all of
>>>> these
>>>> services take a Double typed parameters, or BigDecimal typed  
>>>> ones... and
>>>> never some of each.
>>>>
>>>> Because of this I'm actually tempted to do a branch and get the  
>>>> changes
>>>> going. I'm paying with a few common processes with this in place  
>>>> right
>>>> now.
>>>>
>>>> Any thoughts related to this would be appreciated.
>>>>
>>>> Checking types... what a can of worms!
>>>>
>>>> -David
>>>>
>>>>
>>>>
>>
>>

[Scott Gray]

Thanks David, that all makes perfect sense.

Next question :-)
Some of the minilang operations need to be revised I think, for
example <set> uses string as the default type and <calculate> defaults
to a Double.  I'm wondering if we should make the type attribute
required, I've seen bugs in the past because people haven't bothered
to specify a type and my bet is that people assume in a lot of cases
that the type (for calculate at least) that they are passing in will
be the type to come out.

The other option for calculate is that we could take Groovy's approach
and choose the best type available for the operation if one isn't
specified:
http://groovy.codehaus.org/Groovy+Math#GroovyMath-Mathoperations

WDYT?

Thanks
Scott

2008/10/24 David E Jones <[hidden email]>:

>
> On Oct 23, 2008, at 3:15 AM, Scott Gray wrote:
>
>> Hi David, everyone
>>
>> I thought I'd set about changing pretty much all the doubles in the
>> accounting component to BigDecimal and I've got a couple of questions:
>> 1.  Do we need to follow the standard path we've taken in the past
>> where we deprecate any methods taking or returning doubles and add Bd
>> to the end of a new BigDecimal version?  I guess we do but it's going
>> to take a longer so I figure no harm in asking.
>
> I'm not sure where these got started, but they seem only really useful if
> you want a BigDecimal version of a function AND a Double version of it.
>
> I'd say no, there is no reason to continue following this pattern. It may
> break backward compatibility in esoteric areas, but really this should be
> considered a bug and all code affected should be moved to use BigDecimal
> instead of Double anyway.
>
>> 2.  Slightly OT but I can't see the point of the scaled and rounded
>> ZEROs in the java files, could someone explain the advantage of them
>> over BigDecimal.ZERO?
>
> In older versions of Java there was no BigDecimal.ZERO, so this is probably
> from older code and should updated.
>
>> 3.  Could someone give me an example of when to use
>> floating-point/double over fixed-point/BigDecimal?  I'm not 100% sure
>> of that one.
>
> Since we're doing business applications most things should be fixed point.
> The problem with floating point is that certain decimal numbers are not
> represented accurately and so if used without rounding in a calculation
> you'll get weird results, usually something like a bunch of 9s or 1s at the
> end of a decimal part of the number. This is because a binary number
> representation for non-integer numbers basically look at 1/2 and 1/2 of 1/2
> and 1/2 of 1/2 of 1/2, and not all fractions split up evenly like that, so
> even with a 64 bit floating point number when converted back into a base-10
> number you get something weird.
>
> So when to use a floating point number? When you are dealing with extreme
> numbers where precision beyond a certain number of significant digits
> doesn't matter so much. For financial things ALL digits are significant, so
> a number representation that may cause you to lose some is a BAD thing. For
> scientific stuff sometimes you need numbers that are on the order to 10E30
> or 10E-30 or the like that don't work so well for fixed point things, but
> they only really care about, say, 5 of the 30 or more possible significant
> digits.
>
> -David
>
>
>>
>> 2008/10/22 David E Jones <[hidden email]>:
>>>
>>> I've created a branch, now available at:
>>>
>>> https://svn.apache.org/repos/asf/ofbiz/branches/typecheckcleanup200810
>>>
>>> The branch split was revision 706867, so we can merge from the trunk at
>>> that
>>> rev going forward, and eventually merge back into the trunk once we're
>>> comfortable that enough things have been tested by running, perhaps using
>>> a
>>> bunch of the automated tests too (not sure how well those run in the
>>> trunk
>>> right now though, ie I haven't run them in quite some time!).
>>>
>>> Anyway, the main initial process that I targeted was ecommerce browsing,
>>> add
>>> to cart, and checkout/order. Still need to test basic things like order
>>> fulfillment, purchasing and receiving, and tons of other stuff, but
>>> hopefully most other things won't be impacted as much.
>>>
>>> Any and all help is appreciated!
>>>
>>> -David
>>>
>>> NOTE: I'm still checking out the branch and applying my patch with the
>>> work
>>> so far, so it'll be a few minutes before that's in.
>>>
>>> On Oct 21, 2008, at 10:20 PM, Scott Gray wrote:
>>>
>>>> +1 for making the changes and I'm happy to help with the mundane stuff.
>>>>
>>>> Regards
>>>> Scott
>>>>
>>>> 2008/10/22 David E Jones <[hidden email]>:
>>>>>
>>>>> I added some stuff to make the warnings from entity field type checks a
>>>>> bit
>>>>> more prominent, and include a stack trace to make them easier to track
>>>>> down.
>>>>> I tried throwing exceptions instead of the warning log messages but
>>>>> that
>>>>> pretty much breaks everything, especially because we have lots of
>>>>> BigDecimal/Double problems...
>>>>>
>>>>> On the BigDecimal versus Double stuff, one of the main type change
>>>>> problems
>>>>> comes from this old issue. To help resolve this I'd like to change the
>>>>> java-type of currency-amount and currency-precise to BigDecimal instead
>>>>> of
>>>>> Double.
>>>>>
>>>>> It doesn't make sense to change the "floating-point" type's java-type
>>>>> to
>>>>> BigDecimal, because it really is a floating point like a Double.
>>>>> However,
>>>>> we
>>>>> also have many fields that need decimal precision but really should be
>>>>> fixed
>>>>> point and not floating point... but they really aren't "currency". For
>>>>> these
>>>>> I propose to add a "fixed-point" field type. This would have a
>>>>> BigDecimal
>>>>> java-type and in the database should have a type that is also fixed
>>>>> point
>>>>> (ie not DOUBLE or FLOAT), unless the database has no fixed point types
>>>>> (which is a big minus for those databases). Also, the floating-point
>>>>> type
>>>>> should have a Double java-type and a floating point SQL type.
>>>>>
>>>>> One side effect to changing the java-type on these field types is that
>>>>> it
>>>>> changes services definitions that are based on entity definitions,
>>>>> which
>>>>> causes the service engine to throw exceptions for type checks... and it
>>>>> does
>>>>> throw exceptions instead of just printing warning messages, so it does
>>>>> break
>>>>> stuff when doing that.
>>>>>
>>>>> This is a change that we can't switch back and forth on. Either all of
>>>>> these
>>>>> services take a Double typed parameters, or BigDecimal typed ones...
>>>>> and
>>>>> never some of each.
>>>>>
>>>>> Because of this I'm actually tempted to do a branch and get the changes
>>>>> going. I'm paying with a few common processes with this in place right
>>>>> now.
>>>>>
>>>>> Any thoughts related to this would be appreciated.
>>>>>
>>>>> Checking types... what a can of worms!
>>>>>
>>>>> -David
>>>>>
>>>>>
>>>>>
>>>
>>>
>
>

[Adrian Crum-2]

https://issues.apache.org/jira/browse/OFBIZ-2019



--- On Fri, 10/24/08, Scott Gray <[hidden email]> wrote:

> From: Scott Gray <[hidden email]>
> Subject: Re: Entity Field Type Checking and Move BigDecimal versus Double Stuff
> To: [hidden email]
> Date: Friday, October 24, 2008, 3:03 PM
> Thanks David, that all makes perfect sense.
>
> Next question :-)
> Some of the minilang operations need to be revised I think,
> for
> example <set> uses string as the default type and
> <calculate> defaults
> to a Double.  I'm wondering if we should make the type
> attribute
> required, I've seen bugs in the past because people
> haven't bothered
> to specify a type and my bet is that people assume in a lot
> of cases
> that the type (for calculate at least) that they are
> passing in will
> be the type to come out.
>
> The other option for calculate is that we could take
> Groovy's approach
> and choose the best type available for the operation if one
> isn't
> specified:
> http://groovy.codehaus.org/Groovy+Math#GroovyMath-Mathoperations
>
> WDYT?
>
> Thanks
> Scott
>
> 2008/10/24 David E Jones <[hidden email]>:
> >
> > On Oct 23, 2008, at 3:15 AM, Scott Gray wrote:
> >
> >> Hi David, everyone
> >>
> >> I thought I'd set about changing pretty much
> all the doubles in the
> >> accounting component to BigDecimal and I've
> got a couple of questions:
> >> 1.  Do we need to follow the standard path
> we've taken in the past
> >> where we deprecate any methods taking or returning
> doubles and add Bd
> >> to the end of a new BigDecimal version?  I guess
> we do but it's going
> >> to take a longer so I figure no harm in asking.
> >
> > I'm not sure where these got started, but they
> seem only really useful if
> > you want a BigDecimal version of a function AND a
> Double version of it.
> >
> > I'd say no, there is no reason to continue
> following this pattern. It may
> > break backward compatibility in esoteric areas, but
> really this should be
> > considered a bug and all code affected should be moved
> to use BigDecimal
> > instead of Double anyway.
> >
> >> 2.  Slightly OT but I can't see the point of
> the scaled and rounded
> >> ZEROs in the java files, could someone explain the
> advantage of them
> >> over BigDecimal.ZERO?
> >
> > In older versions of Java there was no
> BigDecimal.ZERO, so this is probably
> > from older code and should updated.
> >
> >> 3.  Could someone give me an example of when to
> use
> >> floating-point/double over fixed-point/BigDecimal?
>  I'm not 100% sure
> >> of that one.
> >
> > Since we're doing business applications most
> things should be fixed point.
> > The problem with floating point is that certain
> decimal numbers are not
> > represented accurately and so if used without rounding
> in a calculation
> > you'll get weird results, usually something like a
> bunch of 9s or 1s at the
> > end of a decimal part of the number. This is because a
> binary number
> > representation for non-integer numbers basically look
> at 1/2 and 1/2 of 1/2
> > and 1/2 of 1/2 of 1/2, and not all fractions split up
> evenly like that, so
> > even with a 64 bit floating point number when
> converted back into a base-10
> > number you get something weird.
> >
> > So when to use a floating point number? When you are
> dealing with extreme
> > numbers where precision beyond a certain number of
> significant digits
> > doesn't matter so much. For financial things ALL
> digits are significant, so
> > a number representation that may cause you to lose
> some is a BAD thing. For
> > scientific stuff sometimes you need numbers that are
> on the order to 10E30
> > or 10E-30 or the like that don't work so well for
> fixed point things, but
> > they only really care about, say, 5 of the 30 or more
> possible significant
> > digits.
> >
> > -David
> >
> >
> >>
> >> 2008/10/22 David E Jones
> <[hidden email]>:
> >>>
> >>> I've created a branch, now available at:
> >>>
> >>>
> https://svn.apache.org/repos/asf/ofbiz/branches/typecheckcleanup200810
> >>>
> >>> The branch split was revision 706867, so we
> can merge from the trunk at
> >>> that
> >>> rev going forward, and eventually merge back
> into the trunk once we're
> >>> comfortable that enough things have been
> tested by running, perhaps using
> >>> a
> >>> bunch of the automated tests too (not sure how
> well those run in the
> >>> trunk
> >>> right now though, ie I haven't run them in
> quite some time!).
> >>>
> >>> Anyway, the main initial process that I
> targeted was ecommerce browsing,
> >>> add
> >>> to cart, and checkout/order. Still need to
> test basic things like order
> >>> fulfillment, purchasing and receiving, and
> tons of other stuff, but
> >>> hopefully most other things won't be
> impacted as much.
> >>>
> >>> Any and all help is appreciated!
> >>>
> >>> -David
> >>>
> >>> NOTE: I'm still checking out the branch
> and applying my patch with the
> >>> work
> >>> so far, so it'll be a few minutes before
> that's in.
> >>>
> >>> On Oct 21, 2008, at 10:20 PM, Scott Gray
> wrote:
> >>>
> >>>> +1 for making the changes and I'm
> happy to help with the mundane stuff.
> >>>>
> >>>> Regards
> >>>> Scott
> >>>>
> >>>> 2008/10/22 David E Jones
> <[hidden email]>:
> >>>>>
> >>>>> I added some stuff to make the
> warnings from entity field type checks a
> >>>>> bit
> >>>>> more prominent, and include a stack
> trace to make them easier to track
> >>>>> down.
> >>>>> I tried throwing exceptions instead of
> the warning log messages but
> >>>>> that
> >>>>> pretty much breaks everything,
> especially because we have lots of
> >>>>> BigDecimal/Double problems...
> >>>>>
> >>>>> On the BigDecimal versus Double stuff,
> one of the main type change
> >>>>> problems
> >>>>> comes from this old issue. To help
> resolve this I'd like to change the
> >>>>> java-type of currency-amount and
> currency-precise to BigDecimal instead
> >>>>> of
> >>>>> Double.
> >>>>>
> >>>>> It doesn't make sense to change
> the "floating-point" type's java-type
> >>>>> to
> >>>>> BigDecimal, because it really is a
> floating point like a Double.
> >>>>> However,
> >>>>> we
> >>>>> also have many fields that need
> decimal precision but really should be
> >>>>> fixed
> >>>>> point and not floating point... but
> they really aren't "currency". For
> >>>>> these
> >>>>> I propose to add a
> "fixed-point" field type. This would have a
> >>>>> BigDecimal
> >>>>> java-type and in the database should
> have a type that is also fixed
> >>>>> point
> >>>>> (ie not DOUBLE or FLOAT), unless the
> database has no fixed point types
> >>>>> (which is a big minus for those
> databases). Also, the floating-point
> >>>>> type
> >>>>> should have a Double java-type and a
> floating point SQL type.
> >>>>>
> >>>>> One side effect to changing the
> java-type on these field types is that
> >>>>> it
> >>>>> changes services definitions that are
> based on entity definitions,
> >>>>> which
> >>>>> causes the service engine to throw
> exceptions for type checks... and it
> >>>>> does
> >>>>> throw exceptions instead of just
> printing warning messages, so it does
> >>>>> break
> >>>>> stuff when doing that.
> >>>>>
> >>>>> This is a change that we can't
> switch back and forth on. Either all of
> >>>>> these
> >>>>> services take a Double typed
> parameters, or BigDecimal typed ones...
> >>>>> and
> >>>>> never some of each.
> >>>>>
> >>>>> Because of this I'm actually
> tempted to do a branch and get the changes
> >>>>> going. I'm paying with a few
> common processes with this in place right
> >>>>> now.
> >>>>>
> >>>>> Any thoughts related to this would be
> appreciated.
> >>>>>
> >>>>> Checking types... what a can of worms!
> >>>>>
> >>>>> -David
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >

     

[Scott Gray]

Ok forget what I said about the set element, it does give out the type
that it took in.  I still think we need to do something about
calculate though.

Regards
Scott

2008/10/25 Scott Gray <[hidden email]>:

> Thanks David, that all makes perfect sense.
>
> Next question :-)
> Some of the minilang operations need to be revised I think, for
> example <set> uses string as the default type and <calculate> defaults
> to a Double.  I'm wondering if we should make the type attribute
> required, I've seen bugs in the past because people haven't bothered
> to specify a type and my bet is that people assume in a lot of cases
> that the type (for calculate at least) that they are passing in will
> be the type to come out.
>
> The other option for calculate is that we could take Groovy's approach
> and choose the best type available for the operation if one isn't
> specified:
> http://groovy.codehaus.org/Groovy+Math#GroovyMath-Mathoperations
>
> WDYT?
>
> Thanks
> Scott
>
> 2008/10/24 David E Jones <[hidden email]>:
>>
>> On Oct 23, 2008, at 3:15 AM, Scott Gray wrote:
>>
>>> Hi David, everyone
>>>
>>> I thought I'd set about changing pretty much all the doubles in the
>>> accounting component to BigDecimal and I've got a couple of questions:
>>> 1.  Do we need to follow the standard path we've taken in the past
>>> where we deprecate any methods taking or returning doubles and add Bd
>>> to the end of a new BigDecimal version?  I guess we do but it's going
>>> to take a longer so I figure no harm in asking.
>>
>> I'm not sure where these got started, but they seem only really useful if
>> you want a BigDecimal version of a function AND a Double version of it.
>>
>> I'd say no, there is no reason to continue following this pattern. It may
>> break backward compatibility in esoteric areas, but really this should be
>> considered a bug and all code affected should be moved to use BigDecimal
>> instead of Double anyway.
>>
>>> 2.  Slightly OT but I can't see the point of the scaled and rounded
>>> ZEROs in the java files, could someone explain the advantage of them
>>> over BigDecimal.ZERO?
>>
>> In older versions of Java there was no BigDecimal.ZERO, so this is probably
>> from older code and should updated.
>>
>>> 3.  Could someone give me an example of when to use
>>> floating-point/double over fixed-point/BigDecimal?  I'm not 100% sure
>>> of that one.
>>
>> Since we're doing business applications most things should be fixed point.
>> The problem with floating point is that certain decimal numbers are not
>> represented accurately and so if used without rounding in a calculation
>> you'll get weird results, usually something like a bunch of 9s or 1s at the
>> end of a decimal part of the number. This is because a binary number
>> representation for non-integer numbers basically look at 1/2 and 1/2 of 1/2
>> and 1/2 of 1/2 of 1/2, and not all fractions split up evenly like that, so
>> even with a 64 bit floating point number when converted back into a base-10
>> number you get something weird.
>>
>> So when to use a floating point number? When you are dealing with extreme
>> numbers where precision beyond a certain number of significant digits
>> doesn't matter so much. For financial things ALL digits are significant, so
>> a number representation that may cause you to lose some is a BAD thing. For
>> scientific stuff sometimes you need numbers that are on the order to 10E30
>> or 10E-30 or the like that don't work so well for fixed point things, but
>> they only really care about, say, 5 of the 30 or more possible significant
>> digits.
>>
>> -David
>>
>>
>>>
>>> 2008/10/22 David E Jones <[hidden email]>:
>>>>
>>>> I've created a branch, now available at:
>>>>
>>>> https://svn.apache.org/repos/asf/ofbiz/branches/typecheckcleanup200810
>>>>
>>>> The branch split was revision 706867, so we can merge from the trunk at
>>>> that
>>>> rev going forward, and eventually merge back into the trunk once we're
>>>> comfortable that enough things have been tested by running, perhaps using
>>>> a
>>>> bunch of the automated tests too (not sure how well those run in the
>>>> trunk
>>>> right now though, ie I haven't run them in quite some time!).
>>>>
>>>> Anyway, the main initial process that I targeted was ecommerce browsing,
>>>> add
>>>> to cart, and checkout/order. Still need to test basic things like order
>>>> fulfillment, purchasing and receiving, and tons of other stuff, but
>>>> hopefully most other things won't be impacted as much.
>>>>
>>>> Any and all help is appreciated!
>>>>
>>>> -David
>>>>
>>>> NOTE: I'm still checking out the branch and applying my patch with the
>>>> work
>>>> so far, so it'll be a few minutes before that's in.
>>>>
>>>> On Oct 21, 2008, at 10:20 PM, Scott Gray wrote:
>>>>
>>>>> +1 for making the changes and I'm happy to help with the mundane stuff.
>>>>>
>>>>> Regards
>>>>> Scott
>>>>>
>>>>> 2008/10/22 David E Jones <[hidden email]>:
>>>>>>
>>>>>> I added some stuff to make the warnings from entity field type checks a
>>>>>> bit
>>>>>> more prominent, and include a stack trace to make them easier to track
>>>>>> down.
>>>>>> I tried throwing exceptions instead of the warning log messages but
>>>>>> that
>>>>>> pretty much breaks everything, especially because we have lots of
>>>>>> BigDecimal/Double problems...
>>>>>>
>>>>>> On the BigDecimal versus Double stuff, one of the main type change
>>>>>> problems
>>>>>> comes from this old issue. To help resolve this I'd like to change the
>>>>>> java-type of currency-amount and currency-precise to BigDecimal instead
>>>>>> of
>>>>>> Double.
>>>>>>
>>>>>> It doesn't make sense to change the "floating-point" type's java-type
>>>>>> to
>>>>>> BigDecimal, because it really is a floating point like a Double.
>>>>>> However,
>>>>>> we
>>>>>> also have many fields that need decimal precision but really should be
>>>>>> fixed
>>>>>> point and not floating point... but they really aren't "currency". For
>>>>>> these
>>>>>> I propose to add a "fixed-point" field type. This would have a
>>>>>> BigDecimal
>>>>>> java-type and in the database should have a type that is also fixed
>>>>>> point
>>>>>> (ie not DOUBLE or FLOAT), unless the database has no fixed point types
>>>>>> (which is a big minus for those databases). Also, the floating-point
>>>>>> type
>>>>>> should have a Double java-type and a floating point SQL type.
>>>>>>
>>>>>> One side effect to changing the java-type on these field types is that
>>>>>> it
>>>>>> changes services definitions that are based on entity definitions,
>>>>>> which
>>>>>> causes the service engine to throw exceptions for type checks... and it
>>>>>> does
>>>>>> throw exceptions instead of just printing warning messages, so it does
>>>>>> break
>>>>>> stuff when doing that.
>>>>>>
>>>>>> This is a change that we can't switch back and forth on. Either all of
>>>>>> these
>>>>>> services take a Double typed parameters, or BigDecimal typed ones...
>>>>>> and
>>>>>> never some of each.
>>>>>>
>>>>>> Because of this I'm actually tempted to do a branch and get the changes
>>>>>> going. I'm paying with a few common processes with this in place right
>>>>>> now.
>>>>>>
>>>>>> Any thoughts related to this would be appreciated.
>>>>>>
>>>>>> Checking types... what a can of worms!
>>>>>>
>>>>>> -David
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
>

[David E Jones]

Yes, set gives you back what you send in unless you specify a type,  
then is tries to convert it.

The calculate operation does use BigDecimal internally and then  
converts to a Double by default. Previous I didn't change the default  
to BigDecimal (actually tried it) because it caused too many problems.  
That would be a good default to change now. Thanks for reminding me  
about that, I had totally forgotten that compromise in previous efforts!

Okay, that change is in the branch (rev 707790)... now to look at the  
problems it caused.

-David


On Oct 24, 2008, at 4:30 PM, Scott Gray wrote:

> Ok forget what I said about the set element, it does give out the type
> that it took in.  I still think we need to do something about
> calculate though.
>
> Regards
> Scott
>
> 2008/10/25 Scott Gray <[hidden email]>:
>> Thanks David, that all makes perfect sense.
>>
>> Next question :-)
>> Some of the minilang operations need to be revised I think, for
>> example <set> uses string as the default type and <calculate>  
>> defaults
>> to a Double.  I'm wondering if we should make the type attribute
>> required, I've seen bugs in the past because people haven't bothered
>> to specify a type and my bet is that people assume in a lot of cases
>> that the type (for calculate at least) that they are passing in will
>> be the type to come out.
>>
>> The other option for calculate is that we could take Groovy's  
>> approach
>> and choose the best type available for the operation if one isn't
>> specified:
>> http://groovy.codehaus.org/Groovy+Math#GroovyMath-Mathoperations
>>
>> WDYT?
>>
>> Thanks
>> Scott
>>
>> 2008/10/24 David E Jones <[hidden email]>:
>>>
>>> On Oct 23, 2008, at 3:15 AM, Scott Gray wrote:
>>>
>>>> Hi David, everyone
>>>>
>>>> I thought I'd set about changing pretty much all the doubles in the
>>>> accounting component to BigDecimal and I've got a couple of  
>>>> questions:
>>>> 1.  Do we need to follow the standard path we've taken in the past
>>>> where we deprecate any methods taking or returning doubles and  
>>>> add Bd
>>>> to the end of a new BigDecimal version?  I guess we do but it's  
>>>> going
>>>> to take a longer so I figure no harm in asking.
>>>
>>> I'm not sure where these got started, but they seem only really  
>>> useful if
>>> you want a BigDecimal version of a function AND a Double version  
>>> of it.
>>>
>>> I'd say no, there is no reason to continue following this pattern.  
>>> It may
>>> break backward compatibility in esoteric areas, but really this  
>>> should be
>>> considered a bug and all code affected should be moved to use  
>>> BigDecimal
>>> instead of Double anyway.
>>>
>>>> 2.  Slightly OT but I can't see the point of the scaled and rounded
>>>> ZEROs in the java files, could someone explain the advantage of  
>>>> them
>>>> over BigDecimal.ZERO?
>>>
>>> In older versions of Java there was no BigDecimal.ZERO, so this is  
>>> probably
>>> from older code and should updated.
>>>
>>>> 3.  Could someone give me an example of when to use
>>>> floating-point/double over fixed-point/BigDecimal?  I'm not 100%  
>>>> sure
>>>> of that one.
>>>
>>> Since we're doing business applications most things should be  
>>> fixed point.
>>> The problem with floating point is that certain decimal numbers  
>>> are not
>>> represented accurately and so if used without rounding in a  
>>> calculation
>>> you'll get weird results, usually something like a bunch of 9s or  
>>> 1s at the
>>> end of a decimal part of the number. This is because a binary number
>>> representation for non-integer numbers basically look at 1/2 and  
>>> 1/2 of 1/2
>>> and 1/2 of 1/2 of 1/2, and not all fractions split up evenly like  
>>> that, so
>>> even with a 64 bit floating point number when converted back into  
>>> a base-10
>>> number you get something weird.
>>>
>>> So when to use a floating point number? When you are dealing with  
>>> extreme
>>> numbers where precision beyond a certain number of significant  
>>> digits
>>> doesn't matter so much. For financial things ALL digits are  
>>> significant, so
>>> a number representation that may cause you to lose some is a BAD  
>>> thing. For
>>> scientific stuff sometimes you need numbers that are on the order  
>>> to 10E30
>>> or 10E-30 or the like that don't work so well for fixed point  
>>> things, but
>>> they only really care about, say, 5 of the 30 or more possible  
>>> significant
>>> digits.
>>>
>>> -David
>>>
>>>
>>>>
>>>> 2008/10/22 David E Jones <[hidden email]>:
>>>>>
>>>>> I've created a branch, now available at:
>>>>>
>>>>> https://svn.apache.org/repos/asf/ofbiz/branches/typecheckcleanup200810
>>>>>
>>>>> The branch split was revision 706867, so we can merge from the  
>>>>> trunk at
>>>>> that
>>>>> rev going forward, and eventually merge back into the trunk once  
>>>>> we're
>>>>> comfortable that enough things have been tested by running,  
>>>>> perhaps using
>>>>> a
>>>>> bunch of the automated tests too (not sure how well those run in  
>>>>> the
>>>>> trunk
>>>>> right now though, ie I haven't run them in quite some time!).
>>>>>
>>>>> Anyway, the main initial process that I targeted was ecommerce  
>>>>> browsing,
>>>>> add
>>>>> to cart, and checkout/order. Still need to test basic things  
>>>>> like order
>>>>> fulfillment, purchasing and receiving, and tons of other stuff,  
>>>>> but
>>>>> hopefully most other things won't be impacted as much.
>>>>>
>>>>> Any and all help is appreciated!
>>>>>
>>>>> -David
>>>>>
>>>>> NOTE: I'm still checking out the branch and applying my patch  
>>>>> with the
>>>>> work
>>>>> so far, so it'll be a few minutes before that's in.
>>>>>
>>>>> On Oct 21, 2008, at 10:20 PM, Scott Gray wrote:
>>>>>
>>>>>> +1 for making the changes and I'm happy to help with the  
>>>>>> mundane stuff.
>>>>>>
>>>>>> Regards
>>>>>> Scott
>>>>>>
>>>>>> 2008/10/22 David E Jones <[hidden email]>:
>>>>>>>
>>>>>>> I added some stuff to make the warnings from entity field type  
>>>>>>> checks a
>>>>>>> bit
>>>>>>> more prominent, and include a stack trace to make them easier  
>>>>>>> to track
>>>>>>> down.
>>>>>>> I tried throwing exceptions instead of the warning log  
>>>>>>> messages but
>>>>>>> that
>>>>>>> pretty much breaks everything, especially because we have lots  
>>>>>>> of
>>>>>>> BigDecimal/Double problems...
>>>>>>>
>>>>>>> On the BigDecimal versus Double stuff, one of the main type  
>>>>>>> change
>>>>>>> problems
>>>>>>> comes from this old issue. To help resolve this I'd like to  
>>>>>>> change the
>>>>>>> java-type of currency-amount and currency-precise to  
>>>>>>> BigDecimal instead
>>>>>>> of
>>>>>>> Double.
>>>>>>>
>>>>>>> It doesn't make sense to change the "floating-point" type's  
>>>>>>> java-type
>>>>>>> to
>>>>>>> BigDecimal, because it really is a floating point like a Double.
>>>>>>> However,
>>>>>>> we
>>>>>>> also have many fields that need decimal precision but really  
>>>>>>> should be
>>>>>>> fixed
>>>>>>> point and not floating point... but they really aren't  
>>>>>>> "currency". For
>>>>>>> these
>>>>>>> I propose to add a "fixed-point" field type. This would have a
>>>>>>> BigDecimal
>>>>>>> java-type and in the database should have a type that is also  
>>>>>>> fixed
>>>>>>> point
>>>>>>> (ie not DOUBLE or FLOAT), unless the database has no fixed  
>>>>>>> point types
>>>>>>> (which is a big minus for those databases). Also, the floating-
>>>>>>> point
>>>>>>> type
>>>>>>> should have a Double java-type and a floating point SQL type.
>>>>>>>
>>>>>>> One side effect to changing the java-type on these field types  
>>>>>>> is that
>>>>>>> it
>>>>>>> changes services definitions that are based on entity  
>>>>>>> definitions,
>>>>>>> which
>>>>>>> causes the service engine to throw exceptions for type  
>>>>>>> checks... and it
>>>>>>> does
>>>>>>> throw exceptions instead of just printing warning messages, so  
>>>>>>> it does
>>>>>>> break
>>>>>>> stuff when doing that.
>>>>>>>
>>>>>>> This is a change that we can't switch back and forth on.  
>>>>>>> Either all of
>>>>>>> these
>>>>>>> services take a Double typed parameters, or BigDecimal typed  
>>>>>>> ones...
>>>>>>> and
>>>>>>> never some of each.
>>>>>>>
>>>>>>> Because of this I'm actually tempted to do a branch and get  
>>>>>>> the changes
>>>>>>> going. I'm paying with a few common processes with this in  
>>>>>>> place right
>>>>>>> now.
>>>>>>>
>>>>>>> Any thoughts related to this would be appreciated.
>>>>>>>
>>>>>>> Checking types... what a can of worms!
>>>>>>>
>>>>>>> -David
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>>