Re: FN-FORUM: ASP Maths Problem!

date posted 1st February 2008 12:32

On Fri, 2008-02-01 at 12:29 +0000, Dom Latter wrote:
> On Friday 01 February 2008 12:40:26 Rob Colton wrote:
>
> > Not sure I understand, isn't it just
> >
> >
>
> Close enough for jazz, but for an *accurate* result you need to use
> integers only, because (in general) *binary* computers don't store
> *decimal* fractions precisely.
>

If you are worried about rounding errors you probably want to use some
sort of decimal type, if your language supports it (I think ASP does,
though I've never used it). It's more intuitive and in the better
implementations (Ada, Java) you can control how things are rounded
(integer arithmetic rounds down, so 99/100 == 0).

This is a good article on all this:

http://www2.hursley.ibm.com/decimal/IEEE-cowlishaw-arith16.pdf

> Not a real issue in this case, but it can be one of those "gotchas"
> that crops up in other problems, so it's a good idea to get in the
> habit of doing integer arithmetic wherever possible, and simplifying
> the equation to reduce rounding errors


> (and save CPU cycles, cos you should).

> So intTotal * 47/200 will do nicely.
>

This could give you 0 depending on the order of evaluation - whether
your compiler sees this as (intTotal*47)/200 or intTotal*(47/200). The
former is more usual, though. Sometimes using floats and rounding right
at the end is the safest course.

Graham

--
Graham Stark, Virtual Worlds, http://www.virtual-worlds.biz
Phone (+44) 01908 618239 Mobile (+44) 07952633185 Skype graham_k_stark