Re: FN-FORUM In VB, 12.8 + 0.576 doesn;t equal 13.376

date posted 22nd January 2002 17:25

What exactly are you trying to fix? :-)

Floating point numbers are dangerous to compare like that as an innocent
looking number when expressed in decimal might resolve to a binary irrational
or recurring fraction, yielding just that sort of strange oddity.

Look up "rounding error" on the web...

I just tried the same thing in TCL on a Linux box and got 1.7763568394e-15.

On Tuesday 22 Jan 2002 4:42 pm, Peter Ebdon wrote:
> I'm using VB 6.0 SP 5
>
> I'm debugging some code and have found that if you type: -
>
> ? (12.8 + 0.576) = 13.376
>
> into the immediate window you get 'False'.
>
> If you calculate (12.8 + 0.576) - 13.376 you get 1.22...E-15
>
> If you convert the numbers to singles it works OK.
>
> Anyone know of a fix?
>
> Cheers,
>
> Pete
>
> _________________________________________
>
> P e t e r E b d o n
> Professional Software Engineering Ltd
>
> Microsoft Certified Professional
>
> Email : [EMAIL REMOVED]
> Tel : +44(0)1235-227279
> Fax : +44(0)1235-227280
> Mobile : +44(0)7973-178945
> Web : www.pseltd.com
>
>
>
> ============================================================
> = Get your message here for as little as £1 per week =
> = =
> = http://www.freelancers.net/cgi/sponsor.cgi?action=show =
> ============================================================
>
> * Free listing for freelancers
> * Free to advertise jobs
> * Free jobs distribution service
> * Free database of 5000 freelancers
>
> Freelancers and Freelance Jobs:
> http://www.freelancers.net
>
> Forum FAQs:
> http://freelancers.netrickery.com
>
> To unsubscribe please email:
> [EMAIL REMOVED]
>
> If you have difficulties unsubscribing please email:
> [EMAIL REMOVED]