VERIFIED SOLUTION i
X

'To Decimal' function not rounding off data correctly during comparisons in Expression calculator in Sagent

Issue

While using 'To decimal (16,3)' function in the Expression calculator transform, output appears as 16.236012 rather than converting it to decimal i.e. (16,3) = 16.236. This behavior is observed while comparing two values like To_Decimal (1.3681,38,3) and To_Decimal (1.3680,38,3).

Cause

For the sake of performance, decimal values are converted to type double when the values are used in calculations or comparisons. However, this process may result in a loss of precision. To ensure a specific accuracy of decimal results, decimal math functionality is enabled. Transforms that do sorts or comparisons with decimal fields may be affected by enabling decimal math, because decimal values are no longer converted to type double.

Resolution

UPDATED: October 23, 2018


Follow these steps:
  1. Navigate to Tools->'Plan Properties'
  2. Enable/choose 'Use decimal math provider'.

Environment Details

Product Feature: Design Studio

Downloads

  • No Downloads