Jellyfish Compliance with SQIDEP · MCU15/0122

6
Technical Papers


                                                 because both false negatives and false            Z score,     should be chosen Z scores
 Table 8. Minimum number of paired
 influent/effluent samples required              positives are important (see section on           from the one-tailed distribution Z-scores
 to establish the mean CRE% for TSS              errors). In many field monitoring programs,       (Teixeira et al., 2009). Given that,
 removal at the 95% Confidence Level.            power issues (such as false negative              confidence level and statistical power
                                      TSS        problems) may actually be more critical           should be balanced to ensure further
                                                 than confidence (Burton and Pitt, 2002).          confidence in the results, but without
 Z-Score (Zα/2)                       1.96
                                                 TYPE I AND TYPE II ERRORS                         making the test too exclusive. Therefore,
 COV                                 0.645                                                         the confidence level can be reduced to
                                                 A Type I error can be defined as the
 % Removal                            41%                                                          90% and the power increased to 80%.
                                                 probability of rejecting a null hypothesis
                                                                                                   Z-scores for these values are:
 Number of paired samples                        that is actually true (a false positive) and
 required to meet 95%                24.32       is typically associated with the confidence
 Confidence Interval                             level, or α value. The confidence level chosen
                                                 is the largest probability of committing an
                                                 allowable Type I error and still rejecting the      Table 10 presents the results for the
    It can be seen in Table 7 that there is
                                                 null hypothesis (Park, 2008).                     number of paired samples required if the
much greater variability around the TSS
treatment removal performance, with per                                                            Type II error is included with an 80% power
                                                    A Type II error can be defined as the          level and balanced with a 90% confidence
cent removal varying from 6% to 89%.
                                                 probability of incorrectly retaining the null     level requirement (using Table 1 data).
It is important to note that the SQIDEP
                                                 hypothesis (a false negative), and is typically
protocol (2014) calls for 15 qualifying                                                              When compared to Table 6, it can be
                                                 associated with statistical power, or β value.
events, a criterion that this data would                                                           seen in Table 10 that when the power is
                                                 The power of the test will determine the
satisfy. However, if we apply the minimum                                                          included, the number of paired samples
                                                 likelihood of selecting an incorrect null
number of paired samples calculations as                                                           required almost doubles, even though the
                                                 hypothesis, and is often used to determine
per the above, we can see in Table 8 that                                                          confidence level is reduced from 95% to
                                                 the required sample size (Park, 2008).
the data set falls well short of the required                                                      90%. The importance of setting the power
number, at 24 paired samples required to             In many environmental programs, the
                                                                                                   to 80% cannot be underestimated, and
establish the mean CRE% for TSS (41%)            importance of a Type II error (β) can be very
                                                                                                   can be a critical element to increasing the
at the 95% confidence level.                     high. This would be equivalent to having
                                                                                                   statistical quality of data presented.
                                                 a null hypothesis of “the water sample
  The importance of calculating a
                                                 is not contaminated”, then retaining the          LIMIT ON MINIMUM NUMBER
minimum number of paired samples
                                                 null hypothesis, when it was actually false.      OF PAIRED SAMPLES
has been clearly established and it will be
                                                 This could lead to environmental damage,
necessary to explore this in great detail for                                                      This approach for assessing proprietory
                                                 health risks to the public, and possibly
the statistical validation of performance                                                          device performance claims recommends
                                                 fines imposed by regulatory agencies, as
data under SQIDEP. Burton and Pitt (2002),                                                         that field-testing is continued until the
                                                 a contaminated discharge of water would
however, take this calculation one step                                                            required number of paired samples is
                                                 occur (Burton and Pitt, 2002). A higher
further as we discuss next.                                                                        met as per the criteria shown for Table
                                                 statistical power (1-β) can be used to
                                                                                                   6 or 10. However, SQIDEP (Stormwater
CONFIDENCE LEVEL AND                             minimise this type of error, and should be
                                                                                                   Australia, 2014) calls for a minimum
STATISTICAL POWER (α                             carefully selected. Typically, a power of
                                                                                                   number of monitoring events that are
AND β VALUES)                                    80% (β = 0.2) is recommended, whereas
                                                 if the Type II error is to be ignored, then a     statistically relevant.
Most statistical tests convey a certain
                                                 power of 50% (β = 0.5) is adopted.                  Given the variable nature of field-
required level of confidence (α). This
indicates and sets the criteria for a              Therefore, if statistical power were to         testing stormwater treatment devices, the
decision about the null hypothesis for the       be included in the above equation used to         number of paired samples required can
test (Privitera, 2011). Statistical power (β),   calculate the minimum number of paired            be very high if the data is not particularly
on the other hand, is frequently ignored,        samples required, the equation becomes:           consistent, as shown by the TSS removal
even though it can sometimes be more                                                               data in Table 8. One solution could be
important than confidence level (Burton                                                            that, for the full performance claims to be
and Pitt, 2002).                                                                                   verified, the 90% confidence interval and
                                                 where                                             80% power requirements must be met.
    Confidence levels are defined as:
                                                                                                   However, if the required number of paired
                                                                                                   samples has not been achieved under
                                                                                                   these statistical criteria, then perhaps the
  Whereas statistical power can be
                                                   It should be noted that while                   overall performance of the device could
defined as:
                                                 should be chosen from the two-tailed              be reduced by a scaling factor. Further

                                                  Table 9. Summary of the available choices with Type I and Type II Errors when
  When evaluating data using a statistical        assessing the null hypothesis.
test, power is the sensitivity of the test
                                                                            Do Not Reject H0                          Reject H0
for rejecting the hypothesis. It is important
that power and confidence be balanced                                       Correct Decision                         Type I Error
                                                  H0 is True
for an effective monitoring program. Most                               1 – α: Confidence Level          α: Size of Test (Significance Level)
studies ignore power, while providing                                          Type II Error                      Correct Decision
a high value (typically 95%) for the level        H0 is False
                                                                      β: Size of Test (Power Level)              1 – β: Power of Test
of confidence. This is a flawed approach



WATER MAY 2015