Validation of results against reference files

Introduction

The ReferenceChecker allows to compare the results produced by a suite against reference files.

A typical usage is the following:

  • During a preliminary generation phase, the reference files are generated, by running a suite in a known configuration.

  • During the actual check phase, the files produced by the suite and compared against the reference files generated before.

These phase are triggered by configuring the variables [reference_checker.check] and [reference_checker.generate] before starting the suite. These options are not mutually exclusive and can be used together, for instance to compare the results against itself, making sure that the system is setup as expected.

In practice, the two flags can be activated by including the configuration file tactus/data/config_files/modifications/use_reference_checker.toml.

If both [reference_checker.check] and reference_checker.generate] are false, the reference_checker is deactivated. Otherwise, reference_checker is activated and a task ReferenceCheck task is added at the end of the suite.

The comparisons are performed directly after each task, and the results are appended to a summary file. At the end of the suite, the summary is analysed, and the suite will fail in case of error.

It is possible to deactivate failure in case of error by setting [reference_checker.suppress_exception] = true. The results will be checked and an error message will be logged but the suite will not fail anymore.

Each suite ran with [reference_checker] activated will produce:

  • a diff file for each comparison performed, generated by the tool (xtool, NormsChecker)

  • a summary file listing all the comparison performed and the results of the comparisons.

Available comparison methods and summary files

The following comparison tools are available:

  • NormsChecker (from DAVAI), to compare norms results in NODE files

  • xtool (from gl) to compare fields store in GRIB and FA format

The available formats for the summary files are json or txt.

Reference checker configuration

General configuration

The reference_checker section is defined in tactus/data/config_files/include/reference_checker.toml.

For instance, the following code defines the configuration of a reference checker that will be used in tactus.

[reference_checker]
  check = true
  generate = true
  rules_active = ["Forecast.fullpos", "Forecast.history", "Forecast.log"]
  create_summary = ["PrepRun"]
  analyze_summary = ["ReferenceCheck"]
  summary_active = ["MainSummary"]
  suppress_exception = false
  label_suffix = "@MEMBER_STR@"

In this configuration file:

  • generate will activate the generation phase

  • check will activate the comparison phase

  • rules_active defines which rules should be used

  • create_summary defines which task will create an initialize the summary files

  • analyze_summary defines in which task the summary will be checked for error

  • summary_active defines which summary should be created.

  • suppress_exception: by default, an exception is raised when an exception is found during analyze_summary. This can be deactivated if suppress_exception = true, if you don’t want that your tests fails in case of difference.

  • label_suffix : a suffix to append to the task name in the summary

Comparison methods

The comparison methods that can be used to compare files are defined as methods [reference_checker.methods.{MethodName}] where {MethodName} is the name of the method.

For instance, the following code defines a new FullPosFields method using xtool to compare GRIB files:

[reference_checker.methods.FullPosFields]
  args_template = "-f1 {test_file} -f2 {reference_file} {file_format} -s -of SCREEN -de -to {tolerance}"
  binary = "@BINDIR_GL@/xtool"
  file_format = "GRIB"
  tolerance = "10"
  tool = "xtool"

The different configuration settings (eg: args_template, binary, tolerance, etc) can be used configure the tools and are specific to each method.

Comparison rules configuration

The files to be compared are defined as rules [reference_checker.task.{TaskName}.{RuleName}] where {TaskName} is the name of the task and {RuleName} is the name of the rule.

For instance, the following code defines a new fullpos rule for the Forecast task, using the FullPostFields method defined above:

[reference_checker.task.Forecast.fullpos]
  inpath = "@ARCHIVE@"
  pattern = "GRIBPF*0006h00m00s"
  generate_folder = "@REFERENCES_GENERATION_FOLDER@/@CASE@/@ARCHIVE_TIMESTAMP@/@MEMBER_STR@"
  method = "FullPosFields"
  reference_folder = "@REFERENCES_FOLDER@/@CASE@/@ARCHIVE_TIMESTAMP@/@MEMBER_STR@"
  result_folder = "@WRK@/checks/Forecast"

The files that will be compared are the one that can be found in the ‘inpath’ directory using the regex pattern provided in the pattern parameter, in this case "GRIBPF*0006h00m00s".

During the generation phase, the file will be copied to the generate_folder defined in the rule. The file will be overwritten if it already exists.

During the comparison phase, the file will be compared against the reference_folder defined in the rule. An error will be raised if the reference file can not be found or if the comparison method reports a difference.

The result_folder will contain the .diff files containing the detailed results of the comparison.

Summary files configuration

Summary files are defined as summary in [reference_checker.summary.{SummaryName}] where SummaryName is the name of the reference summary.

For instance, the following code defines the MainSummary, that will be written in json format:

[reference_checker.summary.MainSummary]
  file = "@CASEDIR@/checks/summary.json"
  format = "json"

The summary file will be stored in the path defined in the file configuration field.

The summary files are created and initialized at the begining of the suite, during the task(s) defined in [reference_checker.create_summary] configuration field.

The summary files are checked at the end of the suite, during the task(s) defined in [reference_checker.analyze_summary].

With the following definition, the summary file will be created in ‘PrepRun’ task and analyzed in ‘ReferenceCheck’ task, that is automatically added to the suite when ‘[reference_checker’] is active. However, it is possible to specify another task, if you prefer to have the tests checked at another place.

[reference_checker]
...
  create_summary = ["PrepRun"]
  analyze_summary = ["ReferenceCheck"]
  summary_active = ["MainSummary"]
  suppress_exception = false
...

The list provided in summary_active is the list of summary that should be produced and analyzed, referenced by their name as defined in [reference_checker.summary.{SummaryName}].

By default, the [reference check] will raise an exception in the task mentionned in the ‘analyze_summary’ list, in case of check failure. This can be deactivated by setting suppress_exception = true.