IEEE.org     |     IEEE Xplore Digital Library     |     IEEE Standards     |     IEEE Spectrum     |     More Sites

Commit 6de05e1b authored by Lars Asplund's avatar Lars Asplund
Browse files

Manage testing of error asserts that can't be fully tested.

Functions that are expected to fail with an assert of severity *error* are normally tested in in two steps:
1. Configure the simulator to stop on error and then apply test input to trigger the assert. The simulator is expected to stop on the assert and this premature simulation exit is logged as an error.
2. Configure the simulator to stop on failure and then apply test input to trigger the assert. The simulator is expected to run to completion and the assert will go undetected. The test is logged as a passing test.

If the first step fails and the second passes we can conclude that the assert severity level is error. A couple of tests will fail in step 2 because the tested function is not designed to handle a non-stopping assert. Specifically, a few functions
will run into an integer out of range problem that will force the simulator to stop regardless of its configuration. This commit changes the tests for these functions such that only step 1 is performed. The price for not running step 2
is that we can't conclude that the assert is of severity error. All we know is that it is of severity error or failure.

The integer out of range condition was not detected by earlier versions of GHDL which caused step 2 to pass as expected.
parent 724684dc
Pipeline #35 canceled with stage
......@@ -122,6 +122,8 @@ def create_project(root, args):
for test_case in testbench.get_tests():
if test_case.name.startswith("Expected to warn"):
levels = ["warning", "error"]
elif test_case.name.startswith("Expected to always fail"):
levels = ["error"]
elif test_case.name.startswith("Expected to fail"):
levels = ["error", "failure"]
else:
......@@ -137,7 +139,9 @@ def check_report(report_file):
""" Report mismatch between test case status and expectation."""
def expected_to_fail(test_case_name):
if "Expected to fail" in test_case_name:
if "Expected to always fail" in test_case_name:
return ("stop@warning" in test_case_name) or ("stop@error" in test_case_name)
elif "Expected to fail" in test_case_name:
return ("stop@warning" in test_case_name) or ("stop@error" in test_case_name)
elif "Expected to warn" in test_case_name:
return ("stop@warning" in test_case_name)
......
......@@ -251,9 +251,9 @@ begin -- architecture testbench
run("Expected to fail: literal error from i-l function") or
run("Expected to fail: literal error from * function") or
run("Expected to fail: literal error from i/l function") or
run("Expected to fail: literal error from i mod l function") or
run("Expected to always fail: literal error from i mod l function") or
run("Expected to fail: literal error from l mod i function") or
run("Expected to fail: literal error from f rem l function") or
run("Expected to always fail: literal error from f rem l function") or
run("Expected to fail: literal error from l rem f function") or
run("Expected to fail: literal error from a resize lit function") or
run("Expected to fail: literal error from f=l function") or
......@@ -4591,11 +4591,11 @@ begin -- architecture testbench
unull := checkint * "0001";
elsif running_test_case = "Expected to fail: literal error from i/l function" then
unull := checkint / "0001";
elsif running_test_case = "Expected to fail: literal error from i mod l function" then
elsif running_test_case = "Expected to always fail: literal error from i mod l function" then
unull := check7uf1 mod "0001";
elsif running_test_case = "Expected to fail: literal error from l mod i function" then
unull := "0001" mod check7uf1;
elsif running_test_case = "Expected to fail: literal error from f rem l function" then
elsif running_test_case = "Expected to always fail: literal error from f rem l function" then
unull := check7uf1 rem "0001";
elsif running_test_case = "Expected to fail: literal error from l rem f function" then
unull := "0001" rem check7uf1;
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment