Stupid NUnit Tricks #3 Foiled!
Here's a bit about finding the calling test fixture name and bugs that result from doing this.
Normally, writing tests that are aware of their environment is not an ideal practice. It invites writing code that performs differently depending on its test runtime environment (ie, if I'm in a test rig, I do abc, but otherwise do cde). We have a valid exception, though. A lot of our tests fall are what we call "stability tests". They perform an operation on an image and make sure that the output of the operation is unchanged. In effect it's like doing Assert.AreEqual(4, 2 + 2, "Whoa! Math failed!"); We have close to 100 tests that work this way. They perform an image command and then run an SHA1 hash on the image data and make sure that everything is sound. Rather than embedding the hash in the test, we do more:
[Test]
public void BugReport2391()
{
AtalaImage image = /* ... code to reproduce bug ... */
StaticTestManager.CompareStoredHash(img);
}
CompareStoredHash walks the stack and finds the first frame with the TestAttribute and also grabs the name of the class and the .NET version and the target architecture and looks for a file named:
UnitTestOutput\CLRn.nDotImagem.m_architecture\ClassNameMethodName.txt
If the file doesn't exist, we create it and write a hash of the image into it and write the image with non-lossy codec.
If the file does exist we compare the hash and if they don't match, we write the new image tagged as a failure and write an image representing the differences between baseline and current. Then we trigger an Assert.Fail() and point to the output in the failure message.
In effect, when a mismatch happens, we get a clear indication of what happened and where to go to get the results.
All of this works because of being able to do this:
StackFrame[] frames = new StackTrace(false).GetFrames();
foreach (StackFrame frame in frames) {
MethodBase mb = frame.GetMethod();
if (HasTestAttribute(mb)) {
return mb.Name;
}
}
NUnit runs tests using MethodInfo.Invoke() - exactly the way I'd do it. The problem is that Invoke() works very differently on Win32 and x64. On Win32, the method called by Invoke() ends up on the stack. On x64 it doesn't.
Apparently - and I haven't dug too deeply into this yet - Invoke calls InvokeMethodFast which does the dirty work. On x64 InvokeMethodFast calls _InvokeMethodFast which apparently tail calls the actual method, leaving no stack trace. Yikes.
I looked for more reflectiony ways of solving this - for example, I thought I could walk back to Invoke and grab the MethodInfo object, but StackFrame doesn't give you parameters - and correctly so. In a beautiful trustworthy world, being able to fully reflect a stack crawl would be awesome. In reality, if I can touch the parameters of all my callers, I can wreak all kinds of havoc. It's a security risk.
There are two other solutions - one is to pass the method name in (I opted for this for the time being). The other involves making the root test be veneer like this:
[MyVeryOwnTestMarker]
private void TestNamedFoo()
{
/* perform real test */
}
[Test]
public void TestNamedFooVeneer()
{
TestNamedFoo();
}
Then I walk back looking for MyVeryOwnTestMarkerAttribute instead of (or in addition to) TestAttribute.
Either way, it's terrible that _InvokeMethodFast doesn't leave the method's stack frame for me - that strikes me as a very serious CLR bug.
Rick pointed out that we're not the only ones who want to get the name of the currently executing test:
Stack Overflow
Andy's John's Blog