Isolator 8.0.5.2 consistently causing tests to fail that don't use Isolator

I have done further investigations and found that a couple of these issues stem from the BinaryWriter problem.

I do have another bizarre one though which I don't even know how to explain except that it has something to do with the attribute we use to mark tests. We inherit from FactAttribute in Xunit and call it TestAttribute and mark our tests with [Test]. If i replace [Test] with [Fact] it fixes the tests. Even weirder is if I create another attribute called MyFactAttribute and mark them with [MyFact] it also fixes the tests.

I have a solution exhibiting the problem. How can I get it to you?

What appears to be happening here is:

1. Typemock does't properly support async test methods with mocking, which means as soon as there is an "await", the test method returns an incomplete Task and Typemock regards the test as finished.

2. Methods are decorated with executors but are not "decorated" by default unless they have an attribute named "TestAttribute" (MethodBaseExtensions.IsTestMethod), otherwise you need to implement a Decorator and override DecorateMethodWhenAttributeIsClass.

3. Since the methods decorated with "FactAttribute" and "MyFactAttribute" are not decorated, they do not perform Isolate.Cleanup(?), which means the async method can keep running without issues after the test returns an incomplete task.

4. Since "TestAttribute" is decorated, Isolate.Cleanup *is* called, which means that the mocks established during the test are torn down before the async method has completed. Then when the async method continues execution and attempts to assert something like Isolate.WhenCalled, it throws an exception that it can't find the mocked method, causing the issue.

This is actually pretty enlightening. Previously we worked around this on async methods by manually calling Isolate.Cleanup (in a hander disposable wrapper class).

Now that I know this, I can fix this with:

 

    [AttributeUsage(AttributeTargets.Assembly | AttributeTargets.Class | AttributeTargets.Method, Inherited = false)]
    public sealed class AsyncAttribute : DecoratorAttribute
    {
        #region Methods
        protected override bool DecorateMethodWhenAttributeIsClass(MethodBase methodBase)
        {
            return true;
        }

        public override object Execute()
        {
            object result = CallDecoratedMethod();
            Task task = (result as Task);

            if (task != null)
                task.Wait();

            return result;
        }
        #endregion
    }

 

And apply it to all classes that have [Isolated]. Any problems with this approach?

Actually the above code won't work because the Isolate attribute has already called Cleanup.

Why is it so hard, TypeMock? :)

OK, so since Isolate.Cleanup can be called inside IsolateAttribute.Execute, I went ahead and did this:

[AttributeUsage(AttributeTargets.Assembly | AttributeTargets.Class | AttributeTargets.Method, Inherited = false)]
public sealed class AsyncIsolatedAttribute : IsolatedAttribute
{
	#region Static Fields
	private static readonly MethodInfo DecoratorAttributeSendTestNameToTracerMethodInfo
		= typeof(AsyncIsolatedAttribute).GetMethod("SendTestNameToTracer", BindingFlags.NonPublic | BindingFlags.Instance);

	private static readonly FieldInfo ExceptionRemoteStackStringFieldInfo
		= typeof(Exception).GetField("_remoteStackTraceString", BindingFlags.Instance | BindingFlags.NonPublic);

	private static readonly MethodInfo IsolateCleanUpMethodInfo
		= typeof(Isolate).GetMethod("CleanUp", BindingFlags.NonPublic | BindingFlags.Static);

	private static readonly MethodInfo IsolatedAttributeGetStrictFlagMethodInfo
		= typeof(IsolatedAttribute).GetMethod("GetStrictFlag", BindingFlags.NonPublic | BindingFlags.Instance);

	private static readonly MethodInfo MockManagerTestModeMethodInfo
		= typeof(MockManager).GetMethod("TestMode", BindingFlags.NonPublic | BindingFlags.Static);

	private static readonly MethodInfo SuggestBehaviorSetArmadilloTestsMethodInfo
		= typeof(MockManager).Assembly.GetType("TypeMock.SuggestBehavior").GetMethod("SetArmadilloTests", BindingFlags.Public | BindingFlags.Static);

	private static readonly List<string> TestAttributes = new List<string>
														  {
															  "TestAttribute",
															  "DataTestAttribute",
															  "FactAttribute",
															  "TheoryAttribute",
														  };
	#endregion

	#region Methods
	protected override bool DecorateMethodWhenAttributeIsClass(MethodBase methodBase)
	{
		foreach (object attribute in methodBase.GetCustomAttributes(false))
		{
			if (TestAttributes.Contains(attribute.GetType().Name))
				return true;
		}

		return base.DecorateMethodWhenAttributeIsClass(methodBase);
	}

	public override object Execute()
	{
		if (!IsTearDown)
		{
			if (!MockManager.IsInitialized)
				MockManager.Init();

			Type declaringType = OriginalMethod.DeclaringType;
			bool isArmadillo = (declaringType != null && declaringType.Assembly.IsDefined(typeof(SuggestBehaviorAttribute), false));

			SuggestBehaviorSetArmadilloTestsMethodInfo.Invoke(null, new object[] { isArmadillo });
			DecoratorAttributeSendTestNameToTracerMethodInfo.Invoke(this, null);

			if (declaringType != null)
			{
				if (declaringType.IsDefined(typeof(SafetyNetAttribute), true))
				{
					SafetyNetAttribute safetyNetAttribute = (SafetyNetAttribute)declaringType.GetCustomAttributes(typeof(SafetyNetAttribute), true)[0];

					if (safetyNetAttribute != null)
					{
						if (safetyNetAttribute.Sandbox)
							TestUtil.AssertRunningInSandbox();
					}
				}
			}
		}

		try
		{
			using ((IDisposable)MockManagerTestModeMethodInfo.Invoke(null, new[] { IsolatedAttributeGetStrictFlagMethodInfo.Invoke(this, null) }))
			{
				object result = CallDecoratedMethod();
				Task task = (result as Task);

				if (task != null)
					task.Wait();

				return result;
			}
		}
		catch (TargetInvocationException ex)
		{
			ExceptionRemoteStackStringFieldInfo.SetValue(ex.InnerException, ex.InnerException.StackTrace + Environment.NewLine);
			ExceptionDispatchInfo.Capture(ex.InnerException).Throw();
			throw;
		}
		finally
		{
			IsolateCleanUpMethodInfo.Invoke(null, new object[] { !HasTearDown || IsTearDown, IgnoreStaleMocks });
		}
	}
	#endregion
}

But then I found that this Execute isn't called. Turns out the framework only does it if there are other decorators. So then I made this:

[AttributeUsage(AttributeTargets.Assembly | AttributeTargets.Class | AttributeTargets.Method, Inherited = false)]
public sealed class EmptyDecoratorAttribute : DecoratorAttribute
{
	#region Static Fields
	private static readonly List<string> TestAttributes = new List<string>
														  {
															  "TestAttribute",
															  "DataTestAttribute",
															  "FactAttribute",
															  "TheoryAttribute",
														  };
	#endregion

	#region Methods
	protected override bool DecorateMethodWhenAttributeIsClass(MethodBase methodBase)
	{
		Validate.Argument.IsNotNull("methodBase", methodBase);

		foreach (object attribute in methodBase.GetCustomAttributes(false))
		{
			if (TestAttributes.Contains(attribute.GetType().Name))
				return true;
		}

		return base.DecorateMethodWhenAttributeIsClass(methodBase);
	}

	public override object Execute()
	{
		return CallDecoratedMethod();
	}
	#endregion
}

Now on my tests I can do:

[AsyncIsolated]
[EmptyDecorator]
[Collection("Isolated")]
public sealed class MyTests
{
	public async Task MyAsyncTest()
	{
		// Mocking and await work fine
	}
}

AsyncIsolated ensures that the Task is waited upon before Cleanup is called. EmptyDecorator ensures that AsyncIsolated is actually used. Collection is an xUnit 2 attribute that stops classes in the "Isolated" collection from running concurrently.

With this, our async tests work great in Typemock 8.0.5.2, xUnit 2, Resharper 9.1 and Visual Studio 2013.

But really TypeMock, support async by default please!

Let me know when the patch for BinaryWriter / StreamWriter are ready since we still have that problem.

Any update on that patch?

I've updated to 8.1.0.17.

First, my trick above doesn't work anymore due to the obfuscation which is annoying (little late now that 8.0.5.2 was released without it, dontchya think?), but I found another way.

Second, there are still issues with BinaryWriter at the least. We have a custom class that extends BinaryWriter. We create it and it calls the BinaryWriter constructor with the "leaveOpen" parameter set to true. This means that you can dispose the BinaryWriter but it doesn't dispose the inner stream. However, with TypeMock enabled, it always disposes the inner stream, which breaks all our IO tests that use BinaryWriter and require the stream to stay open.

Let me know if you need code to reproduce it, but it's pretty straight forward.

I see you've pulled 8.1 from download.

To be clear, this test demonstrates the problem:

    public class TypeMockTests
    {
        #region Methods
        [Fact]
        public void BinaryWriterDoesNotDisposeInnerStreamWhenLeftOpen()
        {
            MemoryStream stream = new MemoryStream();

            // Doesn't throw
            stream.Write(new byte[0], 0, 0);

            using (new BinaryWriter(stream, Encoding.UTF8, true))
            {
            }

            // Shouldn't throw
            stream.Write(new byte[0], 0, 0);
        }
        #endregion
    }

 

8.1.1.3 still has this issue.

Can someone other than me update this question with some progress / feedback please?

Hi Jahmai,

Sorry for the late reply.

We're aware of the issue that followed our fix.

We're currently working on it, I'll update you ASAP.

Thanks Alex, I'll await your update before trying new releases.

Hi Eli,

Will this fix also address the issue I mentioned in this question https://www.typemock.com/answers/19124/methodaccessexception-when-unit-test-sets-content-label-wpf?

Thanks in advance,

Carlos

Sounds great Eli, looking forward to it.

I can only download 8.1.1.9. The installer is only 1.4MB and complains about being invalid when I try to run it.

Installed 8.1.1.10.

Confirmed that both the BinaryWriter and Async test issues are fixed, but...

Regression;

public static readonly variables are null during mocking. Here is a test that demonstrates this issue:

[Isolated]
public class TypeMockTests
{
  [Fact]
  public static void PublicStaticReadOnlyAreNotNull()
  {
      // Faked.Instance.Method() throws NullReferenceException due to Faked.Instance being null
      Isolate.WhenCalled(() => Faked.Instance.Method()).IgnoreCall();
  }

  private class Faked
  {
    public static readonly Faked Instance = new Faked();

    public void Method()
    {
    }
  }
}

Here is the output of the test:

TypeMock.TypeMockException

*** No method calls found in recording block. Please check:

 * Are you trying to fake a field instead of a property?

 * Are you are trying to fake an unsupported mscorlib type? See supported types here: https://www.typemock.com/mscorlib-types

   at _4fOddA7JgGYbOOebRdG4fQREBGkb_._aNtnIwgxjLwUpminjJDGdVtHBTH_._eAtw4rxkQILc50tjMfFmPOUwIJo_(Boolean)

   at _4fOddA7JgGYbOOebRdG4fQREBGkb_._nzVsxF4j04UocypgyEFRFXQby0w_._R6XuDLwlccMpAxcdUJbImOBbMW4b_(Boolean)

   at _4fOddA7JgGYbOOebRdG4fQREBGkb_._nzVsxF4j04UocypgyEFRFXQby0w_._3P6pfP7t5yKdJIrbbfQKhqsDfAu_(Object, Boolean, Func`1, Action, Action, Action, Boolean)

   at _4fOddA7JgGYbOOebRdG4fQREBGkb_._nzVsxF4j04UocypgyEFRFXQby0w_._3P6pfP7t5yKdJIrbbfQKhqsDfAu_(Object, Boolean, Action, Action)

   at _4fOddA7JgGYbOOebRdG4fQREBGkb_._nzVsxF4j04UocypgyEFRFXQby0w_.IgnoreCall()

   at TypeMock.MockManager.getReturn(Object context, String typeName, String methodName, Object methodGenericParams, Boolean isDecorated, Boolean isInterceptedType, Object[] methodArguments)

   at Typemock.Interceptors.Profiler.InternalMockManager.getReturn(Object that, String typeName, String methodName, Object methodParameters, Boolean isInjected, Boolean isInterceptedType)

   at Intrinsic.TypeMock.TypeMockTests.PublicStaticReadOnlyAreNotNull() in TypeMockTests.cs: line 41

System.NullReferenceException occurred

  _HResult=-2147467261

  _message=Object reference not set to an instance of an object.

  HResult=-2147467261

  IsTransient=false

  Message=Object reference not set to an instance of an object.

  Source=Intrinsic.Core.Tests

  StackTrace:

       at Intrinsic.TypeMock.TypeMockTests.<PublicStaticReadOnlyAreNotNull>b__6() in c:RepositoryProjectsCriticalArcDependenciesIntrinsicTestsLibrariesCoreTypeMockTypeMockTests.cs:line 41

  InnerException: 

 

This problem still exists in 8.1.1.11

Can someone please update this question with some progress? TypeMock has been royally broken for us for a few weeks now.

Sorry for the late reply.

We're currently working on it, I'll update you ASAP.