logo1

logo

Noninvasive Unit Testing in ASP.NET MVC4 – A Microsoft Fakes Deep Dive

59 Comments
Posted in Unit Testing

A lot of today’s unit testing technologies require significant invasive code changes in order to unit test appropriately. I’ve always been of the mindset that testing your code should be as noninvasive as possible to the system under test, regardless of how that system is designed.

The ability to test a system shouldn’t be dependent on whether or not that system was designed to be compatible with a certain set of testing tools. The design of systems should instead be driven by the needs of the problem domain, while complexity in applied patterns and concept count should only be escalated as it becomes necessary to do so.

In combining the KISS principle and YAGNI with agile architecture you get an architectural design, that at any given point in time, is the simplest to use, easiest to work with and the most maintainable as is allowable or possible in the problem domain in question.

The KISS principle states that most systems work best if they are kept simple rather than made complex, therefore simplicity should be a key goal in design and unnecessary complexity should be avoided. – Wikipedia

"You ain't gonna need it" or “You aren′t gonna need it” (acronym: YAGNI) is the principle in extreme programming that programmers should not add functionality until it is necessary. – Wikipedia

Up until recently the ability to test without escalation of certain architectural patterns was not possible. Even if you wanted to practice a noninvasive testing style, the means to do so as well as community support weren’t generally available.

Conceptually, noninvasive testing tools and methodologies represent a natural progression/evolution in unit testing practices and principles. This is most visible in the evolution of major commercial testing tools, such as TypeMock’s Isolator and Telerik’s JustMock, that now have features to test/mock everything, not just interfaces and base classes. Now, with the introduction of Microsoft Fakes in Visual Studio 11 (more specifically the ability to detour via shimming), we are given all the tools necessary to accomplish noninvasive unit testing built right in to our development environment.

See my earlier blog posts on Microsoft Fakes for more background information: Using Stubs and Shims to Test with Microsoft Fakes in Visual Studio 11 and Comparing Microsoft Moles in VS2010 to Microsoft Fakes in VS11

Additionally Fakes allows us to take the “mockist” approach of behavior verification described by Martin Fowler in his article Mocks Aren’t Stubs.

But as often as not I see mock objects described poorly. In particular I see them often confused with stubs - a common helper to testing environments. I understand this confusion - I saw them as similar for a while too, but conversations with the mock developers have steadily allowed a little mock understanding to penetrate my tortoiseshell cranium.

This difference is actually two separate differences. On the one hand there is a difference in how test results are verified: a distinction between state verification and behavior verification. On the other hand is a whole different philosophy to the way testing and design play together, which I term here as the classical and mockist styles of Test Driven Development.

Later on in his article Martin provides a more concrete example as he discusses the differences.

The key difference here is how we verify that the order did the right thing in its interaction with the warehouse. With state verification we do this by asserts against the warehouse's state. Mocks use behavior verification, where we instead check to see if the order made the correct calls on the warehouse. We do this check by telling the mock what to expect during setup and asking the mock to verify itself during verification. Only the order is checked using asserts, and if the the method doesn't change the state of the order there's no asserts at all.

With that said I’ll be using Microsoft Fakes to apply noninvasive and mockist testing techniques to test the AccountController of a default MVC 4 project created using the "Internet Application” template, making absolutely no changes at all to the project. This example will use a mixture of both shimming and stubbing from Microsoft Fakes in order to get the job done.

Getting Started

Let’s take a quick look at the class definition for AccountController.

MethodsToTest

Looking through the implementations, a few of the methods are trivial enough for us to skip as part of this example.

public ActionResult ChangePassword() { return View(); }

public ActionResult ChangePasswordSuccess() { return View(); }

[AllowAnonymous]
public ActionResult Login() { return ContextDependentView(); }

[AllowAnonymous]
public ActionResult Register() { return ContextDependentView(); }

Additionally, we’re going to forego testing ContextDependentView, GetErrorsFromModelState & ErrorCodeToString in favor of the more complex methods. That’s not to say you wouldn’t test these methods for appropriate coverage, just that we’re going to exclude them to keep this post somewhat reasonable in length.

Before we get started though, we need to do some basic project setup. 

  • Create a new ASP.NET MVC 4 Application project using the Internet Application template
  • Add a Unit Test Project, I renamed the default cs to AccountsControllerTests
  • Add references to the following items in the Unit Test project
    • the MVC 4 project
    • System.Web
    • System.Web.MVC
  • Additionally I’ll be using NUnit for assertions, so pull down NUnit from Nuget and add the following using statement to the top of the AccountsControllerTests file:
using Assert = NUnit.Framework.Assert;

After all that your solution should look something like this:

InitialSolutionConfig

LogOff Method

We’ll start off with the LogOff method (as seen below), since this is one of the simpler methods we’re going to be looking at.

public ActionResult LogOff()
{
    FormsAuthentication.SignOut();

    return RedirectToAction("Index", "Home");
}

First off, let’s review our goals here. Since our intent with mocking is behavior verification, we want to test both that the correct RedirectToAction was returned and that FormsAuthentication.SignOut() was called. Testing that the correct RedirectToAction was returned seems easy enough, so we’ll start with that.

using System;
using System.Web.Mvc;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using NoninvasiveMVC4Testing.Controllers;
using Assert = NUnit.Framework.Assert;

namespace NoninvasiveMVC4Testing.Tests
{
    [TestClass]
    public class AccountsControllerTests
    {
        [TestMethod]
        public void TestLogOff()
        {
            var accountController = new AccountController();
            var redirectToRouteResult = accountController.LogOff() as RedirectToRouteResult;

            Assert.NotNull(redirectToRouteResult);
            Assert.AreEqual("Index", redirectToRouteResult.RouteValues["Action"]);
            Assert.AreEqual("Home", redirectToRouteResult.RouteValues["controller"]);
        }
    }
}

We get the following results when running our new unit test:

FailedInitialUnitTest

Looking at the stack trace we can see that a NullReferenceException was thrown from FormsAuthentication.SignOut(). This makes sense as technically we’re not in the context of an actual web request and FormsAuthentication depends on a valid HttpContext to be available. This type of problem is common when testing web applications outside of the context of an actual request to a web server.

The traditional guidance on how to test something like this is as follows (see this StackOverflow post for more information):

  • Create a wrapping class around FormsAuthentication with a public method that runs the necessary method
  • Create an interface for this behavior
  • Use dependency injection in our controller to replace the direct call to FormsAuthentication with that of our wrapping class.

Using this formula, our controller code (not the test code mind you) would have to be changed as follows:

public interface IAuthenticationProvider
{
    void SignOut();
}

public class FormsAuthWrapper : IAuthenticationProvider
{
    public void SignOut()
    {
        FormsAuthentication.SignOut();
    }
}

public class AccountController : Controller
{
    private readonly IAuthenticationProvider _authenticationProvider;

    public AccountController(IAuthenticationProvider authenticationProvider)
    {
        _authenticationProvider = authenticationProvider;
    }

    public ActionResult LogOff()
    {
        _authenticationProvider.SignOut();
        return RedirectToAction("Index", "Home");
    }
}

As you can see this pattern is invasive to the system under test and with the purpose of this post being to apply noninvasive testing techniques, we’re going to consider a different way of testing using Microsoft Fakes. Surprisingly enough, it makes short work of these types of scenarios.

The Noninvasive Approach

Let’s start off by putting in what’s minimally necessary to get our test to pass as is. Right click on the System.Web reference in the test project and select Add Fakes Assembly. Once a Fakes assembly is added for System.Web we can use shims in Microsoft Fakes to detour the call to FormsAuthentication.SignOut() to an implementation of our choosing, hopefully one that won’t throw a NullReferenceException.

using System;
using System.Web.Mvc;
using System.Web.Security.Fakes;
using Microsoft.QualityTools.Testing.Fakes;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using NoninvasiveMVC4Testing.Controllers;
using Assert = NUnit.Framework.Assert;

namespace NoninvasiveMVC4Testing.Tests
{
    [TestClass]
    public class AccountsControllerTests
    {
        [TestMethod]
        public void TestLogOff()
        {
            var accountController = new AccountController();
            RedirectToRouteResult redirectToRouteResult;

            //Scope the detours we're creating
            using (ShimsContext.Create())
            {
                //Detours FormsAuthentication.SignOut() to an empty implementation
                ShimFormsAuthentication.SignOut = () => { };
                redirectToRouteResult = accountController.LogOff() as RedirectToRouteResult;
            }

            Assert.NotNull(redirectToRouteResult);
            Assert.AreEqual("Index", redirectToRouteResult.RouteValues["Action"]);
            Assert.AreEqual("Home", redirectToRouteResult.RouteValues["controller"]);
        }
    }
}

That’s simple enough and it does indeed pass.

TestLogOffInitialSuccess

We still have to test that FormsAuthentication.SignOut() was actually called. All we have to do is flip a bit inside of the detoured SignOut method (see lines 28 and 31) and assert it. Here’s the final method.

[TestMethod]
public void TestLogOff()
{
    var accountController = new AccountController();
    var formsAuthenticationSignOutCalled = false;
    RedirectToRouteResult redirectToRouteResult;

    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Detours FormsAuthentication.SignOut() to an empty implementation
        ShimFormsAuthentication.SignOut = () =>
        {
            //Set a boolean to identify that we actually got here
            formsAuthenticationSignOutCalled = true;
        };
        redirectToRouteResult = accountController.LogOff() as RedirectToRouteResult;
        Assert.AreEqual(true, formsAuthenticationSignOutCalled);
    }

    Assert.NotNull(redirectToRouteResult);
    Assert.AreEqual("Index", redirectToRouteResult.RouteValues["Action"]);
    Assert.AreEqual("Home", redirectToRouteResult.RouteValues["controller"]);
}

Testing that FormsAuthentication was called seems pretty trivial here, however we’ll build on this approach as we test more and more complicated methods.

JsonLogin

Moving on, JsonLogin is probably the next simplest method to test in order for us to ease our way into noninvasive testing with Fakes.

[AllowAnonymous]
[HttpPost]
public JsonResult JsonLogin(LoginModel model, string returnUrl)
{
    if (ModelState.IsValid)
    {
        if (Membership.ValidateUser(model.UserName, model.Password))
        {
            FormsAuthentication.SetAuthCookie(model.UserName, model.RememberMe);
            return Json(new { success = true, redirect = returnUrl });
        }
        else
            ModelState.AddModelError("", "The user name or password provided is incorrect.");
    }

    // If we got this far, something failed
    return Json(new { errors = GetErrorsFromModelState() });
}

Right off the bat, it’s pretty clear that MemberShip.ValidateUser and FormsAuthentication.SetAuthCookie will need to be detoured based on our prior experience with the LogOff method. We’ll additionally test that the correct parameters were passed into each.

[TestMethod]
public void TestJsonLogin()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    bool testRememberMe = false;
    string testReturnUrl = "TestReturnUrl";

    var loginModel = new LoginModel
    {
        UserName = testUserName,
        Password = testPassword,
        RememberMe = testRememberMe
    };

    var accountController = new AccountController();
    JsonResult jsonResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.ValidateUser to our mocked implementation
        ShimMembership.ValidateUserStringString = (userName, password) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testPassword, password);
            return true;
        };

        //Sets up a detour for FormsAuthentication.SetAuthCookie to our mocked implementation
        ShimFormsAuthentication.SetAuthCookieStringBoolean = (userName, rememberMe) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testRememberMe, rememberMe);
        };

        jsonResult = accountController.JsonLogin(loginModel, testReturnUrl);
    }
}

Now on to the tricky part, testing JsonResult. JsonResult.Data is of type Object, but is filled with an anonymous type.

JsonResultProperties

return Json(new { success = true, redirect = returnUrl });

This makes it slightly more difficult to get at the properties we want to test.

Possible solutions

First off, we might try to cast JsonResult.Data out of Object into some type we could use to access the fields. This requires some runtime trickery and ends up being a bit of mess. See this StackOverflow post for more info.

private static T CastTo<T>(this Object value, T targetType) 
{ 
    // targetType above is just for compiler magic 
    // to infer the type to cast x to 
    return (T)x; 
}

Unfortunately, this only works if you’re working within the same assembly that defined the original anonymous type.

Next up, we could cleverly stuff a dynamic type with the value from JsonResult.Data and access the properties that way.

dynamic data = jsonResult.Data;
Assert.AreEqual(true, data.success);
Assert.AreEqual(testReturnUrl, data.redirect);

This fails as well, since anonymous types are declared as internal as described by the blog post: Anonymous Types are Internal, C# 4.0 Dynamic Beware!.

JsonLoginDynamicFail

Even though the dynamic data variable has the success property, we don’t have access to it. We could use the assembly attribute InternalsVisibleTo in order to give our testing project access to internal types.

[assembly: InternalsVisibleTo("NoninvasiveMVC4Testing.Tests")]

I don’t consider this to be a bad technique, however since we’re trying to be completely noninvasive, I’m going to opt for a slightly different approach.

We’ll use PrivateObject (MSDN Link) to get at the properties. PrivateObject’s MSDN description:

Allows test code to call methods and properties on the code under test that would be inaccessible because they are not public.

PrivateObject ultimately just uses reflection in order to expose the values we need to test.  The real value is in the fact that it abstracts the reflection code away from us. Here’s the code updated with PrivateObject:

var success = (bool)(new PrivateObject(jsonResult.Data, "success")).Target;
var redirect = (string)(new PrivateObject(jsonResult.Data, "redirect")).Target;

Assert.AreEqual(true, success);
Assert.AreEqual(testReturnUrl, redirect);

And with that we now have successful tests

JsonLoginTestsSuccess

Just for completeness, I’ve put together a test to validate the behavior of an invalid login.

[TestMethod]
public void TestInvalidJsonLogin()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    bool testRememberMe = false;
    string testReturnUrl = "TestReturnUrl";

    var loginModel = new LoginModel
    {
        UserName = testUserName,
        Password = testPassword,
        RememberMe = testRememberMe
    };

    var accountController = new AccountController();
    JsonResult jsonResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.ValidateUser to our mocked implementation
        ShimMembership.ValidateUserStringString = (userName, password) => false;
        jsonResult = accountController.JsonLogin(loginModel, testReturnUrl);
    }

    var errors = (IEnumerable<string>)(new PrivateObject(jsonResult.Data, "errors")).Target;
    Assert.AreEqual("The user name or password provided is incorrect.", errors.First());
}

And there we go, easy as pie.

LogOff_ValidLogin_InvalidLogin_Tests

For the remainder of the post, I’m just going to focus on the “happy” path for brevity. Testing the other paths is relatively straight forward given what we’ve already done.

Login Method

Stepping up in complexity we move on to the Login method.

[AllowAnonymous]
[HttpPost]
public ActionResult Login(LoginModel model, string returnUrl)
{
    if (ModelState.IsValid)
    {
        if (Membership.ValidateUser(model.UserName, model.Password))
        {
            FormsAuthentication.SetAuthCookie(model.UserName, model.RememberMe);
            if (Url.IsLocalUrl(returnUrl))
                return Redirect(returnUrl);
            else
                return RedirectToAction("Index", "Home");
        }
        else
            ModelState.AddModelError("", "The user name or password provided is incorrect.");
    }

    // If we got this far, something failed, redisplay form
    return View(model);
}

MemberShip.ValidateUser and FormsAuthentication.SetAuthCookie are easy enough to test via Shimming. Under normal circumstances Url.IsLocalUrl would be simple to Shim as well. Unfortunately I ran into an issue when faking the System.Web.Mvc assembly containing it. Once you try to instantiate a controller (as part of your test project) after adding a Fakes assembly you get a System.Security.VerificationException: Operation could destabilize the runtime. See my Microsoft Connect submission for more info.

Fortunately enough, there’s a way to mock its implementation using the stubs portion of Microsoft Fakes as opposed to shims. This brings up an interesting dilemma, if a compatible stubbing technique is available should you use that instead of shimming?

I would say the answer is generally “yes” provided that these criteria are met:

  • It doesn’t significantly decrease the readability of the test
  • It doesn’t require excessive measures (such as reflection dumpster diving) to figure out how to do it

Stubbing Around System.Web.Mvc

The first problem we need to solve is that the Url property (of type UrlHelper) is null on our instance of AccountController. The ctor on UrlHelper requires a RequestContext. The ctor on RequestContext requires an HttpContextBase. Since HttpContextBase is an abstract class we can stub it easily and make our way back up the dependency hierarchy.

Decompiling UrlHelper with ILSpy shows us that we’ll need to stub one more item in order to avoid the dreaded NullReferenceException.

public bool IsLocalUrl(string url)
{
    return this.RequestContext.HttpContext.Request.IsUrlLocalToHost(url);
}

We need to make sure that the Request property on HttpContextBase returns a value. As luck would have it the Request property is of type HttpRequestBase and we can easily stub it as well.

[TestMethod]
public void TestLogin()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    bool testRememberMe = false;
    string returnUrl = "/foo.html";

    var loginModel = new LoginModel
    {
        UserName = testUserName,
        Password = testPassword,
        RememberMe = testRememberMe
    };

    var accountController = new AccountController();

    //Setup underpinning via stubbing such that UrlHelper 
    //can validate that our "foo.html" is local
    var stubHttpContext = new StubHttpContextBase();
    var stubHttpRequestBase = new StubHttpRequestBase();
    stubHttpContext.RequestGet = () => stubHttpRequestBase;
    var requestContext = new RequestContext(stubHttpContext, new RouteData());
    accountController.Url = new UrlHelper(requestContext);

    RedirectResult redirectResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.ValidateUser to our mocked implementation
        ShimMembership.ValidateUserStringString = (userName, password) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testPassword, password);
            return true;
        };

        //Sets up a detour for FormsAuthentication.SetAuthCookie to our mocked implementation
        ShimFormsAuthentication.SetAuthCookieStringBoolean = (userName, rememberMe) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testRememberMe, rememberMe);
        };

        redirectResult = accountController.Login(loginModel, returnUrl) as RedirectResult;
    }

    Assert.NotNull(redirectResult);
    Assert.AreEqual(redirectResult.Url, returnUrl);
}

With that, let’s run our tests and make sure everything is working.

TestRunAfterLoginMethod

One thing to notice here, is that the stubbing we’re doing (starting line 20 and continuing to line 27) doesn’t exactly convey what we’re trying to accomplish. All we care about doing is getting Url.IsLocalUrl to return true. Additionally, we had to know quite a bit about the internals of a Controller, UrlHelper, HttpContextBase, HttpRequestBase just to get this behavior to work.

In this scenario it would be preferable, readability wise, just to set and detour Url.IsLocalUrl. In this case our hand was forced since Microsoft Fakes and System.Web.Mvc aren’t currently cooperating, so I’m more than happy that at least a fallback was available.

JsonRegister Method

Both JsonRegister and Register are very similar, so we’ll just hit one of them. There’s really no new concepts here, just reapplying the what we used to test earlier methods.

[AllowAnonymous]
[HttpPost]
public ActionResult JsonRegister(RegisterModel model)
{
    if (ModelState.IsValid)
    {
        // Attempt to register the user
        MembershipCreateStatus createStatus;
        Membership.CreateUser(model.UserName, model.Password, model.Email, 
            passwordQuestion: null, passwordAnswer: null, isApproved: true, 
            providerUserKey: null, status: out createStatus);

        if (createStatus == MembershipCreateStatus.Success)
        {
            FormsAuthentication.SetAuthCookie(model.UserName, createPersistentCookie: false);
            return Json(new { success = true });
        }
        else
            ModelState.AddModelError("", ErrorCodeToString(createStatus));
    }

    // If we got this far, something failed
    return Json(new { errors = GetErrorsFromModelState() });
}

For JsonRegister we’ll need to shim Membership.CreateUser, which is straightforward enough. We’ll need to add a reference to System.Web.Security.ApplicationServices to our testing project for to work with MembershipCreateStatus and we’re good to go.

[TestMethod]
public void TestJsonRegister()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    string testConfirmPassword = "TestPassword";
    string testEmail = "TestEmail@Test.com";

    var registerModel = new RegisterModel
    {
        UserName = testUserName,
        Password = testPassword,
        ConfirmPassword = testConfirmPassword,
        Email = testEmail
    };

    var accountController = new AccountController();
    JsonResult jsonResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.CreateUser to our mocked implementation
        ShimMembership.CreateUserStringStringStringStringStringBooleanObjectMembershipCreateStatusOut =
            (string userName, string password, string email, string passwordQuestion, 
                string passwordAnswer, bool isApproved, object providerUserKey,
                out MembershipCreateStatus @createStatus) =>
            {
                Assert.AreEqual(testUserName, userName);
                Assert.AreEqual(testPassword, password);
                Assert.AreEqual(testEmail, email);
                Assert.Null(passwordQuestion);
                Assert.Null(passwordAnswer);
                Assert.True(isApproved);
                Assert.Null(providerUserKey);
                @createStatus = MembershipCreateStatus.Success;

                return null;
            };

        //Sets up a detour for FormsAuthentication.SetAuthCookie to our mocked implementation
        ShimFormsAuthentication.SetAuthCookieStringBoolean = (userName, rememberMe) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(false, rememberMe);
        };

        var actionResult = accountController.JsonRegister(registerModel);
        Assert.IsInstanceOf(typeof(JsonResult), actionResult);
        jsonResult = actionResult as JsonResult;
    }

    Assert.NotNull(jsonResult);
    var success = (bool)(new PrivateObject(jsonResult.Data, "success")).Target;
    Assert.True(success);
}

Running our tests once again, we can see everything’s passing

TestRunAfterJsonRegisterMethod

ChangePassword Method

The ChangePassword method is slightly more difficult to work with, since we have additional items to fake and stub, but otherwise the concepts are pretty similar.

[HttpPost]
public ActionResult ChangePassword(ChangePasswordModel model)
{
    if (ModelState.IsValid)
    {

        // ChangePassword will throw an exception rather
        // than return false in certain failure scenarios.
        bool changePasswordSucceeded;
        try
        {
            MembershipUser currentUser = Membership.GetUser(User.Identity.Name, 
                userIsOnline: true);
            changePasswordSucceeded = currentUser.ChangePassword(model.OldPassword, 
                model.NewPassword);
        }
        catch (Exception)
        {
            changePasswordSucceeded = false;
        }

        if (changePasswordSucceeded)
            return RedirectToAction("ChangePasswordSuccess");
        else
            ModelState.AddModelError("", "The current password is incorrect or the new password is invalid.");
    }

    // If we got this far, something failed, redisplay form
    return View(model);
}

We need to make sure that User.Identity.Name returns properly. In order to do this, we’re going to have to make sure AccountController’s User property gets populated with an Identity object. Again, due to the MVC faking issue, we’re going to approach this via stubbing, which is slightly less readable and requires some framework dumpster diving, but still gets the job done.

Decompiling down into the Controller class in System.Web.Mvc to see what we need to stub show the following:

public IPrincipal User
{
    get
    {
        if (this.HttpContext != null)
        {
            return this.HttpContext.User;
        }
        return null;
    }
}

Drilling in this.HttpContext

public HttpContextBase HttpContext
{
    get
    {
        if (base.ControllerContext != null)
        {
            return base.ControllerContext.HttpContext;
        }
        return null;
    }
}

The ControllerContext property is settable, so that’s our way in and it has a public ctor taking elements we already have. Additionally, we already have a StubHttpRequestBase which we can set the User property on.

We’ll need to add a Fakes Assembly for mscorlib in order to stub an IPrincipal for the AccountController’s User property. To add a Fakes assembly for mscorlib, add one for the System reference.  System.Web.ApplicationServices needs a Fakes assembly as well in order to shim the ChangePassword method on MembershipUser.

[TestMethod]
public void TestChangePassword()
{
    string testUserName = "TestUserName";
    string testOldPassword = "TestOldPassword";
    string testNewPassword = "TestNewPassword";

    var changePasswordModel = new ChangePasswordModel
    {
        OldPassword = testOldPassword,
        NewPassword = testNewPassword
    };

    var accountController = new AccountController();

    //Stub HttpContext
    var stubHttpContext = new StubHttpContextBase();
    //Setup ControllerContext so AccountController will use our stubHttpContext
    accountController.ControllerContext = new ControllerContext(stubHttpContext, 
        new RouteData(), accountController);

    //Stub IPrincipal
    var principal = new StubIPrincipal();
    principal.IdentityGet = () =>
    {
        var identity = new StubIIdentity { NameGet = () => testUserName };
        return identity;
    };
    stubHttpContext.UserGet = () => principal;

    RedirectToRouteResult redirectToRouteResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        ShimMembership.GetUserStringBoolean = (identityName, userIsOnline) =>
        {
            Assert.AreEqual(testUserName, identityName);
            Assert.AreEqual(true, userIsOnline);

            var memberShipUser = new ShimMembershipUser();
            //Sets up a detour for MemberShipUser.ChangePassword to our mocked implementation
            memberShipUser.ChangePasswordStringString = (oldPassword, newPassword) =>
            {
                Assert.AreEqual(testOldPassword, oldPassword);
                Assert.AreEqual(testNewPassword, newPassword);
                return true;
            };
            return memberShipUser;
        };

        var actionResult = accountController.ChangePassword(changePasswordModel);
        Assert.IsInstanceOf(typeof(RedirectToRouteResult), actionResult);
        redirectToRouteResult = actionResult as RedirectToRouteResult;
    }
    Assert.NotNull(redirectToRouteResult);
    Assert.AreEqual("ChangePasswordSuccess", redirectToRouteResult.RouteValues["Action"]);
}

After running tests, we see that our new unit test is passing.

TestRunAfterChangePasswordMethod

Conclusions

Through the use of Microsoft Fakes and the idea of noninvasive testing, with the mockist approach, we’ve been able to test the AccountController quite thoroughly without any project modifications. I imagine we could of easily hit 100% coverage if that was our goal. The only real issues we ran into were related to beta software.

Oddly enough, I’m glad we ran into the System.Web.Mvc faking issue. This forced us to use stubbing, and ultimately exposed both negative effects on overall readability and increased complexity in terms of the amount of framework decompiling needed to figure out what stubbing was necessary. Shimming in these cases would of better conveyed our intent and abstracted us away from having to deal with the guts of the underlying framework.

With these results in mind, it’s evident that testing tools have truly reached a point where anything can be tested, regardless of design. We’re entering a time where ANY application with ANY architecture can be thoroughly unit tested without even the slightest change to code; a time when the ability to unit test a system is decoupled from the design and architecture of that system.

All of this is for good reason. Today’s testing patterns and practices have arisen from limitations in our capabilities to isolate dependencies when unit testing code. Those limitations have been addressed, it’s time to reevaluate our approaches and move on.

When you drive architecture with the goal of being structurally easier to test, the only thing you end up with is an architecture that is good at being tested. Let architecture be naturally shaped by the needs of the problem domain over time. Let complexity escalate only as needed and simplicity, maintainability and ease of use all be key goals in a system’s design.

From now on, we can definitively say that any constraints or limitations in our abilities to thoroughly test any system with any design, are entirely self imposed.

 

The code for this post is available on GitHub

  • 59 Comments

Using Stubs and Shims to Test with Microsoft Fakes in Visual Studio 11

16 Comments
Posted in Unit Testing

Microsoft Fakes is a full featured mocking framework built into Visual Studio 11. Currently, the available documentation is limited and marked as “preview only”, however it does provide us with some very good descriptions.

Microsoft Fakes is an isolation framework for creating delegate-based test stubs and shims in .NET Framework applications. The Fakes framework can be used to shim any .NET method, including non-virtual and static methods in sealed types.

Additionally here is Wikipedia’s definition of Mock Object for reference

In object-oriented programming, mock objects are simulated objects that mimic the behavior of real objects in controlled ways. A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts.

Shim any .NET Method

The real interesting part here is that Microsoft Fakes can shim “any .NET method, including non-virtual and static methods in sealed types”.  Existing mocking frameworks work by providing “on the fly” implementations to preexisting interfaces with a bit of Dependency Injection to get them situated in the class/method under test.

Microsoft Fakes is going a level deeper by mocking objects with no preexisting interfaces and allowing statics to be mocked in the process. This type of mocking is quite complex to pull off.  So much so, that there are only a handful of products out there that do this.  TypeMock’s Isolator, Telerik’s JustMock and the Microsoft Research project Moles that ultimately lead to Microsoft Fakes. 

TypeMock’s Isolator has been a regular so to speak in unit testing circles for a while now and I’ve actively considered it before, but the starting price of $799 has always been a little steep for my tastes. Telerik’s JustMock is the new kid on the block (past 2-3 years). JustMock has freemium edition supporting general mocking features and a pay version which has support for static, sealed & non virtual mocking with a base price of $299.

The biggest surprise is the Microsoft Research project Moles, which seems to have about the same set of features as Microsoft Fakes, has apparently been available since 11/1/2010 and is compatible with both Visual Studio 2010 & 2008 (based on the Visual Studio 2010 Moles x86 - Isolation Framework for .NET download page).  Perhaps this is the internet’s best kept secret… well, maybe not.  Regardless, I’ll be creating a future blog post going over using Moles in Visual Studio 2010.

Architectural Implications

Without a commercial mocking framework or Microsoft Fakes, if you’re going to do unit testing and you want to do it right as in truly isolate the code under test then you’re going to need to mock dependent objects.  However, just mocking the objects isn’t enough, you also need to instruct the code under test to use the mocks as opposed to its normal implementation.  Typically this requires some form of Dependency Injection. 

For reference here is Wikipedia's definition of Dependency Injection:

Dependency injection (DI) is a design pattern in object-oriented computer programming whose purpose is to reduce the coupling between software components. It is similar to the factory method pattern. Frequently an object uses (depends on) work produced by another part of the system. With DI, the object does not need to know in advance about how the other part of the system works. Instead, the programmer provides (injects) the relevant system component in advance along with a contract that it will behave in a certain way.

To truly isolate code in unit tests you’re typically bound to the following practices as part of Dependency Injection:

  • Any object that needs to be mocked needs to be non static and have an interface
  • You’ll either need to have a Dependency Injection framework as part of the architecture or the right patterns in place to manually resolve and inject as needed
  • Constructors will have to be modified to take in interface types for the dependent objects

That’s a lot of Architectural commitment.  As with any design pattern there are “trade offs” involved in usage.  With Dependency Injection you get reduced coupling and the ability to unit test thoroughly, increasing complexity and reducing maintainability by some level in the process.  The question with any design pattern’s usage is: are the trade offs worth it based on the functionality provided for the problem domain in question? 

Microsoft Fakes changes the trade off evaluation for Dependency Injection.  Currently you trade increased complexity and reduced maintainability for increased code quality brought about by unit testing. 

Microsoft Fakes decouples Dependency Injection from unit testing, therefore changing the value proposition for Dependency Injection to instead be weighed on its own merits of whether or not it adds value to the problem domain in question.

Proper unit testing can now be integrated into any codebase, legacy or new, small or large, and using Dependency Injection or not.  With this in mind Microsoft Fakes is a game changer.  Why a game changer now, presumably years after these features have been available in the wild? 

The answer is widespread availability and acceptance.  Anyone running Visual Studio will be able to run this (hopefully this will be part of the bare SDK as well).  Additionally examples, documentation, blogs and training will be much more readily available.  Faking may very well become a recommended practice along with unit testing at which point it could become as pervasive as general unit testing in .NET solutions.  It’s a lot easier to justify its usage when everyone has access to it, not just those willing to shell out $299 to $799 a seat.

Shims vs Stubs

Let’s take a look at the current MSDN definition for Shims and Stubs

Stub types Stub types make it easy to test code that consumes interfaces or non-sealed classes with overridable methods. A stub of the type T provides a default implementation of each virtual member of T, that is, any non-sealed virtual or abstract method, property, or event. The default behavior can be dynamically customized for each member by attaching a delegate to a corresponding property of the stub. A stub is realized by a distinct type which is generated by the Fakes Framework. As a result, all stubs are strongly typed.

Although stub types can be generated for interfaces and non-sealed classes with overridable methods, they cannot be used for static or non-overridable methods. To address these cases, the Fakes Framework also generates shim types.

Shim types Shim types allow detouring of hard-coded dependencies on static or non-overridable methods. A shim of type T can provide an alternative implementation for each non-abstract member of T. The Fakes Framework will redirect method calls to members of T to the alternative shim implementation. The shim types rely on runtime code rewriting that is provided by a custom profiler.

Delegates Both stub types and shim types allow you to use delegates to dynamically customize the behavior of individual stub members.

Stub types appear to be what most free and/or open source mocking frameworks already do for us today, so there’s really no surprises here.  Shim types on the other hand are where the action is at allowing us to detour static or non-overridable methods. 

MSDN mentions that the detouring in shims does degrade performance slightly.  In the brief amount of testing I’ve done there’s about a 20ms difference between shimmed vs stubbed implementation of the same code, which isn’t a deal breaker in terms of keeping tests fast.  It would make sense the cost could be additive based on the amount of detouring you end up doing.  So if you have the infrastructure to do either or, then pick stubbing over shimming. 

Getting Started with Shims

First off, you’ll need Visual Studio 11, fire it up and create a new C# Class Library project called FakingExample.  For this example I’ve put together a very trivial cart example to play with.

Rename the default class file to CartToShim and modify the code to be as follows:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace FakingExample
{
    public class CartToShim
    {
        public int CartId { get; private set; }
        public int UserId { get; private set; }
        private List<CartItem> _cartItems = new List<CartItem>();
        public ReadOnlyCollection<CartItem> CartItems { get; private set; }
        public DateTime CreateDateTime { get; private set; }

        public CartToShim(int cartId, int userId)
        {
            CartId = cartId;
            UserId = userId;
            CreateDateTime = DateTime.Now;
            CartItems = new ReadOnlyCollection<CartItem>(_cartItems);
        }

        public void AddCartItem(int productId)
        {
            var cartItemId = DataAccessLayer.SaveCartItem(CartId, productId);
            _cartItems.Add(new CartItem(cartItemId, productId));
        }
    }
}

Add the CartItem class.

namespace FakingExample
{
    public class CartItem
    {
        public int CartItemId { get; private set; }
        public int ProductId { get; private set; }

        public CartItem(int cartItemId, int productId)
        {
            CartItemId = cartItemId;
            ProductId = productId;
        }
    }
}

Add the DataAccessLayer class. Don’t worry about the connection string, we’ll never end up hitting it anyway.

using System.Data;
using System.Data.SqlClient;

namespace FakingExample
{
    public static class DataAccessLayer
    {
        public static int SaveCartItem(int cartId, int productId)
        {
            using (var conn = new SqlConnection("RandomSqlConnectionString"))
            {
                var cmd = new SqlCommand("InsCartItem", conn);
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.Parameters.AddWithValue("@CartId", cartId);
                cmd.Parameters.AddWithValue("@ProductId", productId);

                conn.Open();
                return (int)cmd.ExecuteScalar();
            }
        }
    }
}

Next add a Unit Test Project called FakingExample.Tests and rename the default unit test class file to CartToShimTests.

In the FakingExample.Tests project add a reference to the FakingExample project.  You’re solution should look as follows:

InitialSolutionConfig

Right click on the FakingExample reference in FakingExample.Tests references and select the “Add Fakes Assembly”.

AddFakesAssembly

This adds a couple of new items to the project.

AfterFakes

The new references add the following types.

NewTypes

The .fakes file created under the Fakes folder turns out to be an xml file.  The file provides a configuration file for the generation of the fakes assembly. I’ll cover more about what can be done in this file later on in the post.  For now here’s a look at the default code generated output.

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="FakingExample"/>
</Fakes>

Now we’re ready to do some faking.  Let’s take a look at unit testing the AddCartItem method.  AddCartItem calls SaveCartItem on DataAccessLayer, which happens to be static.  We’re going to mock out that database call to isolate the logic in AddCartItem in our unit test. Add the code below to CartToShimTests.

using Microsoft.QualityTools.Testing.Fakes;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace FakingExample.Tests
{
    [TestClass]
    public class CartToShimTests
    {
        [TestMethod]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            //Create a context to scope and cleanup shims
            using (ShimsContext.Create())
            {
                int cartItemId = 42, cartId = 1, userId = 33, productId = 777;

                //Shim SaveCartItem rerouting it to a delegate which 
                //always returns cartItemId
                Fakes.ShimDataAccessLayer.SaveCartItemInt32Int32 = (c, p) => cartItemId;

                var cart = new CartToShim(cartId, userId);
                cart.AddCartItem(productId);

                Assert.AreEqual(cartId, cart.CartItems.Count);
                var cartItem = cart.CartItems[0];
                Assert.AreEqual(cartItemId, cartItem.CartItemId);
                Assert.AreEqual(productId, cartItem.ProductId);
            }
        }
    }
}

Line 13 creates a ShimsContext , which limits the scope of our shimming.  Line 19 defines our shim/detour for the SaveCartItem method.  Notice how we set the new behavior through a property named SaveCartItemInt32Int32. The Int32Int32 on the end is the type signature of parameters accepted by SaveCartItem. Microsoft Fakes has to keep the generated property names unique and predictable for methods, since they could have overloads or be refactored to have them someday.  Now let’s run it in Unit Test Explorer and see if it passes.

TestrunAfterShim

It does indeed pass.  Notice that our shim/detour completely skipped over the SaveCartItem method in DataAccessLayer, if it hadn’t then we would’ve received a nasty exception since “RandomSqlConnectionString” is clearly not a valid connection string . Start to finish, this ends up being pretty easy to setup and use for our simple cart example. 

Stubs Example

For the stubs example our code will be very similar to the shim example however we’ll need to create an ICartSaver interface and inject it into the CartToStub object.  At which point we can stub out ICartSaver and have it return 42 just as we did for shimming.  We’ll manually inject the dependency as opposed to pulling down a DI framework.

Add a new interface ICartSaver to the FakingExample project.

namespace FakingExample
{
    public interface ICartSaver
    {
        int SaveCartItem(int cartId, int productId);
    }
}

Next up add, for completeness (although not necessary) add a CartSaver class to the FakingExample project so we can replicate the implementation of DataAccessLayer.

using System.Data;
using System.Data.SqlClient;

namespace FakingExample
{
    public class CartSaver : ICartSaver
    {
        public int SaveCartItem(int cartId, int productId)
        {
            using (var conn = new SqlConnection("RandomSqlConnectionString"))
            {
                var cmd = new SqlCommand("InsCartItem", conn);
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.Parameters.AddWithValue("@CartId", cartId);
                cmd.Parameters.AddWithValue("@ProductId", productId);

                conn.Open();
                return (int)cmd.ExecuteScalar();
            }
        }
    }
}

Create a new CartToStub class as follows similar to CartToShim but allowing for manual dependency injection of ICartSaver in the constructor:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace FakingExample
{
    public class CartToStub
    {
        public int CartId { get; private set; }
        public int UserId { get; private set; }
        private List<CartItem> _cartItems = new List<CartItem>();
        public ReadOnlyCollection<CartItem> CartItems { get; private set; }
        public DateTime CreateDateTime { get; private set; }
        private ICartSaver _cartSaver;

        public CartToStub(int cartId, int userId, ICartSaver cartSaver)
        {
            CartId = cartId;
            UserId = userId;
            CreateDateTime = DateTime.Now;
            _cartSaver = cartSaver;
            CartItems = new ReadOnlyCollection<CartItem>(_cartItems);
        }

        public void AddCartItem(int productId)
        {
            var cartItemId = _cartSaver.SaveCartItem(CartId, productId);
            _cartItems.Add(new CartItem(cartItemId, productId));
        }
    }
}

Moving back over to the FakingExample.Tests project, add a new unit test file named CartToStubTests.  Create/Modify the AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart method to take into account the new changes.

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace FakingExample.Tests
{
    [TestClass]
    public class CartToStubTests
    {
        [TestMethod]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            int cartItemId = 42, cartId = 1, userId = 33, productId = 777;

            //Stub ICartSaver and customize the behavior via a 
            //delegate, ro return cartItemId
            var cartSaver = new Fakes.StubICartSaver();
            cartSaver.SaveCartItemInt32Int32 = (c, p) => cartItemId;

            var cart = new CartToStub(cartId, userId, cartSaver);
            cart.AddCartItem(productId);

            Assert.AreEqual(cartId, cart.CartItems.Count);
            var cartItem = cart.CartItems[0];
            Assert.AreEqual(cartItemId, cartItem.CartItemId);
            Assert.AreEqual(productId, cartItem.ProductId);
        }
    }
}

Run the unit tests once more to see that everything’s passing.

TestrunAfterStub

Just as with shims, start to finish, this ends up being pretty easy overall for our simple cart example.

The Fakes Xml File

The current MSDN library documentation details the nature of the .fakes xml file as follows:

The generation of stub types is configured in an XML file that has the .fakes file extension. The Fakes framework integrates in the build process through custom MSBuild tasks and detects those files at build time. The Fakes code generator compiles the stub types into an assembly and adds the reference to the project.

The .fakes file provides fine grained control of how stub and shim generation work for a particular assembly.  Luckily enough the file has intellisense and pretty good descriptions when hovering, so exploration of features is rather straight forward.  Expanding out all attributes and elements using intellisense provides us with the following structures.

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="" Location="" Version="" x86=""/>

  <StubGeneration Disable="" SkipVirtualIndexers="" SkipVirtualMethods="">
    <Clear />
    <Add AbstractClasses="" FullName="" Interfaces="" Namespace="" TypeName=""/>
    <Remove AbstractClasses="" FullName="" Interfaces="" Namespace="" Obsolete="" TypeName=""/>
  </StubGeneration>

  <ShimGeneration Disable="">
    <Clear />
    <Add FullName="" Namespace="" TypeName=""/>
    <Remove FullName="" Namespace="" Obsolete="" TypeName=""/>
  </ShimGeneration>

  <Compilation Debug="" DisableCodeContracts="" KeyFile="" ProjectTemplate="">
    <COMReference Guid="" VersionMajor="" VersionMinor="" WrapperTool=""/>
    <Property Condition="" Name="" />
  </Compilation>
</Fakes>

Based on this we see the following features:

  • Either shimming or stubbing can be completely turned off
  • Shims and stubs can be filtered such that only specific items from an assembly are shimmed and/or stubbed
  • Strong signing can be overridden via the KeyFile attribute, based on the MSDN docs the Fakes framework will automatically sign the .Fakes assembly with the same key the source assembly was signed with unless overridden here
  • There’s a facility to reference/deal with a COM component with COMReference if necessary.  Hopefully it won’t be…

Aside from items within the compilation element everything is along the lines of what we’d expect to be available here.  I imagine in most circumstances this file won’t need to be touched, however if you’re faking a larger library like mscorlib, then it would seem almost mandatory to filter the types generated otherwise the compiler could be chewing on it for a while.  The classic DateTime.Now shimming examples should probably be doing this type of filtering.

Conclusion

All in all I’m really excited about Microsoft Fakes.  It has the capacity to drive change in the .NET testing landscape similarly to how adding unit testing to Visual Studio (back in the 2005 version I believe) did so.  Widespread inclusion, usage and education of faking/mocking will be very beneficial for the community and will help drive innovation of testing techniques. Projects which would’ve been considered difficult if not impossible to add some level of unit testing to, without large amounts of refactoring, can now be unit tested easily.  Microsoft Fakes is a definitive win in my book.

 

The code for this post is available on GitHub

  • 16 Comments