Building a Provider Based File Storage System Using .NET

If you haven't read the original article, now is a good time to do so as I won't be repeating a whole lot of it. I'll wait here while you read it.

Done? Good. It means we're all on the same level now. In the original article I introduced two classes: File and FileInfo. The File class is used to store complete details about a file, while the FileInfo class serves as a summary object for a file, without the actual file bytes. The latter class was used to display a list of files in, say, a GridView, without the overhead of the actual bytes associated with the file stored on disk. Besides being a wrapper for some properties such as Id, OriginalName and FileData, these classes also contained methods such as GetItem and GetList to directly access the database. While convenient to write at that time, it also meant the classes are tightly coupled to the chosen database technology (SQL Server in my case) which made them pretty much impossible to reuse. In the new implementation, I separated the File classes from the actual implementation, and let File inherit FileInfo, leading to the class hierarchy with simple Data Transfer Objects (DTO's) shown in Figure 1:

Figure 1

The DatabaseFile and DatabaseFileInfo classes inherit File and FileInfo respectively, and add a constructor that is able to fill the class's properties based on a DbDataReader. You'll see more of this later.

With the File and FileInfo classes separated from the database technology, I was ready to implement a provider to work with them.

Introducing the Provider Model

When ASP.NET was released, it introduced the Provider Model for application services such as Membership and Roles,. According to the Microsoft site, a provider is

A software module that provides a uniform interface between a service and a data source. Providers abstract physical storage media, in much the same way that device drivers abstract physical hardware devices. Because virtually all ASP.NET 2.0 state-management services are provider-based, storing session state or membership state in an Oracle database rather than a Microsoft SQL Server database is as simple as plugging in Oracle session state and membership providers. Code outside the provider layer needn't be modified, and a simple configuration change, accomplished declaratively through Web.config, connects the relevant services to the Oracle providers.

In short, this means that code in your web application talks to a known interface. Under the hood, the configured provider is instantiated and work is delegated to the underlying provider. So, for example, when you call System.Web.Security.Membership.CreateUser( ... ) to create a user, you talk to a static class (Membership) that delegates work to a provider it has access to (which is the System.Web.Security.SqlMembershipProvider class by default, but you can override that). The concrete implementations (SqlMembershipProvider in this example) inherit an abstract base class called MembershipProvider which in turn inherits an abstract base class called ProviderBase. It's this base class that is part of the key to implementing a provider based model. For the Membership services, the object model looks like this:

Figure 2

This figure shows two classes implementing the provider functionality: the SqlMembershipProvider and the ActiveDirectoryMembershipProvider. The code in your own application (such as the CreateUser call shown earlier, but also the built-in controls such as the Login and CreateUserWizard) talks to the static Membership class which in turn talks to one of the configured concrete providers. The MembershipSection class also shown in the diagram is used to read configuration information from the application's config file. You'll see more of this later.

The beauty of this design is that it's extremely easy to swap a provider. In my article Using the Microsoft Access Providers to Replace the Built-In SQL Server Providers I am doing exactly that: replacing the configured provider with one that's built to target a Microsoft Access database instead of the existing SQL Server or Active Directory data stores. To implement a new provider, all you need to do is inherit from the abstract class MembershipProvider, implement all of the members defined in the contract for MembershipProvider and then configure your application to use your custom provider instead. What I like about this design pattern is that you don't need to deal with lots of implementation details of the API; all you do is provide an implementation for the methods defined in the base class and that's it. For example, while the static Membership class has four overloads of CreateUser, the provider implementation only needs to implement one: the most extensive overload.

You can find out more information about the provider model in the following articles:

• Beyond Membership: Using the Provider Design Pattern for Custom Features
• Provider Model Design Pattern and Specification, Part 1
• Provider Design Pattern, Part 2

Implementing Your Own Provider Model

To implement your own provider model for an application,you need to carry out a few steps. I'll briefly list them first and mention the class names I'll be using in my application. In a later section, I'll go over the steps in more detail when discussing my implementation of the File Storage Provider. The goal of the implementation of this provider model is to create a public API similar to that presented in the original article. In other words, the application needs to provide a way to get files, save them in the database and on disk, and delete them again when necessary (this last feature wasn't a part of the original application).

1. Create an abstract class that inherits ProviderBase, and that defines the public contract for the functionality you want to build. This typically means adding abstract members that need to be implemented by the concrete providers. My class is called FileStorageProvider and has methods such as GetFile and SaveFile.
2. Create a collection class that inherits ProviderCollection which is used to store the configured provider(s) in the manager class. My collection class is called FileStorageProviderCollection.
3. Create a configuration class (mine is called FileStorageConfiguration) which inherits ConfigurationSection. This class is used to read configuration information from the application's config file.
4. Create a static class (mine is named FileStorageManager) that serves as the main application's entry point. Add static methods that mimic the methods defined in the provider class. In my case, I added static methods such as GetFile and SaveFile. The methods delegate the work to the configured provider.
5. Create concrete implementations that inherit the provider class. In my sample application, I have three classes that inherit FileStorageProvider: SqlFileStorageProvider, SqlAndDiskFileStorageProvider and AccessFileStorageProvider. The first two target SQL Server, while the latter targets a Microsoft Access database.

Initially this is quite a bit of work. However, adding the next provider to the mix means you just need to execute the last step. So, adding an OracleFileStorageProvider means you just need to inherit FileStorageProvider and implement a few methods. You then need to register the provider in the target application, but you don't have to change a single line of code.

To make the first four steps a lot easier, Microsoft provides the Code Template for Building a Provider-Based Feature. You can find more information about this template here:

• Direct link to the MSI that installs a number of .cs files
• The Provider Toolkit homepage (on the French section of MSDN; couldn't find the English edition; the content is in English though)

The download installs 5 .cs files that serve as the code templates for the classes listed earlier. It also contains a readme file that guides you through changing some code in the template files to match your required behavior. With these code templates, you can have a simple provider with a static DoWork method set up in no time. Extending the model is then quite easy.

For my File Storage Application I used the same template files to create my provider based model. I then named and modified my classes as follows:

Step 1 - Create the FileStorageProvider

This is the class that inherits from the ProviderBase defined in .NET. As explained earlier, this class defines the contract that all concrete providers need to implement. My FileStorageProvider is pretty simple, and looks like this:

public abstract class FileStorageProvider : ProviderBase
{
  public virtual string UploadsFolder { get; protected set; }
  public abstract File GetFile(Guid id);
  public abstract List<FileInfo> GetFileInfoList();
  public abstract bool SaveFile(File file);
  public abstract bool DeleteFile(Guid id);
}

Three of these methods work with the File and FileInfo classes I showed at the beginning of this article.

Step 2 - Create the FileStorageProviderCollection

This code for this class is almost identical to that of the template, except for the FileStorage name and types.

public class FileStorageProviderCollection : ProviderCollection
{
  public override void Add(ProviderBase provider)
  {
    if (provider == null)
    {
      throw new ArgumentNullException("The provider parameter cannot be null.");
    }
    if (!(provider is FileStorageProvider))
    {
      throw new ArgumentException("The provider parameter must be 
                           of type FileStorageProvider.");
    }
    base.Add(provider);
  }

  new public FileStorageProvider this[string name]
  {
    get { return (FileStorageProvider)base[name]; }
  }
  public void CopyTo(FileStorageProvider[] array, int index)
  {
    base.CopyTo(array, index);
  }
}

This class serves as a simple collection for instantiated provider types. It ensures you only add providers of the correct type (I think this code predates the generics introduced in NET 2) and enables you to retrieve a provider by its index. I haven't seen a reason for the CopyTo method so I am not sure why it's included in the template code.

Step 3 - Create the FileStorageConfiguration

Again, this class is almost identical to that of the template. Check out the download at the end of the article for the complete source. This class is used to read configuration information in the application's config file that looks like this

<add name="SqlFileStorageProvider"
    type="Spaanjaars.FilesProvider.SqlFileStorageProvider, Spaanjaars.FilesProvider"
    connectionStringName="DefaultConnectionString"
    description="this came from config" 
/>

Step 4 - Create the FileStorageManager

This is probably the most interesting class from the provider model. This class has two main tasks: first, when instantiated it creates a collection of providers based on the configuration information it finds in the config file for the application. Secondly, it has public and static methods that define the contract your end applications works with. The class contains quite a bit of code, but I'll show it here in its entirety so you know what goes on under the hood.

public static class FileStorageManager
{
  //Initialization related variables and logic
  private static bool isInitialized;
  private static Exception initializationException;
  private static FileStorageProvider defaultProvider;
  private static FileStorageProviderCollection providerCollection;

  static FileStorageManager()
  {
    Initialize();
  }

  private static void Initialize()
  {
    if (!isInitialized)
    {
      try
      {
        //Get the feature's configuration info
        FileStorageConfiguration qc = (FileStorageConfiguration)
                 ConfigurationManager.GetSection("FileStorage");
        if (qc.DefaultProvider == null || 
                  qc.Providers == null || qc.Providers.Count < 1)
        {
          throw new ProviderException("You must specify a valid default provider.");
        }

        //Instantiate the providers
        providerCollection = new FileStorageProviderCollection();
        ProvidersHelper.InstantiateProviders(qc.Providers, 
                 providerCollection, typeof(FileStorageProvider));
        providerCollection.SetReadOnly();
        defaultProvider = providerCollection[qc.DefaultProvider];
        if (defaultProvider == null)
        {
          throw new ConfigurationErrorsException(
               "You must specify a default provider for the feature.", 
               qc.ElementInformation.Properties["defaultProvider"].Source, 
               qc.ElementInformation.Properties["defaultProvider"].LineNumber);
        }
      }
      catch (Exception ex)
      {
        initializationException = ex;
        isInitialized = true;
        throw ex;
      }
    }
    
    isInitialized = true; //error-free initialization
  }

  //Public feature API
  public static FileStorageProvider Provider
  {
    get
    {
      return defaultProvider;
    }
  }

  public static FileStorageProviderCollection Providers
  {
    get
    {
      return providerCollection;
    }
  }

  public static File GetFile(Guid id)
  {
    return Provider.GetFile(id);
  }

  public static bool SaveFile(File file)
  {
    return Provider.SaveFile(file);
  }

  public static bool DeleteFile(Guid id)
  {
    return Provider.DeleteFile(id);
  }

  public static List<FileInfo> GetFileList()
  {
    return Provider.GetFileInfoList();
  }

  public static string UploadsFolder
  {
    get
    {
      return Provider.UploadsFolder; 
    }
  }
}

The Initialize method contains pretty much boiler plate code from the template file for this class. It reads the configuration data using an instance of FileStorageConfiguration and then lets the ProvidersHelper class (part of the .NET Framework) do all the hard work. The helper class parses the type attribute from the configuration section and instantiates the relevant concrete providers based on that type. In the example above, that would mean instantiating a Spaanjaars.FilesProvider.SqlFileStorageProvider. If you have multiple providers listed, all of them are instantiated, but only one is the current provider at any given time. Look at the ProvidersHelper class with Reflector if you want to see the dirty details of the implementation.

The remainder of the code in this class is the public API of the provider. The first two properties, Provider and Providers give access to the currently configured provider, and to all registered providers. Quite often you don't have the need to access them directly in the application that uses the provider model, but they can be useful to carry out some lower level tasks not supported by FileStorageManager directly, or if you want to talk to a provider other than the one configured as the current provider.

The final five members are the part of the API that this article is all about: dealing with files. You see four methods to get a single File, a list of FileInfo instances and to save and delete a file. The UploadsFolder property returns the path of the folder where the files must be saved, in case the configured provider needs to store them on disk; otherwise it returns null.

In order to make use of all this, you need to add the following to your application's .config file:

<configuration>
  <configSections>
    <section name="FileStorage" 
     type="Spaanjaars.FilesProvider.FileStorageConfiguration, 
            Spaanjaars.FilesProvider" allowDefinition="MachineToApplication" />
  </configSections>

  ...

  <FileStorage defaultProvider="SqlFileStorageProvider">
    <providers>
      <add name="AccessFileStorageProvider"
        type="Spaanjaars.FilesProvider.AccessFileStorageProvider, 
            Spaanjaars.FilesProvider"
        connectionStringName="AccessConnectionString"
        SaveFileDataInDatabase="true"
        UploadsFolder="~/Uploads"
        description="Stores meta data in an Access database and 
             files on disk or in the database" 
      />

      <add name="SqlFileStorageProvider"
        type="Spaanjaars.FilesProvider.SqlFileStorageProvider, 
            Spaanjaars.FilesProvider"
        connectionStringName="DefaultConnectionString"
        description="Stores meta data and files in SQL Server" 
      />

      <add name="SqlAndDiskFileStorageProvider"
        type="Spaanjaars.FilesProvider.SqlAndDiskFileStorageProvider, 
            Spaanjaars.FilesProvider"
        connectionStringName="DefaultConnectionString"
        description="Stores meta data in SQL Server, and saves the files to disk"
        UploadsFolder="~/Uploads"
      />
    </providers>
  </FileStorage>
  
  ...
</configuration>    

This code defines three providers: one using Microsoft Access and two targeting SQL Server. The currently active provider is the SqlFileStorageProvider as set by the defaultProvider attribute.

With this configuration (and assembly for the File Storage System referenced by the application), you can now access the public API in the final application like this:

// Save a file 
File myFile = new File( ... Code to instantiate File here ...);
FileStorageManager.SaveFile(myFile);

// Get a file
Guid id = new Guid(Request.QueryString.Get("Id"));
File myFile = FileStorageManager.GetFile(id);

// Get all files (as FileInfo instances)
List<FileInfo> files = FileStorageManager.GetFileList(); 

// Delete a file
Guid id = new Guid(Request.QueryString.Get("Id"));
FileStorageManager.DeleteFile(id);      

While this API is now available as advertised, you still need the concrete implementations of FileStorageProvider in order to actually work with these files. This is discussed next.

Step 5 - Creating Concrete Implementations of FileStorageProvider

If you look at the source code that comes with this article, you find three concrete implementations of the FileStorageProvider, listed below:

Notice how I separated the behavior of storing files on disk or in the database in two separate providers for SQL Server, while the Access implementation can do both at the same time. This just serves as an example. I could have created two separate providers for Access as well, or extended the SqlFileStorageProvider with a switch to determine where to store the actual file data.

Figure 3 shows how the three providers and their base classes are related:

Figure 3

Both SqlFileStorageProvider and AccessFileStorageProvider directly inherit the base provider: FileStorageProvider. SqlAndDiskFileStorageProvider inherits SqlFileStorageProvider and only overrides the two methods that deal with files on disk: DeleteFile and SaveFile. You see how this works a little later.

Most of the code in these providers is pretty standard .NET and ADO.NET code, and most of it looks like the code from the original article. So, instead of showing and discussing the full code, I'll focus on just a few interesting bits and pieces. First, take a look at the Initialize method you find in SqlFileStorageProvider:

public class SqlFileStorageProvider : FileStorageProvider
{
  public string ConnectionString { get; protected set; }
  protected bool SaveFileDataInDatabase { get; set; }

  ...

  public override void Initialize(string name, 
          System.Collections.Specialized.NameValueCollection config)
  {
    SaveFileDataInDatabase = true;

    if ((config == null) || (config.Count == 0))
    {
      throw new ArgumentNullException(
        "You must supply a valid configuration dictionary.");
    }

    if (string.IsNullOrEmpty(config["description"]))
    {
      config.Remove("description");
      config.Add("description", "Put a localized description here.");
    }

    // Let ProviderBase perform the basic initialization
    base.Initialize(name, config);

    //Perform feature-specific provider initialization here
    //Get the connection string
    string connectionStringName = config["connectionStringName"];
    if (String.IsNullOrEmpty(connectionStringName))
    {
      throw new ProviderException(
        "You must specify a connectionStringName attribute.");
    }

    ConnectionStringsSection cs = (ConnectionStringsSection)
    ConfigurationManager.GetSection("connectionStrings");
    if (cs == null)
    {
      throw new ProviderException(
        "An error occurred retrieving the connection strings section.");
    }

    if (cs.ConnectionStrings[connectionStringName] == null)
    {
      throw new ProviderException(
        "The connection string could not be found in the connection strings section.");
    }
    else
    {
      ConnectionString = cs.ConnectionStrings[connectionStringName].ConnectionString;
    }

    if (String.IsNullOrEmpty(ConnectionString))
    {
      throw new ProviderException("The connection string is invalid.");
    }
    config.Remove("connectionStringName");

    //Check to see if unexpected attributes were set in configuration
    if (config.Count > 0)
    {
      string extraAttribute = config.GetKey(0);
      if (!String.IsNullOrEmpty(extraAttribute))
      {
        throw new ProviderException(String.Format(
            "The following unrecognized attribute was found in {0}'s configuration: '{1}'", 
            Name, extraAttribute));
      }
      else
      {
        throw new ProviderException(
            "An unrecognized attribute was found in the provider's configuration.");
      }
    }
  }
}      

The first thing that Initialize does is setting SaveFileDataInDatabase to true. This is an hard coded implementation which means the provider will always store the file data in the database.This property is marked protected which means that classes that inherit SqlFileStorageProvider can override it, which happens to be the case with the SqlAndDiskFileStorageProvider class.

Once again, most of what goes on in Initialize is boiler plate code coming from the Provider Toolkit samples. Basically, the code parses the config file, looking for config elements it recognizes and assigning them to local variables (such as the ConnectionString). Using a Fail Fast mechanism, the code throws exceptions when it encounters invalid values, or when it encounters attributes it doesn't recognize. I really like this style of config parsing, as it means that when your configuration is not right, you get immediate feedback with a crystal clear exception message.

Once Initialize is called, the provider should be in a valid state, after which you can call one of its methods. To see an example, consider this DeleteFile implementation:

public override bool DeleteFile(Guid id)
{
  int affectedRecords;
  using (SqlConnection mySqlConnection = new SqlConnection(ConnectionString))
  {
    using (SqlCommand myCommand = new SqlCommand("sprocFilesDeleteSingleItem", 
               mySqlConnection) { CommandType = CommandType.StoredProcedure })
    {
      myCommand.Parameters.Add(new SqlParameter("@id", SqlDbType.UniqueIdentifier) 
               { Value = id });

      mySqlConnection.Open();
      affectedRecords = myCommand.ExecuteNonQuery();
    }
    mySqlConnection.Close();
  }
  return affectedRecords == 1;
}

This code is pretty much identical to the code from the original article: it instantiates a SqlConnection and a SqlCommand object and then executes a stored procedure that deletes the file from the database.

The other methods (GetFile, GetFileInfoList, and SaveFile) all pretty much follow the same pattern. There's one thing worth noting and that's how GetFile and GetFileInfoList use custom DatabaseFile and DatabaseFileInfo classes to copy the data from a DbDataReader into the class itself. Here's a snippet from the GetFile method:

using (SqlDataReader myReader = myCommand.ExecuteReader())
{
  if (myReader.Read())
  {
    myFile = new DatabaseFile(myReader);
  }
  myReader.Close();
}

The DatabaseFile class simply inherits File, but has a constructor that accepts a DbDataReader like this:

public DatabaseFile(DbDataReader myReader)
{
  Id = myReader.GetGuid(myReader.GetOrdinal("Id")); 
  DateCreated = myReader.GetDateTime(myReader.GetOrdinal("DateCreated"));
  // Other properties here
}  

The GetFile method returns a File instance, so calling code never knows they are dealing with a DatabaseFile. I found this a convenient way to load data in the File class based on some data in the database. Instead of a derived class with a constructor, it would have been easy to create some method like Load in the SqlFileStorageProvider class to copy the data into a File instance. Both methods would result in the same thing: a filled instance of File that can be returned to the calling code. In the former scenario, it's actually a DatabaseFile, but no one knows.

All of the code in SqlFileStorageProvider is designed to store the actual bytes that make up the file in SQL Server. To see how this works, look at the way the parameter for FileData in SaveFile is set up:

myCommand.Parameters.Add(
    new SqlParameter("@fileData ", SqlDbType.VarBinary) 
    { 
      Value = SaveFileDataInDatabase ? (object)file.FileData : DBNull.Value, 
      Size = SaveFileDataInDatabase ? file.FileData.Length : 0 }
    );

Because SaveFileDataInDatabase has a hard coded value of true, the FileData is always sent to the database.

However, because of the SaveFileDataInDatabase property, this provider is technically capable of ignoring the FileData, and sending DBNull.Value instead. That's what I make use of in the inheriting SqlAndDiskFileStorageProvider class. In that class's Initialize method, SaveFileDataInDatabase is always set to false. The class then overrides SaveFile like this:

public override bool SaveFile(File file)
{
  bool result = base.SaveFile(file); 
  if (result)
  {
    // Database update is done; now store the file on disk.
    const int myBufferSize = 1024;
    using (Stream myInputStream = new MemoryStream(file.FileData))
    {
      using (Stream myOutputStream = 
             System.IO.File.OpenWrite(PhysicalUploadsFolder + file.FileUrl))
      {
        byte[] buffer = new Byte[myBufferSize];
        int numbytes;
        while ((numbytes = myInputStream.Read(buffer, 0, myBufferSize)) > 0)
        {
          myOutputStream.Write(buffer, 0, numbytes);
        }
        myInputStream.Close();
        myOutputStream.Close();
      }
    }
  }
  return result;
}  

This method first calls base.SaveFile. That method then stores the meta data in the database. Because SaveFileDataInDatabase is set to false for the derived provider, the FileData is skipped. The same FileData is then stored on disk using a number of Stream classes. The PhysicalUploadsFolder property has been given a value in the Initialize method, based on the configuration element for the configured provider.

DeleteItem follows a similar pattern. It deletes the file from the database, and when that succeeds, it also deletes the file from disk.

The AccessFileStorageProvider class follows a very similar pattern. However, since it can be used to to store files on disk and in the database, its Initialize method has some extra logic to figure that out, and to make sure the UploadsFolder is present, valid and exists. It then uses the SaveFileDataInDatabase property in SaveFile and DeleteFile to determine if the file must be stored on, or deleted from disk.

Extending the Storage System

With the current implementation, you have a very flexible system. The three available providers make it super easy to store meta data in SQL Server or in an Access database. Additionally, through configuration you can determine where the actual files need to be stored: in the database, or on disk. Since the current implementation is close to the original version, I didn't have to make a whole lot of changes to the actual web site that is used to manage the files. I had to change a few types, changed the ObjectDataSource control in Default.aspx and changed a few method calls, but I didn't have to touch any logic.

But what's even better is that it's now very easy to introduce a new provider. If, for example, I wanted to store files in the cloud using Azure, I could build a new FileStorageProvider, called AzureFileStorageProvider for example, register the provider in web.config and be done with it. From then on, files would be stored in, and served from the cloud!

Disclaimer

I have tested most of this code thoroughly enough to know it probably works. However, if you start using this in a real-world application, be sure you know what you're doing and that you understand the code. No warranties whatsoever. If you find a bug or have comments about the current code, please use the Talk Back feature at the end of this article.

Also, the database methods could benefit from some extra level of abstraction. To make each method simple to understand, I am instantiating objects such as connections and commands in each data access method. You could abstract them away to some general helper classes, or use another data access technologies such as Entity Framework or NHibernate.

The sample database that comes with this article is mostly compatible with the one from the original article. I changed the length of the OriginalName column from an nvarchar(50) to an nvarchar(255). I also changed the length of the associated stored procedure parameter in sprocFilesInsertSingleItem and I created a new stored procedure called sprocFilesDeleteSingleItem in order to delete files from the database.

Downloads

You can download the complete code (C# only) for the provider model and the sample site.