The ultimate goal for this API is to enable Cloud Storage Persistence Ignorance. A secondary, and almost equally important, objective for creating this API is to enable testability and thus enhance the over all maintainability of Cloud Applications that make use of Cloud Storage also generally thought of as “shared state”. We have been doing this work on the Windows Azure cloud but in theory this API may be used on any Cloud Storage.
Note: We have created a Cloud Storage API at the consultancy where I work, Dotway. Me and a great colleague Peter von Lochow have been fortunate enough to get some time to fly high up to the clouds. Peter will also be writing one or two blog posts here on my blog as the first ever guest blogger @ Techie.notepad! The time spent in the clouds has been time very well worth spending. We’ve learned a lot and laughed a lot. Guess the thin air at Cloud altitude adds to the development experience because I’ve had a blast! ;~)
Background
This journey started when I combined Windows Azure with the Managed Extensibility Framework: Windows Azure + Managed Extensibility Framework (MEF) = true. The purpose of that post was to enable extensibility on Azure and to break out of the dependency imposed by the RoleManager static class in the Windows Azure SDK. I promised to take two more steps after that first post. NEVER do that! ;~) Kidding; this time I intended to make good on my promise and actually deliver. Just to reiterate:
The overall goal is to create a Windows Azure project template that:
- enables testability
- abstracts away storage
- is extensible and easy to evolve during development
This post is the first in a brief series of posts that take us that second step toward our lofty goal somewhere beyond the clouds: This is the step that makes a Cloud Application Persistence Ignorant and enables testing of these applications without depending on storage. Alright – enough with the poetic crap!
Where is the good stuff?
I’m not going to list the API in detail here in this post. Instead I will go into a few of the details of the API and explain a little about how our reasoning went. The API is available under the license Ms-PL at CodePlex:
CloudStorage.API (Here you can read more details and also get hold of the code.)
Design considerations
The three things you store in clouds these days are Messages (in Message Queues), Blobs (in Blob Containers) and Entities (in Entity Tables). Cloud Storage is typically RESTful in nature. We designed our API to be true to the REST API for the Windows Azure Cloud Storage.
Another consideration we took was to impose as few dependencies on this API as we could. Aside from the .NET Framework core we have only one real dependency and that is to the System.Data.Services.Client. The reason for this dependency is for it’s great usage in accessing Data Entities.
Our implementation
We have today only one partial implementation of this API. That code is not, at time of writing this, but will be published on the Windows Azure Contrib Project site at CodePlex. The main reason we don’t even have one full implementation of our own API is YAGNI. Of course we’ll need it – but not right now! We did however want to publish a complete Cloud Storage API.
We know full well, and brace for impact, that people will say – Dudes YAGNI is creating an API no-one uses! Frameworks are not grown, they are harvested etc. Please look at the non-implemented parts of this API as a theoretical expansion on the things we have implemented. Besides the Azure Storage REST API is there and complete. We now made a programmatic client API of the same size even though we haven’t used all of it in our applications yet.
The implementation we do have uses a lot of Wrapper classes (Adapter Pattern) to apply our API to the Microsoft StorageClient sample. Key word here being the word “sample”. One BIG reason for embarking on this journey is the fact that we do not want to sit down our selves and write a REST client from scratch. First of all that was not our focus. Second Microsoft naturally has a team coding away at a real Storage Client library for Azure which we intend to use once it’s been released. And Third we don’t want to begin coding a big dependency on a sample implementation knowing we’re going to have to exchange it later down the line.
This means: Our implementation of this Cloud Storage API, the one we’ve been using our selves, is incomplete, based on sample code and uses MEF, Managed Extensibility Framework, to enable extensibility which also “automagically” gives us Dependency Injection.
The BIG picture of the CloudStorage.API
The API is divided into three main parts; Blobs, Entities and Messages. There are also a couple of optional parts and a few shared parts.
The API is made up of four different projects: API, API Tests, Extensibility (Optional) and Extensibility Tests. The only part you need to make use of the API is one of the four assemblies – the API.
We intend to publish a small series of blog posts with examples of using each of the main parts so I won’t spend more time here today on the usage of the Blobs, Entities and Messages Storage APIs.
Factories
I will however talk a bit about our use of factories. Since we make heavy use of the Factory method pattern and because the use of factories in software architecture always has been debated I figure it’s pertinent to motivate this design decision a bit. (If you haven’t read Joel Spolsky’s epic post Why I Hate Frameworks you have to go read it – now! But hurry back!)
- One good way to enable extensibility and also make Dependency Injection possible is to make use of the factory pattern.
- Using the factory APIs we’ve supplied is not mandatory. If you don’t like factories – don’t use them.
- One kink in the armor of this approach is that we need some method to access the factories. Yes – we need a factory factory. Our implementation, mentioned above, has provided this for us by means of MEF.
Extensibility
Which brings us to the question of extensibility. We did not want to set the MEF dependency on the API but we did want to enable an extensible view of the usage of the account information required to access a Cloud Store. This little bit of implementation was logically placed together with the API but it is by no means required. Look at is as a small preview to the third step of this journey; Windows Azure Project Extensibility.
Not to go into to much detail here we have an IStorageAccountInfoFactory that gives us a .Get(string storageAccountName) method that can, through use of some MEF Lazy loaded magic get us any set of IStorageAccountInfo we want.
But this is just a sneak peek at our framework code that we will (soonish) publish on the Azure Contrib site.
Help wanted:
We also have a request at this time: Please review our API for another Cloud Storage, apart from Windows Azure Storage, and share the results! We will be only too happy to include you in our project if you do!
In Summary
So there it is – The Cloud Storage API
Now that it’s out there I can honestly tell you that part of the reason for the delay in publishing this is mostly due to the fact that we have had all kinds of labor pains associated with this release. In the end it’s far better to ship sloppy and not remain sloppy than to not ship at all! We don’t feel that this is too sloppy but we’d welcome your feedback!
All in all we’ve enjoyed creating this API and we like using it. If at least one other team out there does too it’s been worth it. Please let us know if you do!
Cheers,
M. http://twitter.com/noopman and P. http://twitter.com/vonlochow
PS. Peter is attending a wedding today. Little did he know there was also going to be a birth tonight. ;~) I wonder what he will think tomorrow, appropriately hung over, when he notices? 10, 9, 8… What did you do? 7, 6, Oh no you didn’t? 5, 4, 3… You didn’t press the BIG RED BUTTON! 2, 1 We have lift-of!
posted @ Saturday, August 15, 2009 11:52 PM