This is the second part of a multi-part article about implementing large file upload for ASP.Net In my previous post I explained the Silverlight bit and this part will be about the WCF backend.

Curious enough, the WCF backend is actually extremely simple. The are only 3 methods that are exposed as a webservice. There 3 methods cover the 3 phases of the file upload process I explained in the previous post. But before I show the service I need to explain some thing else first.

In summary:

• The client request an upload token.
• The client sends the binary file in small blocks using the token.
• After the file is complete, the client confirms the upload.

We took care of those steps with the Silverlight client. On the server side, we need to respond to those steps.

To maintain a record of all the uploads that are in progress, we need to maintain some sort of ‘memory’.  When an upload starts, a new UploadSession object is created. The UploadSession has a couple of properties:

1. The Filename of the file that is being submitted,
2. The ID or token of the session,
3. The State that the upload is currently in (Created, Streaming, Completed or Failed), and finally:
4. The binary Data of the transmitted file.

UploadManager class

To access a session, a singleton class is implemented.It has 3 methods: CreateSession and GetSession, and finally GetUpload():

CreateSession will create a new session, GetSession retrieves a session. Both are declared internal because they are used by the WCF service (we will get to that in a minute). Third, the public GetUpload() is similar to GetSession but is used to retrieve a completed transfer.

UploadSessions

The UploadSessions class is a helper collection class that is used to store the current uploads. In a first version I was keeping all the uploads in a collection that had Timer events to purge old uploads, but later I decided to use the HttpRuntime.Cache. Here’s the collection class:

In essence, it provides strongly-typed access to the cache using the indexer operator. These getter and setter forwards all calls to the HttpRuntime Cache. If you didn't know, this cache is exactly the same class as the HttpContext.Current.Cache, only shorter.

I set the sliding expiration to 5 minutes. This means that the session will be removed is no block is received within 5 minutes, or when the file is not ‘picked up’ by the consuming client after 5 minutes of the upload. This should be sufficient in most cases.

Also, I use the priority to CacheItemPriority.NotRemovable so the server does not expire the session within the timeout period is it low on resources.

(I just noticed that the Remove method is empty…)

Calling the UploadManager from WCF

The WCF service itself is extremely simple. I just forwards all methods to the UploadManager:

Last, but not least

The last step for the service side is UploadSession class itself:

Again, no rocket science. The UploadSession class is just a wrapper class for a MemoryStream. I intend to expand this class so it flushes the buffer to a temp-file if the file becomes too large, but for now it works perfectly.

The reason I am using a in-memory buffer is a) memory is cheap and fast and b) I’m using the uploaded data in combination with Linq. And Linq stored binary data as a Binary() class, which is just a wrapper class for a byte[]. Therefore it has no use in streaming the file to disk first, and then completely loading it back into memory. The disk buffer approach is only handy when lots of file are uploaded at the same time. If a lot of files are uploaded simultaneously, the total memory usage would have been lower during the time of the uploads, at the cost of requiring temporary files and a lot of disk I/O.

Hosting the Service

Now, to host this WCF service, a basic endpoint must be configured in the web.config of the hosting application:

This configuration was generated automatically when I added the service in my project, so it is completely standard. No magic here. No streaming, no large buffers, nothing fancy.

The Silverlight client needs the Url of the WCF service, but I think that was already covered in the previous post.

Wrapping it all up

This is it for the WCF side of it all. As I mentioned before, we are not finished yet. We still need a way to host the Silverlight application, and tell Asp.net that we have received a file. After that we need to pickup the file from the UploadManager. I’ll be showing you a custom server control that does all three of these steps in a simple, reusable Asp.Net control. But that topic will have to wait until the next episode.

Happy uploading!

Yesterday I saw a demo on the new Dynamic Data contribution to Asp.net 3.5. It literally blew me away (really) because that was what I've been doing for most of my time over the last couple of years.

The base

Basically, we develop custom data driven web sites with Asp.net. Most of the time this also involves some kind of back-end content management. What our mini-cms actually did was the following:

• Inspect the database model
• Enrich that model with metadata such as preferred input method, validation and comments and associations,
• Generate a whole lot of Asp.Net controls from the aforementioned properties,
• Inject a lot of role-based security-stuff,
• Present those forms to the end-user to allow basic site management.

So, what we had to build was a configuration file with all the meta data, the database, and of course the web site that got driven by that data.

Not any more. I'm out of a job.

With Visual Studio 2008, Asp.Net 3.5 framework and Dynamic Data the generic application that we wrote can be thrown out of the window, because with the DataContext designer this is not longer something somebody else will pay us to do. As you can see in the demo, by binding your decorated data model to the starter solution, you have an interface in a matter of minutes.

Fortunately the Internet-facing site cannot be generated yet. And, we can embrace the DynamicData technology to our advantage and create sites even faster.

On to the subject of this post!

Silverlight and Dynamic Data

Now this is all nice, but there is a simple problem. If I wanted to use this technology with Silverlight, In the last couple of weeks, I was working on a Silverlight application for our content management concept, and I was wondering if the object model could be easily extracted with the new LINQ To SQL designer. I would have to transfer all meta data and content through a WCF service, right?

Lucky for me, the DynamicData layer isn't even finished and does not yet generate WCF services. So, in order to build a generic Silverlight application you would have to serialize a lot of unknown types into a Silverlight application that was unaware of those types. That is not really what I wanted. I wanted a pre-compiled application that can be configured with just a single configuration property: the data model.

I decided to give it a go, and here's what I came up with. I will try to explain it one step at a time.

Here's my very basic data model:

Just a news item for a particular category that can have zero or more attachments to it. Nothing fancy.

Using the designer, this stuff generates classes for me with the all the properties and data access methods (select, update, delete etc.) while maintaining all the relations.

Here comes the cool bit: Did you know that the DataContext class actually gives us all the meta data without having to use reflection?  Just drill down into DataContext.Mapping and see that you find there:

(edit: since GetTables is a method, it should be written as .GetTables())

Silverlight does not have this nice API, So I created my own WCF contract to serialize these properties in a separate class. Here's a sample of the type specification that I use to transfer to Silverlight (I left the part out that maps the MetaDataMember stuff to my own classes on purpose).

To serialize an item of a particular type I perform a generic LINQ select on the table, I read all the properties of a data item and put them in a dictionary so that Silverlight can read and display those items:

Notice that for this demo I used the Convert.ToString() method on the value to get a nice presentation of the value. In a future version I will need to put the basic values (int, string, DateTime etc.) in the Values collection.

Of course this is not finished by far. I needed a way to decorate the designed classes with validation, descriptions, etc. Because I do not yet have the DynamicData beta installed on my machine I decided to borrow their approach to decorate generated code. Here's a snippet for the NewsItem class:

Because the data context designer generates partial classes, we have the ability to add functionality to those classes. Since I only need to apply some attributes to the generated class I need to define the class again (as partial) and apply my attributes. Since I cannot redefine the properties of that class I have to create an extension class that shadows the properties of the generated class (very nice idea, DynamicData Team!). To use the attributes of the extension class I do need to use reflection, but that is only once. Here's a video of how it works and how it is used. I just copied the concept to there's little more to tell.

A lot of work still needs to be done but I hope you understand that the rest is just more along the same line.

Looks like I still have a job!

From what I've seen, Silverlight 2 beta 2 and Blend 2.5 june preview are amazing.

I've watched some of the new tutorials on silverlight.net, and I especially like the video titled ADD STATES TO A USERCONTROL FOR SILVERLIGHT 2. This demonstrates the fantastic way to add animations, flyout menus etc while having to write very little code or XAML yourself.

I'll definitely be spending some time with these new toys!

Below is a ling to a video that demonstrates how easy it becomes to develop a datadriven Silverlight application.

http://blogs.msdn.com/swiss_dpe_team/archive/2008/03/17/silverlight-2-beta1-wcf-linq-to-sql-a-powerfull-combination.aspx

According to some post on the web, it seems that visual studio 2008 sp1 is out. Biggest noticable change (for me) is that VS2008 Express received the ability to build Web project and class library which make it possible to build Silverlight 2.0 applications with Express. Yeah.

Amidst all excitement on the new silverlight 1.0 service release, I saw something that interested me more than the update itself. Check out this quote:

our team has been cranking on for Silverlight v1.1 (controls, databinding, layout, templates, styling and a package based app model, phew!)

This of course means that we'll be getting a great full featured set when SilverLight 1.1 beta will be released. I am so excited about this next version! What we'll be getting looks cool:

• Controls: We definitely needed controls in 1.1. I have worked on some of my own layout controls and that is a lot of work. I hope they release these controls really soon.
• Databinding: This will of course be much better than doing everything in codebehind
• Layout: This one in combination with the new control set: I'm lovin' it.
• Templates: While trying to build my own controls I was trying to find a solution for templates. Apparently, the silverlight team is way ahead of me. Go! go!
• A package based app model. I have no idea what that'll accomplish.

Sadly enough, when I attended the TechEd Barcelona beginnen november this year, I have not heard one single person talk about these upcoming additions to silverlight. This is actually the first word I hear from the team about future plans with 1.1. So, in one sentence: Please give us another 1.1 alpha! (a beta would be better of course).

Edit: More big news: Visual Studio 2008 has been released. Please be aware that the release notes tell you that the SilverLight tools are not yet updated to work with the release, so if you need to work on SilverLight, you'll have to stick to VS2008 Beta 2 and have a little patience until the SilverLight tools are updated.

In my previous post I rambled about a bug in the Web Application .targets file. I found an easier solution which does not involve patching any existing files.

The solution is to add the dependency projects to the msbuild target list like so:

msbuild /t:ResolveAssemblyReferences;_CopyWebApplication

     /p:OutDir=..\Drop\bin\ /p:WebProjectOutputDir=..\Drop\

Wow. Life CAN be easy.

Update: Even more so, it is better to use ResolveReferences, which also copies any project assembly references you have in your project.

 Update 2: You actually need to add a 'bin' directory to the OutDir folder. 

Note: Please check out this update for an easier solution.

Today was cool. I managed to get my first continuous build server online at work using CruiseControl.net. I was cool, up to the point when I wanted to deploy the application nice and clean.

After a lot of googling, I found out that Visual Studio internally uses a .targets file to deploy web applications. It is (by default) referenced in your .csproj file of your web application so you don't have to add it to your solution file (which fortunately IS a msbuild .targets file with just another name). The target file is located here in C:\Program Files\MSBuild\Microsoft\VisualStudio\v8.0\WebApplications. You don't need to install Visual Studio on your build server, you just need to copy the MSBuild directory to the program files folder of your build server. You will however need to install the 2.0 SDK on the build server. it includes a hidden target named '_CopyWebApplication' which you can call.  Thanks to links here and here.

To deploy your web application, just enter the following commandline with your current directory set to the project directory:

msbuild /t:_CopyWebApplication

Unfortunately, this is not sufficient, because the build file detects if the output directory is the same as the current directory:

<Target Name="_CopyWebApplication" Condition="'$(OutDir)' != '$(OutputPath)'">

This problem is easily solved:

msbuild /t:_CopyWebApplication /property:OutDir=..\Drop\ /property:WebProjectOutputDir=..\Drop\

Which will effectually copy all the files in the directory ..\Drop\ and remove the added _PublishedWebsites path, 

but Alas! There is a 'problem' with this solution because references DLL's are not being copied over. I did a lot of googling again and i managed to solve it.

You need to change the line to:

<Target Name="_CopyWebApplication" Condition="'$(OutDir)' != '$(OutputPath)'" DependsOnTargets="ResolveAssemblyReferences">

By making the _CopyWebApplication dependant on the the ResolveAssemblyReferences target (which is a built-in target), the correct properties @(ReferenceCopyLocalPaths) has been magically filled and all your files will be nicely copied to your drop directory. I guess that this is a nifty bug in the web project build file.

There you have it. All that remains is to add another MSBuild task to your Cruisecontrol.net configuration file that publishes the result. I leave that as an exercise to the interested reader because my dinner is ready.

As promised, the first beta.

Source code and vs2005 solution

SourceMetrics.zip (71,51 kb)

Sample output

codelines.xml (381,00 bytes)

I'm here to tell you about my latest invention: SourceMetrics. It is a commandline tool to scan a source repository (actually just Visual Sourcesafe for now) and report a cool set of statistics about its contents.

Currently, it reports:

• Total number of files
• Total number of bytes
• Total number of lines
• Total number of code lines
• Total number of comment lines

As this tool reports code as Xml, it will be very easy to integrate with build script and tools like CruiseControl. The effect of having some kind of time-related statistics are huge. As first beta, and more info will be posted lated on.