WCF POST test harness

Writing RESTful JSON services with WCF is fun; in fact I do it for a living.  One thing that every developer finds useful is a small program for testing calls, trying new ideas etc so here is my C# test harness for handling POST requests.

private static void PostThis()
    {
      const string baseUrl = "http://127.0.0.1";
      const string port = ":81/";
      const string serviceEndpoint = "Rest.svc/";
      const string serviceAction = "Client/user@email.com/";

      //Create URL to POST request data too.
      string url = baseUrl + port +
                       serviceEndpoint + serviceAction;
      Console.WriteLine("Making POST request to: " + url);
      Uri address = new Uri(url);

      //Create webRequest
      HttpWebRequest request = WebRequest.Create(address)
                                           as HttpWebRequest;

      //the Method property of the request to POST
      request.Method = "POST";
      //Set the ContentType to json
      //our service is expecting WebMessageFormat.Json
      request.ContentType = "application/json; charset=utf-8";

      //define the parameters
      string data = "{\"password\":\"S3cretP@ssw0rd\"}";

      //Create a byte array from your parameters
      byte[] byteData = UTF8Encoding.UTF8.GetBytes(data);

      //Set the content length in the request headers
      //this may require back end service alterations
      request.ContentLength = byteData.Length;

      //Make the request by streaming the data
      using (Stream dataStream = request.GetRequestStream())
      {
        dataStream.Write(byteData, 0, byteData.Length);
        dataStream.Close();
      }

      try
      {
        //Receive the response
        HttpWebResponse response = request.GetResponse()
                                          as HttpWebResponse;

        //Get the response stream
        if (response != null)
        {
          StreamReader reader = new
                                StreamReader(response.GetResponseStream());

          //Write output to the console
          Console.WriteLine();
          Console.WriteLine("Response: ");
          Console.WriteLine(reader.ReadToEnd());

          //Final cleanup
          reader.Close();
          response.Close();
        }
      }
      catch (Exception ex)
      {
        //Display exceptions
        Console.WriteLine(ex.ToString());
      }

      Console.WriteLine("Press enter to quit");
      Console.ReadLine();
    }

Many examples exist on the web for making POST webRequests but most of them specify the incorrect ContentType for JSON WCF services, usually you will find this:

request.ContentType = “application/x-www-form-urlencoded”;

Instead for JSON services you need to use:

request.ContentType = “application/json; charset=utf-8″;

If you use the incorrect ContentType you will get the following http 400 error

“The remote server returned an error: (400) Bad Request”

Simply change the service endpoints constants at the top of the code to your details and test away. I like to run the code in a simple console application. I may do a separate post on the WCF service configuration if I get time.

Also, if you’re looking for a nifty tool to test JSON response data look no further than these guys http://jsonformatter.curiousconcept.com/

Posted in .Net, Microsoft, Testing, WCF | Leave a comment

More Dell BIOS fun “Strike the F1 key to continue”

Following on from my last post about Dell hardware here it would seem that Dell machines do not like active SATA ports with nothing attached to them. I’m posting an update here in the hope that it will help frustrated people fix their issue.

I was using a Dell Dimension 9200 as a host to patch the firmware of some hard disks (see here for all the fun) and had to enable several of the SATA ports not normally needed.  When I returned the donor machine to its owner I had mistakenly left SATA port 4 enabled and the owner was greeted with “Strike the F1 key to continue, F2 to run the setup utility”.

Receiving this error every time you boot is annoying on a desktop, but a serious problem for a server in a data centre, so if you are scratching your head as to why your machine keeps halting on boot with this rather cryptic message check to see you haven’t left a port active after removing disks or altering the configuration of your machine.

Posted in Hardware | Leave a comment

Using Windows Azure tables to persist session data

Being able to store session data is arguably one of the reasons that the Web features the dynamic applications we all use every day.  If it wasn’t for the session we would have a fairly boring static Web experience.

Windows Azure, being built on top of Windows Server 2008, allows ASP.Net developers to utilise all the session features of IIS.  Additionally the developer can extend these options with the Windows Azure table storage to cheaply and easily persist storage of session data in the Cloud.

Session storage for Windows Azure

The scalability of Windows Azure means that you are going to have at least a couple of web roles and possibly even tens or hundreds of web roles for busy sites.  All these individual web roles need access to a central session store, the best place to put each user’s session data is in Windows Azure table storage.

The process of getting started with storing session data in the table store requires the modification of the following files.

Web.config

As per normal ASP.Net web.config settings you need to modify the <sessionState> tag to point to the custom table storage provider.  Application name is the name recorded in the table when the session detail is logged, used to help identify the session.


<system.web>

<sessionState mode="Custom" customProvider="TableStorageSessionStateProvider">

<providers>

<clear/>

<add name="TableStorageSessionStateProvider" type="Microsoft.Samples.ServiceHosting.AspProviders.TableStorageSessionStateProvider" applicationName="myWebAppName"/>

</providers>

</sessionState>

AspProviders

The sample code for the AspProviders project is availbe from Microsoft.  Search for “Windows Azure Platform Training Kit”.  Presently June 2010 is the latest release.  Looking at the TableStorageSessionStateProvider.cs namespace you can see the code still resides in the Microsoft.samples namespace, hopefully this code will be rolled up into the main Windows Azure SDK at some point in the future.

You can either include the AspProviders project into your solution or compile the DLL and include that.  Whatever you choose make sure you add it as a reference to your web role project.

ServiceConfiguration.cscfg

The most important setting here is the “DataConnectionString”.  The AspProviders uses this to pull all the rest of the settings needed to access the table store for your account, the rest of the settings here are standard for Azure.  I’ve shown the values needed for running in the cloud environment when deployed.

<Role name="myWebAppName">

<Instances count="4" />

<ConfigurationSettings>

<Setting name="AccountName" value="yourStorageAccountName" />

<Setting name="AccountSharedKey" value="yourStorageKey" />

<Setting name="BlobStorageEndpoint" value="https://blob.core.windows.net/ " />

<Setting name="QueueStorageEndpoint" value="https://queue.core.windows.net/ " />

<Setting name="TableStorageEndpoint" value="https://table.core.windows.net/ " />

<Setting name="allowInsecureRemoteEndpoints" value="false" />

<Setting name="DataConnectionString" value="DefaultEndpointsProtocol=https; AccountName=yourStorageAccountName; AccountKey=yourStorageSharedKey " />

</ConfigurationSettings>

</Role>

At this point you will need to add the setting name keys into your ServiceDefinition.csdef or your will get a build error.

Global.asax

You will need to set the configuration setting for the Azure CloudStorageAccount before you try and access the table storage.  This is best done in the Global.asax of the web role project

CloudStorageAccount.SetConfigurationSettingPublisher((configName, configSetter) => configSetter(RoleEnvironment.GetConfigurationSettingValue(configName)));

If all has gone to plan you can run your application and connect to the live Windows Azure services.  Using a tool like Cerebrata Cloud Storage Studio you can look in your table store and see that a new table will be automatically created for you called “session”.  If you perform a task requiring state in your application you should see serialised version of your session data stored in the table.

Posted in ASP.net, Azure, Cloud Computing, Microsoft | Leave a comment

Seagate 2TB ST32000542AS CC35 Firmware upgrade

We have a Synology DS1010+ diskstation that has been having trouble maintaining a iSCSI link to a server here at work.  While Synology issued us a patch that seems to have resolved the issue they highlighted that our disks, Seagate 2TB model ST32000542AS firmware CC34, were not running the latest version.

We initially chose these particular Seagate disks as they are on Synology’s approved disk list plus offer low power, low heat output suitable, so we thought, for nearline storage.  This turns out to be incorrect.  Seagate do not recommend these drives in anything other than “Desktop RAID”, meaning RAID 1 or 0 between two disks only.  It is unfortunate then that we are running a five disk RAID 5 array.  Seagate instead recommend their Constellation ES hard disk for higher RAID levels.

There seems to be a lot of confusion out there on the Internet about the firmware upgrade procedure.  The new firmware is advised for all customers with drives which have the following model numbers ST3500412AS, ST31000520AS, ST31500541AS and ST32000542AS, see here.  However, all drive serial numbers within the family product range are not detected as requiring the firmware upgrade by the tool.  Seagate suggest that customers can use the “Force” method to upgrade their disks.

Upgrade procedure

First download the ISO upgrade image here

You need to connect the disk to a native SATA controller; I used an older desktop machine, removed the main hard disk and plugged in our ST32000542AS drive.

You may also need to change the BIOS settings to ATA mode if it is set to RAID or AHCI

First you may attempt to upgrade the disk automatically, simply follow the onscreen instructions.  If not successful you will have to “Force” it.

To “Force” an upgrade

1.Boot from ISO CD.
2.Read disclaimer, press ESC once read.
3.On the option screen (where you can try an auto upgrade) press [Ctrl] + [C] to break to the command line.
4.Press [Y] on the keyboard to confirm
5.Type the following, then press enter:

FDL486A -m Hepburn -f HECC358H.LOD -s -x -b -v -a 20



6.The firmware flash process will begin.  Once successful you will see the prompt.
7.Power down the machine
8.repeat steps 1 to 7 for each additional disk.

For a full rundown of how I discerned that a) the firmware update wasn’t data destructive and b) that you should indeed “force” the firmware see this very full and informative post between another  ST32000542AS user and Seagate.

http://stx.lithium.com/t5/Barracuda-XT-Barracuda-and/Idle-clicking-of-Seagate-Barracuda-LP-HDs-and-DOA-issues/m-p/55298#M20418

Note: In my experience this process is not data or RAID destructive, but may be in your situation.  If your data is valuable to you please back it up first.  Only attempt this firmware upgrade if you feel competent, I will not be held responsible for your actions are a result of this post.

Posted in Backups, Firmware, Hardware, Servers | 12 Comments

Backup options for SQL Azure

SQL Azure makes the move to using Windows Azure easy for many people.  Lots of companies use SQL server and have existing databases and applications in SQL server.  It’s much less daunting moving your SQL server database than re-coding the storage in Windows Azure tables.

One major sticking point with SQL Azure is that there is no native backup.

Microsoft do backup your database on your behalf from time to time but it is not user accessible and Microsoft say that they use this only to restore your data in severe cases of hardware failure.

Realistically nobody is going to run anything other than a tech demo without a backup solution for their SQL database in Azure, so what can be done?

SQL Azure migration Wizard

Back in the early SQL Azure beta days the SQL Azure migration wizard was the only real way to get data in and out of the cloud databases.  Sure you could use raw BCP but the Migration Wizard is built on top of it.

It’s a good tool, especially considering it’s free and seems to have been continually developed.  My main issue was that it was slow, and in older versions I had to copy the entire database every time rather than taking update snapshots.  This leads nicely onto the next backup option.

Red Gate

Red Gate make, amongst other things, excellent SQL server productivity tools.  I had been a fan of SQL Data Compare for years syncing up staging and production databases.  I originally tested their standard SQL Data Compare tool with SQL Azure and it didn’t work, I was initially crushed.

Over a few weeks of emails and some calls expressing how useful their tools would be if SQL Azure was supported Red Gate started a beta program for SQL Data Compare and SQL Compare (schema compare) Sign up to the beta here

SQL Data Compare is my favourite SQL Azure backup tool as it’s very fast.  Rather than copying all the data, you compare the differences between your last local backup SQL database and your SQL Azure database.  The software then snapshots the differences and updates the local database to reflect the changes from the database in the cloud.

There are a whole host of options when comparing the databases, you can select individual tables and columns and even write your own compare where statements to get super granular backups.

Some people may worry about the cost once it comes out of beta but trust me it’s the best tool a DBA can have, it just works, very, very well.

Data Sync for SQL Azure

Data Sync for SQL Azure has been around since November ’09 and it’s pretty much the same as the Migration Wizard.  Its build on top of the Sync Framework 2.0 but it’s not as customisable as SQL Data Compare from RedGate, but it does allow scheduling of synchronisation so feels more like a backup tool.

The best of a bad situation?

I know many people are waiting for the traditional SQL backup / restore interface and the ability to store their SQL Azure backups in the local data centre blob storage.  I guess due to the multi tenancy of the SQL Azure setup this isn’t easy (or they would have done it already, right?)

All three tools are free (or in beta) when this was posted so try all three.  My personal favourite are the Red Gate tools are they are so much more than just a backup tool, but then they are also the most expensive.

Posted in Azure, Backups, Cloud Computing, Microsoft, RedGate, SQL | Leave a comment

Managing the Azure service as a programmer

I’ve been really, really busy this week both in and out of work and have been struggling to find time to get all the interesting things we have been doing with Windows Azure down on paper.

This morning while troubleshooting a broken SAN that looses its iSCSI connection to the server it made me realised just how resilient Windows Azure and SQL Azure are.

Here is a rundown of what we are running in the Cloud:

  • 3x Production servers hosting two web roles and two worker roles
  • 2x development servers hosting dev copies of the above production code
  • 4x SQL Azure databases

Connected to

  • 200GB used of our 100TB storage allocation
  • Worldwide CDN network carrying 220GB/month of traffic from 18 different local data centres

The server instances in Windows Azure are effectively running in an N+1 cluster in what Microsoft calls the ‘Azure Fabric’ so each machine in the group shares the load but as machines fail or are taken down for maintenance another machine is brought online, on the fly, to take on the load up to our configured machine limit.  The SQL Azure databases don’t allow the end users to set the level of redundancy yet, Microsoft has decided that triple redundancy works best for now.

Clustering isn’t easy, managing the hardware is just one part, having spares for the inevitable failures as well as managing patches and configuration changes is a full time job even for a clever person. This is something small companies and start-ups can’t generally afford, thats why I think Windows Azure is a valuable service that allows start-ups to focus on their key product and not infrastructure.

If you add on the management of keeping our 100TB data allocation up 24/7 and the fact that this integrates into the Microsoft worldwide CDN (the same one that powers Windows updates, Zune video and Bing Maps. source zdnet) seamlessly to improve data caching in local environments is a massive undertaking, even for a large multinational company.

My day-to-day involvement in running all this?  Five minutes glancing at our usage reports and checking the Windows Azure service stats to make sure it’s all running smoothly.

As a developer who has worked in small companies before where the DBA is also the SQL programmer and maybe even handles the firewall security too, being able to hand off the hassle of making hardware run smoothly is simply awesome. 

Having Microsoft Azure as a platform to work on gives you such confidence to build great services because I know if we got 100K extra users overnight all I have to do it increase the servers a couple of notches and we can cope.  What I don’t want to worry about is having finance take two weeks to order me servers that arrive un-configured while users struggle and quit the service because its slow.

So as busy as I am this week, one thing I don’t have to worry about is the stability and performance of the Azure platform.

Posted in Azure, Cloud Computing, Hardware, Microsoft, Servers | Leave a comment

Network issues when migrating Hyper-V servers

There are two very important rules for Fridays.  Never do a config change or code release unless absolutely necessary and make sure to reward yourself and your team by having a good lunch or a beer after work.

Sadly this Friday we forgot one of these rules (the good fish & chip lunch was never in doubt!) and ended up leaving the office around midnight after hitting a snag with a relatively simple Hyper-V guest server migration between physical Hyper-V host servers.

Here is a diagram of what we had hoped to achieve.

Microsoft Hyper-V server migration plan

As with most server work of this nature it needs to occur out of hours to minimize disruption.  All we had to do was shutdown one of the Hyper-V guest virtual machines and export it to its new home on the other physical server.  I was confident we could do it in a couple of hours after work on Friday night.

Both physical servers have the same connectivity, one physical management interface and two iSCSI connections to the shared SAN.  These connections are mapped through to the virtual networking inside Hyper-V.  Shown below is the setup of one of the servers, note the static MAC address box has been changed back to dynamic.

The server had originally been running a specific MAC address.  When you first setup the virtual connections they are all named the same with #1, #2 and #3 between the multiple connections.  This makes discovering which connection is for the management interface and which is for the iSCSI interface extremely difficult without a fixed MAC address as a point of reference.

The server exported to the new host no problem and was back up and running in no time, however when we tried to ping the domain controller (also virtual) we got all kinds of mixed errors, destination host unreachable and Request timed out faults.  Other machines in the virtual network could still ping the domain controller and the domain controller could ping both physical host machines.

After much head scratching it was spotted in the configurations that the exported server had the same fixed MAC address as the domain controller!  Somehow it must have got misconfigured or reset when the server was exported.

Personally I blame myself for being so confident that it was such an easy move that we would be finished and in the pub for 8pm that night.  It was such a small mistake that lead to a lot of time being spent chasing ghosts in the networking.  It goes to show there is no such thing as a small job that doesn’t need a good plan to be implemented successfully and on time.

Note: if you like the Viso diagram check out Jonathan Cusson’s Hyper-V template

Posted in Hyper-V, Microsoft, Networking, Servers | Leave a comment

Windows Azure Cloud platform 101

Windows Azure has been available for some time now and there are already a lot of good resources on the Internet.  Having been working with Windows Azure since PDC 2009 I like to think I’ve discovered aspects of Azure not covered in most getting started guides.  I hope to cover most of the topics surrounding Windows Azure and SQL Azure in future posts but for now what exactly makes up the Azure Cloud?

In a sentence Windows Azure is Microsoft’s Cloud computing platform.  The idea of Cloud computing has been around for a while with the likes of Amazons Web Services featuring their EC2 platform and more recently the Rackspace Cloud, but Windows Azure is much simpler that either of these while at the same time being hugely powerful.

The key idea of Windows Azure is to get websites and web services up and operational fast.  As a programmer it allows you to concentrate on the code, removing the need to worry about daily server health checks and other sys admin jobs.  It is a great confidence to know that if you coded it right Microsoft will deliver it right.

As an on-demand service Windows Azure is perfect for fast growth small companies.  We started running on the lowest level of service with our beta program, paying for only the service we needed.  Since then we have expanded the services, added redundancy and extra features like a worldwide CDN and a queue processing service for our email application, but we are still only paying for the capacity we need yet are able to reach a worldwide customer base.

Two things impress me about the Windows Azure service as a on-demand platform.  One, we don’t need to hire additional staff to deal with server incidents as Microsoft maintain the hardware for us.  Second is that if we need additional processing power for the upcomig comic-con or other high volume event we can scale up for the period of the event then scale back down.  Having worked previously in environments where new hardware could take weeks to arrive and configure being able to dial up the service as needed means smooth numbers on our service status.

Having SQL Server in the Cloud in the form of SQL Azure is a massive win for Microsoft and I’m really pleased they got it out.  Originally, pre PDC 2009, Windows Azure Tables would form the only data storage environment.  The only problem is Windows Azure Tables are not relational and many people (including me) had been expecting to just plug their SQL server database into the Cloud and be up and running.

Microsoft made a huge effort to bring a very good version of SQL to the cloud in the form of SQL Azure.  It’s still missing some key features for many larger organisations (such as an integrated backup and restore feature) but there are work arounds that allow for backups which are satisfactory against the alternative of having to employ a SQL server clustering expert and buying three servers outright.

The third part of Microsfts Cloud infrastructure is AppFabric.  It’s had a few name changes through its development (codename “dublin”) and is very closely related to Windows Server 2008 R2 AppFabric so be careful when searching for AppFabric details online.

AppFabric currently comprises two services for the Cloud, the Service Bus and Access Control.  A third and very important Cache service (codename “Velocity”) will hopefully be available soon.  The Service Bus is a powerful Enterprise component allowing you to expose data in a controlled way to 3rd party applications crossing corporate firewall boundaries.  Access Control services allow you to centralise security authorization for RESTful web service that can accept a variety of identity providers (including on-premise domain authorisation).  As more providers become available in future you can allow users to authenticate with these new providers without writing any new service code.

I hope that explains a little more around what the Windows Azure platform is about and I urge you to give the platform a go while Microsoft run some great introductory offers.  See here for details: http://www.microsoft.com/windowsazure/

Posted in Azure, Cloud Computing, Microsoft, Windows | Leave a comment

The dilemma with old hardware

I’m sure may people who have an interest in computing have a collection of hardware similar to this.

stack of old computer hardware

so many good clock cycles here

I’m also sure that any respective wife / partner / flat mate will have told them more than once to clear out “that old junk”.  To us it’s not old junk, its memories of Doom on a Pentium 2 or getting online with an external 3com 33kbps modem.

Time moves on, and it appears to move fastest in the world of computing.  However given a sizable collection of disused and donated hardware it’s possible to make a perfectly good light use web / email PC for somebody in need of a basic computer.

From my collection I created a perfectly usable light usage PC

  • mATX AMD K7VM4 motherboard
  • AMD xp1700+ CPU
  • 512MB DDR RAM
  • GeForce2 AGP graphics
  • Onboard sound
  • 80GB Western Digital IDE hard disk
  • DVD / CD-RW drive IDE

Of course some hardware is beyond practical use, an old AMD-K6-2 runs at 500Mhz, it has less power than my wifes phone.  Equally the machine it came out of runs on an old, inefficient power supply and supports bus technology which simply isn’t practical anymore.

So what to do then?  In the UK and Europe you’re not simply allowed to bin old computer parts.  The WEEE Directive (http://en.wikipedia.org/wiki/Waste_Electrical_and_Electronic_Equipment_Directive) covers all parts of recycling for electrical goods, including computers.

If you bought the computer item from a commercial retailer they must provide you with information on how to safely dispose of the electrical item.  In addition they must provide a free scheme where you can swap old hardware.  There are many caveats to this scheme and the retailer may well only accept equipment they sold or accept whole PCs, but it does apply to online retailers who sell computer equipment.

In reality I’ve found that your local council recycling centre will accept empty computer cases as scrap metal and have special sections for computers and monitors.  You can check online directly to see what your local centre accepts here http://www.valpak.co.uk/dts/page1734.aspx or check out your local council details http://www.recycle-more.co.uk/la/la_detailsla.aspx

As business must now pay to have electronics recycled a downside is that many charities are no longer interested in old hardware as it will cost them to dispose of it if its truly useless.  Other businesses will often no longer give away perfectly good hardware because it has reached its accounting lifespan as they may be liable to cover the recycling costs if they give it away.

Posted in Hardware | Leave a comment

Excellent service at Profile Automotive

I had to get our Focus ST serviced and put through its first MOT, where else to take it but the north of England’s premier tuner, http://www.profile-automotive.co.uk

I had been expecting to have to mooch round the Trafford Centre shopping arcades but Gary, the owner, very kindly offered me a desk, chair and internet access while the work was being done.

All the staff are very welcoming and did a great job on the service, if you’ve got an ST and live in the north it’s a no brainer to have all its work done here.  If I didn’t have a turbulent time ahead I would have been getting my ST fettled as Profile offer some very nice tuning upgrades.

Gary was also very happy to let me take their demo Focus ST out, fitted with their Wolf 295 pack, it certainly goes as well as it looks!

CodeRed: Focus ST

Posted in Cars | Leave a comment