You will find below different links to the blog or Web site which kept Matricis' bloggers interest during the week ending on June, 16th.

Cloud

• Azure VM Access Control with PowerShell (Tyler Doerksen) – Great with this post we now know all the new features that came up last week: PowerShell command.
• 10 Dos and Don'ts for Running Proof-of-Concept Projects in the Cloud – Nice post that highlights the pros and cons.
• Conflicts: Windows Azure Mobile Service to SQLite Data Sync for Windows Phone and WinRT with Portable Class Libraries – Nice post about possible sync conflicts.
• BASE Jumping in the Cloud (Jason Bloomberg) – Interesting post that explains why a re-architecture is more than often needed when moving our application in the cloud.
• Windows Azure: Major Updates for Mobile Backend Development (Scott Gu) – Another big release for Windows Azure that boost the Mobile Services offers.

Databases

• Amazon RDS Lowers On-Demand and Reserved Instance Prices Globally – And the battle for the lowest price continues (that's good for us) this time the databases are targeted.

UX

• Bundling and Minification in ASP.NET MVC (Suprotim Agarwal) – Minification is a important process to have a nice UX. It's now easier than ever.
• JavaScript: Avoiding Repeated Work (David Walsh) – Nice snippets to help us out in  Javascript
• Jeff Gothelf on Lean UX (Martin Cooper) – Nice interview about Lean UX
• NSHipster Quiz #2 (Mattt Thompson) – Are you able to get 100% ? Really hard !

Programming

• Using Duck Typing to Avoid Conditionals in JavaScript – Good explication of Duck Typing method by refractory code step by step.
• Introduction to Unit Testing (Don’t Worry, Your Secret is Safe with Me) (Erik Dietrich) – Interesting post about unit testing. What is it, and what it isn't.
• Claim based Authentication and WIF : Part 1 - First of a series, this post describes what is a claim base process and who are the players involve in it.
• Stop Doing Internet Wrong (Scott Hanselman)- Interesting list of some simple things to do to get a better experience online.
• Using Azure Mobile Services in SharpDevelop – Nice alternative to Visual Studio. Check out the video here to see it developing with Azure.

Miscellaneous

• Marketing 101 for developers (Phil Cross, saraallison Natasha Joseph) – Why a developer should care about marketing? Because we need it? This post answers the question.
• The Importance of Closed Captioning – How To and Why Subtitle (Scott Hanselman) – Really interesting post. Doing subtile look like a lot of effort, but it's worth it. This post contains sample, tutorial and tools references.
• 8+ Practical Tips for Avoiding Pickpockets - Simple tips that can save you a lot of problems when you are traveling.

These posts are from François Boucher and Sebastien Leclerc, and are also posted on http://www.frankysnotes.com/2013/06/reading-notes-99.html.

You will find below different links to the blog or Web site which kept Matricis' bloggers interest during the week ending on June, 9th.

Cloud

• How To: Setup Windows Azure PowerShell  – Quick post that explains how to get ready to play with Azure with PowerShell.
• Cloud Cover Show, Episode 109 – Using Windows Azure Active Directory from Windows Store apps – Nice episode with V. Bertocci that get us up to date on all the news Windows Azure Active Directory (AD) features.
• Running WordPress with SQL Azure – What a great workaround! It could be super if we had this option when creating the application.
• Is Amazon becoming the Microsoft of cloud? Hold that thought | ZDNet (Service Oriented Joe McKendrick) – Interesting post about the future of the major players who wants to rules the cloud.
• Microsoft’s “Understanding the Microsoft Cloud” White Paper Misses the Windows Azure IaaS Boat (Roger Jennings) – Hard critic of a Microsoft White Paper. However, it's true that this paper does reflect Azure IaaS reality.
• Cloudy Thoughts by SyntaxC4 – Quick post that introduces some cool new feature for PHP in Azure…
• Windows Azure PowerShell June 2013 Update for IaaS and PaaS (Michael Washam) – With new Azure capabilities comes great PowerShell functionality… this post is all about that. Also Michal Washam is announcing that he is quitting Microsoft
• Cloud Cover Show, Episode 110 – Windows Azure BizTalk Services – Great show with a demo done by no other than Guru Venkataraman.
• Checking Azure Cloud Service SDK Version with PowerShell (Tyler Doerksen) – Nice and useful script.
• Azure VM Access Control with PowerShell (Tyler Doerksen) – Great with this post we now know all the new features that came up last week: powershell command.
• Windows Azure: Announcing New Dev/Test Offering, BizTalk Services, SSL Support with Web Sites, AD Improvements, Per Minute Billing (Scott Guthrie)- This post is a mandatory reading. Know from the source what's new! So many good things, that I don't know where to start…

Databases

• Azure does Powershell too (Grant Fritchey) – Nice quick introduction to Azure Sql PowerShell cmdlets.

UX

• Principles of Flat Design (Carrie Cousin) – Il faut rester simple…
• The State Of Responsive Web Design (Stéphanie Walter) – Quoi faire avec les images ?
• Pourquoi j'ai choisi de me passer de l'App Store et de l'Android Store (Jérôme Morlon) – Des web app qui vont remplacer les app natives ? Je ne crois pas…

Integration

• Walkthrough of New Windows Azure BizTalk Services – Again a great post/ tutorial from Seroter. This is THE place to start to know more about the Biztalk services (WABS)

Programming

• ELMAH: Error Logging Modules and Handlers for ASP.NET (and MVC too!) (Scott Hanselman) – This post is not really recent, but really interesting. If you still don't ELMAH this post is for you.
• Making Computer Science Cool (Jonathan Rozenblit) – Really good resources post that gives place to get started based on our level of confort. It could also be use at home for our kids… I wonder if this kind of information is also available in other languages…
• TypeScript: Scaling Up JavaScript – Great video from TechEd, it was hard to wait until the end to start doing TypeScript…

Miscellaneous

• Are you secure? WiFi Honeypots, Pineapples and SSL with Troy Hunt – And I though I knew the danger of the WiFi. You must spend 30 minutes and listen this post… and re-share!
• Announcing "Get Involved" from Tekpub – Enhance your career by engaging with your peers (Scott Hanselman & Rob Conery) – Really great video that explains how to be involve in the community, the good way. Really nice work.
• Live Demonstration: Hacker Tools You Should Know and Worry About (Hasain Alshakarti, Marcus Murray) – Wow! Very "perturbing" video that shows a lot of different tools to help you… or not.

These posts are from François Boucher and Sebastien Leclerc, and are also posted on http://www.frankysnotes.com/2013/06/reading-notes-98.html.

Tout d'abord, pourquoi copier un Virtual Hard Drive (disque dur virtuel) ou un blob d'un abonnement Windows Azure à un autre? Ce pourrait être pour faire des sauvegardes, parce que votre période d’essai de Windows Azure se termine, ou pour obtenir une copie de serveurs virtuels de clients pour enquêter sur un problème.
Donc, quand un client m'a demandé s’il était possible de le faire, ma réponse a été: "Oui, à l'aide des outils en ligne de commande pour Azure". Et puisqu’il était à la recherche d'une solution simple, je lui ai écrit un petit script que je vais partager ici afin que tous puisse en profiter.
 

Pour commencer

Pour faire la copie, il vous suffit de faire une commande PowerShell. Afin de pouvoir exécuter cette commande, vous devez avoir le Windows Azure command-line tool (l’outil en ligne de commande de Windows Azure) déjà installé. À la fin de ce billet, je vais donner le script pour son installation.
 

La commande

Les prérequis étant déjà installés, vous pouvez exécuter cette ligne de code, à partir du Windows Azure command-line tool:

azure vm disk upload <source-path> <target-blob-url> <target-storage-account-key>

La source

Dans cette commande, vous devez remplacer <source-path> par l'url du VHD ou du blob que vous souhaitez copier.
Vous pouvez obtenir cet url via le portail de Windows Azure à l'aide de votre compte associé à l’élément que vous souhaitez copier.

1. Sur le côté gauche de l'écran, cliquez sur l'icône de stockage;
2. Puis cliquez sur le nom du stockage;
3. En haut de l'écran, cliquez sur conteneurs;
4. Enfin, une fois que la liste des conteneurs apparaît, cliquez sur le conteneur.

La destination

Maintenant que la source a été identifiée, il faut spécifier la destination. Nous devons remplacer le <target-blob-url> par l'url du conteneur dans l'abonnement Windows Azure destination. Si le conteneur existe déjà, il suffit de connecter à ce compte et de suivre les étapes précédemment décrites. Dans le cas contraire, vous devez créer un nouveau conteneur en utilisant le bouton "+" en bas à gauche de l'écran.

Le plus simple est de créer le conteneur avec un accès public au moment du transfert. Vous pourrez définir cette option lors de la création ou la modifier en utilisant le bouton « Modifier un conteneur », situé en bas de l'écran.
Sélectionnez ensuite Conteneur public, comme propriété d’accès.

La clé

Finalement, nous devons spécifier la clé de compte de stockage et remplacer le <target-storage-account-key> par cette dernière. Vous pouvez trouver la clé par l’entremise du bouton « Gérer les clés » du tableau de bord du stockage. Vous pouvez utiliser la clé primaire ou la clé d'accès secondaire.Installer Windows Azure command-line tool.

 

 

 

 

 

 

 

 

@powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://bit.ly/psChocInstall'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin

@powershell "cinst nodejs.install" && SET PATH=%PATH%;%ProgramFiles(x86)%\nodejs

@powershell -ExecutionPolicy unrestricted "npm install azure -g" && SET PATH=%PATH%;%USERPROFILE%\AppData\Roaming\npm\

@powershell azure

pause

Références

• Windows Azure command-line tool
• Node Packaged Modules or Node.js
• Chocolatey

Vous avez fait le camp d'entraînement (DevCamp) et des ateliers réguliers, mais vous en voulez plus?

Soyez prêts, car à partir du 15 octobre, une nouvelle série virtuelle d’ateliers avancés sur Windows Azure commence. Un expert de Matricis sera là pour répondre à toutes vos questions (en français ou en anglais)  sur le chat avec les experts de la communauté Windows Azure du Canada!

15 octobre – Cloud Variation

Explorez les options d'infrastructure offertes par Windows Azure, telles services infonuagiques, machines virtuelles et sites web. En comparant ces différentes options, vous terminerez l'atelier mieux équipé pour choisir celle qui conviendra le mieux à vos projets.

Enregistrez-vous dès aujourd’hui: aka.ms/cloudvariations

22 octobre – Building Connected Apps with Windows Azure

Découvrez comment les différents services Windows Azure peuvent être intégrés dans des applications pour étendre leurs fonctionnalités. Vous apprendrez à répondre aux besoins communs tels que la capacité de calcul, le stockage, l'authentification, etc. Pour démontrer des concepts, cet atelier se concentre sur Windows 8 et Windows Phone, bien que les concepts puissent être appliqués à n'importe quelle plateforme, y compris Android et iOS.

Enregistrez-vous dès aujourd’hui: aka.ms/connectedappswithazure

29 octobre – Migrating Apps to Windows Azure

Explorez les différentes options de migration d’applications. Soyez au courant des préoccupations et des considérations à garder à l'esprit lors de la planification d'une migration, et apprenez à mettre en œuvre un plan de migration pour migrer les applications locales (ou hébergées traditionnellement) vers Windows Azure.

Enregistrez-vous dès aujourd’hui: aka.ms/migratingtoazure

Pour plus d'informations, vous pouvez aller sur les ateliers de Windows Azure Virtual.

Microsoft BizTalk Server est un produit qui est difficile à ajuster pour améliorer la performance puisque chaque solution demande des paramètres différents. La quantité de messages reçus, leur taille, le type de traitement requis, les besoins de traçabilité et le mode de transport des messages sont toutes des variables qui influenceront les paramètres optimaux d’une solution BizTalk.

Le présent article a pour but d’identifier les paramètres importants à configurer dans le tableau de bord des paramètres BizTalk (BizTalk Settings Dashboard) et d’évaluer l’impact du suivi (tracking) sur la performance.

Séparation des hôtes

Dans tous les environnements BizTalk en production, les fonctions devraient être séparées sur différents hôtes. La configuration de base recommandée est d’avoir un hôte dédié pour la réception, l’envoi, le traitement et le suivi des messages. Ensuite, selon la solution implémentée, les hôtes peuvent être séparés pour différentes raisons :

• la sécurité;
• la taille des messages différente sur certains ports ou orchestrations;
• la conservation de l’ordre des messages;
• optimiser la latence sur certaines parties de la solution;
• etc.

Throttling

Le throttling (limitation en français) est un mécanisme de protection permettant à BizTalk de limiter le débit de messages dans certaines conditions. Ce mécanisme permet aux autres processus BizTalk qui s’exécutent en parallèle de continuer normalement lorsque le serveur est surchargé.

Pour des hôtes 64 bits, les paramètres par défaut doivent être modifiés parce qu’il est possible d’adresser plus de mémoire sur un hôte 64 bits.

Lorsque BizTalk est utilisé en mode « messagerie » (sans orchestration), deux types de throttling peuvent s’appliquer, le « resource based throttling » et le « rate based throttling ». Le premier type se produit lorsque la quantité d’une ressource, mémoire ou CPU, est surutilisée. BizTalk ralentira le débit de messages pour laisser une certaine quantité de ressources aux autres processus.

La limitation basée sur le débit se produit lors de la publication des messages (dans le MessageBox) ou lors de la livraison des messages. Lors de la publication des messages, le throttling se produit lorsque le rythme auquel les messages sont reçus excède le rythme auquel les messages sont publiés. Le même principe est appliqué lors de la livraison des messages. Lorsque le rythme de livraison des messages est en deçà du rythme de traitement des messages, le throttling s’active. Les seuils peuvent être modifiés dans le tableau de bord des paramètres BizTalk.

Intervalles d’interrogation des hôtes

Les intervalles d’interrogation (Polling Intervals) sont les principaux paramètres à modifier lorsque l’on désire réduire le temps de traitement global dans BizTalk. Il faut être vigilant lorsqu’on modifie ces paramètres parce que plus le temps est court, plus le message se traitera rapidement, mais plus le temps est bas, plus SQL Server sera sollicité. Il faut trouver un équilibre selon la solution entre la latence requise et la performance globale de la solution. Généralement, la valeur minimale recommandée est 50 ms.

Une des améliorations de Microsoft BizTalk Server 2010 par rapport à BizTalk 2009 est la possibilité de régler ces paramètres au niveau des hôtes plutôt qu’au niveau du groupe. C’est un élément qui apporte beaucoup de flexibilité lorsque la latence est importante seulement sur certains ports.

Paramétrage des instances d’hôtes

Malheureusement, les paramètres des instances d’hôtes par défaut ne sont pas ceux qui sont recommandés par Microsoft. À chaque fois qu’un hôte est créé, les paramètres doivent être ajustés aux valeurs suivantes :

Ces paramètres sont adéquats pour un serveur avec un seul processeur. Le paramètre « Maximum worker threads » doit être  de 100 par processeur.

Tests avec BizTalk Benchmark Wizard

BizTalk Benchmark Wizard (http://bbw.codeplex.com/) a été utilisé pour vérifier l’impact d’utiliser le mode 32 ou 64 bit et l’impact des paramètres recommandés versus les paramètres par défaut. Les paramètres de throttling n’ont pas été modifiés parce qu’ils n’ont pas d’impact dans un environnement contrôlé où la taille des messages et le débit est constant.

Voir http://blogs.msdn.com/b/ewanf/archive/2010/01/12/benchmark-your-biztalk-server-part-3.aspx pour la description de la solution de BizTalk Benchmark Wizard. L’application envoie le message suivant avec Loadgen.

Environnement :

Machine virtuelle VMware Player avec 2 vCPU, 5 Go de RAM, Windows Server 2008 R2 avec Microsoft BizTalk Server 2010 et SQL Server 2008 R2 installé sur le même serveur.

Paramètres par défaut 64 bits

Paramètres recommandés (100, 25, 100, 25) 64 bits

Paramètres par défaut 32 bits

Paramètres recommandés (100, 25, 100, 25) 32 bits

Avec le scénario de BizTalk Benchmark Wizard avec le serveur de test, l’impact des paramètres est minime. Avec un serveur plus puissant et des messages plus volumineux, l’impact des paramètres aurait probablement été plus significatif.

Impact du suivi sur la performance

Dans la plupart des guides d’optimisation de performance de BizTalk, il est toujours mentionné de réduire le suivi (tracking) au minimum et de le désactiver si possible. En réalité, le suivi est souvent nécessaire pour offrir de la traçabilité et faciliter la résolution de problèmes. Pour réduire son impact, un serveur BizTalk peut être dédié au suivi.

Tests du suivi sur BizTalk Benchmark Wizard

Tous les tests ont été réalisés avec les paramètres recommandés par Microsoft (100, 25, 100, 25) sur des hôtes 32 bits.

Messaging seulement, suivi désactivé

Messaging seulement, suivi des propriétés dans tous les ports et pipelines

Messaging seulement, suivi des propriétés et du corps des messages dans tous les ports et pipelines

Les paramètres des hôtes étaient les mêmes pour tous les tests. Suivre seulement les propriétés des messages et les événements occasionne une pénalité d’environ 25%. Avec des disques séparés pour la base de données de suivi, la pénalité aurait probablement été moindre. En contrepartie, le suivi du corps des messages diminue la performance d’environ 60%.

Le suivi du corps des messages devrait effectivement être réduit à un minimum, mais le coût de suivre les propriétés est relativement faible. À moins que la partie BizTalk de l’environnement d’intégration soit presqu’à pleine capacité, activer le suivi des propriétés sur certains ports ne devrait pas causer de problème de performance. De plus, normalement, le suivi ne sera probablement pas activé sur tous les ports.

Conclusion

Nous avons observé que l’impact des paramètres recommandés était minime et que l’impact sur le suivi était énorme. Les conclusions apportées dans cet article peuvent seulement être appliqués dans une solution réelle si la taille des messages reçus est extrêmement petite et qu’il n'y a que très peu ou pas de traitement. Dans tous les cas, il est préférable d’utiliser les valeurs recommandées par Microsoft comme point de départ. Ensuite, le suivi du corps des messages devrait être éliminé ou réduit au minimum. C’est de loin le paramètre qui influence le plus la performance d’un environnement BizTalk en santé.

La performance d’un environnement d’intégration ne se résume pas au contenu de cet article. Le guide d’optimisation des performances de BizTalk indique toutes les techniques et meilleures pratiques pour améliorer la performance des solutions BizTalk.

S'applique à

Microsoft BizTalk Server version 2009 et 2010

Références et liens intéressants 

http://msdn.microsoft.com/en-us/library/aa561380(v=bts.20).aspx

La solution de BizTalk Benchmark Wizard

http://blogs.msdn.com/b/ewanf/archive/2010/01/12/benchmark-your-biztalk-server-part-3.aspx

Considérations pour un serveur BizTalk 64 bit

http://msdn.microsoft.com/en-us/library/ee308808(v=bts.10).aspx

BizTalk Server 2010 Performance Optimization Guide

http://www.microsoft.com/download/en/details.aspx?id=10855

Configuration du groupe, des hôtes et des instances d’hôtes utilisés dans les tests

Microsoft BizTalk Server orchestrations essentially allow running a sequence of steps in a process, which can involve several systems. Orchestrations can quickly get hard to understand and maintain because of various sources of complexity, as shown below.

This article describes a few practices, in addition to ones that are already well documented^[1], that can be used to reduce the complexity of orchestrations and make them easier to understand.

Variables and messages scopes

In C# code, a good practice is to declare variables as close as possible to where they are used, so that the variable's role becomes more obvious. 

Example C# code where scope is wider than necessary

Improved C# code, where each variable's scope is restricted to where it is being used

An analogous practice in a BizTalk orchestration is to declare variables or messages in the scope where they will be used, instead of declaring all variables and messages at the top level of the orchestrations. It's even usually a good idea to introduce scopes in orchestrations whose role is simply to group a sequence of related steps, including the variables and messages used by these steps.  This way, the variables/messages will be encapsulated in the context where they are relevant. It also improves performance by reducing the number of variables that have to be serialized at each persistence point.

Example orchestration with scope wider than necessary

Refactored example: all messages, variables and correlation sets moved to the scope where they are used

This practice can make a complex orchestration significantly easier to understand, which is important for maintainability. When looking at an orchestration's variables/messages, a developer will want to know:

• What’s the role of the variable/message?
• Where's the variable/message used?
• What are the impacts if a change is made to the way the variable/message is used (for example for a fix, requirements change or refactoring)?

If an orchestration has 30 variables or messages declared in "global" scope (top level), a developer would have to look at each shape to find the answer to these questions. This can make understanding an orchestration very hard and even harder to modify it.

To avoid this problem, it is better to move, as much as possible, variables and message declarations to the smaller scope where they will be used. This way, to analyze the role of a variable/message or the impacts of a change, a developer will only have to look at a few shapes instead of the whole orchestration.

Obviously, there will still be messages at the "global" scope (at least the initial message an orchestration receives), but these should be kept to a minimum.

Split orchestrations into smaller orchestrations

The "Call Orchestration" shape can be used to call orchestrations that contain common code that is reused across several orchestrations. However, its usefulness is not limited to enabling code reuse. An even more beneficial use of this shape is to break down a complex orchestration into smaller ones, even if each smaller orchestration is not reused anywhere else.

Each smaller orchestration can be given a descriptive name saying what it does. This allows having a high level orchestration that's mostly composed of a sequence of calls to other orchestrations. By reading the name of each called orchestration, a developer can understand what the higher level orchestration is doing, without looking at the technical details.This is similar to breaking down, in C#, a complex method into smaller methods.

Note that grouping variables into scopes can be a good first step before splitting some parts of an orchestration into smaller orchestrations: start by dividing the orchestration into scopes, and then move the more complex scopes to a separate orchestration.

The example below shows how the example in the previous section could be refactored into smaller orchestrations:

Refactored orchestration: delegates most of its work to other called orchestrations

Information hiding by using object-oriented classes in orchestrations

Putting messages/variables in orchestration scopes is one basic way to hide information in the context where it is relevant. Splitting complex code in several smaller methods (or smaller orchestrations) is another way. To go further, it's also possible to uses .Net classes to encapsulate (hide information) by moving code and its associated data to a class.

This is a widely used practice in C# applications, but it can also be used with BizTalk orchestrations. To use it, we take advantage of the fact that an orchestration variable can be any .Net class, as long as it's Serializable (marked with the [Serializable]attribute). (non-Serializable classes could be used in atomic scopes, but this is usually not the preferred solution)

A serializable class used as an orchestration variable can encapsulate (by using private members) some state about the process being performed by the orchestration. If the orchestration is dehydrated, all the class member's (including private ones) are serialized and persisted to BizTalk's database. When the orchestration is rehydrated, that state is restored.

This means that the orchestration can delegate some steps of a process to a class, to simplify the orchestration. Also, since the variables required by the class are kept as private fields of the class, the orchestration does not need to know about these variables. Therefore, it is usually a better solution than simply calling classes with static methods, which would require passing the values of several variables between the orchestration and the various methods it invokes throughout a process.

For example, the sequence diagram below shows the interactions between an orchestration and a class with static methods. The orchestration has to keep variables for the return values of each of the class's static methods, and then pass these values to other methods of the same static class.

Overview of procedural interactions between an orchestration and a static class

This can be simplified by keeping intermediate values as private fields or local variables in the class. Also, not all methods of the class need to be public, some can be made private, and only the public methods need to be invoked by the orchestration. Finally, the "MapInternalClientIdToOtherSystemClientId" method should be moved to another class, so that each class has a single responsibility (one class for file name generation, another for Client ID mapping). However, the orchestration needs only to interact with the "FileNameGenerator" class, and the exact type of ClientID needed for the file names (other system ClientID instead of internal ClientID) is a fact hidden to the orchestration.

Refactored object-oriented example, with reduce number of informations that must be known to the orchestration

This allows limiting the impacts of a change. Instead of having ripple effects throughout an orchestration, a change can now be done within a class. The change is also easier to test using simple C# unit tests (for example using NUnit^[2] or MSTest^[3]).

When designing these classes, take in consideration the SOLID principles^[4]. One of these principles is the Single Responsibility Principle, which states that a class should have only one responsibility. Also note that it's possible for a BizTalk orchestration to call a class which itself calls other classes in order to better separate responsibilities across classes, as long as all the classes are Serializable. 

More complex cases could even use objects that are part of a domain model (designed using Domain-Driven Design), or design patterns such as the State pattern. However, this possibility can be taken too far by giving too many responsibilities to BizTalk (the orchestration and its associated classes). Orchestrations are useful to coordinate between different parts of a business process (for example, by calling web services that represent a business application's public interface), but they are usually not the best location for the business logic itself.

Conclusion 

BizTalk Orchestrations, like .Net code, can quickly become hard to maintain if they are too complex. This article described a few practices that are commonly used in object-oriented languages, and has shown how the same practices can be applied to BizTalk orchestrations:

• Limit a variable's scope to the smallest possible scope where the variable is used;

• Breaking down complex components into a set of smaller ones;

• Information hiding using private fields in .Net classes, and public methods that perform work on the object's private state, which prevents the calling code from having to know how the object's state is managed.

Applies to

 All Microsoft BizTalk Server versions from 2004 to 2010.

References

• ^[1] Other well-documented good practices to follow when developing orchestrations:
  • http://social.technet.microsoft.com/wiki/contents/articles/5210.biztalk-server-2010-orchestration-best-practices.aspx
  • http://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2012/07/09/five-rules-to-simplify-large-scale-biztalk-orchestration-development.aspx
• ^[2] MSTest: http://en.wikipedia.org/wiki/Visual_Studio_Unit_Testing_Framework
• ^[3] NUnit: http://www.nunit.org/
• ^[4] SOLID principles:
  • http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod
  • http://lostechies.com/chadmyers/2008/03/08/pablo-s-topic-of-the-month-march-solid-principles/

This serie is not meant to be an exhaustive list of all tools. Some other excellent tools are surely available. If you think I have forgotten one or want me to talk about one, let me know. I will be more than happy to adding it to the list.

Already in the Azure Tools Belt:

• SQL Azure Migration Wizard

What is WASABi?

To achieve elasticity until now, you needed to do it manually through the Azure Management Portal or writing your own code using the REST API. Using WASABi, you just need to define some rules and the application will scale automatically: It can use a schedule or can be triggered by metrics, for example running fewer instances at night or adding extra instances if the CPUs are used at more than 80%.

The auto-scaling application block can be hosted either in a Windows Azure role or in an on-premises application. The auto-scaling application is typically hosted in a separate application from the target application that you want to scale.

Various scenarios are available to help you manage the auto-scaling by dynamically changing instance counts or performing application throttling of web/worker roles. The rules can auto scale based on timetables or metrics collected from the application and/or Windows Azure. You can even use notifications to preview any scaling operations before they take place and you can also use some PowerShell cmdlets to manage the autoscaler. You can constrain the auto scaling by:

• Setting the instance counts upper and lower bounds
• Preventing fast oscillations in the number of role instances with the stabilizer
• Limiting scaling operations acknowledging billing hours

How to use it

For this demo, we will use a simple Hello world application and scale it with a rule via a console application based on time.

Step 1: Put the app in Azure

To get started, an Azure application is needed. Don’t forget to assign a certificate since the console application will need it.

You can then publish the application on the cloud.

Step 2: Adding the scaling application

Now, create a console application and name it AutoScalingConsole. Add the WASABi package by executing: Install-Package EnterpriseLibrary.WindowsAzure.Autoscaling.
It should run without error and your Solution should look like this.

Step 3: Add and configure the rules

Add a new file called Rules.xml and set the property Copy to Output Directory: Copy always. Copy-paste this xml into the rules file.

<?xml version="1.0" encoding="utf-8" ?>
<rules xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/rules">
  <constraintRules>
    <rule name="default" enabled="true" rank="1" description="The default constraint rule">
      <actions>
        <range min="1" max="1" target="AutoscalingApplicationRole"/>
      </actions>
    </rule>
    <rule name="peaktime" enabled="true" rank="10" description="Increase instance count at peak times">
      <timetable startTime="20:00:00"  duration="00:20:00" />
      <actions>
        <range min="2" max="4" target="AutoscalingApplicationRole"/>
      </actions>
    </rule>
  </constraintRules>
</rules>

It contains two rules: A default one that is always active, defining minimum and maximum instance counts of 1, and a second one used for scaling. The variable “Is rank” with a value of one means that it can be overridden by other constraint rules with a higher rank. Naturally, if that rule is applied, there will be only a single instance of the role.

The second rule is named peaktime. This rule has the same target, a higher rank, a minimum value of two, a maximum value of four. Also a timetable makes the rule active for 20 minutes, starting at 10 minutes from the current time.

Step 4: Define the service model

You will now add a new xml file called services.xml and set the property Copy to Output Directory: Copy always. Copy-paste this xml into the services file.

<?xml version="1.0" encoding="utf-8" ?>
<serviceModel xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/serviceModel">

  <subscriptions>
    <subscription name="[yoursubscriptionname]"
                  certificateThumbprint="[yourmanagementcertificatethumbprint]"
                  subscriptionId="[yoursubscriptionid]"
                  certificateStoreLocation="CurrentUser" certificateStoreName="My">
      <services>
        <service dnsPrefix="[yourhostedservicednsprefix]" slot="Staging">
          <roles>
            <role alias="AutoscalingApplicationRole" roleName="AutoscalingApplicationRole" wadStorageAccountName="elazurehol"/>
          </roles>
        </service>
      </services>
      <storageAccounts>
        <storageAccount alias="elazurehol"
                        connectionString="DefaultEndpointsProtocol=https;AccountName=[yourstorageaccountname];AccountKey=[yourstorageaccountkey]">
        </storageAccount>
      </storageAccounts>
    </subscription>
  </subscriptions>
</serviceModel>

In this file, make the following changes:

Replace [yoursubscriptionname] with the name of your Windows Azure subscription and [yoursubscriptionid] with your Windows Azure subscription ID.
Replace [yourmanagementcertificatethumbprint] with your Windows Azure management certificate thumbprint.
Replace [yourhostedservicednsprefix] with the URL prefix of your Windows Azure hosted service.
Replace [yourstorageaccountname] with your Windows Azure storage account name and [yourstorageaccountkey] with your Windows Azure storage account primary access key.

Step 5: Configure the Auto-scaling Application Block

Right-click on the App.config file in Solution Explorer, add one if needed, then click Edit Configuration File. In the Blocks menu, click Add Autoscaling Settings. Now set the rules.xml and services.xml as sources for Rules Store and Service Information Store. Via the File menu, Save then Exit.

To be able to track evolution of the the testing, let’s add some logging. In Visual Studio, double-click on the App.config file to open it in the editor. Then add this system.diagnostics at the end of the file:

<system.diagnostics>
   <sources>
      <source name="Autoscaling General"  switchName="SourceSwitch" switchType="System.Diagnostics.SourceSwitch" />
      <source name="Autoscaling Updates"  switchName="SourceSwitch" switchType="System.Diagnostics.SourceSwitch" />
   </sources>
   <switches>
      <add name="SourceSwitch" value="Verbose, Information, Warning, Error, Critical" />
   </switches>
</system.diagnostics>

Step 6: Try it

You can now run the console application and observe how the auto-scaling rules work with the Azure application. Check the Output window in Visual Studio that logs which rules are being matched.

Conclusion

Using WASABi makes your application elastic but doesn’t make your application scalable, you must therefore design for scalability. If you have any comments, suggestions or experiences to share, feel free to let me know by adding a comment, by e-mail or by the contact page.
 

Where to find More info?

• Microsoft Download Center: http://www.microsoft.com/en-us/download/details.aspx?id=28189
• How to Use the auto-scaling Application Block: https://www.windowsazure.com/en-us/develop/net/how-to-guides/autoscaling
• CodePlex: http://entlib.codeplex.com/
• Nuget: http://nuget.org/packages/EnterpriseLibrary.WindowsAzure.Autoscaling
• How to Create a Storage Account for a Windows Azure Subscription: http://msdn.microsoft.com/en-us/library/windowsazure/gg433066.aspx
• Autoscaling Application Block Logging: http://msdn.microsoft.com/en-us/library/hh680883(v=pandp.50).aspx
• Defining Constraint Rules:http://msdn.microsoft.com/en-us/library/hh680897(v=pandp.50).aspx

One of the questions that clients often ask is: What tools could be used for Windows Azure development. Everyone knows that you need a web browser and a code editor (ex: Visual Studio), but what else? So, I decided to do a serie of posts to present them. I’m therefore beginning with the indispensable SQL Azure Migration Wizard, since it’s used more often at the beginning of a project.

This serie is not meant to be an exhaustive list of all tools. Some other excellent tools are surely available. If you think I have forgotten one or want me to talk about one, let me know. I will be more than happy to adding it to the list.

What is SQL Azure Migration Wizard?

Most of the Azure projects that are starting these days are migration project or at least contain some data migration. The SQL Azure Migration Wizard, like its name implies, is a tool that will take you be the hand and help you migrate your database. In a few clicks the wizard will bring the database schemas and the data to the cloud. Because Azure SQL database is very similar, but not totally compatible to Windows SQL Server 2008, the Wizard will also provide you a list of things that you should take care of like a missing clustered key.

How to use it

First, I will assume that you already have an active Azure account, if you don’t, please follow the instructions on this web page. Then of course you will need the SQLAzureMW tool that you can download on the CodePlex page project. Let’s do a simple migration from a local database to SQL Azure.

	To get started, just double-click on the executable. Some options will be proposed, but in our case we need to select: Analyse / Migrate SQL Database (second choice from the top)
	Then pick your source. One nice thing with keeping the Master database selected, the list of all available databases on the specified server will be populated. Once you are done, click the Connect button.
	The database used in this demo is a subset of Northwind database with three tables: Products, Suppliers, and Categories. Once you have selected what you want to migrate, click the Next button.
	Here three outputs that are generated:                   A result Summary: This explains all the steps made to extract the schemas and data of the database. The analysis will also highlight some incompatibilities and suggest some modifications. In this case, since I have not added any primary keys or indexes, it warned me that a clustered index will be added. SQL Script: A nicely written database creation script that you can save for maintenance. Data file: Since one of the best way to import massive data in SQL Azure database is by using BCP, some data files have been prepared to be imported. Click the Next button to continue.
	For the next step you will need a database server in Azure. So connect to the Azure portal, and from the left panel select Database. You can than create a new server by clicking the Create button. You could also use a server that you already have. Put the server name in your clipboard you will needed.
	Back to our SQLAzureMW, paste the full name of the server in the Server name dropdown. It should be something like xxx.database.windows.net Then enter the credentials of your SQL Server. Note that the username should be follow by “@” and the first part of the server name. Click Connect when you’re done.
	Since no database was created, the list is empty. Let’s create a new database by clicking the Create Database button. A popup window will ask for the name, location and format of the database to be created. Click Create Database, and after few seconds you should be able to click the Next button to continue.
	To see that the new database is really created go on the AzurePortal: you will see it in the database section.
	After a quick confirmation, the migration will be done. A nice summary is displayed so you can see if something went wrong. The migration has been performed successfully, click Exit.
	You can now connect to the new database by SQL Server Management Studio or by the Azure Management Portal. You can see that the three tables are there and the data has also been migrated.
	Like it was supposed to, the index was successfully created. This is really magic!

Conclusion

To conclude, let’s say that this tool is really fantastic and that you should definitely try it if you are planning to do a migration. It will save you a lot of time. Please note that the SQLAzureMW does not manage logins and users. I therefore strongly suggest using the Azure User Management Console – AUMC, available on CodePlex. If you have any comments, suggestions or experiences to share, feel free to let me know by adding a comment, by e-mail or by the contact page.

References

• Windows Azure : https://windows.azure.com
• CodePlex SQLAzureWM: http://sqlazuremw.codeplex.com
• Management Portal: https://manage.windowsazure.com
• Tools and Utilities Support:http://msdn.microsoft.com/en-us/library/windowsazure/ee621784.aspx
• Azure User Management Console – AUMC: http://aumc.codeplex.com

As we were doing many migrations to Azure SQL we build an internal tool to simplify the login and user management. Since only T-SQL commands are available for Azure SQL in SQL Server management Studio, a friendly interface was missing. The AUMC is simply converting your action into T-SQL commands and execute them in the Azure SQL Database.

You can find the Azure User Management Console on CodePlex at:http://aumc.codeplex.com

Connect

When you start the application you will be asked to connect to the Azure SQL database.

Once you log-in you will have the choice to manage Login or User.

Logins

If you click on the button “Logins”, you will see all the existing Logins of the current Azure SQL server. To delete a login just click on this login and then click the button Drop Login on the bottom right of the screen. You will be asked to confirm your request then the list will be refreshing. 

If you want to add a new login then click on “Create New Login” button on the bottom left of the screen. You will need to specify the login name and password in a popup window. 

Users

Users are attached to a database, so after click on the Users button (from the main windows) you will be asked to select a database. In this case one the DbTest database is available since we can’t use the master database. 

Once the database is selected the entire list of available users will be displayed. To delete one user, select this user then click on the “Drop User” button on the bottom right of the windows. 

To edit the role of a specific user only double-click on this one. To add a new User click on the button “Create User” on the bottom left of the windows. You will then be asked to specify the login and the role(s) you want to be attached to this user. 

There's a consensus[1] in the BizTalk community that it’s a good practice to implement pipeline components with a streaming approach. This minimizes the use of memory, by avoiding having a copy of the whole message loaded by each component in a pipeline (which can be a problem for large messages). However, there's not much guidance on how to actually implement streaming. Yossi Dahan has written a good article on this subject (Developing a Streaming Pipeline Component for BizTalk Server) that explains how it can be done, but his sample code is somewhat complex because it manages the output buffer and does the transformation within the same method. 

This article explains the implementation of two classes that encapsulate the complexity of writing to the output buffer and then reading from it, so that streaming components can be simpler to implement.

Applicable contexts

• Transformation of non-XML data ("flat files"). The pipeline could either transform the non-XML to XML, or do some manipulations on the data while keeping it in a non-XML format.
  • Other tools already exist for a similar approach with XML data (see references at the end)
• Encoding/decoding or encryption/decryption of any data stream

Solution

The relationship between the two classes described in this article is shown in the diagram below. To use these two classes, a developer would write a "ConcreteStreamTransformer" that implements abstract methods on the StreamTransformer class.

• StreamTransformer: An abstract base class that decorates a Stream with transformation code. Its implementation of the Stream "Read" method calls our transformation code. However, it doesn't do the whole transformation in one shot. Instead, it will check how many bytes were requested by the caller, and will only call the transformation code enough times to fill the buffer with that number of bytes. It will then return the buffer to this caller, which will be able to do its own incremental processing on this buffer.
  In the diagram above, the transformation code is called twice to fill the buffer requested by the caller. We got lucky in this case, because we filled exactly the right number of bytes, we don't have any leftover bytes that didn't fit in the buffer. To avoid the complexity of managing these leftover bytes, I introduced a second class as an intermediary between the transformation and the output buffer, the PipeStream (see below).

• PipeStream: An intermediary buffer to which the concrete StreamTransformer will write, and which will be consumed as BizTalk (or the next pipeline component in the pipeline) reads from it. Here, "consumed" means that once data is read from this stream, it is removed from memory. This allows working with the same convenience as a MemoryStream to manage a variable size buffer, but without keeping everything in memory (and without swapping to disk as a VirtualStream would do). This pipe allows the transformation code to do a transformation that generates more bytes than was requested by the caller. The caller will only get the number of bytes it requested, and it will get the remaining bytes in later calls.
  
  In this diagram, we have leftover bytes after each transformation. For the transformation code's point of view, it simply writes all the bytes (including the leftovers) to the PipeStream. Then, the PipeStream will return only the correct number of bytes, and keep the remaining ones for the next call to the transformation code. So we have the input string "testing" transformed and returned in 3 steps, as "TES" + "TIN" + "G". This way, BizTalk can start writing "TES" to the send port while waiting for the reception of "tin" and its transformation to "TIN". This reduces latency and avoids keeping the whole "TESTING" string in memory. Obviously, this is a simplified example, there's not much gain in splitting such a small string. In a real solution, we would have larger buffers, for example to process data in blocks of 1024 characters.

To write streaming transformations with these classes, all we have to do is write a "stream decorator" (the ConcreteStreamTransformer) that inherits from the StreamTransformer class and implements its abstract method.

Examples

A simple example would be a transformer that inserts a delimiter at each 1000 characters, so that the file is split into evenly sized "blocks".

public class BlockSplittingStreamTransformer : StreamTransformer
{

	private byte[] blockDelimiter;

	private int blockSize;

	private bool wroteFirstBlock = false;

	public BlockSplittingStreamTransformer(Stream inputStream,

	int blockSize, byte[] blockDelimiter)

	: base(inputStream)

	{

		this.blockSize = blockSize;

		this.blockDelimiter = blockDelimiter;

	}

	protected override bool PartialReadAndTransformInput()

	{

		int bytesToRead = blockSize;

		byte[] inputBuffer = new byte[bytesToRead];

		int nBytesRead = InputStream.Read(inputBuffer, 0, bytesToRead);

		if (nBytesRead < 0)

		{

			if (wroteFirstBlock)

				OutputStream.Write(blockDelimiter, 0, blockDelimiter.Length);

			OutputStream.Write(inputBuffer, 0, nBytesRead);

			wroteFirstBlock = true;

			return true;

		}

		else

		{

			return false; // no more data to transform in the input stream

		}

	}

}

To use this stream in a pipeline component:

	private const int BLOCK_SIZE = 5;

	private readonly byte[] BLOCK_DELIMITER = new byte[] { (byte)'r', (byte)'n' };

	public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)

	{

		Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();

		StreamTransformer transformer =

		new BlockSplittingStreamTransformer(

			originalStream, BLOCK_SIZE, BLOCK_DELIMITER);

		pInMsg.BodyPart.Data = transformer;

		pContext.ResourceTracker.AddResource(transformer);

		return pInMsg;

	}

Here, the transformation is not done by the pipeline component itself. Instead, the pipeline component returns our StreamTransformer and allow BizTalk (or the next pipeline component in the sequence) to incrementally and transparently call the transformation code.

For a more interesting example, we could have a flat file that's too complex to be parsed by BizTalk's flat file features. For example, a weird format that uses:

• a different child delimiter between each field
• allows different delimiters between some fields
• allows different "tag identifiers" for the same type of data

HDR153/20120127:20120130
ITEM9423%10~153.17
LINE9425|20~23.20

This format combines various problematic cases from real-world formats, such as HL7, into one example. Interpreting such a file would be complex with a flat file schema, but it would be simple with regular expressions. We could make a StreamTransformer that applies the regular expressions on the input stream.

	public class RegexStreamTransformer : StreamTransformer

	{

	private StreamReader inputReader;

	private XmlWriter outputXmlWriter;

	public RegexStreamTransformer(Stream inputStream) : base(inputStream)

	{

		inputReader = new StreamReader(InputStream, true);

		this.outputXmlWriter = XmlTextWriter.Create(base.OutputStream);

	}

	// Tag Identifier: HDR

	// First delimiter is "/", second delimiter is "$";

	private Regex headerPattern = new Regex(

		@"^HDR" +

		@"(?<orderid>;[0-9]+)" +

		@"/"; +

		@"(?<orderdate>[0-9]+)" +

		@":" +

		@"(?<deliverydate>[0-9]+)$";

	);

	private Regex orderItemPattern = new Regex(

		// Tag Identifier: Either ITEM or LINE

		@"^(ITEM|LINE)" +

		@"(?<itemid>[0-9]+)" +

		// Delimiter: either % or |

		@"[%|]" +

		@"(?<quantity>[0-9]+)" +

		// Delimiter: ~

		@"~" +

		@"(?<unitprice>[0-9]+.[0-9]+)$"

	);

	private bool wroteXmlRoot = false;

	protected override bool PartialReadAndTransformInput()

	{

		if (!wroteXmlRoot)

		{

			// first write the XML root element before doing any transformation

			outputXmlWriter.WriteStartElement("Order", "http://namespace");

			wroteXmlRoot = true;

			return true; // will continue with the transformation next time the method is called

		}

		string line = inputReader.ReadLine();

		if (line != null)

		{

			var hdrMatch = headerPattern.Match(line);

			if (hdrMatch.Success)

			{

				outputXmlWriter.WriteStartElement("Header", "http://namespace");

				outputXmlWriter.WriteAttributeString("orderId", hdrMatch.Groups["orderId"].Value);

				outputXmlWriter.WriteAttributeString("orderDate", hdrMatch.Groups["orderDate"].Value);

				outputXmlWriter.WriteAttributeString("deliveryDate", hdrMatch.Groups["deliveryDate"].Value);

				outputXmlWriter.WriteEndElement();

			}

			else

			{

				var orderItemMatch = orderItemPattern.Match(line);

				if (orderItemMatch.Success)

				{

					outputXmlWriter.WriteStartElement("Item", "http://namespace");

					outputXmlWriter.WriteAttributeString("itemId", orderItemMatch.Groups["itemId"].Value);

					outputXmlWriter.WriteAttributeString("quantity", itemMatch.Groups["quantity"].Value);

					outputXmlWriter.WriteAttributeString("unitPrice", itemMatch.Groups["unitPrice"].Value);

					outputXmlWriter.WriteEndElement();

				}

			}

			return true; // we may still have more data to transform, will continue on next call

		}

		else

		{

			outputXmlWriter.WriteEndElement(); // close the root element

			// ensure any remaining bytes in the buffer are visible to the base class (StreamTransformer)

			outputXmlWriter.Flush();

			return false; // no more data to transform

		}

	}

}

The code to use this stream in a pipeline component is very similar to the previous example (it only instantiates a different StreamTransformer implementation):

	public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)

	{

		Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();

		StreamTransformer transformer = new RegexStreamTransformer(originalStream);

		pInMsg.BodyPart.Data = transformer;

		pContext.ResourceTracker.AddResource(transformer);

		return pInMsg;

	}

Conclusion

In this article, we've seen how to use two classes that encapsulate the complexity of implementing streaming pipeline components. This technique is especially useful in contexts where we have non-XML data to transform. For XML data, other classes already exist to help achieve a similar approach (see references below).

These classes can be used in a variety of contexts. The context described in this article was BizTalk Send or Receive pipelines. But in fact, the classes don't have a dependency on BizTalk, only on plain .Net streams. This means they could also be used anywhere where .Net streams can be used.

Implementing pipeline components with this approach has several benefits:

• By doing small incremental transformations, the pipeline component uses less memory.

• It also avoids large spikes of CPU usage when a large message is processed all at once, because it distributes CPU usage more evenly in time, allowing better parallelization.

• It also reduces latency by allowing partial data to be processed as it becomes available, instead of waiting for the whole data.

• Finally, the fact that it's transparent to callers means that we can use this pattern in any context where .Net Streams can be used (not only in BizTalk pipeline components)

Note that another frequently used solution to minimize the use of memory in BizTalk pipeline components is  to use a VirtualStream, which swaps its buffer to disk when it gets too large. Like the approach described in this article, it allows reducing the use of memory. However, it lacks the other advantages listed above. In fact, it also increases (instead of decreasing) latency because it has to work with disk I/O, which is slower than memory.

Applies to

All BizTalk versions from 2004 to 2010.

Downloads

The source code for the StreamTransformer and PipeStream classes described in this article, and the associated examples, are available here.
All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.

Other useful references

• Better options for implementing a streaming approach with XML input and outputs (instead of non-XML data as shown in this article):
  • Developing Streaming Pipeline Components – Part 4 – Introducing the XmlTranslatorStream
  • Great functionality for every BizTalk developer (XPathMutatorStream)

• Best practices for optimizing performance of BizTalk Server pipelines

• Developing Streaming Pipeline Components – Parts 1 to 5
• Developing a Streaming Pipeline Component for BizTalk Server The paper that first described the approach used in this article. This article provides tools to simplify the implementation of this approach.