Automated build and deployment for BizTalk using VSTS

Last year I wrote about my experiments about using Powershell scripts to deploy BizTalk solutions and how to manage deploymens (manually) in a controlled way. Since then I have every now and then been thinking about ways to improve the process even more. I have used Visual Studio Team Services (VSTS) in Azure projects for version control, builds and deployment management. What I haven’t used before is the release management functionalities of VSTS.

In a quest for continually educating myself, I figured that what could be better way to get familiar with the release management bits of the service than trying to push the BizTalk deployment story a bit further. Here are the steps that it took to get all the bits set up for the ultimate BizTalk deployment management!

Version control

VSTS provides possibility to host your code within the service. You can use either TFS or GIT repositories and have as many private repositories as you like. But that is just an option and you can just as well host your code in some of the other supported locations (Github, external Git repo, Subversion). I happened to have one very simple BizTalk solution in Github repository, so I decided to use that. You can learn how to set up the connection to your version control from the VSTS documentation.

Setting up automated build

When talking about Continuous Integration (CI), one key element is to have your code automatically built whenever you check code to version control, at least to the branch that then triggers the deployment. This way you can be sure that whatever has been committed to the code base builds and can be deployed. No more missing files in the repo! It’s also considered not cool to break the build, so maybe devs also put a little more effort to the quality of the commits before pushing the button. In this post I will not dive into integration automated tests to the build process, but that is certainly something to concider when setting up the process.

VSTS provides hosted build agents that can be used to actually do the build. Basically it means that you can buy build server time by minute and someone else takes care of the configuration and managing of the server for you. This doesn’t come without a concequece. There’s not much you can do to change the set-up. I faced this when I first tried to build my BizTalk solution using the provided build agent:

error MSB4019: The imported project “C:\Program Files (x86)\MSBuild\Microsoft\BizTalk\BizTalkC.targets” was not found.

It seems that VSTS build agent is not capable of building BizTalk solutions as required build target components are missing. Not much you can do about that in a hosted environment.

Setting up custom build agent

As using hosted build agent is not an option with BizTalk solutions, you need to run your own. Luckily the process of setting up a custom build agent is rather straight forward. All you need is access to a server with the right build targets installed. I used one my BizTalk virtual machines in Azure.

In the Settings -> Agent Queues section of the VSTS portal, you can find a link for downloading agent binary to your server. It comes with simple setup instructions too:

You can find a bit more detailed instructions in the documentation which also helps with the PAT token that is needed during the configuration phase. Seems really straight forward… until you face another problem. I created PAT token so that my agent could connect to my VSTS account as instructed. At the time of figuring this all out, the instructions also contained very detailed scopes that were needed for the build agent to be functional (at the time of writing this, the documentation only says Select the scopes that this token will authorize for your specific tasks). It took a while to find out that there was an error in the documentation and that the documented scopes were not sufficient enough for the agent to work. Giving my PAT token correct scopes resolved my configuration issue and finally my agent was up and running. You can check the status of your agent and that the connection works from the VSTS portal:

One thing to note when configuring the agent. If you decide to run it as a service and intend to use the agent also for managing BizTalk resources during deployments, you need to make sure that you run the agent with a user account that is member of the appropriate BizTalk admin user groups.

Configuring build

Once you have build agent up and running, it is time to configure build definition. From the Build&Release section of the VSTS portal, you can find link for creating a new build definition. Choose Visual Studio template and hit next and you will see screen like this:

Choose the location of your code repository, check Continuous integration checkbox and select to use the your own build agent queue. That’t pretty much it. You can check that connection between your code, VSTS and your agent is working by checking in some code or by manually triggering the build config.

Setting up release management

So now we have set up automated build for the code. It’s time to start thinking how to get that code running in you BizTalk environment. Hopefully you don’t test your code in production and therefore you have multiple environments to manage. You need to configure release definitions in similar manner to the build definitions. So from the release section of the VSTS portal, create a new release plan, add appropriate environments and then start figuring out the tasks you want to run in each environment during the deployment. You can trigger the deployment in each environment differently: either automatically when code is checked in, using schedule, when deployment is successfull in some other environment or when you manually create a new release.

In my test setup I have automatic deployments to test environment, but other environments require manual approval from certain VSTS user. There are tons of settings you can change when it comes to configuring the releases: variables that you give as input, how to handle artifacts, naming conventions, etc. It’s not possible to cover all that in one blog post so I’ll leave some parts as home work for you. The main thing to understand is how you can set up multiple environments, configure when the deployments are triggered and then figuring out the steps that you want to run in each environment.

As you can see, I have set up three environments and my deployment process is really simple. The first task is to copy the build artefacts (basically your BizTalk assemblies) to a certain location that your deployment scripts use. Then I just run my powershell deployment scrips that I have described in my earlier blog post. I use the same agent to run the deployment tasks that I use for building the code, so it already has access to my BizTalk environment. I did need to modify the scripts a bit as now I don’t have to worry about manually handling different versions of the solution or verify the environment from user. Therefore my scripts became quite a bit simpler. What I haven’t tried out is how to run the same scripts on remote servers. This use case obviously becomes relevant if you don’t want to run an agent on all you BizTalk servers (I imagine this being the case) or when you have multiple runtime nodes in your environment. There is task for remote powershell execution also, so I suppose that should not be a major headache to set up eiher.

The really cool part of running the powershell scripts using VSTS is that you get the same output to the release logs that you would normally get to your powershell console. Therefore it is really easy to troubleshoot possible deployment problems without ever logging into the actual BizTalk server. This obviously requires that your scripts give good feedback when they execute.

Is is also really easy to manage and control different versions accross environments. You can see in a clear traffic light type of view the current status of each environment.

Let’s commit!

Now it is time to commit some code to your repository and watch the magic happen! With this approach you can achieve:

• Always up-to-date codebase in version control as all deployments depend on code in your repository
• Automated build of your BizTalk code, no more broken builds
• Automated deployment of BizTalk solutions
• No need to manually handle artifacts
• Versioning follows version control, no need for additional versioning scheme
• No need for manual actions when deploying
• Managing releases in a easy web UI
• Control who can approve deployments
• Only release certain versions to staging and prod, but always have the latest version in test/ci environments
• Control environment specific parameters outside codebase as part of the release configuration
• Easy way to audit the releases (what has been deployed, when it has been deployed, who has authorized the deployment)
• One place to get overview what is going on with you devops process

VSTS story gets even better if you complement the build and deployment part with the project management parts of the service. Then you can follow the whole process from creating the tickets for the developer to the production deployment of the change, all in one place. Pretty sweet I think.

Reality obviously is not as perfect as it may sound. When it comes to complex BizTalk solutions, handling the deployment with a completely automated script is challenging. If you end up having lots of references between your applications and need to update component that other components depend on, most likely you end up doing something manually. Suspended instances also give headache when it comes to executing automated deployment scripts. I have solved that issue by forcing termination of all suspended instances that are related to the components that are being deployed. That might not be a thing you want to do automatically, especially in production environment, without first evaluating the possible downsides of losing information.

It must be said out loud that this approach is sort of a hack. On the other hand, the whole deployment story with BizTalk is rather sad. If you try to achieve some level of automation, you pretty much always end up with some kind of custom solution. Obviously Microsoft hasn’t given too much thought when it comes to combining BizTalk with VSTS. It just isn’t really supported so you are on your own. As BizTalk is still Microsoft’s main integration product, it is quite embarrasing that the platform really hasn’t evolved in years.

The fact still remains that BizTalk is not going away anytime soon. There are lots of customers using it heavily and there is no real competition in the Microsoft world yet (Logic Apps has to come quite a long way…). To me trying to keep up with the rest of the world as much as possible is a way to stay up-to-date and also get hope that things will get better also in the Microsoft integration stack. I think that if you are willing to put in some effort to get everything set up, VSTS could be really good tool for automating BizTalk deployment process as far as it is possible.