New Year tech clean up

Well as another weird year begins I’ve managed to break some time off to do a bit of housekeeping. I’ve remove some old forks, added licenses, tags etc. I’ve also been renaming the default branch to main. git branch -m master main git fetch origin git branch -u origin/main main

Invoke-SqlCmd Failing

A really annoying problem I hit recently with a some Azure Runbooks was the following message occuring: Invoke-Sqlcmd : The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services. After doing some digging, I found out that Invoke-SqlCmd isn’t thread safe, so other runbooks running on that Azure Automation account, could interfere with each other.

HP Laptop review

It’s the time again that I go in search of a new laptop to replace my current one. Following on from my previous success with the Toshiba laptop - buying cheap and upgrading the RAM and HDD. I went in search of a cheap laptop. My requirements remained all but the same except for a Full HD screen. I once again settled on a Intel CPU. I did spent a large amount of time trying to find a AMD Ryzen based laptop and although they seem good I just couldn’t seem to find the right one, most seemed to have a optical drive which nowadays seems like a waste of money and just adds pointless weight to the laptop.

Version control in PowerBI

Power BI is a awesome self-service reporting tool that has a low-level of entry for new comers, that allows end-users (ie Non-IT folks) to deliver elegant insight into data via interactive reports. It’s biggest flaw however is it doesn’t have any kinda of code version control support. Like it’s impossible. Various people/companies have developed hacky solutions however this goes against Power BI core design of being a simple, self service tool.

Auto refreshing PowerBI

One of the problems we hit today was getting a Power BI to automatically refresh. The underlying data is updated every 5 mins, however the actual Power BI only refreshes when the user interacts with the report despite using DirectQuery. To overcome this, we’ve built a simple HTML page that used a bit of JavaScript magic to force a refresh and the new Secure Embed feature. First, we need to get a code for the report, open the report then click File > Embed (You can select ‘Publish to web’ - this means you don’t need to login to view the report, however anyone in the world could access the report)

Microsoft christmas

Power BI Secure embed codes - allows easy sharing of Power BI reports. The public embedded codes - publish to web - has been out for a while that allows you share the Power BI reports that the anyone on the Internet (unauthenticated members of the public) should be able to see. The key difference is the secure part of the name. Users will need to authenticate, this does of course mean they will require either a Power BI Pro license or the report will be using the Power BI Premium capacity.


Monzo is a digital only UK bank. It is fully authorised in the UK by the Prudential Regulation Authority (PRA) and regulated by the Financial Conduct Authority and the PRA. Simply put, if Monzo goes belly up, your money is protected. If you are over 18 years old and have a Android or iPhone (as it’s app only), it’s worth signing up, it takes minutes. Especially whilst they are giving away £10 for new customers!

Power BI folders

One of the annoyingly missing features in Power BI is the ability to organize fields into folders to allow users to easily find them. Luckily the amazingly animated Will Thompson posted an update on Power BI Ideas, which has received over 400 votes, that it has been developed and is available as a preview feature in the Nov'18 Power BI Desktop release. The display folders are part of the new modeling view.

Remember a business is a business

At the end of the day, a business is a business. It’s sole job is to make money. There isn’t anything wrong with this, I like getting paid each month, it enables me to be to, you know, buy food and keep a roof over my head. Once you accept you need money to live in today’s society you can kinda see how a business needs to make money and if you can reduce costs it, brillant.

vardecimal support is enabled for the database but disabled for the project

I had a bit of a weird one where one of the SSDT projects failed to deploy. The package would successfully deploy locally, but failed from the deployment server. An extract from the log: vardecimal support is enabled for the database but disabled for the project. See SQL Server Books Online for more information about how to turn off vardecimal support. The short version of the story is the deployment server was trying to msdb.


Towards the end of last year I was writing .net core AWS Lambda. I was super impressed with how .net core (and even PowerShell) works on Linux. I would have to say C# is by far my favorite programming language, the only exception might be T-SQL and the idea of being about to write code and get it to run on Linux is amazing. Mainly because Linux servers are, to be blunt, cheaper.

Renaming a SQL Server

So you’ve renamed your SQL Server, but something isn’t right. Returns the old name. The fix is simple. You just need to restart the SQL instance and @@servername will return the correct name. More info on the Microsoft Docs. On a similar note, Octopus Deploy has similar annoyance, the server name doesn’t update for the server node. Again simple fix for this simple annoyance: The other thing that can get missed is the SQL Jobs.

SSRS Error - There is an error in XML document (2, 2)

I got this error when opening an SSRS project in Visual Studio. The solution. Upgrade SSDT. Turns out my colleague had a higher version of SSRS component - if you go, within Visual Studio, Help > About it will give you the version number of Visual Studio then the individual component versions.


Here are a few of my predictions based on my experiences over the last few months. First. Azure Data Catalog is going to be massive in the near future (18 months). The (hopefully) future integration with the new SQL Data Discovery and Classification is going to help massively with GDPR. Secondly, Chris Webb is going to become very popular. With Azure Analysis Service, the big question is optimize or scale. The killer feature for any monitoring tool aimed at the BI market is going to have to answer that question.

Bye bye Startcom

In the old days, you would have to throw money at a trusted Certificate Authority (CA) who would sell you a SSL certificate. The alternative was to self-sign or use someone like cacert, the problem is out-of-the-box, no browser\OS supports this so you get a security warning, which isn’t something you want when your trying to be secure. StartCom, was a trusted CA who issued free SSL certificates. Microsoft Internet Explorer, Google Chrome, Mozilla Firefox all trusted StartCom which means you could get a free SSL certificate without the warnings.

Troublesome CSV

Another day, another source of bad data. This time, it was a CSV. As you can see, the format changes mid file. This makes importing tricky. The way I tend to handle importing into SQL is to create staging tables with all the columns defined as varchar, then, once its imported into SQL, then convert it. The problem occurs when you try to determine the number of columns required. Traditionally, we let the system auto determine how many columns we have, and if we have headers, define the names, unfortunately because the format changes we can’t do this otherwise it will ignore the additional columns - assuming the first row doesn’t contain all the columns.


Serverless doesn’t somehow make your code run without physical servers. You still need servers, it’s just your treating your servers like you would code. However, it goes beyond just infrastructure as code (IaaS), it’s utilizing the cloud. Let’s get something clear, the cloud is nothing more than someone else’s server and for this post, I’m defining the cloud as any service provider who allows you run compute by the millisecond. The reason the cloud is such a major thing is that it allows you to treat commodity hardware as a commodity.

Turbo SQL 2017 backups

Another day, another Twitter awesome moment, Parikshit Savjaniwas explaining how backups are now faster in SQL Server 2017. How we made backups faster with SQL Server 2017 — Parikshit Savjani (@talktosavjani) November 23, 2017 For more info on how, read his post. TD;LR - The wait time before data is actually copied is reduced because the amount of buffers to scan is massively reduced. “…requires only 250 buffers to scan as opposed to 500 Million buffers with former algorithm…”

AWS S3 errors

One of the annoying things about AWS S3 is the error messages. Below is an error, see if you can figure out what it’s saying. I’ll give you a hint. We’ve connected and authenticated. Here is another hint, we can read from that bucket. Give up? It’s permissions. Incorrect AWS keys returns a helpful message, so does invalid bucket, as does no access to read bucket

Nice idea, but pretty holely

So the NHS Hack Dayretweet this We're a UK non-profit dedicated to open source in Healthcare. Can anyone help us with some VPS instances? @rackspaceUK @digitalocean @linode — NHoS (@_nhos_) April 30, 2017 Which made me think why is Ubuntu creating a NHS specific distro (OS) and why are they begging for some VPS instances? Turns out, it has nothing to do with either Ubuntu or the NHS. Its a group of people building on-top of Ubuntu, aimed at the NHS.


Idempotent – something that is repeatable, that remains unchanged in value when repeatedly executed either independently or on by itself. SSRS – SQL Server Reporting Services part of Microsoft SQL Server. Used to deliver static reports that can be viewed via a web browser and exported to a number of formats including Excel, PDF and CSV. SSAS – SQL Server Analysis Services part of Microsoft SQL Server. SSIS – SQL Server Integration Services part of Microsoft SQL Server.

Cyber Security Insurance

I’ve been thinking lately about how the web industry needs to change in order to improve. Cyber attacks appears to be on the rise and alot of the time they appear to be caused by someone doing something silly. Perhaps this is because everyone is a web developer, it certainly isn’t down to lack of available resourceson the matter. Thinking positively about cyber security, the card companies banded together to form PCI-DSS – Payment CardIndustry Data Security Standard, with the goal to help businesses process cardpayments securely and reduce cardfraud.

SSRS support for both US Letter and UK A4

One of those annoying things about the US is they use different paper sizes to us Brits. US Letter is 215.9 by 279.4 mm (8.5 by 11.0 inches) The UK equivalent, A4, is 210 by 297 mm (8.26 by 11.69 inches) One of our customers is based in the UK but has remote sales offices in the US, so their SSRS report are set to A4 and when they printed them in the US offices, the footer is cut off.

Windows Insider – new features

Today’s Windows Insider email included a few new cool features, here are my Top 3 Do more at once with the new Compact Overlay window. Keep watching a movie or video chat on one corner of your screen — even when switching apps to check email or browse the web. When an app window enters compact overlay mode, it’ll be shown above other windows so it won’t get blocked. Look for updates to the Movies & TV app and Skype Preview app to take advantage of this feature in the near future.

Rosetta@Home DNS Issue

If you ever need a lesson in the importance of picking a good domain registrar, then read about Rosetta@Homerecent problems. Dear Rosetta@Home participants, We — like many of you who have contacted us — have been extremely frustrated by the long project downtime. We ( had a domain name registration verification lapse, and our registrar ( and ICANN turned off DNS for We went through the steps to

Automating WordPress maintenance

WordPressis an amazing blogging platform. However it does require a fair amount of love. Despite Mythic Beastsmanaging a large portion of my stack (hardware, OS, Apache, PHP, MySQL) and WordPresshaving automatic background updatesI still find myself logging in and finding pending updates for WordPress. The solution was WP-CLI. With the shell add-on, I SSH onto my account, then What the above script does is download WP-CLI, grant it execute permission, then downloads my script and again, gives it execute permission.

MSSQL server failed on Ubuntu on Windows 10

I’ve been getting errors with MSSQL server on Ubuntu on Windows 10. The error, Failed to connect to bus: No such file or directory dpkg: error processing package mssql-server (–remove): subprocess installed post-removal script returned error exit status 1 Processing triggers for libc-bin (2.23-0ubuntu5) … E: Sub-process /usr/bin/dpkg returned an error code (1) I managed to fix it in the end by

Resident parking

Been trying to find out the following from my local council: How many resident parking permits have been issued How many resident parking bays are available What exactly has the money been spent on in terms of fines issued. To say they are being evasive is an understatement. So far, I’ve got, the response of: _This info is fluid as it is not only static residents who can apply for waivers within a given zone but also agencies who provide medical needs and other statutory services.

Cloudflare parser bug

So once again we have another major security leak. You can read about it, hereand below is the email CEO Matthew Prince wrote to customers: Dear Cloudflare Customer: Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare’s systems. If you haven’t yet, I encourage you to read that post on the bug: we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers’ sensitive information could still be available through third party caches, such as the Google search cache.

Caching woes

Caching always seems to cause problems, still, we can’t have it all. Today’s caching problem was to do with RedgateSQL Prompt, a really amazing plugin that helps you write better SQL code. The problem with it is the cache of database object metadata was out-of-date. I had updated a table so when I typed select * then press TAB to expand the * into a list of columns, I got the old names.

Date Format

I’ve had a problem recently when it can to formatting a datetime as just date where FormatDateTime didn’t work The fix was to change it to Format. Oddly, despite the parameter being a datetime, I found I still had to cast it as a date. FormatDateTime was introduced in Vs 2008. I’m not 100% sure why this didn’t work.


Another error I hit Warning 1 [rsRuntimeErrorInExpression] The Value expression for the textrun ‘Textbox28.Paragraphs[0].TextRuns[0]’ contains an error: Overload resolution failed because no Public ‘/’ can be called with these arguments: ‘Public Shared Operator /(d1 As Decimal, d2 As Decimal) As Decimal’: Argument matching parameter ‘d2’ cannot convert from ‘TextBoxImpl’ to ‘Decimal’. C:\Projects\Reports\1. Report.rdl Thankfully Google found Qiuyun answer. I was missing the .Value at the end. So (wrong)

SSRS copy and paste fail

In case you haven’t heard, I’ve started a new job and one of my first tasks was to speed up a SSRS report. One of the first issues I stumbled across was this: An error occurred during local report processing. The definition of the report ‘/1. Report’ is invalid. The Value expression for the textrun ‘Tx412.Paragraphs[0].TextRuns[0]’ contains an error: [BC30456] ‘RdlObjectModel’ is not a member of ‘ReportingServices’ Looking into it the issue it appears its expanded expressions when it got copied – it’s basically making the it fully qualified.

SSRS 2016 by default

For the past 4 months I’ve been using Visual Studio 2015 rather then Visual Studio 2013 and yesterday, I hit a bug with SSRS (shows how much I love SSRS). The bug appeared when it came to deployment The error was deploying SSRS report Exception calling “CreateCatalogItem” with “7” argument(s): “The definition of this report is not valid or supported by this version of Reporting Services. The report definition may have been created with a later version of Reporting

Swollen SSISDB caused by Continuous Deployment

I recently had incident logged where the drive that hosted SSISDB database filled up on the UAT environment. We had SSISDB set to only keep 2 copies of the projects, however the additional copies are only removed when the SQL job is run – SSIS Server Maintenance job. The SQL job is scheduled to run at 00:00 (midnight) every day, which is the default, which runs two steps SSIS Server Operation Records Maintenance

Authenticating IoT

I was re-entering my password into our NowTV boxin the bedroom when it occurred to me. Authentication sucks on the Internet of Things (IoT). The problem is you have a simple device with minimal extras. On the NowTV you have a basic remote that looks like this: Can you imagine entering a password with a length of over 20 that’s a mixture of numbers, special characters and both upper and lower case characters?

Dangers of distribution lists

Distribution list are amazing things, they allow us to send mail to our team without having to remember who exactly is in our team, this is especial true in this modern age where we are part of many different teams, sometimes without even realising. These lists can contain hundreds or even thousands of members and the latest NHS IT bug has left IT departments maybe wanting to double-check their distribution lists – hopefully before the security team comes a knocking.

Microsoft SQL Server on Linux

This is absolutely amazing, Microsoft, running SQL Server, in Linux Redhat (RHEL)!


One of the annoying things about working with multiple languages is they each have their differences – obviously, otherwise there would only be 1 language! Although there is good reason to have these differences they still have elements that are a pain. One of the most annoying things, other then “do you require a semicolon at the end of the statement?” is comments. Comments aren’t executed or complied but they help developers read code.

Where has SIMS Bulk Import gone?

Cut-down version, I’m not working with SIMS anymore and I can’t pass on my work without them getting a large bill from Capita for using the business objects. Years and years ago I was working on a SIMS help desk at a Local Authority (LA). I had a school log a call asking about importing email addresses. Like many schools this school had just purchased a messaging service that allows sending emails and texts, the problem was they needed them in SIMS .

Are hybrid shops missing the point?

With rise of the internet high street businesses are rushing to adapt so they don’t become a thing of the past. However in their rush are they being a storm trooper? Are they missing the point? First, lets look at two successful hybrid business. Hughes, a electrical store based in East Angliaand Argos, a British catalogue retailer operating in the United Kingdom. If we focus on Argos as its more well known business.


So oneof the really nice sponsorsat SQL Saturdayhad a vending machine that was rigged up to release a treat when you sent a tweet. Like so I'm at #sqlsatcambridge home of @redgate #redgatetreat #red668 — Matt Smith (@matt40k) September 10, 2016 and give you… In mine had a SQL Promptlicense. At first I was a bit disappointed, I hoping for SQL Testlicense or a full blown SQL toolbeltbut the more I’ve been using it, the more I’m loving it.

Wooo!!! Fixed in SQL Server 2016! It’s nice to see Microsoft fixing old bugs – the I noticed today they updated the MSDN article about Actions in Multidimensional Models. It now supports HTTPS! Woot!!   

Another year, another SQL Saturday

Unfortunately this year I’ve not been able to make any of the pre-cons, however the (free) community day looks epic – the hardest part is going to be selecting which session to attend, the final one looking the worst. Aghhh!!! I’m going to have to roll a dice or something! CDC – Change Data Capture in Detail Uwe RickenWhat’s new in the SQL Server 2016 for a BI Professional Prathy KamasaniExploring the Azure SQL Database

Care Quality Commission (CQC)

Today one of our older SSIS packages failed which loads data into our warehouse, it turns out one of the fields had been expanded as the business had started to use a different standard for one of the codes. Despite not being told about this, it causing it fail and generate extra work on an already busy Monday it was actually a really good thing. Firstly, once we identified the problem (and fixed it), our main business contact was aware of it, was able to explain why it was happening.

AD tools on Win10

A while ago I got a new laptop at work with Windows 10 Enterprise and for a while I just kept using my old Windows 7 Enterprise laptop for bits I hadn’t yet transferred over. The last thing on the list was AD tools. This should just be a question of installing Remote Server Administration Tools for Windows 10 (KB2693643). Unfortunately Remote Server Administration Tools for Windows 10 is available only in English (United States) for this release.

Why does Bing sucks

The only thing you search for on Bing is Google. Microsoft has for a long time been in the search engine business with numerous rebranding exercises – MSN Search (1998 – 2006), Windows Live Search (2006 – 2007) then Live Search (2007 – 2009) and finally Bing (2009 – present). Despite all the time and money Microsoft has poured into its search engine Google still has 3 times the number of of the search engine market share as Bing.

Build problem and duplicate releases

Yesterday I went to investigate a problem someone had reported only to discover a butt load of releases As you can see, I have more releases then commits. A build was triggered by a commit, but this doesn’t explain why. I tried switching it up so it only builds when you put [build] in the commit message – which resulted in having to commit the yaml file as the feature isn’t available via the web interface, it does make sense to have this under version control.

Toy cars

Over the summer holidays I got my son to create a spreadsheet detailing some of his toy cars which we then imported into PowerBI. Created by a 7 year old in under an hour  

School MIS systems in the England

Updated 10/10/2016 – Added Map using the new Shapes option in PowerBI. Every School in England regularly publish data to the Department of Education and Graham Reed has published a spreadsheet of the software each school used to submit their returns. Below is a PowerBI report I’ve created based on his data. You can also access by clicking here

Incorrect x-axis dates

So yesterday I was playing with PowerBIand I hit a problem. As you can see, the x-axis is wrong. For some reason it wasn’t creating the correct range. After a little while I thought I’d try out Microsoft’s feedback option – the smiley faces – in the top menu you have a smiley face. Click it, then you can send feedback. Sure enough after a little while, Justin Schneider from the Power BI team replied asking a few more questions – basically asking me to check the data was correct, which was in this case and even offered a solution.

SQL Auditing sucks

Back in SQL Server 2008 Microsoft introduced auditing, specifically the Database Audit Specification. It’s pretty good – despite the title I do actually think its nice feature, it pretty much works and doesn’t have much of a performance impact, my problem is not much love has gone into it since it was released in SQL2008. It claims to meet various regulations such as the EU data Protection Directive, HIPAA, PCI DSS and to be fair, I’m sure it does.

Feature request – Add Azure Data Catalog support to SSDT

One of the annoy parts of building a warehouse is building the staging database, its an important first step. The staging database is replica of the source systems. BIMLcan provide away to accelerate this process, however its not perfect. BIML is designed to create SSIS packages – and it does this very, very, well. SQL database objects, not so well. SSDT does this well. The first step in the designing a warehouse is discovery, Azure Data Catalogis an excellent tool for doing the discovery, it allows you to connect to a wide array of data source types and gather the meta data that can be used to build the staging database.

BI resources

Below is a list of useful BI links 

Getting a list of AD Groups and their members using PowerQuery

The Power Query Formula Language (informally known as “M”) is a powerful mashup query language optimized for building queries that mashup data. It is a functional, case sensitive language similar to F#, which can be used with Power Query in Excel and Power BI Desktop. A few days ago I got asked to produce a list of users (and their email address) in a number of AD Groups. I already had a SSIS package that had a script task to pull this data from Active Directory and push it into a SQL database and we have a PowerShell script to get the same data in our code repo.

Is today Day Zero or Day One?

This might sound pretty pointless and silly but when talking about a 20 working day SLA a day can mean alot between a massive fine or compliance. During one of my driving lessonswith Garywe were talking about his recent streak of passes. The conversation then lead on to pass rates and us wondering how this broken down – for example, by gender, ethnicity, age, driving instructor type (intense driving school \ chain \ individual driving instructor \ family etc).

Error building SSDT package

Today my colleague had a problem opening our BI solution, the solution had multiple projects, including 3 SSDT projects. Although the project builds correctly on both my machine, the build machine and another colleague machine it refused to build stating that the reference to the object in another project was invalid. After thinking for a few moments, I remembered I had seem this before. The problem was a bug in SSDT.

Microsoft shows Data Pros some love

Microsoft has released SQL Server 2016 Developer, for free. That’s right, zero, zip, nothing. Completely free. The developer edition is fully featured, it contains the same goodness as the $$$ Enterprise edition. The catch? Its not for production.

Logging ConnectionStrings in SSIS

Another day, another reference to an old Jamie Thomson blog post– today it was getting the ConnectionString to output to the information. Admittedly I wasn’t looking for how to fire the connection string into an information event, just the ConnectionString, still another great idea and its useful for debugging.  

Enterprise vs Enterprise Core

This month saw the release of SQL Server 2016, which from a BI\Report point of view is huge. Once we had access to it yeah, already been on msdn. Can't believe we're being made to wait!!! — Dave Kerby (@davekerby) June 1, 2016 I spotted something odd. So whats the diff? Enterprise is only core licensed - #sqlhelp @sqlserver @SQLServerBI — Matt Smith (@matt40k) June 6, 2016 Now, in SQL Server terms there are two types of licensing

MDSCHEMA_CUBES DMV not returning all cubes

I had previously created a SSIS package with a simple Process Full on the SSAS MD database, however, as the project has progressed this hasn’t been ideal. Its basically all or nothing. In order reduce the damage a failure can cause I’ve setup a Process Full for each dimension and cube so each is processed independently of each other. I’ve used the data from the Analysis Services Dynamic Management Views, or DMV for short, which I’ve used for the documentationto feed the foreach loop.

New laptop, error running SSIS package

So today I went to run a SSIS package on my new laptop and bam, error message. Microsoft.SqlServer.Dts.Runtime.DtsRuntimeException: The package failed to load due to error 0xC0011008 “Error loading from XML. No further detailed error information can be specified for this problem because no Events object was passed where detailed error information can be stored.”. This occurs when CPackage::LoadFromXML fails. —> System.Runtime.InteropServices.COMException: The package failed to load due to error 0xC0011008 “Error loading from XML.

SQL Server vs Containers

I was at event that Simon Sabinarranged, SQL SERVER TOOLS and SSDT shape the futureand one of the questions that was asked, sorry, I forgot who asked (joys of leaving it too long before blogging about it – I want to say Gavin) was, How does (Windows) Containers fit into SQL Server? This is basically Microsoft integrating the Docker technology into the Windows OS, so you in effect, ship a more complete solution.

Power BI – custom visuals

Power BI is Microsoft new cloud based Business Intelligence platform, one of the cool things about it is it built atop internet standards and leveraging open source libraries like D3.jsfor its visuals. Microsoft has continued on the path of open source-ness by open sourcingsome of its codes allowing Joe Public to submit custom visuals to Power-BI, all via GitHub. Yesterday saw the introduction of 4 new visuals, one in particular looked very interesting – Hierarchy Slicerby Jan Pieter Posthuma.

SQL Compare failed

Another day, another problem. Todays problem was incorrect column length which was causing a SSIS package to fail. Historically I use a freeware tool to compare the different environments schema, but lately I just don’t bother, I have continuous deployment so its easier just to hit the deploy button then it is to try and figure it out. Well today I decided I wanted to it old school – although I’ve been using the SSDT SQL compare in my deployments, I’ve not actually used it in terms of viewing the differences.

The connection is broken and recovery is not possible

Msg 0, Level 11, State 0, Line 0 The connection is broken and recovery is not possible. The client driver attempted to recover the connection one or more times and all attempts failed. Increase the value of ConnectRetryCount to increase the number of recovery attempts. In short, you’ve lost your network connection, either your WiFi has dropped out or your network cable has 🙂

Auditing – Who’s viewed my record?

As some of you are aware I currently working for a local authority and anyone who has been following local government news in England will know the bonds between local authority and the health authority, aka the NHS, are growing. As are other parts of the government for that matter, both central, other local authorities as well as charities, but for now, I am focusing on the health part. Specifically the part around what that means.

Auditing – SSAS

There are various types of Auditing in the Microsoft BI stack. There is auditing in SSRS, SharePoint, SSAS and not forgetting SQL has its own auditing. Today I am looking at the SSAS auditing – you can find out more about it on TechNet. Olaf Helperhas published some TSQL code for querying the audit data -on the Script Center. But first, we need a table to store the data, here is a TSQL script for creating the table:

Building a test system

So eariler this month I switched from my Azurehosted TeamCityserver to Appveyor, which freed up some much need Azurecredit. This month, it looks like its going to be going on a Visual Studio Ultimate vm for some SharePoint development. Hopefully this won’t take too long to do as I really want to get on with machine learning after a demo from Ric Howe. I’m also looking at building a few test machines locally – old school I know!

Showing SIMS Bulk Import some love

So today I managed to move SIMS Bulk Importover to Appveyor. So what does this mean? Well it means I don’t have to worry about going over my Azure credit limit each month for starters! Appveyorhas excellent support for GitHub, so each commit is automatically build, tested, and (when I enable it) a new release created. The next release will include Pupil username creation – you can blame\thank Andrew Mulhollandfor this, I saw his project PiNet and thought we can make this better.

Web hosting renewal

One of the bad habbits I have picked up from working in Local Government is reviewing your options, regularly. As I approach my renewal for this site’s web hosting, I decided to revaluate the market. (Short version, I’m with Mythic-Beastsfor another year) Currently, I have the site hosted with Mythic-Beasts. They are pretty cool folks, unfortantely one of there DC had a power cut recently – ok, only a few hours and I pay peanuts so I don’t expect much in terms of reduancy or fancy SLAs, but it did make me look at my options for a bit longer then normal.

Clever SQL Jobs?

Just thinking out load (because its good to get feedback) Just to expand on my tweet. I’ve lot of SSIS packages for load and building my BI warehouse (SSAS cubes), now currently I have a SQL job that has multiple steps For example Load data from Source system into Staging Build ODS intermediate tables Build Warehouse tables And this works for DataMart where they are a denormalized copy of a single source system, but when the source system, or the intermediate tables, are used multiple times, you don’t want to run them multiple times.

DacPac vs SqlCmd variables

I recently discovered the ability to reference other database project (I know, I know), anyway, this cause my lovely automate build\deployment process to fail. Investigating the error led me to, yes, you guessed it a Jamie Thomson’s blog(from 2012). Thankfully, I managed to figure out a clean solution, the DacPac solution on Octopus Deploy library has an option to pass Profile Name, which is the publish profile XML file. He is the cut-down version I’ve used, just the missing variables

Automatic build and deployment process for Microsoft BI

I’ve previously blogged about how I was going to detail how I build and deploy my Microsoft BI projects and to be honest, I’ve been putting it off. The main reason is my work has been looking at Visual Studio Team Services, formally Visual Studio Onlineand I was looking at using the Build functionality that is included. This should make the whole setup process a lot easier for anyone else trying to replicate my setup.

East Anglia SQL Server User Group

Anyone working with SQL Server will be well aware of the mountains of training material, blog posts, videos, code snippets, events and other helpful resources that helps anyone who comes into contact SQL Server regardless if they are a beginner or a seasoned veteran (even MVPs!). One of the pools of knowledge is SQL PASS. is an independent, not-for-profit organization run by and for the community. With a growing membership of more than 100K, PASS supports data professionals throughout the world who use the Microsoft data platform.

Automated SQL Server BI deployments with OctopusDeploy

Today, my pull request for the SSAS deployment script was merged into the Octopus DeployCommunity Library. So what is Octopus Deployand why should I care? Octopus is a friendly deployment automation tool for .NET developers. But don’t be fooled. Octopus Deployisn’t just limited to .NET developers deploying hip new cloud based applications, it can do much, much more. I currently use Octopusto deploy Jasper reports to our 6 HR environments, I also use it to deploy our HR interfaces which are a series of SSIS packages and SQL objects which uses Microsoft SQL Server 2008.


So I’ve been playing wit mapping data, one of the nice things about GitHubis that it supports GeoJSON and automatically renders it With the mapping data already imported into Geographyin SQL Server it was easy enough to convert to GeoJSON (Stackoverflowto the rescue again!)

Getting collation descriptions

Just an updateas I noticed I forgot to say how to get the names and descriptions of the collations your SQL Server supports

Using PowerShell to check setup

So I was handed over some new servers, before I got started I wanted to check everything was as it was suppose to be, so I wrote a few PowerShell scripts to check each server was setup correctly. I wanted my DEV to 100% match my UAT and so on. First, I wanted to check who has local administrator access on the servers. Next, I wanted to check the drives – I wanted to make sure each drive letter was the same across environments and all had a proper label.


I don’t know a lot about Collation – I know how to set it in SQL Server, I know it’s important, it know chaos can ensure if your not careful for example if you set it to CS rather then CI*, so my general tack is to leave it as default and hope it leaves me alone. Today however, this happened Turns out in the VS SSDT package, I have forgotten to set one of the databases collation in project settings to Latin1_General_CI_AS to match the default on SQL Server (rather then the VS SSDT default)

Deploying SQL 2014, looking at SQL 2016 \ PowerBI

My current thoughts on SQL2016/PowerBI as deploying SQL2014. Ok, so we’re looking at moving to pure SQL 2014 for our corporate BI solution. I say pure as we normally use SQL server as the backend then other tools on top, so we’re looking at enabling a lot of extra features we’ve technically had for years but just not enabled\setup\whatever. SQL2016 is on the way and we’re most likely going to upgrade pretty quick, lots of new features that look very useful.

Problem with ISNUMERIC

Todays problem was brought to you by a highly customizable system with poor user validation and a user who doesn’t know the difference between data and formatting. First the system. The supplier has created a highly user customizable system, they can for example create a web form with zero coding – 100% user config. The problem is, technically, it creates a mess. So your nice new form with a number of questions stores all its answers in a single table with the data going into a nvarchar(max) column – which is fine for free-text columns, but not so good for integer fields.

Joys of C#

On Friday, I had a problem raised, they want to change one of the automatic rename jobs which renames and moves export files to a shared drive. Lucky, I had created a SSIS package, which was a SQL Job that runs on a scheduled every 15 mins, which uses a script task to perform the logic. Because we also used OctopusDeploy, this really was going to be a 5 min change.

SQLSaturday #411 – Cambridge 2015

For a while now I’ve been going to the Cambridgeshire SQL Server User Group, which I have found really useful especially as we are at moving towards a more self-service model across IT which requires IT to be a bit more cleverer. Beyond the informative speaker talks you get a really good networking opportunity. The group is run by Mark Broadbent(@retracement) and has recently merged with SharePoint Saturday Cambridge. Although Cambridge is about the same time on the train as London.

Change text direction – Visual Studio 2013 / SQL Server 2014

I’ve spent a while today trying to change the text direction (orientation) of a header on a SSRS report – the option was rather oddly placed in Visual Studio – the option is within the properties (for the text) Localization > Writing Mode > then change from Default to either Rotate270 or Vertical. I was hoping it would be within the Font options. Guess that would be too easy 😉

Missing Reporting data toolbox in Visual Studio 2013 / SQL Server 2014

Todays problem, missing the reporting data toolbox in Visual Studio 2013 – once again, a different menu options. Open the actual report (.rdl) View > Report Data Alternatively press Ctrl + Alt + D

Missing SSIS toolbox in Visual Studio 2013 / SQL Server 2014

Stupidly today, I accidently closed the SSIS toolbox. After a bit of unsuccessful searching in the Visual Studio menus, I gave up and done the IT norm and Google it – hoping I wouldn’t have to reinstall. Luckily I found James Serrahad already blogged about it, the only difference is Microsoft has changed the icon, its still in the same place at least So when is a Visual Studio Toolbox, not a Toolbox?

SSIS jumping in Visual Studio 2013 / SQL Server 2014

One really annoying bug in SSIS when using Visual Studio 2013 / SQL Server 2014 is the object jumping, so when you drag an object like a sequence container down the object shoots to the bottom. Thankfully the fix is simple. Close Visual Studio and apply SQL Server 2014 SP1to your machine.


One of the problems we have is documentation and how to keep it up-to-date. We tend to deliver a new batch of reports as part of a CR (Change Request) then make changes as part of BAU (Business as usual) unless its a particularly complex or time consuming change. The problem is often BAU changes happen rapidly often within a short time frame and time isn’t always allocated to updating documentation.

Small world

So I’ve been following a few folks on Twitter – one of them is Tim Mitchellfrom the US, he tweeted a link to some resourceshe used in a recent SQL Saturdayhe presented at. One of them is Who happen to be based in Holbeach, which is really close to my home town of Wisbech. You go all over the world in search of knowledge, only to find there’s a guru right on your own doorstep.

Failed RavenDB backup

I had a rather odd error with OctopusDeploy where the RavenDBbackups were failing. Turns out the problem was because I had assigned 1 IPv4 address to OctopusDeploy and another IPv4 address for TeamCity. Oddly the RavenDB backup routine referenced the localhost address –, it appeared to be hardcode with no option other then changing port. This was especially weird as the OctopusDeploy service, was able to access it fine. To resolve – at least long enough to get a backup to migrate to v3 which uses SQL Server– was to setup a port proxy so accessing TCP port 10931 on the local loopback would resolve to the same port but on a the actual public IPv4

Moved backup locations

It appears IBM has moved the Cognos backup folder location from: C:\Program Files\ibm\cognos\c10_64\bkp To: C:\Program Files\ibm\cognos\c10_64\uninstall

Swollen Business Objects database

Our new HR system uses SAP Business Objects for transactional reporting, as this was setup by our HR supplier we haven’t really got involved with it beyond writing a few custom reports, turns out we’ve written quite a few, so much so that the Business Objects CMS database has swollen to over 30 GBs. This, as you might have guessed, is causing a few performance problems. After a discussion with the supplier, we found out it was storing all the reports, even the ad-hoc one-off reports and we should have set a parameter to limit it.

New Azure features

Two new Azurefeatures have caught my eye – the first is Logic App. In a nutshell this is workflow. You have connectors and you can add triggers and actions. So you could have a trigger on your HR system that when a new employee starts, it creates a user account then sends a welcome message to the existing staff informing them of the new starters. The other is Data Catalog, this basically allows you to generate meta data about your data sources.

Parking fines issued in Ipswich

I made the above video using Microsoft Excel using data from a FOI request to Ipswich Borough Councilabout the number of Parking fines issued since 2010.

Toshiba laptop update

Just an update to my previous post about the Toshiba Laptop, I’ve been thinking about reinstalling the Toshiba laptop with Ubuntu, but before I do that, I want a backup. Luckily, I still have the original 500GB hdd spare, I just need a USB caddy. Luckily my local Curry’s had a USB3.0 one (PNY SSD and 2.5″ Hard Drive Enclosure and Upgrade Kit), before anyone comments about how I could of got one cheaper off eBay, just remember, I was able to get it same day and it included Acronis.

Technical debt

I was talking to a colleague about one of our suppliers and the progress they were making on a new product they were developing. He was surprised to hear how little progress they appear to have made, they had previously been making huge steps in very little time with very few developers, especially surprising when they have been scaling up the development team over the past few months. I explained they are burning some of the technical debt they have collected whilst they still can.

Toshiba R50-B-12Q

UPDATE: I’ve detailed the Toshiba default build a bit hereAt the beginning of the month I order a new laptop, for the past few weeks I’ve been wanting something more mobile then my desktop, at first I was toying with the idea of selling my desktop and picking up a new Surface. The new Surface 3 which starts at £419 seemed like a good idea what with its sexy full hd touchscreen, however when you scratch the surface (ha ha) it doesn’t look so go

Version Control

I’ve been thinking lately about the adoption of version control. One of the common fears of using version control is that its permanent. All mistakes are visible and are in fact, highlight. Originals are updated with new commits which detail only the change and a nice comment. There is no hiding mistakes. The thing is Mistakes are what make us human. Everyone makes mistakes, we need to accept that fact. In doing so we can move forward faster.

Keyboard mapping [SQL 2014]

I’ve been working on SQL Server 2014 lately and one of the annoying problems I’ve stumbled across using SSMS is that F5 no longer means execute the SQL query, it now means Debug. Which brings up an annoying popup (least for me) about configuring my Windows Firewall. This can be changed in SSMS by going Tools > Customize… > Keyboard… The default is: Debug.Start (F5 (Global)) And the traditional is:

Local Government to go Academy

It appears the Local Government is looking to follow in schools footsteps and do an “Academy” – cutting the red tape and be given more control and freedom.

4th Anniversary of contributing to Rosetta@Home

Tomorrow, Sunday the 7th December 2014, I will be celebrating 4 years of contributing wasted CPU cycles to the excellent Rosetta@Home project. Rosetta@homeproject is determining the 3-dimensional shapes of proteins through research, that may ultimately lead to finding cures for some major human diseases such as HIV, Malaria, Cancer, and Alzheimer’s. The project is lead by Dr David Baker at the Univerity of Washington. You can find out more about the Rosetta@Homeproject on their website – http://boinc.

Technical Analysis on Schools websites in England

So in my last blog postI asked, are you ready for IPv6? The post came about when I was looking at Schools MIS data, which Graham, Joshuaand myselfhave being look at to see who are the big movers and shakers in the Schools administration software (MIS) arena. Data is collected by what software suppliers a school uses to submit the School Census (in England) which is requested under the Freedom of Information (FOI) from Department of Education(DfE)(saving having to FOI every individual school).

Are you ready for IPv6?

For a long while now it has been known that IPv4 will run out of available publicly assignable IP addresses. IPv4 address are 32 bits wide and quite simply can’t cope with the demand of the modern world – despite NAT’ing. IPv6 looks to resolve this by have not only more digits, but also using hexadecimal. For example, an IPv4 address looks like this 192 . 168 . 0 .

Slow progress

I’m currently getting bogged down with other projects and unfortunately my final sprint of SIMS Bulk Importhas grind to a halt. On the plus side I managed to swash a few more bugs last month, get a code signing certificate. This will mean all future releases will be signed as coming from me which is great news, it adds a layer of confidence that the code you run is unaltered by a third party.

TeamCity AssemblyInfo Patcher not working

Tonight I was trying to get TeamCityto auto-update the version number for my SIMS Bulk Importapplication, however the simple AssemblyInfo Patcherjust failed to work. No errors. Nothing. Then I read the manual, again… Note that this feature will work only for “standard” projects, i.e. created by means of the Visual Studio wizard, so that all the AssemblyInfo files and content have a standard location. So I created a new package which placed the AssemblyInfo.

WIX installer why you no MSBUILD no more?

Last night I chopped up my WIX installer project – I basically split the actual installer parts into one file – CoreMsi.wxs, the versions into a variable file – Version.wxi and the file list into a separate fragment file – SimsBulkImport.wxs. This worked. Great I thought, simple PowerShell script to generate the file list, I can even grab the main repository complied bin as a artifact dependency – got to love TeamCity.

SIMS Bulk Import

Overview If your in the UK, the chances are your local school is using SIMS .net. Just checkout the stats on their site. 22,000 schools taking 2,500,000 children’s attendance, every day. That’s impressive.So what is SIMS .net? Well its a MIS system, but what is a MIS system? In the simplest terms its a database that holds the school records, for both students and staff. So it makes sense that you are going to want to interface with it as it’s your at your core when it comes to data sources.

Living in a IPv6 world

I’ve finally gotten round to setting up the VDSMythic-Beastshave kindly donated to host the API backend for one of my open-source projects. A few nights ago I enabled IPv6 and today when I went to run apt-get update it just hung saying: 0% [Connecting to (2a01:450:10:1::10)] As they had previously worked, I suspect it’s IPv6 problem, quick google and… For sites which prefer IPv4 connections change the last line to precedence ::ffff:0:0/96 100 Bingo.

BI for IT operations

I always find it funny when IT departments talk about BI for their company and they don’t actually have a BI solution in place for themselves. Surely you should practice what you preach? So why do IT departments need BI. Well lets look at Rosetta@home. Jul 30, 2014 We are aware of significant network slow-down between the subnet upon which our servers sit and the Internet beyond the UW campus.

PowerShell, DigitalOcean and WordPress

Back in May (2014) I decided to start up a blog, again. I decided to keep it simple and just get a off the shelf hosting package rather then going to extremes like I normally do – configuring a VPS or even a Dedi server! I ended up going with CS New Media– it was just their lite Linux package which I signed up for 3 months. I was tempted to stick with what I know and go with Mythic-Beastslike I normally but I like to tinker and try new things so I went with CS New Mediaafter a failed attempt to spend my dosh with NameCheap– when it came to entering my card details, it just failed to connect to “their” backend and their response was, well it works for me, try another browser – by the time I phoned them I had already tried different browsers, internet connections, computers so at that point I just walked away.


Well my day is finally coming to an end, my bed is calling me after a long day. If I had thought about it more I would have packed a few cans of Pepsi Max and probability took a half day at work, as it was I spent my lunch break by coming home and taking my son to the park*, then heading off to Cambridge after work for mine and my colleagues first Cambridgeshire SQL Server User Group.

The term ‘Invoke-WebRequest’ is not recognized

I’ve currently been working on a PowerShell script to build a virtual server using DigitalOceanexcellent RESTful API. So far this has gone extremely well and I have been impressed with both PowerShell and DigitalOcean’sAPI. Today however I tried running on another machine (an important part of development) only to find it doesn’t work. The first error that appears is: The term ‘Invoke-WebRequest’ is not recognized as the name of a cmdlet, function, script file, or operable progr

Clear Visual Studio recent projects

Another little gem, clearing out those pesky recent projects from your menu on Visual Studio 2012. Fire up RegEdit and navigate to: Files: HKCU\SOFTWARE\Microsoft\Visual Studio\11.0\FileMRUList\ Proj: HKCU\SOFTWARE\Microsoft\Visual Studio\11.0\ProjectMRUList\ Usual disclaimer


Found this little gem – appears they actually made those really cool interfaces they used in Tron Legacy: http://www.

Created new user is disabled

I’ve created a new SQL user along with a new role in our warehouse db project using Visual Studio 2012. CREATE LOGIN [newuser] WITHPASSWORD=’{{ RANDOM PASSWORD }}’,DEFAULT_DATABASE=[Database] GO CREATE USER [newuser] FORLOGIN [newuser] WITHDEFAULT_SCHEMA=[dbo] GO ALTER ROLE [newrole] ADD MEMBER [newuser] GO  Problem is, the user is disabled. When you do it manually via SQL Management Studio, it’s enabled and working. Manually enabling the account doesn’t allow it connect either.

Why Microsoft Business Intelligence is on my radar

In my day job I’ve been looking at SQL Server 2012 recently (I know SQL Server 2014 out before you say anything!), I’ve had a developer license for a while but unfortunately other priorities have been consuming my time and I’ve only managed to get a dev server build, SQL 2012 installed and not much else up till a few days ago. Our current Business Intelligence platform is using SQL Server 2008 (not R2) for the database backend and SQL Server Integration Services (SSIS) BIDS for the ETL.

How not to write a newsletter

I received NameCheap newsletter today full of annoying holes (at least for me), needless to say I hit the unsubscribe button shortly after.  First mistake – mark your calendars. Did they not check their calendar before sending it? The June 10 was 2 weeks ago! $1, one dollar!! You’re trying to promote .UK, use the local currency, using £ (GBP)!   Final mistake I notice (I didn’t look very hard to be honest) was the “SCHOOLSOUT” code.

SOLUS2, will you not just die already!

Well it appears Capita has announced SOLUS2 death day   That’s assuming they don’t put it back. Again. This is bad news. Let me explain why. I’m using Chrome, it’s updated itself I don’t know how many times. If I go and check, all I know or care about is that it is indeed up-to-date, so at some point, it’s updated itself and its worked. If I look at my antivirus I can see it’s updated 3 times today alone.

I’m sorry, but SIMS8 can’t come fast enough

I haven’t really been on SupportNet, Capita SIMSsupport portal, regularly since changing roles about 20 months ago, the only time I pop on nowadays is to find something when I’m helping out my old team during busyhard times (you never really escape at my work). On this occasion I stopped by the forum and one thing jumped out at me. The lack of progress. Now I appreciate I’m about to unload both barrels into my foot, but I think it needs to be said.

Want a job? Head to Superdrug

One of the things you’ll find when your going to interviews for your first job after leaving education is that they ask the same types of questions. The main one that feels me with dread is “Give an example of good customer service you have provided”. I hate that question. Hopefully your interviewer would have been kinda enough to at least state that, as you are ill experienced, you might want to give examples of good customer service you have received.

Why I like Twitter

I have to say I’m really liking Twitter. A lot of my friends can’t see the point. As a social tool it allows gossip to flow really well, how often do we hear that a celeb has tweeted a inappropriate comment or photo. It allows us fans to be alot more closer. As a business tool it allows a new way of interaction, take this example – there is a bug in .

DfE big plans for big data

An interesting debate erupted on EduGeekthis month after Graham Reed, a technologist, made a post on Eduware Consultingblogabout the changing data landscapeand how he saw Department of Education (DfE)data exchange and warehouse projects changing the way MIS work and the way they are used. Now of course we need to take it with a pinch of salt, firstly, this is very much early days. At the moment it’s a vision looking to become a project, and we all know how government projects end up post election time.

Should I purchase Full SQL or is Express enough?

One of the common questions I used to get asked whilst supporting SIMS .netwas, do I need Full SQL. SIMS .netuses Microsoft SQL Serveras it’s relational database management system (RDBMS) so it’s important to ensure the engine is correctly sized for job at hand. Microsoft SQL Serverhas come in a number of editions over the years, I won’t detail all the various editions but the two editions schools should be concerned with is Express and Standard.