Michael J. Swart

March 12, 2019

Lonely Tables in SQL Server

Filed under: SQL Scripts,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 12:00 pm

Takeaway: I provide a script that looks at the procedure cache and reports tables that are never joined to other tables.

Recently, I’ve been working hard to reduce our use of SQL Server as much as possible. In other words, I’ve been doing some spring cleaning. I pick up a table in my hands and I look at it. If it doesn’t spark joy then I drop it.

If only it were that easy. That’s not quite the process I’m using. The specific goals I’m chasing are about reducing cost. I’m moving data to cheaper data stores when it makes sense.

So let’s get tidying. But where do I start?

Getting rid of SQL Server tables should accomplish a couple things. First, it should “move the needle”. If my goal is cost, then the tables I choose to remove should reduce my hardware or licensing costs in a tangible way. The second thing is that dropping the table is achievable without 10 years of effort. So I want to focus on “achievability” for a bit.


What’s achievable? I want to identify tables to extract from the database that won’t take years. Large monolithic systems can have a lot of dependencies to unravel.

So what tables in the database have the least dependencies? How do I tell without a trustworthy data model? Is it the ones with the fewest foreign keys (in or out)? Maybe, but foreign keys aren’t always defined properly or they can be missing all together.

My thought is that if two tables are joined together in some query, then they’re related or connected in some fashion. So that’s my idea. I can look at the procedure cache of a database in production to see where the connections are. And when I know that, I can figure out what tables are not connected.

Lonely Tables

This script gives me set of tables that aren’t joined to any other table in any query in cache

use [your db name here];
SELECT qs.query_hash,
       cast(null as xml) as query_plan
  INTO #myplans
  FROM sys.dm_exec_query_stats qs
 CROSS APPLY sys.dm_exec_plan_attributes(qs.plan_handle) pa
 WHERE pa.attribute = 'dbid'
   AND pa.value = db_id();
WITH duplicate_queries AS
  FROM #myplans
DELETE duplicate_queries
 WHERE n > 1;
UPDATE #myplans
   SET query_plan = qp.query_plan
  FROM #myplans mp
 CROSS APPLY sys.dm_exec_query_plan(mp.plan_handle) qp;
WITH XMLNAMESPACES (DEFAULT 'http://schemas.microsoft.com/sqlserver/2004/07/showplan'),
my_cte AS 
    SELECT q.query_hash,
           obj.value('(@Schema)[1]', 'sysname') AS [schema_name],
           obj.value('(@Table)[1]', 'sysname') AS table_name
      FROM #myplans q
     CROSS APPLY q.query_plan.nodes('/ShowPlanXML/BatchSequence/Batch/Statements/StmtSimple') as nodes(stmt)
     CROSS APPLY stmt.nodes('.//IndexScan/Object') AS index_object(obj)
SELECT query_hash, [schema_name], table_name
  INTO #myExecutions
  FROM my_cte
 WHERE [schema_name] IS NOT NULL
   AND OBJECT_ID([schema_name] + '.' + table_name) IN (SELECT object_id FROM sys.tables)
 GROUP BY query_hash, [schema_name], table_name;
WITH multi_table_queries AS
    SELECT query_hash
      FROM #myExecutions
     GROUP BY query_hash
    HAVING COUNT(*) > 1
lonely_tables as
    SELECT [schema_name], table_name
      FROM #myExecutions
    SELECT [schema_name], table_name
      FROM #myexecutions WHERE query_hash IN (SELECT query_hash FROM multi_table_queries)
SELECT l.*, ps.row_count
  FROM lonely_tables l
  JOIN sys.dm_db_partition_stats ps
       ON OBJECT_ID(l.[schema_name] + '.' + l.table_name) = ps.object_id
 WHERE ps.index_id in (0,1)
 ORDER BY ps.row_count DESC;


So many caveats.
There are so many things that take away from the accuracy and utility of this script that I hesitated to even publish it.
Here’s the way I used the script. The list of tables was something that helped me begin an investigation. For me, I didn’t use it to give answers, but to generate questions. For example, taking each table in the list, I asked: “How hard would it be to get rid of table X and what would that save us?” I found it useful to consider those questions. Your mileage of course will vary.

October 26, 2018

Uncovering Hidden Complexity

Filed under: Miscelleaneous SQL,SQL Scripts,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 12:15 pm

The other day, Erin Stellato asked a question on twitter about the value of nested SPs. Here’s how I weighed in:

Hidden complexity has given me many problems in the past. SQL Server really really likes things simple and so it’s nice to be able to uncover that complexity. Andy Yun has tackled this problem for nested views with his sp_helpexpandview.

Here’s what I came up with for nested anything. It helps unravel a tree of dependencies based on information found in sys.triggers and sys.dm_sql_referenced_entities. With it, you can see what’s involved when interacting with objects. Here’s what things look like for Sales.SalesOrderDetail in AdventureWorks2014. A lot of the resulting rows can be ignored, but there can be surprises in there too.

A lot in there

DECLARE @object_name SYSNAME = 'Sales.SalesOrderDetail';
WITH dependencies AS
    SELECT @object_name AS [object_name],
             QUOTENAME(OBJECT_SCHEMA_NAME(OBJECT_ID(@object_name))) + '.' + 
             as sysname) as [escaped_name],
           object_id(@object_name) AS [object_id],
           1 AS is_updated,
           CAST('/' + CAST(object_id(@object_name) % 10000 as VARCHAR(30)) + '/' AS hierarchyid) as tree,
           0 as trigger_parent_id
      FROM sys.objects 
     WHERE object_id = object_id(@object_name)
    SELECT CAST(OBJECT_SCHEMA_NAME(o.[object_id]) + '.' + OBJECT_NAME(o.[object_id]) as sysname),
           CAST(QUOTENAME(OBJECT_SCHEMA_NAME(o.[object_id])) + '.' + QUOTENAME(OBJECT_NAME(o.[object_id])) as sysname),
           CASE o.[type] when 'U' then re.is_updated else 1 end,
           CAST(d.tree.ToString() + CAST(o.[object_id] % 10000 as VARCHAR(30)) + '/' AS hierarchyid),
           0 as trigger_parent_id
      FROM dependencies d
     CROSS APPLY sys.dm_sql_referenced_entities(d.[escaped_name], default) re
      JOIN sys.objects o
           ON o.object_id = isnull(re.referenced_id, object_id(ISNULL(re.referenced_schema_name,'dbo') + '.' + re.referenced_entity_name))
     WHERE tree.GetLevel() < 10
       AND re.referenced_minor_id = 0
       AND o.[object_id] <> d.trigger_parent_id
       AND CAST(d.tree.ToString() as varchar(1000)) not like '%' + CAST(o.[object_id] % 10000 as varchar(1000)) + '%'
     SELECT CAST(OBJECT_SCHEMA_NAME(t.[object_id]) + '.' + OBJECT_NAME(t.[object_id]) as sysname),
            CAST(QUOTENAME(OBJECT_SCHEMA_NAME(t.[object_id])) + '.' + QUOTENAME(OBJECT_NAME(t.[object_id])) as sysname),
            0 AS is_updated,
            CAST(d.tree.ToString() + CAST(t.object_id % 10000 as VARCHAR(30)) + '/' AS hierarchyid),
            t.parent_id as trigger_parent_id
       FROM dependencies d
       JOIN sys.triggers t
            ON d.[object_id] = t.parent_id
      WHERE d.is_updated = 1
        AND tree.GetLevel() < 10
        AND CAST(d.tree.ToString() as varchar(1000)) not like '%' + cast(t.[object_id] % 10000 as varchar(1000)) + '%'
SELECT replicate('—', tree.GetLevel() - 1) + ' ' + [object_name], 
       [type_desc] as [type],
       tree.ToString() as dependencies       
  FROM dependencies
 ORDER BY tree

July 3, 2018

Shifting Gears in 2018

Filed under: SQLServerPedia Syndication — Michael J. Swart @ 9:00 am

I wanted you to know about some changes coming to this blog. I’m shifting the focus from SQL Server to other technologies. Specifically, I’m going to explore and begin writing more about modern software development including things that have been labeled devops or site reliability engineering.

Shifting Gears

I’ve been looking for a new challenge for a while and I have an opportunity to do that by following the direction set by my company a few years ago. My company is embracing the public cloud for its price, its flexibility and its promise of scalability. Which public cloud? As awesome as Azure is, we’re going all-in AWS.

For me, this means new lessons to learn and new things to write about.

My Audience

My target audience for the new topics include

  • People searching google who want to find the answers to the same questions I learned recently.
  • The developer who is super-familiar with the Microsoft Stack (aka yours truly) but who wants to branch out into a new stack.

I hope that still includes you.

Blogging as a Student

I have no problems blogging as a learner. Just look at Kenneth Fisher (@sqlstudent144) and Pinal Dave (@SqlAuthority). They both began their blogs from the point of view of a learner. That word “student” is even there in Kenneth’s handle. And Pinal’s site is about his “journey to authority”, another colorful expression for learning. And they’ve done it. They’ve both successfully gained a reputation as an authority in their field.

My Topics

I’ve often straddled the line between a Developer and a DBA. I know a little bit about what it takes to keep SQL Server happy and healthy. I look forward to expanding my “Site Reliability Engineering” skills into new areas.

So for the next few weeks, I’ll start by blogging about the tools I use and what it takes to get started on a simple project.

It’s About the Arrows
Software architecture is often over-simplified as drawing boxes and arrows describing things (the boxes) and how they’re organized or how they communicate with each other (the arrows).

One thing I’ve noticed is that programs used to be the hard part. The classes, the objects, the algorithms. Now it seems to me, that the arrows are the hard part. Deployment, security, automation and all that network stuff can’t be deferred to another team.

The Arrows Are The Hard Part

The Arrows Are The Hard Part

I may specialize in something in the future, but for now I have no shortage of topics. I’ve been tracking my google search history: Here’s what that looks like for the past two weeks:

  • youtube getting started terraform aws circleci
  • tf examples getting started
  • terraform tf examples getting started
  • terraform deploy to aws
  • specify descending in primary key
  • codepipeline
  • aws code deploy
  • dynamodb ttl attribute
  • lambda to dynamodb tutorial
  • javascript add 4 months
  • add days to time javascript
  • javascript get guid
  • Handler ‘handler’ missing on module ‘index’
  • TypeError: Date.now is not a constructor
  • Date.now is not a constructor
  • unix timestamp 1 day
  • dynamodb set ttl example js
  • dynamodb DocumentClient
  • specify region in document client
  • aws.config.update region
  • lodash
  • visual studio code
  • visual studio code marketplace tf
  • visual studio code marketplace tf terraform
  • terraform dynamodb attribute type
  • terraform lambda example
  • terraform output arn
  • create role terraform
  • iam_role_policy
  • best way to terraform a role
  • script out role for terraform
  • terraform dynamodb example
  • invoke terraform in aws
  • how to test terraform
  • terraform download
  • aws command line
  • how to create a role using terraform
  • terraform grant a role access
  • deploy a role with terraform
  • create role assume role
  • terraform role trusted entities
  • push a new repository to github
  • provider config ‘aws’: unknown variable referenced ‘aws_secret_key
  • terraform aws credentials
  • aws_profile environment variable
  • specify aws_access_key terraform
  • executable bash script
  • executable bash script windows
  • provider.aws: no suitable version installed
  • no suitable version installed
  • run terraform in circleci
  • run syntax circleci
  • run step syntax circleci
  • specify circleci environement variables
  • set password environment variable circleci
  • terraform “ResourceInUseException: Table already exists: broken_links”
  • terraform “ResourceInUseException: Table already exists:”
  • image hashicorp terraform
  • terraform EntityAlreadyExists
  • terraform backend dynamodb
  • canonical userid s3
  • deploy a lambda function terraform
  • terraform lambda runtime
  • resource “aws_lambda_function”
  • terraform archive_file
  • resource depends on
  • resource depends_on terraform
  • DiffTransformer
  • DiffTransformer trace
  • terraform archive_file example
  • depends_on terraform module
  • path.module terraform
  • windows path vs linux path terraform path.module
  • circleci zip directory
  • zip a file in shell
  • circleci zip
  • zip a file in circleci
  • working_directory circleci
  • zip directory for lambda
  • how to zip a file circleci
  • circleci apt-get zip
  • terraform export environment variables
  • run a shell srcript in terraform
  • steps in circleci
  • circleci artifact directory
  • build-artifacts circleci
  • store_artifacts
  • store variable in circleci
  • create file in terraform
  • output_base64sha256
  • concatenate in terraform
  • Unexpected value for InstanceType
  • Unexpected value for InstanceType terraform
  • terraform apply force
  • use artifacts terraform
  • get artifacts terraform
  • get artifacts circleci
  • use circleci artifacts
  • terraform file contents
  • terraform environment variables
  • use environment variables in terraform
  • var.Circle_artifacts
  • using environment variables in terraform
  • set variables when calling terraform
  • use environment variables in circleci
  • multiline circleci
  • wrap line circleci
  • terraform pass variable to module
  • echo in circleci
  • persist to workspace circleci
  • attach_workspace persist_to_workspace
  • persist_to_workspace
  • debugging circleci
  • git merge all changes into one commit
  • dynamodb materialized views
  • query dynamodb from js
  • query dynamodb from
  • aws_lambda_function filename
  • AWS Lambda Developer Guide
  • bash zip command not found
  • linux create zip file
  • upsert dynamodb
  • updateexpression example js
  • dynamodb docclient javascript update expression
  • use UpdateExpression to increment
  • The provided key element does not match the schema
  • dynamodb multiple key
  • javascript multiline string
  • javascript md5 hash
  • hash a string javascript
  • md5
  • simple hash string javascript
  • hash a string javascript
  • md5 bit length
  • Every entry in that list that doesn’t have an obvious answer is a blog post idea.

    Giving up SQL Server?

    No, not at all, I suspect that most of my day job will still be focused on SQL Server technologies. When I come across something super-interesting. No matter what, I’ll write about it.


    I’m excited. If you find yourself at AWS: Reinvent this fall, then let me know. Maybe we can meet for coffee.

    June 15, 2018

    ORDER BY newid() is an Unbiased Way To Randomize

    Filed under: Miscelleaneous SQL,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 9:47 am

    Mike Bostock is a data-visualization specialist. And it really shows in his blog. Every article is really well designed (which makes sense… many of the articles are about design).

    One of his articles, Visualizing Algorithms has some thoughts on shuffling at https://bost.ocks.org/mike/algorithms/#shuffling.

    He says that sorting using a random comparator is a rotten way to shuffle things. Not only is it inefficient, but the resulting shuffle is really really biased. He goes on to visualize that bias (again, I really encourage you to go see his stuff).

    Ordering by random reminded me of the common technique in SQL Server of ORDER BY newid(). So I wondered whether an obvious bias was present there. So I shuffled 100 items thousands of times and recreated the visualization of bias in a heat map (just like Mike did).

    Here is the heatmap. If you can, try to identify any patterns.

    Order By NewID Bias


      columns are the position before the shuffle,
      rows are the position after the shuffle,
      green is a positive bias and
      red is a negative bias.

    I don’t think there is any bias here. The problem that introduces bias in Mike Bostock’s example is that his “random comparator” that he defined does not obey transitivity. His words. “A comparator must obey transitivity: if a > b and b > c, then a > c.”
    But in SQL Server, because each row is assigned a newid(), ORDER BY newid() doesn’t have that flaw and so it doesn’t have that bias.

    But Be Careful

    Although the method is unbiased, ORDER BY newid() is still inefficient. It uses a sort which is an inefficient way of shuffling. There are alternative shuffle algorithms that are more efficient.
    ORDER BY newid() is good for quick and dirty purposes. But if you value performance, shuffle in the app.

    April 6, 2018

    Are There Any System Generated Constraint Names Lurking In Your Database?

    Names for constraints are optional meaning that if you don’t provide a name when it’s created or cannot afford one, one will be appointed to you by the system.
    These system provided names are messy things and I don’t think I have to discourage you from using them. Kenneth Fisher has already done that in Constraint names, Say NO to the default.

    But how do you know whether you have any?

    Here’s How You Check

    SELECT SCHEMA_NAME(schema_id) AS [schema name],
           OBJECT_NAME(object_id) AS [system generated object name],
           OBJECT_NAME(parent_object_id) AS [parent object name],
           type_desc AS [object type]
      FROM sys.objects
     WHERE OBJECT_NAME(object_id) LIKE 
             type + '\_\_' + LEFT(OBJECT_NAME(parent_object_id),8) + '\_\_%' ESCAPE '\'
           OBJECT_NAME(object_id) LIKE 
              REPLACE(sys.fn_varbintohexstr(CAST(object_id AS VARBINARY(MAX))), '0x', '%\_\_') ESCAPE '\'

    This will find all your messy system-named constraints.
    For example, a table defined like this:

    create table MY_TABLE
      CHECK (id >= 0)

    Will give results like this:

    Happy hunting.

    Update: April 9, 2018
    We can get this info from the system views a little easier as Rob Volk pointed out. I’ve also included the parent object’s type.

    SELECT OBJECT_SCHEMA_NAME(id) AS [schema name],
           OBJECT_NAME(constid) AS [system generated constraint name],
           (select type_desc from sys.objects where object_id = constid) as [constraint type],
           OBJECT_NAME(id) AS [parent object name],
           (select type_desc from sys.objects where object_id = id) as [parent object type]
      FROM sys.sysconstraints
     WHERE status & 0x20000 > 0
       AND OBJECT_NAME(id) NOT IN (N'__RefactorLog', N'sysdiagrams')
     ORDER BY [parent object type], [parent object name], [system generated constraint name];

    March 26, 2018

    T-SQL Options for Comparing “Distinctness”

    Filed under: Miscelleaneous SQL,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 8:40 am

    I had the privilege of listening to Itzik Ben Gan talk about “distinctness” in a talk he gave at PASS Summit. Distinctness is a relationship or comparison between two variables, just like equals (=). But unlike equality, distinctness treats NULLs in a more intuitive way (NULL is not distinct from NULL).

    There’s often confusion because equality in SQL is not like equality in mathematics. In particular equality in SQL doesn’t follow the reflexive property (∀A, A=A).

    Clear right?

    I explore different syntax options to test whether values are distinct or not. Each method has its pros and cons.

    The Setup

    Consider this table.

        AssignedTeamId INT NULL,
        AssignedSubTeamId INT NULL,
        TaskDetails NVARCHAR(2000) NOT NULL,

    When a task is unassigned, the AssignedTeamId and AssignedSubTeamId columns can both be null.
    Our goal will be to select an arbitrary TaskId given parameters @TeamId, @SubTeamId. And when parameters @TeamId and @SubTeamId are both null, I want to return an unassigned task.

    The Equality Join (doesn’t compare distinctness)

    I just want to post this here as an example that doesn’t work.

    DECLARE @TeamId bigint = NULL,
        @SubTeamId bigint = NULL;
    -- this will never return any rows when the parameters are null:
    SELECT TOP 1 TaskId
      FROM Tasks
     WHERE AssignedTeamId = @TeamId
       AND AssignedSubTeamId = @SubTeamId

    PROS: The syntax looks nice and clean.
    CONS: It doesn’t work for nulls.

    The Expanded WHERE Clause

    Well, let’s just write it all out then.

    DECLARE @TeamId bigint = NULL,
        @SubTeamId bigint = NULL;
    SELECT TOP 1 TaskId
      FROM Tasks
     WHERE ( (AssignedTeamId IS NULL AND @TeamId IS NULL) OR AssignedTeamId = @TeamId )
       AND ( (AssignedSubTeamId IS NULL AND @SubTeamId IS NULL) OR AssignedSubTeamId = @SubTeamId )

    There’s no way that syntax is sarg-able. But it turns out that it is. SQL Server works hard and says “I see what you’re trying to do there, I’ll help you out”.
    PROS: It works and it’s sarg-able.
    CONS: That syntax is sooo awkward.

    Using INTERSECT Syntax

    This is a tip that I got straight from Itzik Ben Gan who says he got the idea from Paul White. The idea is that INTERSECT doesn’t use the idea of equality, but of distinctness for it’s comparison. We can use this to create slightly nicer syntax.

    DECLARE @TeamId bigint = NULL,
        @SubTeamId bigint = NULL;
    SELECT TOP 1 TaskId
    FROM tasks
        SELECT assignedTeamId, assignedSubTeamId
        SELECT @TeamId, @SubTeamId

    The syntax is slightly less awkward, and it’s sarg-able. Or should be… But there’s a problem with this query (see if you can find it before reading further). Compare the two query plans. First the expanded where clause:

    The Expanded where clause produces an efficient seek.

    Here’s what the query with the INTERSECT syntax produces:

    The INTERSECT syntax produces an inefficient scan

    The secret to this mystery lies in that filter operator. There’s an implicit conversion there from int to bigint and that can cause a scan of the entire index. With the expanded syntax, SQL Server can handle the conversion gracefully. With the INTERSECT syntax it cannot. This was a really hard-earned lesson for us this week.

    Change the parameters @TeamId and @SubTeamId to INT to match and the query becomes sarg-able again.

    PROS: More elegant syntax and sarg-able (when you’re careful)
    CONS: This syntax causes performance issues with mismatched types. Take extra-special care to make sure types match up.


    Check it:

    DECLARE @TeamId bigint = NULL,
        @SubTeamId bigint = NULL;
    SELECT TOP 1 TaskId
    FROM tasks
    WHERE assignedTeamId IS NOT DISTINCT FROM @TeamId
      AND assignedSubTeamId IS NOT DISTINCT FROM @SubTeamId

    Talk about elegant! That’s what we wanted from the beginning. It’s part of ANSI’s SQL 1999 standard. Paul White tells us it’s implemented internally as part of the query processor, but it’s not part of T-SQL! There’s a connect item for it… err. Or whatever they’re calling it these days. Go read all the comments and then give it a vote. There are lots of examples of problems that this feature would solve.

    PROS: Super-elegant!
    CONS: Invalid syntax (vote to have it included).

    March 15, 2018

    “Failed to initialize deadlock control, Key cannot be null” When Viewing Deadlock Graphs

    Filed under: SQLServerPedia Syndication — Michael J. Swart @ 10:02 am

    I recently got this error in Management Studio when trying to view a deadlock graph that was collected with an extended events session:

    Failed to initialize deadlock control.
        Key cannot be null.
        Parameter name: key

    I found this error in a session that included the xml_deadlock_report event. I chose to “View Target Data” and then clicked the deadlock tab:

    So I couldn’t view the deadlock graph, but the xml for the deadlock was still accessible. I copy-pasted the deadlock graph information into an xdl file, I opened it in management studio and I still got the same error.

    No luck in Management Studio, but SentryOne’s Plan Explorer had no trouble displaying the deadlock graph visually. In fact it kind of helped me figure out what’s going on:

    I noticed a number of things:

    • The resource list is full of “exchangeEvent” elements (which has to do with parallelism) and not the usual key locks I’m used to seeing.
    • The “processes” are all the same process, just different threads (The spids on all processes are the same but the ecids are different)
    • Most importantly, there is no victim here! And our application that issued the query didn’t receive an error from the database.

    My guess is that the deadlock monitor detects this kind of deadlock cycle, but knows how to resolve it without choosing a victim and then issues a deadlock report any way.

    The reason that the management studio can’t display the deadlock is because it assumes that there is at least one victim in a deadlock graph.

    The original deadlock graph starts like this:

     <victim-list />
      <process id="process3ad41344e8" taskpriority="0" logused="10000" ...

    And when I edit it and pick an arbitrary victim like this:

    	<victimProcess id="process3ad41344e8" />
      <process id="process3ad41344e8" taskpriority="0" logused="10000" ...

    then Management Studio is happy again:

    This Kind of Deadlock Is Ignorable

    Take this with a grain of salt, but personally, I ignore these kinds of deadlock because there is no victim and my applications are unaffected.
    I think there are two issues here.

    1. Management Studio can’t handle deadlock graphs with zero victims. But that’s just a UI problem.
    2. This “interquery parallelism” kind of deadlock is resolved internally somehow (yay!) but an xml_deadlock_report event is still raised (boo!)

    Update Turns out this is a bit of a known issue. There’s a request you can vote on to add extra info to the xml_deadlock_report that would allow us to filter these non-deadlocks out: Add columns has_victims and is_intra_query_deadlock to the event xml_deadlock_report

    March 12, 2018

    My Trip to the West Coast

    Filed under: SQLServerPedia Syndication — Michael J. Swart @ 11:22 am

    I was at the MVP Global Summit last week in Redmond, Washington. I was excited to go because I hadn’t been to the summit since 2014. Since I was travelling to the west coast anyway, I applied to speak at SQL Saturday Victoria the weekend before and I got accepted. Then, I expanded my trip to include a visit to the D2L office near Vancouver making it a week and a half tour of the West coast.

    The bonus for me is that I got to visit British Columbia. I’ve spent my whole life in Canada, but I’ve never been to British Columbia and I’m so glad I went. That is one good looking province.

    West Coast D2L

    I started my trip by visiting D2L first.

    Everyone single person there was awesome. D2L has a great work culture and the Vancouver office was no different. They treated me like family.

    On the first day, we did a Q&A on anything database related. There were some good questions and I get the feeling that some of the answers were surprising. For example, I don’t advocate for stored procedures over inline SQL.

    At the end of the week, I got to practice my session that I prepared for SQL Saturday Victoria in front of my colleagues. Now that was interesting. I wrote the talk for a general audience. It’s always useful to imagine an audience member when writing a talk. I imagined someone who wanted to learn more about databases and chose to give up their Saturday to do it. This imaginary audience member chose to come to a talk called “100 Percent Online Migrations” and presumably were interested in schema migrations. So when I gave the talk to my D2L colleagues, it was interesting to be able to get into specifics because I know the business challenges we’re all dealing with. The Vancouver office gave awesome feedback and they didn’t heckle me too much.

    Other sights included a pikachu car, a build-is-broken light (just like home!) and an impromptu tour of the best coffee roaster in Richmond, BC

    Before I left for Victoria, I got to spend a little time in downtown Vancouver

    Downtown Vancouver

    Even though I didn’t have a lot of time in downtown Vancouver, I did get to snap a couple photos before the sun went down. Mostly in and around Gastown.

    Gassy Jack, Steam Clock, Vancouver Street

    Another thing I noticed is that no one seems to smoke tobacco any more.

    Before I left, I did manage to find the alley that Bastian ran down when he was chased by bullies in the Neverending Story. But the alley hasn’t aged well. It didn’t look like luck-dragons frequented that place any more so I didn’t take a picture.

    Then it was time to head to SQL Saturday Victoria

    Ferry to Victoria

    I spent Friday on busses and ferries. Long trips in Ontario are dull. If you’ve seen one corn field, you’ve seen them all. But in British Columbia, the passing scenery is beautiful. Here’s a timelapse video of part of the ferry ride to Swartz Bay (not Swart’s bay).

    SQL Saturday Victoria

    SQL Saturday in Victoria was great. I was relaxed, and the talk (I thought) went really really well.

    Randolph tries not to heckle Michael

    I wasn’t super-thrilled with the turn-out. There were about ten people there (five were fellow speakers). And I have an uncomfortably long streak of having someone fall asleep during my talks. I know I’m not the most dynamic speaker but it’s starting to get on my nerves. That streak remains unbroken thanks in part to Erland’s jetlag.

    But it remained a special SQL Saturday for me. I discovered that the venue was Camosun College in Victoria. Not only is Camosun College a client of D2L, but they’ve got a beautiful campus. Fun fact, spring has already arrived on campus there:


    Thanks to Janice, Scott and everyone else who ran a very very successful event.

    On Sunday, Randolph, Mike and I waited in line for breakfast and walked around Victoria until the evening when we took a boat to meet Angela in Seattle, then on to Bellevue just in time for the MVP Global Summit in Redmond!

    The 2018 MVP Global Summit

    It’s been a while since I attended the MVP Summit. And this one was a good one. Everyone behaved for the most part. In past years, I remember a lot of people looking gift horses in the mouth. The feedback from MVPs to Microsoft sometimes took the form of angry complaints. Even if it was about something they didn’t know existed until that day. There was much less of that this year.

    Personally I got to give feedback about some SQL Server features, and I got to hear about new stuff coming up. I’m always very grateful for Microsoft for putting on this event. I returned home with a lot more blue stuff than I started with.

    What does this photobooth do?

    (Special thanks to Scott, Mike, Angela and Randolph for making this trip extra fun. You’re the best)

    January 17, 2018

    SHA1 Collisions in SQL Server

    Takeaway: It’s been frowned on for a while, but SHA1 is definitely broken for security purposes.

    In October of 2010, Michael Coles created a contest on his blog called “Find a Hash Collision, Win $100“. The contest was part of a discussion at the time about whether the SHA1 hash was useful for detecting changes. For what it’s worth, I still think SHA1 is valuable as a consistency check if not for security.

    At the time no SHA1 hash collisions were known, but in 2017, the news broke that some researchers finally generated a collision. So I looked up the research paper and downloaded the files. I used OPENROWSET to get the binary strings and I created my entry for Michael Coles’ contest:

    --  Begin script
    DECLARE @A varbinary(8000),
          @B varbinary(8000),
          @hA binary(20),
          @hB binary(20);
    -- Replace the ? below with binary strings
    SELECT @A = 0x255044462D312E330A25E2E3CFD30A0A0A312030206F626A0A3C3C2F57696474682032203020522F4865696768742033203020522F547970652034203020522F537562747970652035203020522F46696C7465722036203020522F436F6C6F7253706163652037203020522F4C656E6774682038203020522F42697473506572436F6D706F6E656E7420383E3E0A73747265616D0AFFD8FFFE00245348412D3120697320646561642121212121852FEC092339759C39B1A1C63C4C97E1FFFE017F46DC93A6B67E013B029AAA1DB2560B45CA67D688C7F84B8C4C791FE02B3DF614F86DB1690901C56B45C1530AFEDFB76038E972722FE7AD728F0E4904E046C230570FE9D41398ABE12EF5BC942BE33542A4802D98B5D70F2A332EC37FAC3514E74DDC0F2CC1A874CD0C78305A21566461309789606BD0BF3F98CDA8044629A10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000FFFE00FE0000000000000000FFE000104A46494600010101004800480000FFFE00134372656174656420776974682047494D50FFDB00430001010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFDB00430101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFC20011080008000803011100021101031101FFC40014000100000000000000000000000000000008FFC40014010100000000000000000000000000000009FFFE0006FFFE002FFFDA000C03010002100310000001539DC51FFFC4001510010100000000000000000000000000001626FFFE0006FFFE0033FFDA0008010100010502A953FFC4001F1100000309000000000000000000000000141517001316274563658695FFFE0006FFFE0041FFDA0008010301013F019A8AA56D533BB238739E612166B90E605BFFC4001E11000004070000000000000000000000000014151713274563658594FFFE0006FFFE0040FFDA0008010201013F01984E1555C155B66CDC3E04A21A444C40FFC4001E10000101090000000000000000000000001413001215164462648594FFFE0006FFFE0033FFDA0008010100063F02AD9A4DB175DCE6086D743B05BFFFC40014100100000000000000000000000000000000FFFE0006FFFE0012FFDA0008010100013F216001FFFE0006FFFE002BFFDA000C030100020003000000101FFFC40014110100000000000000000000000000000000FFFE0006FFFE0028FFDA0008010301013F106980FFC40014110100000000000000000000000000000000FFFE0006FFFE0028FFDA0008010201013F106BC7FFC40014100100000000000000000000000000000000FFFE0006FFFE0014FFDA0008010100013F10153FFFD9414E4745FFE000104A46494600010101004800480000FFFE00134372656174656420776974682047494D50FFDB00430001010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFDB00430101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFC20011080008000803011100021101031101FFC40014000100000000000000000000000000000009FFC4001501010100000000000000000000000000000607FFDA000C03010002100310000001524A5FFF00FFC40014100100000000000000000000000000000000FFDA00080101000105027FFFC40018110002030000000000000000000000000000F037A7B7FFDA0008010301013F01DFDBFD9FCFFFC40018110002030000000000000000000000000000F036A6B6FFDA0008010201013F01CA3546287FFFC40018100002030000000000000000000000000000F03767A7FFDA0008010100063F02A99C99898FFFC40014100100000000000000000000000000000000FFDA0008010100013F217FFFDA000C030100020003000000101FFFC40014110100000000000000000000000000000000FFDA0008010301013F100FFFC40014110100000000000000000000000000000000FFDA0008010201013F100FFFC40014100100000000000000000000000000000000FFDA0008010100013F100FFFD90A656E6473747265616D0A656E646F626A0A0A322030206F626A0A380A656E646F626A0A0A332030206F626A0A380A656E646F626A0A0A342030206F626A0A2F584F626A6563740A656E646F626A0A0A352030206F626A0A2F496D6167650A656E646F626A0A0A362030206F626A0A2F4443544465636F64650A656E646F626A0A0A372030206F626A0A2F4465766963655247420A656E646F626A0A0A382030206F626A0A313639330A656E646F626A0A0A392030206F626A0A3C3C0A20202F54797065202F436174616C6F670A20202F5061676573203130203020520A3E3E0A656E646F626A0A0A0A31302030206F626A0A3C3C0A20202F54797065202F50616765730A20202F436F756E7420310A20202F4B696473205B3131203020525D0A3E3E0A656E646F626A0A0A31312030206F626A0A3C3C0A20202F54797065202F506167650A20202F506172656E74203130203020520A20202F4D65646961426F78205B302030203820385D0A20202F43726F70426F78205B302030203820385D0A20202F436F6E74656E7473203132203020520A20202F5265736F75726365730A20203C3C0A202020202F584F626A656374203C3C2F496D302031203020523E3E0A20203E3E0A3E3E0A656E646F626A0A0A31322030206F626A0A3C3C2F4C656E6774682033303E3E0A73747265616D0A710A2020382030203020382030203020636D0A20202F496D3020446F0A510A656E6473747265616D0A656E646F626A0A0A0A0A787265660A30203133200A303030303030303030302036353533352066200A30303030303030303137203030303030206E200A30303030303031383631203030303030206E200A30303030303031383739203030303030206E200A30303030303031383937203030303030206E200A30303030303031393232203030303030206E200A30303030303031393435203030303030206E200A30303030303031393732203030303030206E200A30303030303031393939203030303030206E200A30303030303032303230203030303030206E200A30303030303032303736203030303030206E200A30303030303032313432203030303030206E200A30303030303032333039203030303030206E200A0A747261696C6572203C3C202F526F6F74203920302052202F53697A652031333E3E0A0A7374617274787265660A323339310A2525454F460A,
           @B = 0x255044462D312E330A25E2E3CFD30A0A0A312030206F626A0A3C3C2F57696474682032203020522F4865696768742033203020522F547970652034203020522F537562747970652035203020522F46696C7465722036203020522F436F6C6F7253706163652037203020522F4C656E6774682038203020522F42697473506572436F6D706F6E656E7420383E3E0A73747265616D0AFFD8FFFE00245348412D3120697320646561642121212121852FEC092339759C39B1A1C63C4C97E1FFFE017346DC9166B67E118F029AB621B2560FF9CA67CCA8C7F85BA84C79030C2B3DE218F86DB3A90901D5DF45C14F26FEDFB3DC38E96AC22FE7BD728F0E45BCE046D23C570FEB141398BB552EF5A0A82BE331FEA48037B8B5D71F0E332EDF93AC3500EB4DDC0DECC1A864790C782C76215660DD309791D06BD0AF3F98CDA4BC4629B10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000FFFE00FE0000000000000000FFE000104A46494600010101004800480000FFFE00134372656174656420776974682047494D50FFDB00430001010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFDB00430101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFC20011080008000803011100021101031101FFC40014000100000000000000000000000000000008FFC40014010100000000000000000000000000000009FFFE0006FFFE002FFFDA000C03010002100310000001539DC51FFFC4001510010100000000000000000000000000001626FFFE0006FFFE0033FFDA0008010100010502A953FFC4001F1100000309000000000000000000000000141517001316274563658695FFFE0006FFFE0041FFDA0008010301013F019A8AA56D533BB238739E612166B90E605BFFC4001E11000004070000000000000000000000000014151713274563658594FFFE0006FFFE0040FFDA0008010201013F01984E1555C155B66CDC3E04A21A444C40FFC4001E10000101090000000000000000000000001413001215164462648594FFFE0006FFFE0033FFDA0008010100063F02AD9A4DB175DCE6086D743B05BFFFC40014100100000000000000000000000000000000FFFE0006FFFE0012FFDA0008010100013F216001FFFE0006FFFE002BFFDA000C030100020003000000101FFFC40014110100000000000000000000000000000000FFFE0006FFFE0028FFDA0008010301013F106980FFC40014110100000000000000000000000000000000FFFE0006FFFE0028FFDA0008010201013F106BC7FFC40014100100000000000000000000000000000000FFFE0006FFFE0014FFDA0008010100013F10153FFFD9414E4745FFE000104A46494600010101004800480000FFFE00134372656174656420776974682047494D50FFDB00430001010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFDB00430101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101FFC20011080008000803011100021101031101FFC40014000100000000000000000000000000000009FFC4001501010100000000000000000000000000000607FFDA000C03010002100310000001524A5FFF00FFC40014100100000000000000000000000000000000FFDA00080101000105027FFFC40018110002030000000000000000000000000000F037A7B7FFDA0008010301013F01DFDBFD9FCFFFC40018110002030000000000000000000000000000F036A6B6FFDA0008010201013F01CA3546287FFFC40018100002030000000000000000000000000000F03767A7FFDA0008010100063F02A99C99898FFFC40014100100000000000000000000000000000000FFDA0008010100013F217FFFDA000C030100020003000000101FFFC40014110100000000000000000000000000000000FFDA0008010301013F100FFFC40014110100000000000000000000000000000000FFDA0008010201013F100FFFC40014100100000000000000000000000000000000FFDA0008010100013F100FFFD90A656E6473747265616D0A656E646F626A0A0A322030206F626A0A380A656E646F626A0A0A332030206F626A0A380A656E646F626A0A0A342030206F626A0A2F584F626A6563740A656E646F626A0A0A352030206F626A0A2F496D6167650A656E646F626A0A0A362030206F626A0A2F4443544465636F64650A656E646F626A0A0A372030206F626A0A2F4465766963655247420A656E646F626A0A0A382030206F626A0A313639330A656E646F626A0A0A392030206F626A0A3C3C0A20202F54797065202F436174616C6F670A20202F5061676573203130203020520A3E3E0A656E646F626A0A0A0A31302030206F626A0A3C3C0A20202F54797065202F50616765730A20202F436F756E7420310A20202F4B696473205B3131203020525D0A3E3E0A656E646F626A0A0A31312030206F626A0A3C3C0A20202F54797065202F506167650A20202F506172656E74203130203020520A20202F4D65646961426F78205B302030203820385D0A20202F43726F70426F78205B302030203820385D0A20202F436F6E74656E7473203132203020520A20202F5265736F75726365730A20203C3C0A202020202F584F626A656374203C3C2F496D302031203020523E3E0A20203E3E0A3E3E0A656E646F626A0A0A31322030206F626A0A3C3C2F4C656E6774682033303E3E0A73747265616D0A710A2020382030203020382030203020636D0A20202F496D3020446F0A510A656E6473747265616D0A656E646F626A0A0A0A0A787265660A30203133200A303030303030303030302036353533352066200A30303030303030303137203030303030206E200A30303030303031383631203030303030206E200A30303030303031383739203030303030206E200A30303030303031383937203030303030206E200A30303030303031393232203030303030206E200A30303030303031393435203030303030206E200A30303030303031393732203030303030206E200A30303030303031393939203030303030206E200A30303030303032303230203030303030206E200A30303030303032303736203030303030206E200A30303030303032313432203030303030206E200A30303030303032333039203030303030206E200A0A747261696C6572203C3C202F526F6F74203920302052202F53697A652031333E3E0A0A7374617274787265660A323339310A2525454F460A;
          @hB = HASHBYTES('SHA1', @B);
                      THEN '@A Equals @B'
                      ELSE '@A Is Not Equal To @B'
                      END AS AB_Equal,
                CASE WHEN @hA = @hB
                      THEN '@hA Equals @hB'
                      ELSE '@hA Is Not Equal To @hB'
                      END AS Hash_Equal;
    -- End script

    This gives me the output that wins the contest:

    Unfortunately upon closer inspection, I see that the rules of the contest say that entries must be received prior to midnight U.S. Eastern Standard Time on October 31, 2010.

    Rats, 7 years too late!

    January 15, 2018

    100 Percent Online Deployments: Stage and Switch

    Filed under: Miscelleaneous SQL,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 12:42 pm
    100 Percent Online Deployments
    How to deploy schema changes without scheduled downtime

    In the first draft of this series, this post didn’t exist. I wanted to show a really simple example of a column switch and include it in the Blue-Green (Details) post. I planned for something simple. But I ran into some hiccups that I though were pretty instructive, so I turned it into the post you see here.

    The Plan

    For this demo, I wanted to use the WideWorldImporters database. In table Warehouse.ColdRoomTemperatures I wanted to change the column

    ColdRoomSensorNumber INT NOT NULL,


    ColdRoomSensorLabel NVARCHAR(100) NOT NULL,

    because maybe we want to track sensors via some serial number or other code.

    The Blue-Green plan would be simple:

    The Trouble

    But nothing is ever easy. Even SQL Server Data Tools (SSDT) gives up when I ask it to do this change with this error dialog:

    Never Easy

    There’s two things going on here (and one hidden thing):

    1. The first two messages point out that a procedure is referencing the column ColdRoomSensorNumber with schemabinding. The reason it’s using schemabinding is because it’s a natively compiled stored procedure. And that tells me that the table Warehouse.ColdRoomTemperatures is an In-Memory table. That’s not all. I noticed another wrinkle. The procedure takes a table-valued parameter whose table type contains a column called ColdRoomSensorLabel. We’re going to have to replace that too. Ugh. Part of me wanted to look for another example.
    2. The last message tells me that the table is a system versioned table. So there’s a corresponding archive table where history is maintained. That has to be dealt with too. Luckily Microsoft has a great article on Changing the Schema of a System-Versioned Temporal Table.
    3. One last thing to worry about is a index on ColdRoomSensorNumber. That should be replaced with an index on ColdRoomSensorLabel. SSDT didn’t warn me about that because apparently, it can deal with that pretty nicely.

    So now my plan becomes:

    Blue The original schema

    Aqua After the pre-migration scripts are run

    An extra step is required here to update the new column and keep the new and old columns in sync.

    Green After the switch, we clean up the old objects and our schema change is finished:

    Without further ado, here are the scripts:

    Pre-Migration (Add Green Objects)

    In the following scripts, I’ve omitted the IF EXISTS checks for clarity.

    -- Add the four green objects
    ALTER TABLE Warehouse.ColdRoomTemperatures
    ADD ColdRoomSensorLabel NVARCHAR(100) NOT NULL 
        CONSTRAINT DF_Warehouse_ColdRoomTemperatures_ColdRoomSensorLabel DEFAULT '';
    ALTER TABLE Warehouse.ColdRoomTemperatures
    ADD INDEX IX_Warehouse_ColdRoomTemperatures_ColdRoomSensorLabel (ColdRoomSensorLabel);
    CREATE TYPE Website.SensorDataList_v2 AS TABLE(
        SensorDataListID int IDENTITY(1,1) NOT NULL,
        ColdRoomSensorLabel VARCHAR(100) NULL,
        RecordedWhen datetime2(7) NULL,
        Temperature decimal(18, 2) NULL,
    CREATE PROCEDURE Website.RecordColdRoomTemperatures_v2
        @SensorReadings Website.SensorDataList_v2 READONLY
        --straight-forward definition left as exercise for reader

    Pre-Migration (Populate and Keep in Sync)

    Normally, I would use triggers to keep the new and old column values in sync like this, but you can’t do that with In-Memory tables. So I altered the procedure Website.RecordColdRoomTemperatures to achieve something similar. The only alteration I made is to set the ColdRoomSensorLabel value in the INSERT statement:

    ALTER PROCEDURE Website.RecordColdRoomTemperatures
    @SensorReadings Website.SensorDataList READONLY
        LANGUAGE = N'English'
        BEGIN TRY
            DECLARE @NumberOfReadings int = (SELECT MAX(SensorDataListID) FROM @SensorReadings);
            DECLARE @Counter int = (SELECT MIN(SensorDataListID) FROM @SensorReadings);
            DECLARE @ColdRoomSensorNumber int;
            DECLARE @RecordedWhen datetime2(7);
            DECLARE @Temperature decimal(18,2);
            -- note that we cannot use a merge here because multiple readings might exist for each sensor
            WHILE @Counter <= @NumberOfReadings
                SELECT @ColdRoomSensorNumber = ColdRoomSensorNumber,
                       @RecordedWhen = RecordedWhen,
                       @Temperature = Temperature
                FROM @SensorReadings
                WHERE SensorDataListID = @Counter;
                UPDATE Warehouse.ColdRoomTemperatures
                    SET RecordedWhen = @RecordedWhen,
                        Temperature = @Temperature
                WHERE ColdRoomSensorNumber = @ColdRoomSensorNumber;
                IF @@ROWCOUNT = 0
                    INSERT Warehouse.ColdRoomTemperatures
                        (ColdRoomSensorNumber, ColdRoomSensorLabel, RecordedWhen, Temperature)
                    VALUES (@ColdRoomSensorNumber, 
                            'HQ-' + CAST(@ColdRoomSensorNumber AS NVARCHAR(50)), 
                SET @Counter += 1;
        END TRY
            THROW 51000, N'Unable to apply the sensor data', 2;
            RETURN 1;
        END CATCH;

    That keeps the values in sync for new rows. But now it’s time to update the values for existing rows. In my example, I imagine that the initial label for the sensors are initially: “HQ-1”, “HQ-2”, etc…

    UPDATE Warehouse.ColdRoomTemperatures
    SET ColdRoomSensorLabel = 'HQ-' + CAST(ColdRoomSensorNumber as nvarchar(50));

    Eagle-eyed readers will notice that I haven’t dealt with the history table here. If the history table is large use batching to update it. Or better yet, turn off system versioning and then turn it back on immediately using a new/empty history table (if feasible).


    After a successful switch, the green application is only calling Website.RecordColdRoomTemperatures_v2. It’s time now to clean up. Again, remember that order matters.

    DROP PROCEDURE Website.RecordColdRoomTemperatures;
    DROP TYPE Website.SensorDataList;
    ALTER TABLE Warehouse.ColdRoomTemperatures
    DROP INDEX IX_Warehouse_ColdRoomTemperatures_ColdRoomSensorNumber;
    ALTER TABLE Warehouse.ColdRoomTemperatures
    DROP COLUMN ColdRoomSensorNumber;
    « Newer PostsOlder Posts »

    Powered by WordPress