Michael J. Swart

Careful Hashing

Michael J. Swart — Wed, 15 Feb 2012 17:00:28 +0000

Cryptographic hashing functions like MD5 or SHA-1 are examples of one-way functions that take any data you give it and return a fixed set of bytes (for example, 16 bytes for MD5 and 20 bytes for SHA-1).

I recently had trouble coming up with the same hash value for what I thought were the exact same inputs. Compare the following two examples:

// C# example
byte[] byteArray = Encoding.Unicode.GetBytes("I'm a lumberjack and I'm okay.");
 
SHA1 sha = new SHA1CryptoServiceProvider();
byte[] hashedPasswordBytes = sha.ComputeHash(byteArray);
 
Console.WriteLine(BitConverter.ToString(hashedPasswordBytes));
//30-F1-CE-3E-7E-1F-17-27-A7-B5-97-39-44-D0-FE-EA-DA-69-53-2B

That’s different than this:

-- SQL example
select HASHBYTES('SHA1', 'I''m a lumberjack and I''m okay.')
--0x4D6936CEDD0DE794AC86F9A4099DCBB4EFED8E1F

So what gives? Before reading any further, can you spot the problem? In both examples above, two things are happening. First a string of characters is being converted into a string of bytes and then that string of bytes gets hashed. I’ve learned that it’s not the hash function that’s the problem, it’s converting the character string into binary that is inconsistent.

String Conversions

Maybe that’s not surprising, but still, it’s one more thing to be careful about: converting strings to bytes. SQL Server’s wide characters (NCHARs and NVARCHARs) are equivalent to .Net’s Encoding.Unicode and SQL Server’s single non-unicode characters (CHAR and VARCHAR) (single-byte characters) are equivalent to .net’s Encoding.ASCII.

So for completeness sake, these should be equivalent:

-- wide characters (remember unicode string literals are prefixed with 'N')
select HASHBYTES('SHA1', N'I''m a lumberjack and I''m okay.')
--0x30F1CE3E7E1F1727A7B5973944D0FEEADA69532B
 
-- non-unicode characters:
select HASHBYTES('SHA1', 'I''m a lumberjack and I''m okay.')
--0x4D6936CEDD0DE794AC86F9A4099DCBB4EFED8E1F

And

// Unicode
byte[] byteArray = Encoding.Unicode.GetBytes("I'm a lumberjack and I'm okay.");
SHA1 sha = new SHA1CryptoServiceProvider();
byte[] hashedPasswordBytes = sha.ComputeHash(byteArray);
Console.WriteLine(BitConverter.ToString(hashedPasswordBytes));
//30-F1-CE-3E-7E-1F-17-27-A7-B5-97-39-44-D0-FE-EA-DA-69-53-2B
 
//ASCII
byteArray = Encoding.ASCII.GetBytes("I'm a lumberjack and I'm okay.");
hashedPasswordBytes = sha.ComputeHash(byteArray);
Console.WriteLine(BitConverter.ToString(hashedPasswordBytes));
//4D-69-36-CE-DD-0D-E7-94-AC-86-F9-A4-09-9D-CB-B4-EF-ED-8E-1F

This means that if you’re going to hash strings in SQL Server and compare them to strings that are hashed in C#, you want to use

Encoding.Unicode for NCHARS and NVARCHARS
Encoding.ASCII for CHARS and VARCHARS

This also means that for this use case, I wouldn’t feel safe using other encodings (such as Encoding.UTF8) because SQL Server can’t duplicate that string to binary conversion.

Caveats

Don’t trust me and test it out for yourself!
Collations matter a little (sort order is not relevant here). The code page for your SQL Server collation should match the code page of the .net encoding when dealing with single byte characters. Unicode strings should not care about collation.
If you do find any counter-examples for anything I’ve written here, let me know in the comments.

Microsoft Model Databases: Some Unconventional Alternatives?

Michael J. Swart — Fri, 10 Feb 2012 01:53:12 +0000

It seems like Microsoft has always provided example databases in order to illustrate database concepts.

Pubs: Back when SQL Server 2000 was new, Microsoft provided this sample database and maybe you remember it. Pubs was a really simple database that was designed to represent data that a fictional book publishing company might use. The database had a name (pubs) but I don’t think anyone ever gave the company a name. From here we’re told that it was “used to demonstrate many of the options available” for SQL Server.

Northwind also was available for SQL Server 2000. It was a database that was used for a fictitious company “Northwind Traders”. The company “imports and exports specialty foods from around the world.” It was better than pubs, but that wasn’t saying much. Pubs set a pretty low bar.

AdventureWorks: The Adventureworks database came along with SQL Server 2005 and we were introduced to the bicycle manufacturer Adventure Works Cycle. The database was pretty comprehensive and it touched on a lot more features of SQL Server. I’ve seen countless performance demos that query Sales.SalesOrderDetail. There’s even an Adventureworks Data Warehouse, an Adventureworks OLAP cube and even a light version: AdventureworksLT.

Contoso And lately there’s a new sample database Contoso. It’s another fictional retail company (Contoso Ltd. This time they’re selling electronics and appliances). This database is used for Business Intelligence demos. With Contoso, Microsoft has developed and made a data warehouse and an SSAS cube available.

Presumably, each of these databases is an example of Microsoft putting their best foot forward. Or at least it’s a database which enables them to show off what their database software can do.

Beyond Demos

So those databases are good for demos.

But if we’re looking for other Microsoft-designed databases, we don’t have to stop there. There’s other ones that we can look at. I’m talking about the databases that Microsoft ships along with their products. When looking at these, remember:

There are some good examples, …
We can look at those databases and see how they’re constructed. If you see some interesting db designs, you can use that.
… and there are some not so good examples …
These databases weren’t built as database role models. So there’s bound to be a lot of database design choices that are not ideal. I know that Microsoft has huge pressure to maintain backwards compatibility.
… and that’s okay.
It’s actually a fun exercise to pick out the good designs from the bad. It’s like a smorgasbord, you take the lessons you like and leave the rest.

Some examples of what I’m talking about.

ReportServer

Install and configure an instance of SQL Server Reporting Services. Take a look at the ReportServer. It’s a pretty nice database schema. One thing that was interesting was the choice of identity column data types:

Catalog seems to be one of the main tables here and they’re using uniqueidentifiers (GUIDs) for identity columns. Normally people scoff at that, but I think it makes sense in this case. Catalog is the table that tracks reports, datasources and folder structures. Even at it’s largest, it can’t be that big (I just checked my own installation and the clustered index fits on 1 extent). That means that all the worries about GUIDs for primary keys are mitigated and the choice of using a globally unique identifier might make sense here.
ExecutionLogStorage Contrast that with ReportServer’s largest table, ExecutionLogStorage. Depending on how busy your Reporting Services installation is, this table can grow very large with one row for every report view. When I look at the choice for their primary key, I see that it’s clustered index on a single BIGINT column defined as it’s IDENTITY column. Well done.

Any Database’s Metadata

I like the object oriented influence here:

The relationship between sys.objects –> sys.tables and between sys.objects –> sys.procedures is the subtype relationship class –> subclass. And that shows in the schema, for example, there’s one sys.object row for every sys.table row.
The relationship between sys.foreign_keys –> and sys.foreign_key_columns is the object composition relationship. And that also shows in the schema too. The “primary key” for the table foreign_key_columns is (constraint_object_id, constraint_column_id) and it contains the primary key of its owner.

Another good example is the sys.index_columns table. Look at the schema diagram below. It’s a perfect instance of my favourite DB Design example

Any Sharepoint database

[Sigh] These are mostly counter-examples here, I took a brief glance at the schema once and decided quickly that there are no good lessons. Can anyone tell me if that’s changed?

Removing Duplicate Dimension Rows in SSIS

Michael J. Swart — Thu, 02 Feb 2012 09:00:39 +0000

Something New For Me

I’m relatively new to SQL Server Integration Services (SSIS). And so I want to explain how I solved a problem and invite Business Intelligence (BI) experts (and wannabe experts) to comment or point out things that I missed.

A Short Description of the Problem

So here’s a short description of the problem I’m trying to solve. It involves populating a dimension table. I want a data flow transformation which removes duplicate key values. In other words, every time I come across a new key value for a dimension, I want to pass through that entire row (not just the key values). This is kind of like a cross between T-SQL’s DISTINCT and FIRST_VALUE.

I found out recently that people have treated this topic before:

But because of my bad luck, I only found these posts after I muddled through a solution. But I was surprised to find out my solution had some interesting differences.

The Setup

First I’ll explain my situation. Say I’ve got a flat file containing data about Prime Ministers:

The source data

My main goal is to use this data to populate a data warehouse. That data warehouse has a schema like this:

The Data Warehouse Schema

This example is just for illustration. In real life, the number of fact rows is very large, and the number of dimension rows is relatively small. The time dimension will be pre-populated, but I want to load the other dimensions using values from the flat file. So that means I plan to process the data twice: The first time will be to load these dimensions, and the second time will be to load the fact table. When I process the data for dimensions, I :

Use a lookup transform to find whether that dimension already exists in the warehouse
Use the No Match Output to identify dimension rows I haven’t seen before. Those rows will be used to load the dimension. (i.e. I’m treating the dimension like a Never-Changing-Dimension rather than a Slowly Changing Dimension.)

The problem is that these unmatched rows can be duplicated and I don’t want those duplicate dimensions rows getting into the dimension table.

Things I considered:

I considered using a staging table on the DB side. But that required too much space, the amount of space needed was comparable to the fact table, not the dimension table.
I also considered inserting into dimensions a row-at-a-time using the MERGE statement on the database side. But I didn’t consider that for long. The performance with that strategy was so unacceptable.
I could have used a sort/aggregate, but that was a blocking transformation and I really didn’t need the data sorted.

What I ended up with

I created a script transform which seems to do the trick for what I want. My Data Flow task looks like this:

In those script transforms you see up there, I told SSIS that each script will be used to filter rows out or let them pass through. I did that by setting ExclusionGroup to 1 (by default, it’s normally 0). Here’s where that’s done:

Then in the script, I created a list of seen dimensions. And I let rows pass through if they haven’t yet been seen. The script simply looks like this:

public override void Input0_ProcessInputRow(Input0Buffer Row)
{
 
    if (Row.Party_IsNull)
        return;
 
    if (!seenParty.Contains(Row.Party))
    {
        seenParty.Add(Row.Party);
        Row.DirectRowToOutput0();
    }
 
}

Logically it behaves exactly the same as the sort transformation (as explained in “Dead Easy Duplicate Key Removal” by Todd McDermid). Except for these things:

It seems to perform fast
It’s not a blocking transformation
But you don’t get all that for nothing. What we’re not getting with this method is that the rows aren’t sorted. In most cases, that doesn’t matter.
It takes some memory, so you have to be aware of that. Ask yourself, how well can all unseen dimension keys fit in memory?

Some questions for you BI guys

And here’s where I’d like you to chime in.

What’s the most common approach to this problem?
I get the feeling that I could use the Cache Transform for this technique. But I haven’t tried it.
Are there any new features in SQL 2012 that help?
Am I re-inventing the wheel here?
Am I missing anything else that should be obvious?

Rebuild Your Indexes Online (When You Can)

Michael J. Swart — Wed, 25 Jan 2012 17:00:41 +0000

So I was recently reading SQL Server’s blog, specifically the article Customers using SQL Server 2012 today! and some brave businesses are already using and enjoying some of SQL Server 2012 features that help their databases stay available. I’m excited about the new features too. For example, index rebuilds have been improved. We can now rebuild indexes online that include “blob” columns (like nvarchar(max), image, etc…). This means that (almost) every index can be rebuilt without requiring a table lock for the entire operation. And that’s good news for availability!

This wasn’t the case in earlier versions. In earlier versions, you couldn’t rebuild an index online if it included columns that were large strings. If you tried, you would get this message:

/*
An online operation cannot be performed for index 'ix_t1_1' because the 
index contains column '' of data type text, ntext, image, varchar(max), 
nvarchar(max), varbinary(max), xml, or large CLR type. For a non-clustered 
index, the column could be an include column of the index. For a clustered 
index, the column could be any column of the table. If DROP_EXISTING is 
used, the column could be part of a new or old index. The operation must be 
performed offline. 
*/

Logic to determine when ONLINE=ON is supported

So that means there’s one more thing to check when finding out whether you can rebuild an index on-line. The logic now goes like this. You can rebuild an index online if:

You’re using Enterprise Edition or higher
And you’re using SQL Server 2012 or later
Or you’re using SQL Server 2008 or earlier
And your index is clustered and the table contains no blob columns
Or your index is non-clustered and the index includes no blob columns

Hmm… You can’t really put parentheses in a bullet point list. Here try this flow chart:

Clear as mud

Hmm… that really doesn’t clear things up too well either. And I haven’t even mentioned partitioned indexes.

Just Try It

Maybe you’re like me, you may have to deal with multiple versions, multiple editions, and multiple tables and their indexes. Instead of wading through that logic above, just TRY it and let SQL Server figure it out. Here’s a sproc which takes a table name and index name and tries to rebuild the index online. If it can’t, it builds it offline.

create procedure s_TryRebuildOnlineOtherwiseOffline
(
	@schema sysname = 'dbo',
	@tablename sysname,
	@indexname sysname
)
as
begin
	set @schema = QUOTENAME(@schema);
	set @tablename = QUOTENAME(@tablename);
	set @indexname = QUOTENAME(@indexname);
 
	declare @sqlRebuild nvarchar(max)
	set @sqlRebuild = N'ALTER INDEX ' + @indexname + ' ON ' + @schema + '.' + @tablename + ' REBUILD';
	declare @sqlRebuildOnline nvarchar(max)
	set @sqlRebuildOnline = @sqlRebuild + ' WITH (ONLINE=ON)';
 
	begin try
		EXEC sp_executesql @sqlRebuildOnline;
		print @sqlRebuildOnline;
	end try
	begin catch
		EXEC sp_executesql @sqlRebuild;
		print @sqlRebuild;
	end catch
end
go

Alternatively, you can adjust the script to REORGANIZE an index when it can’t rebuild it online.

By the way, if you use Ola Hallengren’s maintenance scripts, he’s already taken all this logic into account! If you write your own maintenance scripts, feel free to incorporate what I have here.

My favourite search terms

Michael J. Swart — Wed, 18 Jan 2012 17:00:06 +0000

So I recently looked through my web statistics and I wanted to share some of my favourite search terms. These are words or phrases that people have searched for and for good or bad, they've wound up here on my site. Google Analytics helps me browse these search terms and looking through them I learned that

Michael is apparently hard to spell
Some people still include question marks in their search queries. It's quaint. I always assume they're asking "Jeeves"

Any way here are my favourites, in no particular order.

dba humour
trololo
trololo guy
monkey throwing darts
people running from atomic bomb

swart guts

???

is read uncommitted bad?

Yes

rid lookup good or bad

Bad

sql undelete

Nope, sorry about that. Although

how to forget something

Someone googled that. There's a story there.

reporting services is fun

you betcha

10 pockets utility belt

Career in construction or Batman wannabe?

vampire hierarchy

Sorry, you've come to the wrong place, random googler.

cartoon cow tossing dog

(strangely enough) You've come to the right place, random googler.

how to avoid swart

HAHAHA! If you find out, let me know.

my software never has bugs. it just develops random features
and
pivot tables are like good wine you need to learn how to appreciate them

2 things: (1) How did my site come up for these searches and (2) Can we be best friends?

By the way, the phrase "You've come to the right place" reminds me of Engywook, the toothless scientist from the movie The Neverending Story. He's an expert on the Southern Oracle (it's his speciality). So I include him here in the hopes that I get at least one ironic google hit for "Oracle expert":

the (Southern) Oracle Expert

Clearing out the closet…

Michael J. Swart — Tue, 17 Jan 2012 19:31:15 +0000

Hey fellow SQL bloggers,

Have you ever written an amazing blog post that just didn’t pan out because when you reread what you wrote, you found out it was crap? That happens to me sometimes and it’s frustrating. Here are some illustrations that I’ve created in the past that just never made the cut. So I’ll just leave these here.

Clean Your Data

SQLHomies

Just slightly hipper than SQLPeople.net

Love Your Data

Inspired by Karen Lopez’s favourite quote (Maybe this should have gone before the first picture).

Brent Ozar PLF

I admit it, I’m a Brent OPLF fanboy. And once upon a time, Jeremiah Peschka’s and his beard reminded me of Yukon Cornelius:

Thank you notes…

Michael J. Swart — Fri, 13 Jan 2012 17:00:56 +0000

(It’s been a hectic week. The good news is that I’ve got loads of topics to write about. The bad news is that I’ve got no time to do it! Illustrations will make a return next week too because I’m finally getting that Adobe Illustrator license I’ve had my eye on. Maybe I’ll also include a retroactive Jimmy Fallon illustration).

Aaron Bertrand maintains the definitive list of free SQL Server tools. I want to give a shout out to just a couple of them. Without these tools, this week would have been a lot rougher at work for our team.

sp_whoisactive

Thank you Adam Machanic,
For making a lightweight sproc to help me see at a glance the activity on a SQL Server database. (Also kudos to the parameters @get_plans and @find_block_leaders).

If you’re unfamiliar with this stored procedure, start here.

Plan Explorer

Thank you SQL Sentry,
For making a tool that helps me understand everything about a query plan is doing. (Also kudos to the Expressions tab and the Parameters tab).

Plan Explorer presents SQL Server query plans just a bit nicer than SQL Server Management Studio does. When looking at a problematic query plan, I want to go from looking at a plan to a solution as quick as possible. For me this week, it would be safe to say that Plan Explorer cut that analysis time in half.

If you’re unfamiliar with this tool, start here.

Dynamic Management Views

Thank you Microsoft (SQL-Server-2005-and-later-DMVs),
For tracking statistics about query executions and making it possible to examine the history of the db activity without having to trace server activity (so extra kudos to dm_exec_query_stats, dm_exec_cached_plans and dm_exec_query_plan).

I don’t know whether to call these views free. They’re available with any database at SQL Server 2005 (or later). So if you’re using SQL Server, then you can use these DMVs.

If you’re unfamiliar with these views, start here.

Then, Now and Later

Michael J. Swart — Wed, 04 Jan 2012 17:00:37 +0000

So happy New Year! This is the week that a lot of people look back on their year and make plans for their new year. Well, I’m no different. This week I’m taking a break from the technical content to take a look at 2011 and what 2012 might bring.

The Past (Looking back at 2011)

I started blogging a couple years ago. I picked the domain MichaelJSwart.com (with the middle initial) because MichaelSwart.com was taken at the time. I liked the sound of it any way, probably because of Michael J. Fox. My favorite character of his was Marty McFly:

And a year ago, (prompted by Jenn McCown’s t-sql tuesday post about resolutions) I made a new year’s resolution to blog weekly, keep the articles technical and to include illustrations.

And I had an absolute blast doing it! So it was easy to keep it up:

The topics came from whatever interested me. Luckily I have an interesting job with new and different challenges almost daily.
The time came as by treating the blog as a hobby. I figure the average person has that much free time, it’s just a matter of how to spend it.
I like to write when I’m feeling particularly awake.
I like to draw when I’m not because it’s more fun and it probably uses a different part of the brain so that it feels like less mental effort.

I particpated in a couple SQL Saturdays. The first one in Cleveland was great. Among other things, I learned that if you visit Cleveland’s art museum at 10 am on Superbowl Sunday, you have the entire museum to yourself. The second SQL Saturday I attended this year was in my own backyard. I presented for the first time in Toronto and it was exciting.

The Present (this week)

So I was excited to learn this week that I received Microsoft’s Most Valuable Professional (MVP) award for SQL Server (achievement unlocked!). I feel honoured to receive the award and inspired to continue writing. I feel like I’ve learned and benefited from the SQL crowd more than I provided (expertise consumer first, expertise provider second). So many people have helped me but I’m not going to name names. That’s next week – seriously .

The Future (2012)

So I’m going to resolve to continue blogging the way I have for the past year. It’s a challenging (if not a surprising) resolution. It’s challenging even though this resolution boils down to resolving to continue a fun hobby. This starts now but I’m going to hold off on the technical stuff for just one more week: I want to write something about people who inspire me (and why they might inspire you too). Stay tuned.

And also this year, I’m going to try something new. I want to try to participate in a local user group. Easier said than done because for me, there is no local user group and I’m going to do something about that. (Ohioans, I love you but the commute is killer )

It’s all very exciting. Look out 2012, you won’t know what hit you.

A simple message: Merry Christmas

Michael J. Swart — Thu, 22 Dec 2011 15:00:49 +0000

From my family to yours, Merry Christmas.

And just to throw in some SQL, here is a query that will return the complete lyrics to Feliz Navidad, (the most repetitive Christmas song in the world):

select [processing-instruction(complete_lyrics)] = REPLICATE(REPLICATE(REPLICATE('
Feliz Navidad, ', 3) + 'próspero año y felicidad', 2) + REPLICATE(REPLICATE('
I wanna wish you a Merry Christmas, ', 3) + 'from the bottom of my heart', 2),3)
for xml path(''),type

Write Better

Michael J. Swart — Wed, 21 Dec 2011 04:20:14 +0000

I have a confession to make. I suck at writing. In high school, I was never at the top of my English class and my University accepted my application despite my English marks. But even though I majored in Math and Computer Science, my essay-writing days weren’t over. The University I went to required that all students “demonstrate a proficiency in English” before graduating. To demonstrate that, we were required to write an exam, a single essay. I thought I could get by because English is my first language. So I was surprised when I found out that I failed that exam! Ugh…

Then I made a choice which in hindsight turned out to be one of the best things I could have done. In my second year, I signed up for an English course as an elective. It was an introduction to essay writing. I worked hard and did well. In that course I learned a few things I should have learned many years earlier. Those things can be boiled down into:

Have something to write
Write it with the reader in mind
Don’t write anything else

Have something to write

Or in other words have a point. I’m going to repeat that because it’s a lesson I find myself relearning often: Have a point. I need to have something to write more than I need to write something (if that makes sense).

Corollary for bloggers: Don’t feel guilty about writer’s block.

Write it with the reader in mind

If I’m writing a blog article, I try to ask myself “who’s the reader?” Some common readers include these people:

A keen SQL professional googling for a solution. I love writing posts for this person. It usually starts with myself googling for a problem and not finding anything (or being disappointed with what I do find). I like to think that I’m helping people in the same situation I was in. (Examples: Searching Inside Strings: CPU is Eight Times Worse For Unicode Strings, Eliminated Null Values)
Myself: I used to write a lot of articles for myself. They were quick scripts that I could quickly get access to as long as I had internet access. I still use them even today (Examples: Indexing Foreign Keys, Disowning Your Relatives)
Potential employers, clients or trainees: A perfectly valid set of readers, but writing for them is tricky. You’re bragging (which is okay) but you don’t want to appear like you’re bragging (which is not okay) so keep it subtle. Keep the audience in mind. It’s better if the message is “I love this stuff” or “I can help you” rather than “Look how smart I am.”
RSS Readers and Link Followers: Yep, that’s you! (both of you). You enjoy keeping up with SQL Server industry news by following various SQL Server blogs including this one. Something piqued your interest about the title and you started reading (btw, thanks for reading this far!).

Don’t write anything else

This is Mark Twain’s “Employ a simple and straightforward style.” It’s also George Orwell’s “If it is possible to cut a word out, always cut it out.”

Writing clearly goes back to having a point. If a sentence, word or paragraph does not help your point then it probably doesn’t belong. When you omit the unimportant stuff, what’s left is packed with meaning.

One trick I use is to do a brain dump. I quickly type an outline of what I want to write so that I don’t forget anything. Often this simple outline gets included into the post verbatim.