Michael J. Swart

December 12, 2014

Obvious and Not-So-Obvious Writing Tips

Filed under: SQLServerPedia Syndication — Michael J. Swart @ 10:54 am

Takeaway: I leave SQL Server behind this week and I give two tips for technical bloggers,

  1. An obvious tip: Practice a lot
  2. A not-so-obvious tip: Help your readers skip reading your article

First the obvious tip.

Practice in Volume

As far as tips go, practice makes perfect is kind of obvious, and ultimately a little disappointing. Just like “Eat right and exercise”, the phrase “Go practice more” is one of those things that is easier said than done.

I first heard about a Composition Derby when I read The Underachieving School by John Holt. John Holt was an English teacher and author and he describes the Composition Derby as a device he used to help kids practice writing. The kids in his English class get divided into teams and they are asked to write about anything they want (spelling and grammar doesn’t count). At the end of the competition, the team who has written the most words wins. That’s the only criteria, number of words. When kids don’t worry about making mistakes they feel free to practice more. And that frees them to improve faster.

But I think the volume of practice is the key here. I believe in Malcolm Gladwell’s 10,000 hours rule. The rule claims that it takes 10,000 hours to become an expert at something. I like the idea of the 10,000 hour rule, but the one thing I don’t like is that it gives a definite number. Eight hours of writing practice can yield results and 10,000 hours implies a finish line. For example, compare these two illustrations I drew. They both use the same reference photo but they’re spaced apart by about 1,000 hours of practice.
An upset looking E. F. CoddTed Codd

It’s easy to compare illustrations when presented side by side. It’s not as easy to compare writing but feel confident that with practice, you’ll improve and your readers will notice.

Make Your Article Skippable

The second tip is a little counter-intuitive. Make it easy for your readers to skim your article or even skip reading your article all together.

You have something important to write, and I get that. But when thinking about the reader-writer relationship, your article is all about your readers. Their need to read actually outweighs your need to write and ultimately your readers will decide what’s important. I’m notoriously bad at predicting whether a post of mine will be well received or not. And so I make my blog posts skippable. The readers who find what I write important will stick around.

Here are some methods I use that help readers stop reading. Consider using these methods in your own writing

  • Topic sentence (which I frame as a takeaway). Condense your whole blog into a tweet-sized sentence. Give everything away as quickly and clearly as you can. Leave suspense-building for mystery writers. For example, if you only read SQL Server articles, you probably haven’t made it this far. You probably didn’t make it past the first sentence.
  • Organize your article into sections with headings that can stand alone as an outline. It improves skimmability.
  • In general, put a high value on your reader’s time. Make every word count in helping you say the one thing you want to say and don’t say anything else.

Now here’s the crazy part, when you make your article skippable it actually has the opposite effect. These methods I use actually help readers stick around. Readers have a better mental roadmap of the content and they stay (see, you’ve stuck around this far!).

December 3, 2014

Materialized Views in SQL Server

Filed under: Miscelleaneous SQL,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 9:28 am

What’s the difference between Oracle’s “materialized views” and SQL Server’s “indexed views”? They both persist the results of a query, but how are they different? Sometimes it’s difficult to tell.

I'm on the left (or am I?)

I’m on the left (or am I?)

One difference is that SQL Server’s indexed views are always kept up to date. In SQL Server, if a view’s base tables are modified, then the view’s indexes are also kept up to date in the same atomic transaction.

Let’s take a look at Oracle now. Oracle provides something similar called a materialized view. If Oracle’s materialized views are created without the REFRESH FAST ON COMMIT option, then the materialized view is not modified when its base tables are. So that’s one major difference. While SQL Server’s indexed views are always kept current, Oracle’s materialized views can be static.

Static Materialized Views In SQL Server?

Yeah, we just call that a table. You can use a SELECT INTO statement and it’s pretty easy. In fact, for fun I wrote a procedure that does the work for you. Given the name of a view it can create or refresh a table:

/* This is a proof-of-concept and is written for illustration purposes, don't use this in production */
create procedure dbo.s_MaterializeView
  @viewName nvarchar(300),
  @yolo bit = 0 -- use @yolo = 1 to execute the SQL immediately
as
 
declare @persistedViewName nvarchar(300);
 
if not exists (select 1 from sys.views where object_id = object_id(@viewName))
  THROW 50000, N'That @viewName does not exist', 1;
 
select 
  @viewName = QUOTENAME(object_schema_name(object_id)) 
  + N'.'
  + QUOTENAME(object_name(object_id)),
  @persistedViewName = QUOTENAME(object_schema_name(object_id)) 
  + N'.'
  + QUOTENAME(N'persisted_' + object_name(object_id))
from sys.views
where object_id = object_id(@viewName);
 
set xact_abort on;
begin tran
  declare @sql nvarchar(2000);
  set @sql = N'
    IF OBJECT_ID(''' + @persistedViewName + N''') IS NOT NULL
      DROP TABLE ' + @persistedViewName + N';
 
    SELECT *
	INTO ' + @persistedViewName + N'
    FROM ' + @viewName + N';'
 
  if (@yolo = 1)
    exec sp_executesql @sql;  
  else 	
    print @sql;
commit

Which can be used to generate sql something like this:

    IF OBJECT_ID('[dbo].[persisted_vSomeView]') IS NOT NULL
      DROP TABLE [dbo].[persisted_vSomeView];
 
    SELECT *
	INTO [dbo].[persisted_vSomeView]
    FROM [dbo].[vSomeView];

Are Such Static Materialized Views Useful?

Yes:

  • They can be used to get around all the constraints placed on regular indexed views. And if you’ve ever implemented indexed views, you understand that that’s a lot of constraints. I think this benefit is what makes this whole blog post worth consideration.
  • Because it’s static, you can avoid all the potential performance pitfalls that accompany the maintenance of an indexed view (more on this next week).
  • Good or bad, the view doesn’t have to be created with SCHEMABINDING.
  • Indexing is strictly do-it-yourself. Chances are you want more than a single heap of data for your materialized view.

… and no:

  • Most obviously, the data is static, which is another way of saying stale. But notice how Microsoft promotes indexed views. They say that indexed views are best suited for improving OLAP, data mining and other warehousing workloads. Such workloads can typically tolerate staleness better than OLTP workloads. And so maybe materialized views are a feasible alternative to indexed views.
  • You have to manage when these views get refreshed. This means scheduling jobs to do extra maintenance work (yuck). For me that’s a really high cost but it’s less costly if I can incorporate it as part of an ETL process.
  • Using Enterprise Edition, SQL Server’s query optimizer can choose to expand indexed views or not. It can’t do that with these materialized views.

I didn’t write the procedure for any important reason, I just wrote it because it was fun. But I have used this materialized view technique in SQL Server at work and I’ve been quite successful with it. It’s not something that should be used often, but it’s always worth considering if you can understand the trade-offs.

October 3, 2014

Watch Out for Misleading Behaviour From SQL Server

Takeaway: To get consistent behaviour from SQL Server, I share a set of statements I like to run when performing tuning experiments.

Inconsistent Behaviour From SQL Server?

I often have conversations where a colleague wants to understand why SQL Server performs faster in some cases and slower in other cases.

The conversation usually starts “Why does SQL Server perform faster when I…” (fill in the blank):

  1. … changed the join order of the query
  2. … added a transaction
  3. … updated statistics
  4. … added a comment
  5. … crossed my fingers
  6. … simply ran it again

What’s Going On?

It can actually seem like SQL Server performs differently based on its mood. Here are some reasons that can affect the duration of queries like the ones above

  • You changed something insignificant in the query. What you may be doing is comparing the performance of a cached plan with a newly compiled plan. Examples 1 – 4 might fall under this scenario. If that’s the case, then you took a broken thing and gave it a good thump. This percussive maintenance may be good for broken jukeboxes, but maybe not for SQL Server.
  • What about those last two? Say you hit F5 to execute a query in Management Studio, and wait a minute for your results. You immediately hit F5 again and watched the same query take fifteen seconds. Then I like to point out that maybe all that data is cached in memory.

In order to do tune queries effectively, we need consistent behaviour from SQL Server, if only to test theories and be able to rely on the results. SQL Server doesn’t seem to want to give us consistent behaviour…

So Is It Possible To Get Straight Answers?

Best line from all Star Wars

But maybe we can get straight answers from SQL Server. Here’s a test framework that I like to use before all experiments when I want consistent behaviour:

/* Only do this on dev sql servers! */
CHECKPOINT 
DBCC DROPCLEANBUFFERS
DBCC FREEPROCCACHE
SET STATISTICS IO, TIME ON
-- Ctrl+M in Management Studio to include actual query plan

The first two statements are meant to clear SQL Server’s cache of data. Because of write ahead logging, SQL Server will write data changes to disk immediately, but may take its time writing data changes to disk. Executing CHECKPOINT makes SQL Server do that immediately. After the checkpoint there should be no dirty buffers. That’s why DBCC DROPCLEANBUFFERS will succeed in dropping all data from memory.

The DBCC FREEPROCCACHE command will remove all cached query plans.

These commands give SQL Server a fresh starting point. It makes it easier to compare behaviour of one query with the behaviour of another.

The SET STATISTICS IO, TIME ON and the Ctrl+M are there in order to retrieve better information about the performance of the query. Often CPU time, Logical IO, and the actual query plan are more useful when tuning queries than elapsed time.

September 18, 2014

SQL Server Ignores Trailing Spaces In Identifiers

Filed under: Miscelleaneous SQL,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 10:27 am

Takeaway: According to SQL Server, an identifier with trailing spaces is considered equivalent to the same identifier with those spaces removed. That was unexpected to me because that’s not how other programming languages work. My investigation was interesting and I describe that here.

The First Symptom

Here’s the setting, I work with a tool developed internally that reads metadata from a database (table names, column names, column types and that sort of thing). Recently the tool told me that a table had an unexpected definition. In this case, a column name had an extra trailing space. I expected the column name "Id" (2 characters), but my tool was reporting an actual value of  "Id " (notice the blank at the end, 3 characters). That’s what started my investigation.

But that’s really weird. What would lead to a space accidentally getting tacked on to a column name? I couldn’t think of any reason. I also noticed a couple other things. Redgate SQL Compare was reporting no discrepancies and the database users weren’t complaining at all, they seemed just fine. A bug in the in-house tool seemed most likely. My hunch was that there was a problem with the way we collecting or storing these column names (how did a space sneak in there?).

Where Are Column Names Stored?

I wanted to look at the real name of the column – straight from the source – so I ran:

SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_NAME LIKE 'Id%'

It told me that my tool wasn’t wrong. That the column was actually named "Id " with the space. So maybe Red Gate is getting its metadata from somewhere else? I know of a few places to get column information. Maybe Red Gate is getting it from one of those? Specifically I wanted to look closer at these views:

  • sys.columns
  • sys.syscolumns
  • INFORMATION_SCHEMA.COLUMNS

Because these objects are views, I used sp_helptext to learn that all the column names ultimately come from a system table called sys.syscolpars. But sys.syscolpars is a system table and you can’t look at its contents unless you connect to the database using the dedicated administrator connection. And that’s exactly what I did.

I learned that there is only one version of column names, only one place on disk that sql server persists the name of the column. It’s interesting because this implies that Red Gate’s SQL Compare trims trailing spaces from identifier names.

But Doesn’t SQL Server Care?

Well, there’s one way to check:

CREATE TABLE [MyTest] ( [id ] INT );
INSERT INTO [MyTest] VALUES (1);
 
SELECT [id ], [id] -- one column name with a space, one column name without
FROM [MyTest]; 
-- returns a dataset with column names as specified in the query.
go

Just like Red Gate’s SQL Compare, it seems like SQL Server doesn’t care about trailing spaces in identifiers either.

Google? Stackoverflow? Want to Weigh In?

A quick search led me to the extremely relevant Is SQL Server Naming trailing space insensitive?.

And that question has answers which link to the Books Online page documenting Delimited Identifiers. That page claims that “SQL Server stores the name without the trailing spaces.” Hmmm, they either mean in memory, or the page is inaccurate. I just looked at the system tables a moment ago and the trailing spaces are definitely retained.

The stackoverflow question also led me to a reported defect, the Connect item Trailing space in column names. This item was closed as “by design”. So this behavior is deliberate.

What do other SQL Vendors do?

I want to do experiments on SQL databases from other vendors but my computer doesn’t have a large number of virtual machines or playground environments. But do you know who does? SQL Fiddle
It’s very easy to use this site to see what different database vendors do. I just pick a vendor and I can try out any SQL I want. It took very little effort to be able to compile this table:

RDBMS CREATE TABLE... SELECT "id ", "id"...
MySQL Incorrect column name 'id '
Oracle Success "id": invalid identifier
PostgreSQL Success Column "id" does not exist
SQLite Success could not prepare statement (1 no such column: id)
SQL Server Success
ID ID
1 1

And What Does the ANSI standard say?

Look at the variety of behaviors from each vendor. I wonder what the “standard” implementation should be.

Mmmm... SQL Syntax rules.

I googled “ANSI SQL 92″ and found its wikipedia page and that led me to the SQL-92 Standard itself.

ANSI (paraphrased) says that

<delimited identifier> ::= <double quote><one or more characters><double quote>

And it also says explicitly that delimited identifiers can include spaces.

What About String Comparisons In General?

During my experiments on SQL Server I found myself executing this query:

SELECT *
FROM sys.columns
WHERE name = 'Id'

I was surprised to find out that my three-character "Id " column came back in the results. This means that SQL Server ignores trailing spaces for all string comparisons, not just for identifiers.

I changed my google search and looked for “sql server string comparison trailing space”. This is where I found another super-relevant document from Microsoft: INF: How SQL Server Compares Strings with Trailing Spaces.

Microsoft pointed to the ANSI standard again. I mean they explained exactly where to look, they pointed straight to (Section 8.2, , General rules #3) which is the section where ANSI explains how the comparison of two character strings is determined. The ANSI standard says that for string comparisons, the shorter string is effectively padded with trailing spaces so that comparisons can always performed on strings with an equal number of characters. Why? I don’t know.

And that’s where identifier comparisons come in. I found another part of the standard (Syntax rule #11) which tells me that Identifiers are equivalent if they compare as equivalent according to regular string comparison rules. So that’s the link between string comparisons and identifier comparisons.

Summary

There’s a number of things I learned about string comparisons. But does any of this matter? Hardly. No one deliberately chooses to name identifiers using trailing spaces. And I could have decided to sum this whole article up in a single tweet (see the title).

But did you figure out the head fake? This blog post is actually about investigation. The investigation is the interesting thing. This post describes the tools I like to use and how I use them to find things out for myself including:

  • Queries against SQL Server itself, the obvious authority on SQL Server behavior.
    • Made use of sp_helptext
    • Made use of the Dedicated Adminstrator Connection to look at system tables
  • Microsoft’s Books Online (used this twice!)
  • Microsoft Connect
  • Google
  • Stackoverflow
  • SQLFiddle
  • Wikipedia
  • the ANSI Standard

Maybe none of these resources are new or exciting. You’ve likely used many of these in the past. But that’s the point, you can find out about any topic in-depth by being a little curious and a little resourceful. I love to hear about investigation stories. Often how people find things can be at least as interesting as the actual lesson.

June 27, 2014

Trivia about Trivial Plans

Filed under: SQLServerPedia Syndication — Michael J. Swart @ 12:43 pm

Takeaway: I found an example of a query plan which performs better than the “trivial” query plan.

This post is trivia in that it won’t help you do your job as a developer or DBA. But it’s interesting anyway. It offers a look into an interesting part of SQL Server’s query optimizer.

The Setup

I use the 2012 AdventureWorks database and I mess around with the indexes. It’s a setup that Kendra Little developed in order to demonstrate index intersection.

use AdventureWorks2012
GO
 
DROP INDEX Person.Person.IX_Person_LastName_FirstName_MiddleName;
GO
 
CREATE INDEX [IX_Person_FirstName_LastName] ON [Person].[Person] 
( FirstName, LastName ) WITH (ONLINE=ON);
GO
 
CREATE INDEX [IX_Person_MiddleName] ON [Person].[Person] 
( MiddleName ) WITH (ONLINE=ON);
GO

The Trivial Plan

In management studio, include the actual query plan and run this query:

SET STATISTICS IO ON
 
SELECT FirstName, MiddleName, LastName
FROM Person.Person
 
-- 19972 rows returned
-- 1 scan, 3820 logical reads
-- optimization level: TRIVIAL
-- estimated cost: 2.84673

With such a simple query – one against a single table with no filters – SQL Server will choose to scan the narrowest covering index and it won’t bother optimizing the plan any further. That’s what it means to say the optimization level is TRIVIAL.

For this query, the only index that contains all three columns is the clustered one. So it seems there’s no alternative but to scan it. That sounds reasonable right? That’s what we see in the query plan, it looks looks like this:

ClusteredScan

But notice that SQL Server is doing a lot of reading with this plan choice. The table Person.Person has a column called Demographics. This xml field makes the table very wide, so wide that a typical page in Person.Person can only fit about 5 or 6 rows on average.

The Better-Than-Trivial Plan

Now look at this query:

SELECT FirstName, MiddleName, LastName
FROM Person.Person
WHERE FirstName LIKE '%'
 
-- 19972 rows returned
-- 2 scans, 139 logical reads
-- optimization level: FULL
-- estimated cost: 1.46198

The filter is put in place to have no logical effect.  It complicates things just enough so that SQL Server won’t use a trivial plan. SQL Server fully optimizes the query and the query plan now looks like this:

IndexIntersect

Notice that the plan has scans on two indexes nonclustered indexes and a hash join. SQL Server figures (correctly) that scans of two narrow indexes plus a hash join are still cheaper than the single scan of the fat clustered index.

Careful

I don’t think I need to say this, but I do not recommend adding WHERE column like '%' anywhere except maybe in contrived examples for demo purposes.

(MJS — Enjoy the summer, See you in September!)

May 22, 2014

Enabling the New Cardinality Estimator in SQL Server 2014

Filed under: SQLServerPedia Syndication — Michael J. Swart @ 8:33 am

Takeaway: SQL Server 2014 will make use of its newly re-written Cardinality Estimator when the database’s compatibility mode is at least 120. But there’s more to the story.

What’s a Cardinality Estimator (CE)?

Say you’ve been hired to phone everyone on a particular list. If it’s a list of all Americans taller than seven feet, you might manage quite well on your own. But if it’s a list of all Americans shorter than seven feet, you’ll probably need help from others. That’s not surprising because the sizes of the lists are wildly different. One list could have 300 people on it and the other could have 300 million. The expected sizes of the lists influence how you tackle this problem.

This was after phones, but before the do not call list.

SQL Server does the same thing. It uses statistics to find the best ways to execute queries. To find a good query plan, SQL Server often needs to make many choices (which join type, join order, parallelism etc…) It needs to estimate the cost of each choice and it uses educated guesses to evaluate these costs. That’s what the CE was built to do. It provides educated guesses about the number of rows a query plan has to process. That’s why it’s called the cardinality estimator. The accuracy of these estimates will influence the quality of query plans, and consequently, the performance of queries.

With SQL Server 2014, Microsoft released a rewritten version of SQL Server’s CE. I can’t wait to take advantage of it. I’m looking forward to tuning fewer poorly performing queries. Queries that seem to be written well, but are vulnerable to bad query plans.

Risk of Regressions

The CE is part of the query optimizer, so the rewrite represents a significant change to the database engine. And with any pervasive change, there’s always a risk of regressions. While rare, some workloads are expected to perform worse with the new CE. Joe Sack’s excellent white paper Optimizing Your Query Plans with the SQL Server 2014 Cardinality Estimator has some essential tips and suggestions on how to assess and deal with these potential regressions.

Some users may want to continue using the legacy CE. And some users may want to decouple the adoption of the new CE with the adoption of SQL Server 2014. Microsoft anticipated this and so they give DBAs a choice. DBAs have the option to either use the new CE or to stick with the legacy CE.

Enabling the New CE – the Official Details

Simply put, CE behavior can be controlled using the compatibility mode and/or trace flags:

  • The new CE is enabled when compatibility mode is 120 and disabled when it is less than that. The compatibility mode of a database is not modified automatically during an upgrade to 2014, so remember to adjust it accordingly.
  • New trace flags are introduced. Trace flag 9481 can force SQL Server to use the legacy CE when it would otherwise use the new one. Conversely, trace flag 2312 can force SQL Server to use the new CE. And if flags 9481 and 2312 are ever both enabled (in any context), then neither flag takes effect. They cancel each other out and the CE behavior is determined only by the compatibility mode.

Just those two things allow you to influence the CE behavior depending on the granularity you require:

  • For a single query – You could use the QUERYTRACEON hint but it’s not a tempting option. Sysadmin privileges or a forced plan are required.
  • Based on your session – Use session trace flags (again, sysadmin privileges are required).
  • Based on the database you’re connected to – Use compatibility mode.
  • For the whole server – Use global trace flags.

Again, Joe Sack’s white paper explains this in more detail. He provides syntax examples and methods to determine which CE was used based on a query plan.

Corner Use Cases

This leads to some surprising behaviors:

Connect to a System Database to Avoid Compatibility Mode issues

For example, this works:

use master -- in SQL Server 2014, master will always be at compatibility mode 120
GO
 
-- any query (regardless of participating tables) will now use the new CE. e.g.:
SELECT COUNT(*) 
FROM Adventureworks2012.Sales.SalesOrderHeader;

But it’s just a trick and not a technique I would recommend. Besides, this trick doesn’t work when calling stored procedures from other databases.

Using a Trace Flag to Cancel Another One

Trace flags 2312 and 9481 don’t play together well. There is no scenario where one takes precedence over the other. If they’re both enabled, then they cancel each other out:

use Adventureworks2012 -- at compatibility mode 110
GO
DBCC TRACEON( 9481 );
 
SELECT COUNT(*) 
FROM Sales.SalesOrderHeader
OPTION( QUERYTRACEON 2312 ); -- 2312 normally enables the new CE 
-- the 2312 hint is canceled by the 9481 trace flag, the legacy CE is still used.

Again, I avoid this scenario so that I don’t need to worry.

How I Plan To Adopt the New CE

I’d like to begin using the new CE as soon as I upgrade to 2014.

But if I wanted to, I would feel comfortable using compatibility mode as a feature toggle for the new CE. There are other behavior differences between compatibility modes 110 and 120. But I don’t use them and won’t encounter them. They’re obscure and easy to review. So for me, I can ignore those other features and use compatibility mode 120 as the CE feature toggle.

The trace flags 2312 and 9481 are new in SQL Server 2014. So if SQL Server is not at version 2014, it will ignore those trace flags. I intend to do the same no matter what version I’m using. I don’t expect to see many queries showing serious regressions with the new CE, but if I encounter any I’m not going to manage them with these trace flags. Instead, I plan to:

  1. Use hints (whether that means index hints, join hints or query hints) to stabilize the plan temporarily.
  2. Spend time tuning or rewriting the query so that it performs well without these hints.

Further Reading

April 23, 2014

Removing Comments from SQL

Filed under: Miscelleaneous SQL,SQL Scripts,SQLServerPedia Syndication,Technical Articles — Michael J. Swart @ 10:20 am

Check out the following deliberately crazy SQL Script:

create table [/*] /* 
  -- huh? */
(
    "--
     --" integer identity, -- /*
    [*/] varchar(20) /* -- */
         default '*/ /* -- */' /* /* /* */ */ */
); 
go

It’s not surprising that my blog’s syntax colorer has trouble with this statement. But SQL Server will run this statement without complaining. Management Studio doesn’t even show any red squiggly lines anywhere. The same statement without comments looks like this:

create table [/*] 
(
    "--
     --" integer identity, 
    [*/] varchar(20) 
         default '*/ /* -- */' 
); 
go

I want a program to remove comments from any valid SQL and I want it to handle even this crazy example. I describe a handy method that lets me do that.

Using C#

  • In your C# project, find and add a reference to Microsoft.SqlServer.TransactSql.ScriptDom. It’s available with SQL Server 2012’s Feature Pack (search for “ScriptDom” and download).
  • Add using Microsoft.SqlServer.Management.TransactSql.ScriptDom; to your “usings”.
  • Then add this method to your class:
    public string StripCommentsFromSQL( string SQL ) {
     
        TSql110Parser parser = new TSql110Parser( true );
        IList<ParseError> errors;
        var fragments = parser.Parse( new System.IO.StringReader( SQL ), out errors );
     
        // clear comments
        string result = string.Join ( 
          string.Empty,
          fragments.ScriptTokenStream
              .Where( x => x.TokenType != TSqlTokenType.MultilineComment )
              .Where( x => x.TokenType != TSqlTokenType.SingleLineComment )
              .Select( x => x.Text ) );
     
        return result;
     
    }

… and profit! This method works as well as I hoped, even on the given SQL example.

Why I Prefer This Method

A number of reasons. By using Microsoft’s own parser, I don’t have to worry about comments in strings, or strings in comments which are problems with most T-SQL-only solutions. I also don’t have to worry about nested multiline comments which can be a problem with regex solutions.

Did you know that there’s another sql parsing library by Microsoft? It’s found at Microsoft.SqlServer.Management.SqlParser.Parser. This was the old way of doing things and it’s not supported very well. I believe this library is mostly intended for use by features like Management Studio’s Intellisense. The ScriptDom library is better supported and it’s easier to code with.

Let Me Know If You Found This Useful

Add comments below. Be warned though, if you’re a spammer, I will quickly remove your comments. I’ve had practice.

April 11, 2014

Implementing the Recycle Bin Pattern In SQL

Filed under: SQLServerPedia Syndication — Michael J. Swart @ 8:00 am

Kitchener Ontario, recycling since 1983I participated in a week long hackathon recently. It was great to be able to spend the whole week on a self-directed project. I’m excited to write about what my team accomplished, but actually I want to blog about what another team accomplished. That team implemented a really nice “send to recycle bin” feature and they gave me the green light to write about it here.

The recycle bin feature is ultimately a data-hiding feature. Users don’t necessarily want to destroy data, they just don’t want to look at it right now. There are a lot of ways to implement this feature, but one way is by making a few changes in the database (as opposed to the application).

What Needs To Change?

Surprisingly not much. Take your table and give it a nullable RecycleDate column. This is all you need to track the recycled rows. Then create a view that filters out recycled items. That’s pretty much it. Afterwards, if you rename the table, then the view can take its place. This is what that would look like on Adventureworks’ Sales.ShoppingCartItems table:

ALTER TABLE Sales.ShoppingCartItem
  ADD RecycleDate DATE NULL
    CONSTRAINT DF_ShoppingCartItem_RecycleDate DEFAULT NULL;
 
GO
 
EXEC sp_rename 'Sales.ShoppingCartItem', 'AllShoppingCartItems'
 
GO
 
CREATE VIEW Sales.ShoppingCartItem
WITH SCHEMABINDING
AS
    SELECT  ShoppingCartItemID ,
            ShoppingCartID ,
            Quantity ,
            ProductID ,
            DateCreated ,
            RecycleDate
    FROM    Sales.AllShoppingCartItems
    WHERE   RecycleDate IS NULL;
 
GO
 
CREATE PROCEDURE Sales.s_RecycleShoppingCartItem
    (
      @ShoppingCartItemId INT
    )
AS 
    UPDATE  Sales.ShoppingCartItem
    SET     RecycleDate = GETDATE()
    WHERE   ShoppingCartItemID = @ShoppingCartItemId;
 
GO

DML Impact

So what’s the impact on other Delete, Insert, Update or Select statements that are executed against your modified table?

  • Delete statements shouldn’t be affected. You’ll notice that recycle bin contents can’t be deleted via the view. That’s okay.
  • Old Insert statements should work as expected with no adjustments, especially if you name your columns in a column list.
  • Update statements? Check, they’ll continue to work.
  • Select statements will also be unaffected. Especially if you’ve avoided SELECT *.

What About Foreign Keys?

Okay, this is where it gets little tricky. If you don’t use ON DELETE or ON UPDATE clauses with your foreign keys, then you have to be a little careful. I want to show just one example of how things can get a bit messy. Returning to our Adventureworks example, lets think about a query that deletes “shopping carts” as long as it has no items.

DELETE Sales.ShoppingCart
WHERE ShoppingCartId = @ShoppingCartIdToDelete
AND NOT EXISTS
  (
    -- any items in the cart?
    SELECT 1
    FROM Sales.ShoppingCartItem
    WHERE ShoppingCartId = @ShoppingCartIdToDelete
  )

In the old world, this works no problem. But our check for items in the cart misses items that have been recycled and so this query would fail. You’ll have to remember to find queries like this and update them to check Sales.AllShoppingCartItems instead.

Data Lifecycle Policy Concerns

You have a policy right? The lack of one can make it too easy to retain data indefinitely. The concern isn’t necessarily storage, but whether you’re meeting any policies or regulations concerning privacy or other things like that.

The recycle bin feature may make it a little easier to accidentally retain data you didn’t mean to. It may be worth regression testing any delete or purge functionality that you have.

Indexing

Depending on how much data is hidden in the recycle bin, you shouldn’t have to re-evaluate your indexing strategy. Your indexes should probably serve you just as well after this implementation. But if you find yourself storing more than 90% of your data as recycled data, then you may want to start considering re-assessing the table’s indexes. You could consider things like filtered indexes, filtered stats and/or partitioned tables. But before you do, see Data Lifecycle Policy Concerns above.

Other Things To Watch

Any changes to schema or any code should lead to extra testing and the changes I’m proposing are no different.

You have to know your app and environment. Is your recycle bin against a table that participates in downstream Business Intelligence projects? How about Change-Data-Capture? Service Broker? Notification Services? You know better than I do.

Other Reycling Bin Implementations

There are lots of methods.

For example, You don’t have to implement this pattern using SQL. You can implement it in your application. Hiding recycled data via the application makes a lot of sense. Especially if your more of a programmer than a SQL developer (By the way, where’d you come from? Who let you in here?)

It’s worth giving this some thought. Without a recycle bin, the demand to retrieve “deleted” data can be great enough to prompt someone to dig through a restored backup. Digging through restored backups actually counts as a recycle bin implementation even if it is an unintentional and painful one.

March 16, 2014

T-SQL Tuesday #52 Round-Up: Argue Against a Popular Opinion

Filed under: SQLServerPedia Syndication — Michael J. Swart @ 1:52 pm

T-SQL Tuesday LogoThere were thirteen excellent posts this month. (See the invite post here). I loved reading every one of them. It gave a really good perspective on topics that are typically seen as cut and dry. Thank you to each one of the bloggers for putting in the time to participate this month. Here are the posts, each with a mini-review.

Michael J Swart
(@MJSwart)
I Don’t Mind SELECT * Sometimes

The opinion:
The popular opinion is that “SELECT *” should always be avoided. I argue that there are some cases where there are no downsides.
Did I buy it?
I may be a little biased, but I totally bought it. Truer words were never spoken.
Boris Hristov
(@BorisHristov)
Only My Technical Skills Matter

The opinion:
The popular opinion is that only technical skills matter. Boris argues that the most effective professionals develop other skills (like soft skills).
Did I buy it?
For sure. I would also add to his post. It’s easy to agree that soft skills matter. But I’ve recently learned that (holy crap) do they ever matter!
Tech skills are important.
Communication skills can come with enough practice
But tech skills + communication skills + wisdom are a lethal combination.
Russ Thomas
(@SQLJudo)
Eye Heart You Dee Effs

The opinion:
There are a lot of UDF-haters out there in SQL land. Not Russ, he loves them.
Did I buy it?
What say I? I say aye! Well done Russ. Totally got into the spirit of this month’s topic. BTW, he gets the Michael J Swart “Best Article Of The Month” award.
Bob Pusateri
(@SQLBob)
Why I Hate Row Compression

The opinion:
Row compression has been an extremely popular feature since its introduction. But Bob warns us of the risk of using it as spackle for sloppy data modeling.
Did I buy it?
Sloppy design causes future problems. I buy that. Row compression can enable sloppy design. I buy that. The Spock in me says row compression can cause future problems. But the Kirk in me has trouble accepting it.
Thanks for writing the most though-provoking article.
Ken Fisher
(@sqlstudent144)
Keep your head down!

The opinion:
Ken takes a look at people known for their collaborative character and he argues against that behavior with the more isolationist advice: “Keep your head down!”. The sarcasm is called out explicitly and so Ken is actually promoting more collaborative behaviors.
Did I buy it?
His post’s thesis argues with the popular trend of more collaboration. As a blogger myself, thumbs up! It’s was a really well written article, the light-hearted style makes for a good read.
Rob Farley
(@rob_farley)
Scans are better than Seeks. Really.

The opinion:
Rob and I had a great discussion in the week leading up to #tsql2sday. He wants us to believe (with no irony) that scans are better than seeks.
Did I buy it?
After reading the article, yes, I buy it. It turns out there’s a lot of subtlety when talking about seeks and scans. Especially when seeks perform RangeScans or when scans operate on filtered indexes.
Brent Ozar
(@BrentO)
Stop Tuning with Wait Stats Percentages

The opinion:
It’s all in the title. Wait statistics are a popular method of tuning SQL Server. Brent argues that you shouldn’t pay much attention to the reported percentages. Or at least not without some careful context.
Did I buy it?
Sure did. My own take is that wait stats are useful when measured against representative load, when there’s something to tune, and when you look wait rankings over percentages.
Jeremiah Peschka
(@peschkaj)
What Use is an Average?

The opinion:
I invited Jeremiah to contribute this month. Jeremiah has a writing style that doesn’t always show in his more technical blog posts. I was not disappointed. The popular opinion he seems to be arguing against is that the “average” statistic is a worthwhile measure (when applied to query durations for example). He wants us to believe it’s not always enough.
Did I buy it?
Yep, it’s not always enough. And he gets creative with his citations. It was crazy enough Jeremiah… it was crazy enough.
Steve Hood
(@SimpleSQLServer)
Why worry about CXPACKET

The opinion:
The opinion Steve Hood tackles is that the wait type “CXPACKET” is a problem and should cause worry. He argues that it just indicates parallelism which is not necessarily a problem.
Did I buy it?
Yes, In the particular case of CXPACKET, it seems that enough people are tempted to treat the symptom rather than the disease. Steve’s article addresses that.
Steve Jones
(@way0utwest)
Arguments

The opinion:
Steve Jones argues against “Never Turn on Auto Shrink”.
Really?
Really Really?
Did I buy it?
I’m a believer. He gave not one but two scenarios where I would use autoshrink as well.
Mike Fal
(@mike_fal)
Stop depending on “it depends”

The opinion:
So this is interesting. Mike Fal is asking us to get off the fence. It’s popular for DBAs to fall back on “it depends” when cornered for an answer. Mike wants us to “stop it”.
Did I buy it?
Yes, you bet. One of the reasons I picked this topic (“argue against a popular opinion”) is to explore exactly what things depend on. One of the best written articles in this list.
Jes Schultz Borland
(@grrl_geek)
Why Back Up System Databases?

The opinion:
Arguing against public opinion. Jes tells us that that could be the title of her autobiography. The opinion she argues against is that it’s super important to back up your system databases. She thinks it’s just not as important as others want us to believe.
Did I buy it?
Yes, Jes makes a good point. She raises something important. Whether you back up your system databases or script out something to help rebuild a server from scratch. It should be practiced at least once.
Tamera Clark
(@tameraclark)
Don’t Be That Guy

The opinion:
Tamera takes a different tack. Rather than an opinion, she argues against Mr. Popular asking us to not be that guy.
Did I buy it?
She gives 10 pieces of advice to a particular target audience. It’s great advice and it’s a good piece of writing. My only worry is that not enough of her target audience will recognize themselves as the target audience.

March 3, 2014

T-SQL Tuesday #052: Argue Against A Popular Opinion

Filed under: SQLServerPedia Syndication — Tags: — Michael J. Swart @ 10:07 pm

T-SQL Tuesday Logo

Update See the round up post here.
I was asked to host T-SQL Tuesday this month, T-SQL Tuesday #52. This marks my second time hosting and I’m happy to do it again.

Your writing assignment for March 2014 is to

pick a popular opinion and argue against it.

… or at least qualify it. Given any issue, people drift to two kinds of crowds. There’s the “it depends” crowd and there’s the “never ever” crowd. We tend to fall in with one crowd or the other.  This month, I want you to find an “never ever” issue and argue for it (or conversely, find a “always” issue and argue against it).

I wonder how this month will go. It takes guts to go against common wisdom.

You don’t necessarily have to argue against a universal opinion, but it should at least be popular. I think that your choice of opinions is practically limitless:

  • Bob Duffy had a list of 10 interview questions that annoy SQL professionals. With some great topics there including GUIDs, Cursors, and heaps.
  • Google results for “SQL.Server should.never”
  • Fair’s fair. Here are the results for “SQL.Server should.always”
  • Does anyone want to have a shot at redeeming Microsoft Access?
  • Foreign Keys, SchmoreignKeys.
  • Check it out. SQL Server supports varbinary(max)! Ideal for json documents and xml documents (or both!)
  • Shrinking databases and/or log files (because of the fragmentation! God save us all from fragmentation!)

Here’s a little secret. This month’s topic is not for you. It’s for the readers. It’s a chance for you to give them a more nuanced understanding of a topic that they may not have given a lot of thought up until now. I’m a little curious myself.

The rules are the same as always:

Follow These Rules

  1. The post must go live on your blog between 00:00 GMT Tuesday, March 11, 2014 and 00:00 GMT Wednesday, March 12, 2014.
    In other words, set your sql server date, time and timezone properly and run this script:

    IF GETUTCDATE() BETWEEN '20140311' AND '20140312'
    	SELECT 'You Can Post'
    ELSE
    	SELECT 'Not Time To Post'
  2. Your post has to link back to this post, and the link must be anchored from the logo (found above) which must also appear at the top of the post
  3. Leave a comment here (below) or I won’t be able to find your post.

That’s it! Good luck! Can’t wait to see what you have in store.

Your humble host,
Michael J. Swart

Older Posts »

Powered by WordPress