Parallel Programming with 1 Line of Code.

Well maybe not 1 line of code but the new Parallel library in .NET 4.0 makes it dead simple to write parallel processes in our code.  Seriously, this new library is really cool and offers tons of new possibilities.  The main thing that I like about the library is it makes it so easy to take advantage mulitple processors. Most servers these days, and for that matter personal PCs, have multiple processors.  How often are we leveraging those processors in the programs that we write? I bet less often then we should be.  Well, Microsoft has give us the capability to do it with ease.  No excuses now I guess.

Lets take a look at a really “pre-canned” simple example.  Lets say we have a directory filled with a bunch of files that we need to process. The process for each file may take a long period of time.  Let’s compare how we might process the files using the good old for loop with how we might do it using the Parallel class.


List<File> fileList;
fileList = this.loadFilelList( someDirectoryName );
foreach (File file in FileNamesList)
{
FileProcessor.ProcessFile(file);
}


So, if the ProcessFile method take 10 seconds and we have 100 files well that’s a long time ( 1000 seconds for all math wizards ). I am sure there is Big O notation to represent this performance.  I think of it as “damn it take long” and the more files I have the longer it takes.

Now what if we could distribute that processing across multiple processors simultaneously.  Done and done.


Parallel.ForEach(fileNameList, (File file) => {
FileProcessor.ProcessFile(file);
});


What this is doing is that each ProcessFile method is “farmed” off to whatever core processor is available.  If we have 4 then we are reducing our processing time by approximately 75%.  This is approximate and greatly depends on the code, memory etc.

The point is not to get an exact number on how much we improved performance.  We can measure that within our real world applications.The point is to show just how easy it is to write parallel sections of code.  Dead simple right?

A few things to keep in mind.  You should design your code so that each iteration of the loop is completely independent from the others. Any single iteration should not rely on another in order to complete correctly. If we do have shared variables that need to be accessed by each thread then we need to implement a locking scheme.

WebRequest and SSL and forcibly closed by the remote host

Ever run into this Exception when trying to call a service via https?

Here is some boiler plate code to call the the service via a webrequest


var request = (HttpWebRequest)HttpWebRequest.Create(uri);
if (Settings.Credentials != null)
request.Credentials = Settings.Credentials;
var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode != HttpStatusCode.OK)
throw new WebException("Web Request to '" + uri + "' failed with code: " + response.StatusCode);


This looks fine and dandy but when you run the code you get the exception:

“Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host”.

What the heck? To make matters worse you can navigate to the URL in a browser,
enter the credentials and you get the response back.

Well one thing to try is to explicitly set the protocol of the request prior to GetResponse.


System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;


I was running into this issue and by adding this line all my issues went away.

Post JSON to WCF service via JQuery

Real nice and concise article on how to post JSON object to a WCF service.

http://www.primordialcode.com/index.php/2010/01/27/passing-json-serialized-objects-wcf-service-jquery/#comments

SubSonic T4 Templates / Stored Procedures / Nullable Parameters

One thing I noticed recently when working with Subsonic 3.0 and the T4 templates was that there was no way to detect if a stored procedure has nullable parameters.  After a bit of research I found this msdn article that explains:

“SQL Server only maintains default values for CLR objects in this catalog view; therefore, this column has a value of 0 for Transact-SQL objects. To view the default value of a parameter in a Transact-SQL object, query the definition column of the sys.sql_modules catalog view, or use the OBJECT_DEFINITION system function.”

Well that kind of stinks.  I want my cake and I want to eat it too.

I want my generated code to
a) Declare the parameters to stored procedures as nullable types
b) Ensure the code passes these parameters in as nullable types.

In order to accomplish this I changed the following in the T4 Templates.

1. Inside SQLServer.ttinclude

a. Add methods

public string GetStoredProcedureText(string procedureName)
{
string sql;
sql = @"EXEC sp_helptext '" + procedureName + "'";
SqlDataReader sprocReader;
StringBuilder sprocBuilder = new StringBuilder();
using (SqlConnection cn = new SqlConnection(ConnectionString))
{
cn.Open();
SqlCommand cmd = new SqlCommand(sql, cn);
sprocReader = cmd.ExecuteReader();
while (sprocReader.Read())
{
sprocBuilder.Append(sprocReader.GetString(0));
}
return sprocBuilder.ToString();
}
}

public List<string> GetNullableParams(string procedureText)
{
string arguments = string.Empty;
string[] argumentArray = new string[0];
List<string> parameterNullibility = new List<string>();
procedureText = Regex.Replace(procedureText, @"\r\n", " ");
procedureText = Regex.Replace(procedureText, @"\s+", " ");
procedureText = Regex.Replace(procedureText, @"\s*,\s*", ",");
procedureText = Regex.Replace(procedureText, @"/\*.*?\*/", "");
string regex = @"CREATE\s+((PROC)|(PROCEDURE))\s+((\[.+?\]\.?){*}|(\S*))(?<Params>.*?)\s{1}as\s{1}";
RegexOptions options = RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture;
MatchCollection matches = Regex.Matches(procedureText, regex, options);
if (matches.Count > 0 && matches[0].Groups["Params"] != null)
{
Match match = matches[0];
arguments = match.Groups["Params"].ToString();
argumentArray = Regex.Split(arguments, ",");
foreach (string sArgument in argumentArray)
{
MatchCollection paramMatches = Regex.Matches(sArgument, @"(?<ParamName>@\S*)", options);
MatchCollection nullMatches = Regex.Matches(sArgument, @"(?<NullString>=\s*)", options);
if (paramMatches.Count > 0 && paramMatches[0].Groups["ParamName"] != null)
{
bool isNullDefault = (nullMatches.Count > 0 && nullMatches[0].Groups["NullString"] != null);
parameterNullibility.Add(paramMatches[0].Groups["ParamName"].ToString());
}
}
}
return parameterNullibility;
}

b. Change GetSPPArams

List GetSPParams(string spName)
{
var result = new List();
string[] restrictions = new string[3] { null, null, spName };
string procedureText;0
procedureText = GetStoredProcedureText(spName);
List nullableParams = this.GetNullableParams(procedureText);
using (SqlConnection conn = new SqlConnection(ConnectionString))
{
conn.Open();
var sprocs = conn.GetSchema("ProcedureParameters", restrictions);
using (SqlCommand cmd = new SqlCommand(spName, conn))
{
cmd.CommandType = CommandType.StoredProcedure;
SqlCommandBuilder.DeriveParameters(cmd);
}
conn.Close();
foreach (DataRow row in sprocs.Select("", "ORDINAL_POSITION"))
{
SPParam p = new SPParam();
p.SysType = GetSysType(row["DATA_TYPE"].ToString());
p.DbType = GetDbType(row["DATA_TYPE"].ToString()).ToString();
p.Name = row["PARAMETER_NAME"].ToString().Replace("@", "");
p.ParameterMode = GetParamDirection(row["PARAMETER_MODE"].ToString()).ToString();
p.CleanName = CleanUp(p.Name);
if (nullableParams.Contains(row["PARAMETER_NAME"].ToString()))
p.Nullable = true;
result.Add(p);
}
}
return result;
}

2.  Inside Setting.ttinclude

a. Change SSParam class as follows


public class SPParam{
public string Name;
public string CleanName;
public string SysType;
public string DbType;
public string ParameterMode;
public bool Nullable;
}


b. Change ArgList property as follows.


public string ArgList{
get{
StringBuilder sb=new StringBuilder();
int loopCount=1;
foreach(var par in Parameters){
if (par.Nullable)
{
if ( par.SysType.ToString() != "string" && par.SysType.ToString() != "byte[]")
{
sb.AppendFormat("{0}{1} {2}", par.SysType, "?", par.CleanName);
}
else
{
sb.AppendFormat("{0} {1}", par.SysType, par.CleanName);
}
}
else
{
sb.AppendFormat("{0} {1}", par.SysType, par.CleanName);
}
if(loopCount<Parameters.Count)
sb.Append(",");
loopCount++;
}
return sb.ToString();
}
}
}

3.  Inside StoredProcedures.tt change the code to account for 1 and 2 above.

<#@ template language=”C#v3.5″ debug=”False” hostspecific=”True”  #>
<#@ output extension=”.cs” #>
<#@ include file=”SQLServer.ttinclude” #>
<#
var sps = GetSPs();
if(sps.Count>0){
#>
using System;
using SubSonic;
using SubSonic.Schema;
using SubSonic.DataProviders;
using System.Data;

namespace <#=Namespace#>{
public partial class <#=DatabaseName#>DB{

<#  foreach(var sp in sps){#>
public StoredProcedure <#=sp.CleanName#>(<#=sp.ArgList#>){
StoredProcedure sp=new StoredProcedure(“<#=sp.Name#>”,this.Provider);
<#      foreach(var par in sp.Parameters){
if ( par.Nullable == true )
{
if (par.SysType == “string” || par.SysType == “byte[]“)
{
#>

sp.Command.AddParameter(“<#=par.Name#>”,<#=par.CleanName#>,DbType.<#=par.DbType#>,<#=par.ParameterMode#>);
<#                }
else
{
#>
sp.Command.AddParameter(“<#=par.Name#>”,(<#=par.SysType#>?)<#=par.CleanName#>,DbType.<#=par.DbType#>,<#=par.ParameterMode#>);
<#
}

}
else
{
#>
sp.Command.AddParameter(“<#=par.Name#>”,<#=par.CleanName#>,DbType.<#=par.DbType#>,<#=par.ParameterMode#>);
<#
}
}#>
return sp;
}
<#  }#>

}

}
<#  }#>

SubSonic and T4 Templates

Recently I have been using Subsonic 3.0 for a project. I really like this tool because I think it does just what it claims to do, namely generate a DAL layer. Granted the NHibernates / Entity Frameworks of the world may have more bells and whistles but sometimes those features really are not necessary. (I know shoot me but its true)

As I went through the process of generating my code, I noticed that I needed to make a few tweaks to the T4. More specifically, I tweaked the templates in sections that dealt with stored procedures. What I wanted was my generated code to be aware of input/output parameters within the stored procedures.

Here are the tweaks I made to the current T4 templates.

1) Settings.ttinclude

Change the SSParam class as follows

public class SPParam{
public string Name;
public string CleanName;
public string SysType;
public string DbType;
public string ParameterDirection;
}

We add the ParameterDirection property to hold the direction of the stored procedure parameter. A case in point is knowing if a parameter is an input or output param.

2) In SQLServer.ttinclude

Change the method GetSPParams as follows.


List GetSPParams(string spName){
var result=new List();
string[] restrictions = new string[3] { null, null, spName };
using(SqlConnection conn=new SqlConnection(ConnectionString)){
conn.Open();
var sprocs=conn.GetSchema("ProcedureParameters", restrictions);
conn.Close();
foreach(DataRow row in sprocs.Select("", "ORDINAL_POSITION")){
SPParam p=new SPParam();
p.SysType=GetSysType(row["DATA_TYPE"].ToString());
p.DbType=GetDbType(row["DATA_TYPE"].ToString()).ToString();
p.Name=row["PARAMETER_NAME"].ToString().Replace("@","");
p.ParameterDirection = GetParamDirection(row["PARAMETER_MODE"].ToString());
p.CleanName=CleanUp(p.Name);
result.Add(p);
}

}
return result;
}

Add the method

string GetParamDirection(string paramMode)
{
switch (paramMode)
{
case "IN":
return "ParameterDirection.Input";
case "INOUT":
return "ParameterDirection.InputOutput";
case "OUT":
return "ParameterDirection.Output";
case "RETURN":
return "ParameterDirection.ReturnValue";
default:
return "ParameterDirection.Input";
}

}

3. Inside StoredProcedure.tt

Change this line.


sp.Command.AddParameter("<#=par.Name#>",<#=par.CleanName#>,DbType.<#=par.DbType#>,<#=par.ParameterDirection#>);

One thing that stinks is we can’t determine if a parameter has a default value or is a nullable parameter. It would be nice so we could generate parameters as nullable types.

Manage Multiple Config Files in Visual Studio

This post is a quick tip on how to manage multiple app.config or web.config files in Visual Studio. Consider the typical development environment for a project where we have development / QA / Production. Each environment has its own database and other settings. We need a way to be able to run the app but use the pertinent config settings. This technique allows for us to create config files for each environment but still have a clean way to use the respective config.

In short it comes down to leveraging pre-compile events and the configuration manager within Visual Studio.

Check it out as it is a real nice time saver.

http://www.hanselman.com/blog/CommentView.aspx?guid=93bfa4b3-44cd-4681-b70e-f4a2b0386466

JQuery UI and CDN

Stumbled across this this article recently. Apparently Google is not only hosting the JQuery libraries but now also the JQuery UI libraries and CSS files.

http://encosia.com/2009/10/11/do-you-know-about-this-undocumented-google-cdn-feature/

SQL – The Constant in a Changing World

I was recently reading Sam Gentile’s Blog focusing on data.

http://samgentile.com/Web/linq/focusing-on-data-ndash-try-3/

This got me to thinking… SQL seems to be the one constant in an ever evolving technology space. I have seen and used all sorts of languages over my career. The languages all had features that seemed to “wrap” SQL. Similarly we see frameworks introduced that try to make our access to data simpler. Things like LINQ to SQL, ORM tools etc.

So how is it that SQL has stayed consistent all these years? And, if it is consistent, why do we try to keep solving the data access problem? Is it even a problem?

Maybe it’s due to the fact that the problem domain that SQL solves is bounded so there is no need for evolution. I do know this..there has been many a time I have been saved by keeping data access in the database. Maybe saved isn’t the right word but rather able to support change more easily. I still am a big fan of stored procedures. I like the ORM tools that can generate code for my stored procedures.

How about everyone else?

HTTP Request LifeCyle

Someone asked me the other day. “What the heck happens after I type in a web address in my browser and hit submit?”

My initial thought was “yeah, good question”. We take this for granted so much; It is so common place that we forget about all the details under the hood. Kind of like when I turn on the shower in the morning how does the water come out? Where does the water come from.

Now, I know what is happening under the hood but went searching around a bit to just refresh my mental model.

I found this little excerpt on superuser.com (side note..that stackoverlow is such a great resource! Man I wish I came up with that idea.)

I thought this excerpt did a nice job in explaining the entire transaction in layman’s terms. In any case here it is. A nice refreshing look at something we take for granted.

Browser: “Ok, so, I have a user requesting this address: www.cnn.com. I figure since there are no slashes or anything, this is a direct request of a main page. There was also no protocol or port defined, so I’ll assume it’s HTTP and going to port 80… oh well, first things first. Hey DNS, pal, wake up! Where is this www.cnn.com hiding at?”

DNS: “Right… wait a sec, I’ll ask the ISP servers. Ok, it looks like 157.166.226.25.”

Browser: “Ok. Internet Protocol Suite, your turn! Call 157.166.226.25, please. Send them this HTTP header. It’s asking for the basic structure and content of their main page so I know what else to fetch… oh well, not that you’d care about this I guess. “

TCP/IP: “What do you mean my turn? Like I wasn’t just working my back off right there for the DNS? God, what does it take to get a bit of appreciation here…”

Browser: …

TCP/IP: “Yeah, yeah… Connecting… I’ll just ask the gateway to forward it. You know, it isn’t all that easy, I’ll have to divide your pretty request there into multiple parts so it reaches the end, and assemble any stuff they send back from all the thousands of packages I get… ah, right, you don’t care. Figures.”

Meanwhile, at the CNN headquarters, a message finally ends up at the door of the Web Server.

CNN Web Server: “Nzhôô! A customer! He wants news! The Front Page! How about it?”

CNN Server Side Script Engine: “Right, will do! Front page, right?”

CNN Database Server: “Yey! Work for me! What content do you need?”

CNN Server Side Script Engine: “… um, sorry DB, I have a copy of front page right here in my cache, no need to compile anything. But hey, take this user ID and store it, I’ll send it to the customer too, so we know who we’re talking to later on.”

CNN Database Server: “Yey!”

Back at the user’s computer…

TCP/IP: “Ooookay, here comes the reply. Oh boy, why do I have a feeling this’ll be a big one…”

Browser: “Uh, wow… this has all sorts of javascript code… bunch of images, couple of forms… Right, this’ll take a while to render. Better get to it. Hey, IP system, there’s a bunch more stuff you’ll need to get. Let’s see I need a few stylesheets from i.cdn.turner.com – via HTTP and ask for the file /cnn/.element/css/2.0/common.css. And then get some of those scripts at i.cdn.turner.com too, I’m counting six so far…”

TCP/IP: “I get the picture. Just give me the server addresses and all that. And wrap that file stuff within the HTTP request, I don’t want to deal with it.”

DNS: “Checking the i.cdn.turner.com… hey, bit of trivia, it’s actually called cdn.cnn.com.c.footprint.net. IP is 4.23.41.126″

Browser: “Sure, sure… wait a sec, this’ll take a few nsec to process, I’m trying to understand all this script…”

TCP/IP: “Hey, here’s the CSS you asked for. Oh, and… yeah, those additional scripts also just came back.”

Browser: “Whew, there’s more… some sort of video ad!”

TCP/IP: “Oh boy, what fun that sounds like…”

Browser: “There’s all sorts of images too! And this CSS looks a bit nasty… right, so if that part goes there, and has this line at the top… how on earth would that fit anymore… no, I’ll have to stretch this a bit to make it… Oh, but that other CSS file overrides that rule… Well, this one ain’t going to be an easy piece to render, that’s for sure!”

TCP/IP: “Ok, ok, stop distracting me for a sec, there’s a lot to do here still.”

Browser: “User, here’s a small progress report for you. Sorry, this all might take a few secs, there’s like 140 different elements to load, and going at 16 so far.”

One or two seconds later…

TCP/IP: “Ok, that should be all. Hey, listen… sorry I snapped at you earlier, you managing there? This sure seems like quite the load for you too.”

Browser: “Phew, yeah, it’s all these websites nowdays, they sure don’t make it easy for you. Well, I’ll manage. It’s what I’m here for.”

TCP/IP: “I guess it’s quite heavy for all of us these days… oh, stop gloating there DNS!”

Browser: “Hey user! The website’s ready – go get your news!”

WPF vs. WPF XBAP vs. Silverlight vs. ASP.NET MVC

A nice small comparison of when to use WPF / WPF XBAP / Silverlight / ASP.NET

WPF vs WPF XBAP vs Silverlight

Shared via AddThis