Category: Software Engineering

This is what I do for a living. Like most things, it can be a little bit of column A and a little bit of column B. Current obessions of mine include Test Driven Design, Domain Driven Design, Distributed Systems, Information Retrieval, Model Driven Design, and Programming Languages.

Are you really Open Source?

I’m a bit groggy and out of it from pulling an all-nighter to build a software that I needed for a client proposal.  I won’t name any names, but, this “open source” product was not easy to build.  First off, when I went to a link that says “download” I was thinking that I was going to see one or more tar gzips.  Well this product had none.  In fact all I could do was to use firefox download manager to download several folders on the svn website for the project. 

Seondly, those files were marked as debian packages and since I was running Ubuntu 9.1 I thought I was cooking with gas.  Those downloads took about 20 minutes and I spent the last week trying to deploy them to no success.  I spent the last week pulling my hair out before accepting that the debian packages were hopelessly broken and not even rebuilding them from the source had any hopes of fixing them.  I went through the torture of downgrading my perl install to 5.8.8 because I assumed there was a library issue.  Actually the core issue is that the dependencies for the debian packages were farked.  In fact, even after adding the developer’s debian site to my Synaptic source list I discovered how broken the packages were:  The two main packages had circular dependencies.  Unfortunately, no package manager on earth can resolve circular dependencies since such a thing is illogical.

Thirdly, after giving up on using debs (and somewhat messing up my perl installation with a polluted deploy of perl 5.8.8–that I must uninstall) I moved to CPAN.  This is where I discovered how truly devious the developers were.  I could understand if someone had not updated their debian and tar gzip deliverables, but, for the CPAN deployment to be broken actually takes some effort from the developers themselves to ensure that the package is uninstallable.  What makes this worse is that to even get to the point of the CPAN deployment I had to download via SVN the trunk for the application.  This is not documented in any of the developer’s documentation.  I am sorry, but, this developer is giving a big middle finger to the Open Source community by not fixing whatever distribution mechanism they are using for their software.  I appreciate the hard work that they have certainly put into this product, but, to deliberately (all my problems were not by accident and were not coincidence, they have deliberately broken the automated build and deploy mechanisms) make it hard to find and use their software they might as well not bothered to declare it “Open Source.”  Open source is not just a phrase, its about a certain respect that developers have for themselves and the greater user community.  A respect that means the developers have faith that others will benefit more from the fruits of their labor rather than restricting use to a chosen few.  Not showing this respect is dishonest and not in keeping with the spirit of the Open Source community.

Seriously, I would have found it easier to download and build the Windows 7 source code than this application.  That is a shameful indictment of this developer and their business practices.  Its no surprise that they offer pay for hosting for the software (I’m guessing the debian packages broke on the same day they announced the hosting plans).

After much hand-wringing, the software finally builds under Ubuntu 9.1.  I have not had the luck of deploying it into Apache, but, I don’t expect much problems with that (though I do expect to have to hack through their setup scripts to make sure files get copied properly).  I had to spend the whole night going through missed dependencies in CPAN (I even had to reverse engineer a few build scripts).  I was considering some sort of partnership with the developer, but, now I don’t respect them enough to do so.  Instead, I will hack their software myself.  Of course I’ll make sure there is a working debian for whatever changes I make available on my site.

Goodbye, and thanks for the fish!!!

Exceptions leading to failure in AdonetAppender.DoAppend: Pattern layout in log4net

If you use log4net then you have to be careful of using patternlayouts for parameters for Adonetappenders.  Log4net can work with all native .net datatypes including nullables.  However, a layout like this:
<parameter>
        <parameterName value=”@UidSubmission”/>
        <dbType value=”Int32″/>
        <layout type=”log4net.Layout.PatternLayout”>
          <conversionPattern value=”%X{uidsubmission}” />
        </layout>
      </parameter>

while valid for a simple type like an int, may generate an exception if it is a nullable or reference type.  This is because patternlayout infers a cast to string when binding parameters.  If the value of the parameter is null, that tostring cast will fail.  They make no mention of that in the patternlayout documentation that is here.  For reference types and nullables you would be better served by using a rawpropertylayout.  This binds directly to the datatype and does not do any casting.  The syntax is a bit unusual (it helps to bind to the log4net.xsd so you can see how to use this).  An example of such a parameter is below:
   <parameter>
        <parameterName value=”@DischargeDate”/>
        <dbType value=”DateTime” />
        <layout type=”log4net.Layout.RawPropertyLayout”>
          <key value=”dischargedate”/>
          </layout>
      </parameter>    
The error you will get in your log4net logs will look like this:
og4net:ERROR [AdoNetAppender] Failed in DoAppend
System.Data.SqlClient.SqlException: Error converting data type nvarchar to datetime.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at log4net.Appender.AdoNetAppender.SendBuffer(IDbTransaction dbTran, LoggingEvent[] events)
   at log4net.Appender.AdoNetAppender.SendBuffer(LoggingEvent[] events)
   at log4net.Appender.BufferingAppenderSkeleton.Append(LoggingEvent loggingEvent)
   at log4net.Appender.AppenderSkeleton.DoAppend(LoggingEvent loggingEvent)

MsBuild, CCNet, VS2008

If you ever want to get this all working. Please ignore all the blogs that try to indicate that somehow you can use a custom task library or maybe some feature of msbuild to do deployments. The best blog I have found on getting this all to work including deployment is here. Lets face it. Unless you have Team Foundation running, Microsoft wants you to jump through hoops to get continuous integration to work. Depend on CCNet and ignore having too much in MSBuild.

Creating Custom Activities can make you rich

After pulling out my hair to understand what it takes to make an activity without code-behind appear in the toolbar of Visual Studio 2008, I resigned myself to inevitability: ○ Unless you want to write your own XOML then you better use the code-behind format if you want to create a workflow. ○ Creating a custom activity is easy. Creating a custom activity that will have behavior and attributes like built in activities will take a little more time. ○ Most things you want to do in workflow can be accomplished using the default activities so don’t create your own custom activities if you don’t have to. ○ There is no such thing as truly dynamic properties in workflow. You will have to use any of the patterns for state (like State for example). I’ve finally been able to coble together a workable solution for a client that needed dynamic logic and dynamic variables. They also wanted to create the workflows graphically. The solution we proposed was to use custom activities that are arranged into a workflow using Visual Studio 2008. The clients would drop our custom activities onto a Sequential Workflow with code behind. They would then use the Rules Editor to define conditions that were a requirement of the Decision type activities. The client would then automatically create a XOML and a .rules file directly from inside Visual Studio 2008. The XOMl, .rules, and our activity library dll would be all that is required for another developer to create our workflow inside any Workflow container. This is powerful stuff because that is allowing custom logic and rules to be embedded into any system with

XML source and multiple data outputs for elements.

If you have ever had to deal with a typical XML file for data loading for ETL tasks in SSIS then  you have encountered its default behavior of creating an output for every element.  This is not the nicest behavior because it is doubtful that anyone would have a database so normalized.  While working on such an issue we discovered a useful trick.  First off, SSIS has not left you in no mans land.  If you have generated outputs for each element you can write all the elements to a Data Destination and use a SQL join statement to reconstitute the data in whatever way you want.  SSIS automatically adds IDs to all the tables it creates so that you can do join statements based on the hierarchy of the XML.  For example, if your xml hierarchy looks like this:

Parent Element->

                                Child Element->

                                                              Inner Child Element

 

Then the tables that SSIS would create will have an ID element for InnerChildElement that makes it joinable to Child Element and an ID field inside Child Element that makes it joinable to Parent Element.  This can be a life saver as you will just have to do one join to get the data back in a form that is interpretable as hierarchy.  This added ID field allows a second possibility:  Using Merge Joins on the output to create denormalized tables.  The SSIS Team actually have a blog entry that goes over this (http://blogs.msdn.com/mattm/archive/2007/12/11/using-xml-source.aspx).

Long story short, XML source does exactly what you will want it to for importing XML data.  That and the XSD validation capability of XML Task (note watch your constraints after the XML task to ensure that if you are validating xml you will get the proper success result) makes SSIS a complete XML handling solution.                    

 

SubTyping in ActiveRecord isn’t too clear.

I’ve spent quite a lot of time messing around the methods for subtyping in ActiveRecord.  The clearest method I’ve found is using concrete classes (what Fowler calls them).  For this to work you simple have a inheritance relationship between the base type and the subclasses and have the AR attribute on all the classes (I get a weird error if the base class doesn’t have this attribute).

E.g. 
[ActiveRecord(“BaseType”)]
public class BaseType
{}

[ActiveRecord(“FirstSubType”)]
public class FirstSubType:BaseType
{}

[ActiveRecord(“SecondSubType”)]
public class SecondSubType:BaseType
{}

You end up with a little bloat in terms of sourcefiles and database tables, but, at least you have a very clear subtyping relationship.  I had wasted a lot of time with the AnyToAny relationship syntax.  Note:  There is no documentation whatsoever for how to use AnyToAny relationships in AR.  I reccomend searching the Subversion repository test cases for examples of how to use that.  Even the forums lacked any good description of how to code it.  Honestly this is one of those features that I would almost switch to NHibernate if I can’t work it.

More on Monorail

More on Monorail

          

Why are people still using ASP.Net WebForms?
 

Finally cooking with Monorail and ActiveRecord has come to the party.  Can’t believe that I just got log4net working yesterday and I’ve had log messages  in code from the start.  Turns out that you have to take the log4net config out of the web.config and put it in its own log4net.config file in the same directory as web.config.   I still haven’t figured out how to turn on event logging appender yet, but, I’m fine with file logging for now. 
Man, when it all comes together its a thing of beauty.  I’m making use of the paginationhelper now.  I have to use the caching one because of multiple search params, but, I could get away w/out the caching one if I had to.  I should be able to get deep into the SCORM stuff later today after cleaning up one or two things.  I’ve been mostly happy with everything except that I’m using sourcesafe.  2007 and Sourcesafe is still a hot mess.  On the next project, I’m definetly switching to subversion.   Friends don’t let friends sourcesafe. 

Thoughts on software

.Net web development frameworks

          

Why no one uses webforms
Castle Framework   Who gives a damn about viewstate

Went through the usual suspects.  I was looking for something that supported MVC, TDD, domain driven design and had nhibernate support.  Castle handily has support for all this.  Took a look at the spring project, but, I’m not that into aspect oriented (Can you name one major aspect oriented developed project?).  Are there too many frameworks and methodologies out there?  There is one train of thought that anything worthwhile is necessarily simple in its makeup and explanation.  I agree with this except I don’t think the proper methodologies are emphasized in computer science.  People are very caught up in buzz words, but, they don’t realize that there are a lot of people working on ways of making their work easier and worthwhile.  That it comes out in a variety of shapes and forms, is not an indication of merit, but, a reflection of how many smart people there are working on all this.  A few things that can help the development of maturity in software engineering:

  1. The cathedral and bazaar conflict has to leave the software engineering mindset.  This is a concern that is outside of our work.  We make the products.  Let the bankers, financiers and the like figure out how to sell it.  Software Engineers can no more impact the software engineering marketplace than automotive engineers can impact the automotive marketplace:  sure, we can design new forms and create new trends, but, when it comes down to it, its all about what some individual wants and how much they are willing to pay for it.
  2. The software engineering licensing movement has to move forward.  There are too many kids with bazookas out there.  In the last 60 years we have managed to create a wasteland of broken promises and even more broken systems.  A certain seriousness has to enter into the approach to software development.  Every day, system criticality increases as IT drives more and more of human endeavor.  We have to get serious about creating good software and to do that we need good software engineers.
  3. Intellectual property has to be recognized as an asset of the species rather than as corporate asset.  This may seem to contradict point one, but, let me clarify.  I said nothing about making all software free.  I believe developers should be paid for their work.  However, there must be a new model for what happens to software once its developers have received payment for their work.  Perhaps, we should all agree on some fair use clause that kicks in after X amount of years.  I am very opposed to the idea of perpetual copyrights to software.  Software has to be seen as a human intellectual product.  As such, it should only be protected for a limited amount of time and then released like all other human intellectual products.  The benefits to humanity of fair use far outweigh any financial incentive that any single developers every see in their lifetime (Bill Gates included).  Of course, this may mean a radical adjustment to the economic model for software.  However, this is happening as we speak anyway.  Software as a Service is the latest incarnation of transformations that are already happening.