Category: Development

Navigating the AI Revolution in Coding: Promise, Potential, and Pitfalls.

AI-based code generators, like GitHub’s Copilot and various other popular tools, have shown impressive capabilities in assisting developers with coding tasks, intelligently. Previously I wrote an article on how we can use GitHub CoPilot as an AI pair programmer for faster and more effective programming. As we basked in the AI’s potential, skepticism arose. Questions of reliability, security, impact on employment, creativity.

What could be the Pitfalls ?

Code Quality & Reliability: The AI-generated code isn’t perfect and can overlook coding best practices, leading to verbose or inefficient code. There’s also the risk of planting unintended bugs, an inevitable coding nightmare.

Security Threats: Technological advancement brings along potential security vulnerabilities. AI’s code suggestions might fall into this category, and the risk of sensitive data leakage is undeniably a concern.

Skill Erosion: We’re at risk of losing essential coding skills with AI in the driver’s seat. It’s akin to using calculators in math classes, where students may memorize formulas but miss out on understanding the underlying concepts.

Ethical & Licensing Dilemmas: Copyrighted code is an intellectual asset. An AI suggesting a similar line of code raises matters of intellectual property rights, introducing ethical and licensing dilemmas.

Bias: AI is only as good as the data it’s trained on. If the training data carries biases, the AI-produced code might inherit them, causing ethical issues.

Economic Impact: Technological progress has consequences. In this case, the possible reduction in need for entry-level programming jobs, which can impact the tech industry’s employment chart.

Trust & Dependency: A growing dependency on AI might lead to blind trust causing issues in the final product. It’s a case of ‘too much of a good thing.’

Stifled Creativity: Innovation and creativity are yet human realms. AI can’t whip up innovative or off-the-wall solutions. Are we stifling our creativity?

Understanding the Promise

Any modern technology brings both excitement and skepticism in the beginning however as time passes, society often adjusts to its presence, weighing its benefits against its drawbacks, and finding ways to integrate it into everyday life. To understand and verify AI’s promise to modern coding, an experiment and study was conducted by Cornell University in 2022, whose results were published as a paper in Feb 2023:

  • Objective: This research focuses on the potential of generative AI tools to boost human productivity.
  • Methodology: A controlled experiment was conducted with GitHub Copilot, an AI-assisted programming tool. Developers were tasked with quickly implementing an HTTP server in JavaScript.

Experiment Setup: During the trial, developers were tasked to implement an HTTP server in JavaScript as swiftly as possible. The treatment group had access to GitHub Copilot, and was given a brief introduction video. In contrast, the control group, which didn’t have Copilot access, was unrestricted in other ways (e.g., they could use online search or Stack Overflow).

Any modern technology brings both excitement and skepticism in the beginning however as time passes, society often adjusts to its presence, weighing its benefits against its drawbacks, and finding ways to integrate it into everyday life.

Results:

Those who had access to GitHub Copilot (the treatment group) finished the task 55.8% faster than those who did not (the control group). The performance difference between the two groups was both statistically and practically significant. The confidence interval for this measurement is between 21% and 89%. This means that based on the sample data and the statistical analysis performed, we are reasonably confident that the true performance improvement of the treatment group (with Copilot) over the control group is somewhere between 21% and 89%.

  • The blue bar represents the control group, which did not have access to GitHub Copilot.
  • The green bar represents the treatment group, which had access to GitHub Copilot. This group finished the task 55.8% faster.
  • The black error bar on the green bar represents the confidence interval for the performance improvement, ranging from 21% to 89%.
  • If the treatment group was 89% faster, then they would take only 11% of the time the control group took.
  • If the treatment group was 21% faster, then they would take 79% of the time the control group took.

Diverse Effects: The AI tool had varied impacts based on developer backgrounds. Those with less programming experience, older developers, and individuals who spend more hours programming daily benefited the most. This suggests that such tools can expand access to software development careers.

Developers with Less Programming Experience (Junior Developers):

  • Observation: Junior developers or those with less programming experience derived significant benefits from the tool.
  • Implication: GitHub Copilot likely provided these developers with code suggestions and solutions that they might not have been aware of or would have taken longer to derive on their own. This assistance can accelerate their learning curve and increase their efficiency.

Older Developers:

  • Observation: Older developers benefited more from the tool compared to their younger counterparts.
  • Implication: The reasons for this can be multi-fold. Older developers might have vast experience in programming but may not be as updated with newer frameworks, libraries, or best practices. GitHub Copilot can bridge this gap by suggesting modern code solutions. Alternatively, older developers might be more methodical and take advantage of AI assistance to validate or enhance their code.

Developers Who Program More Hours Per Day:

  • Observation: Those who spend more hours programming daily saw more benefits from the tool.
  • Implication: Developers who are more engaged in coding tasks can leverage GitHub Copilot more frequently, extracting maximum utility from its suggestions. It could serve as a continuous companion, optimizing their workflow and reducing the time spent on debugging or searching for solutions online.
A group of deligted programmers …..generated by AI using Midjourney

Other Observations: The paper mentions heterogeneous effects, suggesting there might be varying results based on different conditions or types of developers. There’s also a mention that AI pair programmers might be helpful for individuals transitioning into software development careers.

Affiliations: The authors are affiliated with institutions like Microsoft Research, GitHub Inc., and MIT Sloan School of Management.

Limitations of the study

The research is in its early stages, involving a select group of participants working on a non-commercial project. As such, the product is not designed for public or production deployment. Therefore, concerns regarding licensing, legal implications, and security were not factors considered in this study. Although crafting an HTTP server with JavaScript presents its own challenges, real-world commercial endeavors often encompass intricate systems, necessitating collaboration across diverse teams. In a typical enterprise environment, developers do more than just code solutions for business challenges; they also craft various test cases, from unit to integration tests. Consequently, this study does not evaluate metrics such as code quality or testability.

Harnessing the Potential

Given this brief overview, it seems the study provides empirical evidence for the positive impact of generative AI tools, on developer productivity. The resultant surge in productivity is not merely a metric; it translates to valuable time that can be redirected towards innovative, business-centric endeavors rather than being mired in the repetitive grind of manual coding. While these AI tools aren’t silver bullets for intricate projects, they serve as adept and intelligent CoPilots. For organizations to truly harness this potential, it’s imperative for leadership and engineering heads to foster a culture of vigilant oversight, continuous training, and rigorous code reviews. This proactive approach can not only maximize the advantages of these tools but also address the challenges highlighted earlier in the discussion.

Happy programming !

Resources

Here is a link to the study paper available on the web:

Cornell study: Impact of AI tools on Developer productivity

Please note there is a link to the downloadable PDF at the right hand top.

Exploring GitHub Copilot, Ask Copilot

GitHub Copilot is the new AI powered programming tool, which can be used to assist with various programming tasks within the development environment you are using. The AI tool gets integrated within your code editor and provides assistance throughout, without leaving the code editor window. Rightly named, it functions like a co-programmer pairing in real time- creating, enhancing and documenting code.

It is based on OpenAI Codex, a powerful artificial intelligence system that can generate code from natural language prompts. OpenAI Codex is trained on billions of lines of code from public GitHub repositories and other sources, so it can understand the context and syntax of different programming languages. Copilot uses OpenAI Codex to suggest code snippets. Open AI Codex can understand various programming languages, frameworks, and domains, and can even answer questions about code. It is a powerful tool for developers, learners, and anyone who wants to create software with ease and fun.

The above image was created with DALL-E, prompt to DALL-E was:
AI pair programming with Github copilot anime cartoon

Copilot can help you with various coding tasks, such as writing functions, tests, loops, queries, algorithms, and more. It can also generate comments, documentation, and examples for your code. It is not a replacement for human programmers, but rather a tool that augments their capabilities and creativity. It is still in technical preview and may not always produce correct or optimal code. You should always review and test the code generated by CoPilot before using it in your projects. You should also follow the best practices and guidelines for your programming language and framework, as well as the license and terms of service of CoPilot.

I explored it with Visual Studio 2022 and C# code, however Copilot works with various code editors and programming languages. As for setup, it works as an extension for Visual Studio and Visual Studio Code. At the time of this writing, you have to install it with Visual Studio 2022 version 17.4.4 or later and follow some steps. GitHub and Microsoft have several articles on how to set it up with Visual Studio, however at the end of this article I have listed links which will help you set up GitHub CoPilot with Visual Studio.

Briefly here are the steps.

  • Sign up for the GitHub Copilot technical preview waitlist at https://copilot.github.com/. You will receive an email from GitHub when you get access to the preview.
  • Install Visual Studio 2022 version 17.4.4 or later from https://visualstudio.microsoft.com/downloads/.
  • In Visual Studio 2022, go to Extensions > Manage Extensions and search for GitHub Copilot. Download and install the extension.
  • Restart Visual Studio 2022 and authorize the GitHub Copilot extension with your GitHub account.
  • Accept the telemetry terms to join the GitHub Copilot technical preview.
Created by DALL-E Prompt used : “Step by step instructions to install represented as digital art

Once you have done these steps, you can start using Copilot to write code faster and smarter. You can use natural language comments or method names to trigger suggestions, and press Tab to accept a suggestion. You can also use the Copilot panel to see alternative suggestions or tweak the settings. Copilot will suggest code and entire functions based on your context and comments. You can press Tab to accept a suggestion, or keep typing to ignore it.

The most useful feature of GitHub Copilot is Ask Copilot- which you can initiate by Right clicking on the code editor to show the context menu which will present the Ask Copilot context menu option.

This is where it gives you the experience of ChatGPT embedded within your IDE and Code editor helping you add, refine and document code.

GitHub Copilot is not a replacement for human programmers, but rather a tool that augments their capabilities and creativity. It is still in technical preview and may not always produce correct or optimal code. You should always review and test the code generated by GitHub CoPilot before using it in your projects. You should also follow the best practices and guidelines for your programming language and framework, as well as the license and terms of service of GitHub CoPilot.

For example some prompts I tried are:

Create a Startup class for this project

Refactor Program.cs to use the Startup class

Add a xunit based unit test for my StreamController class

Create a new class Sale which has properties Item and Amount

The above prompt creates a Sale class with the properties and relevant types.

As you can see in the image above you get an option to either a Create new file or Insert at Caret. Create a new file adds the class to the project.

Insert at Caret will insert the code where the cursor is on the editor.You can keep enhacing and improving the same code.

Then I asked “Add a property Active to the class Sale” and it produced the output as in the above image.

It automatically inferred Active as a boolean and allows to insert to the class. While this is a simple and straight forward feature it can save typing time by creating classes, interfaces and other code, and add them to the project or existing code. This takes the drudgery out of programming by automating them and saving time for harder and more complex programming.

Here are some of the other tasks I was able to accomplish:

  1. Created multiple API controller classes and added constructors with dependency injection and added Get methods.
  2. Added XUnit based test code for the controller classes.
  3. Asked the copilot to explain code, framework classes and create documentation.
  4. Improved code to make it more succint.
  5. Asked to explain the meaning of a code block.
  6. Given a block of code, wrap it around a try-catch block.

Another task I asked Copilot to do was code review the below class:

My prompt was:

Do a code review of the class StreamController

Here is the class:

[ApiController]
    [Route("[controller]")]
    public class StreamController : Controller
    {
        private readonly OpenAIAPI _openAIApi;
        private readonly Conversation _conversation;    

        public StreamController(OpenAIAPI openAIApi)
        {
            _openAIApi = openAIApi;
            _conversation = _openAIApi.Chat.CreateConversation();
        }

        [HttpGet("{inputParam}")]
        public async IAsyncEnumerable<string> Get(string inputParam)
        {

            _conversation.AppendUserInput(inputParam);

            var sb = new StringBuilder();

            await foreach (var res in   _conversation.StreamResponseEnumerableFromChatbotAsync())
            {
                sb.Append(res.ToString());
            }

            yield return sb.ToString();

        }
    }
}

Here are the code review comments by Copilot:

I also explored using natural language commenting for it to suggest a code snippet. For example as shown in the screenshot below I typed a comment in the code as here below :

//This method makes a Post call to OpenAI api and returns the response

The Copilot typed up the rest of the code for me line by line as I accepted it by pressing Tab.

Some tasks I could not accomplish were like adding a Unit test project to an existing Solution. While it can generate unit test code for individual classes and methods it cannot add a scaffolded unit test project to the existing solution. I was hoping that it can create a unit test project and add the generated classes to it. I was also not able to refactor an existing Program.cs to create a new Startup, add that to the project and refactor it appropriately to use that Startup class without me interving manually to add the services to the service collection and other start up code. While it can understand the context of the program and create code and classes based on that it cannot accomplish multiple steps of refactor or creating a basic templated project and adding generated code to it.

It’s hard to say whether some of these tasks are possible or not in the future based on how the models are trained, however I am hoping there will be many more automations to come which will eliminate the drudgery out of repetetive and mundane coding tasks and make them more fun, faster and easier to accomplish. It is also an extremely useful tool to learn programming and understand frameworks, sdks and APIs you are using to build your solutions.

The fact that you can remain within your code editor and find all the answers to the questions without having to switch to google search, ChatGPT, Stack Overflow etc. is a tremendous time saver – plus the added benefit of generating code and adding to the ongoing code base. There is already a lot of interest around Copilot, many companies have made this a part of their day today programming.

These are some of my initial experiments with it. I used a simple and brand new project to explore the tool and some of it’s features. It would be interesting to see how this can be used to review and optimize existing complex code. I am still in the process of exploring it further, and give the right natural language comments and the right prompts to Ask Copilot.

I highly recommend you use this for your programming projects and efforts.

Happy programming !

Some useful resources:

(1) GitHub Copilot in Visual Studio 2022 – Visual Studio Blog.
(2) Quickstart for GitHub Copilot – GitHub Docs.
(3) Getting started with GitHub Copilot – GitHub Docs.

(4) CoPilot chat feature private preview

Unit testing: climbing the Uphill road

This article is not about discussing the basics of unit testing or simplifying the concepts, nor it is about merits of the practice. Unit testing methods have evolved quite a bit, with many books and online content. However, the real world practice of Unit testing can be an uphill task in several contexts, due to different reasons, even after how the ecosystem around Unit testing has matured quite a bit.

Unit testing is both an Architectural as well as DevOps concern. Tactically both these departments strive to get the software development operations to effectively incorporate Unit testing.

What are some of the challenges ?

  1. Brownfield vs. Greenfield : Unit testing is easier to adopt when you are starting a new application. The reason being an application needs to be written in a certain way, applying principles like S.O.L.I.D to be able to Unit test it effectively. There are a few key programming knowledge areas which need to be understood well before starting Unit testing correctly, like Interface driven programming and Dependency Injection. The industry is full of so called ‘Brownfield’ applications or simply existing production applications which may not have been written using some of the best practices. These Brownfield applications can become major barriers of entry into Unit testing. Software Engineers who are involved day to day supporting these applications often find it difficult to introduce Unit testing into them because they need to be refactored before doing so.
  2. Time : Unit testing applications can be time consuming, requiring additional development time to be factored in. Often this upfront time is seen as additional cost to development without realizing that, there could be long term benefits like reduced QA footprint and automated regression tests. Hence a technical leadership which creates a culture of Unit testing by coaching the stakeholders on benefits of Unit testing has to be fostered by organizations.
  3. Training: To Unit test effectively, software engineers need to have knowledge of Unit testing basics like creating Mocks, Code Coverage and the AAA pattern ( also known as Arrange, Act and Assert ) . The lack of this knowledge itself can be a barrier even if the first two barriers do not exist. Hence training on Unit testing fundamentals and the available Unit testing frameworks needs to happen.
The Uphill task of Unit testing

The industry is full of so called ‘Brownfield’ applications or simply existing production applications which may not have been written using some of the best practices. These Brownfield applications can become a major barriers of entry into Unit testing. Software Engineers who are involved day to day supporting these applications often find it difficult to introduce Unit testing into them because they need to be refactored before doing so.

All the above challenges need to be overcome to create an atmosphere and mechanism of regular Unit testing.

How do we overcome the challenges ?

  1. A brownfield application which does not lend itself to Unit testing through it’s design will have be refactored first – piecemeal by piecemeal to facilitate the testing. Refactoring can include anywhere from making the code more object oriented, removing deep coupling, adding dependency injection and creating interfaces. Brand new code within the project can be written keeping Unit testing in mind. The more monolithic and deep coupled the code, more refactoring has to be done. It is best to chew at it slowly – over a period of time code will improve. Scrapping a brownfield completely to start a new greenfield application is not always an easy decision. Hence refactoring existing code for improvement one step at a time, can be more cost effective and beneficial.
  2. ‘Time’ is the one of the most negotiated element during software development. Most stakeholders like Product Owner and end clients will want the application deployed ‘yesterday’. In such a situation anything that adds more time to development can mean negotiation, conflict and stress. It is very normal for software engineers to give up on Unit testing when time is a constraint. A strong technical leadership committed to building consensus over Unit testing is critical to overcoming the time constraint- time spent upfront will save time later. The most significant benefits are as stated earlier Reduced QA time, Automatic regression and Better product quality.
  3. Unit testing does not come naturally as part of software development. It is still a habit which has to be developed and cultivated- training is also required to understand techniques of Unit testing. Lack of knowledge itself can inhibit Unit testing to happen even if the above two situations don’t exist. Often the difference between Unit testing and Integration testing are not understood. A culture of training and learning Unit testing has to be created. Team members who are ahead in Unit testing can be appointed as ‘coaches’ in Unit testing. Unit testing demos, presentations and discussions must be organized regularly to create the energy and enthusiasm around it. Technical leaders must ‘call out’ above and beyond Unit testing efforts; Incentivize or reward Unit testing efforts. An overarching policy of ‘incremental’ code coverage can be set, gradually increasing percentage code coverage. As needed ‘time’ must be allocated for the purpose of unit testing within the development cycles with the help of project managers.

A 100% code coverage may or may not be necessary based on the type of code or type of project. It’s a great goal to have, however it’s more important to test code for where rules, algorithms, calculations and complex logic need to be tested thoroughly. If time is a constraint decide where there is more value in Unit testing, and Unit test that part of the application.

An overarching policy of ‘incremental’ code coverage can be set, gradually increasing percentage code coverage. This way as the team matures into Unit testing, code coverage requirements are increased. Team members who are ahead in Unit testing can be appointed as ‘coaches’ in Unit testing. Unit testing demos, presentations and discussions must be organized regularly to create the energy and enthusiasm around it. Technical leaders must ‘call out’ above and beyond Unit testing efforts; Incentivize or reward Unit testing efforts.

A culture change like Unit testing can only happen when there are strong technical leads committed to the quality of the software, willing to go above and beyond to achieve that excellence. They have to be leaders willing to see beyond self-interest and work towards consistently evangelizing the practice.

Collaborating teams build systems that collaborate

Information systems of a company are a reflection of the communication patterns within the organization. The more teams communicate with one other cohesively, these systems will connect and grow together more as a result. In today’s world of digital transformation and data intelligence, it is imperative that multi function organizations communicate and collaborate, so systems built by the development staff will bring the necessary business intelligence and scale. This cohesive intelligence is critical to further growth. It is very common in multi function organizations for ‘Silos’ to develop over a period of time. These ‘Silos’ focus successfully on one aspect of the business, however they fail to connect with other systems in the workflow such that it reflects the customer journey. This ‘disconnect’ can create many black-boxes producing their own data, which as a whole brings minimal value in the long run.

It is very interesting that many popular methodologies and architectural patterns have evolved which fundamentally solve this problem of communication. Let’s take DevOps for example. DevOps is designed to have better collaboration between Development teams and Network operations. Although they are independent in function, lack of communication between them results in surprises, errors and continuous finger pointing. This can be very expensive operationally leading to client dissatisfaction. Today DevOps has had great success which leads to test, build and deployment automation. This in turn increases speed-to-market and system reliability.

In today’s world of digital transformation and data intelligence, it is imperative that multi function organizations communicate and collaborate, so systems built by the development staff will bring the necessary business intelligence and scale as well.

Agile development methodologies solve the collaboration problems between all disciplines of an organization. The SCRUM methodology with it’s different ‘ceremonies’ brings all relevant departments together in one room regularly. From Sprint Planning to Sprint Retrospective, different departments like products, development, QA, and user experience come together to discuss topics from requirements to ‘lessons learnt’. This collective, methodical way of communication leads to better product definition, risk mitigation, estimation and release planning. Everyone is in a continuous improvement and feedback loop, improving deliveries significantly.

collaboration

Let’s take Micro-services architecture, for example. This style of architecture was embraced by many companies in an effort to have small teams become autonomous by fostering a loosely coupled architecture connected via independent services. Again, at a high level team structures and their communication patterns dictate how the Enterprise architecture shapes. Netflix adopted the Micro-services architecture to increase speed-to-market which led to a social culture of small autonomous teams collaborating together.

Companies don’t just build products and offer services , they build professional communities and offer personal growth.

Any organization which looks to build information systems that work together, while they are independent units, has to build a communication culture and team structure which can facilitate such systems. Cohesive systems bring business intelligence while loose coupling allows them to scale. Cohesion comes from continuous collaboration and loose coupling comes from small autonomous teams. Adopt methodologies and architectural patterns which bring teams and people together. Companies don’t just build products and offer services , they build professional communities and offer personal growth.

Program to an interface , not an implementation

I have been lately discussing SOLID in my previous posts – some of the basic principles of OOD . Recently while doing my project work , we had to decide between Entity Framework  or NHibernate which are popular ORM layers in .Net framework .  The decision was not easy , we got stuck at Second Level caching support …and could not reach any decision right away. I started thinking that may be for now we have to write our layer in such a way that we choose one ORM , and then we can switch to a different one worst case if need be. Ideally a situation like this should not arise , but if it does happen , you need to be able to do it.  Once again I felt how a very fundamental OOD principle could come in handy –

Program to an interface , not an implementation.

These words above can be very ambiguous and abstract when you just read them .

However when seen from a context , you understand how powerful this is , and how it can increase flexibility and ability to switch . Seen from my own situation where we needed the flexibility to switch data access mechanism if need be, let’s understand this by constructing a scenario. For example , a common problem could be implementing a subscription mechanism for a data service that your system may provide. Users can subscribe to the data service and get notified when a change occurs. We need to persist the subscription information to the database , where you will need to use some database access mechanism.

As usual there is a presentation layer that gathers some subscription information and passes it along to the data service which persists the information to the database . For simplicity’s sake , let’s say we have a SubscriptionService , which is our data service for persisting subscriptions . We can discuss this further in the context of an ASP.Net MVC application – where a View is our Presentation layer through which Subscription details are collected. A controller action method is typically invoked to pass along the information to the server.

We can assume for our article purposes that our application provides notifications on Food Recalls. So the member user to our website subscribes to recalls for a certain type of food issued by different organizations like FDA etc. The Subscription in simplicity could be :

Please note : all code is pseudo only , this is not a working solution that can be downloaded and built.

public class RecallsSubscriptionService
{
    Boolean Subscribe( int userId, string food_type, string organization)
    {
           // data access code to insert subscriptions
           return false; 
    }         
}

Most of the times the controller action method would look something like this:

public class SubscriptionController : Controller
{
        [HttpPost]
        public ActionResult Subscribe(int userId, string food_type, string organization)
        {
            try
            {
                 SubscriptionService myservice = new SubscriptionService();
                 myservice.Subscribe(userId, food_type, organization);

                return RedirectToAction("Index");
            }
            catch
            {
                return View();
            }
        }
}

So , we make a decision we will use Entity Framework 6 to implement our data access to store subscription information. The above code will work fine . The code goes into production . However later we decide that due to certain project requirements we have to switch to NHibernate. This is a tough situation – our controller here , who is a client of the SubscriptionService , is heavily dependent on instantiating this specific service which accesses the database with Entity Framework 6. Although we expected the problem could arise we didn’t design our system for this flexibility. Even though we encapsulated the data access code into a Service , we did not free the client from the implementation of data access because the client program is indirectly tied to EF implementation.

We need to make some changes , so our controller is independent of the future changes that could happen to the database access mechanism. To bring transparency into how data access may happen from a client’s perspective we need to introduce interfaces. We write an interface instead for the SubscriptionService :

interface IRecallsSubscriptionService
{
   Boolean Subcribe(int userId, string food_type, string organization);       
}

We could write two implementations of the above interface or we could just use the same implementation we had earlier , and replace it with NHibernate.

public class RecallsSubscriptionServiceEF
{
       Boolean Subscribe(int userId, string food_type, string organization) 
       { 
                //   implement using EF ;
                return false;
       }         
}

public class RecallsSubscriptionServiceNH
{
       Boolean Subscribe(int userId, string food_type, string organization) 
       {
              //implement using NH ;
              return false;
       }         
}
/*( not that the above approach is recommended , where you keep 
two implementations available for the same requirement. 
This is for understanding purposes only )*/

The above mechanism allows us to drive the Subscribe method call through the IRecallsSubscriptionService interface. In order to effectively use the method above , we will need to pass IRecallsSubscriptionService as a parameter to the controller constructor and use dependency injection in this scenario to have the right concrete class to be instantiated during run-time. I will not dive deeper into that because it is outside of the scope of this topic.

This is the whole basis of Dependency Inversion and Injection ( this is also the ‘D’ in SOLID ): here we program to an interface , not an implementation as we saw above. This gave us the flexibility to change our implementation completely without changing the client code. So the basic idea is programming to an interface decouples the client from the internal implementation freeing calling programs from knowing the internals leading to flexibility in changing implementation as the need arises.

Even in scenarios where there are published APIs , when you use objects  go up in hierarchy to program to the interface as opposed to the concrete class. So if you are using a List object and comparing individual strings-  then if this object implements IComparer , then program to IComparer –

instead of writing:

List myList = new List();
myList.Compare( myList[0] , myList[1]);

//Write as :

IComparer myList = new List();
myList.Compare( myList[0], myList[1] );

//This gives you the freedom to use a different flavor of Compare method if you need to ,

IComparer myList = new Array();
myList.Compare( myList[0], myList[1] );

The advantage of the above is ability to switch different comparison methods if they are available. And also test program can easily switch different implementations and see what works the best.

We also need to understand that “Program to an interface” should not be taken literally . Even Abstract classes can act as interfaces in this context and give us the same flexibility.

Interfaces should be used wisely and only when needed. If there is no potential for an implementation to ever change there is no need to bother with interfaces. For example , a Utility class that converts a string to a number. There is no need to write interface to this . Interfaces should also be used in projects where testability is a big concern and mocking may be required.

SOLID conclusions with ISP and DIP

In my last post we went over LSP ( Liskov Substitution principle ) , how it helps achieve Subtype Polymorphism. We learnt how to think in terms of client programs and how client’s call to  programs can influence the quality and design of applications. The SRP and OCP are foundational principles on which LSP and the next one we discuss today , ISP manifest themselves.  It is not enough if Single Responsibility and Open Closed are understood – the rest of the three principles are where you see their applications and extensions. LSP , ISP and DIP all three teach us how to design from a client’s point of view. Uncle Bob calls it as “Clients exerting forces “  . 

The interface-segregation principle (ISP) states that no client should be forced to depend on methods it does not use

ISP helps formulate mechanisms on how interfaces should be segregated so the same interface can be useful to different clients. It’s a very powerful thought that can bring strong results. Single Responsibility talks about writing interfaces that have very cohesive functions . ISP takes it one step further and gives concrete ideas that give rise to reusable interfaces . This is where the relationship and the difference between SRP and ISP exists .

So how does ‘Segregating’ Interfaces help the clients ? Interfaces are meant to represent a certain type of behavior to the client and by virtue of their contractual representation promote pluggability. So when interfaces represent more than one type of behavior , they provide more than what a client needs . This leads to the interface becoming incapable of being used for plug-in purposes. Also they become ‘Fat’ in nature , with unncessary behavious not useful in the client’s context. Basically this is solidifying more on the SRP and OCP and stressing a whole lot more on how interface contracts should represent one type of behavior. Almost all patterns have a basis in SRP , OCP and ISP. Let’s say we are writing a class to represent a Persistence Medium. One of the most popular persistence medium is Database . So is XML . And a lot of applications use simple JSON files to store simple content.  Say , we start out by writing a PersistenceMedium class. So a simpe IPersistenceMedium looks like this :

Code is pseudo only , to describe the principle , not a working , compiled example :

// IResult is an imaginary interface 

interface IPersistenceMedium
{
       string fileName { get; set; }
       string connectionString { get; set; }

       void Open();       
       void IResult ExecuteQuery(string query);
       void IResult ReadFile();
       void Close();
}

//So our database class can be written like this:

public class Database : IPersistenceMedium
{
    private string _connectionString;

    public string connectionString
    {   get{ return _connectionString; }  
        set{ _connectionString = value; }    
    }

    public void Open(){  /* database open connection code */ }
    public IResult ExecuteQuery( string query );
    public void Close(){  /* database close connection code */ }
}

// The JSONStore can be written like this :

public class JSONStore : IPersistenceMedium
{
     private string _fileName;

     public string fileName
     {   get{ return _fileName; }  
         set{ _fileName = value; }    
     }

      public void Open(){  /* open a JSON file code */}
      public IResult ReadFile();
      public void Close(){  /* close file code  */}
}

See what happened above ? The Database class could not use ReadFile() and fileName and JSONStore could not use ExecuteQuery() and connectionString . Although both are variants of a PersistenceMedium behaviour , they are unnecessarily clubbed together from the perspective of Database and JSONStore classes which are the clients of IPersistenceMedium.

A Better way would be :

interface IFileConnection 
{ 
     string fileName { get; set; }
     void Open();
     void Close();
}

interface IDatabaseConnection 
{   
      string connectionString {  get ; set; }   
      void Open();
      void Close();
}

interface IDBOperation
{
     IResult ExcecuteQuery();
}

interface IFileOperation
{
      IResult ReadFile();
}

Now this provides interfaces that can be specifically used for Database or JSONStore because we segregated them keeping in mind different clients / client groups. The interfaces which can be used for JSONStore can be used for XML or any other type of file store. Similarly the interfaces that got written for Database can be used for any type of database , SQL , Oracle or NoSQL . The interface is less fat , with interfaces ‘Segregated’ strictly based on different client usages . This is SRP in effect, by using ISP.  What are the different ways of using these with the Clients ? One obvious way is multiple inheritance.

public class JSONStore : IFileConnection , IFileOperation
{
    private string _fileName;
    public string fileName
    {   get{ return _fileName; }  
        set{ _fileName = value; }    
    }

    public void Open(){ /* open file*/ }
    public IResult ReadFile(){ /* Read File */}
    public void Close() { /* Close File */ }

}

public class Database : IDatabaseConnection , IDatabaseOperation
{
    private string _connectionString;
    public string connectionString
    {   get{ return _connectionString; }  
        set{ _connectionString = value; }    
    }

    public void  Open() { /* Open Connection */}
    public IResult ExecuteQuery(){  /* execute query to obtain results */}
    public void Close(){ /* Close Connection */}

}

The above JSONStore could be very well be written as FileStore , in that case different FileStores can be used and IResult representing different result types form different types of files.

There are different ways to implement the above without multiple inheritance. One way is to use something like a Strategy pattern where IFileOperation has concrete classes for JsonFileOperation and XmlFileOperation – because each may have subtle read differences. XML requires special parsing and JSON is more of a string representation.

We have been able to establish in many different ways the value of SOLID – how it  leads to good design practices and understanding of patterns.

I will conclude without diving into Dependency Inversion – there are fantastic blogs including Uncle Bob’s report. Dependency Inversion distinguishes itself by establishing how layers should interact  without depending on each other. Again a powerful concept that has led to great programming practices including Test Driven Development and several patterns that facilitate the principle. SOLID is definitely worth your time !

Finding the way to Design Patterns : more SOLID

So continuing into the process of finding our way into design patterns from my last post , we will try and unfold SOLID a bit more. Some of the SOLID principle interpretations and applications can be very subjective and a matter of debate. How do you know the classes that you wrote have adhered to the Single Responsibility Principle ? Is there a way to determine ? What does Single Responsibility exactly mean?  How far do you try and take these things…blah ..blah.

There needs to be a balance in everything of course.One needs to find a mid way between over-engineering and under-engineering. We need to create small classes with very few cohesive/related functions without getting carried away by having so many classes that we cannot manage those either. At the same time , understanding exactly how to create classes with just one relevant behavior or few cohesive functions can get tricky. It’s important to ask the question while writing a class , what is it that will potentially change in this class later , that can be re-factored into a separate class , so minimal changes / or no changes need to be made to the existing implementation when the change needs to be made.

It’s important to ask the question while writing a class , what is it that will potentially change in this class later , that can be re-factored into a separate class , so minimal changes / or no changes need to be made to the existing implementation when the change needs to be made.

If we take the Logging example that we discussed in the previous post , we discussed three functions a logger can do. Initializing a medium to log , Formatting to the medium itself  and Writing to the medium . So, they all look like related functions don’t they ? So they all can go in one class , which is a violation of SRP . If we take one step further into re-factoring  perhaps it’s easy to recognize right away that there are different mediums to log , each requires a different method to write to itself. So we create separate classes for the medium . Yes , we applied SRP here.

What about formatting ?  Can formatting change ? Yes, it can . Say we need to add a new logging medium called Mobile Device. To the mobile device , you want to send text messages whenever severe conditions occur. Doesn’t the format of what goes into a database differ from a mobile device ?  Also , later down the line the potential users of our Logger may require that they want the format of the message a little different. Now if we did not separate the format function into it’s own class , we have it as part of the class that writes to the medium or worse as part of the Logger class. Logger class is actually our client facing class that the client would use to log messages . Just to change  formatting , we have to go and change this Logger class or the medium classes.

At this point we need to consider  whether it’s worth writing something like this:


   1:  interface IFormatter {
   2:   
   3:  public string Format( string message );
   4:   
   5:  }
   6:

and the logger could do something like this :


   1:  public class Logger{
   2:   
   3:  public Write( IFormatter formatter){}
   4:   
   5:  }
   6:

Or let’s go couple of steps further along these lines,


   1:  public interface ILogMedium{
   2:   
   3:  void Write(String Message);
   4:   
   5:  }
   6:

   1:  public class LogToDatabase : ILogMedium{
   2:   
   3:  public void Write(string message){
   4:   
   5:  // medium specific logging
   6:   
   7:  }
   8:   
   9:  }
  10:

   1:  public class Logger{
   2:   
   3:  ILogMedium _logTodb;
   4:   
   5:  IFormatter _formatter;
   6:   
   7:  public void Logger( ILogMedium logToMedium,  IFormatter formatter )
   8:   
   9:  {
  10:   
  11:  _logTodb = DBMediumProvider.Create(); // DBMediumProvider could be a creation class
  12:   
  13:  _formatter = DBFormatProvider.Create(); // DBFormatProvider could be a creation class
  14:   
  15:  }
  16:   
  17:  public Write(String message){
  18:   
  19:  _logTodb.Write( _formatter.Format(message) );
  20:   
  21:  }
  22:   
  23:  }
  24:

Please note : The code above is not a working solution. It is only like pseudo code to demonstrate the thought process .

The advantage of the above is we kept the formatter completely separate from the medium , as well as the Logger class. This allows us to plug in new mediums , formatters and configure them based on our needs. Also , if there are bugs it’s easier to fix a specific class as opposed to one big class which could potentially break other functions.

This brings us to the Open Closed principle , which is the next one and the ‘O’ in SOLID as we all know.

A consistent use of SRP can lead to OCP , which simply states: A class should be open for extension but closed for changes. This does not mean that classes should be completely sealed for change , it means that the class should be in a state where only bug fixes should be made and new functions should be added via new classes , minimizing implementation changes to the existing classes.

Words should not be taken literally here : once you wrote the class does not mean that it cannot be changed at all : however we should get to a point that , when behavior changes in predictable ways, you should not have to make several changes into a single class or several classes in the system. An ideal situation would be where you achieve the change by adding new class / code rather than changing existing code . For anyone who has been in programming for a few years supporting production systems , this will make a lot of logical sense.

Whatever we discussed above with Logger example with reference to SRP will apply to OCP as well , because OCP can be achieved by applying SRP consistently. So had we written the formatting in the Logger or the individual medium classes we would not be able to add new formatters or change existing formatting without changing the Logger or the mediums classes , plus mix and match formatting with mediums . The way we achieved OCP with Logger example is we gave the Logger a Single Responsibility of writing to the medium. We gave the Medium classes a Single Responsibility of initializing / creating the medium and specifically write to that medium as well. And then OCP came into effect when we made it possible for new formatters to be added to the system by simply implementing the interface IFormatter and adding that class to the system as a plugin. So , when a user wants to use the new Formatter , she can do it through the configuration system and that formatter will get automatically used which needs to be implemented through some creation classes of course , which is a separate topic – an example is Factory Method pattern.

Several design patterns use the SRP and OCP , and following the above two will put you in the mode of clean and efficient code . Some popular patterns like Strategy , Factory are all based on OCP . We will continue with this discussion in the next post , happy programming !

Few references :

Wikipedia definition Open-Closed

Interesting read on OCP by Jon Skeet

Finding the way to Design Patterns : SOLID

In my previous post on design patterns I discussed about why it’s a challenge for developers / organizations to adopt patterns as part of their development practices. I also had suggested that a dedicated effort to re-factor code should be made continuously as part of software life cycle especially in the early to mid-stages of development. Having said that, how do you go about re-factoring? How do some of the object oriented principles and design patterns help with re-factoring? So, if re factoring is important, the object oriented mechanisms that allow and facilitate this are important as well. In conclusion the learning needs to happen, slowly and steadily – as a result it will start becoming a habit to incorporate some of these patterns related principles.

I guess the challenge is where do you begin? There is a ton of information on the web, it’s all too overwhelming, scattered and fragmented. You need a have a structure and if you just pick up the Gang of Four book (our bible for design patterns), it can seem rather academic and intimidating for the first timer, abstract as well. The book of course is great, however it’s hard to jump into patterns pragmatically just by reading it.

As in my previous post in my concluding paragraph I had said that getting familiar with SOLID principles developed by Robert. C. Martin is great way to get started with principles that will lead the way towards patterns programming later.  Before expanding upon technically on what SOLID is about, I would like to discuss its importance in the whole OO programming space. If you come from a OO language background like Java, C++, C# etc. you already are familiar with Encapsulation, Inheritance, Polymorphism as foundational principles of OO and they are a daily part of your programming life. SOLID takes it one step further, lays out 5 principles that you can apply to re-factor and improve the code making it maintainable, reusable and efficient.

So when you start applying SOLID, you basically are applying some of the fundamental principles on which Design Patterns are developed. SOLID stands for Single Responsibility, Open-Closed principle, Lishkov Substitution principle, Interface Segregation and Dependency Inversion.

If you just take the first two: Single Responsibility and Open-Closed principle, just there you will improve the structure and quality of your classes.

Single Responsibility states: a class should have only a single responsibility.

Say, you are writing a Log class whose job is simply to write messages to different logging mediums. However you also gave it the responsibility of formatting the messages for these mediums, because it’s part of logging function. In addition, you gave it the responsibility of initializing and choosing the medium to log into, for ex: Event logs, Database, Log files etc. So now the Logger class has multiple reasons to change – one is, Write to the log medium, another is formatting the message: perhaps different mediums require different formatting and on top of that initializing the medium . So your class has now grown into one giant monolithic program that has multiple reasons to change – formatting for each medium, initializing the medium and logging to the medium. So if you need to change only one aspect : say change the formatting messages in database , you have to change that class and the rest of the code could possibly break because you needed to make changes for one aspect. We come across situations like this all the time in production code where we change one thing and potentially something else breaks , without us intending it . This sort of programming makes the code not only fragile, but totally un-pluggable.

Ideally in this situation, the Logger class should just take on the responsibility of writing to the medium being free of what medium it is writing to and what formatting that particular medium needs. Although it seems like they are all related functions , they merit to become individual classes based on their specific function and behavior. More so, ideally you should write an interface that can be implemented to write to different log mediums.

public interface ILog
{
void Write(String Message);
}

As you see , when you just take this one principle and follow it while writing the code you will see that you have written classes that are light weight and each is meant to do one particular job – this makes the design pluggable , reusable and efficient  . You can create classes that are specific to a medium – so adding mediums for logging becomes easy later. Also if there is a change required in it , say how you format the message for that medium, you change only the corresponding medium or the formatting class, avoiding the risk of the rest of the code breaking due to the one change you made in one big class.

As you go further and start applying one by one each principle you will see that certain patterns are shaping up, and possibly they can be applied to common scenarios. I guess I will stop here for now, and conclude that the first step is definitely to understand SOLID and start applying it in your programming life seriously.

Below is a great Video on Single Responsibility Principle by Robert C. Martin himself to learn more on it and get started :

We will discuss more on the rest of the SOLID in the upcoming post …until then happy programming.

Forms Authentication and Membership : Still going strong

ASP.net and .Net framework have come a long way.  Asp.net 1.0 went through a major change with the introduction of ASP.Net 2.0 having several boiler plate features which did not exist in the 1.0 version. Providers were a major component of these features with Membership and Roles added on top of Forms Authentication which was introduced with the 1.x version itself. Since then the combination of Forms authentication with Membership and Role Providers are a strong and proven mechanism that can be used to take care of Authentication as well as Authorization needs of a .Net web application. Together they can be used to Validate a User with Membership and then Create / Manage a Authentication ticket with Forms Authentication.

I just recently started an ASP.Net MVC3 project with .Net framework 4. The MVC3 project template gives you a choice to add all the basic code to implement your security via Forms Authentication / Membership . I have had 4 successful projects with Forms / Membership – 2 were even integrated with SSO. I had all the reason to go for it because I felt very comfortable with it. I must say it took me only at most an hour to configure everything the way I wanted it to work. I did not have to struggle with anything or learn any new technology tricks. I loved that feeling , because every step of the way otherwise you end up learning something new with every version that’s introduced. I don’t consider that a bad thing , however as much as changing technologies are good to have,  stable features give you the comfort and speed required to get the project going. ASP.Net MVC has kept all the goodness except that now the security functions are called via Controller / Action methods.

Forms Authentication and Membership Timeline


Together they can be used to Validate a User with Membership and Create / Manage a Authentication ticket with Forms Authentication.

ASP.Net MVC has kept all the goodness except that now the security functions are called via Controller / Action methods.


If you are using Single Sign On with some kind of federated database , it’s really easy to integrate the two together as well. Just make sure that the credentials which exist in the federated database also exist in the ASP.Net membership database. Once the credentials are authenticated in the SSO database , you take the same credentials and validate them against the Membership database and assign Roles accordingly if you have them. The only catch is if the SSO system creates an authentication ticket that has it’s own expiry , you need to make sure you are signed out of the SSO system when the user signs out of forms authentication system. Forms authentication system creates it’s own ticket – which has it’s own expiry.

Not that the Forms authentication / Membership system is suitable for all systems – several larger corporations have their own home grown authentication and authorization systems. Their custom needs may be complicated enough not be able to use Forms Authentication / Membership . However for all small and mid sized applications, where a reliable boiler plate security mechanism is required , the Forms Authentication / Membership works without any major issues. It is also comforting to see that it is stable enough for Microsoft to have kept it the same way without having the need to make many changes through version 4. Just thought I should discuss this via blog because good to see something stable and strong in an ever changing world of technology.

Technical Knowledge : Breadth and Depth

I could have titled this blog “Professional Knowledge : Breadth and Depth “ – however thought that since my blog focuses on a niche readership,  “Technical Knowledge” would sound more appropriate . Anyway, often times at work discussions come up on how mastering a certain subject area leads to growth , especially for technology professionals. Depth plays a key role in technology professions in leading to both job satisfaction and growth. So is depth more important that breadth? I think both are equally important, and they both contribute to an individual’s job performance and satisfaction.

Why Depth?

1) Depth in a subject area establishes you as an expert – the deeper you drill within a subject area focusing on niche fields the better you will do at the design and execution level. Depth is instrumental in achieving high quality results, thus better performance by you as a professional. Say for example , you develop a mastery on ‘Design Patterns’  with regular practice and study – your expertise in that area could play a pivotal role in robust architecture and solid implementation of the architecture which will take the project a long way for years to come.

2) All technical projects / undertakings are team efforts – not individual heroic endeavors. An ‘expert’ in each area of the technical solution will be required to fill the places as opposed to some one who just knows a little bit of all. For example , a software project may have a ‘Single Sign On’ expert just handling the Authentication aspect of the solution where he/she is the ultimate ‘go to’ person  for it. Hence having a niche expertise helps you build your value within  the organization almost always to fill in that spot of your expertise across different projects. A key to job security especially for those who work as consultants.

3) Depth can easily make you a ‘Mentor’ or ‘Leader’ in your area , very important for your own job satisfaction . Let me give you my own example in this case. All along in my career , I made careful choices while consulting – I chose to take up projects which were ‘ASP.Net and C# ‘ centric – this quickly helped me establish myself as a mentor as well as a lead in those areas allowing me to provide expert help and guidance across projects. This leads to great satisfaction , sense of value and achievement.

“Having a niche expertise helps you build your value within  the organization almost always to fill in that spot of your expertise across different projects.”

Why Breadth ?

1) Breadth gives you a vision , an ability to see beyond the purview of a narrow expertise. For example, a business problem can be solved either with a Client Server Architecture or Service Oriented Architecture. If you do not keep up at least with what SOA means , it’s application at a high level , your problem solving abilities are stunted by lack of broader knowledge.

2) If depth establishes you as an expert , breadth can establish you as a sound thinker. A ‘subject matter expert’ may not necessarily be able to lead the overall project , whereas breadth

sound thinker

helps in being a leader and decision maker for the whole project. A typical software project has BAs, QA , Developers, Designers , Project Managers etc. – a professional with a broad understanding of how each works can listen and guide everyone effectively .

3) Breadth just makes you a lot better communicator – it’s easy to communicate in areas where you are deep , however breadth enhances the conversation power many fold because you can quote examples from beyond your forte. A great asset for meetings , hallway brainstorms and white board discussions. This can lead you to opportunities that you want to explore to grow as a professional.

 “Breadth gives you a vision , an ability to see beyond the purview of a narrow expertise.”

The power of knowledge is immense in IT : having both depth and breadth will round you up as a sound professional as well as a leader. Practice gives depth and study can give breadth , combined the result is a highly valued professional.  I always feel in technology you can ‘Read your way up’ –  a small investment in time of may be even an hour a day makes a big difference , it keeps you abreast with the new stuff.  I think if one has the time , if say you work 9 hours day , may be extending it to 10 hours by adding that extra reading time at work itself will better prepare you with your project work in applying and understanding better techniques.