Development, Object Oriented Programming, SDLC, Software Architecture

Unit testing: climbing the Uphill road

This article is not about discussing the basics of unit testing or simplifying the concepts, nor it is about merits of the practice. Unit testing methods have evolved quite a bit, with many books and online content. However, the real world practice of Unit testing can be an uphill task in several contexts, due to different reasons, even after how the ecosystem around Unit testing has matured quite a bit.

Unit testing is both an Architectural as well as DevOps concern. Tactically both these departments strive to get the software development operations to effectively incorporate Unit testing.

What are some of the challenges ?

  1. Brownfield vs. Greenfield : Unit testing is easier to adopt when you are starting a new application. The reason being an application needs to be written in a certain way, applying principles like S.O.L.I.D to be able to Unit test it effectively. There are a few key programming knowledge areas which need to be understood well before starting Unit testing correctly, like Interface driven programming and Dependency Injection. The industry is full of so called ‘Brownfield’ applications or simply existing production applications which may not have been written using some of the best practices. These Brownfield applications can become major barriers of entry into Unit testing. Software Engineers who are involved day to day supporting these applications often find it difficult to introduce Unit testing into them because they need to be refactored before doing so.
  2. Time : Unit testing applications can be time consuming, requiring additional development time to be factored in. Often this upfront time is seen as additional cost to development without realizing that, there could be long term benefits like reduced QA footprint and automated regression tests. Hence a technical leadership which creates a culture of Unit testing by coaching the stakeholders on benefits of Unit testing has to be fostered by organizations.
  3. Training: To Unit test effectively, software engineers need to have knowledge of Unit testing basics like creating Mocks, Code Coverage and the AAA pattern ( also known as Arrange, Act and Assert ) . The lack of this knowledge itself can be a barrier even if the first two barriers do not exist. Hence training on Unit testing fundamentals and the available Unit testing frameworks needs to happen.
The Uphill task of Unit testing

The industry is full of so called ‘Brownfield’ applications or simply existing production applications which may not have been written using some of the best practices. These Brownfield applications can become a major barriers of entry into Unit testing. Software Engineers who are involved day to day supporting these applications often find it difficult to introduce Unit testing into them because they need to be refactored before doing so.

All the above challenges need to be overcome to create an atmosphere and mechanism of regular Unit testing.

How do we overcome the challenges ?

  1. A brownfield application which does not lend itself to Unit testing through it’s design will have be refactored first – piecemeal by piecemeal to facilitate the testing. Refactoring can include anywhere from making the code more object oriented, removing deep coupling, adding dependency injection and creating interfaces. Brand new code within the project can be written keeping Unit testing in mind. The more monolithic and deep coupled the code, more refactoring has to be done. It is best to chew at it slowly – over a period of time code will improve. Scrapping a brownfield completely to start a new greenfield application is not always an easy decision. Hence refactoring existing code for improvement one step at a time, can be more cost effective and beneficial.
  2. ‘Time’ is the one of the most negotiated element during software development. Most stakeholders like Product Owner and end clients will want the application deployed ‘yesterday’. In such a situation anything that adds more time to development can mean negotiation, conflict and stress. It is very normal for software engineers to give up on Unit testing when time is a constraint. A strong technical leadership committed to building consensus over Unit testing is critical to overcoming the time constraint- time spent upfront will save time later. The most significant benefits are as stated earlier Reduced QA time, Automatic regression and Better product quality.
  3. Unit testing does not come naturally as part of software development. It is still a habit which has to be developed and cultivated- training is also required to understand techniques of Unit testing. Lack of knowledge itself can inhibit Unit testing to happen even if the above two situations don’t exist. Often the difference between Unit testing and Integration testing are not understood. A culture of training and learning Unit testing has to be created. Team members who are ahead in Unit testing can be appointed as ‘coaches’ in Unit testing. Unit testing demos, presentations and discussions must be organized regularly to create the energy and enthusiasm around it. Technical leaders must ‘call out’ above and beyond Unit testing efforts; Incentivize or reward Unit testing efforts. An overarching policy of ‘incremental’ code coverage can be set, gradually increasing percentage code coverage. As needed ‘time’ must be allocated for the purpose of unit testing within the development cycles with the help of project managers.

A 100% code coverage may or may not be necessary based on the type of code or type of project. It’s a great goal to have, however it’s more important to test code for where rules, algorithms, calculations and complex logic need to be tested thoroughly. If time is a constraint decide where there is more value in Unit testing, and Unit test that part of the application.

An overarching policy of ‘incremental’ code coverage can be set, gradually increasing percentage code coverage. This way as the team matures into Unit testing, code coverage requirements are increased. Team members who are ahead in Unit testing can be appointed as ‘coaches’ in Unit testing. Unit testing demos, presentations and discussions must be organized regularly to create the energy and enthusiasm around it. Technical leaders must ‘call out’ above and beyond Unit testing efforts; Incentivize or reward Unit testing efforts.

A culture change like Unit testing can only happen when there are strong technical leads committed to the quality of the software, willing to go above and beyond to achieve that excellence. They have to be leaders willing to see beyond self-interest and work towards consistently evangelizing the practice.

Git, Software Configuration Management, Source Control

Git with Posh-Git, simplifying setup

Git has become one of the most popular source control systems. Thousands of enterprises are using Git for source control management. There are several reasons why Git is a compelling tool to consider for source control. Git being a distributed version control system, the central copy of the code is replicated as a local repository in the work space of the local machine. This facilitates working in offline mode with faster branching, history etc. which are otherwise expensive operations in Centralized repository systems like TFVC or Subversion.  It has extensive integration with many tools like Project Management tools, Build Automation tools and other surrounding tools for ALM and DevOps. Microsoft offers Git as a source control option along with TFVC as part of VSTS and TFS.

Knowledge of Git helps even if that’s not the version control being used for one’s work projects. One can go create public repositories for Open source work, or download / fork other public repositories and contribute to them. Git and GitHub are a great option to source control personal projects.

Many times, it may become a bit of a challenging transition to use Git if one is used to TFVC as a source control on WIndows. TFVC is so much a GUI based tool where Git is best used via command line. Git’s origins are in Linux based systems, and it’s client interfaces were designed to be used in Linux systems. Hence for Windows based development, to use Git as a Source control system, the setup is not as straightforward to understand. While there are many websites which go through setting up Git from scratch on Windows, this is an attempt to explain the steps in more detail so setting up Git for command line use is better understood and it’s easier to cross that initial hump.

Also, Git is best used with command line, since it has been designed that way. For Windows, use Posh-Git which provides an excellent command line interface. Here are the detailed steps for setting up Git and Posh-Git so Git can be used with command line.

1.Install Git for Windows from the website: 

Choose the below option from the Wizard. This will allow you to install Git Bash and Posh-Git.

gitposh.png

Git Bash is an application for Microsoft Windows environments which provides an emulation layer for a Git command line experience. Bash is an acronym for Bourne Again Shell.  Bash is a popular default shell on Linux and macOS. Git Bash is a package that installs Bash, some common bash utilities, and Git on a Windows operating system.

However, it is recommended to use Git with Posh-Git. It’s a PowerShell environment for Git. It’s an open source project hosted on GitHub. Here is the Post-Git GitHub link.

Excerpt from the Post-Git Github site about what Posh-Git offers for Git:

“posh-git is a PowerShell module that integrates Git and PowerShell by providing Git status summary information that can be displayed in the PowerShell prompt, e.g.:

C:\Users\Keith\GitHub\posh-git [master ≡ +0 ~1 -0 | +0 ~1 -0 !]>

posh-git also provides tab completion support for common git commands, branch names, paths and more. For example, with posh-git, PowerShell can tab complete git commands like checkout by typing git ch and pressing the tab key. That will tab complete to git checkout and if you keep pressing tab, it will cycle through other command matches such as cherry and cherry-pick. You can also tab complete remote names and branch names e.g.: git pull or<tab> ma<tab> tab completes to git pull origin master.“

Just to clarify, Git will work both through Git Bash and Posh-Git. However Posh-Git has a better interface which we will show later towards the end of the article.

After Git for Windows installation is completed,

2. Now you need to install Post-Git to be able to use Git from PowerShell.

While all instructions are there on the posh-git GitHub website, below are detailed step by step instructions for easier navigation through all the steps:

Open PowerShell as administrator.

Set Execution Policy to RemoteSigned so Posh-Git can be safely downloaded. Basically  RemoteSigned execution policy protects from running unsigned PowerShell scripts downloaded from the internet.

remotesigned.png

Now download and install posh-git. Run the below command.

PowerShellGet\Install-Module posh-git -Scope CurrentUser -AllowPrerelease -Force

This may give a AllowPrerelease parameter is not recognized error as below:

PowerShell PostGit install command

If the AllowPrerelease parameter is not recognized, update your version of PowerShellGet to >= 1.6 e.g. #

Install-Module PowerShellGet -Scope CurrentUser -Force -AllowClobber

AllowRelease

Once again run the command to get Posh-Git as below.

Install Posh-Git

3. After you have installed posh-git, you need to configure your PowerShell session to use the posh-git module. 

The first step is to import the module into your PowerShell session which will enable git tab completion. You can do this with the command:

Import-Module posh-git

This will  manually execute the Import-Module command every time you open a new PowerShell prompt. It’s better if posh-git to be available in all PowerShell hosts (console, ISE, etc).

For that purpose, execute 

Add-PoshGitToProfile -AllHosts.

This will add a line containing Import-Module posh-git to the file $profile.CurrentUserAllHosts.

importmoduleallhosts.png

Check the entry to import module exists powershell profile script by executing the below command in power shell:

PowerShellFileView

4. Check Git Version in Power Shell :

PowerShell GitVersion

Now , Git can be used with Power Shell and Posh-Git. A project can be created and GitHub can be used as a remote repository. It can be used with Git Bash as well, however it is recommended to keep the Posh-Git as primary CLI and Git-Bash can be the secondary CLI.

To illustrate one example how Posh-Git and Git Bash command line interfaces compare, and why Posh-Git is considered better- see below diagram where for a ‘test’ Git project how the ‘master’ branch shows up where two files have been added to the branch. Posh-Git clearly displays the 2 files added.

The Git websites cover many of the Git commands to set up initial projects very well. Learning the Git commands at the Git website and using other online tools will help in becoming further very comfortable with Git.

Software Architecture, Technology Leadership

‘Core Architecture’ function : why, what and how

Core Architecture function is not a new concept in many software companies. Systems which support a business are engineering products which ideally are built on top of many building blocks. The more the complexity of systems, more you need common building blocks which act as ‘Services’ or ‘Micro Services’, allowing new applications to be built quickly with them. While many companies have come to learn the value of establishing a core architecture function, several organizations have not yet realized the benefit of investing in such a structure. The reasons can be different depending on the company history and culture.

Why do we need a Core architecture group?

This model facilitates cost effective application development by promoting re-usability, and reducing redundancy. Often groups within a company are seen solving the same problems over and again in different ways. While autonomy and reinvention is necessary for creativity and improvement, it can be expensive in many cases when instead of  an existing solution being reused, it’s built all over again. In this age of Agility in producing digital products, reinventing the wheel often means lag in time-to-market with high upfront cost in development. Organizations when are totally reliant on automation and digitization, having building blocks or services which provide a ‘platform’ for quick application development is essential for nimbleness and speed to market.

In this age of Agility in producing digital products, reinventing the wheel often means lag in time-to-market with high upfront cost in development. Organizations when are totally reliant on automation and digitization, having building blocks or services which provide a ‘platform’ for quick application development is essential for nimbleness and speed to market.

It is also important that today’s digital products address the concerns of scale upfront. Products have to be built to scale, and a core dedicated architecture function is better positioned to streamline the architecture of applications in that direction. Development leads can use the guidance and assistance of dedicated architecture functions in addressing these concerns.

Software engineering and architecture standards, guidelines have to be established, shared, ensured and enforced as needed. Technology leaders who are empowered to make executive decisions must develop and communicate certain high level coding and architecture standards and practices, which keep the underlying code consistent, standard and of high quality.

What are the responsibilities of a core architecture group ?

A Core architecture group is a highly functional group which interacts and collaborates with all the stakeholders. The responsibilities will entail both strategic and tactical.

  1. Understand thoroughly the business capabilities of the organization, how information flows.
  2. Map and list all the core services that cut across all business capabilities of the company.
  3. Communicate on an ongoing basis the required services and their value.
  4. Discover already existing core / common services by collaborating with different teams. There could be multiple flavors of the same service, however bringing them all to the front will likely cause some of them to be reused. Those which are the most used can be narrowed down as final core services. Hence create a common repository of all these services, catalog and document them. Share the knowledge and documentation, provide links via source control, shared libraries or API endpoints.
  5. Plan and build new core services and publish.
  6. Evangelize core services, be the facilitator of systems integration of these services. This is a challenge in itself. It might be easier to build services, however harder to get them integrated into existing systems- an easier strategy then is to integrate services in new applications.

corearchitecture

In addition as the team grows and matures, it can take on other architecture functions like:

  1. Ensure application and systems architecture will be able to scale via reviews, testing and metrics. Consistently work with development leads in that direction.
  2. Establish and ensure coding and architecture standards.

The function can extend to encompass more tenets of architecture like performance, resilience, reliability and more.

How do you go about building a core architecture group ?

Execution is key- if a core architecture function is integrated well with the day to day development operations, a solid foundation is built for systems for the future.

  1. If budget allows it, form a team of software architects and engineers dedicated to this function. This can be a very small group to begin with, even starting with one expert architect can be good. And then this can be grown to a bigger group as needed.
  2.  If not, appoint technical leads , lead developers or technical managers as part time architects who take on the job of core architecture. There might be challenges with time management with day to day work, however this is a collective effort. Individual responsibilities can be assigned. These responsibilities and tasks are not outside of mainstream development work, they get integrated into the development pipeline and followed up on. This way there is full accountability. In fact a structure like this can be an advantage because systems integration is easier, the managers and leaders are already involved in developing the core services themselves.

Technology leadership of a company has to be vested in the idea of a Core architecture group. This is critical to the success of Systems that support the business functions. Product definition, product development and systems architecture all need to work together towards building products which scale and stand the test of time.

Technology Leadership

The value of Technology Road maps

We are going through the fourth industrial revolution- everything is digital and the world is technology driven no matter what we are doing. It’s common to see business intelligence, cloud related advertisements on TV , Billboards like direct consumer based products. Every company sees itself as a software company aiming to bring technology to the front and center. However many organizations, especially if they started as non-technology companies, have challenges bringing technology in their DNA because it’s not common to discuss and implement technology initiatives on their day to day – this kind of ‘tech-centric’ behavior is not natural to them.

What can be done to start making technology as the core function in the company ?

Assuming a technology team already exists, we can make a reasonable assumption that there exists a ‘Product Road map’ which the technology teams follow to build products. The product road map typically translates into product backlogs which are then turned into ‘technical’ solutions. This has almost become common knowledge.

While the tech team is implementing the product backlogs, it is very easy to get lost in the daily operations of releases, production support and feature enhancements. Technology evolves very fast and requires constant training, upgrade and alignment. One needs to take charge of that and make sure it’s part of road map and goals.

What is a Technology Road map ?

Product road maps are typically built by Product managers which are ‘Product’ features and priorities, with dates of expected releases. Product road maps can often be confused with Technology road maps. Unless the company culture organically has grown to include technical initiatives within product road maps, many technical initiatives may not make it to the product road map or backlog – which means they become a ‘nice-to-have’ and low priority items which get done only during free time – which may never happen !

Technology road maps include tool upgrades, platform upgrades, migrations, technology stack upgrades, technical debts, technical skills training, tools evaluations etc. Time and priority needs to be allocated to each of this. These type of items typically don’t get added to the Product road map, or even if they do get added their priority can get bumped down against product related priorities. The reason being ‘Product’ is directly related to the bread and butter, whereas the technology behind it is a support function however crucial it may be.

Technology road maps include tool upgrades, platform upgrades, migrations, technology stack upgrades, technical debts, technical skills training, tools evaluations etc. Time and priority needs to be allocated to each of this.

Below is an example of a simplest technology road map which has a mix of infrastructure, DevOps and Software Engineering timelines. This is a very high level road map created with no particular road map creation tool just for demonstration.

simple_roadmap

 

Each line item can be drilled down further into tasks. For each system there could be granular tasks associated. The chief product owner and and the technology head can get together and combine the product road map with the technology road map to come up with an integrated map.

How does one build a Technology Road map ?

  1. Technology leaders, Engineering managers, Technical architects must all collaborate together to come up with a 2-5 year technology road map.  The road map should be based on business goals, product goals and the technology vision around the goals. When technology leads together contribute to this road map, there is shared accountability, as well as consensus.
  2. If the company product line includes multiple systems, each system can have it’s own map. So, there can be a system specific road map and another high level road map which can apply to all the systems.
  3. A technology road map can be a combination of networking, security initiatives along with all other technology items. For example you may decide that in the next five years you want to migrate your source control from TFS to Git. You may also have a need to decide that you want to take your MSMQ servers to the AWS or Azure cloud. For that you may have the need to move Active Directory to the cloud first. They all together will constitute a ‘IT Road map’ or ‘Technology Road map’.
  4. Ultimately the Technology road map is owned by the CTO or the CIO . The road map is a vast matrix of technology initiatives over a period of 2 to 5 years with different categories and timelines. Each category might be a department or a system or a high level technical initiative.

roadmap

How do you execute on the technology road map ?

The items in a technology road map must be further divided into tasks which can be integrated or weaved into the product road map.  The resource allocation as well as priority allocation has to happen accordingly with a shared accountability of product managers, engineering managers and project managers.

The product managers, project managers and the rest of the business stakeholders have to buy into technology road map. The tasks within the technology road map should get baked into the ‘Sprints’ or the cycle of ‘Product development’. This way technology becomes a mainstream function aligned with latest industry trends. No technical debts are accumulated, causing a disaster or halt which can be costly to the bottom line. Not paying timely attention to pure technology initiatives can lead to unrecoverable technical damage. The value of technology road map in this digital age is very high and mission critical.

 

Software Architecture, Technology Leadership

Building a strong technology team
Every company today is a technology or software company. Software automation is the backbone of business operations today. The lines between business operations and software development are blurring because the two groups work together more closely than ever. Technology leaders have a tremendous responsibility of developing a futuristic digital vision, and accordingly building a strong technology team which can accomplish the vision.
vision-exercise-1
The goal of building a 5 year technology vision which translates into a solid technology road map can be lost even in companies who believe they are ‘tech’ companies. The daily pressures of production support, development operations and releases can bog down most of the technology managers. However a strong technology leader makes it her most important responsibility to build a technology vision and road map which takes the company to the next level of success step by step, avoiding major technical debts. The leader must develop a vision along with her other technology leaders who are her reports or peers so they can be the torch bearers of the vision. The vision has to trickle down and across.
The product vision and technology vision will together guide in building a team which will execute both, to a successful outcome.
A technology road map addresses technical debts, refactoring, migrations and upgrades, new technology evaluations, tool assessment and introductions, and any other relevant technology initiatives which are essential to technology growth for the company. The product vision and technology vision will together guide in building a team which will execute both, to a successful outcome.
So, what does a strong technology team composition look like ?
People talent is core to a company’s growth-  developing a team which aligns with the best technology practices and future vision is key to building a strong team. A technology leader, manager can strengthen her team by assessing what the current team composition and talent spread is. This assessment exercise should be periodically repeated until an optimal skill and experience mix is reached.
                                                      team    

A technology leader can strengthen her team by assessing what the current team composition and talent spread is. This assessment exercise should be periodically repeated until an optimal skill and experience mix is reached.

 

  1. Make a list of skill sets you want to have in your team.
  2. Determine which skill sets exist and which do not.
  3. Where are the strengths ? Where are the weaknesses- you want to induce talent in the team which will complement the existing skill set. Hire people from outside who have these new talents, when they come, they naturally rub off on the existing team. This is a huge motivator – peers who bring new ideas and skills from which one can be inspired and challenged. A natural growth occurs.
  4. Determine what experience levels exist. Too many Juniors can lack the experience and domain expertise to deliver well to your business context. Too many seniors can create Silos- they are opinionated and  often comfortable in their set old ways. Hence a careful examination of team composition is always necessary in determining team morale and growth.
  5. Attract and hire fresh talent- fresh graduates and juniors are highly motivated, open and enthusiastic. You need a good mix of Juniors, Mid-Level and Senior staff. You have to have few Senior experienced leaders and mentors, some Juniors and rest Mid-level. A 30 % experienced, 40% mid, and 30% Junior approximately can be a healthy composition which will keep the team motivated , mentored and productive. An imbalance or dominance of any category can lead to side effects which are not healthy.
  6. Form a ‘Core architecture team or function’. Break ‘Silos’ by creating a formalized core architecture team, function or platform which establishes re-usability, reduces redundancy, and builds common services.

Form a ‘Core architecture team or function’. Break ‘Silos’ by creating a formalized core architecture team, function or platform which establishes re-usability, reduces redundancy, and builds common services.

What about professional growth, nurturing home grown talent ? 

A leader has to create an impact on the team in many different ways for professional development.

  1. Lead by example : you can demonstrate that as a leader you walk the talk. Demos, presentations, proof of concepts provide actual examples of what is being proposed and evangelized. People follow examples better than persuasion.
  2. Be hands on, deep and wide technically. A technology leader when ‘speaks’ to technology very well, often can make much greater impact.
  3. Appoint coaches on topics to promote learning. For example , say you want to spread Unit testing practices. Pick people in the team who are ahead in that, as appointed coaches. Likewise for example  ‘Agile’, ‘Build automation’ or any best practices where learning needs to be facilitated.  This can also foster a culture of learning from peers.
  4. Call out, above and beyond efforts.  Provide a platform for star performers and outliers to ‘show and tell’. Interesting events like Lunch n learns, ‘Coffee chats’ or other group meetings can be organized where team exchanges ideas in a more casual setting.
  5. Consensus building: let’s face it, in technology multiple view points exist. People can get passionate about their opinions- a leader may need to be patient and persuasive which of course is not easy. Often there is one solution which suits the context most, towards which the consensus has to be built. White boarding the context and various solutions, and mapping the most suitable in that situation will help build consensus.

One article can not cover the vast topic of how strong technology teams can be built, however everything starts with a vision and a road map. The leader then aligns the team with this vision consistently.  An assessment and measure of team skill and experience spread, will help carve a plan for building a strong technology team. Stay tuned for more on this topic – happy team building !

Design Patterns, Development, Software Architecture, Technology Leadership

Collaborating teams build systems that collaborate

Information systems of a company are a reflection of the communication patterns within the organization. The more teams communicate with one other cohesively, these systems will connect and grow together more as a result. In today’s world of digital transformation and data intelligence, it is imperative that multi function organizations communicate and collaborate, so systems built by the development staff will bring the necessary business intelligence and scale. This cohesive intelligence is critical to further growth. It is very common in multi function organizations for ‘Silos’ to develop over a period of time. These ‘Silos’ focus successfully on one aspect of the business, however they fail to connect with other systems in the workflow such that it reflects the customer journey. This ‘disconnect’ can create many black-boxes producing their own data, which as a whole brings minimal value in the long run.

It is very interesting that many popular methodologies and architectural patterns have evolved which fundamentally solve this problem of communication. Let’s take DevOps for example. DevOps is designed to have better collaboration between Development teams and Network operations. Although they are independent in function, lack of communication between them results in surprises, errors and continuous finger pointing. This can be very expensive operationally leading to client dissatisfaction. Today DevOps has had great success which leads to test, build and deployment automation. This in turn increases speed-to-market and system reliability.

In today’s world of digital transformation and data intelligence, it is imperative that multi function organizations communicate and collaborate, so systems built by the development staff will bring the necessary business intelligence and scale as well.

Agile development methodologies solve the collaboration problems between all disciplines of an organization. The SCRUM methodology with it’s different ‘ceremonies’ brings all relevant departments together in one room regularly. From Sprint Planning to Sprint Retrospective, different departments like products, development, QA, and user experience come together to discuss topics from requirements to ‘lessons learnt’. This collective, methodical way of communication leads to better product definition, risk mitigation, estimation and release planning. Everyone is in a continuous improvement and feedback loop, improving deliveries significantly.

collaboration

Let’s take Micro-services architecture, for example. This style of architecture was embraced by many companies in an effort to have small teams become autonomous by fostering a loosely coupled architecture connected via independent services. Again, at a high level team structures and their communication patterns dictate how the Enterprise architecture shapes. Netflix adopted the Micro-services architecture to increase speed-to-market which led to a social culture of small autonomous teams collaborating together.

Companies don’t just build products and offer services , they build professional communities and offer personal growth.

Any organization which looks to build information systems that work together, while they are independent units, has to build a communication culture and team structure which can facilitate such systems. Cohesive systems bring business intelligence while loose coupling allows them to scale. Cohesion comes from continuous collaboration and loose coupling comes from small autonomous teams. Adopt methodologies and architectural patterns which bring teams and people together. Companies don’t just build products and offer services , they build professional communities and offer personal growth.

Design Patterns, Development, Object Oriented Programming, SDLC, Software Architecture

Program to an interface , not an implementation

I have been lately discussing SOLID in my previous posts – some of the basic principles of OOD . Recently while doing my project work , we had to decide between Entity Framework  or NHibernate which are popular ORM layers in .Net framework .  The decision was not easy , we got stuck at Second Level caching support …and could not reach any decision right away. I started thinking that may be for now we have to write our layer in such a way that we choose one ORM , and then we can switch to a different one worst case if need be. Ideally a situation like this should not arise , but if it does happen , you need to be able to do it.  Once again I felt how a very fundamental OOD principle could come in handy –

Program to an interface , not an implementation.

These words above can be very ambiguous and abstract when you just read them .

However when seen from a context , you understand how powerful this is , and how it can increase flexibility and ability to switch . Seen from my own situation where we needed the flexibility to switch data access mechanism if need be, let’s understand this by constructing a scenario. For example , a common problem could be implementing a subscription mechanism for a data service that your system may provide. Users can subscribe to the data service and get notified when a change occurs. We need to persist the subscription information to the database , where you will need to use some database access mechanism.

As usual there is a presentation layer that gathers some subscription information and passes it along to the data service which persists the information to the database . For simplicity’s sake , let’s say we have a SubscriptionService , which is our data service for persisting subscriptions . We can discuss this further in the context of an ASP.Net MVC application – where a View is our Presentation layer through which Subscription details are collected. A controller action method is typically invoked to pass along the information to the server.

We can assume for our article purposes that our application provides notifications on Food Recalls. So the member user to our website subscribes to recalls for a certain type of food issued by different organizations like FDA etc. The Subscription in simplicity could be :

Please note : all code is pseudo only , this is not a working solution that can be downloaded and built.

public class RecallsSubscriptionService
{
    Boolean Subscribe( int userId, string food_type, string organization)
    {
           // data access code to insert subscriptions
           return false; 
    }         
}

Most of the times the controller action method would look something like this:

public class SubscriptionController : Controller
{
        [HttpPost]
        public ActionResult Subscribe(int userId, string food_type, string organization)
        {
            try
            {
                 SubscriptionService myservice = new SubscriptionService();
                 myservice.Subscribe(userId, food_type, organization);

                return RedirectToAction("Index");
            }
            catch
            {
                return View();
            }
        }
}

So , we make a decision we will use Entity Framework 6 to implement our data access to store subscription information. The above code will work fine . The code goes into production . However later we decide that due to certain project requirements we have to switch to NHibernate. This is a tough situation – our controller here , who is a client of the SubscriptionService , is heavily dependent on instantiating this specific service which accesses the database with Entity Framework 6. Although we expected the problem could arise we didn’t design our system for this flexibility. Even though we encapsulated the data access code into a Service , we did not free the client from the implementation of data access because the client program is indirectly tied to EF implementation.

We need to make some changes , so our controller is independent of the future changes that could happen to the database access mechanism. To bring transparency into how data access may happen from a client’s perspective we need to introduce interfaces. We write an interface instead for the SubscriptionService :

interface IRecallsSubscriptionService
{
   Boolean Subcribe(int userId, string food_type, string organization);       
}

We could write two implementations of the above interface or we could just use the same implementation we had earlier , and replace it with NHibernate.

public class RecallsSubscriptionServiceEF
{
       Boolean Subscribe(int userId, string food_type, string organization) 
       { 
                //   implement using EF ;
                return false;
       }         
}

public class RecallsSubscriptionServiceNH
{
       Boolean Subscribe(int userId, string food_type, string organization) 
       {
              //implement using NH ;
              return false;
       }         
}
/*( not that the above approach is recommended , where you keep 
two implementations available for the same requirement. 
This is for understanding purposes only )*/

The above mechanism allows us to drive the Subscribe method call through the IRecallsSubscriptionService interface. In order to effectively use the method above , we will need to pass IRecallsSubscriptionService as a parameter to the controller constructor and use dependency injection in this scenario to have the right concrete class to be instantiated during run-time. I will not dive deeper into that because it is outside of the scope of this topic.

This is the whole basis of Dependency Inversion and Injection ( this is also the ‘D’ in SOLID ): here we program to an interface , not an implementation as we saw above. This gave us the flexibility to change our implementation completely without changing the client code. So the basic idea is programming to an interface decouples the client from the internal implementation freeing calling programs from knowing the internals leading to flexibility in changing implementation as the need arises.

Even in scenarios where there are published APIs , when you use objects  go up in hierarchy to program to the interface as opposed to the concrete class. So if you are using a List object and comparing individual strings-  then if this object implements IComparer , then program to IComparer –

instead of writing:

List myList = new List();
myList.Compare( myList[0] , myList[1]);

//Write as :

IComparer myList = new List();
myList.Compare( myList[0], myList[1] );

//This gives you the freedom to use a different flavor of Compare method if you need to ,

IComparer myList = new Array();
myList.Compare( myList[0], myList[1] );

The advantage of the above is ability to switch different comparison methods if they are available. And also test program can easily switch different implementations and see what works the best.

We also need to understand that “Program to an interface” should not be taken literally . Even Abstract classes can act as interfaces in this context and give us the same flexibility.

Interfaces should be used wisely and only when needed. If there is no potential for an implementation to ever change there is no need to bother with interfaces. For example , a Utility class that converts a string to a number. There is no need to write interface to this . Interfaces should also be used in projects where testability is a big concern and mocking may be required.

Design Patterns, Development, Object Oriented Programming, OO, SDLC, Software Architecture

SOLID conclusions with ISP and DIP

In my last post we went over LSP ( Liskov Substitution principle ) , how it helps achieve Subtype Polymorphism. We learnt how to think in terms of client programs and how client’s call to  programs can influence the quality and design of applications. The SRP and OCP are foundational principles on which LSP and the next one we discuss today , ISP manifest themselves.  It is not enough if Single Responsibility and Open Closed are understood – the rest of the three principles are where you see their applications and extensions. LSP , ISP and DIP all three teach us how to design from a client’s point of view. Uncle Bob calls it as “Clients exerting forces “  . 

The interface-segregation principle (ISP) states that no client should be forced to depend on methods it does not use

ISP helps formulate mechanisms on how interfaces should be segregated so the same interface can be useful to different clients. It’s a very powerful thought that can bring strong results. Single Responsibility talks about writing interfaces that have very cohesive functions . ISP takes it one step further and gives concrete ideas that give rise to reusable interfaces . This is where the relationship and the difference between SRP and ISP exists .

So how does ‘Segregating’ Interfaces help the clients ? Interfaces are meant to represent a certain type of behavior to the client and by virtue of their contractual representation promote pluggability. So when interfaces represent more than one type of behavior , they provide more than what a client needs . This leads to the interface becoming incapable of being used for plug-in purposes. Also they become ‘Fat’ in nature , with unncessary behavious not useful in the client’s context. Basically this is solidifying more on the SRP and OCP and stressing a whole lot more on how interface contracts should represent one type of behavior. Almost all patterns have a basis in SRP , OCP and ISP. Let’s say we are writing a class to represent a Persistence Medium. One of the most popular persistence medium is Database . So is XML . And a lot of applications use simple JSON files to store simple content.  Say , we start out by writing a PersistenceMedium class. So a simpe IPersistenceMedium looks like this :

Code is pseudo only , to describe the principle , not a working , compiled example :

// IResult is an imaginary interface 

interface IPersistenceMedium
{
       string fileName { get; set; }
       string connectionString { get; set; }

       void Open();       
       void IResult ExecuteQuery(string query);
       void IResult ReadFile();
       void Close();
}

//So our database class can be written like this:

public class Database : IPersistenceMedium
{
    private string _connectionString;

    public string connectionString
    {   get{ return _connectionString; }  
        set{ _connectionString = value; }    
    }

    public void Open(){  /* database open connection code */ }
    public IResult ExecuteQuery( string query );
    public void Close(){  /* database close connection code */ }
}

// The JSONStore can be written like this :

public class JSONStore : IPersistenceMedium
{
     private string _fileName;

     public string fileName
     {   get{ return _fileName; }  
         set{ _fileName = value; }    
     }

      public void Open(){  /* open a JSON file code */}
      public IResult ReadFile();
      public void Close(){  /* close file code  */}
}

See what happened above ? The Database class could not use ReadFile() and fileName and JSONStore could not use ExecuteQuery() and connectionString . Although both are variants of a PersistenceMedium behaviour , they are unnecessarily clubbed together from the perspective of Database and JSONStore classes which are the clients of IPersistenceMedium.

A Better way would be :

interface IFileConnection 
{ 
     string fileName { get; set; }
     void Open();
     void Close();
}

interface IDatabaseConnection 
{   
      string connectionString {  get ; set; }   
      void Open();
      void Close();
}

interface IDBOperation
{
     IResult ExcecuteQuery();
}

interface IFileOperation
{
      IResult ReadFile();
}

Now this provides interfaces that can be specifically used for Database or JSONStore because we segregated them keeping in mind different clients / client groups. The interfaces which can be used for JSONStore can be used for XML or any other type of file store. Similarly the interfaces that got written for Database can be used for any type of database , SQL , Oracle or NoSQL . The interface is less fat , with interfaces ‘Segregated’ strictly based on different client usages . This is SRP in effect, by using ISP.  What are the different ways of using these with the Clients ? One obvious way is multiple inheritance.

public class JSONStore : IFileConnection , IFileOperation
{
    private string _fileName;
    public string fileName
    {   get{ return _fileName; }  
        set{ _fileName = value; }    
    }

    public void Open(){ /* open file*/ }
    public IResult ReadFile(){ /* Read File */}
    public void Close() { /* Close File */ }

}

public class Database : IDatabaseConnection , IDatabaseOperation
{
    private string _connectionString;
    public string connectionString
    {   get{ return _connectionString; }  
        set{ _connectionString = value; }    
    }

    public void  Open() { /* Open Connection */}
    public IResult ExecuteQuery(){  /* execute query to obtain results */}
    public void Close(){ /* Close Connection */}

}

The above JSONStore could be very well be written as FileStore , in that case different FileStores can be used and IResult representing different result types form different types of files.

There are different ways to implement the above without multiple inheritance. One way is to use something like a Strategy pattern where IFileOperation has concrete classes for JsonFileOperation and XmlFileOperation – because each may have subtle read differences. XML requires special parsing and JSON is more of a string representation.

We have been able to establish in many different ways the value of SOLID – how it  leads to good design practices and understanding of patterns.

I will conclude without diving into Dependency Inversion – there are fantastic blogs including Uncle Bob’s report. Dependency Inversion distinguishes itself by establishing how layers should interact  without depending on each other. Again a powerful concept that has led to great programming practices including Test Driven Development and several patterns that facilitate the principle. SOLID is definitely worth your time !

Design Patterns, Object Oriented Programming, OO, Software Architecture

Way to Patterns : even more SOLID

In the last post we saw how OCP facilitates extensibility and reusability . OCP is the foundation on which several patterns have been written. The Strategy pattern is a great example of OCP where subclasses are written based on different algorithms. The manifestation of OCP happens in the third SOLID principle , the ‘L’ as we know ,  Liskov Substituion principle – perhaps the more involved and less understood principles in SOLID .

We saw how SRP can lead to OCP – LSP takes OCP and establishes clear rules that will ensure polymorphism is accomplished correctly. LSP attempts to achieve what we call subtype polymorphism through it’s rules. In short we can represent this in pseudocode – a client call to a subtype method call , through a base class interface:

public class Supertype{   public virtual outparam SomeMethod( inparam );   }     
public class Subtype : Supertype { public override outparam SomeMethod( inparam );  } 

 //Client call:   
 Supertype baseType = new Subtype();    
 outparam = baseType.SomeMethod(inparam);

Liskov Substitution statement  translates to : Subtype(derived type) must be behaviorally equal to their base types. They must be usable through the base type interface without the need for the user to know the difference.

LSP needs to be understood from a client’s perspective . Client here means : Calling programs , Users of your interfaces and abstract classes within the organization  or outside the organization. A Client perceives the behavior of a class through methods of a class : arguments passed , value returned and any state changes after the method execution. So basically to be able to use a subclass / subtype in place of a superclass/supertype the sub type successfully needs to preserve argument requirements and return expectations by the client.  A client code that gets written keeping in mind the Supertype to call it’s method should not change or break when replaced with subclass method calls . Compilers do enforce Signature compliance when methods are overridden from abstract classes . However internally within the implementation if arguments or return values were treated in a manner that could break the client code , the code is in violation of LSP because essentially the contract with the client was broken . Compilers typically will not catch these violations.

It is the programmers responsibility to ensure LSP compliance for the most part.  In order for the internal subtype behavior to keep in consistency with supertype behavior , these principles were formed.

There is a very subtle and interesting nuance that one needs to understand here. You could very well write a ‘Superclass’ , and then write a ‘Subclass’ – override the Superclass method to give a specific implementation. Use the subclass in your programs to execute the subclass specific method. You may never see anything wrong up until you expect the subclass to be a ‘subtype’ of the Superclass which is a Supertype. It’s when a ‘subclass’ is expected to become a ‘subtype’ is when LSP comes into play.

A lot of blogs have been written to explain these rules.  I have cited at the end some good ones in order to understand them with code examples : instead in this blog let’s try and understand some of the confusing rules that mostly I have seen people asking questions about .  Let’s just list all of them first :

  • Contravariance of method arguments in the subtype.
  • Covariance of return types in the subtype.
  • No new exceptions should be thrown, unless the exceptions are subtypes of exceptions thrown by the parent.
  • Preconditions cannot be strengthened in the subtype
  • Postconditions cannot be weakened in the subtype
  • Invariants must be preserved in the subtype.
  • History Constraint – the subtype must not be mutable in a way the supertype wasn’t.

Now, let’s take just the two which seem to confuse most people :

  • Preconditions cannot be strengthened in the subtype.
  • Postconditions cannot be weakened in the subtype.

Preconditions apply to arguments which will be used as part of the implmentation. Postconditions mostly relate to return values or the state after the implementation is executed. Preconditions are requirements on users of the functions, while postconditions are requirements on the functions themselves. Preconditions get executed before the actual implementation is executed , whereas postconditions are executed after.

Preconditions cannot be strengthened in the subtype.

Wikipedia explains this as :

In the presence of inheritance, the routines inherited by descendant classes (subclasses) do so with their preconditions in force. This means that any implementations or redefinitions of inherited routines also have to be written to comply with their inherited contract. Preconditions can be modified in redefined routines, but they may only be weakened. That is, the redefined routine may lessen the obligation of the client, but not increase it.

What is the obligation of the client ?  The arguments that need to be passed  , is the obligation of the client. If the preconditions  are set in such a way in the subclass method  that the choice of the arguments which can be passed from the client is lesser or restricted then you actually strengthened the precondition .  This increases the obligation of the client.

Let’s understand this with an example. We will modify the IFormatter interface from the last post to an abstract base class with some implementation.

 abstract class Formatter{            

         public virtual string Format( String message)
         {
                if ( String.IsNullOrEmpty( message ) ) 
                    throw new  Exception ();
                // do formatting
         }      

     }
    // strengthened precondition
    public class MobileFormatter : Formatter{

         public override string Format( String message)
         {
                if ( String.IsNullOrEmpty( message ) || message.Length > 250  ) 
                    throw new  Exception ();
                // do formatting
         }

    }
    // weakened precondition
    public class MobileFormatter : Formatter
    {
        public override string Format(String message)
         {
                if ( message == null  ) 
                   throw new  Exception ();
                // do formatting
         }

    }

As we see above the MobileFormatter placed more restriction on the arguments in the strengthened precondition – this will force the client to change their code to accommodate for this if they want to avoid getting an exception.  So behaviorally the base and subtype are different.

In the weakened precondition what happened is that the client now does not need to accommodate the code for MobileFormatter, the argument that gets passed to MobileFormatter , works for base Formatter as well because the validation in Formatter is stronger or the validation in MobileFormatter is weaker.

Postconditions cannot be weakened in the subtype.

Wikipedia explains this as :

In the presence of inheritance, the routines inherited by descendant classes (subclasses) do so with their contracts, that is their preconditions and postconditions, in force. This means that any implementations or redefinitions of inherited routines also have to be written to comply with their inherited contract. Postconditions can be modified in redefined routines, but they may only be strengthened. That is, the redefined routine may increase the benefits it provides to the client, but may not decrease those benefits.

Let’s understand this with a code example:

   abstract class Formatter
    {

        public virtual string Format(String message)
        {

            // do formatting
            return message.Trim();
        }

    }
    // weakened postcondition
    public class MobileFormatter : Formatter{

         public override string Format( String message)
         {

                //do formatting
             return message;
         }

    }
    // strengthened postcondition
    public class MobileFormatter : Formatter
    {
        public override string Format(String message)
        {

            //do formatting
            return message.Trim().PadLeft(5);
        }
    }

What we see above in the MobileFormatter is that the postcondition got weakened by removing the Trim method. This provides the client less than what was provided in terms of the result .

Then to correct it , we strengthened the postcondition by adding left padding. This does not require the client to change any code , however the code gets the extra benefit of padding. The above example is rather crude but serves the purpose of explaining .

There is a very interesting pattern called Template pattern accomplishes LSP via template methods written inside a base class which are overridden in derived classes.  For now , this is enough to contemplate about – more in the next blog .

Here are some really good blogs on Liskov that discuss other rules as well:

http://www.ckode.dk/programming/solid-principles-part-3-liskovs-substitution-principle/#contravariance

http://msdn.microsoft.com/en-us/magazine/hh288081.aspx

Until then happy programming !

Design Patterns, Development, Object Oriented Programming, OO, SDLC, Software Architecture

Finding the way to Design Patterns : more SOLID

So continuing into the process of finding our way into design patterns from my last post , we will try and unfold SOLID a bit more. Some of the SOLID principle interpretations and applications can be very subjective and a matter of debate. How do you know the classes that you wrote have adhered to the Single Responsibility Principle ? Is there a way to determine ? What does Single Responsibility exactly mean?  How far do you try and take these things…blah ..blah.

There needs to be a balance in everything of course.One needs to find a mid way between over-engineering and under-engineering. We need to create small classes with very few cohesive/related functions without getting carried away by having so many classes that we cannot manage those either. At the same time , understanding exactly how to create classes with just one relevant behavior or few cohesive functions can get tricky. It’s important to ask the question while writing a class , what is it that will potentially change in this class later , that can be re-factored into a separate class , so minimal changes / or no changes need to be made to the existing implementation when the change needs to be made.

It’s important to ask the question while writing a class , what is it that will potentially change in this class later , that can be re-factored into a separate class , so minimal changes / or no changes need to be made to the existing implementation when the change needs to be made.

If we take the Logging example that we discussed in the previous post , we discussed three functions a logger can do. Initializing a medium to log , Formatting to the medium itself  and Writing to the medium . So, they all look like related functions don’t they ? So they all can go in one class , which is a violation of SRP . If we take one step further into re-factoring  perhaps it’s easy to recognize right away that there are different mediums to log , each requires a different method to write to itself. So we create separate classes for the medium . Yes , we applied SRP here.

What about formatting ?  Can formatting change ? Yes, it can . Say we need to add a new logging medium called Mobile Device. To the mobile device , you want to send text messages whenever severe conditions occur. Doesn’t the format of what goes into a database differ from a mobile device ?  Also , later down the line the potential users of our Logger may require that they want the format of the message a little different. Now if we did not separate the format function into it’s own class , we have it as part of the class that writes to the medium or worse as part of the Logger class. Logger class is actually our client facing class that the client would use to log messages . Just to change  formatting , we have to go and change this Logger class or the medium classes.

At this point we need to consider  whether it’s worth writing something like this:





   1:  interface IFormatter {
   2:   
   3:  public string Format( string message );
   4:   
   5:  }
   6:




and the logger could do something like this :






   1:  public class Logger{
   2:   
   3:  public Write( IFormatter formatter){}
   4:   
   5:  }
   6:




Or let’s go couple of steps further along these lines,






   1:  public interface ILogMedium{
   2:   
   3:  void Write(String Message);
   4:   
   5:  }
   6:








   1:  public class LogToDatabase : ILogMedium{
   2:   
   3:  public void Write(string message){
   4:   
   5:  // medium specific logging
   6:   
   7:  }
   8:   
   9:  }
  10:








   1:  public class Logger{
   2:   
   3:  ILogMedium _logTodb;
   4:   
   5:  IFormatter _formatter;
   6:   
   7:  public void Logger( ILogMedium logToMedium,  IFormatter formatter )
   8:   
   9:  {
  10:   
  11:  _logTodb = DBMediumProvider.Create(); // DBMediumProvider could be a creation class
  12:   
  13:  _formatter = DBFormatProvider.Create(); // DBFormatProvider could be a creation class
  14:   
  15:  }
  16:   
  17:  public Write(String message){
  18:   
  19:  _logTodb.Write( _formatter.Format(message) );
  20:   
  21:  }
  22:   
  23:  }
  24:




Please note : The code above is not a working solution. It is only like pseudo code to demonstrate the thought process .


The advantage of the above is we kept the formatter completely separate from the medium , as well as the Logger class. This allows us to plug in new mediums , formatters and configure them based on our needs. Also , if there are bugs it’s easier to fix a specific class as opposed to one big class which could potentially break other functions.


This brings us to the Open Closed principle , which is the next one and the ‘O’ in SOLID as we all know.


A consistent use of SRP can lead to OCP , which simply states: A class should be open for extension but closed for changes. This does not mean that classes should be completely sealed for change , it means that the class should be in a state where only bug fixes should be made and new functions should be added via new classes , minimizing implementation changes to the existing classes.


Words should not be taken literally here : once you wrote the class does not mean that it cannot be changed at all : however we should get to a point that , when behavior changes in predictable ways, you should not have to make several changes into a single class or several classes in the system. An ideal situation would be where you achieve the change by adding new class / code rather than changing existing code . For anyone who has been in programming for a few years supporting production systems , this will make a lot of logical sense.


Whatever we discussed above with Logger example with reference to SRP will apply to OCP as well , because OCP can be achieved by applying SRP consistently. So had we written the formatting in the Logger or the individual medium classes we would not be able to add new formatters or change existing formatting without changing the Logger or the mediums classes , plus mix and match formatting with mediums . The way we achieved OCP with Logger example is we gave the Logger a Single Responsibility of writing to the medium. We gave the Medium classes a Single Responsibility of initializing / creating the medium and specifically write to that medium as well. And then OCP came into effect when we made it possible for new formatters to be added to the system by simply implementing the interface IFormatter and adding that class to the system as a plugin. So , when a user wants to use the new Formatter , she can do it through the configuration system and that formatter will get automatically used which needs to be implemented through some creation classes of course , which is a separate topic – an example is Factory Method pattern.


Several design patterns use the SRP and OCP , and following the above two will put you in the mode of clean and efficient code . Some popular patterns like Strategy , Factory are all based on OCP . We will continue with this discussion in the next post , happy programming !


Few references :


Wikipedia definition Open-Closed


Interesting read on OCP by Jon Skeet