Artificial Intelligence, Artificial Intelligence, Development, Software Architecture

Navigating the AI Revolution in Coding: Promise, Potential, and Pitfalls.

AI-based code generators, like GitHub’s Copilot and various other popular tools, have shown impressive capabilities in assisting developers with coding tasks, intelligently. Previously I wrote an article on how we can use GitHub CoPilot as an AI pair programmer for faster and more effective programming. As we basked in the AI’s potential, skepticism arose. Questions of reliability, security, impact on employment, creativity.

What could be the Pitfalls ?

Code Quality & Reliability: The AI-generated code isn’t perfect and can overlook coding best practices, leading to verbose or inefficient code. There’s also the risk of planting unintended bugs, an inevitable coding nightmare.

Security Threats: Technological advancement brings along potential security vulnerabilities. AI’s code suggestions might fall into this category, and the risk of sensitive data leakage is undeniably a concern.

Skill Erosion: We’re at risk of losing essential coding skills with AI in the driver’s seat. It’s akin to using calculators in math classes, where students may memorize formulas but miss out on understanding the underlying concepts.

Ethical & Licensing Dilemmas: Copyrighted code is an intellectual asset. An AI suggesting a similar line of code raises matters of intellectual property rights, introducing ethical and licensing dilemmas.

Bias: AI is only as good as the data it’s trained on. If the training data carries biases, the AI-produced code might inherit them, causing ethical issues.

Economic Impact: Technological progress has consequences. In this case, the possible reduction in need for entry-level programming jobs, which can impact the tech industry’s employment chart.

Trust & Dependency: A growing dependency on AI might lead to blind trust causing issues in the final product. It’s a case of ‘too much of a good thing.’

Stifled Creativity: Innovation and creativity are yet human realms. AI can’t whip up innovative or off-the-wall solutions. Are we stifling our creativity?

Understanding the Promise

Any modern technology brings both excitement and skepticism in the beginning however as time passes, society often adjusts to its presence, weighing its benefits against its drawbacks, and finding ways to integrate it into everyday life. To understand and verify AI’s promise to modern coding, an experiment and study was conducted by Cornell University in 2022, whose results were published as a paper in Feb 2023:

  • Objective: This research focuses on the potential of generative AI tools to boost human productivity.
  • Methodology: A controlled experiment was conducted with GitHub Copilot, an AI-assisted programming tool. Developers were tasked with quickly implementing an HTTP server in JavaScript.

Experiment Setup: During the trial, developers were tasked to implement an HTTP server in JavaScript as swiftly as possible. The treatment group had access to GitHub Copilot, and was given a brief introduction video. In contrast, the control group, which didn’t have Copilot access, was unrestricted in other ways (e.g., they could use online search or Stack Overflow).

Any modern technology brings both excitement and skepticism in the beginning however as time passes, society often adjusts to its presence, weighing its benefits against its drawbacks, and finding ways to integrate it into everyday life.

Results:

Those who had access to GitHub Copilot (the treatment group) finished the task 55.8% faster than those who did not (the control group). The performance difference between the two groups was both statistically and practically significant. The confidence interval for this measurement is between 21% and 89%. This means that based on the sample data and the statistical analysis performed, we are reasonably confident that the true performance improvement of the treatment group (with Copilot) over the control group is somewhere between 21% and 89%.

  • The blue bar represents the control group, which did not have access to GitHub Copilot.
  • The green bar represents the treatment group, which had access to GitHub Copilot. This group finished the task 55.8% faster.
  • The black error bar on the green bar represents the confidence interval for the performance improvement, ranging from 21% to 89%.
  • If the treatment group was 89% faster, then they would take only 11% of the time the control group took.
  • If the treatment group was 21% faster, then they would take 79% of the time the control group took.

Diverse Effects: The AI tool had varied impacts based on developer backgrounds. Those with less programming experience, older developers, and individuals who spend more hours programming daily benefited the most. This suggests that such tools can expand access to software development careers.

Developers with Less Programming Experience (Junior Developers):

  • Observation: Junior developers or those with less programming experience derived significant benefits from the tool.
  • Implication: GitHub Copilot likely provided these developers with code suggestions and solutions that they might not have been aware of or would have taken longer to derive on their own. This assistance can accelerate their learning curve and increase their efficiency.

Older Developers:

  • Observation: Older developers benefited more from the tool compared to their younger counterparts.
  • Implication: The reasons for this can be multi-fold. Older developers might have vast experience in programming but may not be as updated with newer frameworks, libraries, or best practices. GitHub Copilot can bridge this gap by suggesting modern code solutions. Alternatively, older developers might be more methodical and take advantage of AI assistance to validate or enhance their code.

Developers Who Program More Hours Per Day:

  • Observation: Those who spend more hours programming daily saw more benefits from the tool.
  • Implication: Developers who are more engaged in coding tasks can leverage GitHub Copilot more frequently, extracting maximum utility from its suggestions. It could serve as a continuous companion, optimizing their workflow and reducing the time spent on debugging or searching for solutions online.
A group of deligted programmers …..generated by AI using Midjourney

Other Observations: The paper mentions heterogeneous effects, suggesting there might be varying results based on different conditions or types of developers. There’s also a mention that AI pair programmers might be helpful for individuals transitioning into software development careers.

Affiliations: The authors are affiliated with institutions like Microsoft Research, GitHub Inc., and MIT Sloan School of Management.

Limitations of the study

The research is in its early stages, involving a select group of participants working on a non-commercial project. As such, the product is not designed for public or production deployment. Therefore, concerns regarding licensing, legal implications, and security were not factors considered in this study. Although crafting an HTTP server with JavaScript presents its own challenges, real-world commercial endeavors often encompass intricate systems, necessitating collaboration across diverse teams. In a typical enterprise environment, developers do more than just code solutions for business challenges; they also craft various test cases, from unit to integration tests. Consequently, this study does not evaluate metrics such as code quality or testability.

Harnessing the Potential

Given this brief overview, it seems the study provides empirical evidence for the positive impact of generative AI tools, on developer productivity. The resultant surge in productivity is not merely a metric; it translates to valuable time that can be redirected towards innovative, business-centric endeavors rather than being mired in the repetitive grind of manual coding. While these AI tools aren’t silver bullets for intricate projects, they serve as adept and intelligent CoPilots. For organizations to truly harness this potential, it’s imperative for leadership and engineering heads to foster a culture of vigilant oversight, continuous training, and rigorous code reviews. This proactive approach can not only maximize the advantages of these tools but also address the challenges highlighted earlier in the discussion.

Happy programming !

Resources

Here is a link to the study paper available on the web:

Cornell study: Impact of AI tools on Developer productivity

Please note there is a link to the downloadable PDF at the right hand top.

Artificial Intelligence, Software Architecture

Embark on an AI API journey with OpenAI: A detailed Guide for Beginners

Are you ready to unleash your creativity and build amazing applications with the power of AI?

Artificial Intelligence based APIs are a way for developers to access the functionality of pre-trained AI models without the need for extensive knowledge of machine learning algorithms. They help break the barrier of entry into creating AI applications and facilitate the adoption of AI development for non-AI engineers. These APIs can be integrated into a variety of applications and are typically categorized based on their functionality. They enable us to create applications with smarter artificial intelligence features such as natural language processing, image recognition, data analysis, image creation and many other use cases. As of this writing, there are several APIs available in the market each offering a set of capabilities and models. In this blog, we will discuss OpenAI API, one of the top AI API offerings available as of today.

One of the hardest thing to do in technology is disrupt yourself

Matt Mullenwegg

What is OpenAI API? 

OpenAI API is a service that lets you access the most advanced and known artificial intelligence models in the world through programming interfaces. OpenAI is creator of the now very popular ChatGPT. They have a strong community and documentation around their APIs. It is fairly sophesticated with SDKs available for various technology platforms, which can be leveraged for individual use or for Enterprise applications. This makes the OpenAI API an excellent candidate for learning and leveraging, to create new applications or enhance existing applications.

These are some models which OpenAPI currently supports:

MODELSDESCRIPTION
GPT-4A set of models that improve on GPT-3.5 and can understand as well as generate natural language or code
GPT-3.5A set of models that improve on GPT-3 and can understand as well as generate natural language or code
DALL·EA model that can generate and edit images given a natural language prompt
WhisperA model that can convert audio into text
EmbeddingsA set of models that can convert text into a numerical form
ModerationA fine-tuned model that can detect whether text may be sensitive or unsafe
GPT-3LegacyA set of models that can understand and generate natural language
DeprecatedA full list of models that have been deprecated
Courtesy of OpenAI API documentation website.

You can further fine-tune the models with your own data to suit your specific needs and preferences. You can combine various models to create interesting outputs. The OpenAI API is easy to use and integrate with your existing workflows.

So now that we have established the value OpenAI API let’s understand how to get started with them.

How to register on the OpenAI API website? 

To use OpenAI API, follow these steps:

  1. Visit openai.com and sign up for an account.
  2. Provide your name, email address, and create a password.
  3. Agree to the terms of service and privacy policy.
  4. Check your email for a confirmation link.
  5. Click on the link to activate your account.
  6. Once activated, you will be redirected to the OpenAI dashboard.

How to generate a key for the API? 

  1. Go to the OpenAI dashboard.
  2. Click on “API keys” in the navigation menu.
  3. Locate the “Create an API key” button and click on it.
  4. A pop-up window will appear with your new API key.
  5. Copy the API key, which will look something like this: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
  6. Keep the API key safe and secure, as it grants access to your OpenAI API.
  7. You can now use this API key to authenticate your requests to the OpenAI API.

Remember, it’s important to never share your API key with anyone to maintain the security of your API access.

How does OpenAI charge for the usage of API? 

OpenAI API is not free, but it has a generous free tier that lets you experiment and learn without paying anything. The free tier gives you 5,000 tokens per month. A token is a unit of measurement that represents the amount of data processed by OpenAI API.

 If you exceed the free tier limit, you will be charged $0.06 per 1,000 tokens. You can monitor your usage and billing on the OpenAI dashboard. You can also set up alerts and limits to avoid unexpected charges. 

Let’s understand tokens a little bit more. A character is a single symbol or letter in a text, such as “a” or “.” A token is a group of characters that form a meaningful unit, such as “cat” or “hello”. Tokens can vary in length depending on the context and the model. For example, the word “cat” might be one token for some models, but two tokens for others if they split it into “c” and “at”. 

Created using DALL-E.

“A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words).”

A token can be as short as one character or as long as one word. For example, “ChatGPT is great!” is considered as five tokens: [“ChatGPT”, ” “, “is”, ” “, “great!”].

Remember, both input and output tokens count towards the total. For example, if you send 10 tokens as a prompt and receive 20 tokens as a response, you’ll be billed for 30 tokens.

The cost per token varies depending on your usage tier. You can find detailed pricing information on the OpenAI Pricing page.

Some interesting Use cases of Open AI API

OpenAI API provides a wide range of AI capabilities that can be used in various use cases. Some of the common use cases for OpenAI API include:

  1. Text generation: OpenAI API can generate human-like text, making it useful for applications such as chatbots, content creation, and language translation.
  2. Text analysis: OpenAI API can analyze text to extract information, classify text, and perform sentiment analysis. This can be useful for applications such as social media monitoring, customer feedback analysis, and market research.
  3. Image recognition: OpenAI API can recognize and classify images, making it useful for applications such as visual search, image tagging, and object detection.
  4. Speech recognition: OpenAI API can transcribe speech to text, making it useful for applications such as voice assistants, voice search, and voice commands.
  5. Data analysis: OpenAI API can analyze data to extract insights, make predictions, and perform anomaly detection. This can be useful for applications such as predictive maintenance, fraud detection, and customer segmentation.

Possibilities are endless based on the context and creativity, how you can leverage these to enhance existing solutions or create new solutions.

Start with a simple program

Once all the above steps have been taken, you are ready to start programming with them.

Open a code editor like Visual Studio Code and write the Open API calling application in your programming language of choice.

Here is a sample Python program which calls an OpenAI API. As you can see it’s very simple and straight forward. All Open AI APIs are centered around completion APIs. Please note you can use any programming language to call these API, however Python is widely used by OpenAI themselves to show examples and various blogs on the web. This makes it a very convenient language to try various models, templates and settings easily. Visual Studio Code is a popular IDE which is free and supports various languages including Python. You can easily install Python extension in Visual Studio Code and with few steps start programming in it.

If you haven’t already started learning Python you may want to start doing since it is one of the most popular programming languages being used today. Here is a survey result from StackOverflow conducted in 2022.

Courtesy Stack Overflow : https://survey.stackoverflow.co/2022#most-popular-technologies-language

Python will also likelihood remain a dominant player in the coming years due to widespread use in AI, ML, Data Science etc. We will not go into too much detail into how to set up Python locally, however here is a sample program how to make a call to Open AI completion API with Python- as you can see it’s very simple and straightforward.

import openai

openai.api_key = "your open api key here"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"

prompt = "Hello"
prompt += "\nTranslate this into 1.French, 2.Spanish and 3.Hindi:"
prompt += "\n1."

print(prompt)

response = openai.Completion.create(
    engine="davinci",
    prompt=prompt,
    temperature=0.1,
    max_tokens=100,
    top_p=1,
    frequency_penalty=0,
    presence_penalty=0
)
print(response)  # Print the server response
print(response['choices'][0]['text'])

Here is an output of the program above:

Why is prompt engineering critical in using OpenAI API? 

Prompt engineering is a skill that every developer should learn and master. Why? Because it can help you create amazing applications that leverage the power of natural language processing and artificial intelligence. It is the art and science of designing and optimizing prompts, which are inputs that guide the behavior of a natural language model. A prompt can be a question, a command, a statement, or anything else that elicits a response from the model. You are already using prompts if you are using ChatGPT or other similar AI tools.

By learning prompt engineering, you can unlock the full potential of natural language models customize the outputs of the models to suit your needs and preferences, such as tone, style, format, length, and content. Prompt engineering is not only useful, but also fun and creative. You can experiment with different prompts and see how the models react. You can also challenge yourself to create prompts that produce the best results for your specific goals and scenarios. 

Prompt engineering is the future of natural language processing and artificial intelligence. It’s a crucial skill for developers working with AI language models because the quality of the output heavily depends on the input prompt.

With Prompt Engineering you can get:

  1. Better Results: A well-crafted prompt can guide the model to generate more accurate and relevant responses.
  2. Efficiency: It can help reduce the number of tokens used, thereby saving costs.
  3. Control: It allows developers to have more control over the model’s behavior and output.

Prompt engineering involves understanding the model’s behavior, experimenting with different prompts, and iterating based on the results. It’s a blend of art and science, and mastering it can greatly enhance the effectiveness of your AI applications.

Use the Presets / Templates to learn

The OpenAI API Playground is a user-friendly interface that allows developers to interact with OpenAI’s powerful models. One of the key features of the Playground is the use of Presets or Templates. These are pre-configured settings that help users get started with the API without needing to manually set all the parameters. They also give you some basic ideas on how to constructing your prompts for desired results.

Presets or Templates are designed to provide a starting point for common tasks. They are set up with optimal parameters for specific use-cases, such as drafting an email, writing Python code, or translating languages. By selecting a preset, the Playground automatically fills in the necessary parameters, such as the prompt, temperature, and max tokens.

Here’s how you can use Presets/Templates:

  1. Go to OpenAI API platform’s playground page.
  2. Select a Preset: When you open the Playground, you’ll see a dropdown menu labeled ‘Presets’. Click on it to see a list of available templates.
  3. Choose Your Task: The list includes a variety of tasks. Select the one that best matches your needs. The Playground will automatically populate the parameters based on your selection.
  4. Customize If Needed: Although the preset provides a good starting point, you can still customize the parameters to better suit your specific needs. For instance, you can modify the prompt, adjust the temperature to make the output more or less random, or change the max tokens to control the length of the output.
  5. Run the Model: Once you’re satisfied with the settings, click ‘Run Model’ to see the results. You can experiment with different settings and run the model multiple times to get the best output.

So, Presets in the OpenAI API Playground are a great way to get started with the API. They simplify the process of setting up the model, allowing you to focus on getting the results you need. Whether you’re a seasoned developer or a beginner, these presets can help you make the most of the powerful capabilities of the OpenAI API.

Here is a Preset example of chat conversation to generate SQL :

The playground also gives you the Python code you can use for the Preset prompts and play with it in code:

So I think this is enough motivation and information to get started with OpenAI APIs. While this writing was more to generate interest in OpenAI APIs, there are various other APIs which offer individual capabilities and use cases. The best APIs which suit your scenario the most should be selected for production applications. Here are some popular platforms which provide AI APIs today:

  • Google Cloud AI Platform: This platform offers a suite of services specifically targeted at building, deploying, and managing machine learning models in the cloud. official website.
  • Amazon Rekognition: This is an image recognition software that uses machine learning to automate and lower the cost of image recognition and video analysis. official website.
  • IBM Watson: This is a question-answering computer system capable of answering questions posed in natural language. It was developed in IBM’s DeepQA project by a research team led by principal investigator David Ferrucci. official website.
  • Microsoft Azure Cognitive Services: This is a comprehensive suite of out-of-the-box and customizable AI tools, APIs, and models that help modernize your business processes faster. official website.
  • Clarifai: This is an artificial intelligence company that specializes in computer vision and uses machine learning and deep neural networks to identify and analyze images and videos. The company offers its solution via API, mobile SDK, and on-premise solutions. official website.
  • TensorFlow: This is an open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks. official website.

I hope this will motivate you to jump into AI programming with AI APIs.

Happy Programming!

Artificial Intelligence, Development, Git, Software Architecture

Exploring GitHub Copilot, Ask Copilot

GitHub Copilot is the new AI powered programming tool, which can be used to assist with various programming tasks within the development environment you are using. The AI tool gets integrated within your code editor and provides assistance throughout, without leaving the code editor window. Rightly named, it functions like a co-programmer pairing in real time- creating, enhancing and documenting code.

It is based on OpenAI Codex, a powerful artificial intelligence system that can generate code from natural language prompts. OpenAI Codex is trained on billions of lines of code from public GitHub repositories and other sources, so it can understand the context and syntax of different programming languages. Copilot uses OpenAI Codex to suggest code snippets. Open AI Codex can understand various programming languages, frameworks, and domains, and can even answer questions about code. It is a powerful tool for developers, learners, and anyone who wants to create software with ease and fun.

The above image was created with DALL-E, prompt to DALL-E was:
“AI pair programming with Github copilot anime cartoon“

Copilot can help you with various coding tasks, such as writing functions, tests, loops, queries, algorithms, and more. It can also generate comments, documentation, and examples for your code. It is not a replacement for human programmers, but rather a tool that augments their capabilities and creativity. It is still in technical preview and may not always produce correct or optimal code. You should always review and test the code generated by CoPilot before using it in your projects. You should also follow the best practices and guidelines for your programming language and framework, as well as the license and terms of service of CoPilot.

I explored it with Visual Studio 2022 and C# code, however Copilot works with various code editors and programming languages. As for setup, it works as an extension for Visual Studio and Visual Studio Code. At the time of this writing, you have to install it with Visual Studio 2022 version 17.4.4 or later and follow some steps. GitHub and Microsoft have several articles on how to set it up with Visual Studio, however at the end of this article I have listed links which will help you set up GitHub CoPilot with Visual Studio.

Briefly here are the steps.

  • Sign up for the GitHub Copilot technical preview waitlist at https://copilot.github.com/. You will receive an email from GitHub when you get access to the preview.
  • Install Visual Studio 2022 version 17.4.4 or later from https://visualstudio.microsoft.com/downloads/.
  • In Visual Studio 2022, go to Extensions > Manage Extensions and search for GitHub Copilot. Download and install the extension.
  • Restart Visual Studio 2022 and authorize the GitHub Copilot extension with your GitHub account.
  • Accept the telemetry terms to join the GitHub Copilot technical preview.
Created by DALL-E Prompt used : “Step by step instructions to install represented as digital art“

Once you have done these steps, you can start using Copilot to write code faster and smarter. You can use natural language comments or method names to trigger suggestions, and press Tab to accept a suggestion. You can also use the Copilot panel to see alternative suggestions or tweak the settings. Copilot will suggest code and entire functions based on your context and comments. You can press Tab to accept a suggestion, or keep typing to ignore it.

The most useful feature of GitHub Copilot is Ask Copilot- which you can initiate by Right clicking on the code editor to show the context menu which will present the Ask Copilot context menu option.

This is where it gives you the experience of ChatGPT embedded within your IDE and Code editor helping you add, refine and document code.

GitHub Copilot is not a replacement for human programmers, but rather a tool that augments their capabilities and creativity. It is still in technical preview and may not always produce correct or optimal code. You should always review and test the code generated by GitHub CoPilot before using it in your projects. You should also follow the best practices and guidelines for your programming language and framework, as well as the license and terms of service of GitHub CoPilot.

For example some prompts I tried are:

“Create a Startup class for this project“

“Refactor Program.cs to use the Startup class“

“Add a xunit based unit test for my StreamController class“

“Create a new class Sale which has properties Item and Amount“

The above prompt creates a Sale class with the properties and relevant types.

As you can see in the image above you get an option to either a Create new file or Insert at Caret. Create a new file adds the class to the project.

Insert at Caret will insert the code where the cursor is on the editor.You can keep enhacing and improving the same code.

Then I asked “Add a property Active to the class Sale” and it produced the output as in the above image.

It automatically inferred Active as a boolean and allows to insert to the class. While this is a simple and straight forward feature it can save typing time by creating classes, interfaces and other code, and add them to the project or existing code. This takes the drudgery out of programming by automating them and saving time for harder and more complex programming.

Here are some of the other tasks I was able to accomplish:

  1. Created multiple API controller classes and added constructors with dependency injection and added Get methods.
  2. Added XUnit based test code for the controller classes.
  3. Asked the copilot to explain code, framework classes and create documentation.
  4. Improved code to make it more succint.
  5. Asked to explain the meaning of a code block.
  6. Given a block of code, wrap it around a try-catch block.

Another task I asked Copilot to do was code review the below class:

My prompt was:

“Do a code review of the class StreamController“

Here is the class:

[ApiController]
    [Route("[controller]")]
    public class StreamController : Controller
    {
        private readonly OpenAIAPI _openAIApi;
        private readonly Conversation _conversation;    

        public StreamController(OpenAIAPI openAIApi)
        {
            _openAIApi = openAIApi;
            _conversation = _openAIApi.Chat.CreateConversation();
        }

        [HttpGet("{inputParam}")]
        public async IAsyncEnumerable<string> Get(string inputParam)
        {

            _conversation.AppendUserInput(inputParam);

            var sb = new StringBuilder();

            await foreach (var res in   _conversation.StreamResponseEnumerableFromChatbotAsync())
            {
                sb.Append(res.ToString());
            }

            yield return sb.ToString();

        }
    }
}

Here are the code review comments by Copilot:

I also explored using natural language commenting for it to suggest a code snippet. For example as shown in the screenshot below I typed a comment in the code as here below :

//This method makes a Post call to OpenAI api and returns the response

The Copilot typed up the rest of the code for me line by line as I accepted it by pressing Tab.

Some tasks I could not accomplish were like adding a Unit test project to an existing Solution. While it can generate unit test code for individual classes and methods it cannot add a scaffolded unit test project to the existing solution. I was hoping that it can create a unit test project and add the generated classes to it. I was also not able to refactor an existing Program.cs to create a new Startup, add that to the project and refactor it appropriately to use that Startup class without me interving manually to add the services to the service collection and other start up code. While it can understand the context of the program and create code and classes based on that it cannot accomplish multiple steps of refactor or creating a basic templated project and adding generated code to it.

It’s hard to say whether some of these tasks are possible or not in the future based on how the models are trained, however I am hoping there will be many more automations to come which will eliminate the drudgery out of repetetive and mundane coding tasks and make them more fun, faster and easier to accomplish. It is also an extremely useful tool to learn programming and understand frameworks, sdks and APIs you are using to build your solutions.

The fact that you can remain within your code editor and find all the answers to the questions without having to switch to google search, ChatGPT, Stack Overflow etc. is a tremendous time saver – plus the added benefit of generating code and adding to the ongoing code base. There is already a lot of interest around Copilot, many companies have made this a part of their day today programming.

These are some of my initial experiments with it. I used a simple and brand new project to explore the tool and some of it’s features. It would be interesting to see how this can be used to review and optimize existing complex code. I am still in the process of exploring it further, and give the right natural language comments and the right prompts to Ask Copilot.

I highly recommend you use this for your programming projects and efforts.

Happy programming !

Some useful resources:

(1) GitHub Copilot in Visual Studio 2022 – Visual Studio Blog.
(2) Quickstart for GitHub Copilot – GitHub Docs.
(3) Getting started with GitHub Copilot – GitHub Docs.

(4) CoPilot chat feature private preview

Development, Object Oriented Programming, SDLC, Software Architecture

Unit testing: climbing the Uphill road

This article is not about discussing the basics of unit testing or simplifying the concepts, nor it is about merits of the practice. Unit testing methods have evolved quite a bit, with many books and online content. However, the real world practice of Unit testing can be an uphill task in several contexts, due to different reasons, even after how the ecosystem around Unit testing has matured quite a bit.

Unit testing is both an Architectural as well as DevOps concern. Tactically both these departments strive to get the software development operations to effectively incorporate Unit testing.

What are some of the challenges ?

  1. Brownfield vs. Greenfield : Unit testing is easier to adopt when you are starting a new application. The reason being an application needs to be written in a certain way, applying principles like S.O.L.I.D to be able to Unit test it effectively. There are a few key programming knowledge areas which need to be understood well before starting Unit testing correctly, like Interface driven programming and Dependency Injection. The industry is full of so called ‘Brownfield’ applications or simply existing production applications which may not have been written using some of the best practices. These Brownfield applications can become major barriers of entry into Unit testing. Software Engineers who are involved day to day supporting these applications often find it difficult to introduce Unit testing into them because they need to be refactored before doing so.
  2. Time : Unit testing applications can be time consuming, requiring additional development time to be factored in. Often this upfront time is seen as additional cost to development without realizing that, there could be long term benefits like reduced QA footprint and automated regression tests. Hence a technical leadership which creates a culture of Unit testing by coaching the stakeholders on benefits of Unit testing has to be fostered by organizations.
  3. Training: To Unit test effectively, software engineers need to have knowledge of Unit testing basics like creating Mocks, Code Coverage and the AAA pattern ( also known as Arrange, Act and Assert ) . The lack of this knowledge itself can be a barrier even if the first two barriers do not exist. Hence training on Unit testing fundamentals and the available Unit testing frameworks needs to happen.
The Uphill task of Unit testing

The industry is full of so called ‘Brownfield’ applications or simply existing production applications which may not have been written using some of the best practices. These Brownfield applications can become a major barriers of entry into Unit testing. Software Engineers who are involved day to day supporting these applications often find it difficult to introduce Unit testing into them because they need to be refactored before doing so.

All the above challenges need to be overcome to create an atmosphere and mechanism of regular Unit testing.

How do we overcome the challenges ?

  1. A brownfield application which does not lend itself to Unit testing through it’s design will have be refactored first – piecemeal by piecemeal to facilitate the testing. Refactoring can include anywhere from making the code more object oriented, removing deep coupling, adding dependency injection and creating interfaces. Brand new code within the project can be written keeping Unit testing in mind. The more monolithic and deep coupled the code, more refactoring has to be done. It is best to chew at it slowly – over a period of time code will improve. Scrapping a brownfield completely to start a new greenfield application is not always an easy decision. Hence refactoring existing code for improvement one step at a time, can be more cost effective and beneficial.
  2. ‘Time’ is the one of the most negotiated element during software development. Most stakeholders like Product Owner and end clients will want the application deployed ‘yesterday’. In such a situation anything that adds more time to development can mean negotiation, conflict and stress. It is very normal for software engineers to give up on Unit testing when time is a constraint. A strong technical leadership committed to building consensus over Unit testing is critical to overcoming the time constraint- time spent upfront will save time later. The most significant benefits are as stated earlier Reduced QA time, Automatic regression and Better product quality.
  3. Unit testing does not come naturally as part of software development. It is still a habit which has to be developed and cultivated- training is also required to understand techniques of Unit testing. Lack of knowledge itself can inhibit Unit testing to happen even if the above two situations don’t exist. Often the difference between Unit testing and Integration testing are not understood. A culture of training and learning Unit testing has to be created. Team members who are ahead in Unit testing can be appointed as ‘coaches’ in Unit testing. Unit testing demos, presentations and discussions must be organized regularly to create the energy and enthusiasm around it. Technical leaders must ‘call out’ above and beyond Unit testing efforts; Incentivize or reward Unit testing efforts. An overarching policy of ‘incremental’ code coverage can be set, gradually increasing percentage code coverage. As needed ‘time’ must be allocated for the purpose of unit testing within the development cycles with the help of project managers.

A 100% code coverage may or may not be necessary based on the type of code or type of project. It’s a great goal to have, however it’s more important to test code for where rules, algorithms, calculations and complex logic need to be tested thoroughly. If time is a constraint decide where there is more value in Unit testing, and Unit test that part of the application.

An overarching policy of ‘incremental’ code coverage can be set, gradually increasing percentage code coverage. This way as the team matures into Unit testing, code coverage requirements are increased. Team members who are ahead in Unit testing can be appointed as ‘coaches’ in Unit testing. Unit testing demos, presentations and discussions must be organized regularly to create the energy and enthusiasm around it. Technical leaders must ‘call out’ above and beyond Unit testing efforts; Incentivize or reward Unit testing efforts.

A culture change like Unit testing can only happen when there are strong technical leads committed to the quality of the software, willing to go above and beyond to achieve that excellence. They have to be leaders willing to see beyond self-interest and work towards consistently evangelizing the practice.

Git, Software Configuration Management, Source Control

Git with Posh-Git, simplifying setup

Git has become one of the most popular source control systems. Thousands of enterprises are using Git for source control management. There are several reasons why Git is a compelling tool to consider for source control. Git being a distributed version control system, the central copy of the code is replicated as a local repository in the work space of the local machine. This facilitates working in offline mode with faster branching, history etc. which are otherwise expensive operations in Centralized repository systems like TFVC or Subversion.  It has extensive integration with many tools like Project Management tools, Build Automation tools and other surrounding tools for ALM and DevOps. Microsoft offers Git as a source control option along with TFVC as part of VSTS and TFS.

Knowledge of Git helps even if that’s not the version control being used for one’s work projects. One can go create public repositories for Open source work, or download / fork other public repositories and contribute to them. Git and GitHub are a great option to source control personal projects.

Many times, it may become a bit of a challenging transition to use Git if one is used to TFVC as a source control on WIndows. TFVC is so much a GUI based tool where Git is best used via command line. Git’s origins are in Linux based systems, and it’s client interfaces were designed to be used in Linux systems. Hence for Windows based development, to use Git as a Source control system, the setup is not as straightforward to understand. While there are many websites which go through setting up Git from scratch on Windows, this is an attempt to explain the steps in more detail so setting up Git for command line use is better understood and it’s easier to cross that initial hump.

Also, Git is best used with command line, since it has been designed that way. For Windows, use Posh-Git which provides an excellent command line interface. Here are the detailed steps for setting up Git and Posh-Git so Git can be used with command line.

1.Install Git for Windows from the website: 

Choose the below option from the Wizard. This will allow you to install Git Bash and Posh-Git.

gitposh.png

Git Bash is an application for Microsoft Windows environments which provides an emulation layer for a Git command line experience. Bash is an acronym for Bourne Again Shell.  Bash is a popular default shell on Linux and macOS. Git Bash is a package that installs Bash, some common bash utilities, and Git on a Windows operating system.

However, it is recommended to use Git with Posh-Git. It’s a PowerShell environment for Git. It’s an open source project hosted on GitHub. Here is the Post-Git GitHub link.

Excerpt from the Post-Git Github site about what Posh-Git offers for Git:

“posh-git is a PowerShell module that integrates Git and PowerShell by providing Git status summary information that can be displayed in the PowerShell prompt, e.g.:

C:\Users\Keith\GitHub\posh-git [master ≡ +0 ~1 -0 | +0 ~1 -0 !]>

posh-git also provides tab completion support for common git commands, branch names, paths and more. For example, with posh-git, PowerShell can tab complete git commands like checkout by typing git ch and pressing the tab key. That will tab complete to git checkout and if you keep pressing tab, it will cycle through other command matches such as cherry and cherry-pick. You can also tab complete remote names and branch names e.g.: git pull or<tab> ma<tab> tab completes to git pull origin master.“

Just to clarify, Git will work both through Git Bash and Posh-Git. However Posh-Git has a better interface which we will show later towards the end of the article.

After Git for Windows installation is completed,

2. Now you need to install Post-Git to be able to use Git from PowerShell.

While all instructions are there on the posh-git GitHub website, below are detailed step by step instructions for easier navigation through all the steps:

Open PowerShell as administrator.

Set Execution Policy to RemoteSigned so Posh-Git can be safely downloaded. Basically  RemoteSigned execution policy protects from running unsigned PowerShell scripts downloaded from the internet.

remotesigned.png

Now download and install posh-git. Run the below command.

PowerShellGet\Install-Module posh-git -Scope CurrentUser -AllowPrerelease -Force

This may give a AllowPrerelease parameter is not recognized error as below:

PowerShell PostGit install command

If the AllowPrerelease parameter is not recognized, update your version of PowerShellGet to >= 1.6 e.g. #

Install-Module PowerShellGet -Scope CurrentUser -Force -AllowClobber

AllowRelease

Once again run the command to get Posh-Git as below.

Install Posh-Git

3. After you have installed posh-git, you need to configure your PowerShell session to use the posh-git module. 

The first step is to import the module into your PowerShell session which will enable git tab completion. You can do this with the command:

Import-Module posh-git

This will  manually execute the Import-Module command every time you open a new PowerShell prompt. It’s better if posh-git to be available in all PowerShell hosts (console, ISE, etc).

For that purpose, execute 

Add-PoshGitToProfile -AllHosts.

This will add a line containing Import-Module posh-git to the file $profile.CurrentUserAllHosts.

importmoduleallhosts.png

Check the entry to import module exists powershell profile script by executing the below command in power shell:

PowerShellFileView

4. Check Git Version in Power Shell :

PowerShell GitVersion

Now , Git can be used with Power Shell and Posh-Git. A project can be created and GitHub can be used as a remote repository. It can be used with Git Bash as well, however it is recommended to keep the Posh-Git as primary CLI and Git-Bash can be the secondary CLI.

To illustrate one example how Posh-Git and Git Bash command line interfaces compare, and why Posh-Git is considered better- see below diagram where for a ‘test’ Git project how the ‘master’ branch shows up where two files have been added to the branch. Posh-Git clearly displays the 2 files added.

The Git websites cover many of the Git commands to set up initial projects very well. Learning the Git commands at the Git website and using other online tools will help in becoming further very comfortable with Git.

Software Architecture, Technology Leadership

‘Core Architecture’ function : why, what and how

Core Architecture function is not a new concept in many software companies. Systems which support a business are engineering products which ideally are built on top of many building blocks. The more the complexity of systems, more you need common building blocks which act as ‘Services’ or ‘Micro Services’, allowing new applications to be built quickly with them. While many companies have come to learn the value of establishing a core architecture function, several organizations have not yet realized the benefit of investing in such a structure. The reasons can be different depending on the company history and culture.

Why do we need a Core architecture group?

This model facilitates cost effective application development by promoting re-usability, and reducing redundancy. Often groups within a company are seen solving the same problems over and again in different ways. While autonomy and reinvention is necessary for creativity and improvement, it can be expensive in many cases when instead of  an existing solution being reused, it’s built all over again. In this age of Agility in producing digital products, reinventing the wheel often means lag in time-to-market with high upfront cost in development. Organizations when are totally reliant on automation and digitization, having building blocks or services which provide a ‘platform’ for quick application development is essential for nimbleness and speed to market.

In this age of Agility in producing digital products, reinventing the wheel often means lag in time-to-market with high upfront cost in development. Organizations when are totally reliant on automation and digitization, having building blocks or services which provide a ‘platform’ for quick application development is essential for nimbleness and speed to market.

It is also important that today’s digital products address the concerns of scale upfront. Products have to be built to scale, and a core dedicated architecture function is better positioned to streamline the architecture of applications in that direction. Development leads can use the guidance and assistance of dedicated architecture functions in addressing these concerns.

Software engineering and architecture standards, guidelines have to be established, shared, ensured and enforced as needed. Technology leaders who are empowered to make executive decisions must develop and communicate certain high level coding and architecture standards and practices, which keep the underlying code consistent, standard and of high quality.

What are the responsibilities of a core architecture group ?

A Core architecture group is a highly functional group which interacts and collaborates with all the stakeholders. The responsibilities will entail both strategic and tactical.

  1. Understand thoroughly the business capabilities of the organization, how information flows.
  2. Map and list all the core services that cut across all business capabilities of the company.
  3. Communicate on an ongoing basis the required services and their value.
  4. Discover already existing core / common services by collaborating with different teams. There could be multiple flavors of the same service, however bringing them all to the front will likely cause some of them to be reused. Those which are the most used can be narrowed down as final core services. Hence create a common repository of all these services, catalog and document them. Share the knowledge and documentation, provide links via source control, shared libraries or API endpoints.
  5. Plan and build new core services and publish.
  6. Evangelize core services, be the facilitator of systems integration of these services. This is a challenge in itself. It might be easier to build services, however harder to get them integrated into existing systems- an easier strategy then is to integrate services in new applications.

corearchitecture

In addition as the team grows and matures, it can take on other architecture functions like:

  1. Ensure application and systems architecture will be able to scale via reviews, testing and metrics. Consistently work with development leads in that direction.
  2. Establish and ensure coding and architecture standards.

The function can extend to encompass more tenets of architecture like performance, resilience, reliability and more.

How do you go about building a core architecture group ?

Execution is key- if a core architecture function is integrated well with the day to day development operations, a solid foundation is built for systems for the future.

  1. If budget allows it, form a team of software architects and engineers dedicated to this function. This can be a very small group to begin with, even starting with one expert architect can be good. And then this can be grown to a bigger group as needed.
  2.  If not, appoint technical leads , lead developers or technical managers as part time architects who take on the job of core architecture. There might be challenges with time management with day to day work, however this is a collective effort. Individual responsibilities can be assigned. These responsibilities and tasks are not outside of mainstream development work, they get integrated into the development pipeline and followed up on. This way there is full accountability. In fact a structure like this can be an advantage because systems integration is easier, the managers and leaders are already involved in developing the core services themselves.

Technology leadership of a company has to be vested in the idea of a Core architecture group. This is critical to the success of Systems that support the business functions. Product definition, product development and systems architecture all need to work together towards building products which scale and stand the test of time.

Technology Leadership

The value of Technology Road maps

We are going through the fourth industrial revolution- everything is digital and the world is technology driven no matter what we are doing. It’s common to see business intelligence, cloud related advertisements on TV , Billboards like direct consumer based products. Every company sees itself as a software company aiming to bring technology to the front and center. However many organizations, especially if they started as non-technology companies, have challenges bringing technology in their DNA because it’s not common to discuss and implement technology initiatives on their day to day – this kind of ‘tech-centric’ behavior is not natural to them.

What can be done to start making technology as the core function in the company ?

Assuming a technology team already exists, we can make a reasonable assumption that there exists a ‘Product Road map’ which the technology teams follow to build products. The product road map typically translates into product backlogs which are then turned into ‘technical’ solutions. This has almost become common knowledge.

While the tech team is implementing the product backlogs, it is very easy to get lost in the daily operations of releases, production support and feature enhancements. Technology evolves very fast and requires constant training, upgrade and alignment. One needs to take charge of that and make sure it’s part of road map and goals.

What is a Technology Road map ?

Product road maps are typically built by Product managers which are ‘Product’ features and priorities, with dates of expected releases. Product road maps can often be confused with Technology road maps. Unless the company culture organically has grown to include technical initiatives within product road maps, many technical initiatives may not make it to the product road map or backlog – which means they become a ‘nice-to-have’ and low priority items which get done only during free time – which may never happen !

Technology road maps include tool upgrades, platform upgrades, migrations, technology stack upgrades, technical debts, technical skills training, tools evaluations etc. Time and priority needs to be allocated to each of this. These type of items typically don’t get added to the Product road map, or even if they do get added their priority can get bumped down against product related priorities. The reason being ‘Product’ is directly related to the bread and butter, whereas the technology behind it is a support function however crucial it may be.

Technology road maps include tool upgrades, platform upgrades, migrations, technology stack upgrades, technical debts, technical skills training, tools evaluations etc. Time and priority needs to be allocated to each of this.

Below is an example of a simplest technology road map which has a mix of infrastructure, DevOps and Software Engineering timelines. This is a very high level road map created with no particular road map creation tool just for demonstration.

simple_roadmap

 

Each line item can be drilled down further into tasks. For each system there could be granular tasks associated. The chief product owner and and the technology head can get together and combine the product road map with the technology road map to come up with an integrated map.

How does one build a Technology Road map ?

  1. Technology leaders, Engineering managers, Technical architects must all collaborate together to come up with a 2-5 year technology road map.  The road map should be based on business goals, product goals and the technology vision around the goals. When technology leads together contribute to this road map, there is shared accountability, as well as consensus.
  2. If the company product line includes multiple systems, each system can have it’s own map. So, there can be a system specific road map and another high level road map which can apply to all the systems.
  3. A technology road map can be a combination of networking, security initiatives along with all other technology items. For example you may decide that in the next five years you want to migrate your source control from TFS to Git. You may also have a need to decide that you want to take your MSMQ servers to the AWS or Azure cloud. For that you may have the need to move Active Directory to the cloud first. They all together will constitute a ‘IT Road map’ or ‘Technology Road map’.
  4. Ultimately the Technology road map is owned by the CTO or the CIO . The road map is a vast matrix of technology initiatives over a period of 2 to 5 years with different categories and timelines. Each category might be a department or a system or a high level technical initiative.

roadmap

How do you execute on the technology road map ?

The items in a technology road map must be further divided into tasks which can be integrated or weaved into the product road map.  The resource allocation as well as priority allocation has to happen accordingly with a shared accountability of product managers, engineering managers and project managers.

The product managers, project managers and the rest of the business stakeholders have to buy into technology road map. The tasks within the technology road map should get baked into the ‘Sprints’ or the cycle of ‘Product development’. This way technology becomes a mainstream function aligned with latest industry trends. No technical debts are accumulated, causing a disaster or halt which can be costly to the bottom line. Not paying timely attention to pure technology initiatives can lead to unrecoverable technical damage. The value of technology road map in this digital age is very high and mission critical.

 

Software Architecture, Technology Leadership

Building a strong technology team
Every company today is a technology or software company. Software automation is the backbone of business operations today. The lines between business operations and software development are blurring because the two groups work together more closely than ever. Technology leaders have a tremendous responsibility of developing a futuristic digital vision, and accordingly building a strong technology team which can accomplish the vision.
vision-exercise-1
The goal of building a 5 year technology vision which translates into a solid technology road map can be lost even in companies who believe they are ‘tech’ companies. The daily pressures of production support, development operations and releases can bog down most of the technology managers. However a strong technology leader makes it her most important responsibility to build a technology vision and road map which takes the company to the next level of success step by step, avoiding major technical debts. The leader must develop a vision along with her other technology leaders who are her reports or peers so they can be the torch bearers of the vision. The vision has to trickle down and across.
The product vision and technology vision will together guide in building a team which will execute both, to a successful outcome.
A technology road map addresses technical debts, refactoring, migrations and upgrades, new technology evaluations, tool assessment and introductions, and any other relevant technology initiatives which are essential to technology growth for the company. The product vision and technology vision will together guide in building a team which will execute both, to a successful outcome.
So, what does a strong technology team composition look like ?
People talent is core to a company’s growth-  developing a team which aligns with the best technology practices and future vision is key to building a strong team. A technology leader, manager can strengthen her team by assessing what the current team composition and talent spread is. This assessment exercise should be periodically repeated until an optimal skill and experience mix is reached.
                                                      team    

A technology leader can strengthen her team by assessing what the current team composition and talent spread is. This assessment exercise should be periodically repeated until an optimal skill and experience mix is reached.

 

  1. Make a list of skill sets you want to have in your team.
  2. Determine which skill sets exist and which do not.
  3. Where are the strengths ? Where are the weaknesses- you want to induce talent in the team which will complement the existing skill set. Hire people from outside who have these new talents, when they come, they naturally rub off on the existing team. This is a huge motivator – peers who bring new ideas and skills from which one can be inspired and challenged. A natural growth occurs.
  4. Determine what experience levels exist. Too many Juniors can lack the experience and domain expertise to deliver well to your business context. Too many seniors can create Silos- they are opinionated and  often comfortable in their set old ways. Hence a careful examination of team composition is always necessary in determining team morale and growth.
  5. Attract and hire fresh talent- fresh graduates and juniors are highly motivated, open and enthusiastic. You need a good mix of Juniors, Mid-Level and Senior staff. You have to have few Senior experienced leaders and mentors, some Juniors and rest Mid-level. A 30 % experienced, 40% mid, and 30% Junior approximately can be a healthy composition which will keep the team motivated , mentored and productive. An imbalance or dominance of any category can lead to side effects which are not healthy.
  6. Form a ‘Core architecture team or function’. Break ‘Silos’ by creating a formalized core architecture team, function or platform which establishes re-usability, reduces redundancy, and builds common services.

Form a ‘Core architecture team or function’. Break ‘Silos’ by creating a formalized core architecture team, function or platform which establishes re-usability, reduces redundancy, and builds common services.

What about professional growth, nurturing home grown talent ? 

A leader has to create an impact on the team in many different ways for professional development.

  1. Lead by example : you can demonstrate that as a leader you walk the talk. Demos, presentations, proof of concepts provide actual examples of what is being proposed and evangelized. People follow examples better than persuasion.
  2. Be hands on, deep and wide technically. A technology leader when ‘speaks’ to technology very well, often can make much greater impact.
  3. Appoint coaches on topics to promote learning. For example , say you want to spread Unit testing practices. Pick people in the team who are ahead in that, as appointed coaches. Likewise for example  ‘Agile’, ‘Build automation’ or any best practices where learning needs to be facilitated.  This can also foster a culture of learning from peers.
  4. Call out, above and beyond efforts.  Provide a platform for star performers and outliers to ‘show and tell’. Interesting events like Lunch n learns, ‘Coffee chats’ or other group meetings can be organized where team exchanges ideas in a more casual setting.
  5. Consensus building: let’s face it, in technology multiple view points exist. People can get passionate about their opinions- a leader may need to be patient and persuasive which of course is not easy. Often there is one solution which suits the context most, towards which the consensus has to be built. White boarding the context and various solutions, and mapping the most suitable in that situation will help build consensus.

One article can not cover the vast topic of how strong technology teams can be built, however everything starts with a vision and a road map. The leader then aligns the team with this vision consistently.  An assessment and measure of team skill and experience spread, will help carve a plan for building a strong technology team. Stay tuned for more on this topic – happy team building !

Design Patterns, Development, Software Architecture, Technology Leadership

Collaborating teams build systems that collaborate

Information systems of a company are a reflection of the communication patterns within the organization. The more teams communicate with one other cohesively, these systems will connect and grow together more as a result. In today’s world of digital transformation and data intelligence, it is imperative that multi function organizations communicate and collaborate, so systems built by the development staff will bring the necessary business intelligence and scale. This cohesive intelligence is critical to further growth. It is very common in multi function organizations for ‘Silos’ to develop over a period of time. These ‘Silos’ focus successfully on one aspect of the business, however they fail to connect with other systems in the workflow such that it reflects the customer journey. This ‘disconnect’ can create many black-boxes producing their own data, which as a whole brings minimal value in the long run.

It is very interesting that many popular methodologies and architectural patterns have evolved which fundamentally solve this problem of communication. Let’s take DevOps for example. DevOps is designed to have better collaboration between Development teams and Network operations. Although they are independent in function, lack of communication between them results in surprises, errors and continuous finger pointing. This can be very expensive operationally leading to client dissatisfaction. Today DevOps has had great success which leads to test, build and deployment automation. This in turn increases speed-to-market and system reliability.

In today’s world of digital transformation and data intelligence, it is imperative that multi function organizations communicate and collaborate, so systems built by the development staff will bring the necessary business intelligence and scale as well.

Agile development methodologies solve the collaboration problems between all disciplines of an organization. The SCRUM methodology with it’s different ‘ceremonies’ brings all relevant departments together in one room regularly. From Sprint Planning to Sprint Retrospective, different departments like products, development, QA, and user experience come together to discuss topics from requirements to ‘lessons learnt’. This collective, methodical way of communication leads to better product definition, risk mitigation, estimation and release planning. Everyone is in a continuous improvement and feedback loop, improving deliveries significantly.

collaboration

Let’s take Micro-services architecture, for example. This style of architecture was embraced by many companies in an effort to have small teams become autonomous by fostering a loosely coupled architecture connected via independent services. Again, at a high level team structures and their communication patterns dictate how the Enterprise architecture shapes. Netflix adopted the Micro-services architecture to increase speed-to-market which led to a social culture of small autonomous teams collaborating together.

Companies don’t just build products and offer services , they build professional communities and offer personal growth.

Any organization which looks to build information systems that work together, while they are independent units, has to build a communication culture and team structure which can facilitate such systems. Cohesive systems bring business intelligence while loose coupling allows them to scale. Cohesion comes from continuous collaboration and loose coupling comes from small autonomous teams. Adopt methodologies and architectural patterns which bring teams and people together. Companies don’t just build products and offer services , they build professional communities and offer personal growth.

Design Patterns, Development, Object Oriented Programming, SDLC, Software Architecture

Program to an interface , not an implementation

I have been lately discussing SOLID in my previous posts – some of the basic principles of OOD . Recently while doing my project work , we had to decide between Entity Framework  or NHibernate which are popular ORM layers in .Net framework .  The decision was not easy , we got stuck at Second Level caching support …and could not reach any decision right away. I started thinking that may be for now we have to write our layer in such a way that we choose one ORM , and then we can switch to a different one worst case if need be. Ideally a situation like this should not arise , but if it does happen , you need to be able to do it.  Once again I felt how a very fundamental OOD principle could come in handy –

Program to an interface , not an implementation.

These words above can be very ambiguous and abstract when you just read them .

However when seen from a context , you understand how powerful this is , and how it can increase flexibility and ability to switch . Seen from my own situation where we needed the flexibility to switch data access mechanism if need be, let’s understand this by constructing a scenario. For example , a common problem could be implementing a subscription mechanism for a data service that your system may provide. Users can subscribe to the data service and get notified when a change occurs. We need to persist the subscription information to the database , where you will need to use some database access mechanism.

As usual there is a presentation layer that gathers some subscription information and passes it along to the data service which persists the information to the database . For simplicity’s sake , let’s say we have a SubscriptionService , which is our data service for persisting subscriptions . We can discuss this further in the context of an ASP.Net MVC application – where a View is our Presentation layer through which Subscription details are collected. A controller action method is typically invoked to pass along the information to the server.

We can assume for our article purposes that our application provides notifications on Food Recalls. So the member user to our website subscribes to recalls for a certain type of food issued by different organizations like FDA etc. The Subscription in simplicity could be :

Please note : all code is pseudo only , this is not a working solution that can be downloaded and built.

public class RecallsSubscriptionService
{
    Boolean Subscribe( int userId, string food_type, string organization)
    {
           // data access code to insert subscriptions
           return false; 
    }         
}

Most of the times the controller action method would look something like this:

public class SubscriptionController : Controller
{
        [HttpPost]
        public ActionResult Subscribe(int userId, string food_type, string organization)
        {
            try
            {
                 SubscriptionService myservice = new SubscriptionService();
                 myservice.Subscribe(userId, food_type, organization);

                return RedirectToAction("Index");
            }
            catch
            {
                return View();
            }
        }
}

So , we make a decision we will use Entity Framework 6 to implement our data access to store subscription information. The above code will work fine . The code goes into production . However later we decide that due to certain project requirements we have to switch to NHibernate. This is a tough situation – our controller here , who is a client of the SubscriptionService , is heavily dependent on instantiating this specific service which accesses the database with Entity Framework 6. Although we expected the problem could arise we didn’t design our system for this flexibility. Even though we encapsulated the data access code into a Service , we did not free the client from the implementation of data access because the client program is indirectly tied to EF implementation.

We need to make some changes , so our controller is independent of the future changes that could happen to the database access mechanism. To bring transparency into how data access may happen from a client’s perspective we need to introduce interfaces. We write an interface instead for the SubscriptionService :

interface IRecallsSubscriptionService
{
   Boolean Subcribe(int userId, string food_type, string organization);       
}

We could write two implementations of the above interface or we could just use the same implementation we had earlier , and replace it with NHibernate.

public class RecallsSubscriptionServiceEF
{
       Boolean Subscribe(int userId, string food_type, string organization) 
       { 
                //   implement using EF ;
                return false;
       }         
}

public class RecallsSubscriptionServiceNH
{
       Boolean Subscribe(int userId, string food_type, string organization) 
       {
              //implement using NH ;
              return false;
       }         
}
/*( not that the above approach is recommended , where you keep 
two implementations available for the same requirement. 
This is for understanding purposes only )*/

The above mechanism allows us to drive the Subscribe method call through the IRecallsSubscriptionService interface. In order to effectively use the method above , we will need to pass IRecallsSubscriptionService as a parameter to the controller constructor and use dependency injection in this scenario to have the right concrete class to be instantiated during run-time. I will not dive deeper into that because it is outside of the scope of this topic.

This is the whole basis of Dependency Inversion and Injection ( this is also the ‘D’ in SOLID ): here we program to an interface , not an implementation as we saw above. This gave us the flexibility to change our implementation completely without changing the client code. So the basic idea is programming to an interface decouples the client from the internal implementation freeing calling programs from knowing the internals leading to flexibility in changing implementation as the need arises.

Even in scenarios where there are published APIs , when you use objects  go up in hierarchy to program to the interface as opposed to the concrete class. So if you are using a List object and comparing individual strings-  then if this object implements IComparer , then program to IComparer –

instead of writing:

List myList = new List();
myList.Compare( myList[0] , myList[1]);

//Write as :

IComparer myList = new List();
myList.Compare( myList[0], myList[1] );

//This gives you the freedom to use a different flavor of Compare method if you need to ,

IComparer myList = new Array();
myList.Compare( myList[0], myList[1] );

The advantage of the above is ability to switch different comparison methods if they are available. And also test program can easily switch different implementations and see what works the best.

We also need to understand that “Program to an interface” should not be taken literally . Even Abstract classes can act as interfaces in this context and give us the same flexibility.

Interfaces should be used wisely and only when needed. If there is no potential for an implementation to ever change there is no need to bother with interfaces. For example , a Utility class that converts a string to a number. There is no need to write interface to this . Interfaces should also be used in projects where testability is a big concern and mocking may be required.