In an agile and rapid software development environment, it is hard to keep control of Quality Assurance(QA) and staging builds when you have too many projects running and several stakeholders (other developers, QA team, Account Managers, Senior Management, Project Managers, Designers ...etc) who need to check, verify and validate builds before moving to client environments. With manual deployment, someone who has access to the staging servers has to create a build package and perform the deployment procedure - this can be different depending on the web project type. But what if (s)he were not available? What if there were too many projects that would consume a lot of unnecessary time?
You can give more people access to the staging servers, developers for instance. But what if developers made an urgent hotfix directly on those servers for other stakeholders to check and then forgot to update the code repository with that code?
We built a NodeJS application that acts as a build/deploy composer that theoretically can handle any type of web solutions and can run on any type of Operating System: Microsoft Windows, Mac OS or Linux. We called it a composer because it is not a native build server for a specific technology, it is rather hybrid and combines tools from different technologies to build the solutions and deploy them.
The diagram below shows, on a high level, the main components of the approach and how they interact together.
To put a little context to the diagram, we highlighted each step in this sequence:
The NodeJS application will be running on your staging servers, wherever they are, whether they are local or hosted on an online service. The only requirement is that the NodeJS application should have a publicly accessible URL.
Git hooks are a facility that Git repositories provide as a way of communicating updates for integration with external parties. Some hooks are preconfigured by the hosts to communicate and integrate with specific systems such as Campfire, Grove or Twitter. A very famous hook is email, which comes without any configuration. If you choose to watch (or follow) a repository for instance, you get email notifications to the email address registered with your Git account.
In addition to the preconfigured Git hooks, there is a POST hook that is generic and could be consumed and used as suited for your business(this is the one we used). POST hook does an HTTP POST request that contains detailed information about a Git update in HTTP form encoding that could be received by any application.
For instance, if you host your code repositories on BitBucket, you can configure a POST hook by following these steps:
In the link above, username and password are optional. If you wish to use basic HTTP authentication to have more security, you can enter the credentials there which you will validate in the NodeJS server. The port field is also optional depending on your URL.
The NodeJS server acts as the middleware that connects your Git actions to the staging environment. It consumes the Git Hook then receives, decodes and parses the posted data. After processing, it determines which command to execute depending on the type and technology of the code repository.
The first thing NodeJS server does is the security checks:Check the incoming HTTP request verb. It must be always POST, otherwise it throws an HTTP verb not allowed, 405 HTTP codeCheck the source IP addresses against the whitelisted IP's that are allowed to send data to the NodeJS server. Those IP address must be preconfigured in both your staging server's firewall and the NodeJS application. Note that those IP could be changed so you might need to keep an eye on them. If the source IP was not recognized on the NodeJS application level, it will return an unauthorized HTTP error code, 401 HTTP codeCheck the HTTP basic authentication credentials that come within the HTTP POST request, which you enter when you setup the Git Hook in your repository settings. If provided credentials are incorrect the NodeJS application throws an unauthorized HTTP error code, 401 HTTP code
Once all security checks are validated and passed, the NodeJS server proceeds by parsing the POST data using an application/x-www-form-urlencoded parser.
You can use NodeJS modules to simplify your application logic like express. body-parser, basic-auth ...etc.
Configuration data is the backbone that controls the flow of the server. It is a JSON object that is stored in the server and given no public access. It contains mainly two types of data (in addition to anything you want to keep configured).
Server-specific configuration such as flags - e.g. Great if you want to check for HTTP basic authentication or not (with allowed credentials) or check for whitelisting or not (with whitelisted IP addresses), ...etc. It also contains the local paths of build/deploy commands specific to each supported technology.
Repository-specific configurations - these contain an array of repositories that you need to automate their build and deployment. For each repository object you need:
Commands are script files that contain instructions related to a specific technology that build and deploy a web solution. Those instructions are written in an Operating System specific language - in Microsoft Windows you can use command line or PowerShell, for MacOS and Linux you can use bash scripts.
For instance, you can build and deploy .NET web solutions using Microsoft Build Tool with command line script using this instruction (executed after the Git pull request succeeds): msbuid_path project_file_path_in_repository "/p:Configuration=staging_configuration" /p:DeployOnBuild=true "/p:PublishProfile=staging_profile". You can use additional parameters as well to specify how you want the build log to be handled. The configuration and publish profiles has to be present in the repository (they can be created in development machines using Microsoft Visual Studio) and they help determine the publish (i.e. deploy) path, the build configuration that would publish the configuration files (i.e. web.config) in the proper transformation.
The command script's instructions should be written in a way to read parameters like the project file path from input arguments, and those arguments will be read by the NodeJS server from the configuration file and passed to the execution command as arguments. You can use the child_process NodeJS module to handle executing the command scripts.
January 9, 2020
Subscribe to our blog to receive relevant news and tips about digital transformation, app development, website development, UX, and UI design. Promise we won't spam you.