Bunk8s: Enabling Easy Integration Testing of Microservices in Kubernetes
Extracted source text ยท arXiv 2207.06811
\documentclass[10pt, conference]{IEEEtran}
\usepackage{todonotes}
\usepackage[nocompress]{cite}
\usepackage{url}
\usepackage{enumitem}
\usepackage{epstopdf}
% \epstopdfDeclareGraphicsRule{.tif}{png}{.png}{convert #1 \OutputFile}
% \epstopdfDeclareGraphicsRule{.tiff}{png}{.png}{convert #1 \OutputFile}
% \PrependGraphicsExtensions{.tif, .tiff}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{amsmath}
\usepackage{tabu}
\usepackage{multirow}
\usepackage{adjustbox}
\usepackage{fontawesome}
% \usepackage{minted}
% \usepackage[frozencache=true,cachedir=minted-cache]{minted}
% \usepackage[finalizecache,cachedir=minted-main]{minted}
\usepackage{algpseudocode}
\usepackage{soul}
\usepackage[bookmarks=false]{hyperref}
\usepackage{footnote}
\usepackage{lipsum}
\usepackage[ruled,vlined, linesnumbered]{algorithm2e}
%\setlength{\belowcaptionskip}{2pt}
\usepackage{subcaption}
\usepackage{listings}
\usepackage{booktabs}
%\usepackage{stfloats}
\usepackage{tikz}
\usetikzlibrary{positioning}
\soulregister\cite7
\soulregister\ref7
\soulregister\pageref7
\soulregister\secref7
\soulregister\tabref7
% For tables or so
\newcommand{\Cline}[2]{%
\noalign{\global\setlength{\Oldarrayrulewidth}{\arrayrulewidth}}%
\noalign{\global\setlength{\arrayrulewidth}{#1}}\cline{#2}%
\noalign{\global\setlength{\arrayrulewidth}{\Oldarrayrulewidth}}}
\setlength{\tabcolsep}{1.0em}
\renewcommand{\arraystretch}{1.25}
\newcommand*\circled[1]{\tikz[baseline=(char.base)]{
\node[shape=circle,draw,inner sep=0.75pt, text=white,fill=black] (char) {#1};}}
% Please leave notes to a specific co-author by using these macros
% \newcommand{\todomc}[1]{\todo[color=cyan!60,inline,size=\small]{Mohak: #1}}
% \newcommand{\todomg}[1]{\todo[color=yellow!60,inline,size=\small]{Dr. Gerndt: #1}}
\newcommand{\todomc}[1]{\todo[color=cyan!60,inline,size=\small]{Mohak: #1}}
\newcommand{\todochris}[1]{\todo[color=yellow!60,inline,size=\small]{Christoph: #1}}
\newcommand{\commentanshul}[1]{\todo[color=red!60,inline,size=\small]{Anshul: #1}}
\newcommand\copyrighttext{%
\footnotesize \copyright 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes, creating new
collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted
component of this work in other works.}
\newcommand\copyrightnotice{%
\begin{tikzpicture}[remember picture,overlay]
\node[anchor=south,yshift=10pt] at (current page.south) {\fbox{\parbox{\dimexpr\textwidth-\fboxsep-\fboxrule\relax}{\copyrighttext}}};
\end{tikzpicture}%
}
\definecolor{ao}{rgb}{0.0, 0.5, 0.0}
% \definecolor{db}{rgb}{0.0, 0.0, 1.0}
\definecolor{db}{rgb}{0.2, 0.2, 0.6}
\definecolor{cadmiumgreen}{rgb}{0.0, 0.42, 0.24}
\newcommand{\scorep}{\mbox{Score-P}}
% The red color should be removeded at the very end. Until then it makes it easier to spot referencing issues
\newcommand{\figref}[1]{Figure~\ref{#1}}
%\newcommand{\figref}[1]{\textcolor{red}{Figure~\ref{#1}}}
\newcommand{\tabref}[1]{Table~\ref{#1}}
%\newcommand{\tabref}[1]{\textcolor{red}{Table~\ref{#1}}}
\newcommand{\secref}[1]{Section~\ref{#1}}
\renewcommand{\algref}[1]{Algorithm~\ref{#1}}
\SetAlFnt{\footnotesize}
% \newcommand\copyrighttext{%
% \footnotesize \copyright 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including
% reprinting/republishing this material for advertising or promotional purposes, creating new
% collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted
% component of this work in other works.}
% \newcommand\copyrightnotice{%
% \begin{tikzpicture}[remember picture,overlay]
% \node[anchor=south,yshift=10pt] at (current page.south) {\fbox{\parbox{\dimexpr\textwidth-\fboxsep-\fboxrule\relax}{\copyrighttext}}};
% \end{tikzpicture}%
% }
\begin{document}
\title{Bunk8s: Enabling Easy Integration Testing of Microservices in Kubernetes}
\author{\IEEEauthorblockN{Christoph Reile\IEEEauthorrefmark{1}, Mohak Chadha\IEEEauthorrefmark{1}, Valentin Hauner\IEEEauthorrefmark{2}, Anshul Jindal\IEEEauthorrefmark{1}, Benjamin Hofmann\IEEEauthorrefmark{2}, Michael Gerndt\IEEEauthorrefmark{1}\\}
\IEEEauthorblockA{\IEEEauthorrefmark{1}Chair of Computer Architecture and Parallel Systems, Technische Universit{\"a}t M{\"u}nchen \\
Garching (near Munich), Germany \\ \IEEEauthorrefmark{2}MaibornWolff GmbH, Germany \\}
Email: \IEEEauthorrefmark{1}\{firstname.lastname\}@tum.de, \IEEEauthorrefmark{2}\{firstname.lastname\}@maibornwolff.de}
% make the title area
\maketitle
%Uncomment before final submission
% \pagenumbering{gobble}
\copyrightnotice
% \copyrightnotice
% As a general rule, do not put math, special symbols or citations
% in the abstract
% \input{sections/00_abstract}
\begin{abstract}
Microservice architecture is the common choice for cloud applications these days since each individual microservice can be independently modified, replaced, and scaled. However, the complexity of microservice applications requires automated testing with a focus on the interactions between the services. While this is achievable with end-to-end tests, they are error-prone, brittle, expensive to write, time-consuming to run, and require the entire application to be deployed. Integration tests are an alternative to end-to-end tests since they have a smaller test scope and require the deployment of a significantly fewer number of services. The de-facto standard for deploying microservice applications in the cloud is containers with Kubernetes being the most widely used container orchestration platform. To support the integration testing of microservices in Kubernetes, several tools such as Octopus, Istio, and Jenkins exist. However, each of these tools either lack crucial functionality or lead to a substantial increase in the complexity and growth of the tool landscape when introduced into a project. To this end, we present \emph{Bunk8s}, a tool for integration testing of microservice applications in Kubernetes that overcomes the limitations of these existing tools. \emph{Bunk8s} is independent of the test framework used for writing integration tests, independent of the used CI/CD infrastructure, and supports test result publishing. A video demonstrating the functioning of our tool is available from \url{https://www.youtube.com/watch?v=e8wbS25O4Bo}.
\end{abstract}
\begin{IEEEkeywords}
DevOps, Integration Testing, Kubernetes, Microservices
\end{IEEEkeywords}
% \IEEEpeerreviewmaketitle
% \copyrightnotice
% \thispagestyle{empty}
% \todochris{Write details about the supervisors from MaibornWolff Valentin, Benjamin in the author section}
% Valentin Hauner, valentin.hauner@maibornwolff.de, Senior Software Engineer at MaibornWolff GmbH since 2016, Munich
% Benjaming Hofmann, benjamin.hofmann@maibornwolff.de, Senior Lead Test Architect at MaibornWolff GmbH since 2018, Munich
\input{sections/01_introduction}
\input{sections/02_background_related_work}
\input{sections/03_system_design}
% \input{sections/04_evaluation}
\input{sections/05_case_study}
\input{sections/06_why_bunk8s}
\input{sections/07_conclusion_future_work}
\input{sections/08_acknowledgement}
% \section{Acknowledgement}
% This work was supported by the funding of the German Federal Ministry of Education and Research (BMBF) in the scope of the Software Campus program. Google Cloud credits were provided by the Google Cloud Platform research credits.
\bibliographystyle{IEEEtran}
\thispagestyle{empty}
\bibliography{parallelpgm}
% \listoftodos
\end{document}
\section{Introduction}
\label{sec:intro}
%Think about changing title
%explaining one line diff between broad/narrow in intro
% Containerizing components of an application is a popular strategy to enable the deployment of a component as an independent, isolated service, where each instance of a service runs inside a container \cite{miciocont}.
% This deployment strategy is often used for applications that are based on a microservice architecture, where an application is structured "as a set of loosely coupled, collaborating services"\cite{micioarch}.
% Nearly 90\% of deployed containers are orchestrated \cite{datadog}. With a 75\% usage share, the most utilized orchestration platform is Kubernetes \cite{sysdig}, which can be deployed and operated on-premise or in a public cloud, such as Microsoft Azure or Amazon Web Services. They also offer a wide range of managed services, that can be used to build an infrastructure for an application and DevOps tools \cite{aws-product-catalog} \cite{azure-product-catalog}.
% If a microservice-based application is deployed to a Kubernetes cluster, it may rely on infrastructure services that run outside of Kubernetes, for example, services that provide storage or network functionalities.
% If the Kubernetes cluster runs in a public cloud, the public cloud can provide those infrastructure services, but only the infrastructure's API can be accessed. It is not possible to test the interaction of modules from application microservices with modules from infrastructure services in isolation from other modules of the infrastructure service.
% Therefore, testing interactions of application microservices with infrastructure services can only be accomplished by running end-to-end tests or broad integration tests. The former are brittle, expensive to write, time-consuming to run \cite{fowlerpyr} and require the whole application to be deployed.
% If instead only subsystems of the application, in which changes were made, are tested with broad integration tests from outside the cluster, another impediment arises.
% The subsystem may only consist of backend services, which do not possess interfaces that can be accessed from outside of Kubernetes via its public interfaces.
% This makes it necessary to add services to the testing scope, that build communication paths to the backend services.
% This in turn may increase the testing scope to such a high degree, that it leads to the same downsides as with end-to-end tests and a less effective test suite. Alternatively, additional public interfaces could be added by deploying more Ingresses to Kubernetes, that point to the backend services. However, since Kubernetes' Ingresses only support routing HTTP or HTTPS traffic inside the cluster, it may be not possible to build a connection with a backend service, that can not communicate via HTTP or HTTPS.
% A solution to this problem would be to run the tests inside of Kubernetes, thereby making the deployment of additional services or the reconfiguration of Kubernetes obsolete, since tests can directly access interfaces of deployed services. This allows for reduced scopes of systems under test. There already are multiple tools that allow to run integration tests inside of Kubernetes, namely Kyma Octopus, Istio Testing Framework and Jenkins, however each of them comes with it's own downsides. These downsides are, for example, missing features that are crucial to the testing process or dependencies on other technologies which would serverely increase a project's complexity if introduced.
% The main contributions of this paper are:
% \begin{enumerate}
% \item Implementation of a tool called Bunk8s that enables broad integration testing of microservice applications in Kubernetes.
% \item A comparison of Bunk8s with the existing tools, showing that Bunk8s compensates for their shortcomings.
% \item Demonstration of the functioning of Bunk8s by running integration tests for a room tracking application.
% \end{enumerate}
% Linux containers, due to their portability and high availability, have become the de-facto standard for developing, testing, and deploying a wide range of applications from enterprise to web services in cloud environments~\cite{cloud_container}. This is because containers enable users to package their application along with its custom software dependencies as a single unit into easy-to-deploy images.
%Linux containers, due to their portability, isolation, and high availability, have become the de-facto standard for developing, testing, and deploying a wide range of applications from enterprise to web services in cloud environments~\cite{cloud_container}. This is because containers enable users to package their application and its custom software dependencies as a single unit into easy-to-deploy images. Moreover, containers are a natural fit for applications based on a microservice architecture in the cloud due to their smooth integration with container orchestration platforms~\cite{kubesphere} for efficient resource management. In a microservice architecture, software systems are broken down into independently deployable components that can be directly translated into services and automatically scaled on-demand. Components are independently replaceable and upgradable units of software~\cite{fowlermic}, consisting of several modules~\cite{clemson}. Recent trends~\cite{datadogreport} show that nearly $90$\% of all deployed containers are orchestrated, with Kubernetes (k8s)~\cite{kuber} being the most utilized container orchestration platform~\cite{sysdig}.
Microservices-based applications are typically composed of independently deployable components that can be directly translated into services and automatically scaled on-demand. Components are independently replaceable and upgradable units of software~\cite{fowlermic}, consisting of several modules~\cite{clemson}. Linux containers, due to their portability, isolation, and high availability, have become the de-facto standard for developing, testing, and deploying such applications in cloud environments~\cite{cloud_container}. This is because containers enable users to package their application and its custom software dependencies as a single unit into easy-to-deploy images. Moreover, containers are a natural fit for microservices-based applications in the cloud due to their smooth integration with container orchestration platforms~\cite{kubesphere} for efficient resource management. Recent trends~\cite{datadogreport} show that nearly $90$\% of all deployed containers are orchestrated, with Kubernetes (k8s)~\cite{kuber} being the most utilized container orchestration platform~\cite{sysdig}.
% Applications based on a microservice architecture are structured as a set of loosely coupled, independently deployable, and collaborating services~\cite{microservicearch} that can be automatically scaled on-demand.
Developers can deploy k8s on-premise or can utilize the services offered by most commercial cloud providers such as Microsoft (Azure Kubernetes Service)~\cite{aks}. If a microservice-based application is deployed to a k8s cluster, it may rely on backend services that run outside the k8s cluster. For instance, services that provide storage or networking functionalities. For commercial cloud providers, these services are self-managed, with only the service's live API available for the developer to access. As a result, testing the interactions of modules from a microservice application with the modules from the backend service requires the entire backend service to be deployed. It cannot be done in isolation with only specific modules of the backend service. Therefore, testing the interactions of a microservice application with backend services can only be accomplished by running end-to-end tests or broad integration tests. Note that, integration tests can be classified into narrow and broad depending on their granularity (\S\ref{sec:integeration_testing}).
End-to-end tests are brittle, expensive to write, time-consuming to execute, and require the deployment of the entire application~\cite{fowlerpyr}. Broad integration testing of microservices in k8s can be done in two ways, i.e., from inside or outside the cluster. Running broad integration tests for microservices from outside the k8s cluster has several drawbacks, making them as inefficient as end-to-end tests. For example, the lack of public interfaces in the services which allow them to be accessed from outside the k8s cluster. As a result, the standard way of broad integration testing of microservices is from inside the k8s cluster. For the integration testing of microservices from inside the k8s cluster, several tools such as Octopus~\cite{octopus}, Istio~\cite{istio}, and Jenkins~\cite{jenkins} exist. However, the existing tools lack crucial functionalities or are dependent on particular infrastructure for the execution of these tests. To this end, we present \emph{Bunk8s} that overcomes the limitations of these existing tools.
% For this, several tools such as Octopus, Istio, and Jenkins exist. However,
% \todomc{Work on intro later}
Towards easy integration testing in Kubernetes, our key contributions are:
\begin{itemize}
\item We implement and present \emph{Bunk8s}\footnote{https://github.com/Teiktos/bunk8s}, a tool for broad integration testing of microservices in Kubernetes.
\item \emph{Bunk8s} is independent of the test framework used for writing integration tests, independent of the used CI/CD infrastructure, and supports publishing of test results. Thus, making the inclusion of \emph{Bunk8s} in a project relatively easy.
\item We demonstrate the functioning of \emph{Bunk8s} by running broad integration tests for a real-world microservice application on a commercial cloud provider, i.e., Azure.
\end{itemize}
The rest of the paper is structured as follows. \S\ref{sec:integeration_testing} provides a background on integration testing. In \S\ref{sec:bunks}, the system design of \emph{Bunk8s} and its workflow for integration testing of microservices are described. In \S\ref{sec:case_study}, we demonstrate the functioning of \emph{Bunk8s} for the integration testing of a real-world microservice application. \S\ref{sec:whybunk8s} motivates the need for \emph{Bunk8s} and presents a comparison with other tools for integration testing of microservices. Finally, \S\ref{sec:futurework} concludes the paper and presents an outlook.
% Moreover, they can be easily scaled to meet application demand using container orchestration platforms such as Kubernetes~\cite{kuber}, offered as a service by commercial cloud providers such as Microsoft (Azure Kubernetes Service)~\cite{aks}.
% \section{Background \& Related Work}
% \label{sec:related_work}
% There already exist tools and test frameworks, that follow the approach of running integration tests inside Kubernetes. However, all of them are either limited in their applicability, since they require a specific tech stack to be executable or lack functionalities, that are crucial to the testing process.
% \subsection{Kyma Octopus}
% One of those testing tools is Octopus, where tests are defined with custom resource definitions and for each integration test, a pod is deployed to Kubernetes. It supports rerunning tests to prevent flaky test results. After the tests are finished, it is only reported, if the test successfully finished or failed. The logs that the tests create are not extracted from the containers and therefore are not published. This does not allow an in-depth analysis of why exactly a test failed \cite{github-octopus}\cite{kyma-octopus}. This framework does not support mocking.
% \subsection{Istio Testing Framework}
% The Istio testing framework is supposed to test microservice applications, that use the Istio service mesh to handle communication between microservices. The framework is an expansion of the Go testing package and therefore only supports writing integration tests in Go. Furthermore, it requires Prow as CI/CD tool and GitHub to execute the tests automatically \cite{istiotest}. This framework also does not support mocking.
% \subsection{Jenkins}
% There exists another approach, that is based on Jenkins and allows to run tests inside of Kubernetes, as well as mocking \cite{jenkins-itest} \cite{jenkins-itest-git}. However, the test runner containers can only be used to test the interaction of containers within pods, respectively the interaction of modules of microservices. This makes this approach only suited for narrow integration testing of microservices, not broad integration testing. Additionally, in order to use it, Jenkins would have to be introduced into a project and at least parts of the CI/CD pipeline would have to be migrated to Jenkins if it is not already in use.
\section{Background}
\label{sec:integeration_testing}
\begin{figure}[t]
\centering
\includegraphics[width=0.55\columnwidth]{figures/narrow_vs_broad_integration_testing.png}
\caption[Room Tracking System Components]{Narrow and Broad integration testing.}
\label{fig:narrow_broad_int_tests}
\end{figure}
% In microservice architectures, software systems are broken down into independently deployable components, which can directly be translated into microservices. They are independently replaceable and upgradable units of software \cite{fowlermic}, consisting of several modules \cite{clemson}.
% Integration tests are "typically used to verify interactions between layers of integration code and the external microservices to which they are integrating"\cite{clemson}. They can be further categorized by their granularity, into narrow and broad integration tests. The former refers to only testing the modules of microservice, that communicate with each other.
Integration tests are generally used to verify the interactions between the layers of integration code and the external microservices to which they are integrating~\cite{clemson}. They can be further categorized by their granularity, into narrow and broad integration tests. Narrow integration tests refer to testing only the modules of microservices that communicate with each other. Therefore, they only require instantiating
the modules under test~\cite{fowlerinttest}. Microservice applications that run in commercial cloud offerings of k8s may integrate with services that run outside of k8s, e.g., Azure Cosmos DB, Azure Service Bus. Such services only provide developers with their live versions and it is not possible to instantiate only modules of these services that can integrate with the components of the microservice application. As a result, testing the interaction of microservices and services external to Kubernetes is not possible with narrow integration tests. On the other hand, broad integration tests refer to testing code paths through several services and all of their modules. Moreover, broad integration tests can be of varying granularity, depending on the scope of the subsystem under test~\cite{clemson}. They require live versions of the components under test, a substantial test environment and network access \cite{fowlerinttest}. The difference in the code paths used for writing narrow and broad integration tests is shown in Figure~\ref{fig:narrow_broad_int_tests}.
% For developers, only live versions of these services are available and it is not possible to instantiate only modules that can integrate with the components of the microservice application.
% The latter refers to testing code paths through several services and all of their modules, not limited to communication modules. Broad integration tests can be of varying granularity, depending on the scope of the subsystem under test \cite{clemson}. The difference in exercised code paths of narrow and broad integration tests is depicted in \autoref{fig:narrow_broad_int_tests} \cite{fowlerinttest}.
% Microservice applications that run in public cloud offerings of Kubernetes may integrate with services that run outside of Kubernetes. In the context of Microsoft Azure, these services could be storage services, e.g. Data Lake or Azure Blob Storage or message services, e.g. Queue Storage. For those services, only the live versions are available for developers, but it is not possible to instantiate only modules that can integrate with components of the microservice application. Therefore, testing the interaction between microservices and those services external to Kubernetes is not possible with narrow integration tests but only with broad integration tests or end-to-end tests. However end-to-end test are brittle, expensive to write and time-consuming to run, since they require the whole application to be deployed. These downsides do not apply to broad integration tests, due to their smaller testing scope.
%cannot test specific modules without deploying the entire service -> clariy
% Using broad integration tests in your test suite allows to prevent the occurrence of an ineffective test suite.
% Integration tests are "typically used to verify interactions between layers of integration
% code and the external components to which they are integrating"[5]. They exercise
% communication paths in subsystems, whereby the tests can be of any granularity,
% depending on the scope of the subsystem under test [5]. They can be further categorized
% by their granularity, into narrow and broad integration tests. The former refers to only
% testing the modules of components, that communicate with each other. The latter refers
% to testing code paths through several components and all of their modules, not limited
% to communication modules. .
\section{Bunk8s}
\label{sec:bunks}
% In this section, the functional and non-functional requirements on Bunk8s will be elicited and the system design will be presented using a component diagram. Afterwards, the technical implementation of Bunk8s will be explained and elaborated on by describing step-by-step how a testing process with Bunk8s works.
In this section, we first describe our design goals for the development of \emph{Bunk8s}. Following this, we present its system design. Finally, we describe in detail, the workflow of \emph{Bunk8s} for the broad integration testing of microservices in k8s.
\subsection{Design Goals}
% The goal is to design Bunk8s so that it enables broad integration tests while compensating for the shortcomings of the tools discussed in the previous chapter. From this objective, the following requirements can be derived.
% Bunk8s must be applicable independently of the programming languages services are written in, as well as independently of the testing framework that is used to write or generate tests. Furthermore, it must not be dependant on specific CI/CD tools. These requirements are derived from the limitations of the testing approaches with Jenkins and the Istio testing framework. Istio requires prow as CI/CD tool and only supports writing tests with the Go testing framework. Additionally, it requires the Istio service mesh to be implemented into the application and does not work without Istio or with another service mesh.
% The testing with Bunk8s must also support the extraction and publishing of test results, which is not supported by Kyma Octopus.
% Additionally, Bunk8s must support starting multiple test runs at the same time and be compatible with all container runtimes, that can be installed into Kubernetes nodes, namely containerd, CRI-O, and Docker \cite{ch4-tka-containerruntimes}.
% The communication between components that run inside Kubernetes and components that run inside of Kubernetes must be via HTTP or HTTPS since Kubernetes' ingresses only expose HTTP and HTTPS routes from the outside of the cluster to the Kubernetes services.
% Lastly, Bunk8s must be portable between different public cloud offerings of Kubernetes.
% To enable the broad integration testing of microservices in Kubernetes and to facilitate the ease-of-use for utilizing \emph{Bunk8s} in a project, we chose the following design goals:
To facilitate the ease-of-use for utilizing \emph{Bunk8s} in a project and to overcome the limitations of the current tools for the integration testing of microservices in k8s (\S\ref{sec:whybunk8s}), we chose the following design goals:
\begin{itemize}
\item Independence of the testing framework used for writing and generating tests.
\item Independent of the specific CI/CD tools used within a project.
\item Compatible with different container runtimes such as containerd~\cite{containerd} and CRI-O~\cite{crio}.
\item Portable across the different Kubernetes offerings by the commercial cloud providers such as AKS~\cite{aks}, GKE~\cite{gke}.
\item Support for publishing of test results.
\end{itemize}
\subsection{System Design}
\label{sec:sysdesign}
% The underlying approach with Bunk8s is to create a tool, that allows deploying test runner containers in test runner pods to a testing namespace. In this namespace, the services under test are deployed. The test runner containers will then execute the tests. The setup of cloud infrastructure and the deployment of the services under test is not a functionality of Bunk8s and should be handled by tools that are suited for that task, e.g. Helm or Terraform\footnote{https://www.terraform.io/}.
\emph{Bunk8s} allows deploying test runner containers in test runner pods to a testing namespace. In this namespace, the microservices under test are deployed. Note that, test runner containers are written by developers and contain the integration tests for the microservice application. Following this, the tests are executed by the test runner containers. Setting up of the cloud infrastructure and the deployment of the microservices under test is not a functionality of \emph{Bunk8s} and should be done with tools such as Helm~\cite{helm} or Terraform~\cite{terraform}. The different components of \emph{Bunk8s} are shown in Figure~\ref{fig:bunk8s_component}. It consists of two components, i.e., \circled{1} the Launcher and \circled{2} the Coordinator. The Launcher runs in the CI/CD pipeline while the Coordinator runs in the k8s cluster.
% It consists of three components, i.e., \circled{1} the Launcher, \circled{2} the Coordinator, and \circled{3} the Test Runner. All components of \emph{Bunk8s} are implemented using \texttt{Go}.
\begin{figure}[t]
\centering
\includegraphics[width=0.65\columnwidth]{figures/component_diagram.png}
\caption[Bunk8s components]{Component diagram of \emph{Bunk8s}.}
\label{fig:bunk8s_component}
\end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[width=\columnwidth]{figures/example_application_testrun_communication.png}
% \caption[Bunk8s components]{Component diagram of Bunk8s}
% \label{fig:bunk8s_component}
% \end{figure}
% Bunk8s will consist of two components, a launcher running in the pipeline and a coordinator running in the Kubernetes cluster. A component diagram is given in Figure \ref{fig:bunk8s_component}.
\subsubsection{The Launcher}
\label{sec:launcher}
% The Launcher runs outside of Kubernetes as part of the continuous integration process in a pipeline.
% After the container images of the services under test are built and pushed to a container registry by the pipeline, the Bunk8s launcher is started, which starts the testing process.
% It loads a configuration file, which provides the information that is required by the launcher in order to connect to the coordinator. This includes the IP or URL of the Kubernetes ingress that has the Bunk8s coordinator configured as the backend service. It also provides information required by the controller for the test run. This includes names of the test runner images, in which namespace to deploy the test runner pods, the name of the test runner pods and a timeout.
% The Launcher connects to the coordinator via HTTP and sends the data.
% After the testing is finished and the coordinator responds to the launcher, the launcher calls the \textit{kube-apiserver} and extracts the test results from the test runner containers via HTTP and stores them on the pipeline runner. After the extraction, the test results can be published with the pipelines default test result publishing functionality, e.g. GitLab artifacts.
% The launcher will be designed to run in a container, e.g. as a Docker container or another arbitrary container runtime, making it platform independent and easily adaptable to other container runtimes.
The Launcher runs outside of k8s as part of the continuous integration (CI) process in a pipeline. After the container images of the microservices under test are built and pushed to a container registry by the pipeline, the \emph{Bunk8s} launcher is started, which in turn starts the testing process. The launcher takes a \texttt{YAML} configuration file as input, an example of which is shown in Listing~\ref{lst:config_file}. The configuration file provides the relevant information that is required by the launcher for connecting to the coordinator (\S\ref{sec:coordinator}), i.e., its IP address and port number. The developers can also provide a TLS certificate file for an authorized connection that is placed in the container image of the launcher. The configuration file also includes information required by the coordinator for running the integration tests. This includes the names of the test runner images, the namespace to which the test runner pods should be deployed, the name of the test runner pods, and a test timeout value. Note that, the list \texttt{testRunnerPods} allows defining multiple test runner pods which can be deployed in different namespaces, while the list \texttt{containers} allows defining multiple test runner containers per pod. The launcher connects to the coordinator via HTTP and sends the configuration data. After the testing has finished, the coordinator responds to the launcher. Following this, the launcher calls the \textit{kube-apiserver} and extracts the test results from the test runner containers via HTTP, and stores them on the pipeline runner. Pipeline runner is an agent that executes the CI/CD workflow. After the extraction, the test results can be published with the pipeline's default test result publishing functionality such as GitLab artifacts~\cite{gitlabar}. The launcher is designed to run in a container which makes it platform-independent and easily adaptable to other container runtimes.
% The launcher takes a \texttt{YAML} configuration file as input that provides the relevant information that is required by it for connecting to the coordinator (\S\ref{sec:coordinator}), i.e., its IP address and port number. The configuration file also includes information required by the \texthl{controller} for running the integration tests. This includes the names of the test runner images, the namespace to which the test runner pods should be deployed, the name of the test runner pods, and a test timeout value.
% \begin{figure}[t]
% \centering
% \begin{minted}[frame=single, framesep=2mm, fontsize=\tiny, linenos, breaklines, obeytabs=true,tabsize=4, numbersep=-10pt]{yaml}
% launcherConfig:
% coordinatorIp: # IP or URL of the Ingress
% coordinatorPort: # Port on the Ingress
% certFile: # Name of the CAs .pem certificate file
% coordinatorConfig:
% testRunnerPods:
% - podName: # Name of a test runner pod
% namespace: # Namespace the test runner pod is deployed in
% testTimeout: # Timeout for the test runner pod
% containers:
% - containerName: # Name of a test runner container
% image: # Name of a test runner container image
% startupCommands: # Startup commands for the test runner container image. Replaces the images entrypoint.
% startupCommandsArgs: # Arguments for the startup commands
% testResultPath: # directory inside the test runner container image, where the test results are stored
% \end{minted}
% \caption{Example configuration file taken as input by the \emph{Bunk8s} launcher.}
% \label{lst:config_file}
% \end{figure}
\lstset{ %
basicstyle=\ttfamily\footnotesize,
commentstyle=\color{blue}\ttfamily,
frame=single,
language=Bash,
showstringspaces=false,
numbers=left,
xleftmargin=2.6em,
framexleftmargin=2.7em,
morekeywords={blue},
morestring=[s][\color{Gray}]{<}{>}
}
\begin{lstlisting}[caption={
Example configuration file taken as input by the \emph{Bunk8s} launcher.
}, float, floatplacement=t, captionpos=b, basicstyle=\ttfamily\tiny, belowskip=-1.5 \baselineskip, frame=single, numbers=left,breaklines,language=Bash, xleftmargin=2.6em,
framexleftmargin=2.7em, label={lst:config_file}]
launcherConfig:
coordinatorIp: # IP or URL of the Ingress
coordinatorPort: # Port on the Ingress
certFile: # Name of the CAs .pem certificate file
coordinatorConfig:
testRunnerPods:
- podName: # Name of a test runner pod
namespace: # Namespace the test runner pod is deployed in
testTimeout: # Timeout for the test runner pod
containers:
- containerName: # Name of a test runner container
image: # Name of a test runner container image
startupCommands: # Startup commands for the test runner container image. Replaces the images entrypoint.
startupCommandsArgs: # Arguments for the startup commands.
testResultPath: # Directory inside the test runner container image, where the test results are stored.
\end{lstlisting}
% After the testing is finished and the coordinator responds to the launcher, the launcher calls the \textit{kube-apiserver} and extracts the test results from the test runner containers via HTTP and stores them on the pipeline runner.
\subsubsection{The Coordinator}
\label{sec:coordinator}
The \emph{Bunk8s} Coordinator handles the deployment of the test runner containers and watches their state until they have finished. It consists of two sub-components. First, an RPC server for communication and synchronization between the launcher and the coordinator. Second, a k8s clientset that connects to the \textit{kube-apiserver} via HTTP, sends a request to create the test runner pods, and watches their state. The coordinator's RPC server is exposed to the launcher via an Ingress. On test completion, the coordinator sends a response to the Launcher, containing the information that is required to extract the test results from the test runner containers. The test runner container image that is deployed in a pod by the \emph{Bunk8s} coordinator must be built and pushed to a container registry from which k8s can pull container images. Since the integration tests are executed inside a container that is separate from the other components of \emph{Bunk8s}, the developers can use any test framework or programming language to write their tests.
% An RPC server, which allows the synchronization of the test extraction by the launcher, and a Kubernetes clientset, that connects to the \textit{kube-apiserver} via HTTP and sends a request to create the test runner pods and watches the test runner pod's state. The coordinator's RPC server is exposed to the launcher via an Ingress.
% After the test runner pods finish, the coordinator sends a response to the Launcher, containing the data that is required to extract the test results from the test runner containers.
% \subsubsection{The Test Runner}
% % The test runner container image that is deployed in a pod by the coordinator must be built and pushed to a container registry, from whichKubernetes can pull container images. Since the tests are run in a container that is separated from the other components of Bunk8s, it isnot required to use a specific test framework or programming language to write tests.
% The test runner container image that is deployed in a pod by the \emph{Bunk8s} coordinator must be built and pushed to a container registry from which k8s can pull container images. Since the integration tests are executed inside a container that is separate from the other components of \emph{Bunk8s}, the developers can use any test framework or programming language to write their tests.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{figures/communication_diagram_6.png}
\caption[Bunk8s components]{Workflow of Integration testing with \emph{Bunk8s}.}
\label{fig:bunk8s_workflow}
\end{figure}
\subsection{Integration testing with Bunk8s}
\label{sec:testingworkflow}
To install \emph{Bunk8s} in a k8s cluster, the launcher and coordinator (\S\ref{sec:sysdesign}) images must be built using \texttt{Dockerfiles}. The launcher image must be uploaded to a container registry from where it can be pulled by the pipeline runners, while the coordinator image must be uploaded to a registry to which the k8s cluster has access. For a quick setup of the \emph{Bunk8s} coordinator, we provide a pre-configured Helm chart in our repository (\S\ref{sec:intro}). Note that, the coordinator can be deployed in any k8s namespace without affecting its functionality. However, it is recommended that the coordinator is deployed in a namespace of its own. The project admin is responsible for creating the role-based access control rules (RBAC) for the coordinator pod in its namespace. The detailed workflow of the testing process with \emph{Bunk8s} is shown in Figure~\ref{fig:bunk8s_workflow}.
At the beginning \circled{1}, the \emph{Bunk8s} launcher container is started by using a launch script. The launcher takes a \texttt{YAML} configuration file as input (\S\ref{sec:sysdesign}) which is mounted into the container at startup. The \texttt{YAML} parser parses the configuration file, checks if it is valid \circled{2}, and creates a \texttt{Go} struct from the configuration data \circled{3}. Following this, the \texttt{gRPC} client is created that receives the test run configuration created in the previous step \circled{4}. The \texttt{gRPC} client calls the \texttt{DeployTestRunner} RPC method registered in the \texttt{gRPC} server that invokes several functions defined in the kubeclient package \circled{5}. For brevity, those function names have been omitted from Figure~\ref{fig:bunk8s_workflow}. As part of the RPC method in \circled{5}, the kubeclient does several HTTP requests to the kube-apiserver. First, it does a \texttt{GET} request for each namespace provided in the configuration data \circled{6}. After it receives a response, it checks if the given namespaces already exist or not. If not, the test run is terminated and the coordinator sends an error code as a \texttt{gRPC} reply to the launcher. Second, it does a \texttt{GET} request for each pod from the configuration data \circled{7}. On receipt of the response, it checks whether the pods already exist within the namespace that they should be scheduled in. If the pods do not exist, the kubeclient creates k8s API pod objects. Third, it does a \texttt{POST} request for each pod object created in the previous step \circled{8}. After this, the test runner pods are created by the kube-apiserver \circled{9}. Finally, the kubeclient creates a watcher for each test runner pod. After all the tests have finished, the \texttt{DeployTestRunner} RPC method ends \circled{10}. Following this, the \texttt{gRPC} server sends the server response to the \texttt{gRPC} client which is written to the logs of the \emph{Bunk8s} launcher container \circled{11}. The server reply is stored as a JSON file by the launch script (\circled{12}) and then read by the data extraction script (\circled{13}). After this, the data extraction script executes kubectl which does an HTTP \texttt{POST} request to the kube-apiserver to copy the test results from the test runner pods to the pipeline runner \circled{14}. The test results are extracted by the kube-apiserver from the sidecar containers inside the test runner pods \circled{15}. Following this, the test results are stored on the pipeline runner \circled{16}. Finally, the data extraction script does an HTTP \texttt{DELETE} request to the kube-apiserver (\circled{17}) to delete the test runner pods (\circled{18}). The results of the integration tests can now be published using the CI pipeline's publishing mechanisms.
% test runner pods are deleted by the kube-apiserver
%After all tests have finished
\section{Case Study: Room Tracking System}
\label{sec:case_study}
% In this section, the successful implementation of Bunk8s will be demonstrated by
% running broad integration tests for a microservice example application. This serves as
% an example of how to use Bunk8s and shows that the requirements defined in chapter 4
% are met.
In this section, we demonstrate the usage of \emph{Bunk8s} for the broad integration testing of a real-world microservice application.
\subsection{Experimental Setup}
\label{sec:env}
% For this demonstration, a remote test environment is available. The environment includes GitLab and a Kubernetes namespace in an Azure Kubernetes Service (AKS), as well as an Azure Container Registry. GitLab is a DevOps platform, that provides remote Git repositories, as well as CI/CD pipelines \cite{ch5-gi-whatisgitlab}. The example application, as well as Bunk8s, are located in a GitLab Git repository and are deployed to the AKS with GitLab's built-in pipeline functionality.
To demonstrate the functioning of \emph{Bunk8s}, we create a Kubernetes cluster using the Azure Kubernetes Service (AKS)~\cite{aks}. We use GitLab as the DevOps platform since it provides remote Git repositories and CI/CD pipelines~\cite{ch5-gi-whatisgitlab}. As the registry for containers, we use the Azure Container Registry~\cite{acr}. The chosen application and \emph{Bunk8s} are deployed to the AKS by utilizing the pipeline functionality present in GitLab. Although we use AKS for our experiments, \emph{Bunk8s} is compatible with commercial k8s offerings of all cloud providers.
\subsection{Application Design}
\label{sec:design}
\begin{figure}[t]
\centering
\includegraphics[width=0.75\columnwidth]{figures/example_application_component.png}
\caption[Room Tracking System Components]{Room Tracking Application Microservices.}
\label{fig:example_components}
\end{figure}
% The example application is a room tracking system, that is used to train developers on the architecture of microservices. A user can hold a card up to a badge reader inside a room and that room will be marked as occupied by the application. Users can also access a dashboard via a browser, that gives an overview of the room utilization. Furthermore, when a user enters a room, it is announced on a shout website, separate from the dashboard.
The application we use is a room tracking system. A user can hold a card up to a badge reader inside a room and that room will be marked as occupied by the application. Users can also access a dashboard via a browser that provides an overview of the room utilization. Furthermore, when a user enters a room, it is announced on a shout website that is separate from the dashboard.
Figure~\ref{fig:example_components} gives an overview of the different microservices present in the room tracking system. It consists of two frontend services, i.e., \textit{Dashboard} and \textit{Shout}, three backend services, i.e., \textit{Device}, \textit{Account}, and \textit{Room}, and two infrastructure services, i.e., \textit{RabbitMQ} and \textit{MongoDB}. The frontend services \textit{Dashboard} and \textit{Shout} provide the dashboard and shout websites. \textit{RabbitMQ} provides message queues for the communication between the \textit{Device}, \textit{Dashboard}, \textit{Shout} and the \textit{Room} service. When a user enters a room, a badge reader device sends a badge reader event to the \textit{Device} service. Following this, the \textit{Device} service creates a device event containing the card and room numbers. The device event is then sent to the \textit{Room} service via the message queue. The \textit{Room} service updates the room allocation in \textit{MongoDB} and creates a room event that describes which user entered or left the room and forwards it to the \textit{Dashboard} and \textit{Shout} services via the two different message queues. After receiving the room event, the \textit{Dashboard} service updates a counter that tracks the number of users that have checked into a room. On the other hand, the \textit{Shout} service requests the name of the user from the room event from the \textit{Account} service. The \textit{Account} service provides the functionality for creating, modifying, and listing the user accounts that are stored in the \textit{MongoDB} service. Note that, \texttt{AMQP}~\cite{standard2012oasis} is a message protocol for transferring messages, while the wire protocol is the default communication protocol used by MongoDB~\cite{wire}.
%What is wire protocol
%Mention AMQP
%Wire protocol -? used by MongoDB
% The frontend services \textit{Dashboard} and \textit{Shout} provide the dashboard and shout websites. \textit{RabbitMQ} and \textit{MongoDB} are infrastructure services. \textit{RabbitMQ} provides message queues for the communication between the \textit{Device}, \textit{Dashboard}, \textit{Shout} and the \textit{Room} service.
% Each component corresponds to a single microservice, that can be deployed to Kubernetes. All logging functionality is omitted from the application.
% The frontend services Dashboard and Shout provide the previously mentioned dashboard and shout websites. The services RabbitMQ and MongoDB are infrastructure services. RabbitMQ provides message queues for the communication between Device and Room service, as well as between the Room service and the Dashboard and Shout service. MongoDB provides a MongoDB database. Device, Room, and Account are backend services.
% When a user enters a room, a badge reader device sends a badge reader event to the Device service. It then creates a device event, containing the card and room numbers.
% This device event is then sent to the Room service via a message queue provided by the RabbitMQ service. The Room service receives the device event and updates the room allocation in MongoDB. It also creates a room event, which describes which user entered or left a room and sends it to the Dashboard and Shout services via two different message queues. Afterwards, it updates the room allocation in MongoDB. When the Dashboard service receives the room event, it updates a counter of how many users are checked into a room. After the Shout service receives the room event, it requests the name of the user from the room event from the Account service. The Account service allows to create, modify and list user accounts, which are stored in the MongoDB service.
\subsection{Broad Integration Testing Results}
% Broad integration tests cover multiple code paths through several microservices and all of their components (\S\ref{sec:integeration_testing}). As a result, multiple services of the room tracking application will be under test (\S\ref{sec:design}). Since \emph{Bunk8s} \hl{enables testing of backend subsystems without requiring the reconfiguration of ingresses in k8s, as well as the testing of backend services that do not provide an HTTP or HTTPS interface, the services \textit{Dashboard}, \textit{Shout}, and \textit{Account} are not tested and omitted from deployment}. As a result, the microservices under test are \textit{Device}, \textit{Room}, \textit{RabbitMQ}, and \textit{MongoDB}.
Broad integration tests cover multiple code paths through several microservices and all of their components (\S\ref{sec:integeration_testing}). As a result, multiple services of the room tracking application will be under test (\S\ref{sec:design}). Since \emph{Bunk8s} enables testing of backend services without requiring the reconfiguration of ingresses in k8s, as well as the testing of backend services that do not provide an HTTP or HTTPS interface, the services \textit{Dashboard}, \textit{Shout}, and \textit{Account} are not tested and omitted from deployment. To this end, the microservices under test are \textit{Device}, \textit{Room}, \textit{RabbitMQ}, and \textit{MongoDB}.
% \begin{figure}[t]
% \centering
% \begin{minted}[frame=single, framesep=2mm, fontsize=\tiny,linenos,breaklines, obeytabs=true,tabsize=4, numbersep=-10pt]{text}
% === RUN TestDeviceServiceReceiveDeviceEvent
% --- PASS: TestDeviceServiceReceiveDeviceEvent (0.27s)
% === RUN TestRoomServiceRoomEventResponse
% === RUN TestRoomServiceRoomEventResponse/Queue_Dashboard
% === RUN TestRoomServiceRoomEventResponse/Queue_Shout
% --- PASS: TestRoomServiceRoomEventResponse (0.07s)
% --- PASS: TestRoomServiceRoomEventResponse/Queue_Dashboard (0.00s)
% --- PASS: TestRoomServiceRoomEventResponse/Queue_Shout (0.00s)
% === RUN TestMongoDBRoomAllocationEntry
% --- PASS: TestMongoDBRoomAllocationEntry (0.09s)
% PASS
% ok showcase 0.435s
% \end{minted}
% \caption{Test result logs after successful completion of the broad integration tests with \emph{Bunk8s}.}
% \label{lst:succresult}
% \end{figure}
\begin{lstlisting}[caption={
Test result logs after successful completion of the broad integration tests with \emph{Bunk8s}.
}, float, floatplacement=t, captionpos=b, basicstyle=\ttfamily\tiny, belowskip=-1.5 \baselineskip, frame=single, numbers=left, xleftmargin=2.6em,
framexleftmargin=2.7em, label={lst:succresult}]
% === RUN TestDeviceServiceReceiveDeviceEvent
% --- PASS: TestDeviceServiceReceiveDeviceEvent (0.27s)
% === RUN TestRoomServiceRoomEventResponse
% === RUN TestRoomServiceRoomEventResponse/Queue_Dashboard
% === RUN TestRoomServiceRoomEventResponse/Queue_Shout
% --- PASS: TestRoomServiceRoomEventResponse (0.07s)
% --- PASS: TestRoomServiceRoomEventResponse/Queue_Dashboard (0.00s)
% --- PASS: TestRoomServiceRoomEventResponse/Queue_Shout (0.00s)
% === RUN TestMongoDBRoomAllocationEntry
% --- PASS: TestMongoDBRoomAllocationEntry (0.09s)
% PASS
% ok showcase 0.435s
\end{lstlisting}
% \begin{figure}[t]
% \centering
% \begin{minted}[frame=single, framesep=2mm, fontsize=\tiny,linenos,breaklines, obeytabs=true,tabsize=4, numbersep=-10pt]{text}
% [...]
% === RUN TestMongoDBRoomAllocationEntry
% main_test.go:152: Failed to create MongoDB client: Failed to ping MongoDB: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongodb://mongo.bunk8s-fe.svc.cluster.local, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp mongodb://mongo.bunk8s-fe.svc.cluster.local: connect: connection refused }, ] }
% --- FAIL: TestMongoDBRoomAllocationEntry (10.00s)
% FAIL
% ok showcase 10.101s
% \end{minted}
% \caption{Test result logs if a broad integration test failed with \emph{Bunk8s}.}
% \label{lst:notsuccresult}
% \end{figure}
\begin{lstlisting}[caption={
Test result logs after successful completion of the broad integration tests with \emph{Bunk8s}.
}, float, floatplacement=t, captionpos=b, basicstyle=\ttfamily\tiny, belowskip=-1.5 \baselineskip, frame=single, numbers=left,breaklines,tabsize=4, xleftmargin=2.6em,
framexleftmargin=2.7em, label={lst:notsuccresult}]
[...]
=== RUN TestMongoDBRoomAllocationEntry
main_test.go:152: Failed to create MongoDB client: Failed to ping MongoDB: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongodb://mongo.bunk8s-fe.svc.cluster.local, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp mongodb://mongo.bunk8s-fe.svc.cluster.local: connect: connection refused }, ] }
--- FAIL: TestMongoDBRoomAllocationEntry (10.00s)
FAIL
ok showcase 10.101s
\end{lstlisting}
%More explanation of the integration tests
For writing the integration tests, we use Go and its default testing package. We generate a badge reader event which is sent to the \textit{Device} service. Following this, the contents of the room events created by the \textit{Room} service are checked for validity. Finally, we check the validity of the room allocated in the \textit{MongoDB} database. As described in \S\ref{sec:testingworkflow}, the test results are extracted and stored in artifact store provided by GitLab. Listing~\ref{lst:succresult} shows the logs of the test results after all broad integration tests have completed successfully. \emph{Bunk8s} also extracts test results if a test fails. For instance, Listing~\ref{lst:notsuccresult} shows logs of the test \texttt{TestMongoDBRoomAllocationEntry} when no connection to the \textit{MongoDB} service could be established. As a result, the test result publishing supported by \emph{Bunk8s} allows developers to easily analyse and debug their applications.
% caption={Test results after successful Broad Integration tests with \emph{Bunk8s}},label={lst:succresult}, captionpos=b
% Since we use AKS (\S\ref{sec:env}) for our k8s cluster, we replace the \textit{MongoDB} and \textit{RabbitMQ} services by the Azure Cosmos DB and Azure Service Bus respectively. Both of these services run outside of the k8s cluster on the Azure cloud platform.
\label{sec:results}
\section{Why Bunk8s?}
\label{sec:whybunk8s}
\begin{table}[t]
\centering
\begin{adjustbox}{width=9cm,center}
\begin{tabu}{ |c|c|c|c|c| }
\hline
\textbf{Supported Features} & \textbf{Bunk8s} \textbf{[This Work]} & \textbf{Octopus}~\cite{octopus} & \textbf{Istio}~\cite{istio} & \textbf{Jenkins}~\cite{jenkins} \\
\tabucline{-}
Broad integration testing & \faThumbsOUp & \faThumbsOUp & \faThumbsDown & \faThumbsDown\\
\tabucline{-}
Narrow integration testing & \faThumbsDown & \faThumbsDown & \faThumbsOUp & \faThumbsOUp\\
\tabucline{-}
Test framework independent & \faThumbsOUp & \faThumbsOUp & \faThumbsDown & \faThumbsOUp \\
\tabucline{-}
CI/CD Infrastructure independent & \faThumbsOUp & \faThumbsOUp & \faThumbsDown & \faThumbsDown\\
\tabucline{-}
Test result publishing & \faThumbsOUp & \faThumbsDown & \faThumbsOUp & \faThumbsOUp\\
\tabucline{-}
\end{tabu}
\end{adjustbox}
\caption{Comparison of the different features supported by \emph{Bunk8s} with the other available tools for integration testing. \faThumbsOUp Supported. \faThumbsDown No support. }
\label{table:comparison}
\end{table}
To support the integration testing of microservices inside k8s several tools exist. These include Octopus~\cite{octopus}, Istio~\cite{istio}, and Jenkins~\cite{jenkins}. Octopus allows developers to define integration tests with custom resource definitions~\cite{crd}. For each test, Octopus deploys a pod to k8s. Furthermore, it supports re-running tests to prevent flaky test results. The Istio testing framework allows developers to test microservice applications that use the Istio service mesh to handle the communication between microservices. Developers can also use Jenkins to run integration tests inside k8s by using the different Kubernetes plugins~\cite{jenkins-itest}. A comparison of the different features supported by \emph{Bunk8s} with the other tools is shown in Table~\ref{table:comparison}.
Both Istio and Jenkins only support narrow integration testing (\S\ref{sec:integeration_testing}). For Istio, this is possible due to the presence of sidecar containers for pods. On the other hand, both Octopus and \emph{Bunk8s} support broad integration testing which enable the testing of interactions between the application microservices and the backend services. For all the tools shown in Table~\ref{table:comparison}, the developers can use any programming language and test framework for writing integration tests except Istio. Since Istio is an expansion of the Go testing package it only supports writing integration tests in Go. Moreover, Istio is dependent on the usage of Prow~\cite{prow} as the CI/CD system for running the integration tests~\cite{istiotest}.
Similarly, using Jenkins would require it's introduction into a project and migration of at least parts of the CI/CD pipeline to Jenkins if not already in use. In contrast, both \emph{Bunk8s} and Octopus can be used with any CI/CD tool that supports Docker containers. A major drawback of using Octopus is that after a test is finished it only reports whether a test failed or was successful and does not extract the test logs from the containers. This prevents an in-depth analysis of test failures~\cite{kyma-octopus}. Moreover, Octopus is not open-source and charges for its services. On the other hand, \emph{Bunk8s} is completely open-source and supports publishing of test results since they are extracted from the test runner pod and are stored on the pipeline runner (\S\ref{sec:sysdesign}). Following this, the supported pipeline mechanisms for test result publishing can be utilized. As a result, the different features supported by \emph{Bunk8s} make it easier for it to be integrated into an existing projects pipeline.
% All tools shown in
%improve reasoning for broad integeration testing
% Octopus allows developers to define integration tests with custom resource definitions~\cite{crd}. For each test, Octopus deploys a pod to k8s. Furthermore, it supports re-running tests to prevent flaky test results. However, after a test is finished Octopus only reports whether a test failed or was successful and does not extract the test logs from the containers. This prevents an in-depth analysis of test failures~\cite{kyma-octopus}.
% The Istio testing framework allows developers to test microservice applications that use the Istio service mesh to handle the communication between microservices. The framework is an expansion of the Go testing package and only supports writing integration tests in Go. Moreover, it requires Prow as the CI/CD tool and Github to execute the tests automatically~\cite{istiotest}. With Jenkins the developers can also run integration tests inside of k8s. However, it only supports narrow integration testing, i.e., the test runner containers can only be used to test the interaction of containers within pods. Moreover, using Jenkins would require it's introduction into a project and migration of at least parts of the CI/CD pipeline to Jenkins if not already in use.
% for using Jenkins it would have to be introduced into the
\section{Conclusion \& Future Work}
\label{sec:futurework}
% Conclusion
% In the scope of this paper, Bunk8s was developed. It is a testing tool for broad integration testing of microservices deployed in Kubernetes. It is independent of the test framework that is used to write or generate tests, independent of the infrastructure or platform on which Kubernetes runs and supports test result publishing.
In this paper, we implemented and presented \emph{Bunk8s}, a tool for broad integration testing of microservices in Kubernetes. \emph{Bunk8s} is independent of the test framework used for writing or generating tests, independent of the infrastructure or platform on which Kubernetes runs, and supports test result publishing. \emph{Bunk8s} is well suited for developers planning to write a new test suite as well as for developers who wish to revise and improve the already existing test suite of a microservice application. In the future, we plan to extend \emph{Bunk8s} to support mocking of services. Furthermore, we plan to add support for narrow integration testing in \emph{Bunk8s} to enable testing of multi-container pods. Support for FaaS based CI/CD workloads is another future direction~\cite{jindal2021function}.
% Bunk8s consists of a launcher, which must be run in a pipeline, and a coordinator, which must be deployed to Kubernetes. The launcher reads a configuration file for a test run and sends the configuration data to the coordinator, which then starts a test runner pod, watches its state, and sends a reply to the launcher after the test runner pod has finished. The launcher then extracts the test results from the test runner pod, stores them on a pipeline runner, and shuts it down. This process enables easy broad integration testing of microservices, as shown in the case study.
% Bunk8s is recommended for developers who start creating a new test suite, as well as for developers who need to revise and improve already existing test suites of applications. If the test suite consists of too many end-to-end tests and only few broad integration tests, Bunk8s allows migrating the end-to-end tests downwards. Furthermore, if integration tests were previously run locally with locally started services, Bunk8s enables shipping the integration testing process to the pipeline with Kubernetes. This may result in reducing the time that is required to run integration tests and allows running tests in a test environment that has a higher parity to the production environment of the application.
% %Future Work
% There are multiple features of Bunk8s that can be added in the future. Since Bunk8s allows to test subsystems of a microservice application, it may be necessary to add mocking of services to the feature set of Bunk8s.
% However, this would require further research, if already existing solutions for mocking are sufficiently compatible with the testing process with Bunk8s or if a functionality for mocking would have to be added to Bunk8s itself.
% Since pods provide "highly-coupled container-to-container communication" features \cite{ch4-tka-clusternetworking}, as well as shared storage for containers in the form of volumes \cite{ch2-tka-pods}, support for narrow integration testing could be added in order to enable testing multi-container pods. This would probably require expanding multiple components of Bunk8s. On the part of the coordinator, a new remote procedure method would have to be added to the gRPC server. Furthermore, the structure of the configuration file, as well as Bunk8s' data model might have to be adapted. In order to find a suited approach on how to implement this feature for Bunk8s, further research must be conducted, which is out of the scope of this paper.
% It is highly likely that further adjustments to Bunk8s are necessary based on feedback from developers, after they integrated Bunk8s into their project. This includes improving the installation process, improving usability for developers, e.g. reworking the configuration file, and adding new requested features. In this regard, another step towards simplifying the installation process for developers is to add container images for other container runtimes.
% Based on this paper, a performance evaluation of Bunk8s can be conducted, which should measure several important metrics, e.g. test throughput. The insights generated from the performance evaluation can be used to further improve Bunk8s.
% In addition to improving Bunk8s, it is important to keep the tool up-to-date. Since Kubernetes is under continuous development, it is important to be aware of any changes to the API. Furthermore, the gRPC framework and other libraries used must be continuously updated to their current versions, as there may be security vulnerabilities or compatibility issues.
% \section{Acknowledgement}
% This work was supported by the funding of the German Federal Ministry of Education and Research (BMBF) in the scope of the Software Campus program.
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{fowlermic}
\BIBentryALTinterwordspacing
M.~Fowler and J.~Lewis. (25.05.2014) Microservices. [Online]. Available:
\url{https://martinfowler.com/articles/microservices.html}
\BIBentrySTDinterwordspacing
\bibitem{clemson}
\BIBentryALTinterwordspacing
T.~Clemson. (2014) Testing strategies in a microservice architecture. [Online].
Available: \url{https://martinfowler.com/articles/microservice-testing/}
\BIBentrySTDinterwordspacing
\bibitem{cloud_container}
{14 Tech Companies Embracing Container Technology.},
\url{https://learn.g2.com/container-technology}, accessed 09/29/2021.
\bibitem{kubesphere}
A.~Beltre, P.~Saha, and M.~Govindaraju, ``Kubesphere: An approach to
multi-tenant fair scheduling for kubernetes clusters,'' in \emph{2019 IEEE
Cloud Summit}, 2019, pp. 14--20.
\bibitem{datadogreport}
{DataDog Container Report},
\url{https://www.datadoghq.com/container-report-2020/#2}, accessed on
11/11/2021.
\bibitem{kuber}
{Kubernetes.}, \url{https://kubernetes.io/}, accessed 09/29/2021.
\bibitem{sysdig}
\BIBentryALTinterwordspacing
Sysdig. (2021) Sysdig 2021 container security and usage report. [Online].
Available:
\url{https://sysdig.com/wp-content/uploads/2021-container-security-and-usage-report.pdf}
\BIBentrySTDinterwordspacing
\bibitem{aks}
{Azure Kubernetes Service.},
\url{https://azure.microsoft.com/en-us/services/kubernetes-service/},
accessed 09/29/2021.
\bibitem{fowlerpyr}
\BIBentryALTinterwordspacing
M.~Fowler. (1.05.2012) Testpyramid. [Online]. Available:
\url{https://www.martinfowler.com/bliki/TestPyramid.html}
\BIBentrySTDinterwordspacing
\bibitem{octopus}
{Octopus}, \url{https://github.com/kyma-incubator/octopus}, accessed on
11/11/2021.
\bibitem{istio}
{Istio}, \url{https://istio.io/}, accessed on 11/11/2021.
\bibitem{jenkins}
{Jenkins}, \url{https://www.jenkins.io/}, accessed on 11/11/2021.
\bibitem{fowlerinttest}
\BIBentryALTinterwordspacing
M.~Fowler. (16.01.2018) Integrationtest. [Online]. Available:
\url{https://martinfowler.com/bliki/IntegrationTest.html}
\BIBentrySTDinterwordspacing
\bibitem{containerd}
{Containerd.}, \url{https://containerd.io/}, accessed 09/29/2021.
\bibitem{crio}
{CRI-O-OCI-based implementation of Kubernetes Container Runtime Interface.},
\url{https://github.com/cri-o/cri-o}, accessed 09/29/2021.
\bibitem{gke}
{Google Kubernetes Engine.}, \url{https://tinyurl.com/5as5db77}, accessed
09/29/2021.
\bibitem{helm}
{Helm}, \url{https://helm.sh/}, accessed on 11/11/2021.
\bibitem{terraform}
{Terraform}, \url{https://www.terraform.io/}, accessed on 11/11/2021.
\bibitem{gitlabar}
{GitLab Artifacts},
\url{https://docs.gitlab.com/ee/ci/pipelines/pipeline_artifacts.html},
accessed on 11/11/2021.
\bibitem{ch5-gi-whatisgitlab}
\BIBentryALTinterwordspacing
\relax{GitLab Inc.} (2021) What is gitlab? [Online]. Available:
\url{https://about.gitlab.com/what-is-gitlab/}
\BIBentrySTDinterwordspacing
\bibitem{acr}
{Azure Container Registry},
\url{https://azure.microsoft.com/en-us/services/container-registry/},
accessed on 11/11/2021.
\bibitem{standard2012oasis}
O.~Standard, ``Oasis advanced message queuing protocol (amqp) version 1.0,''
\emph{International Journal of Aerospace Engineering Hindawi www. hindawi.
com}, vol. 2018, 2012.
\bibitem{wire}
{MongoDB wire protocol},
\url{https://docs.mongodb.com/manual/reference/mongodb-wire-protocol/},
accessed on 11/11/2021.
\bibitem{crd}
{Custom Resource Definition in Kubernetes}, \url{https://tinyurl.com/puxwmfu4},
accessed on 11/11/2021.
\bibitem{jenkins-itest}
\BIBentryALTinterwordspacing
B.~Szeti. (29.06.2018) Running integration tests in kubernetes. [Online].
Available:
\url{https://opensource.com/article/18/6/running-integration-tests-kubernetes}
\BIBentrySTDinterwordspacing
\bibitem{prow}
{Prow}, \url{https://github.com/kubernetes/test-infra/tree/master/prow},
accessed on 11/11/2021.
\bibitem{istiotest}
\BIBentryALTinterwordspacing
S.~Landow. Istio test framework. [Online]. Available:
\url{https://github.com/istio/istio/wiki/Istio-Test-Framework}
\BIBentrySTDinterwordspacing
\bibitem{kyma-octopus}
\BIBentryALTinterwordspacing
A.~Szecowka and K.~Zydek. How we approached integration testing in kubernetes,
and why we stopped using helm tests. [Online]. Available:
\url{https://kyma-project.io/blog/2020/1/16/integration-testing-in-k8ss}
\BIBentrySTDinterwordspacing
\bibitem{jindal2021function}
\BIBentryALTinterwordspacing
A.~Jindal, M.~Gerndt, M.~Chadha, V.~Podolskiy, and P.~Chen, ``Function delivery
network: Extending serverless computing for heterogeneous platforms,''
\emph{Software: Practice and Experience}, vol.~51, no.~9, pp. 1936--1963,
2021. [Online]. Available:
\url{https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.2966}
\BIBentrySTDinterwordspacing
\end{thebibliography}