A Reproducibility Study of "IP Spoofing Detection in Inter-Domain Traffic"
Extracted source text · arXiv 1911.05164
%\documentclass[9pt,sigconf,letterpaper,balance,nonacm]{acmart}
\documentclass[sigconf]{acmart}
% Disable / remove copyright boxes
\setcopyright{none}
\settopmatter{printacmref=false}
\renewcommand\footnotetextcopyrightpermission[1]{}
% Increase margin between text and footer
\setlength{\footskip}{20pt}
% Add CCR footer
\usepackage{fancyhdr}
%\fancypagestyle{plain}{%
%\fancyhf{} %
%\fancyfoot[L]{ACM SIGCOMM Computer Communication Review}%
%\fancyfoot[R]{Volume TBA Issue TBA, January TBA}%
%}
\pagestyle{plain}
% Add CCR footer on first page
%\fancypagestyle{firstpagestyle}{%
%\fancyhf{} %
%\fancyfoot[L]{ACM SIGCOMM Computer Communication Review}%
%\fancyfoot[R]{Volume TBA Issue TBA, January TBA}%
%}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\input{commands}
\usepackage{booktabs}
\usepackage{balance}
\usepackage{url}
\usepackage{graphicx}
%\usepackage{subfigure}
\usepackage{caption}
%\captionsetup[subfigure]{position=b}
\usepackage[position=b]{subcaption}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{float}
\usepackage{makecell}
\usepackage{color, colortbl}
\definecolor{Gray}{gray}{0.9}
\usepackage{pgfplots}
\usetikzlibrary{positioning}
\usepackage{tikz}
\usepackage{tikzpeople}
\usetikzlibrary{arrows, backgrounds, fit, positioning, quotes, shapes, patterns, hobby}
\usepackage{standalone}
\usepackage{geometry}
\graphicspath{{figs/}}
\begin{document}
% Only using the short title field to remove the line break.
\title[A Reproducibility Study of ``IP Spoofing Detection in Inter-Domain Traffic'']{A Reproducibility Study of \\\ ``IP Spoofing Detection in Inter-Domain Traffic''}
\author {Jasper Eumann}
\affiliation{%
\institution{HAW Hamburg}
}
\email{jasper.eumann@haw-hamburg.de}
\author {Raphael Hiesgen}
\affiliation{%
\institution{HAW Hamburg}
}
\email{raphael.hiesgen@haw-hamburg.de}
\author {Thomas C. Schmidt}
\affiliation{%
\institution{HAW Hamburg}
}
\email{t.schmidt@haw-hamburg.de}
\author {Matthias W\"ahlisch}
\affiliation{%
\institution{Freie Universit\"at Berlin}
}
\email{m.waehlisch@fu-berlin.de}
\input{abstract}
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10003033.10003039.10003045.10003046</concept_id>
<concept_desc>Networks~Routing protocols</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10003033.10003079.10011704</concept_id>
<concept_desc>Networks~Network measurement</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10003033.10003083.10003014.10003015</concept_id>
<concept_desc>Networks~Security protocols</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10003033.10003106.10010924</concept_id>
<concept_desc>Networks~Public Internet</concept_desc>
<concept_significance>300</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[500]{Networks~Routing protocols}
\ccsdesc[500]{Networks~Network measurement}
\ccsdesc[500]{Networks~Security protocols}
\ccsdesc[300]{Networks~Public Internet}
\keywords{Spoofing, inter-domain routing, IXP, DDOS}
\maketitle
\input{intro}
\input{methodology}
\input{implementation}
\input{results}
\input{conclusion}
\input{acks}
\balance
\bibliographystyle{ACM-Reference-Format}
\bibliography{rfcs,security,internet,own,tmp}
\end{document}
\newcommand{\todo}[1]{\textcolor{red}{\textbf{TODO:} #1}}
\newcommand{\mw}[1]{\textcolor{teal}{\textbf{Matthias:} #1}}
\renewcommand{\comment}[1]{{\color{magenta}#1}}
\usepackage{pifont}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{56}}%
\newcommand{\pmark}{\ding{218}}%
\usepackage{xspace}
\newcommand{\eg}{\textit{e.g.,}~}
\newcommand{\ie}{\textit{i.e.,}~}
\newcommand{\cf}{\textit{cf.,}~}
\newcommand{\etc}{\textit{etc.}~}
\newcommand{\one}{({\em i})\xspace}
\newcommand{\two}{({\em ii})\xspace}
\newcommand{\three}{({\em iii})\xspace}
\newcommand{\four}{({\em iv})\xspace}
\newcommand{\five}{({\em v})\xspace}
\makeatletter
\renewcommand{\paragraph}[1]{\vspace*{0.03in}\noindent{\bf #1.}\hspace{0.25ex \@plus1ex \@minus.2ex}}
%\startsection{NAME}{LEVEL}{INDENT}{BEFORESKIP}{AFTERSKIP}{STYLE}
%\z@ => shorthand for zero points, no indent
\makeatother
\begin{abstract}
IP spoofing enables reflection and amplification attacks, which cause major threats to the current Internet infrastructure.
Detecting IP packets with incorrect source addresses would help to improve the situation. This is easy at the attacker's network, but very challenging at Internet eXchange Points (IXPs) or in transit networks.
In this reproducibility study, we revisit the paper \textit{Detection, Classification, and Analysis of Inter-Domain Traffic with Spoofed Source IP Addresses} published at ACM IMC~2017~\cite{lskrf-dcait-17}.
Using data from a different IXP and from a different time, we were not able to reproduce the results.
Unfortunately, our analysis shows that the current state of art does introduce a methodology that does not comply with common real-world deployment.
\end{abstract}
\section{Introduction}
\label{sec:intro}
IP spoofing injects packets that include an IP source address, which is not advertised to the routing by its origin network.
Consequently, any reply is directed not to its origin but to a different destination.
In combination with a distributed amplification, in which small requests trigger much larger replies, this leads to serious denial of service attacks in the current Internet~\cite{jkkrs-mtuam-17,rowrs-adads-15}.
The most effective mitigation of reflection attacks~\cite{rowrs-adads-15} is ingress filtering at the network of the attacker~\cite{RFC-2827,RFC-3704}.
This solution, however, is not sufficiently deployed~\cite{ffgmr-manrs-18}.
An alternative solution~\cite{lskrf-dcait-17} proposes a heuristic at central Internet eXchange Points (IXPs) following the observation that a valid packet should flow compliant with control plane information, and hence should reach the IXP via a customer cone that contains its origin. This paper recently published at ACM IMC 2017 claims that a method is presented ``to passively detect packets with spoofed IP addresses [...] and minimize false positive inferences''~\cite[\S~1]{lskrf-dcait-17}. Central to this approach is a reliable inference of customer cones from BGP data, which poses the major challenge.
In this paper, we report on our attempts to reproduce the current state of the art, based on a different team and setup~\cite{bbfkp-dbgre-19}.
At different times, we analyze data from a large regional IXP instead of data from a large European IXP, which should not affect the validity of the method.
Unfortunately, our findings largely differ from those presented in IMC'17, even though we explore various ways of inferring the customer cones. In particular, spoofed traffic classified in our experiments exceeds the values of IMC'17 by orders of magnitude with a traffic mix that strongly indicates a dominant portion of false positives. We identify plausible reasons for these discrepancies from further analyses.
It is worth noting that our insights are independent of the vantage point and time but highlight intrinsic drawbacks of the previous methodology.
Later research~\cite{mlhcb-cisti-19} will confirm our results.
In the remainder, we recap the methodology in Section~\ref{sec:methodology}, introduce our implementation and data sources in Section~\ref{sec:implementation}, present and discuss results in Section~\ref{sec:results}, and conclude in Section~\ref{sec:conclusion}.
\section{Recap of IMC'17 Methodology}
\label{sec:methodology}
The objective of the proposed approach~\cite{lskrf-dcait-17} is to sort \textbf{invalid} (spoofed) traffic from \textbf{regular} (non-spoofed). Before classifying packets into theses two categories the traffic is sanitized by filtering \textbf{bogon} packets, \ie packets with addresses from private networks and other ineligible routable prefixes~\cite{RFC-1918,RFC-5735,RFC-6598}, as well as \textbf{unrouted} packets, \ie packets from sources that do not show any announcements.
For the remaining packets entering the IXP via an IXP member, a check is required whether each packet arrives via a customer cone that covers the prefix of the origin AS. Such a customer cone includes all ASes that receive (indirect) upstream via the IXP member and includes transitive~peering.
Due to the limited visibility of BGP relations and the lack of vantage points, it is a major challenge to correctly infer these cones. Three approaches are proposed in~\cite{lskrf-dcait-17} (names taken from the paper):
\paragraph{(1) Naive Approach} Built from public BGP information, this approach considers a packet valid if it originates from an AS that is part of an announced path for its source prefix. It aims to reflect the topology but falls short in representing business relationships between ASes accurately. Live data provides sufficient information to deploy it.
\begin{figure*}[t]
\begin{subfigure}{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{week_naive_2018-02-19-2018-02-26.pdf}
\caption{Naive Approach}
\label{fig::week_naive}
\end{subfigure}\hfill
\begin{subfigure}{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{week_caida_2018-02-19-2018-02-26.pdf}
\caption{CAIDA Customer Cone}
\label{fig::week_cc}
\end{subfigure}\hfill
\begin{subfigure}{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{week_full_2018-02-19-2018-02-26.pdf}
\caption{Full Cone}
\label{fig::week_full}
\end{subfigure}\hfill
\caption{Time series of classified traffic distributions}
\label{fig::week}
\end{figure*}
\paragraph{(2) CAIDA Customer Cone} In contrast to the naive cone, CAIDA represents the business relationships rather than the topology. It is created by information such as community strings, directly reported relations, and historic information. Further details are available in \textit{AS Relationships, Customer Cones, and Validation} by Luckie et al.~\cite{lhdgc-arccv-13}. % CAIDA publishes a cone monthly
\paragraph{(3) Full Cone} This extended cone is built from the assumption that ASes neighboring in an announcement are tranistively peering. Built from public BGP announcements this approach adds transitive relationships between all peers. Even though this might misinterprete or miss business relationships it results in the largest cone.
\paragraph{\textbf{Multi-AS Organization Extension}:} This add-on can be combined with the CAIDA Customer Cone and the Full Cone. It adds information about sibling ASes by building connections between ASes belonging to the same multi-AS organization~\cite{chkw-tam-10}, thus allowing a bidirectional data exchange between them. \medskip
Using these three cones packets can be classified either as \textbf{invalid} (spoofed) or as \textbf{regular}.
The full cone approach is the main method examined in the IMC'17 paper and the basis for most of its evaluation. In our reproducibility study, we consider the various approaches of cone construction and analyse its different impacts on the packet classification.
\section{Implemention and Data Sources}
\label{sec:implementation}
\paragraph{Software}
This study performed replicability and reproducibility work in two phases. First, we replicated the results by applying scripts kindly provided by the IMC'17 authors~\cite{Lichtblau18} to our data sets. These scripts supported constructing the \textit{full cone} from BGP data, which is taken as the indicator of \textit{invalid} packets. We augmented these scripts with helper tools for
\one reading output from tool \texttt{bgpdump}, and \two detecting \textit{bogon} and \textit{unrouted} packets. This step served as baseline to verify that our data can be processed as expected. All further results in this paper are based on the second phase, the reproduction.
For the reproduction, we re-implemented all methods to construct all cones (\ie Naive, CAIDA, and Full cone) and to detect bogon and unrouted packets.
Based on this reimplementation, we added enhanced features for classifying payloads of spoofed traffic using libpcap\footnote{\url{https://www.tcpdump.org/}}.
While carefully confirming consistency with the original scripts, our extended toolset allowed for a more accurate analysis of the classification, as discussed later in more detail.
The software is available at\\ \url{https://github.com/inetrg/reproducibility-study-ixp-spoofing}.
%We extended the implementation with missing functionality, including tools to build the cones for the Naive Cone and CAIDA Cone.
%For extended measurements and further analyses in this paper, we with enhanced features for classifying payloads of spoofed traffic using libpcap\footnote{\url{https://www.tcpdump.org/}}.
\paragraph{Data sources}
Our traffic analysis is based on sampled flow data from a large regional Internet Exchange Point in Europe.
We consider two different time periods, February~19-25, 2018 and June~1-7,~2019. All results shown in this paper are based on the week in Februrary 2018, while we used the June 2019 data to verify the stability of our results.
To construct the cones and to identify unrouted prefixes, we utilize BGP data from all route collectors available in BGPStream~\cite{okggd-bsflh-16}, for the corresponding weeks as well as one day before and one day~after.
\section{Results}
\label{sec:results}
\begin{table}[b]
\caption{Comparison of the different classification results for anomalous
traffic. Significant differences highlighted in~gray.}
\label{tab::results}
\begin{tabular}{l l l l l}
\toprule
& \multicolumn{2}{c}{IMC 2017} & \multicolumn{2}{c}{This study}
\\ \cmidrule(r){2-3} \cmidrule(l){4-5}
& Bytes & Packets & Bytes & Packets
\\
\midrule
Bogon & 0.003\% & 0.02\% & 0.0009\% & 0.0022\%
\\
Unrouted & 0.004\% & 0.02\% & 0.00001\% & 0.0001\%
\\ \midrule
%\parbox[b][4mm]{2mm}{\multirow{2}{*}{\rotatebox[origin=c]{90}{Invalid}}}
Invalid
\\
\quad Naive & 1.1\% & 1.29\% & 0.579\% & 1.537\%
\\ %\midrule
\quad CAIDA & 0.19\% & 0.3\% & 0.955\% & 1.563\%
\\
\quad Full & \cellcolor{Gray}{0.0099\%} & \cellcolor{Gray}{0.03\%} & \cellcolor{Gray}{0.2\%} &
\cellcolor{Gray}{0.488\%}
\\
\bottomrule
\end{tabular}
\end{table}
We show a first glance of the overall results in Table~~\ref{tab::results}. Our results resemble those of IMC'17 for the amount of invalids based on the naive and CAIDA cones but diverge significantly for invalids based on the full cone and the classification of bogon and unrouted traffic.
\begin{figure*}
\begin{subfigure}{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{traffic_naive_2018-02.pdf}
\caption{Naive Approach}
\label{fig::traffic_naive}
\end{subfigure}\hfill
\begin{subfigure}{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{traffic_caida_2018-02.pdf}
\caption{CAIDA Customer Cone}
\label{fig::traffic_cc}
\end{subfigure}\hfill
\begin{subfigure}{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{traffic_full_2018-02.pdf}
\caption{Full Cone}
\label{fig::traffic_full}
\end{subfigure}\hfill
\caption{CCDF of the fractions of anomalous traffic per IXP member AS}
\label{fig::traffic}
\end{figure*}
\begin{figure*}
\begin{subfigure}[b]{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{sizes_naive_2018-02.pdf}
\caption{Naive Approach}
\label{fig::sizes_naive}
\end{subfigure}\hfill
\begin{subfigure}[b]{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{sizes_caida_2018-02.pdf}
\caption{CAIDA Customer Cone}
\label{fig::sizes_cc}
\end{subfigure}\hfill
\begin{subfigure}[b]{.33\textwidth}
\centering
\includegraphics[width=\linewidth]{sizes_full_2018-02.pdf}
\caption{Full Cone}
\label{fig::sizes_full}
\end{subfigure}\hfill
\caption{CDF of packets sizes by category}
\label{fig::sizes}
\end{figure*}
It is worth noting that the method of classifying bogon and unrouted traffic is well-known and not specific to the IMC'17 proposal. Observed divergences are not surprising but rather reflect different states of deployment in time and place. For example, as bogon traffic is a typical anomaly from misconfigured NATs, fluctuations between deployments are likely.
The decline of invalids in the full cone, however, do surprise. IMC'17 finds 20 times less invalid bytes after extending the CAIDA cone with additional peering links, whereas we see only a reduction to more than one fifth. Similarly, packet numbers drop for IMC'17 down to one tenth, while ours decrease by a factor of three.
The largely enhanced impact of the IMC'17 full cone leads to a notable reduction of traffic classified as invalid. Neither by adding (fairly effectless) multi-AS organisation extensions (see Section \ref{sec:methodology}), nor by varying input data sets, we could reproduce these strong effects.
In the remainder of this section, we will analyze the invalid traffic and the impact of cone construction in more detail.
\subsection{Discrepancies for Invalid Traffic}
Figure~\ref{fig::week} displays the underlying time series of classified traffic for the different cone types. Absolute numbers have been rescaled by a common factor to fit the IMC'17 range. Corresponding results from IMC'17, which are only available for the full cone approach, are indicated by the shaded regions.
It is clearly visible that the different, stepwise increasing of cones enfold a moderate impact on the traffic fraction that is classified as invalid. Packet numbers marked as invalid drop less than one order of magnitude when moving from the naive to the full cone approach. This is in contrast to the IMC'17 results, which show invalids of almost two orders of magnitude less than our results for the full cone.
We compare the fractions of anomalous traffic as contributed from the IXP members in Figure~\ref{fig::traffic}. A large group of IXP members issue only a very small portion of invalid packets---some of which disappearing for the full cone approach. Still no member has more than 1\% of its traffic classified as either bogon, unrouted, or invalid. This is in strong contrast to the results of IMC'17, where a few members emit close to 100 \% of their entire traffic to invalid, i.e., almost all traffic of these ASes is identified as spoofed.
Given these significant discrepancies between our reproduced results and IMC'17 for the fraction and the distribution of traffic classified as invalid, we question the correctness of classification by taking a closer look at the invalid packets.
First, we inspect the observed packet sizes per category. Considering that spoofed packets are often used for amplification attacks, a larger amount of small invalid packets would be in support of the classification results. Figure~\ref{fig::sizes} shows the packet size distributions.
All three approaches exhibit a similar distribution of regular packet sizes with most packets larger than 1200 bytes. In contrast, bogon and unrouted traffic is overwhelmingly made up of small packets. Invalid packets tend to be smaller but vary between the approaches. Still sizes of invalid packets show a wide distribution with significant portions of large packets. This again is in contrast to the IMC'17 results, which show a sharp cut-off for packet sizes larger than $\approx 900$~Bytes.
\subsection{Traffic Mix Reveals False Positives}
\begin{table*}
\caption{Traffic mix per protocol and destination port of invalid packets from the reproduced full cone}
\label{tab::traffic_mix}
\begin{tabular}{l c c c c c c c c}
\toprule
\multirow{2}{*}{ICMP}
& & & & & & & & total \\
& & & & & & & & 0.37~\% \\\midrule
\multirow{2}{*}{UDP}
& 53 & 123 & 161 & 443 & 19302 & ephemeral & other & total \\
& 1.18~\% & $< 0.1$~\% & 0.35~\% & 19.73~\% & 0.18~\% & 0.94~\% & 0.81~\% & 20.36~\%\\\midrule
\multirow{2}{*}{TCP}
& 80 & 443 & 27015 & 10100 & & ephemeral & other & total \\
& 3.50~\% & 62.29~\% & 0.00~\% & 0.00~\% & -- & 6.75~\% & 13.67~\% & 79.45~\% \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table*}[htb!]
\caption{False positive indicators in traffic of the reproduced full cone}
\label{tab::false_positve_idicators}
\begin{tabular}{ l r r r r r r}
\toprule
& SSL over TCP & HTTP response & ICMP echo reply & TCP ACK & malformed \\
\midrule
Naive Approach & 3.985\% & \cellcolor{Gray}{0.174\%} & 0.056\% & \cellcolor{Gray}{86.188\%} & 0.000\% \\
CAIDA Customer Cone & 4.166\% & 0.134\% & 0.070\% & 69.197\% & 0.000\% \\
CAIDA (multi-AS ext.) & 4.166\% & 0.134\% & \cellcolor{Gray}{0.081\%} & 80.148\% & 0.000\% \\
Full Cone & 6.395\% & 0.117\% & 0.043\% & 76.079\% & 0.001\% \\
Full (multi-AS ext.) & \cellcolor{Gray}{6.512\%} & 0.029\% & 0.044\% & 77.350\% & 0.001\% \\
\bottomrule
\end{tabular}
\end{table*}
We are now diving deeper into packet inspection of the traffic classified as invalid and want to understand its characteristics. Table~\ref{tab::traffic_mix} explores the traffic mix and lists the top destination port distributions of invalid UDP and TCP packets. We cannot equivalently compare to the IMC'17 results, as their traffic mix has not fully been disclosed.
Strikingly, we find the majority of invalid traffic to be HTTPS over TCP followed by Quick over UDP, and plain HTTP according to the transport ports. Typical amplification/reflection attack patterns such as DNS (UDP 53) and NTP (UDP 123) do not stand out in our data, even though they were reported to dominate in the IMC'17 results. On the overall, almost 80~\% of TCP traffic raises doubts about its spoofed nature. Rather, this strongly indicates that the traffic classified as invalid from our data set mainly consists of regular Web flows and hence has been classified erroneously based on the previously proposed methodology.
As a further attempt to assess the invalid traffic as classified from our full cone approach, we test for specific indicators of unspoofed traffic. We selected the five indicators (1) SSL over TCP, (2) HTTP responses, (3) ICMP echo replies, (4) TCP packets carrying ACKs, and (5) malformed packets (\eg transport port 0) as used by CAIDA. These indicators are not rigorous, but strong.
Injecting packets into existing TCP connections requires some guesswork and is not easily deployed on a large scale. TCP packets that carry HTTP responses or ACKs might be less likely to be spoofed. The presence of an encrypted channel only strengthens this assumption. Packets that can easily be dropped by the receiver and neither provoke a reply nor require action are not attractive for spoofing either. ICMP echo replies are an example for this category. Looking at the problem from the opposite direction, malformed packets that would disrupt communication are more likely to be spoofed than part of regular traffic. As part of this analysis we looked for packets that use port zero or the same destination and source address.
% TODO: Should we mention ACK storms here?
% We also used TCP ack packets as an indicator though, these could possibly be misused to create a TCP ack storm. However, this is only possible with a man in the middle attack~\cite{Hubballi18} and in this case the attacker is between the victim and the measuring point, so it is not possible to detect the fake packets with a cone-based approach.
Our findings are summarized in Table~\ref{tab::false_positve_idicators}. They clearly indicate that for any selected cone more than 80~\% of the packets carry a distinct characteristic of unspoofed traffic. We conclude that applying the method of spoofing detection at IXPs presented in \cite{lskrf-dcait-17} cannot be reproduced on a quantitative scale, and most likely leads to results dominated by false positives---and thus contradicts the original claims.
Pursuing further investigations and private communications with the authors, we tried to investigate the reasons for these dissappointing results. Aside from minor inaccuracies of the method such as \one inaccurate timing and updates as well as \two disregards of BGP withdraw messages in the public dump files, and \three inaccurate modeling of transitivity in the BGP routing graph~\cite{swscb-cuflsr-16}, the construction of the customer cones always suffers severely from the lack of visibility of BGP relations. This well-known problem cannot be resolved on a general level. Instead, the authors of IMC'17 decided to manually inspect the traffic flows and adjust peering relations accordingly (\cf Section~4.4, ``Missing AS Links:'' in~\cite{lskrf-dcait-17}). Such manual adjustments of data sets are understandable for the individual case under exploration. They discard, however, the presented method from future use in real deployment.
% We analyze the invalid traffic for all approaches and count how many of these indicators exist in a fraction of all invalid packets. We see that in all approaches a lot of indicators are visible. Table~\ref{tab::false_positve_idicators} shows false positives and packet healthy indicators for all approaches. For TCP and UDP port 0 indicates a greater value of the traffic being spoofed. In contrast, a larger value for the false positive indicators suggests spoofed traffic. According to this classifies the full cone with multi-AS extension least false positive and therefore most spoofed traffic.
\section{Conclusion}
\label{sec:conclusion}
%In the present study, we aimed at reproducing the results of methods for identifying spoofed traffic at IXPs presented at IMC'17 for different but equivalent data sets. Using both, scripts provided by the authors and an extended, independently written tool set, we could reproduce certain general properties of the paper, but failed to reproduce the core outcome of the method. Instead, on the basis of further analyses, we could derive strong indicators that the proposed method cannot be generalized beyond individually handcrafted examinations.
%Reproducibility taught us: in trying to repeat paper \textit{Detection, Classification, and Analysis of Inter-Domain Traffic with Spoofed Source IP Addresses}~\cite{lskrf-dcait-17} we found that the manual component of the methodology (i) represents the major challenge in terms of repeatability, and has (ii) a significant effect on the results; this highlights challenges in deploying approaches based on current methods in an automated fashion.
Following a purely algorithmic approach, we could not reproduce the results of IMC'17. Instead, the majority of packets identified as ``invalid'' appeared as false positives.
In trying to reproduce the paper \textit{Detection, Classification, and Analysis of Inter-Domain Traffic with Spoofed Source IP Addresses}~\cite{lskrf-dcait-17}, we found that the manual component of the methodology \one represents the major challenge in terms of reproducability, and has \two a significant effect on the results; this highlights challenges in deploying approaches based on current methods in an automated fashion.
%Accordingly, we find that the approach proposed at IMC'17 does not comply with common real-world deployment.
\subsection*{Acknowledgments}
We explicitly thank Franziska Lichtblau and Florian Streibelt for patiently answering our questions during this study.
Philipp Richter is gratefully acknowledged for concluding remarks.
We thank Alberto Dainotti and Alistair King for organizing the DUST workshop at CAIDA, which enabled discussions on the results in a wider scope.
Finally, we thank the reviewers of the ACM IMC 2019 Poster Track for their valuable feedback.
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{16}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showDOI \undefined \def \showDOI #1{#1}\fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[\protect\citeauthoryear{Bajpai, Brunstrom, Feldmann, Kellerer, Pras,
Schulzrinne, Smaragdakis, W{\"a}hlisch, and Wehrle}{Bajpai
et~al\mbox{.}}{2019}]%
{bbfkp-dbgre-19}
\bibfield{author}{\bibinfo{person}{Vaibhav Bajpai}, \bibinfo{person}{Anna
Brunstrom}, \bibinfo{person}{Anja Feldmann}, \bibinfo{person}{Wolfgang
Kellerer}, \bibinfo{person}{Aiko Pras}, \bibinfo{person}{Henning
Schulzrinne}, \bibinfo{person}{Georgios Smaragdakis},
\bibinfo{person}{Matthias W{\"a}hlisch}, {and} \bibinfo{person}{Klaus
Wehrle}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{The Dagstuhl Beginners Guide to Reproducibility
for Experimental Networking Research}}.
\newblock \bibinfo{journal}{\emph{ACM SIGCOMM Computer Communication Review}}
\bibinfo{volume}{49}, \bibinfo{number}{1} (\bibinfo{date}{January}
\bibinfo{year}{2019}), \bibinfo{pages}{24--30}.
\newblock
\newblock
\shownote{Editorial note.}
\bibitem[\protect\citeauthoryear{Baker and Savola}{Baker and Savola}{2004}]%
{RFC-3704}
\bibfield{author}{\bibinfo{person}{F. Baker} {and} \bibinfo{person}{P.
Savola}.} \bibinfo{year}{2004}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{Ingress Filtering for Multihomed
Networks}}}.
\newblock \bibinfo{type}{RFC} 3704. \bibinfo{institution}{IETF}.
\newblock
\bibitem[\protect\citeauthoryear{Cai, Heidemann, Krishnamurthy, and
Willinger}{Cai et~al\mbox{.}}{2010}]%
{chkw-tam-10}
\bibfield{author}{\bibinfo{person}{Xue Cai}, \bibinfo{person}{John Heidemann},
\bibinfo{person}{Balachander Krishnamurthy}, {and} \bibinfo{person}{Walter
Willinger}.} \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{{Towards an AS-to-Organization Map}}. In
\bibinfo{booktitle}{\emph{Proc. of the 10th ACM IMC}}.
\bibinfo{publisher}{ACM}, \bibinfo{address}{New York, NY, USA},
\bibinfo{pages}{199--205}.
\newblock
\bibitem[\protect\citeauthoryear{Cotton and Vegoda}{Cotton and Vegoda}{2010}]%
{RFC-5735}
\bibfield{author}{\bibinfo{person}{M. Cotton} {and} \bibinfo{person}{L.
Vegoda}.} \bibinfo{year}{2010}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{Special Use IPv4 Addresses}}}.
\newblock \bibinfo{type}{RFC} 5735. \bibinfo{institution}{IETF}.
\newblock
\bibitem[\protect\citeauthoryear{Ferguson and Senie}{Ferguson and
Senie}{2000}]%
{RFC-2827}
\bibfield{author}{\bibinfo{person}{P. Ferguson} {and} \bibinfo{person}{D.
Senie}.} \bibinfo{year}{2000}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{Network Ingress Filtering: Defeating
Denial of Service Attacks which employ IP Source Address Spoofing}}}.
\newblock \bibinfo{type}{RFC} 2827. \bibinfo{institution}{IETF}.
\newblock
\bibitem[\protect\citeauthoryear{Freedman, Foust, Greene, Maddison,
Robachevsky, Snijders, and Steffann}{Freedman et~al\mbox{.}}{2018}]%
{ffgmr-manrs-18}
\bibfield{author}{\bibinfo{person}{David Freedman}, \bibinfo{person}{Brian
Foust}, \bibinfo{person}{Barry Greene}, \bibinfo{person}{Ben Maddison},
\bibinfo{person}{Andrei Robachevsky}, \bibinfo{person}{Job Snijders}, {and}
\bibinfo{person}{Sander Steffann}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{Mutually Agreed Norms for Routing Security
(MANRS) Implementation Guide}}}.
\newblock \bibinfo{type}{RIPE Documents} ripe-706. \bibinfo{institution}{RIPE}.
\newblock
\urldef\tempurl%
\url{https://www.ripe.net/publications/docs/ripe-706}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Jonker, King, Krupp, Rossow, Sperotto, and
Dainotti}{Jonker et~al\mbox{.}}{2017}]%
{jkkrs-mtuam-17}
\bibfield{author}{\bibinfo{person}{Mattijs Jonker}, \bibinfo{person}{Alistair
King}, \bibinfo{person}{Johannes Krupp}, \bibinfo{person}{Christian Rossow},
\bibinfo{person}{Anna Sperotto}, {and} \bibinfo{person}{Alberto Dainotti}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{{Millions of Targets Under Attack: A Macroscopic
Characterization of the DoS Ecosystem}}. In \bibinfo{booktitle}{\emph{Proc.
of the 2017 Internet Measurement Conference}} \emph{(\bibinfo{series}{IMC
'17})}. \bibinfo{publisher}{ACM}, \bibinfo{address}{New York, NY, USA},
\bibinfo{pages}{100--113}.
\newblock
\bibitem[\protect\citeauthoryear{Lichtblau, Streibelt, Krüger, Richter, and
Feldmann}{Lichtblau et~al\mbox{.}}{2017}]%
{lskrf-dcait-17}
\bibfield{author}{\bibinfo{person}{Franziska Lichtblau},
\bibinfo{person}{Florian Streibelt}, \bibinfo{person}{Thorben Krüger},
\bibinfo{person}{Philipp Richter}, {and} \bibinfo{person}{Anja Feldmann}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{{Detection, Classification, and Analysis of
Inter-Domain Traffic with Spoofed Source IP Addresses}}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2017 Internet Measurement
Conference}} \emph{(\bibinfo{series}{IMC '17})}. \bibinfo{publisher}{ACM},
\bibinfo{address}{New York, NY, USA}, \bibinfo{pages}{86--99}.
\newblock
\bibitem[\protect\citeauthoryear{Lichtblau, Streibelt, Krüger, Richter, and
Feldmann}{Lichtblau et~al\mbox{.}}{2018}]%
{Lichtblau18}
\bibfield{author}{\bibinfo{person}{Franziska Lichtblau},
\bibinfo{person}{Florian Streibelt}, \bibinfo{person}{Thorben Krüger},
\bibinfo{person}{Philipp Richter}, {and} \bibinfo{person}{Anja Feldmann}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \bibinfo{title}{transitive closure cone}.
\newblock
\newblock
\urldef\tempurl%
\url{https://gitlab.inet.tu-berlin.de/thorben/transitive_closure_cone}
\showURL{%
\tempurl}
\newblock
\shownote{Accessed: 2019-08-28.}
\bibitem[\protect\citeauthoryear{Luckie, Huffaker, Dhamdhere, Giotsas, and
claffy}{Luckie et~al\mbox{.}}{2013}]%
{lhdgc-arccv-13}
\bibfield{author}{\bibinfo{person}{Matthew Luckie}, \bibinfo{person}{Bradley
Huffaker}, \bibinfo{person}{Amogh Dhamdhere}, \bibinfo{person}{Vasileios
Giotsas}, {and} \bibinfo{person}{kc claffy}.}
\bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{{AS Relationships, Customer Cones, and
Validation}}. In \bibinfo{booktitle}{\emph{Conference on Internet Measurement
Conference}} \emph{(\bibinfo{series}{IMC'13})}. \bibinfo{publisher}{ACM},
\bibinfo{address}{New York, NY, USA}, \bibinfo{pages}{243--256}.
\newblock
\bibitem[\protect\citeauthoryear{M{\"u}ller, Luckie, Huffaker, kc~claffy, and
Barcellos}{M{\"u}ller et~al\mbox{.}}{2019}]%
{mlhcb-cisti-19}
\bibfield{author}{\bibinfo{person}{Lucas M{\"u}ller}, \bibinfo{person}{Matthew
Luckie}, \bibinfo{person}{Bradley Huffaker}, \bibinfo{person}{kc claffy},
{and} \bibinfo{person}{Marinho Barcellos}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{Challenges in Inferring Spoofed Traffic at IXPs}}.
In \bibinfo{booktitle}{\emph{Proc. of ACM CoNEXT}}. \bibinfo{publisher}{ACM},
\bibinfo{address}{New York, NY, USA}.
\newblock
\newblock
\shownote{to be published.}
\bibitem[\protect\citeauthoryear{Orsini, King, Giordano, Giotsas, and
Dainotti}{Orsini et~al\mbox{.}}{2016}]%
{okggd-bsflh-16}
\bibfield{author}{\bibinfo{person}{Chiara Orsini}, \bibinfo{person}{Alistair
King}, \bibinfo{person}{Danilo Giordano}, \bibinfo{person}{Vasileios
Giotsas}, {and} \bibinfo{person}{Alberto Dainotti}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{{BGPStream: A Software Framework for Live and
Historical BGP Data Analysis}}. In \bibinfo{booktitle}{\emph{Proc. of the
2016 Internet Measurement Conference}} \emph{(\bibinfo{series}{IMC '16})}.
\bibinfo{publisher}{ACM}, \bibinfo{address}{New York, NY, USA},
\bibinfo{pages}{429--444}.
\newblock
\bibitem[\protect\citeauthoryear{Rekhter, Moskowitz, Karrenberg, de~Groot, and
Lear}{Rekhter et~al\mbox{.}}{1996}]%
{RFC-1918}
\bibfield{author}{\bibinfo{person}{Y. Rekhter}, \bibinfo{person}{B. Moskowitz},
\bibinfo{person}{D. Karrenberg}, \bibinfo{person}{G.~J. de Groot}, {and}
\bibinfo{person}{E. Lear}.} \bibinfo{year}{1996}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{Address Allocation for Private
Internets}}}.
\newblock \bibinfo{type}{RFC} 1918. \bibinfo{institution}{IETF}.
\newblock
\bibitem[\protect\citeauthoryear{Ryba, Orlinski, W{\"a}hlisch, Rossow, and
Schmidt}{Ryba et~al\mbox{.}}{2015}]%
{rowrs-adads-15}
\bibfield{author}{\bibinfo{person}{Fabrice~J. Ryba}, \bibinfo{person}{Matthew
Orlinski}, \bibinfo{person}{Matthias W{\"a}hlisch},
\bibinfo{person}{Christian Rossow}, {and} \bibinfo{person}{Thomas~C.
Schmidt}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{Amplification and DRDoS Attack Defense --
A Survey and New Perspectives}}}.
\newblock \bibinfo{type}{Technical Report} arXiv:1505.07892.
\bibinfo{institution}{Open Archive: arXiv.org}.
\newblock
\urldef\tempurl%
\url{http://arxiv.org/abs/1505.07892}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Schlamp, W{\"a}hlisch, Schmidt, Carle, and
Biersack}{Schlamp et~al\mbox{.}}{2016}]%
{swscb-cuflsr-16}
\bibfield{author}{\bibinfo{person}{Johann Schlamp}, \bibinfo{person}{Matthias
W{\"a}hlisch}, \bibinfo{person}{Thomas~C. Schmidt}, \bibinfo{person}{Georg
Carle}, {and} \bibinfo{person}{Ernst~W. Biersack}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{CAIR: Using Formal Languages to Study
Routing, Leaking, and Interception in BGP}}}.
\newblock \bibinfo{type}{Technical Report} arXiv:1605.00618.
\bibinfo{institution}{Open Archive: arXiv.org}.
\newblock
\urldef\tempurl%
\url{http://arxiv.org/abs/1605.00618}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Weil, Kuarsingh, Donley, Liljenstolpe, and
Azinger}{Weil et~al\mbox{.}}{2012}]%
{RFC-6598}
\bibfield{author}{\bibinfo{person}{J. Weil}, \bibinfo{person}{V. Kuarsingh},
\bibinfo{person}{C. Donley}, \bibinfo{person}{C. Liljenstolpe}, {and}
\bibinfo{person}{M. Azinger}.} \bibinfo{year}{2012}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{IANA-Reserved IPv4 Prefix for Shared
Address Space}}}.
\newblock \bibinfo{type}{RFC} 6598. \bibinfo{institution}{IETF}.
\newblock
\end{thebibliography}